COMPUTER SCIENCE, TECHNOLOGY AND APPLICATIONS
COMPUTER ANIMATION
No part of this digital document may be reproduced, stored in a retrieval system or transmitted in any form or by any means. The publisher has taken reasonable care in the preparation of this digital document, but makes no expressed or implied warranty of any kind and assumes no responsibility for any errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of information contained herein. This digital document is sold with the clear understanding that the publisher is not engaged in rendering legal, medical or any other professional services.
COMPUTER SCIENCE, TECHNOLOGY AND APPLICATIONS Additional books in this series can be found on Nova’s website at:
https://www.novapublishers.com/catalog/index.php?cPath=23_29&seriesp= Computer%20Science%2C%20Technology%20and%20Applications&sort=2a&page=1
Additional ebooks in this series can be found on Nova’s website at:
https://www.novapublishers.com/catalog/index.php?cPath=23_29&seriespe= Computer+Science%2C+Technology+and+Applications
COMPUTER SCIENCE, TECHNOLOGY AND APPLICATIONS
COMPUTER ANIMATION
JARON S. WRIGHT AND
LLOYD M. HUGHES EDITORS
Nova Science Publishers, Inc. New York
Copyright © 2010 by Nova Science Publishers, Inc.
All rights reserved. No part of this book may be reproduced, stored in a retrieval system or transmitted in any form or by any means: electronic, electrostatic, magnetic, tape, mechanical photocopying, recording or otherwise without the written permission of the Publisher. For permission to use material from this book please contact us: Telephone 6312317269; Fax 6312318175 Web Site: http://www.novapublishers.com NOTICE TO THE READER The Publisher has taken reasonable care in the preparation of this book, but makes no expressed or implied warranty of any kind and assumes no responsibility for any errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of information contained in this book. The Publisher shall not be liable for any special, consequential, or exemplary damages resulting, in whole or in part, from the readers’ use of, or reliance upon, this material. Any parts of this book based on government reports are so indicated and copyright is claimed for those parts to the extent applicable to compilations of such works. Independent verification should be sought for any data, advice or recommendations contained in this book. In addition, no responsibility is assumed by the publisher for any injury and/or damage to persons or property arising from any methods, products, instructions, ideas or otherwise contained in this publication. This publication is designed to provide accurate and authoritative information with regard to the subject matter covered herein. It is sold with the clear understanding that the Publisher is not engaged in rendering legal or any other professional services. If legal or any other expert assistance is required, the services of a competent person should be sought. FROM A DECLARATION OF PARTICIPANTS JOINTLY ADOPTED BY A COMMITTEE OF THE AMERICAN BAR ASSOCIATION AND A COMMITTEE OF PUBLISHERS. LIBRARY OF CONGRESS CATALOGINGINPUBLICATION DATA Computer animation / editors, Jaron S. Wright and Lloyd M. Hughes. p. cm. ISBN 9781612090788 (eBook)
Published by Nova Science Publishers, Inc.
New York
CONTENTS Preface
vii
Chapter 1
Computer Animation Applied to the Recovery of Preindustrial Heritage: A New Approach José Ignacio RojasSola and Francisco Javier ContrerasAnguita
Chapter 2
Virtual Engineering in Augmented Reality Pier Paolo Valentini, Eugenio Pezzuti and Davide Gattamelata
57
Chapter 3
A Survey of Popular 3D SoftBody Animation Compression Approaches S. Ramanathan and A.A. Kassim
85
Chapter 4
Virtual Emotion to Expression: A Comprehensive Dynamic Emotion Model to Facial Expression Generation Using the MPEG4 Standard Paula Rodrigues, Asla Sá and Luiz Velho
113
Chapter 5
ExampleBased PerformanceDriven Animation of an Anatomical Face Model Yu Zhang
129
Chapter 6
Dynamics for Managing Occlusion of Buildings in Panoramic Maps Neeharika Adabala
145
Chapter 7
ConstraintBased and FeatureBased CAD Systems and Applications Ioannis Fudos and Vasiliki Stamati
157
Chapter 8
Computer Aided Geometric Design with PowellSabin Splines Hendrik Speleers, Paul Dierckx and Stefan Vandewalle
177
Chapter 9
An Ontology of ComputerAided Design Udo Kannengiesser and John S. Gero
209
Index
1
235
PREFACE During the last decades, computeraided engineering (CAE) methodologies have deeply changed the way of designing and developing products, systems and services. Thanks also to significant hardware and software improvements, CAE techniques are widely used by designers from the early conceptual phases up to the final stages of engineering processes. At the industry level, these methodologies have become a fundamental tool to be competitive and to ensure high quality standards. In industrial engineering, computeraided methodologies typically are instrumental for design teams in shape modeling, behavioral simulations, digital mockups and realistic animations. They are able to follow the development of a product from conception to production, also managing its lifecycle. Character animation is one of the key research areas in computer graphics and multimedia. It has applications in many fields, ranging from entertainment, games, virtual presence and others. This new important book gathers the latest research from around the globe in this dynamic field. The heritage of the preindustrial period is today coming under examination more often, as engineering must accept the study of its evolution as a discipline, from a technical as well as a historical perspective. Engineering therefore provides industrial Archaeology and the history of technology with an important element in order to complete the study of industrial heritage. These studies are generally considered from the perspectives of history, ethnography, philology and architecture, but do not usually include studies from an engineering perspective. Chapter 1 provides a detailed examination of the infographic work carried out on a Manchegan windmill (La Mancha – Quixote), as an example of preindustrial heritage, in order to obtain a computer animation, so that the procedure followed can be extrapolated to other examples of preindustrial heritage. One of the reasons for choosing the windmill is that flour mills represented an important nucleus of the economy and of the industrial and social development of society. For this reason its study is important, especially for industrial history. The study and analysis of these windmills is especially important owing to their general state of abandonment and deterioration, including analysis of the techniques used in their construction and those used in the working of the windmill. Computer animation is a key element in the recovery of this interesting preindustrial heritage. In addition, the chapter discusses the advantages of this technique compared with others such as virtual reality, and why the majority of museum interpretation centres already possess these tools.
viii
Jaron S. Wright and Lloyd M. Hughes
CADCAE (ComputerAided Design/ComputerAided Engineering) techniques provide through computer animation a fundamental tool to present an integral study from the perspective of engineering of any example of preindustrial heritage. The importance of this chapter resides in that it presents in an innovative and structured way the procedure for generating a computer animation of preindustrial heritage. In Chapter 2 the authors discuss several approaches in order to integrate computeraided engineering instruments into Augmented Reality environment. Engineers and designers often develop their creative ideas in front of a computer monitor using mouse and keyboard. Although the integration between numerical computation and graphics leads to the generation of very realistic digital mockups, they are still far from the real context and the user has limited interaction with them. The purpose is to illustrate how recent development in computer graphics and image processing can improve the realism and interactivity with digital mockups. Starting from the interactive modeling of 3d shapes, the chapter presents some examples about the integration of realtime mechanism motion simulation, structural and fluid dynamics analysis postprocessing. In Chapter 3, the authors review 3D dynamic mesh compression algorithms and investigate how vertex clustering, which chiefly contributes to animation coding complexity, affects compression performance. The authors finally conclude this chapter with observations that need to be effectively addressed by future 3D animation coding algorithms. In Chapter 4 the authors present a framework for generating dynamic facial expressions synchronized with speech, rendered using a tridimensional realistic face. Dynamic facial expressions are those temporalbased facial expressions semantically related with emotions, speech and affective inputs that can modify a facial animation behavior. The framework is composed by an emotion model for speech virtual actors, named VeeM (Virtual emotiontoexpression Model), which is based on a revision of the emotional wheel of Plutchik model. The VeeM introduces the emotional hypercube concept in the R4 canonical space to combine pure emotions and create new derived emotions. The VeeM model implementation uses the MPEG4 face standard through a innovative tool named DynaFeX (Dynamic Facial eXpression). The DynaFeX is an authoring and player facial animation tool, where a speech processing is realized to allow the phoneme and viseme synchronization. The tool allows both the definition and refinement of emotions for each frame, or group of frames, as the facial animation edition using a highlevel approach based on animation scripts. The tool player controls the animation presentation synchronizing the speech and emotional features with the virtual character performance. Finally, DynaFeX is built over a tridimensional polygonal mesh, compliant with MPEG4 facial animation standard, what favors tool interoperability with other facial animation systems. Recent development of physicsbased face modeling that emulates the anatomical structure including skin, muscles, and skull allows us to create detailed, realistic animations. However, synthesis of facial expressions on such complex models often involves significant manual work due to the difficulty in determining appropriate values of the muscle actuation parameters. Chapter 5 presents an examplebased performancedriven method to automatically estimate facial muscle actuation parameters from markerless video footage. The authors method is based on an efficient face tracker which uses a facial deformation subspace model. During the training phase of the tracker a set of templates associated with the subspace basis is computed to alleviate the online computation. At runtime, the tracking algorithm establishes temporal correspondence of the face region in the video sequence by
Preface
ix
simultaneously determining both motion and appearance parameters. Using a set of example pairs that consist of the appearance and animation parameters corresponding to the key expressions, we learn the relationship between facial appearances and animation parameters. It enables the animation parameters to be computed in realtime from the appearance parameters obtained by the tracker, allowing animation of the anatomical model at interactive rates. Panoramic maps depict urban areas in oblique view. This form of cartography was prevalent from the late sixteenth century to the early nineteenth century, when there were not many skyscrapers in urban areas. But oblique view maps in the current urban scenarios suffer from loss of details due to occlusion among closely located multistory buildings. In Chapter 6 the authors leverage the time dimension to overcome the clutter in space dimension by introducing functional dynamics. The authors define a parameter called occlusion index for an urban scene at a given viewpoint. Solving the problem of occlusion involves devising methods for visualizing the urban scene that reduce/minimize the occlusion index. They explore occlusion reduction techniques that involve selecting optimal viewpoints, displacing buildings, making buildings transparent and changing building heights. The authors demonstrate these approaches by presenting screen shots of the solution applied to a prototype city block, and discuss the advantages and disadvantages of these solutions. This work is pioneering in its approach to applying animation in cartography, which has previously used animations only to depict timedependent phenomena or flythroughs. A new generation of Computer Aided Design systems has become available in which geometric constraints can be defined to determine properties of large designs. The new design concept, often called constraintbased design or design by features offers users the capability of easily defining and modifying a design, but introduces the problem of solving complicated, not always well defined, constraint problems. Traditional parametric models can also be enhanced to partially support declarative constraintbased descriptions. In Chapter 7 the authors provide an overview of representation schemes for CAD applications. Then they present a survey of methods for geometric constraint solving appropriate for Computer Aided Design. The authors demonstrate how these representations and constraint solving methods can be combined or adapted to support a broad range of CAD applications by presenting two example cases of successfully using a featurebased constraintbased representation scheme to support two different CAD applications. PowellSabin splines are bivariate C1continuous quadratic splines defined on an arbitrary triangulation. Their construction is based on a particular split of each triangle in the triangulation into six smaller triangles. In Chapter 8 the authors give an overview of the properties of PowellSabin splines in the context of computer aided geometric design. These splines can be represented in a compact normalized Bspline basis with an intuitive geometric interpretation involving control triangles. Using these triangles one can interactively change the shape of the splines in a predictable way. The authors describe the simple subdivision rules for PowellSabin splines, and discuss some applications. The authors consider a new efficient spline visualization technique based on subdivision. The authors also look at two useful generalizations of the PowellSabin splines, i.e., QHPS splines and NURPS surfaces. The QHPS splines are a hierarchical variant of PowellSabin splines. They have very similar properties as the PowellSabin splines, and their hierarchical nature allows a local refinement of the spline in a very straightforward way. The NURPS surface is the rational extension of
x
Jaron S. Wright and Lloyd M. Hughes
the PowellSabin spline. By means of weights they give extra degrees of freedom to the designer for the modelling of surfaces. Chapter 9 develops an ontology of computeraided design, based on the functionbehaviourstructure (FBS) ontology. It proposes two complementary views of the process of design. The objectcentred view applies the FBS ontology to the artefact being designed. Integrating an ontology of three “design worlds”, this view establishes a framework of designing as a set of transformations between the function, behaviour and structure of the design object, driven by interactions between the three design worlds. Building on this framework, the processcentred view applies the FBS ontology to the activities defined by the objectcentred view. This increases the level of detail and provides a more welldefined set of representations of these activities. The authors ontological framework can be used to provide a better understanding of the functionalities required of existing and future computeraided design support.
In: Computer Animation Editors: J.S. Wright and L.M. Hughes, pp. 156
ISBN: 9781607415596 © 2010 Nova Science Publishers, Inc.
Chapter 1
COMPUTER ANIMATION APPLIED TO THE RECOVERY OF PREINDUSTRIAL HERITAGE: A NEW APPROACH José Ignacio RojasSola* and Francisco Javier ContrerasAnguita University of Jaén, Department of Engineering Graphics, Design and Projects, Campus de las Lagunillas, s/n, Jaén 23071, Spain
Abstract The heritage of the preindustrial period is today coming under examination more often, as engineering must accept the study of its evolution as a discipline, from a technical as well as a historical perspective. Engineering therefore provides industrial Archaeology and the history of technology with an important element in order to complete the study of industrial heritage. These studies are generally considered from the perspectives of history, ethnography, philology and architecture, but do not usually include studies from an engineering perspective. This chapter provides a detailed examination of the infographic work carried out on a Manchegan windmill (La Mancha – Quixote), as an example of preindustrial heritage, in order to obtain a computer animation, so that the procedure followed can be extrapolated to other examples of preindustrial heritage. One of the reasons for choosing the windmill is that flour mills represented an important nucleus of the economy and of the industrial and social development of society. For this reason its study is important, especially for industrial history. The study and analysis of these windmills is especially important owing to their general state of abandonment and deterioration, including analysis of the techniques used in their construction and those used in the working of the windmill. Computer animation is a key element in the recovery of this interesting preindustrial heritage. In addition, the chapter discusses the advantages of this technique compared with others such as virtual reality, and why the majority of museum interpretation centres already possess these tools.
*
Email address:
[email protected] Tel: +34953212452; Fax: +34953212334; Corresponding author. Professor Dr. José Ignacio RojasSola. University of Jaén, Department of Engineering Graphics, Design and Projects, Campus de las Lagunillas, s/n, Jaén 23071. Spain.
2
José Ignacio RojasSola and Francisco Javier ContrerasAnguita CADCAE (ComputerAided Design/ComputerAided Engineering) techniques provide through computer animation a fundamental tool to present an integral study from the perspective of engineering of any example of preindustrial heritage. The importance of this chapter resides in that it presents in an innovative and structured way the procedure for generating a computer animation of preindustrial heritage.
Introduction A Google search for the term “computer animation” shows up 2,650,000 results, while a search for the term “heritage” gives 117,000,000 results. A search for both terms together gives 91,400 results (search carried out on the 22nd December 2008). This shows the increasing importance of heritage in any of its facets. This importance can also be seen in the numerous prestigious international congresses on the subject, such as “World Heritage in the Digital Age" (organized by UNESCO's World Heritage Centre) or VAST Conferences (International Symposium on Virtual Reality, Archaeology and Cultural Heritage). We must also consider the existence of highimpact journals, such as the Journal of Cultural Heritage (JCR) among others, the large number of websites dedicated to the issue [1], [2], or the European Union’s 7th Framework [3]. The UNESCO World Heritage [4] defines heritage as “our legacy from the past, what we live with today, and what we pass on to future generations”. In terms of the virtual heritage which concerns us here, researchers believe that it can serve to encourage people to visit the actual site, and can provide a complement to such a visit [5]; visitors can benefit from the changes and opportunities it offers [6]. Current trends in work on virtual heritage point to three different steps: complete 3D documentation, 3D representation (from historical reconstruction to visualization) and 3D publication (from immersive reality to augmented reality) [7]. Many applications have been developed which deal with historical sites or buildings, and in 2000 it was already forecast that in the following decade work would be centered on virtual industrial heritage [8]. Industrial heritage has a close relationship with Industrial Archaeology. Much has been written on this subject, defining it variously as the discovery, analysis, record and preservation of past industrial remains [9], the discovery, cataloguing and study of physical remnant of the industrial past, in order to learn about significant aspects of the world of work and technical and production processes [10], or the study of material culture and aspects linked to production, distribution and consumption, in the future and un connections with the past [11]. Today there are many examples of industrial heritage that are about to disappear, and in many cases in ruins. Many organizations are working to study and analyze these cases, some linked to industrial archaeology, such as TICCIH (The International Committee for the Conservation of Industrial Heritage) [12], AIA (Association for Industrial Archaeology  UK) [13], or linked to the history of technology, such as SHOT (Society for the History Of Technology  USA) [14], as well as branches of UNESCO which study the many aspects of heritage; architectural, industrial, cultural, ethnographic, to name but a few. The recovery of heritage is in many cases linked to the history of technology, as it is a fundamental element in the study of the technological evolution of any invention. Engineering Graphics, and more specifically infographic techniques, play an essential role in the study of the history of technology, given the universal character of graphic language, as is
Computer Animation Applied to the Recovery of Preindustrial Heritage
3
shown by the large number of articles in print which deal with graphic reconstructions of various inventions and devices [15  19]. However, in many cases the efforts of conservationists, archaeologists and restorers are not enough. In particular, the heritage provided by ruined buildings and constructions, whether architectural or industrial, is often lost owing to the interests of urban development or the lack of a renovation project which could give life to the area and bring opportunities for work. This loss is more clearly shown in the case of preindustrial heritage1, which has been part of production processes, not only because of the wideness of its scope, but also because older machines suffer greater deterioration when they are no longer used. On many occasions initiatives are in put in place to conserve examples of industrial heritage, for example Museums of Science and Technology, which are becoming increasingly frequent, as they are a way of safeguarding a form of culture linked to the socioeconomic development of a given area [20]. This example is all the more evident in the case of elements related to protoindustrialization (windmills, watermills, fulling mills, or oil presses, among others), as they date from the preindustrial period, and their age makes them more susceptible to deterioration and disappearance. The role of synthesis images in the conservation of industrial heritage has grown exponentially in recent years. They allow an area, building or object to be preserved and interpreted in ways otherwise impossible to imagine, using photographic techniques. The most important factor is however that when using virtual models, it is not necessary to disturb or modify the original item. There are also other advantages, stemming from the computer animation itself. Firstly there is a sociocultural objective in the conservation of the ‘collective historical memory’ of an area where a given type of heritage was prevalent, providing information on the evolution of that society. Secondly, there is a clear educational objective in showing details of an abandoned culture [21]. Thirdly, there is also technological interest, as the use of computer animation techniques and processes provides valued knowhow. A computergenerated image should be as faithful as a figure in a journal, although this is rarely possible [22]. In sum, whenever an element of a society’s heritage is lost, it also becomes impossible to study, analyze and value its impact on that society. This chapter presents a new approach to the use of computer animation techniques applied to an element of preindustrial heritage, the Manchegan windmill (La Mancha, Spain), which were built in the 16th century, and some of which remain in nearperfect condition today. The specific windmill under study is the ‘Sardinero’, one of the 10 which still stand in the area of Campo de Criptana (Ciudad Real, Spain); it has also been declared of special cultural interest by the Spanish Government. These famous windmills appear in the masterpiece of Spanish literature, Don Quixote, by Miguel de Cervantes.
The Windmill The windmill is one of the devices that has most been used over the centuries to obtain flour, an essential part of the human diet. A detailed study of their working mechanisms [2325] and
1
The following is applicable to both preindustrial and industrial heritage.
4
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
their design codes allows us to develop a computer model and conserve completely this example of heritage, leaving a legacy which can be studied by future generations.
Architecture Although there are various different types of windmills both in Spain and in other countries, the original architecture of a Manchegan windmill has three different floors. The ground floor (cuadra) is where the cereal was received and the canvas sails were stored. The first floor (camareta) is where the flour was packed into sacks, and the second floor (moledero) housed all the machinery necessary for milling the cereal [26]. The windmill had a cylindrical masonry tower about 8 m in height, capped with a conical cover (windmill cap) made of zinc, about 3.5 m in height. This rested on a ring on the top of the tower, which allowed this part of the windmill to turn to face the prevailing wind.
Working The way a Manchegan windmill worked can be explained using the following photographs, which were taken by the author. Figure 1 shows an exterior view of the ‘Sardinero’ windmill in Campo de Criptana. The photograph shows different functional elements of the windmill, such as the sails (which would be covered with canvas to increase their surface area), the windshaft and the windmill cap.
WIND SHAFT
WINDMILL CAP
SAILS
UPPER WINDOWS
Figure 1. View of the Windmill.
Computer Animation Applied to the Recovery of Preindustrial Heritage
5
Figure 2. Closeup view of the sails.
Figure 3. Closeup view of the join between the sails and the windshaft.
Each windmill has two rotation systems: a horizontal system, formed by the windshaft, the sails and other elements which will be described below; and a vertical system formed by the windmill cap and the tailpole. This vertical rotation system allowed the windmill cap to turn so that the sails faced the prevailing wind. This was done by the miller, who would use
6
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
the 12 small upper windows around the windmill to determine which way the wind was blowing. Figure 2 shows the geometry of the sails, which have become deformed over time. They measure some 16 m from tip to tip, and are formed by 2 central stocks, and each of the sails had the central stock, four long ways struts and 19 crossways struts, giving rigidity to the sail. Figure 3 shows the detail of the join between the sails and the windshaft, and the three struts which provide rigidity to the sails.
TAILPOLE
FIRST FLOOR WINDOW
ENTRANCE
STONE MARKERS
Figure 4. Front View of the Entrance.
Figure 5. Space for storing sail canvasses.
TRIPOD
Computer Animation Applied to the Recovery of Preindustrial Heritage
7
Figure 4 is a view of the front of the windmill, showing the only entrance to the ground floor (cuadra), the window of the first floor (camareta) which provided the only source of natural light, the tailpole, which allowed the windmill cap to turn into the wind, the tripod or support for the tailpole, and the 12 stone markers which marked the 12 possible positions of the tailpole.
Figure 6. Entrance to the ground floor, with the counterweight which was used to separate the milling stones.
Figure 7. Spiral staircase leading to the first floor.
8
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
In the entrance to the ground floor there was an area where the sail canvasses were stored (Figure 5), and where the counterweight hung (Figure 6). This counterweight could be pulled down manually in order to separate the two milling stones.
Figure 8. Beams (marranos) supporting the milling floor.
Figure 9. View of the flour channel where the flour was put into sacks.
Computer Animation Applied to the Recovery of Preindustrial Heritage
9
The spiral staircase (Figure 7) which leads up to the first floor runs alongside the two huge beams (marranos), which supported the second floor (milling floor) (Figure 8). On this floor there was a flour channel (Figure 9) through which the milled flour passed directly to be put into sacks. The mechanism which separated the milling stones (relief mechanism) (Figure 10) was located on the milling floor, and was activated by the counterweight shown in Figure 6. Figure 11 shows the runner and the bedstone, between which the cereal was milled. These stones were normally grooved to aid the milling process. The photograph also shows the outlet for flour which led to the flour channel shown in Figure 9.
MECHANISM FOR SEPARATING MILLING STONES
Figure 10. Stairway to milling floor.
RUNNER
BEDSTONE
FLOUR CHANNEL
Figure 11. View of the two milling stones and the outlet for flour.
10
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
Figure 12 shows gearing between the wallower and the brake wheel (fixed to the windshaft), which were the gear wheels which transmitted movement to the milling mechanism. The brake wheel had 40 cogs, and the wallower had 8 segments in which the cogs fitted, giving a gear ratio of 1:5 (8/40). Therefore, when the sails turned (normally at around 9 rpm), the brake wheel transmitted movement to the wallower, which in turn drove the rotation of the runner stone via the iron wallower axle. A further important element in the working mechanism was the brake rim, which was activated using a set of struts and a rope. Figure 13 shows the joint between the tailpole and the windmill cap, which were joined by a wooden block called the fraile. This linked the tailpole to the roof structure of the windmill, which was strengthened by wooden ribs. IRON WALLOWER AXLE
COG
BRAKE RIM
WALLOWER
BRAKE WHEEL
Figure 12. Detail of the gearing between the wallower and the brake wheel. WINDMILL CAP STRUCTURE
TAILPOLE
FRAILE
Figure 13. Detail of the joint between the tailpole and the windmill cap at the fraile.
Computer Animation Applied to the Recovery of Preindustrial Heritage
11
Figure 14 shows the hopper where the cereal was housed and the channel which fed the cereal into the central hole in the runner stone. Lastly, Figure 15 shows the roof structure (formed by perpendicular beams called madres and manzanos), which was the wooden structure on which the ribs of the roof section of the windmill rested, as well as the tailpole, which allowed the windmill cap to turn. It turned on a ring (rueda terrera), which was greased in order to avoid excessive friction. The photograph also shows the windshaft and one of the two stones on which it rested, the forestone (fuélliga), and the tailstone (rabote).
CHANNEL HOPPER CENTRAL HOLE IN THE RUNNER STONE
Figure 14. Hopper and channel feeding the hole in the runner stone.
ROOF STRUCTURE
WINDSHAFT
TAILSTONE
Figure 15. View of the roof structure, windshaft and tailstone.
12
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
Methodology Computer animation forms part of an innovative methodology for the conservation, diffusion and updating of industrial heritage [27]. In order to systemize the process of acquisition, classification, treatment and distribution of the available sources of information dealing with the object of study (plans, photographs, documents, slides, analogue recordings, hardcopy texts) and thereby both improve the conservation of this material and help to generate formats with higher added value, a working methodology was developed, as shown in Figure 16. UPDATING
EXECUTION
VERIFICATION
TARGET MATERIAL
Corrective actions
Figure 16. Diagram of the methodology proposed for the recovery and updating of industrial heritage.
The Updating stage is divided into three sections: • • •
Location, classification, nomenclature and storage. Digitalization of the original material. Classification in digital repositories.
The Execution stage has four steps: • • • •
Identification of the technical requirements. Definition of functional features. Workflows between applications. Analysis of the creation and publication processes.
The Verification stage has two parts: • •
Development of tests. Analysis and verification.
This methodology provides an organized sequence of procedures, structured in three stages. A computer animation forms part of the first stage of the procedure, digitalization of the original material, as it consists of creating a sequence of frames in sequence, which when played at an adequate speed forms a video animation. There are many programs for modeling, synthesizing images and computer animation, and a comparative study of them [28] shows the most well known characteristics of each. Two of the most outstanding are Autodesk 3ds MaxTM and Autodesk MayaTM. Although either of the two could have been chosen for this study, Autodesk 3ds Max was chosen owing to the need to create particles (grains of wheat and flour). One of the critical phases of this process was the generation of digital models of the object, in order to create realistic images from the original sources. The applications and processes used in this task are described in the following sections.
Computer Animation Applied to the Recovery of Preindustrial Heritage
13
The work process followed these steps: 1. General outline of the virtual recreation of the ‘Sardinero’ windmill 2. Creation of CAD model with AutoCAD and import to Autodesk 3ds Max 2.1. Fieldwork 2.2. Modeling 2.2.1. From AutoCAD, by exporting .3ds files 2.2.2. From Autodesk 3ds Max, by importing .dwg files 3. Cameras and illumination 3.1. Camera movement. Creation of path 3.2. Illumination 4. Animation of working parts 4.1. Runner Stone raising mechanism 4.2. Brake rim mechanism 5. Materials and maps. Mapping coordinates 6. Creation of textures 7. Rendering and video creation 8. Postproduction
Development 1. General Outline of the Virtual Recreation of the ‘Sardinero’ Windmill It is recommendable, when working with industrial heritage, to perform two sequences or videos: a virtual ‘static’ view, which shows the object and its surroundings, and a second, ‘dynamic’ view, showing the working of the object, following the logical order of the productive process. This is how the work has been carried out in the case of the ‘Sardinero’ windmill studied here, establishing a playback speed of PAL frames of 25 frames per second. Given the nature of this work we decided to create a single file in Autodesk 3ds Max which included both sequences (static and dynamic) so as not to have to make adjustments in texture and illumination in various files. Lastly, we chose .avi as the file format, defined by Windows as its Video for Windows technology, as it is a format which is compatible with most video players. The file was created from the frames rendered individually in .png format.
2. Creation of CAD Model with AutoCAD and Import to Autodesk 3ds Max Although there are many existing procedures to digitalize industrial heritage objects in 3D [29], such as Empirical techniques, Topographic techniques, Laser scanning techniques or Photogrammetry, we have used empirical techniques, owing to their ease of use, their transferability and to the fact that precision measurement was not a determining factor. In addition, the geometry is relatively simple, with a cylindrical tower which could easily be modeled using CAD techniques, and from the perspective of engineering graphics this technique allows us to obtain all types of views, perspectives and sections of the windmill. This in turn allows us to make comparisons with other forms.
14
José Ignacio RojasSola and Francisco Javier ContrerasAnguita Two examples of the plans obtained are shown in Figures 17 and 18.
Figure 17. Section Perspective of the windmill modeled in 3D.
Figure 18. Exploded view of the horizontal rotation system of the windmill.
Computer Animation Applied to the Recovery of Preindustrial Heritage
15
The development of the empirical approach applied to the ‘Sardinero’ windmill is based on two fundamental previous sections: fieldwork and graphic reconstruction.
2.1. Fieldwork The fieldwork necessary for the project includes both taking photographs and drawing sketches of the building and its mechanisms. The quality of the computer animation depends on that of the photographs, as the texture captured from them is applied to the model in order to provide a high degree of realism in the final video.
Figure 19. Transition from sketch to 3D CAD model.
We used a Nikon D200 digital camera to take the photographs, with an ISO setting of 800. This allowed us to obtain clear images, and we took around 500 photographs of the
16
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
exterior and of the three floors of the windmill. The windmill was measured using a 10 m tape measure in order to draw the sketches. The inside areas and mechanisms of the windmill were measured using an engineer’s scale. As the windmill is not a precisionbuilt construction, various references were taken and measurements had to be adjusted. The precision of the final model depends on the accuracy of the sketches and measurements taken. It is also necessary to bear in mind that the geometrical data obtained also allow us to infer certain technological considerations, and to make possible comparisons with other types of windmills.
2.2. Modeling After sketching, the next step is modeling, which is necessary in the graphic reconstruction to plan the virtual tour of the windmill. From the perspective of engineering, modeling is a powerful tool which allows for an accurate study of each part of the machinery, as well as giving an overall idea of how the different parts of the mechanism worked together. Some of the measurements taken ‘in situ’ were not completely accurate, and so further measurements had to be taken in order to shed light on certain assembly details which were not totally clear. The program used for modeling was AutoCAD. To obtain a model which is a faithful as possible to the original is a complex task, as there are often limiting factors, such as the measurement of certain elements which cannot be measured by hand, and other techniques have to be used. In this way the CAD model is obtained from the handdrawn sketches (Figure 19). Given that AutoCAD and Autodesk 3ds Max were developed by the same company, it is easy to exchange information between them; for example, a point with coordinates x, y, z in AutoCAD corresponds exactly to another with coordinates u, v, w in Autodesk 3ds Max. This is an added advantage, because although there are neutral exchange files such as IGES, STEP or VDAS, these files sometimes produce a loss of information. This exchange of information can be made in two ways:
2.2.1. From AutoCAD, by Exporting .3ds Files Using this method it is possible export all the parts which are necessary, avoiding the need to debug later all the nonnecessary elements, but they must by solids surfaces, lines, 3D polylines or 3D faces, among others. However, there are some disadvantages: Sometimes when working with complex geometry AutoCAD cannot obtain the 3ds file format. In order to avoid in Autodesk 3ds Max curved surfaces which have a multisided appearance, it is necessary to increase a variable in AutoCAD (facetrees) from a default value of 0.5 to a value of 10, which causes a notable slowing of the program. For these reasons, we chose the second option:
2.2.2. From Autodesk 3ds Max, by Importing .dwg Files The model created in AutoCAD can be imported directly with the extension .dwg, although it is necessary beforehand to configure the .max receiver file in Autodesk 3ds Max with a series of options such as the measurement units, considerations about AutoCAD primitives, geometry, layering and rendering options of the splines.
Computer Animation Applied to the Recovery of Preindustrial Heritage
17
Once the model has been imported, the screen is divided into four windows, called graphic windows (Figure 20), in order to create the sequences from different angles and perspectives. The active window is marked with a thick grey line, and all options can be accessed from a contextual menu using the right mouse button.
Figure 20. Working Screen in Autodesk 3ds Max.
3. Cameras and Illumination Cameras can be used to obtain personalized views of a scene much in the same way as with real cameras. Here, they need lens adjustments which are measured in millimeters. Autodesk 3ds Max has two types of cameras: Target and Free. The first is centered on the given object and the area around it, giving an independent animation of the object, while the free type simple records a scene in the direction in which it is pointing, without being linked to a specific object. In the case of the ‘Sardinero’ windmill, we used target cameras to obtain general plans of the exterior views, and of the first and second floors. Free cameras were used to focus in specific elements or movements, for example the counterweight relief mechanism used to life the runner stone. Although Autodesk 3ds Max provides a wide variety of groups of lenses, from 35 mm to 200 mm, in the windmill cameras with a focal distance of 24.29 mm were used, which is the default setting, in order to obtain wide angle views of the scene. As well as the lens, it is necessary to adjust the field of vision (FOV) which is measured in degrees (Figure 21). This
18
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
is linked directly to the focal length and measures the visible part of the scene. In the case of the default focal distance, the program adjusts the value directly to 45º.
Figure 21. Camera with adjusted FOV.
3.1. Camera Movement. Creation of Path Although a free camera is usually the better option if it is in movement and a target camera is more useful in situations where the camera does not move, we chose to use a target camera to produce the static video of the windmill (that is, a virtual visit where the windmill is not working), animating both the ‘body’ of the camera and the objective. It is very important to maintain a constant and appropriate speed during the path of the camera, and therefore movement constraints were used, which link objects to others or to the path of the camera. Autodesk 3ds Max offers different constraints, such as: • • • • • • •
Attachment constraint Surface constraint Path constraint Position constraint Link constraint LookAt constraint Orientation constraint
Computer Animation Applied to the Recovery of Preindustrial Heritage
Figure 22. Path of the camera following a spline curve.
Figure 23. Dummy linked to the camera lens and moved through the scene.
19
20
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
In this case, the path constraint has been used, so that the camera follows a spline curve previously created in Autodesk 3ds Max (Figure 22). Using the appropriate commands, the following path was created, which is the path of the camera outside and inside the ‘Sardinero’ windmill. Once the path has been generated, this has to be assigned to the ‘body’ of the camera. Although this can be done directly, in this case it was done indirectly, by assigning the path to a false object (Dummy) and then linking the camera to this object. This allows us to create camera travelling, at the same time as the camera uniformly followed its path; this is very useful to position objects and measure dimensions. A simple cube was used as the Dummy, with a pivot point in the centre, which was not rendered and which had no parameters. The link was then made between objects lower and higher in the kinematic chain. Finally, camera lens was animated independently of the ‘body’ of the camera using key frames. A helper was also linked, and moved through the scene by movement transformations, which does not imply any changes in the geometry of the object, but rather a modification of its initial state. The following figure shows the situation of the camera which travels the path through the windmill.
3.2. Illumination Illumination is the most intricate and complex part of the creation of any scene, as it forms the basis of the work carried out with textures and materials, and also determines to a large extent the rendering. In cases of complex examples of industrial heritage such as a windmill, simplicity should be a key factor, in order to find an optimum balance between rendering time and the quality of the result. The configuration of the materials in the scene will also be a conditioning factor. In this example, we have used different types of lights from Autodesk 3ds Max, including the Daylight system to simulate natural sunlight (Figure 24) with its various options, in which the software simulates the position of the sun at a specific time and date, and from a specific direction. It is also necessary to activate shadows by selecting the ray traced type (Figure 25), which are very accurate, as AutoDesk 3ds Max calculates the shadows according to each ray of light which enters the scene. In addition, we activated the option which determines the transition between bright areas and areas without illumination, and the exponential attenuation of light with distance. The other values are default values, and the color and intensity of the light are according to the geographical location selected earlier. It is also necessary to include fill light which does not generate shadows in areas where the principal light does not provide illumination. This fill light projects light from a defined area rather than from a single point, and with a lower intensity than that of the principal light. In the windmill these lights have been placed in each of the small upper windows (Figure 26).
Computer Animation Applied to the Recovery of Preindustrial Heritage
Figure 24. Simulation of sunlight.
Figure 25. Selection of Ray Traced Shadows.
21
22
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
Figure 26. Fill light situated in the upper windows.
4. Animation of Working Parts The following working subgroups were studied and animated: 1. Runner Stone raising mechanism. Inverse kinematics was used, which allow the designation of the movements of objects higher in the kinematics chain through the movement of the objects lower in the kinematics chain. 2. Brake rim mechanism. Here, inverse kinematics was also used, as well as Free Form Deformation (FFD), which allows elastic deformations of objects. 3. Creation and animation of ropes. Here the Reactor module has been used, and so once the approximate forms have been created, gravity is applied, giving a realistic curve. 4. Brake wheel–wallower. In this case we have used forward kinematics, that is, to determine the movements of the objects lower in the kinematics chain acting on the objects higher in the kinematics chain. 5. Obtaining flour from grains of wheat. This operation was carried out in two phases: in the first a Reactor module was used to achieve the effect of the grains of wheat, stored in the hopper, falling into the channel through an opening and from there falling to the milling stones. In the second phase, the particle systems were introduced, which are elements which generate groups of objects called particles, which behave as a single unit, and which allow the creation of realtime simulations of natural phenomena such as rain, dust and snow, among others.
Computer Animation Applied to the Recovery of Preindustrial Heritage
23
6. Movement of the sail canvases. Here an independent simulation system called Cloth is used, which allows the creation and animation of deformable material. Autodesk 3ds Max also has a specific modifier called Garment Maker, which transforms geometric primitives into material patterns. As it would take up too much space to give a detailed explanation of each of these subgroups, we have chosen two as examples: the runner stone raising mechanism and the brake rim mechanism.
4.1. Runner Stone Raising Mechanism In order to animate this mechanism which separates the milling stones, we used inverse kinematics, which allows us to determine the movements of objects higher in the kinematics chain by controlling the objects lower in kinematics chain. This is more effective than forward kinematics. Autodesk 3ds Max includes various methods to animate using inverse kinematics, such as IK Solvers, and traditional methods, Interactive IK and Applied IK. IK solvers are helpers which apply inverse kinematics to systems of linked objects. For example, there is a HistoryDependent solver which is recommended in mechanical systems with sliding joints in inverse kinematics, as it has controls for damping, priority, and spring back.
Figure 27. Elements in the scene.
24
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
Figure 28. Control Element and actions on the other elements.
Interactive IK allows the positioning of a hierarchy linked to objects in different frames, and Autodesk 3ds Max interpolates all the key frames. This is not an accurate method, although it uses a minimum number of keys. Finally, applied IK is a method which applies a solution in a range of frames, calculating the keys in each frame; it is more accurate that Interactive IK, although it creates a large number of key frames. In this case we used inverse kinematics applying both methods. Once the necessary links between the objects had been established using Interactive IK, their behavior was observed, and the animation was carried out using Applied IK. To animate the movement of the separation mechanism of the milling stones, the raising mechanism of the runner stone, the elements have to be renamed, as when the AutoCAD model is imported into Autodesk 3ds Max, a predetermined name is given to all elements in the scene, and these names are not clear when there are many elements. Therefore, it is necessary to redesignate all the elements (Figure 27). Then, the control element for inverse kinematics is established. This is the element which will be animated manually, to be used as the basis for the animation (Figure 28). The next step is to determine the links between the control element and the other elements. This is the most difficult and timeconsuming step, as it is necessary to use helpers and to relocate the pivot points of some objects. We added 6 helpers, to allow for the interconnection between all the elements and the combination of movements of some of them, for example in the runner stone which must turn and rise at the same time.
Computer Animation Applied to the Recovery of Preindustrial Heritage
25
Figure 29. Positions of helpers.
Figure 29 shows the functions of the helpers in the mechanism: Helper 01: Linked to the runner stone (including rings), the wallower (including rings and cogs) and to the iron wallower axle, which is the object higher in the kinematics chain. It allows the transmission of circular movement to these elements. Helper 02: Linked in the same way as helper 01, and transmits higher and lower movement hierarchy to this set of elements. Helper 03: Linked to the leverbeam joint, acting as a link between this and the other elements. Helper 04: Linked to the exterior raising beam, allowing for two pivot points on this element. Helper 05: Linked to the exterior raising beam, at the point where this joins the interior raising beam. Its function is to connect these two beams. Helper 06: This helper is lower on the hierarchy than the interior raising beam and its function is to create two pivot points and also to connect the interior and exterior raising beams. In addition, it is necessary to link two elements which are already joined, helper 04 and helper 03. In the same way, helper 06 is linked to helper 05, and helper 02 to helper 05. It is then necessary to define the constraints of the joints of each element, as each has six degrees of movement: rotation and movement along the X, Y and Z axes. The Rotational Joints and Sliding Joints options were used (Figure 30).
26
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
Each joint has three sections referring to each of the three axes, and if the Active option is deselected, that axis is constrained; that is, if the active option of the x axis of the interior raising beam is deselected in the rotational joints window, the element cannot turn on this axis. In the same way, if the same option is deselected in the sliding joints window, the joint cannot slide along this axis. This is how the joints are defined for the rest of the elements of the mechanism.
Figure 30. Rotational & Sliding Joints.
Computer Animation Applied to the Recovery of Preindustrial Heritage
27
The rings of the runner stone, the wallower, and the screw and bolts which form part of the mechanism do not have defined joints, as they are linked solidly to elements which already have these joints defined. Once the links and joints have been established (rotational and sliding) interactive IK is used to check that the elements of the mechanism move correctly. The button select and rotate shows another transformation of the three available in Autodesk 3ds Max, which does not imply any change in the geometry of the object, but rather a modification of its initial state (Figure 31). It can be seen that from a certain angle of the control element, the joints do not function as in real life; specifically, from 30º, the elements begin to intersect with one another. To solve this problem, other animation tools can be used such the Reactor plugin, which allows the creation of key frames when objects interact according to the laws of physics.
Figure 31. Inverse Kinematics and select and rotate buttons.
However, it would be timeconsuming to configure the scene using the Reactor module, and given that the runner stone has a movement of approximately 1 cm, for which the angle at which the control element turned was not more than 6º, this simulation is unnecessary. Therefore it is only necessary to apply the inverse kinematic solution using applied IK, which can be applied to any range of frames. The required animation is therefore obtained, as the program calculates the key frames for the other elements according to the control element and the links established.
28
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
4.2. Brake Rim Mechanism Inverse Kinematics has also been used to simulate the mechanism which brakes the brake wheel. In addition, a Free Form Deformation (FFD) modifier has been used to achieve the elastic deformation of the brake rim (Figure 32). In this case the elements which are present in the animation are the linking beam, counterweight beam, hook joint with windmill cap, counterweightlinking beam joint, linking beamcounterweight joint, pin flange, pin and bolt (Figure 33). The ring and the rim itself will be animated once the movement of the other parts has been determined. The control element is the linking beam, which is also the reallife control element (Figure 34). The links are then made between the control element and the other elements. A helper has been added, not to link objects, but to make possible the presence of two pivot points on the counterweight beam. This beam is then established as higher in the kinematics chain than the pin, pin flange, bolt, hook joint with windmill cap, counterweightlinking beam joint, as well as the helper, and lastly, the control element is designated as higher in the kinematics chain than the linking beamcounterweight joint. The counterweightlinking beam joint is linked to the linking beamcounterweight joint; specifically, the counterweightlinking beam joint is made to follow the linking beamcounterweight joint, and finally, the constraints of the joints are defined in these elements in the same way as before, using the Rotational Joints and Sliding Joints options.
Figure 32. Brake rim mechanism.
Computer Animation Applied to the Recovery of Preindustrial Heritage
29
Figure 33. Elements of the brake rim mechanism.
As before, correct movement is checked using interactive inverse kinematics, turning the control element with respect to its y axis, and observing the behavior of the other elements. Applied IK is then used as it can be applied to a given range of frames. Once the animation of the beams has been completed, the brake rim and its metal ring are animated. For this, the FFD modifier is used, as it can model rounded deformations without arrises, adjusting the control points of a lattice.
30
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
Figure 34. Control element and actions on the other elements.
Figure 35. FFD modifer button.
Computer Animation Applied to the Recovery of Preindustrial Heritage
31
Figure 36. Cylindrical Geometry of the FFD modifier, and surface adjustment button.
Figure 37. Modifier adjusted to fit the geometry of the rim.
The selection which is closest to the geometry of this example is cylinder type, FFd (cyl). The geometry of the modifier is then situated in the desired location (Figure 36).
32
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
Figure 38. Control points selected two by two.
Figure 39. Effect of the modifier on the geometry after moving the control points.
Computer Animation Applied to the Recovery of Preindustrial Heritage
33
The lattice is then selected and its resolution and size is configured, so that it coincides as much as possible with the brake rim and the ring. The higher the resolution, the better the results in the elastic deformation of the object, but more time is required. In this case, the resolution of the lattice was set at 42 control points, giving a perfect fit with the geometry of the rim, and also allowing us to animate it quickly. Figure 37 shows the geometry of the FFD modifier (cyl) after making these adjustments. The brake rim and its ring are then linked to the animation, similarly to the way objects are linked using the command Select and Link. Finally, the lattice control points are animated using key frames, defining an initial and final state using movement transformation. Therefore, the control points should be selected two by two (Figure 38) in order to achieve a good result, and to ensure that the elastic deformation of the rim coincides with the movement of the beams. Figure 39 shows the effect of the FFD modifier on the geometry of the brake rim after moving the control points vertically upwards.
5. Materials and Maps. Mapping Coordinates Autodesk 3ds Max uses materials to cover objects to imitate the effect of light on these objects. Maps or textures are elements which are applied to materials in order to achieve a realistic appearance, using mapping coordinates which are defined as how the objects are aligned using threedimensional coordinates u, v and w.
Figure 40. Material selection window in Autodesk 3ds Max.
34
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
Autodesk 3ds Max has various types of materials, and their choice depends to a great extent on the rendering motor used and the type of illumination, among other factors. In the case of our windmill we have used standard materials to give realism to the animation, which are found in the Material Editor (Figure 40). Below is a description of the process used for the ‘dustcover’, a piece of wood which covered the milling stones. In many cases it is necessary to create a new material because it does not exist in the library contained in the software. In our case, as the element which is to be textured is formed by a series of wooden staves, a material was created for each stave with a similar texture, so that the final appearance of the surface of the element does not have repeated patterns. It is also necessary to define the shader to be used, as this is the algorithm which calculates the appearance of the material according to the specified parameters. In this case the Blinn shader has been used (the default setting), as it renders simple circular projections and softens adjacent surfaces. This shader has color panels to configure the Ambient, Diffuse and Specular colors, which determine the appearance of the final color of an object. However, we used maps and textures taken from images of the original model in the texturizing process, using the Diffuse component through another shader. It is also possible to obtain better results by configuring some aspects of the indirect illumination in the rendering motor, although this takes more time. This decision depends ultimately on the designer. It is then necessary to define the mapping coordinates in the element to be texturized. We have used the UVW Map modifier, with a projection gizmo, which defines how the map will be projected onto the surface and how the material will be applied (Figure 41).
Figure 41. Projection Gizmo and material assignment.
Computer Animation Applied to the Recovery of Preindustrial Heritage
35
Figure 42. Adjustment of dimensions of gizmo to those of the image.
Figure 43. Final Result of Mapping.
However, the visualization of the texture on the surface of the element it not correct, as the adaptation of the gizmo to its geometry implies uneven steps, and therefore when the
36
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
texture is applied it seems stretched or compacted. Final adjustments have to be made to ensure that the dimensions of the gizmo are proportional to those of the image (Figure 42). Figure 43 shows the final result, which is very similar to real life.
6. Creation of Textures Maps or textures are applied to the materials to obtain a realistic effect. In our case the textures (Figure 44) are taken from digital photographs taken of the real object and edited with Adobe PhotoshopTM.
Figure 44. Texture taken from digital photograph.
. Figure 45. Continued on next page.
Computer Animation Applied to the Recovery of Preindustrial Heritage
37
Figure 45. Exterior ground of the windmill processed by Adobe Photoshop.
Figure 46. Wooden door from which repetition texture is extracted.
The digital model was divided into a number of pixels with color and intensity, giving images which Adobe Photoshop can work with. These textures can be repeated indefinitely without the sensation of a repeated pattern. This is shown in Figure 45 applied to the ground of the exterior of the windmill. The upper part of the image has not been processed with
38
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
Adobe Photoshop and edges of the images do not match; however, the lower part of the image shows how this problem is solved.
Figure 47. Image with corrected perspective and area to be cut selected.
Figure 48. Rendered image with texture applied.
Computer Animation Applied to the Recovery of Preindustrial Heritage
39
The first step when using Adobe Photoshop is to select the color mode RGB (RedGreenBlue) and the 8 bits /channel option, which is a standard mode found in televisions and color monitors. Files used should be saved in format .psd. The first step is extract the area of the photograph (texture) which is to be applied, using the lens correction and cut tools. The example shows the process used for the wooden door of a food store (Figure 46). The lens correction tool is used to correct the perspective, eliminating the divergence of parallel line, and the cut tool is then used to extract the desired area of the photograph (Figure 47). As digital photographs are already illuminated, it is necessary to adjust the brightness in order to ensure that the textures extracted are not excessively bright. The brightness has therefore been reduced by between 10% and 20% in photographs taken inside the windmill, and between 25% and 30% in photographs taken outside. Once this has been done, the image is cut and adjusted, obtaining a texture which can be repeated on the model without clear edges. This is undoubtedly one of the most complex stages of the work, to give a texture which has a high degree of realism (Figure 48).
7. Rendering and Video Creation Rendering is a process which calculates the properties of objects before they are shown on screen, that is, it generates a synthesis of the scene created. Autodesk 3ds Max has a rendering motor called Mental Ray and an additional plugin called VRay, which give excellent results thanks to the representation of light through rays. Rendering is done in the active window, which is marked by a thick border (Figure 49).
Figure 49. Active window where the scene is rendered.
40
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
Figure 50. Render process window.
Figure 51. Video Post process window.
Computer Animation Applied to the Recovery of Preindustrial Heritage
41
The Rendering dropdown menu includes the Render command where the configuration is done, defining the rendering motor, the range of frames to render, the format and size of the images, among other factors. We chose the format .png (Portable Network Graphics), with a color configuration RGB 26 bits (16.7 million), with alpha channel and interlinking activated, which is one of the best image formats for computer animation. The resolution of the image is 768x576 pixels, with a width to height ratio of 4:3, although the final video format is a DVD. Once the configuration has been done, the rendering process itself starts, and a dialogue box shows the adjustments made and the progress of the process (Figure 50). Before rendering the final image is configured, defining the range of color and the output levels of the final image. This stage is very important, as it controls the clarity of the colors of the scene (not the illumination), the intensity of the tones, the intensity of the standard lights, and adjusts the colors so that they correspond to an exterior scene. Once the rendering process is complete, the frames are linked together using the Video Post command, which allows the inclusion of many effects; this command is also found in the Rendering menu. The first operation to obtain video is to include the rendered images to make up the list of images. Then the output format AVI (Audio Video Interleave) is set, as it a simple and standard digital video format, and a compression codec is chosen to reduce the size of the final video so that it is more manageable. The resolution of the final video file is also set, in this case PAL 768x576 pixels. We obtained a video of 720 MB for the static sequence (a virtual tour with the windmill not working) and a video of 8.41 GB for the dynamic sequence (virtual tour with of the working windmill).
42
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
The sequence is executed and the following window appears (Figure 51). The following 25 images obtained using rendering show the degree of realism obtained in the computer animation process.
Computer Animation Applied to the Recovery of Preindustrial Heritage
43
44
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
Computer Animation Applied to the Recovery of Preindustrial Heritage
45
46
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
Computer Animation Applied to the Recovery of Preindustrial Heritage
47
48
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
Computer Animation Applied to the Recovery of Preindustrial Heritage
49
50
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
Computer Animation Applied to the Recovery of Preindustrial Heritage
51
52
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
Computer Animation Applied to the Recovery of Preindustrial Heritage
53
54
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
8. Postproduction The final stage in any Project in the graphic conservation of industrial heritage is the post production or editing of the video, to create an audiovisual document combining effectively audio, video and text. The object is to give a clear idea of the heritage and its setting in the production process. This stage has been carried out with Adobe Premiere, which is a very versatile and intuitive program with a wide range of video, audio and transition effects. The final video has the configuration DVPAL with the Standard 48 kHz, allowing it to be shown anywhere using any equipment, and with options which allow the user to personalize the video. First, the sound and video files are loaded, and the title and subtitles are created using a text editor. These are then added to the timeline, to set the order and timing of each element, and the transitions between videos are established. Lastly, the video is exported with the necessary settings.
Conclusion This project shows that to create a realistic computer animation requires a great deal of time and effort. This process is usually carried out by a team of designers equipped with powerful computers with various highspeed microprocessors, sufficient RAM and graphic cards with large amounts of memory. In our case, two people and three computers took almost one year to complete the project. Another important conclusion to be drawn is the importance of the technical training of the person who carries out the virtual recreation of the apparatus and devices which make up the element of preindustrial heritage, as in order to create a truetolife animation it is necessary to know how these devices worked and were originally designed. Without such knowledge, it would be impossible for example to reproduce real working speeds of the machinery in the animation. At the same time it is extremely useful to be familiar with forward and inverse kinematics, as it makes working with Autodesk 3ds Max much easier. Generating a high quality computer animation requires great effort, not only in learning the main software packages used, but also in learning to use other graphic programs, such as video editing software and photographic software, which offer many possibilities. There has been a great deal of progress in the field of virtual reality, for example augmented reality, which is especially useful in the design of virtual scenes where reallife images are mixed with virtual images. However, computer animation using specific software still gives very high quality results, which makes it very useful when the objective is to show in detail how old machinery worked and its environment. Computer animation using specific software provides a better solution when dealing with complex machinery than a virtual reality in which the user interacts with the system, as the user would need to know in detail how the machinery worked. For example, in the case of a windmill, how to use the regulation elements such as the counterweight which operates the raising mechanism of the runner stone or the mechanism which controls the brake rim; this is specialist knowledge which a normal user is unlikely to possess.
Computer Animation Applied to the Recovery of Preindustrial Heritage
55
Therefore we have produced an audiovisual animation with as many camera angles as necessary, which shows quickly and intuitively all the details of the working of the machinery and its environment, of which it is not necessary for the user to have expert knowledge. This shows how nonimmersive virtual reality using computer animations can provide many advantages.
Funding This research was funded by the Spanish National R&D Plan (HUM200600377), “Estudio históricotecnológico y representación gráfica de la evolución en el diseño de los molinos de viento en la mancha, en la España de los siglos XVI y XVII, mediante técnicas de Dibujo Asistido”, of the Research Projects Subdepartment of the Universities Department of the Ministry of Science and Innovation.
References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18]
http://www.virtualheritage.net/ http://www.itabc.cnr.it/VHLab http://cordis.europa.eu/fp7/ict/telearndigicult/home_en.html (Cultural Heritage & Technology Enhanced Learning) http://whc.unesco.org/en/35/ Refsland, S. T., Ojika, T., Addison A. C., & Stone, R. J. (2000) Virtual heritage: Breathing new life into our ancient past. IEEE Multimedia, 7, 2021. Arnold, D. (2001). Virtual heritage: Challenges and opportunities. Digital content creation (281293). New York: SpringerVerlag. Addison, A. C. (2000). Emerging trends in virtual heritage. IEEE Multimedia, 7, 2225. Stone, R. J., & Ojika, T. (2000). Virtual Heritage: What Next?. IEEE Multimedia, 7, 7374. Buchanan, R. A. (1972). Industrial Archaeology in Britain. Harmondsworth: Penguin Books. Hudson, K. (1971). A guide to the industrial archaeology of Europe. London: Adams & Dart. Carandini, A. (1984). Arqueología y cultura material. Barcelona: Mitre. http://www.mnactec.cat/ticcih/ http://www.industrialarchaeology.org.uk/ http://www.historyoftechnology.org/ RojasSola, J. I. (2005). Ancient technology and ComputerAided Design: olive oil production in Southern Spain. Interdisciplinary Science Reviews, 30, 5967. RojasSola, J. I., & DomeneGarcía, J. (2005). Engineering and Computeraided design: A Study of watermills in Southeastern Spain. Interciencia, 30, 745751. RojasSola, J. I., SuárezQuirós, J., & RubioGarcía, R. (2007). The tradition of fulling mills: a study from engineering. Interciencia, 32, 675678. RojasSola, J. I., & LópezGarcía, R. (2007). Engineering graphics and watermills: Ancient technology in Spain. Renewable Energy, 32, 20192033.
56
José Ignacio RojasSola and Francisco Javier ContrerasAnguita
[19] Pennestri, E., Pezzuti, E., Valentini, P. P., & Vita, L. (2006). Computeraided virtual reconstruction of Italian ancient clocks. Computer animation and virtual worlds, 17, 565572. [20] RojasSola, J. I. (2006). Cultural heritage and information technologies: improvement proposal for science and technology museums and interactive Centers of Venezuela. [21] RojasSola, J. I., & LópezGarcía, R. (2007). Computeraided design in the recovery and analysis of industrial heritage: Application to a watermill. International Journal of Engineering Education, 23, 192198. [22] Bakker, G., Meulenberg, F., & De Rode, J. (2003). Truth and Credibility as a Double Ambition: Reconstruction of the Built Past, Experiences and Dilemmas. Journal of Visualization and Computer Animation, 14, 159167. [23] RojasSola, J. I., & AmezcuaOgáyar, J.M. (2005). Graphical and Technical study of windmills in Spain. Interciencia, 30, 339346. [24] RojasSola, J. I., & AmezcuaOgáyar, J.M. (2005). Southern Spanish windmills: technological aspects. Renewable Energy¸30, 19431953. [25] RojasSola, J. I., GómezElvira González, M.A., & PérezMartín, E. (2006). Computeraided design and engineering: a study of windmills in La Mancha (Spain). Renewable Energy, 31, 14711482. [26] RojasSola, J. I., & AmezcuaOgáyar, J.M. (2005). Origin and expansion of windmills in Spain. Interciencia, 30, 316325. [27] SuárezQuirós, J., RojasSola, J. I., RubioGarcía, R., MartínGonzález, S., & MoránFernández, S. (2009). Teaching applications of the new computeraided modelling technologies in the recovery and diffusion of the industrial heritage. Computer Applications in Engineering Education, 17¸ 455466. [28] http://www.tdt3d.be/articles_viewer.php?art_id=99 [29] Pavlidis, G., Koutsoudis, A., Arnaoutoglou, F., Tsioukas, V., & Chamzas, C. (2007). Methods for 3d Digitization of Cultural Heritage. Journal of Cultural Heritage, 8, 9398.
In: Computer Animation Editors: J.S. Wright and L.M. Hughes, pp. 5783
ISBN: 9781607415596 © 2010 Nova Science Publishers, Inc.
Chapter 2
VIRTUAL ENGINEERING IN AUGMENTED REALITY Pier Paolo Valentini*, Eugenio Pezzuti and Davide Gattamelata University of Rome “Tor Vergata”, Department of Mechanical Engineering Via del Politecnico, 1 – 00133 Rome, Italy
Abstract In this chapter the authors discuss several approaches in order to integrate computeraided engineering instruments into Augmented Reality environment. Engineers and designers often develop their creative ideas in front of a computer monitor using mouse and keyboard. Although the integration between numerical computation and graphics leads to the generation of very realistic digital mockups, they are still far from the real context and the user has limited interaction with them. The purpose is to illustrate how recent development in computer graphics and image processing can improve the realism and interactivity with digital mockups. Starting from the interactive modeling of 3d shapes, the chapter presents some examples about the integration of realtime mechanism motion simulation, structural and fluid dynamics analysis postprocessing.
Keywords: Virtual Engineering; Augmented Reality; Simulation; ComputerAided Design
1. Introduction 1.1. The Role of Virtual Engineering During last decades computeraided engineering (CAE) methodologies have deeply changed the way of designing and developing products, systems and services [1]. Thank also to significant hardware and software improvements, CAE techniques are widely used by the designers since the early conceptual phases up to the final stages of engineering processes. At the industry level, these methodologies have become a fundamental tool to be competitive and to ensure high quality standards. In industrial engineering, computeraided methodologies typically are instrumental for design teams in shape modeling, behavioral simulations, digital *
Email address:
[email protected] (Address all correspondence to this author)
58
Pier Paolo Valentini, Eugenio Pezzuti and Davide Gattamelata
mockups up and realistic animations. They are able to follow the development of a product from conception to production, also managing its lifecycle. For this reason, the science that support the use of these tools is called Virtual Engineering. It means that all the tasks which are typical of an engineer or a designer can be arranged in a virtual way, using informatics and computers. The benefits of this approach are many. First of all there is a saving of time: many solutions can be tested, compared and optimized without building physical prototypes. As a consequence there is a saving of money because a digital mockup is much less expensive than a physical one. On the other hand, a meaningful and reliable virtual simulation needs accurate study of the involved phenomena, definition of parameters, simplifications. The available computing capabilities allow to produce very realistic models and animations, mimic the real behavior of a system. Examples of these model are reported in Figure 1.
Figure 1. Examples of digital mockup built and simulated using virtual engineering techniques.
Virtual Engineering in Augmented Reality
59
1.2. Augmented Reality The Augmented reality (AR) is an emerging field of the visual communication and information technologies [24]. It deals with the combination of real world images and computer generated data. Although the idea of virtual reality can be dated in the last century and the development of portable displays is dated 1966, the phrase “augmented reality” was coined by Prof. Tom Caudell only in 1990 considering an application developed at Boeing to help workers in assembling aircraft components. At present, most AR research is concerned with the use of live video imagery which is digitally processed and "augmented" by the addition of computer generated graphics. The idea behind the augmented reality is simple, but its development has been slowed down by the inadequacy of hardware resources to support real time heavy computation. Only during last decades, thanks to the increasing of hardware performance, the research in the field of augmented reality boosted [5]. With an AR system, the user can extend the visual perception of the world, being supported by additional information and virtual objects. The level of details of the augmented scene has to be very realistic in order to give the user the illusion of a unique real world. There are different types of AR applications depending on the level of details, graphics effects and interactivity (Figure 2). A generic AR implementation is depicted in Figure 3. The user interacts with the application. This one has to manage the presence of interfaces, ensure a correct tracking of user and objects in the real scene and compute an adequate and realistic collimation between real world and augmented contents. At the first level there are the tracking and the registration technologies; the first aims to calculate the user’s point of view with respect to the scene while the registration deals with the collimation of virtual world objects with the real environment. The humanscene interfaces can be mechanical, electromagnetic or optical devices. The second basic element for an AR system is the capability of real time rendering. The detail of rendered scene depends on hardware capabilities. The display technology ensures the immersion of the user in the scene. The ways to capture images from the real world, process and project again to the user can be different [6]. Three different technologies are currently implemented: • • •
the video based system; the optical seethrough system; the video seethrough system.
In the first one, as shown in the left side picture of Figure 2, the images coming from the real world are augmented with simple graphics data and streamed on a monitor. This approach is often used in sports to superimpose data in a common television broadcasting. It is quite simple, but the user is not immersed in the scene at all. In optical seethrough systems, the information is not directly captured from the real world, but the augmented contents are added by projecting them onto a semitransparent visor which naturally mixes the real perception with the augmented one. The user feels immersed in the scene, but there are some limitations on the quality of the images and some difficulties in collimating real and virtual objects.
60
Pier Paolo Valentini, Eugenio Pezzuti and Davide Gattamelata
Figure 2. Examples of augmented reality applications. On the left: a simple video augmented application in which a virtual line is added to the video stream in a swimming competition. On the right: a complex augmented scene for training in medicine (courtesy of Institute for Computer Graphics and Vision, Austria).
Figure 3. Layout of a generic highend Augmented Reality System.
The video seethrough system (see Figure 2, on the right and the scheme depicted in Figure 4) is based on the use of one or two cameras which acquire an image stream from the real world. The stream is processed by a computer which add virtual contents, producing an augmented image stream which is projected again to the user by means of a blind visor. It allows to increase the level of immersion in the scene and the quality of the images depends on the resolution of the visor. As witnessed by several investigations, the video seethrough systems revealed to be more suitable for different applications, especially in the engineering field.
Virtual Engineering in Augmented Reality
61
Figure 4. AR Video seethrough system.
Scientific literature reports an increasing interest for the development of applications of augmented reality in many different fields [79]. The AR has been used in medicine and surgery [10] to improve the reliability of complex clinical treatments and assist operations (imageguided surgery). Moreover, for military purposes, the army is already using AR displays in cockpits where screened information are shown to the pilot on the windshield of the cockpit or the visor of their flight helmets. Other fields of interests in AR technologies are the robotics [11] and telerobotics in which an augmented display can assist the user with a visual image of the remote workspace to guide the robot movements. AR is also useful in maintenance and assembling activities [1214] where technicians can approach a new or unfamiliar piece of equipment simply putting on an AR display, instead of opening several repair manuals, and visualizing information directly on the desired object. AR applications have been developed in architecture [15] for perceiving the structural modifications inside and outside an house, by superimposing walls and furniture to the current solution and perceiving the results in a realistic way. There are also applications in elearning [1518], manufacturing [1920], services and logistics [2122], arts [23], navigation [24], etc. The most of all these applications deals with the merging in the real world object, scene and animation which have been modeled and simulated outside the system. It means that the user perceives a real scene augmented with precomputed object. His interaction with them is often limited to exploration. In order to extend the application of augmented reality in engineering field we have to move some steps forward. A designer or an engineer cannot be limited to the exploration of the scene, but he has to interact with objects to modify, create, animate and materialize his creativity [2529]. For this purpose the augmented reality has to be enriched with tools (haptic devices) which allow to interact with the objects in the scene. The most important interaction is about the tracking of user’s position in the augmented scene. There are five types of tracking devices that are implemented, depending on the methodology used for measuring the
62
Pier Paolo Valentini, Eugenio Pezzuti and Davide Gattamelata
position in the space: mechanical, electromagnetic, optical, acoustic and inertial. The first ones use a multi degreeoffreedom linkage to compute the position in the space of a pointer. They have the advantage to be very precise and rather cheap but they have a small workspace and limit the user’s movement. The electromagnetic devices are comprised of an emitter and a receiver. The emitter generates a magnetic field which is captured by the receiver. The changing of the acquired signal is converted to information about the position and attitude of the receiver. Due to its small size, the receiver can be easy put on by the user or attached to a tracking stick. On the other hand, the electromagnetic devices are sensible to interference of the magnetic field due to electronic equipments or bulky ferromagnetic objects. Optical tracking systems are more complex. They require the use of several markers and two or more highspeed cameras. Their precision depends on the resolution of the cameras and the size of the markers. They allow wide working area, but require a specific setup to ensure that the markers are always visible to cameras. Acoustic devices are comprised of an emitter and several receiver. The emitter generates an acoustic signal whose time of flight is acquired by each receiver and converted in spatial position information. They are quite cheap devices but cannot ensure great precision and they are sensible to temperature and humidity variation and to the presence of echoes. Inertial devices use accelerometers and gyroscopes to measure the position and the attitude. They require a frequent calibration but offer good precision. In a general application, the interaction with the scene has to fulfill the following requirements: • • • •
the devices have to be easy to use; the devices have to be precise and be allow an acquisition with a high frequency; their application has to be intuitive and natural without limit the user’s movement; they have to support the ability and the intent of the designer being an alley and not an obstacle.
Unfortunately many of these devices or system are often much expensive and their cost limits the usage to large research facilities or large industries. The AR system has some similarities with the Virtual Reality (VR) one. The main difference is that in the VR the perceived world is fully virtual (generated by one or more rendering pipelines), while in the AR the virtual world merges the real one. This mixing involves not only the generation of rendered graphics, but also complex procedures of registering the real and the virtual video stream. Moreover, in order to give the most natural perception of the augmented world to the user, all these activities need to be real time processed. Such demanding requirements slowed down the advancements and researches of AR systems with respect to the VR ones.
1.3. Motivation and Objectives The motivations of this chapter come from all the considerations discussed in the previous section and are fuelled by the idea that future of virtual engineering will be on virtual and augmented platforms in order to increase the user’s interaction and the level of realism and perception.
Virtual Engineering in Augmented Reality
63
According to many researchers, the AR system can be the future of ComputerAided Design (CAD) technologies and Virtual Engineering. Presently CAE applications support the designer through numerical computation and computer graphics. Often engineers and designers develop their creative ideas in front of a computer monitor using mouse and keyboard. Although the integration between numerical computation and graphics leads to the generation of realistic digital mockups, they are still far from the real context and the user has a limited interaction with them. This limitation can generate problems (nonconformities, unexpected behaviour and appearance, for instance) when the designed products have to be integrated in the real world. For overcoming this disadvantage, new instruments, based on AR systems can be set up. The contents of the chapter concern with the discussion about possible implementation of virtual engineering tools in an augmented reality environment. Three aspects will be faced: the modeling of three dimensional shapes, the simulation of physical motion of mechanisms and the visualization of engineering structural and fluiddynamics investigations. Both hardware and software details will be discussed, proposing an implementation of a lowcost system.
2. 3D Modelling in Augmented Reality The first step toward the building of a virtual prototype is the modeling of shapes. Standard applications implemented into ComputerAided Design systems make use of keyboard, mouse and, in some cases, of spatial pointers. The results of modeling actions can be viewed by the designer on the pc monitor. They have a good level of realism but they are limited inside the monitor. The designer with his personal ability has to extrapolate these results and imagine how they will fit in the real world. In order to overcome this limitation, virtual shapes can be merged in the real world using augmented reality techniques. This solution still limits the creativity of the designer because he has to model using a monitor interface and then project the shapes in the real world. A more interactive solution involves the possibility to create virtual shapes directly in the real environment by passing the use of external interfaces. In order to arrange a system able to allow the user a realtime modeling and visualization, appropriate hardware has to be set up and software has to be implemented. In the following section a low cost solution, that has been implemented by the authors, will be described.
2.1. Hardware Setup The implemented system is made of input, processing and output devices. Input devices have to acquire a real world video stream and user actions. Output devices have to project an augmented perception of the real world enriched with virtual objects. Processing units have to manage inputs coming from different devices, store and arrange data flows and render the augmented video stream. The input video device of the implemented system is a Microsoft LifeCam VX6000 USB 2.0 camera, able to catch frames up to 30 Hz with a resolution of 1024x768 pixels. This camera has been rigidly mounted on the Head Mounted Display (Figure 5). The position tracking system has been developed starting from an electromagnetic 6 degrees of freedom
64
Pier Paolo Valentini, Eugenio Pezzuti and Davide Gattamelata
sensor named Flock of Birds by Ascension (http://www.ascensiontech.com) in combination with a wood pen (named “IPen”). Flock of Birds is an electromagnetic tracker similar to those described in the introduction. Its receiver (a small cube) is rigidly constrained to the IPen and the emitter is placed inside the design space. Another input device is a standard pc keyboard for user key command. The processing unit is a personal computer with a Pentium IV quad core processor, 3 Gb RAM, NVidia Quadro FX3700 graphic card and the operative system is Windows XP professional and the development suite for programming is Microsoft Visual Studio 2005. The output display is an Head Mounted Display equipped with OLed displays (Z800 3D visor by Emagin  http://www.3dvisor.com/). It is able to support stereovision up to a resolution of 800x600 each eye. The emitter of the electromagnetic tracker has to be rigidly fixed inside the working area. Since it has a transmission range of less than 1 m, it can be useful to place in the middle of the area. The presence of metallic objects strongly affects the measurement, so it is important to ensure a correct shielding or an appropriate distance of ferromagnetic parts from the working zone. In the working area, in a place that can be always seen by the camera, a patterned marker has to be located. Its presence allows the computation of the relative position between the camera and the real world as explained in the following section. The user has to wear the head mounted display and the camera and has to grab the IPen in a normal way (Figure 5).
2.2. Software Setup Since the system is assembled from scratch, a reliable software has to be implemented in order to manage the various data flows, to control the devices and to generate an augmented realtime video stream. Since the driver of the electromagnetic tracker were available in C++, all the procedures have been implemented in the same programming language.
Figure 5. Input and output devices of the implemented system.
In the system there are three main data flows. The first is the video stream coming from the camera. This video is processed in order to find out the relative position between the camera and the real world. Since the camera is fixed on the head mounted display, it is also
Virtual Engineering in Augmented Reality
65
the relative position between the user and real world. The computation is possible thanks to the ArtToolkit libraries. Their source code (freely available together with documentation at http://sourceforge.net/projects/artoolkit) is widely used to developed augmented reality applications. The ArtToolkit routines can analyze the scene, recognize a patterned square marker (as that in Figure 4), and find out the spatial transformation between the camera and the real world. At the same time another data flow comes from the tracking system. So it is possible to locate the tip P of the IPen in the world reference frame of the emitter (OXYZ) as (see Figure 6):
{P}O− XYZ = {Op }O− XYZ + [T ]IPen {P}O − x y z p
(1)
p p p
where:
{*} A are the coordinate of the generic vector with respect to A system of coordinates; [T ]IPen is the spatial transformation between the reference frame of the IPen sensor and
that of the emitter
{ }
Both O p
O − XYZ
and [T ]IPen can be computed with the embedded driver of the tracker.
Figure 6. World, emitter and IPen reference frames.
In order to collimate the three reference frame of the camera, the world and the IPen, another transformation has to be computed. It relates the emitter with the world patterned marker. Since these two reference frames are fixed in space, this transformation ( {OW }O − XYZ and
[T ]marker )
can be computed offline when setting up the working
environment. By doing this it is possible to locate the IPen tip P in the reference frame of the marker as:
{r '}O − XYZ = {P}O− XYZ − {OW }O− XYZ {P}O
W
= [T ]marker {r '}O − XYZ = [T ]marker {r '}O − XYZ −1
− xW yW zW
T
(2) (3)
66
Pier Paolo Valentini, Eugenio Pezzuti and Davide Gattamelata
2.3. Examples In this section some examples of modeling will be discussed. With reference to Figure 7 one can notice how to sketch planar geometry such as lines and polygons. The user can draw using the IPen in the same way he can do with a pencil and a paper sheet. The graphic modeler, based on OpenGL, render the scene, augmented with sketched objects. The augmented scene also includes virtual tripods in order to underline where the world systems of coordinate is located (on the patterned marker and where the IPen is pointing). Figure 8 shows examples of three dimensional geometry. Starting from sketches, the user can extrude 1D and 2D entities, building surfaces and solids. It can be done simply dragging the IPen throughout the path of extrusion in a natural way. The results can be visualized in preview during the modeling, so the user can adjust and correct his operation in real time. Starting from the picking of points in space, the user can also define free form surfaces (Figure 9). These entities are very suitable for aesthetic design purposes, reverse engineering and advanced modeling. The user can define and modify control points of surface, simply moving the IPen in the working area and selecting locations.
Figure 7. Modelling of objects in plane (lines, on the left and a triangle, on the right).
Virtual Engineering in Augmented Reality
67
3. Simulating and Animating in AR The capabilities of augmented reality can be also used to support engineering simulation. In particular, this section focuses on the main three types of simulation often performed by engineers: the movement of a linkage (kinematics and dynamics), the deformation of an object subjected to loads, and the dynamics of fluids inside or outside objects. When a designer prepares a virtual model for one of these simulations, he has to define geometries, constraints and boundary conditions. All these tasks can be supported by the implemented AR system which can serve as pre and postprocessor. Moreover, the great advantage that the augmented reality can give to simulation is in the visualization of results, i.e. in postprocessing. In this way the user can integrate in the real world the visual results of several numerical analyses in a more realistic way.
Figure 8. Modelling of 3D extruded objects (a surface based on a spline, on the left and two cylinders, on the right).
68
Pier Paolo Valentini, Eugenio Pezzuti and Davide Gattamelata
Figure 9. Modelling of 3D complex objects (a free form surface based on control points, on the left and a patched surface to be integrated in an existing models of a car, on the right).
3.1. Multibody Animation With multibody simulations engineers analyze the kinematics and dynamics of moving parts and linkages. Starting from the knowledge of mass and inertial properties of each component, topological constraints (joints and fixtures) and external actions (forces, torques, motors), it is possible to simulate the movement of a system in a reliable way. Readers interested in techniques for deducing the governing equations and solution strategies can refer to referenced books [3031]. The AR can support multibody simulations in two different ways. The first is about the possibility to project on the real world the results coming from a precomputed simulation. It concern the rendering on the scene of all the objects involved in the simulation, whose position is updated according to the results of the computation. This implementation is similar to that of the common post processing software for visualizing graphics results. The only difference lays in the merging of the simulated system in the real
Virtual Engineering in Augmented Reality
69
world. The advantage is to perceive the interaction with the real world and check working spaces, possible interferences, etc. Although useful, this approach does not use all the potential of AR. Let us consider a practical application in order to illustrate this first possible integration between motion simulation and augmented reality. The purpose is to simulate a robot that has to be mounted on flange by means of a revolute joints. The flange is present in the real world as a physical component (Figure 10). The robot and its movement have to be added as virtual shapes. The first step is to built the virtual parts. This task can be done using the modeling techniques illustrated in the previous section or also using a computeraided design program. In this case it is useful to export geometries using .vrml (Virtual Reality Markup Language) file. This format is quite common and can be exported by almost all solid modelers. Dealing with .vrml is useful for rendering geometries using OpenGL libraries. The second step is to perform the numerical computation using a specific solver. In order to prepare the input file for simulation, it is useful to consider how to relate the real world to the virtual world. The main issue is to collimate the two main reference frames. This operation can be made using communication reference frames in the same way of substructuring large assemblies. In other worlds, we can consider the virtual world (and the objects in it) as a subsystem of the real world. The relation between the main system and a subsystem is controlled by several constraint equations acting on the communication reference frames. For the specific example of the robot, it can be convenient to locate the real world communication frame on the flange when the revolute joint has to be implemented. Similarly, we can define the virtual communication frame coincident to the inertial reference frame of the virtual system. When building the equations for simulation, we have to create a fictitious revolute joint between the virtual inertial reference frame and a reference frame on the first link of the robot. This approach can be resumed in the following five steps: 1. Before the simulation starts, the geometries and topological properties (joints and connections) have to be defined, built and stored in files; 2. The real scene has to contain information for collimating the real world to the virtual objects (communication frames); 3. The equations of motion of the investigated mechanisms have to be built and externally solved by means of a specific solver; 4. After the numerical solution, simulation results have to be accessible to AR executable. 5. Each frame acquisition, virtual objects have to be rendered on the scene in the correct position and attitude according to the simulation results, considering the position of the communication frames and using OpenGl ModelView transformations. The definition of the real world reference frame can be done in two different ways. In the first, the patterned marker can be placed on the flange where the communication frame is desired (Figure 10). The second way is about the picking of points defining the communication frame by the use of IPen and recording the coordinates of the origin and main axes. As a result, the augmented scene includes the real world with the virtual robot. Since the movement of the manipulator has been simulated solving the equations of motion, the augmented scene will show an animation of the robot. By this way the user can visualize from different points of view, the motion of the manipulator directly in the real world looking at its
70
Pier Paolo Valentini, Eugenio Pezzuti and Davide Gattamelata
performance, verifying working space, checking possible interference with real objects and the aesthetic impact on the real environment. The augmented video can be enriched with other visual information on kinematic and dynamic parameters as velocity, acceleration, force, torque, joint reaction, etc. This can be performed rendering on the scene both vectors (for the direction) and numerical values (for the amplitude) as static overlays. A smarter way to enhance the multibody simulation is to introduce interactivity. It means that the user does not only watch the augmented scene, but interact with it. Let us imagine the kinematic simulation of a robot arm whose endeffector can be grabbed by the user and moved. The purpose of the simulation is to compute attitude and position of all the links in order to obtain the required position and orientation of the endeffector. This simulation can be supported by augmented reality by introducing in the real scene the virtual robot that can be interactively manipulated by the user using virtual sensors. Of course, the simulation of the mechanism has to be computed real time in order to update the scene with quick information. This idea can be implemented following five steps (Figure 11): 1. Before the simulation starts, the geometries and topological properties (joints and connections) have to be defined, built and stored in files; 2. The real scene has to contain information for collimating the real world to the virtual objects and the virtual sensor(s) for the interactive action of the user; 3. Each frame, the position and attitude of all markers in the scene have to be acquired and mathematical transformations between camera and markers have to be computed. 4. Each frame, starting from the previous recognition and calculation, dynamic or kinematics equations have to be solved in order to compute the correct position and attitude of all the virtual bodies in the scene; Each frame acquisition, virtual objects have to be rendered on the scene in the correct position and attitude.
Figure 10. Continued on next page.
Virtual Engineering in Augmented Reality
71
Figure 10. Simulation of the movement of a manipulator in augmented reality. Starting from a real environment and adding collimated virtual objects (on the top), an augmented animation can be built (at the bottom).
Figure 11. Activities for implementing interactive simulation in augmented reality.
The input and output procedures (data acquisition, markers recognition, rendering of the virtual objects) can be implemented using the ArtToolkit open source library. Between the input procedures (data acquisition and markers recognition) and the output procedures (rendering of the virtual objects on the input scene) a specific kinematic or dynamic solution has to be implemented. This portion of the algorithm depends on the specific mechanism to be simulated. Since it has to be performed between the acquisition of two consecutive frames, all the equations need a realtime solution [3234]. For this purpose it is useful to optimize the solution strategy and kinematics simulations are more suitable
72
Pier Paolo Valentini, Eugenio Pezzuti and Davide Gattamelata
because involve the solution of a system of non linear equations, instead of a system of differentialalgebraic equations. In order to explain in details all the steps to implement a multibody simulation in an Augmented Reality environment we develop an example about a 3D manipulator.
Figure 12. The 3D manipulator of the example, and its closure loop.
With reference to Figure 12, let us consider a robot with 4 bodies connected with 3 revolute joints and 1 spherical joint. According to Grubler’s count the mechanism has 6 degrees of freedom:
dof = 6 ⋅ nlink − ∑ i =1
joints
( 6 − fi ) = 6
Virtual Engineering in Augmented Reality
73
where nlink is the number of moving parts, joints is the number of kinematic pairs and f i are the degrees of constraint of the ith pair. It means that, in order to define in a unique way the position in space of the manipulator, we have to prescribe 6 independent parameters (i.e. position and attitude of the endeffector). For an interactive simulation it means that the user can freely choose the position and attitude of the end effector and the Augmented Reality scene has to be able to include such a 6 d.o.f. sensor. This sensor can be represented by a patterned marker. The first step in building the model is the construction of geometries of each link that can be stored in .vrml (Virtual Reality Markup Language) file. This format is quite common and can be exported by almost all solid modelers. Dealing with .vrml is useful for rendering geometries using OpenGL. The second step is about the preparation of the scene. We need a marker (marker 0) to define the position and orientation of the manipulator world coordinate system (i.e. its position inside the scene) and another marker (marker 1) to define the position and the orientation of the endeffector that will work as a sensor (Figure 12, on the right). The third step is about the implementation of the system of constraint equations which can be built considering the closed loop of vectors (Figure 12, on the right). Several approaches can be used for building the system of equations. It is useful, for the subsequent graphical operations, to use the 4x4 homogeneous transformation matrix [T ]l1−l 2 to express
the relative position and attitude between two generic links 2 and 1. The first 3x3 portion of this matrix is used to define the relative orientation between the two reference frames attached to the two links. The last column is used to describe the relative position between the origins of the coordinate frames. The last row of the matrix is [0 0 0 1]:
⎡[ Orientation ]3x3 [ Position ]3x1 ⎤ [T ]l1−l 2 = ⎢ ⎥ ⎣
0 0 0
1
⎦
Looking at the tip point P on the link 2 (center of the spherical joint) we can deduce its position with respect to the marker 0 as:
{P}marker0 = [T ]0−l 0 ⋅ [T ]l 0−l1 ⋅ [T ]l1−l 2 ⋅ {P}link 2
(1)
where:
{P}marker0 is the position vector of point P in marker 0 (world) coordinate system; {P}link 2 is the position vector of point P in link 2 (local) coordinate system; [T ]0−l 0 is the homogeneous transformation matrix between link 0 and marker 0. It is a function of the parameter revolute joint;
which describes the relative rotation at their relative
74
Pier Paolo Valentini, Eugenio Pezzuti and Davide Gattamelata
[T ]l 0−l1 is the homogeneous transformation matrix between link 1 and link 0. It is a function of the parameter revolute joint;
which describes the relative rotation at their relative
function of the parameter revolute joint;
which describes the relative rotation at their relative
[T ]l1−l 2 is the homogeneous transformation matrix between link 2 and link 1. It is a
Looking at the same point P, but on the slider and considering that the slider is attached to the marker 1, we can deduce its position with respect to the marker 0 as:
{P}marker0 = [T ]0−1 ⋅{P}slider
(2)
where:
{P}slider is the position vector of point P in slider or marker 1 (local) coordinate system; [T ]0−1 is the homogeneous transformation matrix between marker 1 and marker 0. It is a function of the 6 independent parameters which describes the relative position and the relative rotation between the two markers. These parameters can be considered as the input of the kinematic analysis because can be freely chosen by the user to move the manipulator in space.
Since at point P, link 1 is connected to slider by means of a spherical joint, we can obtain the closure loop equation of the mechanism as:
[T ]0−l 0 ⋅ [T ]l 0−l1 ⋅ [T ]l1−l 2 ⋅ {P}link2 − [T ]0−1 {P}slider
⎧0 ⎫ ⎪0 ⎪ ⎪ ⎪ =⎨ ⎬ ⎪0 ⎪ ⎩⎪1 ⎭⎪
(3)
The system of equations in (3) can be solved for the unknown kinematic parameters starting from the knowledge of the position and attitude of the marker 1 (and the endeffector slider) with respect to the marker 0. In order to compute this information we have to know the relative transformation between marker 1 and marker 0. ARToolkit deals with matrices which are similar to the homogeneous ones. They are 3x4 transformation matrices, containing information about the relative position and attitude as the homogeneous ones but without the last dummy row. Quaternions and position vector can be extracted from these matrices by using arUtilMat2QuatPos Artoolkit procedure. The relative position of the camera with respect of marker 0 ( [T ]c −0 ) and marker 1
( [T ]c −1 ) can be computed using an Artoolkit procedure arGetTransMat. It computes the camera position and attitude in function of detected markers. Their inverse matrices
Virtual Engineering in Augmented Reality ( [T ]0−c and
[T ]1−c )
75
represent the relative transformations between the markers and the
camera. The relative transformation between the marker 1 and marker 0 can be computed as:
[T ]0−1 = [T ]c−0 ⋅ [T ]1−c
(4)
This relation is used to compute the coordinates of point P on slider (fixed to the marker 1) with respect to the marker 0. By doing this, the equations (3) can be solved for the unknown angles. At this point the virtual position and attitude of each link of the mechanism are known. The next step is to compute the projection of geometries in the augmented scene. Since the renderer is OpenGl engine [35], we have to deal with the ModelView projection matrix which maps the 3d points coordinates into 2d (scene) coordinates. The first step concerns with the loading of the projection matrix computed from the perspective of the marker 0. This task can be performed by using arglCameraViewRH whose output is a vector of 16 elements containing the transformation and scale information to project the object from the 3D coordinate system of the marker 0 into the camera view plane. This transformation can be used to relate the position and attitude of each objects from the virtual world coordinate system and the marker 0. The next steps concern the computation of the transformation in order to draw the manipulator link in the scene, using the OpenGl operators glRotated and glTranslated. The first performs a rotation of an angle about a specified axis, the second performs a translation of a specified amplitude along a specified direction. For the base it is sufficient to perform a rotation about the z axis of the marker 0 of an angle: glRotated(alpha,0.0,0.0,1.0);
For the first link three transformations need: a translation a1 along z axis of marker 0, a rotation about the same z axis of an first revolute joint:
angle and a rotation of a
angle about the axis of the
glTranslated(0.0,0.0,a1); glRotated(alpha,0.0,0.0,1.0); glRotated(beta,0.0,1.0,0.0);
For the second link, three transformations need: a rotation of an angle about the z axis of the marker 0, a translation to the center of the revolute joint between the 1st and the 2nd links and a rotation of an 2nd links:
angle about the axis of the revolute joint between the 1st and the
glRotated(alpha,0.0,0.0,1.0); glTranslated(l1*cos(beta),0.0,l1*sin(beta)+a1); glRotated(gamma,0.0,1.0,0.0);
The endeffector, since it is fixed to the marker 1, can be projected using marker 1 projection matrix without applying any further transformation.
76
Pier Paolo Valentini, Eugenio Pezzuti and Davide Gattamelata Screenshots of animation are reported in Figure 13.
Figure 13. Snapshots of the interactive animation.
3.2. FEM Pre and Postprocessing By means of finite element analyses engineers analyze the mechanical stresses inside a structure and its deformation to applied loads. Starting from the knowledge of elastic properties of each part, topological constraints between parts and presence of external actions (forces, torques, enforced displacements) it is possible to simulate the deformation of a system and
Virtual Engineering in Augmented Reality
77
assess the level of mechanical stresses inside it. Readers interested in techniques for deducing the governing equations and solution strategies can refer to referenced book [36]. Let us consider a practical application in order to illustrate how the augmented reality can support this kind of engineering simulation. The purpose is to simulate a steel bar that is pinned on two mountings and it is loaded with a force. The two mountings and the bar are physically present in the real world (Figure 14). The force, the resultant deformation and the stress field have to be added as virtual contents. The first step is to built the virtual parts. This task can be done using the modeling techniques illustrated in the previous section, using the IPen to acquire the geometry of the bar. The second step is to define the location and the amplitude of the force. This can be also done with the IPen, by pointing on the force location on the real world and defining load amplitude using the keyboard. Then, we have to perform the numerical computation using an external FEM solver. The last step is the projection of the numerical results in the real scene. Generally, the main results of such simulations concern with a deformed shape of the structure (that reproduces the deformation in an amplified way) that is coloured with a palette indicating the stress level. The information has to be added to the real world. Again, we have to collimate the two main reference frames of the real world and the virtual one.
Figure 14. Simulation of the deformation and stresses of a pinned bar in augmented reality. Starting from a real environment and adding collimated virtual objects (on the top), an augmented animation can be built (at the bottom) .
78
Pier Paolo Valentini, Eugenio Pezzuti and Davide Gattamelata
This operation can be made using communication reference frames in the same way of the previous discussion. For the specific example of the bar, it can be convenient to put a visible patterned marker locate near the physical bar (Figure 14), defining the real world communication frame. Similarly, we can define the virtual communication frame at the corresponding location in the virtual world. By this way it is possible to locate the virtual deformed and colored bar in the real scene with a simple coordinate transformation (translation and rotation) using OpenGl operators glRotated and glTranslated. As a result, the augmented scene includes the real world with the virtual bar (Figure 14, at the bottom). The scene can be also enriched including an animation of the deformed shape instead of static geometry. It is useful especially for vibration analyses where the visualization of modal shape is very important. By this way, the user can visualize from different points of view, the deformation and the stress level of the bar directly in the real world looking at its performance, verifying excessive stresses or deformation. The implementation can be summarizes in the following six steps: 1. The real scene has to contain information for collimating the real world to the virtual objects. A patterned marker or a coordinate system defined by picking points with IPen can be used; 2. The geometry of component(s) under investigation has to be acquired; 3. Boundary conditions (constraints and loads) have to be defined too; 4. The computation of deformed shapes and stress level can be performed using an external FEM solver; 5. The results coming from step 4 has to be converted into coloured .wrml entities (i.e. deformed shapes with stress contour plot); 6. The .wrml entities have to be collimated with the real world and rendered on the augmented scene. The augmented video can be enriched with other visual information on the exact numerical values of stresses (i.e. adding a virtual legend), constraints and restraints forces, etc. This can be performed rendering on the scene both vectors (for the direction) and numerical values (for the amplitude).
3.3. CFD Postprocessing With computational fluiddynamics engineers analyze liquid and gasses flows inside or outside parts. Starting from the knowledge of fluid properties, boundary conditions (flow rate, pressure, temperature, etc.) and wall properties, it is possible to simulate the kinematic and thermodynamic behavior of fluids and their interaction with solids. Readers interested in techniques for deducing the governing equations and solution strategies can refer to referenced book [37]. Let us consider a practical application in order to illustrate how the augmented reality can support this kind of engineering simulation. The purpose is to simulate an external air flow around a cylinder on a table. Both table and cylinder are physically present in the real world (Figure 15). The stream line and the pressure field on the surfaces have to be added as virtual contents. The first step is to built the virtual parts. This task can be done using the modeling
Virtual Engineering in Augmented Reality
79
techniques illustrated in the previous section, using the IPen to acquire the geometry of the cylinder and of the table. The second step is to define all the bounding conditions for the simulation. Because there are many values to be defined, it is convenient to complete the model outside the augmented reality environment. Then we have to perform the numerical computation using an external CFD solver. The last step is the projection of the numerical results in the real scene. Generally, the main results of such simulations concern with several stream lines describing the fluid trajectories which are coloured with a palette indicating the velocity, pressure or temperature value. The information has to be added to the real world. Once again, we have to collimate the two main reference frames of the real world and the virtual one. This operation can be made using communication reference frames in the same way of the previous discussions. For the specific example of the cylinder, it can be convenient to put a visible patterned marker locate near the physical cylinder (Figure 15), defining the real world communication frame. Similarly, we can define the virtual communication frame at the corresponding location in the virtual world. By this way it is possible to locate the virtual stream lines and surface plots in the real scene with a simple coordinate transformation (translation and rotation) using OpenGl operators glRotated and glTranslated.
Figure 15. Simulation of the air flow around a cylinder on a table in augmented reality. Starting from a real environment and adding collimated virtual objects (on the top), an augmented animation can be built (at the bottom) .
80
Pier Paolo Valentini, Eugenio Pezzuti and Davide Gattamelata
As a result, the augmented scene includes the real world with the virtual surfaces touched by the fluid (cylinder and table) (Figure 15, at the bottom). The scene can be also enriched including an animation of the stream lines. By this way the user can visualize from different points of view, the fluid stream around the cylinder and the pressure acting on the boundary surfaces directly in the real world. The implementation can be summarizes in the following six steps: 1. The real scene has to contain information for collimating the real world to the virtual objects. A patterned marker or a coordinate system defined by picking points with IPen can be used; 2. The geometry of component(s) under investigation has to be acquired; 3. Boundary conditions (fluid flows, pressure openings, wall properties, etc.) have to be defined too; 4. The computation of fluid field and pressure on surfaces can be performed using an external FEM solver; 5. The results coming from step 4 has to be converted into coloured .wrml entities (i.e. stream lines, surfaces coloured according to pressure values); 6. The .wrml entities have to be collimated with the real world and rendered on the augmented scene. The augmented video can be also enriched with other visual information on the exact numerical values of fluid or surface parameters adding a virtual legend (i.e for describing pressure, velocities, temperature, etc.).
4. Conclusion The integration between computer aided engineering tools and augmented reality has revealed to be a valid instrument in supporting designers and users in modeling, testing and reviewing their products. With the integration of specific hardware devices as trackers and sensors, the user can interact with the scene in an immersive way. For the modeling of shapes, a magnetic sensor can be useful to acquire a precise position in space of a virtual pen in order to allow the user to sketch virtual objects directly on the real world. Moreover the comprehension of the results of engineering simulations as motion analyses, structural investigations, fluid dynamics computations can be improved by enhanced visualization and interaction with real and virtual objects. The discussed instrument and methodologies can be also useful for the collaborative design. Very often, scientists or engineers teams work on the same project at different locations. In this case, all the designers can wear an AR subsystem and all these subsystems can communicate among them. Imagine that a group of designers is working on the model of a complex device for their customers. The designers and customers want to perform a joined design review even though they are physically separated. If each of them is equipped with an augmented reality display this could be accomplished. The physical prototype that the designers have mocked up is imaged and displayed in the client’s AR system in 3D. The client may look at different aspects of it, testing engineering performances and checking its integration to the real world.
Virtual Engineering in Augmented Reality
81
The future of this combination between AR and Virtual Engineering is very promising. The discussed examples are only a small part of the capabilities of such integration. The main future challenges are about the manipulation of objects, the capabilities for virtual assembling and the fully integration of numerical methodology without requiring external solvers.
References [1]
[2] [3] [4] [5] [6] [7]
[8] [9]
[10]
[11]
[12] [13]
[14]
Bernard A. (2005). Virtual engineering: methods and tools. Proceedings of the Institution of Mechanical Engineers, Part B: Journal of Engineering Manufacture Vol. 219 (5), pp. 413422. Azuma R.T. (1997). A survey of augmented reality. Teleoperators and Virtual Environments, 6(4), pp. 355–385. Bimber O., Raskar R. (2005). Spatial Augmented Reality: Merging Real and Virtual Worlds, A K Peters, Ltd. Vallino J. (1998). Interactive augmented reality. PhD thesis, Department of Computer Science, University of Rochester, USA. Azuma R., Baillot Y. et al. (2001). Recent advances in augmented reality. IEEE Computer Graphics 21(6), pp. 34–47. Haniff D., Baber C., Edmondson W. (2000). Categorizing augmented reality systems. Journal of Three Dimensional Images 14(4), pp. 105– 109. Klinker G., Ahlers K.H. et al. (1997). Confluence of computer vision and interactive graphics for augmented reality, PRESENCE: teleoperations and virtual environments. Special issue on Augmented Reality, 6(4), pp. 433–451. Fotis Liarokapis (2007). An augmented reality interface for visualizing and interacting with virtual content, Virtual Reality 11, pp. 23–43. Muller A., Conrad S., Kruijff E. (2003). Multifaceted interaction with a virtual engineering environment using a scenegraphoriented approach. Proceedings of the 11th Int. Conf. in Central Europe on Computer Graphics, Visualization and Computer Vision, Czech Republic. Samset E., Talsma A., Elle O., Aurdal L., Hirschberg H., Fosse E. (2002). A virtual environment for surgical image guidance in intraoperative MRI, Computer Aided Surgery 7(4), pp.187– 196. Stilman M., Michel P., Chestnutt J., Nishiwaki K., Kagami S., Kuffner J.J. (2005). Augmented Reality for Robot Development and Experimentation. Tech. Report CMURITR0555, Robotics Institute, Carnegie Mellon University. Ong S.K., Pang Y., Nee, A.Y.C. (2007). Augmented Reality Aided Assembly Design and Planning. Annals of the CIRP Vol. 56/1 pp.4952. Pang Y., Nee A.Y.C., Ong S.K., Yuan M.L. YoucefToumi K. (2006). Assembly Feature Design in an Augmented Reality Environment, Assembly Automation, 26/1, pp. 3443. Sharma R. and Molineros J. (1995). Computer visionbased augmented reality for guiding manual assembly. PRESENCE: Teleoperators and Virtual Environments, n. 3, pp. 292317.
82
Pier Paolo Valentini, Eugenio Pezzuti and Davide Gattamelata
[15] Webster A., Feiner S., MacIntyre B., Massie W., Krueger T. (1996). Augmented reality in architectural construction, inspection, and renovation. Int. Proc. Of Third Congress on Computing in Civil Engineering ASCE 3, Anaheim, CA, pp. 913919. [16] Liarokapis, Petridis P., Lister P.F., White M. (2002). Multimedia augmented reality interface for Elearning (MARIE). World TransEng Technol Educ 1(2), pp.173–176. [17] Pan Z., Cheok A.D., Yang H., Zhu J.,_, Shi J. (2006). Virtual reality and mixed reality for virtual learning environments. Computers & Graphics 30, pp. 20–28. [18] Kaufmann H., Schmalstieg D., Wagner, M. (2000). Construct3D: A Virtual Reality Application for Mathematics and Geometry Education. Education and Information Technologies 5(4), pp. 263276. [19] Dangelmaier W., Fischer M., Gausemeier J., Grafe M., Matysczok C., Mueck B. (2005). Virtual and augmented reality support for discrete manufacturing system simulation. Computers in Industry 56, pp. 371–383. [20] Ong S.K., Nee A.Y.C. (2004). Virtual and Augmented Reality Applications in Manufacturing. Springer, London, UK. [21] Reif R., Walch D. (2008). Augmented & Virtual Reality applications in the field of logistics. Visual Computing 24, pp. 987–994. [22] Friedrich W. (2004). ARVIKA—Augmented Reality for Development, Production and Service. Publicis Corporate Publishing, Erlangen. [23] Liarokapis F., Sylaiou S., et al. (2004). An interactive visualization interface for virtual museum. Proceedings of the 5th international symposium on Virtual Reality, Archaeology Cultural Heritage, Brussels and Oudenaarde, pp 47–56. [24] Narzt W., Pomberger G., Ferscha A., Kolb D., Muller R, Wieghardt J., Hortner H., Lindinger C. (2006). Augmented reality navigation systems. Univ Access Inf Soc 4, pp. 177–187. [25] Klinker G., Stricker D., Reiners D. (1999). Optically based direct manipulation for augmented reality, Computers & Graphics 23, pp. 827830. [26] Bowman D.A. (1996). Conceptual Design Space: Beyond Walkthrough to Immersive Design. In Bertol D., Designing Digital Space, John Wiley & Sons, New York. [27] Bowman D.A. (1999). Interaction Techniques for Common Tasks in Immersive Virtual Environments: Design, Evaluation, and Application. Ph.D. thesis, Virginia Polytechnic & State University. [28] Liang J. and Green M. (1994). JDCAD: a highly interactive 3D modeling system. Computers & Graphics 18(4), pp. 499506. [29] Raskar R., Welch G., Cutts M., Lake A., Stesin L., Fuchs H. (1998). The office of the future: a unified approach to imageBased modeling and spatially immersive displays. Proceedings of SIGGRAPH 98, Orlando, FL, July 19–24. [30] Garçıa de Jálon J., Bayo E. (2004). Kinematic and Dynamic Simulation of Multibody Systems – the RealTime Challenge. SpringerVerlag, New York. [31] Haug E.J. (1989). ComputerAided Kinematics and Dynamics of Mechanical Systems. Allyn and Bacon, Boston, MA. [32] Korkealaakso P.M., Rouvinen A.J., Moisio S.M., Peusaari J.K. (2007). Development of a realtime simulation environment. Multibody System Dynamics, 17, pp. 177–194. [33] Naya M.A., Dopico D., Perez J.A., Cuadrado J. (2007). Realtime multibody formulation for virtualrealitybased design and evaluation of automobile controllers.
Virtual Engineering in Augmented Reality
[34]
[35] [36] [37]
83
Proceedings of the Institution of Mechanical Engineers, Part K: Journal of Multibody Dynamics, Vol. 221 (2), pp. 261276. Isnard F., Dodds G., Vallée C., Fortuné D. (2000). Realtime dynamics simulation of a closedchain robot within a virtual reality environment. Proceedings of the Institution of Mechanical Engineers, Part K: Journal of Multibody Dynamics Vol. 214(4), pp. 219232. Wright R.S., Lipchak B. (2004). OpenGl Super Bible, third edition, Sams Publishing, USA. Zienkiewicz O.C. (1977). The Finite Element Method in Engineering Science. McGrawHill Publ., London. Anderson J.D. (1995). Computational Fluid Dynamics, Mc Grow Hill Inc., USA.
In: Computer Animation ISBN 9781607415596 c 2010 Nova Science Publishers, Inc. Editors: J.S. Wright and L.M. Hughes, pp. 85111
Chapter 3
A S URVEY OF P OPULAR 3D S OFTB ODY A NIMATION C OMPRESSION A PPROACHES S. Ramanathan and A.A. Kassim National University of Singapore, Dept. of Electrical and Computer Engineering, Singapore
1.
Introduction
The world of Computer Graphics has seen rapid developments over the past few decades. Since the appearance of 3D computer animation in the science fiction movie, Futureworld (1976), animations have been successfully employed in Toy Story (1995), Shrek (2001) and Happy Feet (2006). Contemporary animations are synthesized by deforming 3D objects, typically modeled using triangular meshes. A triangular mesh is defined by its geometry (location of vertices), connectivity (how the triangles are connected), surface color, normal and texture properties. An isolated triangle requires more than 100 bytes for its description, since the geometry, normal and texture components are represented using floating point numbers. For accurate representation of real world objects, 3D mesh models typically require many thousands of vertices and triangles. Therefore, 3D meshes demand extensive storage memory and transmission bandwidth, which makes 3D animation compression1 a very relevant problem that has generated much interest in recent years. Besides entertainment and video games, 3D meshes and animations are used in biometrics, computeraided design (CAD) and medicine. 3D softbody animations are synthesized by applying nonlinear deformations on a static mesh to generate realistic, lifelike motion. Complex, realistic and human like motion of these characters is achieved using physicallybased [41, 9] or synthetic [30] animation techniques, by moving the 3D mesh vertices in separate trajectories, as compared to rigidbody transformations where the whole mesh deforms homogeneously. As shown in Fig. 1, mesh deformations can be achieved by either 1
In the 1980s, the need for efficient multimedia storage and transmission led to the development of the popular JPEG (Joint Photographic Experts Group) and MPEG (Moving Picture Experts Group) techniques.
86
S. Ramanathan and A.A. Kassim i. changing the mesh geometry while keeping the connectivity constant where mesh is deformed by changing the position of the mesh vertices. These animations are termed dynamic geometry sequences. ii. altering mesh geometry as well as connectivity  where deformations are accompanied by movement of the mesh vertices as well as addition/deletion of new/existing vertices which modifies the existing connectivity.
While it is intuitive to treat coding of 3D animations (essentially 3D video) similar to 2D video compression [24], the two problems are inherently different. The key to efficient 3D animation coding is the compact representation of the intermesh motion using a few parameters analogous to motion vectors in video compression. A number of animation coding algorithms [25, 1, 46] have attempted to address the compact 3D motion representation problem using the video coding approach involving the following steps (i) Segmentation  a necessary preprocessing step to efficiently represent the motion between consecutive video frames. The current video instance (frame) is segmented into smaller blocks (of size 8 × 8 or 16 × 16 pixels) during this process. (ii) Motion prediction  which allows for compact interframe motion representation in terms of motion parameters. The motion parameters are computed by finding the bestmatch block in the temporal reference for each block in the current frame. (iii) Parameter encoding  where the amount of information required to represent the motion parameters is further reduced using data compression techniques such as Huffman/Arithmetic coding [45]. However, the arbitrary topology (i.e., arrangement of vertices in space) of 3D meshes makes it difficult to efficiently segment 3D meshes and thereby translate well known video compression techniques to 3D. The difficulty involved in segmenting nonplanar 3D meshes as compared to trivial planar image segmentation is illustrated in Fig. 2. Therefore, a number of new methods have been proposed for efficient 3D dynamic mesh coding. MPEG4 Part 25 [19] and www.3dcompression.com are resources dedicated to research on 3D animation compression. In this chapter, we review 3D dynamic mesh compression algorithms and investigate how vertex clustering, which chiefly contributes to animation coding complexity, affects compression performance. We finally conclude this chapter with observations that need to be effectively addressed by future 3D animation coding algorithms.
2. 2.1.
An Overview of Mesh Coding Algorithms Static Mesh Compression
Over the past decade, much research has been focused on compression of 3D static meshes and a majority of these [7, 8, 10, 11, 26, 34, 40, 42] address efficient encoding of mesh connectivity. Using spiralling treebased encoding schemes, popular algorithms like Cutborder[11] and Edgebreaker [34] achieve lossless connectivity compression with only up
A Survey of Popular 3D SoftBody Animation Compression Approaches
87
(a)
(b) Figure 1. Examples of animations synthesized by (a) changing mesh geometry as seen from frames 95 and 105 of the Chicken animation or (b) changing both mesh geometry and connectivity (around the mouth) for realistic facial expression generation.
(a)
(b)
Figure 2. (a) Image segmentation into equalsized blocks is a simple preprocessing step to motion prediction in video coding. A segmented frame from the Foreman sequence. (b) Segmentation of the Dino 3D mesh (2039 vertices, 3999 triangles) into 29 pieces using spectral mesh decomposition [28]. Since 3D meshes are nonplanar, segmenting 3D meshes into coherent pieces is a nontrivial and computeintensive task.
88
S. Ramanathan and A.A. Kassim
to 1.52 bits per mesh triangle. A few multiresolution geometrycumconnectivity representation techniques [17, 32] have been proposed to facilitate progressive data transmission and achieve a compression efficiency of around 410 bits per vertex (bpv). Compression of mesh geometry is only a supplementary to the connectivity coding scheme in these algorithms. Geometry compression, which involves coding of the floating point (x,y,z) vertex coordinates, is inherently lossy and has been attempted using predictive coding as well as signal processingbased techniques. Predictive coding [42] exploits correlation in the mesh data by predicting a vertex position using the positions of its neighboring vertices. Prediction errors are quantized and entropy coded for compact representation. A typical compression efficiency of 712 bpv is obtained using this scheme. Spectral compression [48] is a popular signal processingbased mesh compression method, where the mesh geometry is projected onto an orthonormal basis and reconstructed using a small number of components in the basis. This method achieves a compression efficiency of around 14 bpv for perceptually lossless encoding. A waveletbased geometry compression technique [2] achieves a compression efficiency of 8 bpv. Since the number of components for geometry reconstruction can be adaptively varied, [48] and [2] can also be used for multiresolution mesh representation. Other techniques that tackle the problem of mesh representation with various levels of detail are [32, 8].
2.2.
Dynamic Mesh Coding Algorithms
As noted above, there are two types of 3D animation sequences  (i) dynamic geometry sequences where mesh motion is achieved by moving the mesh vertices with time, and (ii) dynamic geometrycumconnectivity sequences where mesh motion is accompanied by changes in mesh geometry as well as connectivity. Dynamic geometry compression algorithms can be grouped into three major classes based on their implementation Registrationbased, Predictionbased and PCAbased multiresolution representation. Examples of these algorithms are discussed below. 2.2.1.
RegistrationBased Compression
In Lengyel’s pioneering work on registrationbased dynamic geometry compression [25], he proposes the segmentation of the mesh into smaller submeshes and represents the motion of each of these submeshes using rigidbody affine transforms. His compression mechanism yields an efficiency of 3.45 bpvf (bpv per frame) for the Chicken animation with 16 and 4 bits used for affine and vertex quantization respectively.Ibarria et al. report a compression efficiency ranging from 1.37 to 2.91 bpvf for their Dynapack algorithm [18] when the quantization ranges from 7 to 13 bits for test animations. Their algorithm exploits spacetime coherence in dynamic geometry by predicting the position of each vertex v in frame f from three of its neighbors in f and the positions of v and its neighbors in the previous frame. A video codinglike method which segments the 3D mesh into blocks and computes motion vectors and error residuals for each mesh block is proposed by Ahn et al. [1]. A compression efficiency of 9.6 bpvf is obtained using the encoding scheme that consists of I
A Survey of Popular 3D SoftBody Animation Compression Approaches
89
(Intra), P (Predicted) and B (Bidirectionally predicted) meshes for the Chicken animation. In Gupta et al.’s dynamic geometry compression scheme [13], the mesh is partitioned into segments, and the displacement of vertices in each segment is computed using Iterative Closest Point (ICP)based registration. The encoding scheme describes mesh motion using a few affine parameters and residual errors to achieve a compression efficiency of 2.5 bpvf for the Chicken animation. 2.2.2.
PredictionBased Compression
Another interesting work on dynamic geometry compression is that of Yang et al. [46], based on vertexwise motion vector (MV) prediction. Each vertex is given a motion vector obtained from the neighborhood of the vertex, defined as the set of all vertices within a threshold distance around the vertex. This their coding procedure requires a third of the bitrate compared to [25] for the same quality of animation reconstruction measured in terms of SignaltoNoise ratio (SNR). Stefanoski et al. propose a connectivitybased prediction technique in [38], where prediction is performed in a frametoframe fashion using the previous frame and the partly decoded current frame. Mesh connectivity is used to determine the order of vertex compression and the spatialcumtemporal dependency between vertex locations is exploited using a nonlinear spatiotemporal predictor with angle preserving properties. They report a 25% improvement in compression performance over competing prediction schemes like [18], especially for high quality animation reconstruction. Muller et al. [31] propose another predictionbased compression algorithm using Differential Pulse Code Modulation (DPCM) where errors in prediction from the previously decoded mesh are clustered in an octree. Only a representative from each cluster is used for further processing which results in a significant reduction in bitrate. 2.2.3.
Multiresolution Representation
Recently, multiresolution mesh representation for bandwidth limited streaming applications has generated much interest. A notable work on multiresolution representation of dynamic geometry is that of Alexa et al. [3] who propose a Principal Component Analysis (PCA)based compact animation representation scheme where each mesh in the animation sequence is projected on a basis of n PCA eigenvectors. The animation may be reconstructed using k eigenvectors where k << n. Higher the k, greater the level of detail. Another example of waveletbased multiresolution encoding is that of Guskov et al. [14], which exploits parametric coherence in mesh sequences using an anisotropic wavelet transform and progressively encodes wavelet details. Payan et al. [33] propose another waveletbased multiresolution representation scheme based on a temporal lifting scheme that exploits the temporal redundancy in dynamic geometry. In [21], Karni et al. propose a compression scheme that employs a combination of Principal Component Analysis (PCA) and Linear Predictive Coding (LPC). Recently, localized PCAbased dynamic geometry coding techniques have yielded good compression performance. Sattler et al. [35] propose animation compression using Clustered PCA (CPCA) where the mesh is first segmented into meaningful components based on vertex motion analysis and PCA is applied on each of these components. This compression scheme outperforms both pure PCAbased and PCA+LPC approaches while achieving better animation
90
S. Ramanathan and A.A. Kassim
reconstruction. Another Localized PCA Analysis (LPCA)based compression scheme is proposed by Amjoun et al. [4]. Upon clustering the mesh using local similarity properties, a local coordinate system is defined for each cluster with respect to which the cluster motion is encoded using PCA. LPCA coding achieves better compression performance compared to CPCAbased compression for similar quality of reconstructed animation. 2.2.4.
Other Coding Algorithms
Varakliotis et al. propose animation encoding with RTP packetization in [43], and recommend insertion of I frames in the encoded/transmitted mesh sequence to maintain animation smoothness. A Differential Pulse Code Modulation (DPCM)based encoder is used to compress the animation, whose compression efficiency is low. Main contributions of this work include (i) Analysis of the tradeoff between compression performance and reconstructed animation quality and (ii) Introduction of the Peak Mean Square Error (PMSE)based distortion metric to tackle degradation of animation smoothness under noise. MPEG4 Part 25 [19] presents generic tools for dynamic 3D mesh compression using BoneBased Animation (BBA), which involves decomposition of geometric motions in the animation to elementary transformations, and Framebased Animation Mesh Compression (FAMC), where the animation is divided into segments that can be decoded independently. A spatially and temporally scalable compression scheme for 3D animations using FAMC, where the original animation is reconstructed at multiple layers corresponding to different spatial resolutions is proposed in [39]. Boulfani et al. [5] propose a 3D dynamic mesh compression scheme where geometry compensation is performed upon clustering the mesh using motion characteristics followed by application of the scanbased wavelet transform. 2.2.5.
Encoding 3D Dynamic Meshes with Changing Connectivity
Very few works deal with compression of animations with changing connectivity. Shamir et al. [36] suggest a multiresolution representation scheme for animations with changing connectivity using the TDAG data structure, which can be incrementally constructed even as the input mesh is processed. Gupta et al. [12] propose an Iterative Closest Point (ICP) registrationbased geometrycumconnectivity coding scheme for dynamic 3D MMs. As in [13], the current and previous frames are partitioned to generate submeshes and intermesh correspondences are computed using ICP to identify the added/deleted vertices over time. Subsequently, the errors in geometry and connectivity prediction are encoded and transmitted.
3.
Vertex Clustering for Dynamic Geometry Coding
In this section, we investigate how vertex clustering, which involves grouping of mesh vertices for motion prediction, affects dynamic geometry compression. We first discuss the impact of vertex clustering on registrationbased dynamic mesh coding with ICPbased compression [13] as an example, and see how PCA coding approaches developed on similar lines [35, 4] have yielded superior compression performance for dynamic geometry animations in general.
A Survey of Popular 3D SoftBody Animation Compression Approaches
91
Vertex clustering techniques group mesh vertices into a number of sets, where the grouping may done on the basis of (a) mesh topology (b) mesh geometry or (c) semantic mesh segmentation.
3.1. 3.1.1.
Overview of Vertex Clustering Techniques TopologyBased Clustering
Topologybased clustering techniques partition the mesh based on vertex adjacency. They are more popularly known as ”Graph Partitioning Techniques” and many algorithms have been developed to solve the graph partitioning problem [16, 23]. The graph partitioning problem was formulated to solve a compute intensive problem using a number of parallel processors where the objective was to share the workload equally among the processors while keeping number of interprocessor communications small. In the case of the 3D triangular mesh, the aim is to divide it into equalsized pieces with minimum number of edges between the pieces. To this end, the multilevel kway graph partitioning algorithm divides the mesh into k roughly equal partitions such that the number of cut edges connecting vertices in different partitions is minimized. In MPEG video compression [24], the image is divided into smaller, equalsized blocks for efficient motion prediction. Likewise, decomposition of the mesh into equalsized segments can be achieved using the topologybased multilevel kway graph partitioning algorithm [16]. Given the mesh geometry V , the function of the graph partitioning algorithm is to divide the mesh, consisting of n vertices, into k subsets, V1 , V2 , . . . , Vk such that Vi ∩ Vj [
= φ f or i 6= j
(1)
Vi  = n/k
(2)
Vi = V
(3)
i=1..k
S where Vi  denotes the cardinality of the ith cluster and Vi denotes the union of the k clusters. In this mode of clustering, vertices are clustered with their connected neighbors as given by the mesh connectivity and no knowledge of mesh geometry is required. For a graph G(V, E) containing vertex set V and edge set E, the graph partitioning algorithm first coarsens the original graph G0 = G(V 0 , E 0 ) into a series of coarse graphs Gi = G(V i , E i ), such that the number of vertices at Gi is approximately half the number of vertices at G(i−1) i.e., Vi  ≈ 12 Vi−1 . The graph is coarsened by performing a series of edge contractions. A maximal set of edges, no two of which are incident on the same vertex, are first determined and these edges are contracted. This coarsening procedure maps each vertex in the fine graph Gi−1 to a unique vertex in the coarse graph Gi and therefore, graph topology is preserved. The coarsening terminates when the original graph has been coarsened to Gm = G(V m , E m ) where Vm  is typically a small number. The coarsest graph Gm is now segmented using a spectral partitioner [15] which uses eigenvectors of the graph Laplacian for partitioning. The partitions obtained for the coarsest graph are propagated back to the finer graphs by projecting the k partitions onto Gm−1 , Gm−2 , . . . , G0 . The projected partitioning onto Gi−1 is occasionally refined using local refinement heuristics based on the KerninghanLin (KL) algorithm [23]. Vertices are
92
S. Ramanathan and A.A. Kassim
incrementally swapped among the partitions to reduce the number of cut edges connecting vertices in different partitions. The mesh is finally divided through recursive bisection into k sets each containing about V0 /k vertices. The clusters generated by the kway graph partitioning algorithm for various meshes are shown in Fig 3. Overall, multilevel kway partitioning performs better than competing inertial or spectral bisection approaches [37] in terms of execution time and the partition quality (based on the number of cut edges). However, the vertex clusters obtained by minimization of the number of cutedges are ineffective for determining the mesh motion. This is evident from Fig 2 where vertices belonging to distinct mesh regions are clustered together (parts of the nose, forehead and cheeks in the face; pelvis and thigh regions for blade) while vertices corresponding to the same region fall in different clusters (mouth region of the face and the claws for the chicken). Clearly, the clustered vertices do not undergo homogeneous motion. Also, when the same clusters are used for the entire sequence, detecting the coherent motion regions becomes difficult and consequently, the compression performance is affected as we will see in latter sections where we analyze related experimental results.
(a)
(b)
(c)
(d)
Figure 3. Clusters generated by topologybased kway partitioning algorithm for (a) Chicken  32 partitions (b) Face  8 partitions (c) Dinopet  32 partitions and (d) Blade  8 partitions.
3.1.2.
GeometryBased Clustering
Geometrybased clustering involves grouping of vertices based on their positional closeness and is independent of the mesh connectivity. The Lloyd’s kmeans algorithm [29] is a popularly used geometrybased clustering technique. Given a set of n data points {xi } in ddimensional space and the required number of clusters k, the problem is to determine a set of k centers {cj } such that the mean squared distance of each point to its nearest center, termed the average distortion D, is minimum. The algorithm works as follows. The initial k cluster centers are chosen at random and the data points {xi } are partitioned into k clusters by assigning each point to the cluster containing the closest ci . The set of data points to which ci is the nearest center is known
A Survey of Popular 3D SoftBody Animation Compression Approaches
93
as the neighborhood of ci and is denoted by V (ci ). Once the initial centers and their neighborhoods have been determined, the algorithm proceeds by moving the ci ’s to the centroid of their clusters and recomputing V (ci ) for each of the ci ’s. This process iterates until convergence is achieved or the mean distortion D achieves a local minimum. A summary of the Lloyd’s algorithm is presented below. • Step 1: Initialize {cj } by selecting the cj ’s at random. • Step 2: Determine the neighborhood V (cj ) for each of the cj ’s by assigning the xi ’s to their closest center. V (cj ) = {xi : d(xi , cj ) ≤ d(xi , ck ), f or all k 6= j}
(4)
• Step 3: Move each of the cj ’s to the centroid of V (cj ). cj =
X 1 (xi ), xi ∈ V (cj ) V (cj )
(5)
i
• Step 4: Repeat Step 2 and Step 3 until mean distortion D is minimum i.e., D=
k X 1X 1 (xi − cj )2 = Dmin k V (cj ) j=1
(a)
(6)
x∈V (cj )
(b)
(c)
Figure 4. Illustration of Lloyd’s clustering for k=3. (a) The initial cluster centers in red, blue and green and their computed neighborhoods (b) Centers are moved to the centroid of the cluster and the data points are reassigned to the nearest centers. (c) Final clusters. Fig 4 illustrates the working of the Lloyd’s algorithm. In the context of mesh partitioning, the clusters themselves are more important than the cluster centers. For kmeans clustering, it can be proved that the local minimum distortion measure would correspond to a ”centroidal Voronoi” configuration [20], where each data point is closer to its cluster center than any other cluster center. The partitions move closer to this configuration at every step until convergence, and the final clusters would correspond to the local energy minima, even when the initial centers are badly chosen. However, slightly different initial
94
S. Ramanathan and A.A. Kassim
partitionings do not produce the same set of clusters. Also, while the final partitioning is definitely better than the initial partitioning, it need not correspond to the global minimum. Nevertheless, this is not a significant problem since data repartitioning may be performed later, as explained in the next section. Since vertex clustering is independent of the mesh connectivity, vertex neighbors have to be computed explicitly. For 3D meshes, computing nearest neighbors is not a trivial problem. The Lloyd’s implementation in [20] computes nearest neighbors using a kdtree (k dimensional tree) data structure. Also, since the data points do not change throughout the cluster computation process, the kdtree needs to be computed only once. The clusters at every step are determined by computing the nearest center for each of the tree nodes. Since clusters are determined based on the vertex positions, the cluster configurations will vary for different meshes in the animation sequence (Fig 5). Clustering based on vertex proximity produces better quality partitions whose vertices are more likely to undergo homogeneous motion.The cluster sizes are variable and the mesh can be segmented into an arbitrary number of clusters. A noticeable improvement in compression performance is observed when geometrybased clustering is employed, especially for highmotion sequences, as noted in later sections. However, since the vertex clusters do not correspond to the distinctive mesh components, the general performance of geometrybased clustering is inferior to that of semantic mesh decomposition for dynamic mesh coding.
(a)
(b)
(c)
(d)
Figure 5. Segmentation using Lloyd’s clustering for frames (a) 70 and (b) 120 of the Chicken animation (maximum cluster size = 100); frames (c) 0 and (d) 505 of the Face animation (maximum cluster size = 75).
3.1.3.
SpectralBased 3D Mesh Segmentation
A third set of techniques perform Semantic Mesh Decomposition to segment the mesh into meaningful components. These techniques exploit both topology and geometry features to generate components that represent distinctive features of the 3D polygonal mesh. Unlike in geometry or topologybased clustering, where the number of clusters is user specified, most of the semantic mesh decomposition algorithms [28, 22, 27] automatically determine the number of vertex clusters based on homogeneity of the mesh regions. The mesh components represent the distinctive regions of the object consistent with human perception, which defines boundaries along concavities of the surface. They can be used to establish
A Survey of Popular 3D SoftBody Animation Compression Approaches
95
shape correspondence, and in most cases, also correspond to regions capable of undergoing independent motion (Fig 6). Liu and Zhang proposed a spectral clustering approach to mesh decomposition in [28], where, in order to segment a 3D mesh with n faces along the edges, the n×n affinity matrix W is initially constructed for the dual of the mesh graph to group faces closer to each other. Each vertex in the dual graph corresponds to a mesh face and two vertices are connected if and only if the corresponding mesh faces are adjacent to each other. For grouping of faces, the pairwise face distance measure used in [22] is used to define the affinity matrix. The distance measure between mesh faces fi and fj is defined as the shortest path between their dual vertices given by
Dist(i, j) = weight(dual(fi ), dual(fj )) = δ
Geod(fi , fj ) Ang Dist(αij ) +(1−δ) (7) avg(Geod) avg(Ang Dist)
Here, Geod(fi , fj ) is the geodesic distance between fi and fj while the angular distance is defined as Ang Dist(αij ) = η(1 − cosαij )
(8)
where αij is the angle between the normals for adjacent faces fi and fj . Since the angular distance plays a more important role for visually meaningful segmentation, δ is set to a value close to zero. Also, a smaller value of η favors concavities and therefore it is set in the range 0.1 ≤ η ≤ 0.2. On obtaining the pairwise face distances, the affinity matrix is defined by the Gaussian kernel W (i, j) = e
−Dist(i,j) 2σ 2
(9)
It can be easily seen that 0 < W (i, j) < 1 and takes larger values for faces closer to each other. A suitable value for the width of the Gaussian, σ, is empirically set to 1 P 2 1≤i,j≤n Dist(i, j). n W (i, j) encodes the likelihood of faces i and j belonging to the same patch. The nor1 1 malization of the affinity matrix is performed as N = D− 2 W D− 2 where D is the diagonal matrix whose ith diagonal element is the sum of the ith row of W , the vertex degree at node i. N possesses desirable properties in the context of spectral clustering [44] and W Nij = √D ijD . Let V be n×k matrix formed using the k leading eigenvectors of N . Then, ii
jj
the n × n matrix Q = V V T represents the most energy preserving projection of N to rank k. Normalizing the rows of V to unit length gives Vˆ , whose rows vˆ1 . . . vˆn (of dimension k) represent the embedding of the Wij s onto the kdimensional unit sphere centered at origin. ˆ = Vˆ Vˆ T is known as the association matrix whose elements Q ˆ ij = vˆi vˆj T = cos θij Q are the cosine of the angle between unit vectors vˆi andPvˆj . As N is projected to successively lower rank k, the sum of squared angle cosines i,j (cosθij )2 is strictly increasing [6]. Point pairs likely to be clustered together will move towards each other as k decreases, while other pairs will move further apart. Therefore, clustering points in kdimensional space is easier than clustering the original data and is accomplished by performing kmeans clustering on the rows of Vˆ .
96
S. Ramanathan and A.A. Kassim
The spectral clustering algorithm performs a semantic decomposition of the mesh with the generated components corresponding to the salient object features, capable of undergoing independent motion, as shown in Fig 6. The algorithm tends to segment the mesh in a hierarchical fashion on varying the number of eigenvectors chosen for V . The computation of the shortest distance face pairs and the affinity matrix W are of complexity O(n2 log(n)) and O(n2 ) respectively but this computation time is greatly reduced in the implementation described in [47]. Also, a recursive 2way spectral cut procedure used in [47] overcomes the problem of choosing the optimal k for clustering and produces better quality partitions. Since the components generated upon mesh decomposition correspond to the salient features of the object, the same set of vertex clusters can be used for performing motion estimation for the entire animation sequence. As the components can describe the piecewise affine mesh motion effectively, it is possible to encode the animation more efficiently. Experimental results confirm the superior compression performance obtained through semantic mesh decomposition compared to kway partitioning or Lloyd’s clustering for dynamic geometry compression.
(a)
(b)
(c)
(d)
Figure 6. Segmentation of (a) Chicken (59 components), (b) Face (6 components), (c) Dolphin (7 components) and (d) Dinopet (29 components) meshes through spectral clustering. A number of mesh segments e.g., fins of the dolphin, limbs of the dinosaur can undergo independent motion.
3.1.4.
Analysis of RegistrationBased Coding Algorithms
A majority of the registrationbased dynamic geometry compression algorithms use topologybased clustering for segmenting meshes. Lengyel’s [25] algorithm uses a greedy vertex clustering approach based on the triangulation of the original mesh. Predictionbased geometry compression algorithms [46, 38] define vertex neighborhoods for prediction based on mesh connectivity . Ahn et al. [1] segment the mesh by converting the triangular mesh structure into a linear triangle strip form. The triangle strip is divided into blocks such that each block has same number of vertices. Gupta et al. [13] use the multilevel kway graph partitioning technique [16] that generates clusters of approximately equal sizes. Since the
A Survey of Popular 3D SoftBody Animation Compression Approaches
97
mesh connectivity remains constant for dynamic geometry, topologybased clustering needs to be performed only for the first mesh in the sequence (I mesh) and the clusters remain fixed thereafter for the entire sequence. We find that mesh segmentation based on the fixed mesh topology is unsuitable for compressing mesh sequences with changing mesh geometry. Efficient detection of mesh pieces that have moved over time is possible only when the components generated upon clustering roughly represent these pieces. When the mesh undergoes arbitrary deformation, the vertex clusters undergoing coherent motion will be different at different times. Clearly, it is impossible for a given set of clusters generated using topologybased partitioning to represent the coherent motion regions at all times. Clustering the mesh on the basis of positional proximity instead of graph adjacency is more suited for encoding dynamic geometry sequences. Alternatively, a fixed set of clusters will effectively describe the piecewise affine mesh motion in animations only if they correspond to the distinctive mesh components that can undergo independent motion. Our experimental results (Section 3.7) confirm that geometrybased clustering and semantic mesh decomposition techniques produce better compression performance than topologybased clustering. The next section briefly discusses ICPbased dynamic mesh coding [13] and related performance metrics while the following sections demonstrate how potent vertex clustering improves compression performance of registrationbased and PCAbased dynamic mesh coding algorithms.Finally, we conclude with observations that will require critical consideration during the development of an efficient 3D animation coding standard.
3.2.
ICPbased 3D Dynamic Geometry Compression
The vertex clustering algorithms described in the previous section simplify the problem of determining the regions of coherent motion by segmenting the mesh into pieces that can possibly undergo affine motion. In order to determine the actual regions that have moved, motion prediction needs to be performed. For 3D dynamic geometry, the intermesh motion is typically small. The mesh motion can be completely described using a few affine transformations and residual errors, and this compact representation leads to compression. An efficient dynamic geometry compression algorithm that performs systematic motion estimation is described in [13]. The algorithm uses the multilevel kway graph partitioning algorithm for initially segmenting into pieces, and for each piece in the current mesh, the corresponding piece in the temporal reference is detected using the Iterative Closest Point (ICP) algorithm. Using the results of ICPbased registration, the motion segmentation module segments the mesh vertices into three distinct sets based on their motion characteristics. • First set (Type 1)  consisting of clusters of vertices, such that the motion of each of the clusters may be described accurately using the associated affine transform Ai . The reconstruction error for the vertices falling in this set is less than a threshold, τ . • Second set (Type 2)  consisting of clusters of vertices, such that each cluster has an affine transformation matrix Ai associated with it. In addition, residual errors also need to be encoded for accurately representing the vertex positions. The reconstruc
98
S. Ramanathan and A.A. Kassim tion error for the vertices falling in this set using the affine transform alone is less than 20 τ . • Third set (Type 3)  consisting of vertices whose motion cannot be described effectively using affine transforms. These are encoded using DPCMbased techniques.
The ability of ICPbased motion segmentation to divide the vertices in the mesh into distinct sets helps achieve efficient compression. This is because the motion of Type 1 and Type 2 vertices, which constitute over 70% of the total, can be described using a few affine transformations and residual errors. Also, the residual errors can be adaptively coded using variable number of bits for different groups of vertices in order to maintain the animation smoothness. ICPbased compression produces two types of meshes: I and P . I (Intra) meshes are encoded using static mesh compression techniques and complete information can be obtained by decoding the I mesh without any reference. P (Predicted) meshes contain only the differences from the temporally previous I or P mesh and chiefly contribute to compression. The difference needs to be added to the reference mesh data in order to obtain the complete P mesh information. Although dynamic geometry compression is inherently lossy, it is still acceptable if the reconstructed animation is ’perceptually lossless’, as there is a tradeoff between compression and quality. For P meshes, the following information needs to be encoded: • Affine matrices and vertices associated with each affine transform matrix. • Error values associated with the vertex positions for Type 2 and Type 3 vertices. To compactly encode the above information, the following procedure is used: • Vertices whose motion can be described using affine transforms are given the symbol P. Every P vertex is associated with a patch index to denote the associated affine transform. When the patch index of the vertex hasn’t changed from the reference, a symbol N is used to signify no change and the previous patch index is used. • Type 2 vertices are represented using the symbol E to denote error information. • Type 3 vertices encoded using DPCM techniques are assigned the symbol D. • As affine transforms associated with the vertex clusters exhibit a high spatiotemporal correlation, the differences between the affine matrices are quantized and encoded. • The number of bits used to encode error data for Type 2 and Type 3 vertices is determined by the PSNR. More bits are added to encode error for vertices whose PSNR is below a certain threshold. • Vertex symbols, affine matrices and residual errors are encoded using Arithmetic coding. The process of reconstructing mesh geometry from P meshes is outlined in Fig 7. When there is large intermesh motion, the same vertex may be associated with a number of affine transforms and many vertices may require error information to be encoded, which results
A Survey of Popular 3D SoftBody Animation Compression Approaches
99
in considerably lower compression ratios. Therefore, such meshes are encoded as I meshes for which only the spatial coherence in mesh geometry is exploited. The Edgebreaker algorithm [34] is used for encoding I meshes. Insertion of I meshes helps maintain animation smoothness with a marginal reduction in compression. Also, periodic transmission of I meshes is necessary while transmitting data over noisy channels and for enabling random access to the animation sequence.
Figure 7. Reconstruction of mesh geometry from P meshes.
3.2.1. Performance Metrics The Signal to Noise Ratio (SNR) is extensively used for evaluating the performance of dynamic geometry coding algorithms. The SNR and PSNR (Peak Signal to Noise Ratio) are considered objective measures for comparing the quality of the compressed data against the original data. The following definition for Peak Signal to Noise Ratio (PSNR) [43] is for evaluating the reconstruction quality of the encoding scheme. P SN R = −10 log10 P M SE
(10)
where P M SE is the Peak Mean Square Error per vertex given by P M SE =
1 Nn
Pn
1 j=1 3
P
i=x,y,z (vji (t) R2
− v¯ji (t))2
(11)
where R is the maximum intermesh displacement for the entire animation sequence, Nn and n denote the number of vertices that have moved between two consecutive meshes and the total number of mesh vertices respectively. The PSNR provides a quantitative measure of the animation smoothness. Another performance metric used for comparing various compression algorithms is the compression ratio which is defined as the size of the original data to the encoded data. The
100
S. Ramanathan and A.A. Kassim
perframe compression ratio (CR) is calculated as follows: CR =
(Bits for raw data) (Encoded vertex data bits + Affine transform bits + Error bits)
(12)
A number of encoding schemes also use Distortion Factor da (also termed KGerror ), to evaluate reconstruction quality and is defined as da = 100
ˆ kB − Bk kB − C(B)k
(13)
where B is a 3V × F matrix representing the geometry of the V vertices in the F frames ˆ represents the reconstructed animation geometry and C(B) of the original animation, B contains the average vertex positions for the animation. Likewise, the compression performance is alternatively measured using encoded Bits per Vertex per Frame, (bpvf ), which is related to the compression ratio as given by the following equation.
bpvf =
3.3.
Bits for encoding each vertex Raw data bits (total) = = Compressed bits (total) F ( ) FV
PF
96
i=1
CRi
(14)
F
Impact of Vertex Clustering on Compression Performance
In this section, we study the impact of vertex clustering on dynamic geometry compression by comparing the aforementioned performance metrics for the clustering schemes described previously. 3.3.1.
Test Animations
Four animation sequences, namely, Chicken2 , Face3 , Cow and Dance were used for comparing the clustering schemes. The Chicken animation contains 400 frames with each mesh in the sequence consisting of 3029 vertices and 5664 triangles. The animation is highly nonlinear with the motion becoming extremely rapid after frame 260. The Face sequence contains a realistic animation of a talking human face in various poses and exhibiting various facial expressions as well. There are 952 frames in the sequence with 757 vertices and 1468 triangles per mesh. The Cow (2904 vertices, 5804 triangles, 204 frames) animation is also a high motion sequence while the Dance sequence (7061 vertices, 14118 triangles, 201 frames) depicts a person performing various dance movements. Due to the similar nature of motion throughout the Dance animation, only the first 100 meshes were used in the experiments. 3.3.2.
Experimental Results
ICPbased dynamic geometry compression [13] using kway partitioning, with about 100 vertices per cluster, achieves an average compression of 45 for the Chicken animation. It can 2
The Chicken, created by Andrew Glassner, Tom McClure, Scott Benza, and Mark Van Langeveld for Microsoft Corporation (1996). 3 The Face sequence is the property of Visage Technologies.
A Survey of Popular 3D SoftBody Animation Compression Approaches
101
be observed that the Lloyd’s clustering (Fig 4) and spectral decomposition (Fig 6) schemes are able to segment the chicken’s neck from its torso more effectively compared to kway topology partitioning (Fig 3). The motion in frames 4060 and 200230 is mainly localized around the neck of the chicken as seen in Fig 8.
Figure 8. Frames 48, 56, 203 and 221 of the Chicken animation. For P frames, the perframe compression is directly proportional to the number of Type 1 and Type 2 vertices and inversely proportional to the number of Type 3 vertices. The number of Type 1 vertices, in turn, depends on how well the affine transforms computed using ICP based registration can represent the piecewise motion of the mesh. There exists a direct relationship between the number of Type 1 vertices registered using ICP and the accuracy with which the initial clusters input to ICP can represent the independent mesh regions. To illustrate this point, frame sequences 4060 and 200230 were encoded using Lloyd’s clustering (100 vertices per cluster) and spectral mesh decomposition (59 mesh components). Table 1 shows that the number of Type 1 vertices registered using ICP are much higher when Lloyd’s and spectral clustering are used instead to kway graph partitioning for both frame sequences. Better clustering produces more Type 1 vertices, and hence, better compression performance. For frames 4060, all mesh vertices are encoded as either Type 1 or Type 2. However, for frames 200230, affine transforms can effectively describe motion of only few mesh vertices, and therefore, there are many Type 3 vertices. For this frame sequence, while the number of Type 1 vertices registered using Lloyd’s and spectral clustering are about the same, more Type 2 vertices generated using spectral clustering produces higher compression. The latter part of the Chicken animation (frames 261399) is characterized by extensive motion. In these frames, reconstruction errors are associated with a large number of vertices and the reconstructed animation is noisy. Additional bits need to be encoded to improve animation quality at the expense of compression performance for these frames as seen from Fig 9. PSNR calculations are used to measure and improve the animation smoothness for this set of frames. For each frame in the animation, we measure the PSNR for Type 1, Type 2, Type 3 vertices and the entire reconstructed frame for analysis. For ensuring smooth animation reconstruction, the overall minimum PSNR varies for different sequences and depends on the nature of the mesh motion. A minimum PSNR of 35 db is required for the Chicken sequence (R = 0.7662), while a PSNR threshold of 20 db is sufficient for the low
102
S. Ramanathan and A.A. Kassim
motion Face sequence (R = 0.0673). The frame PSNR can be improved by (i) allocating extra bits for encoding Type 2 and Type 3 vertices (ii) using a smaller error threshold τ to register Type 1 vertices and (iii) transmission of I meshes. Examples of (i) and (ii) are shown in Fig 9. For frames 261399, Lloyd’s clustering performs better than spectral clustering and provides the best compression performance (Table 1). This is because the motion in these frames is concentrated around the chicken’s wings which is not well segmented by spectral clustering. The rapidness in motion also necessitates a number of meshes in the animation to be coded as I meshes which correspond to the minima in the compression curve. The performance of various dynamic geometry coding schemes for the Chicken animation is presented in Table 2. Clearly, the use of Lloyd’s clustering over kway partitioning improves performance of ICPbased compression by 3.8% under similar distortion (da ). Table 1. Impact of vertex clustering on compression performance for frame sequences (a) 4060, (b) 200230 and (c) 261399 of the Chicken animation. The table contains the mean values of Type 1 count, Type 2 count, CR and PSNR for the frame sequence under consideration. The number of Type 1 vertices and compression ratios increase with improvement in quality of input clusters. Frame nos. 4060
200230
261399
Clust. Algo.
Type 1 count
Type 2 count
CR
PSNR
kway
733
2296
52
66.1
Lloyd’s
746
2283
52.2
66.5
Spectral
839.6
2189.4
52.8
67
kway
470
1787
45.6
63.7
Lloyd’s
604
1632
47.2
63.9
Spectral
603
1925
48.3
64.6
kway
264
1406
38.3
51.1
Lloyd’s
357
1299
39.9
50.8
Spectral
296
1407
39.1
51.1
For the Face sequence, the interframe motion is very low and very few vertices are registered with error threshold τ = ν/4, where ν is average interframe motion, for many frames. The compression results for the different clustering schemes for τ = ν/4, ν/3 and ν/2 are shown in Table 3. Clearly, spectral clustering produces higher compression performance than Lloyd’s or kway partitioning. The ability of the spectral clustering algorithm to accurately segment the various face regions (Fig 6) enables the ICP module to register a maximum number of Type 1 vertices. This leads to a major improvement in the compression performance even when the encoded mesh is small in size. The compression obtained using spectral clustering is 9.7%, 15% and 7.8% higher than kway partitioning for τ equal to ν/4, ν/3 and ν/2 respectively. However, the compression obtained using Lloyd’s and kway partitioning is very similar even though Lloyd’s clustering produces better quality
A Survey of Popular 3D SoftBody Animation Compression Approaches
(i)
(ii)
(i)
(a)
103
(ii)
(b)
Figure 9. (a) Reconstructed Chicken frame 286 with (i) PSNR = 25.7 db (CR = 46.6) and (ii) PSNR = 43.2 db using τ = ν/11 and 6 bits for error encoding (CR = 46.3). (b) Reconstructed frame 320 with (i) PSNR = 31.8 db (CR = 49.5) and (ii) PSNR = 39.7 db using τ = ν/7 and 5 bits for error encoding (CR = 49.3).
(a)
(b)
(c)
Figure 10. Frames of the Cow and Dance animations partitioned using (a) kway (b) Lloyd’s and (c) Spectral clustering.
104
S. Ramanathan and A.A. Kassim
Table 2. Performance of various dynamic geometry compression algorithms for the Chicken animation (uncompressed file size = 13.9 MB). Table contains mean values of CR and da (wherever available). Compression algorithm
CR
da
Motion compensated compression [1]
10
Motion vector predictionbased compression [46]
18.3
Timedependent geometry compression [25]
27
PCA Representation [3]
39.8
Connectivityguided connectivity compression [38]
33
0.13
Clustered PCA Analysis [35]
34.3
0.076
Partitioning based compression [13]
45.3
0.11
[13] with spectral
46.6
0.12
[13] with Lloyd’s
46.7
0.12
Local PCA Analysis [4]
64
0.057
clusters. This is possibly because a marginal improvement in cluster quality does not significantly improve the number of Type 1 vertices for the lowmotion, smallsized Face mesh sequence. A PSNR threshold of 20 db is sufficient to smoothly reconstruct the animation. Table 3. Compression performance of the different clustering schemes algorithms at various values of τ for the Face animation. Table contains mean values of CR and PSNR. τ ν/4
ν/3
ν/2
Clust. algo.
Type 1 count
CR
PSNR
kway
182
41.9
44.3
Lloyd’s
165
41.2
44.5
Spectral
268
45.9
43.8
kway
372
53.3
39.9
Lloyd’s
367
52.9
40
Spectral
526.8
61.3
39.2
kway
670
69.5
34.4
Lloyd’s
675
69.2
33.9
Spectral
710
74.9
34.9
Some partitioned frames of the Cow and the Dance animations are shown in Fig 10.
A Survey of Popular 3D SoftBody Animation Compression Approaches
105
A minimum PSNR of 35 db is required to smoothly reconstruct the animation for both sequences. It is evident from the figure that kway and Lloyd’s produce more mesh clusters compared to spectral clustering. As vertex clustering is performed solely based on proximity for kway and Lloyd’s clustering, the cluster sizes affect the compression and SNR performance as observed in [13]. While ICP works well on smallsized clusters, a large number of mesh clusters are associated with increased processing time and reduced compression performance (as more affines need to be encoded). Also, largesized clusters produce registration errors and consequently, a degradation in compression and SNR performance. We observe that the best compression and SNR performance is achieved for cluster sizes of 100 and 125 for kway and Lloyd’s clustering respectively. Table 4. Compression performance of the different clustering schemes algorithms for the Cow and Dance animations. Mean values of CR and PSNR are listed in the table. Anim. Cow
Dance
Clust. algo.
Type 1 count
Type 2 count
CR
(% incr.)
PSNR
kway
351
1386
40.3

43
Spectral
332.3
1597
41.7
3.5
44.7
Lloyd’s
368
1581
42
4.2
42.5
kway
312
4038
41.4

42.2
Lloyd’s
414
3993
41.9
1.2
42.9
Spectral
740
4765
45.8
10.6
41.5
The Cow animation sequence is characterized by high motion and I meshes need to be encoded frequently after frame 100. Lloyd’s clustering enables most efficient encoding of the mesh motion as shown in Table 4. The number of Type 1 vertices is minimum for spectral clustering but outperforms kway partitioning as there are more registered Type 2 vertices. The poor performance of spectral compression for the Cow and the latter part of the Chicken animations underline the limitations of semantic mesh decomposition. This is because the mesh decomposition is purely based on the intrinsic geometric structure of the mesh. While it is difficult to achieve accurate segmentation of the mesh into distinctive components, efficient segmentation of coherent motion regions can only be performed by exploiting the motion cues available from the animation. Overall, about a 4% improvement in compression performance is obtained when Lloyd’s clustering is used instead of graph partitioning. On the other hand, spectral clustering performs exceedingly well for the Dance animation. The increased number of ICP registered Type 1 vertices using spectral clustering improves compression performance by over 10% for the Dance animation as shown in Table 4.
3.4.
Comparison with PCABased Algorithms
As seen from the experimental results, the clustering scheme employed to segment the mesh for motion prediction greatly affects compression performance. The kway parti
106
S. Ramanathan and A.A. Kassim
(a)
(b)
Figure 11. (a) Clusters generated for the Chicken and Cow in [35]. (b) Vertex clusters for the Chicken, Cow and Dance meshes in [4]. Figures adapted from [35, 4]. tioning, Lloyd’s clustering and Spectral mesh decomposition are static mesh segmentation techniques that segment the mesh to be encoded into smaller pieces without any temporal considerations. For encoding motion in dynamic mesh sequences, temporal cues can also be used to group vertices likely to undergo similar motion. Motionbased segmentation can facilitate identification of those ”pieces” that cannot be easily detected using static mesh decomposition. For example, for the human figure in the Dance sequence, segmentation of the arms and limbs is achieved by spectral clustering (Fig 10). Motion cues can be used to achieve further segmentation around articulated joints like the elbow and knee, and clearly, the segmented parts will correspond to the coherent motion regions better. Next, we look at two recent algorithms that segment the mesh based on motion characteristics to reinforce the idea that meaningful mesh segmentation greatly impacts performance of 3D dynamic mesh coding algorithms . Two approaches that perform motionbased clustering to efficiently represent motion have been found to achieve high compression performance. Sattler et al. [35] propose the clustered PCA (CPCA) approach to dynamic geometry compression that can identify the mesh parts undergoing coherent motion over time. The vertex trajectories are clustered using Lloyd’s clustering [29] in combination with PCA to segment the coherent mesh parts. Each mesh part is then compressed using PCA on the complete animation as performed in [3]. This method results in higher compression than standard PCA and PCA+LPC approaches while producing lesser distortion. Also, Amjoun et al. [4] propose local PCAbased compression, where the mesh is segmented into clusters based on local motion characteristics and a local coordinate system is defined for each cluster, with respect to which the cluster motion is encoded. Table 5 compares the performance of ICPbased compression using spectral clustering with CPCA and LPCAbased compression for similar distortion. From the table, it is evident that for the bpvf values for ICPbased compression using spectral decomposition are much lower than those for Clustered PCA for comparable values of distortion. While LPCAbased compression performs better than CPCA or ICPbased coding for the Chicken sequence, ICP coding with spectral clustering achieves maximum compression for the Cow animation. This could be attributed to the inadequate segmentation achieved by the pure motionbased clustering schemes in [35, 4] as shown in Fig. 11. While motionbased clustering can produce meaningful segmentation of the mesh into components, e.g., wings and legs of the Chicken, more coherent mesh segments are obtained for
A Survey of Popular 3D SoftBody Animation Compression Approaches
107
Table 5. Comparison of CPCA and LPCAbased compression with ICP basedcompression using Spectral clustering for the Chicken and Cow animations. Animation
Chicken
Cow
CPCA
LPCA
ICP
bpvf
da
bpvf
da
bpvf
da
4.7
0.076
3.5
0.008
2.16
0.12
2.8
0.139
1.5
0.057
1.72
0.26
7.4
0.16
6.8
0.128
2.9
0.33
3.8
0.50
4.1
0.47
2.3
0.47
2.0
7.4
2.2
1.22
1.9
0.9
the Cow using spectral decomposition (Fig 10) compared to pure motionbased clustering. Therefore, an ideal vertex clustering scheme for 3D dynamic mesh coding needs to exploit both structural and motionbased cues for maximal compression performance.
4.
Conclusion
3D softbody animation compression is a nontrivial as nonplanar 3D mesh deformations cannot be described using wellknown video motion prediction techniques. A number techniques have been proposed for efficiently coding 3D animations, and in particular, 3D dynamic geometry. These 3D dynamic mesh coding algorithms can be divided into three major classes registrationbased, predictionbased and PCAbased multiresolution representation. Efficient vertex clustering for motion prediction enhances compression performance achieved using registration and pcabased coding. Our experimental results clearly demonstrate that meaningful mesh segmentation enhances compression performance of dynamic mesh coding algorithms. Using spectral mesh decomposition, which segments the mesh based on structural cues, performance of registrationbased dynamic geometry coding improves by as much as 10% (for the Dance animation), while CPCA and LPCAbased compression algorithms, that use motion cues for vertex clustering, also perform significantly better compared to other PCAbased coding schemes. Nevertheless, vertex clustering exclusively using structural or motion cues does not produce the best clusters for motion prediction Lloyd’s clustering outperforms spectral decomposition for the highmotion Chicken animation while cluaters generated using motion cues are inadequate for the Cow sequence. A hierarchical mesh segmentation scheme that initially segments the mesh based on structural cues followed by generation of finer vertex clusters through motion analysis appears to be best suited for dynamic geometry coding. Detecting motioncoherent vertex clusters could be the key in solving the animation compression problem. Recently proposed Bonebased animation [19], which involves detection of the elementary transformations that constitute mesh motion, offers an exciting prospect in this regard.
108
S. Ramanathan and A.A. Kassim
References [1] J. H. Ahn, C. S. Kim, C. C. Kuo, and Y. S. Ho. Motion compensated compression of 3d animation models. Electronic Letters, 37(24):1445–1446, 2001. [2] A.Khodakovsky, P.Schroder, and W.Sweldens. Progressive geometry compression. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 271–278, 2000. [3] M. Alexa and W. Muller. Representing animations by principal components. In EUROGRAPHICS, volume 19(3), pages 411–418, 2000. [4] Rachida Amjoun and Wolfgang Straβer. Efficient compression of 3d dynamic mesh sequences. Journal of the WSCG, 2007. [5] Y. Boulfani, F. Payan, and M. Antonini. Temporal waveletbased compression of 3d animated meshes using motionbased clustering. In Proceedings of the Workshop TAIMA’07. Tunisia, May 2007. [6] M. Brand and K. Huang. A unifying theorem for spectral embedding and clustering. In Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics, 2003. [7] M. Chow. Geometry compression for realtime graphics. In Proceedings of Visualization’97, 1997. [8] D. CohenOr, O. Remez, and D. Levin. Progressive compression of arbitrary triangular meshes. In Proceedings of Visualization ’99, pages 67–72, 1999. [9] G. Debunne, M. Desbrun, M.P. Cani, and A. H. Barr. Dynamic realtime deformations using space and time adaptive sampling. In SIGGRAPH 2001, Computer Graphics Proceedings, pages 31–36, 2001. [10] M. Deering. Geometry compression. In Proceedings of SIGGRAPH ’95, pages 13–20, 1995. [11] S. Gumhold and W. Strasser. Real time compression of triangle mesh connectivity. In Proceedings of SIGGRAPH ’98, pages 133–140, 1998. [12] S Gupta, K Sengupta, and A.A. Kassim. Registration, partitioning based compression of 3d dynamic data. IEEE Transactions on Circuits and Systems for Video Technology, 13(11):1144–1155, 2003. [13] Sumit Gupta, Kuntal Sengupta, and A.A. Kassim. Compression of 3d dynamic geometry data using iterative closest point algorithm. Computer Vision and Image Understanding, 87:116–130, 2002. [14] Igor Guskov and Andrei Khodakovsky. Wavelet compression of parametrically coherent mesh sequences. In SCA ’04: Proceedings of the 2004 ACM SIGGRAPH/Eurographics symposium on Computer animation, pages 183–192, 2004.
A Survey of Popular 3D SoftBody Animation Compression Approaches
109
[15] Bruce Hendrickson and Robert W. Leland. An improved spectral graph partitioning algorithm for mapping parallel computations. SIAM Journal on Scientific Computing, 16(2):452–469, 1995. [16] Bruce Hendrickson and Robert W. Leland. A multilevel algorithm for partitioning graphs. In Supercomputing, 1995. [17] Hugues Hoppe. Progressive meshes. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 99–108, 1996. [18] L. Ibarria and J. Rossignac. Dynapack: Spacetime compression of the 3d animation of triangle meshes with fixed connectivity. In Proceedings of the ACM SIGGRAPH Symposium on Computer Animation, 1999. [19] B. Jovanova, M. Preda, and F. Preteux. Mpeg4 part 25: A generic model for 3d graphics compression. In 3DTV08, pages 101–104, 2008. [20] Tapas Kanungo, David M. Mount, Nathan S. Netanyahu, Christine D. Piatko, Ruth Silverman, and Angela Y. Wu. The analysis of a simple kmeans clustering algorithm. In Proc of the 16th Annual Symposium on Computational Geometry, pages 100–109, 1991. [21] Z. Karni and C. Gotsman. Compression of soft body animation sequences. Computers and Graphics, 28:25–34, 2004. [22] Sagi Katz and Ayellat Tal. Hierarchical mesh decomposition using fuzzy clustering and cuts. ACM Transactions on Graphics, 22(3):954–961, 2003. [23] B. W. Kernighan and S. Lin. An efficient heuristic procedure for partitioning graphs. The Bell System Technical Journal, 49(2):291–307, 1970. [24] Rob Koenen. Overview of the MPEG4 standard. Moving Picture Experts Group, 2000. [25] J. Lengyel. Compression of time dependent geometry. In Symposium on Interactive 3D Graphics, pages 89–95, 1999. [26] J. Li and C.C. Kuo. A dual graph approach to 3d triangular mesh compression. In Proceedings of the IEEE International Conference on Image Processing, pages 891– 894, 1998. [27] X. Li, T. Toon, T. Tan, and Z. Huang. Decomposing polygon meshes for interactive applications. In Proceedings of the Symposium on Interactive 3D Graphics, pages 35–42, 2001. [28] R. Liu and H. Zhang. Segmentation of 3d meshes through spectral clustering. In Proceedings of Pacific Graphics, pages 298–305, 2004. [29] Stuart P. Lloyd. Least squares quantization in pcm. IEEE Transactions on Information Theory, 18(2):129–137, 1982.
110
S. Ramanathan and A.A. Kassim
[30] S. Moradoff and D. Lischinski. Synthesis of textural motion with hard constraints. In Proceedings of the 4th IsraelKored 6inational conference on geometric modelling and computer graphics., pages 123–128, 2003. [31] K. Muller, A. Smolic, M. Kautzner, P. Eisert, and T. Wiegand. Predictive compression of dynamic 3d meshes. In ICIP05, pages 589–592, 2005. [32] R. Pajarola and J. Rossignac. Compressed progressive meshes. IEEE Transactions on Visualization and Computer Graphics, 6(1):79–93, 2000. [33] F. Payan and M. Antonini. Waveletbased compression of 3d mesh sequences. In Proceedings of IEEE ACIDCAICMI’2005, 2005. [34] J. Rossignac. Edgebreaker: Connectivity compression for triangle meshes. IEEE Transactions on Visualization and Computer Graphics, 5(1):47–61, 1999. [35] Mirko Sattler, Ralf Sarlette, and Reinhard Klein. Simple and efficient compression of animation sequences. In SCA ’05: Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation, pages 209–217, 2005. [36] Ariel Shamir, Chandrajit Bajaj, and Valerio Pascucci. Multiresolution dynamic meshes with arbitrary deformations. In Proceedings of the conference on Visualization ’00, pages 423–430, 2000. [37] H.D. Simon. Partitioning of unstructured problems for parallel processing. In Proc. Conference on Parallel Methods on Large Scale Structural Analysis and Physics Applications, pages 135–148. Pergammon Press, 1991. [38] N. Stefanoski and J. Ostermann. Connectivityguided predictive compression of dynamic 3d meshes. In ICIP06, pages 2973–2976, 2006. [39] N. Stefanoski and J. Ostermann. Spatially and temporally scalable compression of animated 3d meshes with mpeg4/famc. In ICIP08, pages 2696–2699, 2008. [40] G. Taubin and J. Rossignac. Geometric compression through topological surgery. ACM Transactions on Graphics, 17(2):84–115, 1998. [41] D. Terzopoulos, J. Platt, A. Barr, and K. Fleischer. Elastically deformable models. In SIGGRAPH ’87: Proceedings of the 14th annual conference on Computer graphics and interactive techniques, pages 205–214, 1987. [42] C. Touma and C. Gotsman. Triangle mesh compression. In Proceedings of Graphics Interface, pages 26–34, 1998. [43] S. Varakliotis, J. Ostermann, and V. Hardman. Coding of animated 3d wireframe models for internet streaming applications. In Proc. International Conference on Multimedia and Expo., pages 353–356, 2001. [44] Yair Weiss. Segmentation using eigenvectors: A unifying view. In International Conference on Computer Vision, pages 975–982, 1999.
A Survey of Popular 3D SoftBody Animation Compression Approaches
111
[45] Ian H. Witten, A. Moffat, and Timothy C. Bell. Managing Gigabytes: Compressing and Indexing Documents and Images. Morgan Kaufmann Publishers, San Francisco, CA, 1999. [46] J.H. Yang, C.S. Kim, and S.U. Lee. Compression of 3d triangle mesh sequences based on vertexwise motion vector prediction. IEEE Transactions on Circuits and Systems for Video Technology, (12):1178–1184, December 2002. [47] H. Zhang and R. Liu. Mesh segmentation via recursive and visually salient spectral cuts. In Proceedings of Vision, Modeling and Visualization, pages 429–436, 2005. [48] Z.Karni and C.Gotsman. Spectral compression of mesh geometry. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 279–286, 2000.
In: Computer Animation ISBN 9781607415596 c 2010 Nova Science Publishers, Inc. Editors: J.S. Wright and L.M. Hughes, pp. 113127
Chapter 4
V IRTUAL E MOTION TO E XPRESSION : A C OMPREHENSIVE DYNAMIC E MOTION M ODEL TO FACIAL E XPRESSION G ENERATION U SING THE MPEG4 S TANDARD Paula Rodrigues1,∗, Asla S´a2,† and Luiz Velho3,‡ 1 Informatics Department, PUCRio, Brazil 2 TecGraf, PUCRio, Brazil 3 IMPA  Instituto de Matem´atica Pura e Aplicada, Brazil
Abstract In this paper we present a framework for generating dynamic facial expressions synchronized with speech, rendered using a tridimensional realistic face. Dynamic facial expressions are those temporalbased facial expressions semantically related with emotions, speech and affective inputs that can modify a facial animation behavior. The framework is composed by an emotion model for speech virtual actors, named VeeM (Virtual emotiontoexpression Model), which is based on a revision of the emotional wheel of Plutchik model. The VeeM introduces the emotional hypercube concept in the R4 canonical space to combine pure emotions and create new derived emotions. The VeeM model implementation uses the MPEG4 face standard through a innovative tool named DynaFeX (Dynamic Facial eXpression). The DynaFeX is an authoring and player facial animation tool, where a speech processing is realized to allow the phoneme and viseme synchronization. The tool allows both the definition and refinement of emotions for each frame, or group of frames, as the facial animation edition using a highlevel approach based on animation scripts. The tool player controls the animation presentation synchronizing the speech and emotional features with the virtual character performance. Finally, DynaFeX is built over a tridimensional polygonal mesh, compliant with MPEG4 facial animation standard, what favors tool interoperability with other facial animation systems.
Keywords: Facial Animation, Talking Heads, Expressive Virtual Characters. Email address:
[email protected] Email address:
[email protected] ‡ Email address:
[email protected]
∗ †
114
1.
Paula Rodrigues, Asla S´a and Luiz Velho
Introduction
Character Animation is one of the key research areas in Computer Graphics and Multimedia. It has applications in many fields, ranging from Entertainment, Games, Virtual Presence and others. Within the general area of character animation, the modeling and animation of faces is perhaps the single most important and challenging topic. This is because the expressiveness and personality of a character is communicated by facial expressions. The research in face modeling and animation dates back to the seminal work of Frederic Parke in the early 1970’s [9]. Since that time, the area experienced a very intense development. Practically all problems related to generating the shape and motion of faces have been deeply studied. This body of research includes a plethora of techniques for capturing the geometry and appearance of human faces, learning facial expressions, modeling muscles and the dynamics of deformations, together with realistic rendering methods for skin and hair. Despite of the amazing progress in the area of facial animation, there is one problem which is still open, and poses a great challenge to researchers: it is how to incorporate emotion on animated characters! This is the crucial step toward believable virtual characters. While a talented artist with the help of powerful modeling and animation tools can manually create a very expressive character, the same is not true for an automatic or even semiautomatic animation system. It is our intent in this paper to address the challenge of generating believable virtual characters automatically by incorporating a computational emotion model. We propose a comprehensive emotion model for facial animation that considers the various aspects of an expressive character. The model is implemented using a system based on the guidelines of the MPEG4 standard for faces. The rest of the paper is structured as follows: in next section the emotion models and related work are discussed. In Section 3. we propose an emotion space named emotion Hypercube. The emotion hypercube enables pure emotion combinations in order to generate derived emotions in a natural way. We then describe a derived emotion classification scheme based on the proposed space. In Section 4. some affective phenomenon, like mood and personalitity, are incorporated into the model. In Section 5. the VeeM (Virtual emotiontoexpression Model) is formalized, it consists of a representation of a facial expression from an emotion description with dynamic features. Section 6. presents an overview of the MPEG4 standard to facial animation. In Section 7. we explain how the proposed model is implemented. Finally, conclusions and future work are discussed in the Section 8..
2.
Emotion Models and Related Work
Several models have been proposed to explain what is an emotion and how it is represented [13]. Here we summarize the main approaches related to this topic. Basic Emotion is probably the most wellknown emotion approach. The reason for this is its association with universal recognized emotions [5]. Nevertheless, there is not a consensus for defining which are the basic emotions yet.
Virtual Emotion to Expression
115
Figure 1. Six universal basic emotions defined by Ekman (surrounded by a dashed line) and additional Plutchik basic emotions (accepted and aware). As discussed in [7], the basic emotion approach aims to build a psychologically irreducible emotion set, which means that these emotions cannot be derived by any other emotion and new emotions are derived from them. Note that these considerations matches to the mathematical definition of a basis. As mentioned above, the best known method used to study basic emotion is by observing facial expressions. Through this, Ekman [5] defined six universal emotions: anger, fear, disgust, surprise, joy and sadness, illustrated in Figure 1. An extension of Ekman’s model to the basic emotions representation is the approach proposed by Plutchik [11], where two additional basic emotions are defined (emphasized in Figure 1): anticipation (also referred as aware, curiosity or interest) and acceptance (also referred as trust). Plutchik describe its basic emotion as pairs of opposite emotions. Plutchik emotions are disposed in a wheel of opposed pairs, as illustrated in Figure 2. Derived emotions are defined as the combination of two neighbor basic emotion or as a basic emotion intensity variation. In the emotion literature, the Plutchik wheel is considered to be enough to span most of human emotion state. Examples of computational systems that use the basic emotion approach to generate their facial expressions are: SMILE [6], eFASE [3], EEFAS [15], Cloning Expression [12], the MPEG4 Standard [8] and the CSLU Toolkit [2]. In addition to any model of emotion, the emotion perception becomes single for each person due to factors such as mood and personality. The approach proposed in [10] is to model mood as a simple and unique dimension: good mood and bad mood. A more complete approach proposed by Thayer in [16] uses emotion spaces to represent mood in two dimensions (calm/tense and energy/tired), resulting in four mood emotional states: Energeticcalm, Energetictense, Tiredcalm and Tiredtense. An example of computational system that incorporates mood to the character using Thayer’s model to generate dynamic facial expressions is the DER [14] [15]. Personality is another important aspect to define the action and reaction of each person as unique even when submitted to the same situation of another person. Until now, there is not a formal consensus to define the personality trait of a person, however the Big
116
Paula Rodrigues, Asla S´a and Luiz Velho
Figure 2. Plutchik wheel. Five (or Big OCEAN) model is wellknown. In this model, each first letter of OCEAN word defines a dimension in the personality trait: Openess to experience, Conscientiousness, Extraversion, Agreeableness, Neuroticism 1 . Emotions are not static. They are experienced by each individual differently because of characteristics such as personality and mood, referred here as affective phenomenas. Additionally, affective phenomena also interferes in the reaction of each person when receives a stimulus, defining the emotion sustaining time. Our aim is to propose a computational system which incorporates and implements a robust model based on basic emotions. So, the Plutchik model [11] is revisited and generalized to allow the description of new emotions from the eight basic emotions as well as the incorporation of emotion dynamics in a comprehensive manner in order to allow the automatic generation of believable virtual characters.
3.
The Emotion Hypercube
The emotion description space proposed in this paper is a reinterpretation of Plutchik’s emotional wheel. We consider a family of emotions as a set of emotions composed by different intensity levels of a given basic emotion Ei . Plutchik has considered a discrete set of three levels of intensity of a given basic emotion, namely, an attenuation of the pure basic emotion, the pure basic emotion itself and an extrapolation of the pure basic emotion, as described in Table 1. Assuming that the Plutchik’s set of basic emotions is psychologically irreducible, our 1
More information about BIG OCEAN model can be found at http://www.answers.com/topic/bigfivepersonalitytraits (accessed in 22jan2008).
Virtual Emotion to Expression
family
Table 1. Basic emotions attenuation basic emotion extrapolation
(axis)
(αi  < 1)
(αi  = 1)
(αi  > 1)
1 (x+) 2 (y+) 3 (z+) 4 (w+) 5 (x) 6 (y) 7 (z) 8 (w)
serenity annoyance acceptance distraction pensivenes apprehension boredom interest
joy anger trust surprise sadness fear disgust anticipation
ecstasy rage admiration amazement grief terror loathing vigilance
117
goal is to define a basis that represents the space of derived emotions. In order to do so, we define an emotion axis, denoted by e, as composed by a pair of opposed families as stated in Plutchik emotional wheel. Thus, the eight basic emotions are arranged in 4 emotion axes denoted by x, y, z e w. The level of intensity of an emotion axis is modeled as a continuum parameter represented by the real value αi , where αe ∈ [−γ, +γ], with γ ≥ 1. A basic emotion is mapped to the intensity level 1 and its opposed basic emotion is mapped to 1. The neutral emotion is mapped to level 0. Though the intensity level αi  = 1 matches to a basic emotion, αi  < 1 is the emotion attenuation and αi  > 1 is the emotion extrapolation (See Table 1). The more obvious space to be adopted to represent the space of emotions is Rn , where n is the number of opposed pairs of emotions to be considered in the model. Since we adopt 4 emotion axes, limited to the interval [−γ, +γ] we obtain an emotion hypercube H = [−γ, +γ] × [−γ, +γ] × [−γ, +γ] × [−γ, +γ] A given emotional stimulus can then be completely defined by a vector of intensity levels ~u = (αx , αy , αz , αw ) that represents the character’s state of emotion. The emotion hypercube H is a comprehensive emotion space useful to facial expression generation. Observe that the proposed space can be easily extended to more than four axes if a new pair of opposed basic emotions is incorporated, observing that the new axis should be independent from the previously defined ones in order to preserve the property that the set of emotion axes is a basis to H.
3.1.
Derived Emotions
The combination of basic emotions is an adequate approach to use with virtual talking heads. Plutchik [11] states that two basic emotions can be combined if they are not opposed to each other. The emotion hypercube H leads to a simple way to derive combined emotions. For instance, binary combinations can be defined by setting two intensity levels to zero. Thus, given two non opposite basic emotions Ei and Ej , their combination is defined by their non zero intensity values αi and αj . Although the original Plutchik model restricts combinations to adjacent basic emotions, we do not adopt this restriction in the proposed model.
118
Paula Rodrigues, Asla S´a and Luiz Velho
We emphasize that by using the hypercube model, nary combinations, where n is greater than two, are easily stated, although in the literature combinations of more than two emotions are not considered, probably due to the combinatorial growth of the number of derived emotions to be considered and semantically interpreted. Observe also that the restriction to not combine opposite emotions is intrinsic to the disposition of opposite emotions in the same axis.
3.2.
Binary Derived Emotions Taxonomy
In this section we propose a natural and complete taxonomy of the binary derived emotions that extends Plutchik’s work [11] by defining additional semantic interpretations. Two emotion axes ei and ej define an emotion plane of derived emotions Πij . We will refer to each quadrant of a plane as a sector of derived emotion. The combination of the 4 axis, 2 by 2, results in 6 derived planes. In Table 2 each sector of each plane is named with a semantic interpretation of the derived emotion. We emphasize that the semantic interpretation of each sector corresponds to the combination of two basic emotions with level of intensity αi  = 1. If the basic emotion intensity is attenuated or extrapolated the semantic interpretation of the derived emotion may change.
4.
Modeling Affective Phenomena in H
Personality, mood, physical environment and others factors interfere on the expression of emotion. We refer to such factors collectively as affective phenomena and when applied to an individual character as its affective pattern. Thanks to affective phenomena people under the same emotion stimulus react and feel it in different ways and intensities according to its affective pattern. Affective phenomena are more enduring emotions and usually occur as an emotional background of much lower intensity than emotional episodes. In order to model the influence of an affective phenomena on an emotional episode we assume that the basic emotions parameters are defined considering a neutral character while an affective pattern is modeled as a distortion (warping) of the original emotion space.
4.1.
Affective Pattern Description
Suppose that in a given instant a happiness stimulus is augmented, then it is reasonable to think that the level of intensity related to joy also augments even if the affective pattern of the character is biased to sadness. Thus it is reasonable to state that an affective pattern can be defined by a set of monotonic functions fi : [−γ, γ] → [−γ, γ]. The new intensity level on each emotion axis is α˜i = fi (αi ), corresponding to the original stimulus i with i = x, y, z, w is one of the emotion axes from H. Then, the vector ~u ˜ = (α˜x , α˜y , α˜z , α˜w ) is an instance of the affective pattern in H. This approach to modeling affective patterns is general and capable to describe nonneutral characters behavior. The set of functions χ = {fx , fy , fz , fw } characterize an affective pattern. The difficulty resides in determine such functions in a meaningful manner.
Virtual Emotion to Expression
119
Table 2. Taxonomy Derived
basic
basic
derived
Plane
emotion i
emotion j
emotion
Πxy
Joy Joy Sadness Sadness
Fear Anger Fear Anger
thrill negative pride despair envy
Πxz
Joy Joy Sadness Sadness
Trust Disgust Trust Disgust
love morbidness sentimentalism remorse
Πxw
Joy Joy Sadness Sadness
Anticipation Surprise Anticipation Surprise
optimism absent pessimism disappointment
Πyz
Fear Fear Anger Anger
Trust Disgust Trust Disgust
submission distress dominance contempt
Πyw
Fear Fear Anger Anger
Anticipation Surprise Anticipation Surprise
anxiety awe aggression outrage
Πzw
Trust Trust Disgust Disgust
Anticipation Surprise Anticipation Surprise
positive pride curiosity cynicism defeat
Note that the physical environment can also be described by a set of such functions. For example if a character is in a formal environment the emotion expression tends to be attenuated, that is, α˜i < αi , thus the χ set should be a set of emotion attenuation functions.
4.2.
The Dynamics of an Affective Pattern
It is important to notice that affective patterns are not static. For instance, personality traits evolve in lifetime while mood can change during the day time or as a reaction to an emotional episode. The emotion dynamics can be modeled as a function of emotion and time f : (H, t) → H. Semantic aspects also need to be taken into account in the context of emotion dynamics. For example: a character cannot mix opposite emotion families and cannot swap them
120
Paula Rodrigues, Asla S´a and Luiz Velho
Figure 3. Emotion vector generation. frequently, otherwise a conflict or emotional instability is established. The combination of basic emotions with the farthest families generates destructive emotions, because they are close to a conflict state.
5.
VeeM: Virtual Emotion to Expression Model
Figure 4. VeeM architecture. The H space is used to define a given character’s state of emotion in an instant of time. In order to simulate a believable character animation, the affective patterns and its dynamic characteristics as well as dynamic characteristics such as head movements and eyes blink have to be combined together. The dynamics of the facial expression of a given emotion will be treated by VeeM. In Figure 4 a schematic view of the proposed model is illustrated.
Virtual Emotion to Expression
121
There are other equally important dynamic characteristics to be considered when aiming to generate a believable character, namely the speech dynamics, the eyes movements dynamics and the head movement dynamics, refered in literature as nonverbal movements. The difference between affective patterns and nonverbal movements is their domain of action, instead of affect an emotional state, the nonverbal movements act directly on the facial expression domain F. Thus, can be modeled as a function of emotion and time g : (H, t) → F. Speech interferes on mouth movements and a lot of work has been done in order to define the movements that characterize it. The subject is complex and depends on the spoken language and character’s culture. But the consensus is to define visemes to represent the phonemes and combine them to produce speech visualization. A viseme is a visual representation of a phoneme that describes the facial movements that occur alongside the voicing of phonemes. Head and eyes movements can be treated as random functions (random noise) considering the fact that it is uncommon to keep them fixed. Other complementary approach is to model them as directed reactions that simulate the attentional focus. In both cases the function that model these behaviors should interfere in the head and eyes position as well as the motion velocity. Speech and nonverbal characteristics such as eyes and head movements should be combined with the emotion description in order to produce the resultant facial expression [13]. VeeM also incorporates dynamics of reaction to an emotion episode (directly related to stimulus or facts which elicits an emotion). The emotion expression reaction depends on the affective pattern. The reaction curve rj (t), similarly to the description of emotional stimulus given by Picard [10], has 3 stages: onset (usually very fast), sustain and decay stages, as illustrated in Figure 5.
Figure 5. Emotion reaction curve. Emotion transition is incorporated as a blending between two subsequent emotions.
6.
The MPEG4 Standard
Once the domain of emotion description H has already been defined we now turn to the description of the range space, that is, the space where the emotions are to be visualized:
122
Paula Rodrigues, Asla S´a and Luiz Velho
the facial expression space F. In general, different works in facial animation generation develop their own facial model without worrying about compatibility. Often the only common approach among these works is the emotion model, which usually are the six Ekman’s basic emotions. Aiming to define a standard, the MPEG4 [4] [8] agreed a set of control points to define a facial model proposing a facial polygonal mesh that can be considered universal. It is important to mention that the MPEG4 facial animation standard is the first effort in this direction. The MPEG4 specifies a face model in its neutral state, a number of feature points (FPs) on this neutral face as reference points, and a set of facial animation parameters (FAPs), each corresponding to a particular facial action that deforms the face model starting from the neutral state. In this work we identify the facial expression space F to the space of FAPs. A neutral face in the MPEG4 standard must consider the following properties: • Gaze is in the direction of zaxes; • All face muscles are relaxed; • Eyelids are tangent to the iris; • The pupil is one third the diameter of the iris; • Lips are in contact; the line of the lips is horizontal and the same height at lip corners; • The mouth is closed and the upper teeth touch the lower ones; and • The tongue is flat, horizontal with the tip of the tongue touching the boundary between upper and lower teeth. In order to define FAPs for arbitrary face models, MPEG4 defines facial animation units (FAPUs) that servers to scale FAPs for any face model. FAPUs are defined as fractions of distances between key facial features. These features, such as eye separation are defined on a face model that is in the neutral state. From the FAPUs definition, MPEG4 specifies 84 FPs on the neutral face. The main purpose of these FPs is to provide spatial reference for defining FAPs. FPs are arranged in groups such as cheeks, eyes and mouth. The location of these FPs has to be known for any MPEG4 compliant face model. The FAPs are based on the study of minimal perceptible actions and are closely related to muscle actions [8]. The 68 parameters are categorized into 10 groups related to parts of the face (Table 3). FAPs represent a complete set of basic facial actions including head motion, eye and mouth control. The FAP group 1 contains two highlevel parameters: visemes and expressions. The MPEG4 standard defines 14 visemes to represent english phonemes [13] [8]. The expression parameter defines the six basic facial expressions. Facial expressions are animated by a value defining the excitation of the expression. A benefit of using FAP Group 1 is that each facial model preserves its personality in a sense that a specific face model preserves its particular version of facial expression.
Virtual Emotion to Expression
123
Table 3. FAPs groups Group
Number of FAPs
1. visemes and expressions 2. jaw, chin, inner lowerlip cornerlips, midlip 3. eyeballs, pupils, eyelids 4. eyebrow 5. cheeks 6. tongue 7. head rotation 8. outerlip positions 9. nose 10. ears
2 16 12 8 4 5 3 10 4 4
FAP groups 2 to 10 are considered lowlevel parameters. They specify precisely how much a FP of a face has to be moved for a given amplitude [8]. An MPEG4 facial expression is then obtained by moving the Feature Points (FP) associated to the FAPs. Each basic emotion has a set of FAPs defined to produce its correspondent facial expression. We call signal the facial expression related to a specific emotion and we denote the set of FAP values that define the emotion j as ~vj . The facial animation sequence is obtained by specifying FAP values at each time instant, ~vjt , according to an input timeline. We adopt the MPEG4 Standard as our space of facial expressions.
7.
VeeM applied on an MPEG4 Face Model
As already mentioned, the VeeM permits to generate different facial emotions taking into account affective patterns and the emotion dynamics. A challenge turns into emotion visualization using an MPEG4 face model. That is, to set FAPs parameter values from the defined emotion. In this work we adopt the open source Xface [1] face model. The model had to be extended since the original implementation defines only the six Ekman’s basic emotions as conventioned in MPEG4 standard. Figure 6 shows the eight basic emotions of VeeM specified in the Xface MPEG4 polygonal mesh. Speech is a key element to generate a dynamic and natural facial animation combined with the character’s emotional state. The implemented system uses an audio file with the character’s speech as input. This file goes through a speech recognition stage generating, as output, the speech phonemes [13]. The produced phonemes are mapped into the 14 MPEG4 visemes, each one mapped into a set with 26 FAPs that define the mouth region. The FAPs values generated at each stage of facial animation specification (verbal expressions, nonverbal expressions and emotions) need to be blended to define the final value
124
Paula Rodrigues, Asla S´a and Luiz Velho
Figure 6. VeeM basic emotions viewed in a MPEG4 facial animation. that a FAP takes in each animation frame. For each animation frame, this blending is done in two stages (Figure 7): • Facial expression for the emotion (pure or derived); and • Facial expression for the resulting emotion and viseme blending.
Figure 7. FAP blending to generate the final facial expression for each animation frame. The first stage can receive as input a single vector of FAPs, two vectors of FAPs or nothing. Whatever the input, the results are always two vectors with dimension 68: a vector containing the mask of FAPs (signaling if a FAP participates or not in the current frame) and another with the values for the participants FAPs.
Virtual Emotion to Expression
125
If a vector is not provided as input, the first stage result is the FAPs vector for natural emotion and a mask containing the value 0 for the 26 FAPs of the mouth region and for the FAP 1 (viseme), and the value 1 for the other FAPs. This configuration purpose is the face remains in the natural state, and its expression influenced only by the viseme FAPs. In the case where a single FAPs vector is provided, the first stage output is the FAPs vector itself and a mask containing a value of 1 for all FAPs, with the exception of FAP 1 (viseme). When the first stage receives two vectors of FAPs, a blending is necessary to define the derived emotion. This blending is obtained by calculating the mean between the values of each FAP, except for FAPs 1 and 2 which are the highlevel FAPs. In FAP 1 the value 0 is set, once it represents the viseme. In FAP 2, which describes the emotion, the value 0 is assigned, because this FAP does not has influence in the animation, since only lowlevel FAPs are considered by the module of synchronization. If the two FAPs vectors are provided as input, the output mask passes through the same process as in the case of only one input vector. The second blending stage generates the resulting FAPs vector using as input the first stage output, the viseme FAPs vector and a viseme contribution parameter indicating a factor blending, denoted by β, where 0 ≤ β ≤ 1. Before applying the visemeemotion blending rule, the second stage creates a mask for the viseme FAPs vector. If there is a viseme FAPs vector, the generated mask vector has value 1 for the 26 FAPs related to mouth region and value 0 for the other FAPs. If there is not a viseme FAPs vector specified as input of second stage, the mask is created with all elements of the vector having value 0. The blending rule is simple. Denoting F APemo and M ASKemo , respectively, the resulting emotion FAPs vector and mask vector; F APvis and M ASKvis , respectively, the viseme FAPs vector and mask, both created in second stage; and β as the viseme contribution factor, it is possible to apply for each index i of resulting FAPs vector F APres the following algorithmic logic: • if (M ASKvis = 0) ⇒ F APresi = M ASKemoi ∗ F APemoi • if ((M ASKvis 6= 0) ∧ (M ASKemo = 0)) ⇒ F APresi = M ASKvisi ∗ F APvisi • if ((M ASKvis 6= 0) ∧ (M ASKemo 6= 0)) ⇒ F APresi = (1 − β) ∗ M ASKemoi ∗ F APemoi + β ∗ M ASKvisi ∗ F APvisi With this rule, it is possible to note that the value β = 1 is related to the blending visemes and emotions strategy where visemes is overlapping emotions, ignoring the emotion influence in the generation of facial expression in mouth region. The β value can be set for a qualitative analysis of the visual results obtained. Good results have been achieved with values ranging from 0.6 ≤ β ≤ 0.7. Once FAP values for each animation frame are generated, the next step is the synchronization between the audio speech file and each frame. In the beginning of animation presentation, a thread for audio presentation is initiated in parallel with the thread used to control the face. The thread responsible for the face control synchronizes FAPs with the
126
Paula Rodrigues, Asla S´a and Luiz Velho
audio verifying the machine clock and using the frame frequency defined in the animation. At each iteration, the elapsed time from the start of the animation is calculated and the corresponding frame is loaded. If the frame is different from the previous one, it is taken in FAP structure received as a parameter. This method generates the FAPs mapping in the mesh and the new expression is rendered.
8.
Conclusion
This paper introduced a new emotion model for the generation of facial expressions in virtual characters. The proposed Virtual emotiontoexpression Model (VeeM) is based on a generalization of Plutchik‘s emotional wheel [11] to the emotional Hypercube H ⊂ R4 . This mathematical formulation allows the combination of the 8 pure emotions in a general and coherent way. Furthermore, it sets the ground for a comprehensive framework which integrates emotions and affective phenomena by timevarying functions f : (H, t) → H as well as the nonverbal movements g : (H, t) → F, from the configuration space H to the space of facial expressions F. The dynamics of expressions is modeled by considering the temporal properties of functions ft in the space H and gt in the space F. Facial expressions are defined in our framework using the MPEG4 standard. Consequently, another relevant contribution of this paper is a computational methodology that incorporates VeeM expressions into the representation of a face under the MPEG4 guidelines. Our system generates animations of believable virtual characters from emotional and verbal elements. This is done by mapping emotion and speech to FAPs at each frame of the animation with lipsync and expression dynamics. An important aspect that we plan to address in the future is the development of tests to validate the combination between visemes and facial expressions in VeeM. Another avenue for future work is the investigation of derived emotions resulting from the combination of more than two axis of the emotional Hypercube. Further research for the VeeM framework includes: a detailed analysis of warpings of the emotional hypercube to model affective phenomena; an indepth study of how to effectively model the dynamics of expressions through timedependent properties of functions in the emotional space; and a validation of the best strategies to perform transition between emotional states. Finally, our end goal is to fully exploit the potential of VeeM in actual applications.
References [1] Balci, K.: Xface: MPEG4 based open source toolkit for 3D Facial Animation. Proceedings of the working conference on Advanced visual interfaces, 399–402 (2004). [2] Cole, R.: Tools for research and education in speech science. Proceedings of the International Conference of Phonetic Sciences (1999). [3] Deng, Z., Neumann, U.: eFASE: Expressive Facial Animation Synthesis and Editing with PhonemeIsomap Control. Proceedings of ACM SIGGRAPH/Eurographics Symposium on Computer Animation (2006).
Virtual Emotion to Expression
127
[4] Ebragimi, T., Pereira, F.: The MPEG4 Book. Prentice Hall PTR, 1st edition (2002). [5] Ekman, P.: Universal and cultural differences in facial expressions of emotion. Nebraska Symposium on Motivation. Ed. Lincoln, 207–283 (1971). [6] Karla, P. et al.: SMILE: A Multilayered Facial Animation System. Proceedings of IFIP WG 5 (10), 189–198 (1991). [7] Ortony, A., Turner, T.J.: What’s basic about basic emotion? American Psychological Association, Inc. 97 (3), 315–321 (1990). [8] Pandzic, I.S., Forchheimer, R.: MPEG4 Facial Animation: The Standard, Implementation and Applications. John Wiley and Sons, Ltd (2002). [9] Parke, F.: A parametric model for human faces. PhD Thesis, University of Utah (1974). [10] Picard, R. W.: Affective computing. Cambridge, Mass. : M.I.T. Press (1997). [11] Plutchik, R.: A general psychoevolutionary theory of emotion. Emotion: Theory, research, and experience. Theories of emotion Vol 1, 3–33 (1980). [12] Pyun, H. and et al.: An ExampleBased Approach for Facial Expression Cloning. Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, 167–176 (2003). [13] Rodrigues, P.S.L.: A System for Generating Dynamic Facial Expressions in 3D Facial Animation with Speech Processing. PhD Thesis, PUCRio, Brazil (2007). [14] Tanguy, E. A. R.: Emotions: the Art of Communication Applied to Virtual Actors. PhD Thesis, Department of Computer Science; University of Bath; CSBU200606 (ISSN 17409497) (2006). [15] Tanguy, E., Willis, P., Bryson, J.: A Dynamic Emotion Representation Model Within a Facial Animation System. Techinical Report, CSBU200514 (2005). [16] Thayer, R.E.: The origin of everyday moods. Oxford University Press (1996).
In: Computer Animation ISBN 9781607415596 c 2010 Nova Science Publishers, Inc. Editors: J.S. Wright and L.M. Hughes, pp. 129144
Chapter 5
E XAMPLE BASED P ERFORMANCE D RIVEN A NIMATION OF AN A NATOMICAL FACE M ODEL Yu Zhang Institute of High Performance Computing, Singapore 117528
Abstract Recent development of physicsbased face modeling that emulates the anatomical structure including skin, muscles, and skull allows us to create detailed, realistic animations. However, synthesis of facial expressions on such complex models often involves significant manual work due to the difficulty in determining appropriate values of the muscle actuation parameters. This paper presents an examplebased performancedriven method to automatically estimate facial muscle actuation parameters from markerless video footage. Our method is based on an efficient face tracker which uses a facial deformation subspace model. During the training phase of the tracker a set of templates associated with the subspace basis is computed to alleviate the online computation. At runtime, the tracking algorithm establishes temporal correspondence of the face region in the video sequence by simultaneously determining both motion and appearance parameters. Using a set of example pairs that consist of the appearance and animation parameters corresponding to the key expressions, we learn the relationship between facial appearances and animation parameters. It enables the animation parameters to be computed in realtime from the appearance parameters obtained by the tracker, allowing animation of the anatomical model at interactive rates.
1.
Introduction
Realistic modeling and animation of human faces has been one of the most interesting problems in computer graphics. In particular, synthesizing facial expressions of virtual characters has experienced increased attention for its important applications in entertainment (e.g., movies and computer games), advanced manmachine interfaces, electronic commence, telepresence and shared virtual world systems, and facial expression recognition. However, the task of modeling the expressive human face by computer remains a
130
Yu Zhang
major challenge. First, facial movement is a product of the underlying skeletal and muscular forms, as well as the mechanical properties of the skin and subcutaneous layers. This is very complex, because there are numerous specific muscles and important interactions between muscles and bone structure. Second, the human visual system is very sensitive to the nuances of facial expressions that it can perceive, and the slightest deviation from real facial appearance or movement can be immediately detected as wrong by the most casual viewer. Advances in facial animation systems show the potential of physicsbased approaches, where an anatomically accurate model of facial musculature, passive tissues and underlying skeletal structure is simulated [15, 16, 18, 25, 30, 33]. This kind of techniques can be used to create detailed, realistic animations. However, as the model becomes more complex, animating detailed models of this sort becomes more difficult, requiring complex coordinated stimulation of the underlying musculature. Although the Facial Action Coding System (FACS) [10] that codes facial movements in small units provides guidance to activate individual muscles for synthesizing specific expressions, extensive manual intervention is still required due to the difficulty in determining appropriate muscle parameter values. Once a model exists, it is often desirable to automatically determine muscle contractions from real facial motion data. One solution to this problem comes from the performancedriven animation approach, in which video footage recording the performance of a human actor is used to control the animation of a synthetic model. The face can be tracked throughout the video by recovering the position and expression at each frame. This information can then be used to estimate animation parameter values. While various techniques have been used for performancedriven animation, most existing ones use colored markers painted on the actor’s face to aid the face tracker. Once the position of the markers has been determined, the position of the face and the facial features can be derived easily. However, the use of markers on the face is intrusive and limits the type of video that can be processed. In this paper, we present an examplebased performancedriven animation method by automatically determining facial muscle activations that track markerless video footage of a person’s face. Our method consists of several stages: facial deformation subspace construction, facial motion tracking, and expression retargeting. Fig. 1 shows a block diagram of the system architecture. In facial deformation subspace construction, we build a lowdimensional linear subspace that models image variation due to nonrigid facial deformations. The subspace is trained offline by processing a video sequence of the person with different expressions. In face tracking procedure, the deformation subspace model is incorporated into an efficient tracking algorithm which establishes temporal correspondence of the face region in the video sequence by simultaneously determining both motion and appearance parameters with no more computation that would be required. During the training phase of the tracking algorithm a set of templates associated to the subspace basis is computed to alleviate the online computation. In expression retargeting, a set of example face images that show key expressions are selected. Their appearance parameters in the facial deformation subspace together with the corresponding animation parameters of an anatomical 3D face model are used to learn the relationship between the animation parameters and the appearance parameters. At runtime, it allows us to efficiently estimate the animation parameters from the appearance parameters provided by the tracker.
Hamiltonian Mechanics
131
Figure 1. Overview of the examplebased performancedriven expression synthesis system. The paper is organized as follows. Section 2. reviews the previous and related work. Section 3. presents our anatomical face model and creation of key example expressions. Construction of the deformation subspace is explained in Section 4.. Section 5. details the facial motion tracking algorithm. Section 6. describes efficient estimation of animation parameters in the expression retargeting process. Experimental results are shown in Section 7.. Section 8. presents conclusions and proposes avenues for future work.
2.
Previous and Related Work
Realistic facial animation remains as a fundamental challenge in computer graphics. Since the pioneering work of Parke [24], a large body of literature on modeling and animating faces has been published in the last four decades. A good overview can be found in the textbook by Parke and Waters [23] and in the survey by Noh and Neumann [21]. In the context of this paper, we focus on publications that address performancedriven facial animation and musclebased face modeling. Remarkably, one of the oldest publications in this context is the one that uses threedimensional sparse motion capture marker data to control facial movement of computergenerated models [32]. The system synthesizes expressions by changing texture coordinates calculated from the positions of the markers on the performer’s face. Eisert and Girod [9] model a face with a Bspline surface, and analyze facial expressions into feature point positions to estimate the facial animation parameters of the MPEG4 standard. Guenter [13] capture both 3D geometry and shading information of a human face, and reproduce photorealistic expressions. In all of these methods, the locations of the markers are used to drive the 3D model. Since the markers usually are quite sparse compared to the dense surface mesh of the model, an interpolation function is typically used to deformed the mesh so that vertices in between the markers are displaced properly. Another category of performancedriven animation is to synthesize expressions by blending premodeled key expressions. The animation is achieved by computing a set of
132
Yu Zhang
blending weights that minimize the Euclidean distance between the corresponding markers on the actor’s face and the 3D model. Pighin et al. [27] reconstruct the geometry and texture of an individual face from several face images taken from different viewangles. They also model basic expressions and generate novel expressions by blending them. Later, they propose a method to find the blending weights by minimizing an error function over the set of premodeled expressions and face positions spanned by the model [26]. Kouadio et al. [17] animate a synthetic character by extracting the interpolation weights from the feature points traced by an optical capture system. Choe and Ko [5] develop an artistintheloop method for analyzing captured expressions. The expressions are synthesized by a linear combination of the elements in a muscle actuation basis which consists of face shapes resulting from the contraction of each single facial muscle. Typically, the basis elements need to be resculpted a number of times to obtain satisfactory results. In the approach proposed by Chuang and Bregler [6], the 2D key expressions are automatically found from the tracking data, and the corresponding 3D key shapes of a face model are created manually. Facial animation is produced by applying the blending weights recovered from facial feature decomposition to the 3D key shapes. However, a complete bank of 3D key shapes must be built for any new subject, which is a tedious task. Some approaches involve mapping the motion of a facial expression from the source model to the target model directly [20]. Since the target model may have different shape, the source motion vectors need to be transformed to follow the curvature of the new face shape. In order to generate delicate skin deformations, dense mesh motion is required as input. But this may not be available from some motion capture systems. Moreover, dense correspondences between the source and target models should be established for motion retargeting, which is difficult if the source and target shapes are very different. In facial animation, desire for improved realism has driven researchers to extend geometric models with physical models of facial anatomy which attempt to emulate the influence of muscle contraction onto the skin surface by approximating the biomechanical properties of skin [15, 16, 18, 25, 30, 33]. Animating a detailed musclebased model can be rather difficult since facial muscles contract in a complex coordinated manner to generate expressions. A solution to this problem is to automatically determine muscle activations from the facial motion data. Terzopoulos and Waters [31] extract muscle contraction parameters based on the position of facial features tracked by snakes. Morishima et al. [19] use 2D marker positions as input for a neural network which estimates muscle actuation parameters. Both of these approaches require heavy makeup of the actor’s face. Some techniques compute an optical flow from the video sequence and decompose the flow into muscle activations. Essa et al. [12] use a physical face model and develop a system to estimate muscle contractions that match optical flow input based on feedback control theory. Decarlo and Metaxas [8] employed a similar model that incorporates variations in head shape using anthropometric measurements. In [1, 4], a 2D quasistatic finite element model is used to simulate movements of the lips [1] or the facial skin surface [4]. Given marker data, the authors use a steepest descent iterative solver to calculate the lip model parameters or the facial muscle activations that best track the motion data. More recently, Sifakis et al. [29] employ an optimization framework to determine muscle activations that track a sparse set of surface landmarks, and used it for speech animation [28]. However, the computational complexity of the nonlinear optimization process makes this method unsuitable to
Hamiltonian Mechanics
133
retarget facial expressions in realtime.
3. Creating Key Expressions on An Anatomybased Face Model We have developed an anatomybased face model for physicallybased facial animation [33]. The model encapsulates three structural layers: skin, muscles, and skull (see Fig. 2). The skin surface is represented as a triangular mesh, consisting of 4,517 triangles. The edges and vertices of the skin mesh are converted to nonlinear springs and point masses to simulate dynamic deformation of the soft tissue. A layer of 23 muscles is attached to the skull and inserted into the skin to control facial movement. Our muscle models simulate the distribution of muscle force exerted on the facial skin. When muscles contract, the surrounding skin tissue is dynamically deformed under a field of muscle forces. An animation of facial expressions is carried out by a deformation of the skin mesh resulting from the combined contractions of a set of muscles based on the FACS [10]. The skull is also represented as a triangular mesh. During the runtime of an animation, articulation of the jaw causes motion of the mouth, and shape of the skull constrains skin deformation, preventing skull penetration. The reader is referred to [33] for detailed description of this model. Frontalis Inner Frontalis Major Frontalis Outer Corrugator Supercilliary Orbicularis Oculi Nasalis Zygomaticus Minor Zygomaticus Major Orbicularis Oris Risorius Depressor Anguli Mentalis
Figure 2. The anatomybased face model. Left: face geometry. Right: multilayer anatomical structure of the skin, muscles and skull. The anatomybased face model is animated to generate a set of key expressions, which is done once offline. Ekman [10] illustrated that facial expressions result from the actuation of a single or multiple facial muscles. Using FACS, expressions are coded as combinations of the Action Units and levels of muscle activation, which serves as an excellent guide to the key expression modeling job. With a windows GUI, different expressions can be readily synthesized on the face model using the muscle and jaw parameters. For each muscle, the degree of the muscle contraction is controlled by the parameter muscle contraction rate which is defined between 0 (passive) and 1 (maximally active). The motion of the jaw is realized by a 3D coordinate transformation which is controlled by six parameters: three rotation angles and three translation parameters. Rigid movements of two eyes are controlled by four transformation parameters. The total 33 facial animation parameters are grouped into a vector p = [p1 , p2 , . . . , pm ]T , where m = 33. Following the categorization of emotions in psychological study [10], we create a set of 54 key expressions which are believed to correspond to situations eliciting different kinds of emotion. Fig. 3 shows some
134
Yu Zhang
examples. Each key expression is rendered into an image, and its corresponding animation parameter vector, pi , is recorded.
Figure 3. Examples of key expressions generated on the anatomybased face model.
4.
Face Deformation Subspace Model
The tracking model is trained from an annotated training set which contains a number of images from the training video sequence. In acquisition of the training data, the subjects were asked to make all kinds of expressions including the key expressions. To obtain the training set, a handful of M frames from the video sequence that exhibit pronounced differences are manually labeled with 65 feature points which are located on the eyebrows, eyes, nose, mouth, and outline of the face. Feature points are distributed evenly along each contour (see an example in Fig. 4).
(a)
(b)
Figure 4. A training face image (a) and the feature points (b). Assume that the feature point set of the training frames be given as {Pi }i=1,...,M where i )) ∈ R2K is a sequence of K (K=65) points in the image Pi = ((xi1 , y1i ), . . . , (xiK , yK ¯ plane. Let P be the mean shape of the feature point set. P¯ is calculated after Pi are aligned to remove the affine motion of the face. Each training image is then warped correspondingly from its original feature point set Pi to the mean shape P¯ by using a thin plate spline approach [3]. After this normalization procedure, we define a target region R ∈ RN which is the patch of N image pixels enclosed by the bounding box of P¯ . Let the set {Ri }i=1,...,M be a set of M target regions in the warped training images. For each image in the set {Ri } we construct a 1D vector by scanning it in the standard lexicographic order. We assume that the number of training images, M , is less than the number of pixels, N . The average over
Hamiltonian Mechanics 135 PM M formed vectors is given by Φ0 = (1/M ) i=1 Ri . Each formed vector differs from the average by the vector dRi = Ri − Φ0 . We arrange the deviation vectors into a matrix D = [dR1 , dR2 , . . . , dRM ]. Principal component analysis (PCA) of the matrix D yields a set of M principal orthogonal modes of variation in {Ri }, Φj , and their corresponding eigenvalues λj . Φj are sorted according to the decreasing order of their eigenvalues. The PCA model is obtained as: R = Φ0 +
M X
αj Φj = Φα
(1)
j=1
where Φ = [Φ0 , Φ1 , . . . , ΦM ] is the matrix consisting of the average and M principal modes of variation in the training set, and α = [1, α1 , . . . , αM ]T is the vector of appearance parameters. The projection from R to α is α = ΦT R
(2)
By truncating the expansion of Eq. 1 at j = k we introduce an error whose magnitude decreases when k is increased. We choose the k such that k X j=1
λj ≥ τ
M X
λj
(3)
j=1
where τ defines the proportion of the total variation exhibited in the training set (98% in our case). By this, a k(<< N ) dimensional deformation subspace is defined by the k basis vectors, and each training image, Ri , is represented as a point in the subspace in Rk . The linear subspace basis, Φ, models nonrigid deformation of the face in generating expressions.
5.
Tracking the Face
Let I(x, t) be the pixel value at the location x = (x, y)T in the image acquired at time t. Over time, the relative motion between the subject and the camera causes the image of the face to shift. We use a warping function f (x, β) to model the rigid motion of the face, where β = [β1 , β2 , . . . , βl ]T is the motion parameter vector, with f (x, 0) = x. f is assumed to be differentiable in both x and β. Tracking a face amounts to recovering the motion parameter vector for each image in the tracking sequence. We assume that the only changes in images of the face are completely described by f , i.e., there are no changes in the illumination of the face. Our tracking model is represented by the image constancy assumption I(f (x, β t ), t) = [Φαt ](x),
∀x ∈ R
(4)
where I(f (x, β t ), t) is the image acquired at time t rectified with motion model f (x, β t ) and motion parameters β t . By [Φα](x) we denote the value of Φα for the pixel with position x in the image. Intuitively, Eq. (4) states that the rigidly rectified image I(f (x, β t ), t) can be expressed as a linear combination of the subspace basis vectors Φ.
136
Yu Zhang
Tracking a face consists of estimating for each frame in the sequence the values of the motion parameter vector β and appearance parameter vector α to minimize the following least squares objective function X O(α, β) = (I(f (x, β t ), t) − [Φαt ](x))2 =k I(β t , t) − Φαt k2 (5) x∈R
where I(β t , t) is the image of the target region, under the rameters β, in vector form in an N dimensional space: I(f (x1 , β t ), t) I(f (x2 , β ), t) t I(β t , t) = .. . I(f (xN , β t ), t)
change of coordinates with pa
(6)
Minimizing Eq. (5) can be a difficult task as it defines a nonconvex cost function. In the absence of a good starting point, some costly global optimization approaches are required to solve this problem. In our case, by taking advantage of the continuity of face motion in the tracking sequence, we recast the tracking problem as one of determining a vector of offsets, ∆β, such that β t+∆t = β t + ∆β from a frame acquired at t + ∆t. Incorporating this modification into Eq. (5) and using a first order Taylor series expansion, we reduce the problem to a linearized version O(α, ∆β) ≈k I(β t , t + ∆t) + J∆β − Φαt+∆t k2
(7)
where J ∈ RN ×l is the Jacobian matrix of I with respect to the components of β, i.e., J = ∂I(β,t) ∂β βt . Such a linearization enables us to apply continuous optimization procedures to the tracking problem. The optimization scheme we use first assumes α constant and uses the most recent appearance parameter estimate αt to rectify the target region. Then the solution for ∆β can be obtained by solving the set of equations ∇O = 0: ∆β = −(JT J)−1 JT [I(β t , t + ∆t) − Φαt ]
(8)
We define the error vector as the difference between the rectified image and the linear combination of the subspace basis vectors. e(t + ∆t) = I(β t , t + ∆t) − Φαt
(9)
Thus, the solution of Eq. (5) at time t + ∆t given a solution at time t is β t+∆t = β t − (JT J)−1 JT e(t + ∆t)
(10)
From Eq. (10), we see that the obstacle to efficiently tracking the face region through the image sequence is the computational cost of estimating J for each frame which involves the calculation of the image gradient vector. However, it is possible to reduce this computation by factoring J. Each element of J can be written as sij = Iβj (f (xi , β), t) = ∇f I(f (xi , β), t)T fβj (xi , β)
(11)
Hamiltonian Mechanics
137
By differentiating both sides of Eq. (4) with respect to x, we obtain ∇f I(f (x, β), t)T fx (x, β) = ∇x Φα From Eq. (11) and (12), we get J(α, β) =
∇x Φ(x1 )αfx (x1 , β)−1 fβ (x1 , β) ∇x Φ(x2 )αfx (x2 , β)−1 fβ (x2 , β) .. . ∇x Φ(xN )αfx (xN , β)−1 fβ (xN , β)
(12)
(13)
Therefore, J can be expressed in terms of the gradient of the subspace basis vectors, ∇x Φ, which are constant, and the appearance and motion parameters, (α,β), which are timevarying. If we choose a motion model f such that αfx (x, β)−1 fβ (x, β) = Λ(x)Γ(α, β),
(14)
where Λ and Γ are the matrices depending only on image coordinates and parameters (α,β), respectively. J can then be factored into ∇x Φ(x1 )Λ(x1 ) ∇x Φ(x2 )Λ(x2 ) J(α, β) = (15) Γ(α, β) = J0 Γ(α, β) .. . ∇x Φ(xN )Λ(xN )
where ∇x Φ(xi ) is the Jacobian of Φ with respect to image coordinates, J0 is a constant matrix, and Γ is a timevarying matrix. The columns of J0 can be regarded as a set of fixed template images and can be precomputed offline. By exploiting this factorization, from (8), an efficient solution for ∆β can be obtained: ∆β = −(ΓT ΩΓ)−1 ΓT JT0 e
(16)
where Ω = JT0 J0 . J0 can be precomputed and stored, and only e and Γ need to be evaluated online at (αt , β t ). Our optimization scheme then assumes β constant and computes the minimum of O(α, ∆β) with respect to α to obtain the solution for αt+∆t : αt+∆t = ΦT [I(β t , t + ∆t) + J∆β]
(17)
The term J∆β represents the pixel value variation in I due to a motion of magnitude ∆β. Intuitively Eq. (17) states that the appearance parameters are computed by projecting into the subspace Φ the rectified image corrected to take into account the incremental motion ∆β. In our experiments, we use a projective motion model, f (x, β) = Hxh , where H is a 3 × 3 projective transformation matrix containing l = 8 motion parameters and homogeneous image coordinates xh are related to x by: xh = (r, s, λ)T → x = (r/λ, s/λ)T = (x, y)T ; λ 6= 0. The above optimization is performed using the GaussNewton approach. With obtained αt+∆t , we iteratively estimate ∆β using (16). Normally in two to three iterations the convergence is reached.
138
6.
Yu Zhang
Facial Expression Retargeting
The face tracker described in Section 5. provides the vector of subspace coefficients, αt , which encapsulates deformation appearance of the face at time t. αt is used to estimate animation parameters of the anatomybased face model. We select a set example face images {Rj }j=1,...,n (n = 54) corresponding to the key facial expressions from the normalized training data set {Ri }i=1,...,M , and project each example image into the constructed deformation subspace to obtain its appearance parameters. αj = ΦT Rj (18) Let A∈ R(k+1)×n and P∈ Rm×n be the matrices obtained by storing columnwise the computed appearance parameters of the example images and prestored facial animation parameters corresponding to the key facial expressions, respectively. We construct a matrix G∈ R(k+m+1)×n : A α1 · · · αn = (19) G= WP W(p1 · · · pn ) where W is a diagonal matrix of weights for the facial animation parameters to compensate the difference in scale between the animation parameters and appearance parametersz. We set W = diag(w) where w2 is the ratio of the total variation of the appearance parameters to the total variation of the animation parameters. Applying PCA on G, we obtain a matrix Ψ ∈ R(k+m+1)×(q+1) which consists of the mean vector of examples and q (q < n) eigenvectors corresponding to the q largest eigenvalues of the covariance matrix GGT : Ψα (20) Ψ= Ψp Each example pair (αj ,pj ) can be approximated as: αj = Ψγj wpj
(21)
where γ j = [1, γj,1 , . . . , γj,q ]T is the vector of coefficients. By this, a concatenated parameter vector, which is originally in R(k+m+1) , can be represented as a point in the low dimensional parameter subspace in R(q+1) . Ψ parameterizes the example parameter vectors and represents the relationship between the appearance parameters in A and the animation parameters in P. For each frame in the tracking sequence, once the vector of appearance parameters is obtained by the tracker, it can be represented by a linear combination of the parameter subspace basis vectors: αt = Ψα γ t , where γ t is the unknown. γ t is solved by γ t = pinv(Ψα )αt
(22)
where pinv(·) is the matrix pseudoinverse operator using SVD. The vector of facial animation parameters corresponding to the tracked face at time t is then computed as: 1 1 pt = Ψp γ t = Ψp pinv(Ψα )αt = Cαt (23) w w where the constant matrix C∈ Rm×(k+1) is precomputed offline. Table. 1 shows the implementation steps of our efficient performancedriven facial animation algorithm.
Hamiltonian Mechanics
139
Table 1. The steps of our facial animation algorithm. Process Offline 1. Compute the gradient of the subspace basis ∇x Φ(x) and matrix Λ(x).
2. Compute and store J0 and Ω.
3. Compute the parameter subspace basis Ψ. 4. Compute and store C.
Runtime 1. Reconstruct the image vector Φαt . 2. Use the motion parameter βt to compute I(βt , t + ∆t). 3. Compute e(t + ∆t) according to Eq. (10). 4. Compute Γ(αt , βt ). 5. Compute ∆β according to Eq. (16). 6. Compute βt+∆t = βt + ∆β. 7. Compute αt+∆t according to Eq. (17). 8. Go to step 5 to compute ∆β using αt+∆t until convergency. 9. Compute the facial animation parameters pt+∆t according to Eq. (23).
7.
Results
Our facial animation system is programmed with C++/OpenGL and runs on a 3.2GHz PC with 1GB memory. In order to evaluate the accuracy of estimation of facial animation parameters, we use a synthetic sequence as the input. The anatomybased face model is animated to generate two synthetic image sequences: a training sequence (944 frames) and a tracking sequence (1865 frames). The facial expressions in the training sequence include the key expressions and are different from those in the tracking sequence. We select 134 normalized images from the training sequence to train the tracker. The target face region contains N = 147 × 220 pixels. These selected images allow us to form the deformation subspace, Φ, for tracking. The set of 54 normalized images of the key expressions and the prestored animation parameter vectors is used to compute the parameter subspace, Ψ, for facial expression retargeting.
Figure 5. Some tracked images from the synthetic sequence.
140
Yu Zhang
Figure 6. Errors of estimated animation parameters.
(a)
(b)
Figure 7. Example frames from two live tracking sequences and expressions synthesized on the 3D face model. Fig. 5 shows some tracking frames from the synthetic sequence. We assess the reanimation by measuring the maximum, mean, and root mean square (RMS) errors from the
Hamiltonian Mechanics
141
Table 2. Parameter values of two experiments and performance of our system. Notation: tracking time (Tt ), retargeting time (Tr ) and expression simulation time (Ts ).
estimated animation parameters to the ground truth. Fig. 6 plots the normalized errors of all parameters (the ratio between measured errors and the maximum parameter range). The result shows that the overall parameter estimation is accurate. Some relatively large errors (e.g., the maximum errors of the parameters 38, 18 and 19 which control the peripheral muscles) occur when the model in the tracking sequence has a large outofplane rotation (> 20 degrees). We also test our system using the live video sequences of different subjects. Fig. 7 shows some tracking frames and animation snapshots. Note that we do not estimate 3D head motion, only the inplane 2D rotation and translation extracted during the tracking stage are used to produce the global head motion. The first tracking video (male) consists of 5,100 frames, acquired by a Sony VL500 firewire camera at 30 fps. The second video (female) consists of 5,400 frames. Also, long sequences are used for training the tracker and the expression retargeting module. In particular, a set of images corresponding to the key expressions are manually selected from the training sequence. Together with the stored animation parameter vectors, the appearance parameters of these images are used to form the example parameter pairs for building the parameter subspace. Table 2 shows the parameter values used in two experiments and the average time of each process for resynthesizing facial animation. With the proposed algorithm we can achieve standard video rate performance (30Hz) for face tracking and estimation of animation parameters. The bulk of computation is consumed by the physicallybased expression simulation [33]. Nevertheless, we can still achieve an average frame rate of about 15 fps animation speed on the current experimental platform.
8.
Conclusion
We have presented an efficient examplebased method to synthesize facial expressions on an anatomical model by retargeting captured performance. Our method automatically determines facial muscle activations from markerless video footage. In offline processing, we build a linear subspace model to compactly describe appearance variation due to facial deformations. Based upon this model, an efficient face tracker is used to track the target face region by simultaneously solving for both the motion and appearance parameters. Efficiency is gained by precomputing a set of motion template resulting from factorization of the image Jacobian used in minimizing the tracking error function. Using a set of example pairs that consist of the appearance and animation parameters corresponding to the
142
Yu Zhang
key expressions, we learn an expression retargeting matrix. Given the appearance parameters provided by the tracker, facial animation parameters are estimated in realtime, and the anatomical model is animated at an interactive rate. One improvement of the existing system consists in splitting the face into the upper region (including eyes and forehead) and the lower region (including nose, mouth, and chin). This allows a more compact and accurate model of the regions of interest. It enables us to use a subspace of much lower dimension (i.e., number of eigenvectors) to model the appearance of a target region with lower dimensionality (i.e., length of an eigenvector), which would speed up the tracking process. Moreover, we would like to automatically select key expressions of the actor from the training sequence. One possibility is to build a personalized model by conforming the anatomical 3D face model to the face shape in video footage, and generate key expressions on this model. The corresponding video image can then be found by minimizing an objective function that measures the similarity between the expression in the video and 2D projection of the synthesized expression.
References [1] S. Basu, N. Oliver, and A. Pentland. “3D modeling of human lip motion.” Proc. ICCV’98, pp. 337343, 1998. [2] M. J. Black and A. D. Jepson. “Eigentracking: Robust matching and tracking of articulated objects using a viewbased representation.” International Journal of Computer Vision, 26(1): 6384, 1998. [3] F. L. Bookstein. “Principle warps: Thin plate splines and the decomposition of deformations.” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2(6):567585, 1989. [4] B. Choe, H. Lee, and H.S. Ko. “Performancedriven musclebased facial animation.” Journal of Visualization and Computer Animation, 12: 6779, 2001. [5] B. Choe and H. S. Ko. “Analysis and synthesis of facial expressions with handgenerated muscle actuation basis.” Proc. Computer Animation’01, pp. 1219, 2001. [6] E. Chuang and C. Bregler. “Performance driven facial animation using blendshape interpolation.” Standford University Computer Science Technical Report, CSTR200202, April 2002. [7] T. F. Cootes, G. J. Edwards, and C. J. Taylor. “Active appearance models.” Proc. ECCV’98, vol. 2, pp. 484498, 1998. [8] D. DeCarlo and D. Metaxas. “Deformable modelbased shape and motion analysis from images using motion residual error.” Proc. ICCV’98, pp. 113119, 1998. [9] P. Eisert and B. Girod. “Analyzing facial expression for virtual conferencing.” IEEE Computer Graphics and Application, 18(5):7078, 1998. [10] P. Ekman and W. V. Friesen, Facial Action Coding System, Consulting Psychologists Press Inc., Palo Alto, California 94306, 1978.
Hamiltonian Mechanics
143
[11] R. Enciso, J. Li, D. Fidaleo, TY. Kim, JY. Noh, and U. Neumann. “Synthesis of 3D faces.” International Workshop on Digital and Computational Video, December 1999. [12] I. Essa and A. Pentland. “Coding, analysis, interpretation, and recognition of facial expressions.” IEEE Tran. Pattern Analysis and Machine Intelligence, 19(7):757763, July 1997. [13] B. Guenter, C. Grimm, D. Wood, H, Malvar, F. Pighin. “Making faces.” Proc. SIGGRAPH’98, pp. 5566, July 1998. [14] G. Hager and P. Belhumeur. “Efficient region tracking with parametric models of geometry and illuminations.” IEEE Tran. Pattern Analysis and Machine Intelligence, 20(10):10251039, 1998. [15] K. K¨ahler, J. Haber, H. Yamauchi, and H.P. Seidel. “Head shop: Generating animated head models with anatomical structure.” Proc. ACM SIGGRAPH Symp. on Comput. Anim., pp. 5564, 2002. [16] R. Koch, M. Gross, and A. Bosshard. “Emotion editing using finite elements.”Proc. EUROGRAPHICS’98, pp. 295302, 1998. [17] C. Kouadio, P. Poulin, and P. Lachapelle. “Realtime facial animation based upon a bank of 3D facial expression.” Proc. Computer Animation’98, pp. 128136, 1998. [18] Y. Lee, D. Terzopoulos, and K. Waters. “Realistic modeling for facial animation.” Proc. SIGGRAPH’95, pp. 5562, August 1995. [19] S. Morishima, T. Ishikawa, and D. Terzopoulos. “Facial muscle parameter decision from 2D frontal image.” Proc. the Int. Conf. on Pattern Recognition, vol. 1, 160162, 1998. [20] J. Y. Noh and U. Neumann. “Expression cloning.” Proc. SIGGRAPH’01, pp. 277288, August 2001. [21] J. Y. Noh and U. Neumann. A survey of facial modeling and animation techniques. USC Technical Report 99705, USC, Los Angeles, CA, 1999. [22] J. Ohya, Y. Kitamura, H. Takemura, H. Ishi, F. Kishino, and N. Terashima. “Virtual space teleconferencing: Realtime reproduction of 3d human images.” Journal of Visual Communications and Image Representation, 6(1): 125, 1995. [23] F. I. Parke and K. Waters. Computer Facial Animation. AK Peters, Wellesley, MA, 1996. [24] F. I. Parke. Computer generated animation of faces. Master’s thesis, University of Utah, Salt Lake City, 1972. [25] S. Platt and N. Badler. “Animating facial expressions.” Proc. SIGGRAPH’81, pp. 245252, 1981.
144
Yu Zhang
[26] F. Pighin, R. Szeliski, and D. H. Salesin. “Resynthesizing facial animation through 3d modelbased tracking.” Proc. ICCV’99, pp. 143150, 1999. [27] F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, and D. H. Salesin. “Synthesizing realistic facial expressions from photographs.” Proc. SIGGRAPH’98, pp. 7584, July 1998. [28] E. Sifakis, A. Selle, A. RobinsonMosher, and R. Fedkiw. “Simulating speech with a physicsbased facial muscle model.” Proc. ACM SIGGRAPH/Eurographics Symp. on Comput. Anim.’06, pp. 261270, 2006. [29] E. Sifakis, I. Neverov, and R. Fedkiw. “Automatic determination of facial muscle activations from sparse motion capture marker data.” Proc. SIGGRAPH’05, pp. 417425, 2005. [30] D. Terzopoulos and K. Waters. “Physicallybased facial modeling, analysis and animation.” Journal of Visualization and Computer Animation, vol.1, pp. 7380, 1990. [31] D. Terzopoulos and K. Waters. “Analysis and synthesis of facial image sequences using physical and anatomical models.” IEEE Tran. Pattern Analysis and Machine Intelligence, 15(6):569579, June 1993. [32] L. Williams. “Performancedriven facial animation.” Proc. SIGGRAPH’90, pp. 235242, August 1990. [33] Y. Zhang, E. C. Prakash, and E. Sung. “Efficient modeling of an anatomybased face and fast 3D facial expression synthesis.” Computer Graphics Forum, 22(2): 159169, June 2003.
In: Computer Animation ISBN 9781607415596 c 2010 Nova Science Publishers, Inc. Editors: J.S. Wright and L.M. Hughes, pp. 145156
Chapter 6
DYNAMICS FOR M ANAGING O CCLUSION OF B UILDINGS IN PANORAMIC M APS Neeharika Adabala∗ Microsoft Research India, “Scientia”, 196/36, 2nd Main, Sadashivnagar, Bangalore 560080
Abstract Panoramic maps depict urban areas in oblique view. This form of cartography was prevalent from the late sixteenth century to the early nineteenth century, when there were not many skyscrapers in urban areas. But oblique view maps in the current urban scenarios suffer from loss of details due to occlusion among closely located multistory buildings. In this work we leverage the time dimension to overcome the clutter in space dimension by introducing functional dynamics. We define a parameter called occlusion index for an urban scene at a given viewpoint. Solving the problem of occlusion involves devising methods for visualizing the urban scene that reduce/minimize the occlusion index. We explore occlusion reduction techniques that involve selecting optimal viewpoints, displacing buildings, making buildings transparent and changing building heights. We demonstrate these approaches by presenting screen shots of the solution applied to a prototype city block, and discuss the advantages and disadvantages of these solutions. This work is pioneering in its approach to applying animation in cartography, which has previously used animations only to depict timedependent phenomena or flythroughs.
1.
Introduction
Panoramic maps were a vibrant form of representing urban locales from the late sixteenth century to the early nineteenth century [8]. They depicted cities in oblique view and included trees, people, horse carts on roads, etc., composed together aesthetically. The beauty of panoramic maps makes them popular wall hangings to this day. These maps in oblique view are also appealing because they represent landmarks in threedimensions, making them easier to recognize for map users while finding their way in a city. ∗
Email address:
[email protected]
146
Neeharika Adabala
Making of panoramic maps disappeared with time as it involved extensive manual effort by skilled artists. Also current day urban areas contain several buildings that occlude each other, making oblique view maps unattractive and lacking in information. Recent advances in computer graphics and computer vision have enabled creation of largescale urban models [15, 13, 14] that can be used to generate panoramic maps. These techniques significantly reduce the manual effort of panoramic map artists. However, the problem of extensive building occlusion severely limits the usefulness of the maps. Therefore there is a need to explore techniques to overcome this limitation, and increase the visibility of buildings. A simple way to address the problem of occlusion is to modify the viewpoint of the map. This approach is applicable for either static maps printed on a paper or dynamics maps displayed by online mapping services. However, it is not always possible to find an acceptable solution as one or more buildings may be occluded in every possible viewpoint. Therefore a solution cannot be guaranteed by this approach. When it is not possible to find an ideal viewpoint, then it is clear that there is no possible solution in the three dimensions of space. We overcome this problem by resorting to the time dimension  we do this by introducing the concept of functional dynamics into the rendering and solve the occlusion problem. The solutions can take various forms, the specific techniques that we describe in this work include: displacing buildings to improve visibility, viewpointdependent building transparency and altering building heights. These solutions are especially relevant in the context of the recent trend towards providing online mapping services on digital displays. These online maps no longer have to be static. The modifications based on the proposed approaches are applied to entities (buildings) present in the map resulting in a time varying appearance of the map. The above time variation in maps during user interaction is different from the typical dynamics that are often incorporated in contemporary maps to depict timedependent phenomena (example: the evolution of a storm, or the expansion of an empire with time). We note the distinction of the dynamics introduced by us to solve the problem of occlusion by emphasizing that this dynamic is unrelated to the physical concept of time lapsed. Therefore the animation is said to be functional and has the sole function of reducing occlusion between buildings. This differentiation is important when developing GI systems that need to visualize large volumes of information by employing both functional dynamics and animations of timedependent phenomena. Functional dynamics impacts the user interaction with the geographic information system; an effective implementation of functional dynamics results in conveying more information to the user with lesser interaction. The rest of the paper is organized as follows, the following section gives an overview of related work. In section 3 we define a parameter called occlusion index, which characterizes the extent to which the buildings are occluded for a given viewpoint. The visibility of the buildings in a map can be optimized by selecting a configuration that minimizes the occlusion index. Section 4 to 7 describes techniques to manipulate the oblique view maps to minimize the value of occlusion index when rendering of an urban region model. We present example results on a hypothetical synthesized city block to illustrate the working of the proposed technique in section 8 and give conclusions in section 9.
Dynamics for Managing Occlusion of Buildings in Panoramic Maps
2.
147
Related Work
Several approaches to largescale urban modeling have been developed and are described in [10]. Other interesting results on nonphotorealistic rendering of city models inspired by panoramic maps are described in Buchholz et al. [2] and techniques for rendering them stylistically are presented in the work of Adabala [1] and D¨ollner and Walther [4]. A technique for procedurally creating city models is presented in [16]. The rendering of the models [16] generated in birdseye view results in impressive panoramic view of urban areas, however the resulting images cannot be used as maps as several of the buildings occlude each other. The problem of being able to use the rendered results as maps has not been addressed. Landmarks are best represented by threedimensional models in oblique view, they have a key role in userfriendliness of a map and are often seen in tourist maps. Therefore techniques to make oblique view maps from models of cities is attractive. There has been significant amount of work done in the area of determining optimal viewpoints for rendering of graphics models. These include the work of PerePau V´azquez et al. [17] where they develop a method of determining the optimal viewpoint for viewing a geometric model by defining viewpoint entropy. The entropy attempts to capture the degree of visibility of features of the geometric model, higher entropy of a viewpoint indicates greater visible information. A large body of work exists in computer vision literature on finding optimal viewpoints at which to place cameras and lights such that the resulting image is most suitable to apply computer vision algorithms. Capture of edges and features of objects is crucial for identifying objects. Therefore these algorithms have mainly focused on single objects rather than scenes where elements occlude each other [19]. All research in the context of aspect graphs in computer vision is relevant to this work, the concept of aspect graph is very general and has to be adapted significantly to be applied in a particular context. Another group of pertinent algorithms in computer vision are developed in the context are robot path planning; indoor environments form the main subject of these studies [12]. Rendering algorithms like radiosity and raytracing often solve the visibility problem in the context of whether light from a one surface reaches another [7]. Similarly shadow casting algorithms also solved the visibility problem. The work by Wonka and Schmalstieg [18] solve a problem of visibility in the context of walkthroughs in urban environments. An survey of walkthrough related visibility results can be found in [3]. A summary of several studies on visibility can be found in the excellent multidisciplinary survey of visibility studies by Durand [5, 6]. The other interesting group of problems that require viewpoint analysis are the problems that can be mapped onto the art gallery problem. Several analytical solutions have been proposed for the problem that are based on polygon triangulation [11]. The solutions limit themselves to twodimensions by considering the art gallery as a polygon. Interesting solutions have been developed for this problem, however they cannot be easily applied in the context of our problem. All these visibility related solutions treat the graphics models as a collection of triangles and define optimization techniques applicable to the triangles. The number of triangles can be large making the optimization algorithms slow. Also, the triangles cannot be associated with the semantics of the model they represent. In this work we introduce a simple concept
148
Neeharika Adabala
of occlusion index that is specific to urban models. The occlusion index provides a tool that enables us to retain the semantic significance of each building as an entity while optimizing visibility. The work on automatically generating tourist maps [9] explores the problem of reducing occlusion in tourist maps, and addresses the problem by adopting the traditional technique used by artists that involves expanding the roads. This technique considers the static maps and emphasizes on the visibility of roads. The idea of using functional dynamics to overcome occlusion and improve visualization has not been explored prior to this work.
3.
The Occlusion Index
In our approach buildings within the region of interest are represented by their bounding boxes for the purpose of visibility computations. This is quite accurate for rectangular buildings blocks, and for more irregular shaped buildings also it is acceptable. A general rule of thumb is that such a bounding box is suitable for all buildings that can be modeled completely by extruding a footprint till the final height with no modifications in cross section. This approximation can lead to faulty results when the building tapers towards the top. In this case if the full height of the building is considered in the bounding box, then the solutions can be erroneous for the visibility of the tapering portion. A viewpoint in which a small portion of the tapered tip of the building is visible may be selected as an optimal viewpoint. We over come this problem by limiting the height of the bounding box to the height of the building that is not tapered when it is being evaluated for visibility. We use the bounding box for the full height of the building in computations where it occludes other buildings so that occlusion by the tapering part is not ignored in the computation. Using a variable bounding box, illustrated in figure 1, is a significant approximation to visibility computations on actual geometry, but we err on the side of being more conservative in our estimates in this approach. This leads to reasonable solutions while keeping the computations simple. Once the bounding boxes that represent the buildings have been identified, we determine the extent of occlusion of buildings for a given view point using these bounding boxes. For this purpose we define a parameter called occlusion index that measures that is an aggregate of the farction of buildings occluded for a given viewpoint of the urban region. The building P occlusion boi of a building i for a viewpoint is defined by boi = j (j6=i) BlkF racij where BlkF racij is the fraction of visible area of building i that is blocked by building j. We apply the models transformation matrix and the projection matrix to compute the screen coordinates of each of the models, and the value of BlkF racij is computed in the screen coordinates space. In the case of buildings with tapering roofs or buildings that require the use of the variable bounding box approach, we use the smaller height value shown in figure 1 while computing the value of BlkF racij of the building i, and the larger height value when it occurs as the building j while occluding the building i behind it. Thus we are able to separate the height property of a building during visibility computation, and occlusion computation.
Dynamics for Managing Occlusion of Buildings in Panoramic Maps
149
Figure 1. Bounding boxes of buildings when building cannot be thought of as an extrusion of its footprint. When we want to find the solution for a subset of total number of buildings in the scene then we limit the computation of boi to the buildings of interest. However the computation of values BlkF racij is performed over all buildings j in the scene. The occlusion index for a given viewpoint and position of buildings is given by O = P i boi where the summing is over all buildings i in the scene for which boi is computed. In the following four sections we describes ways in which this occlusion index can be exploited to reduce occlusion among buildings in panoramics maps.
4.
Modification of Viewpoint
An obvious way to handle occlusion is to change the viewpoint [6], for this the occlusion index is computed repeatedly for rotating viewpoints. The viewing direction that has the minimum value of occlusion index is selected as the optimal viewpoint. The algorithm for determining the optimal viewpoint is outlined below: 1. Select region or buildings for which optimal viewpoint is to be determined. Region can be selected by truncating an urban scene to an extended bounded region around the current visible portion of environment on the screen. If buildings are to be selected this can be done by querying a database that contains additional information about the functionality of the buildings. 2. Select the angle of elevation and zoom level at which to display the panoramic map of region. 3. Determine bounding boxes for buildings. 4. Rotate 360 degrees about the center of the selected region, and evaluate occlusion index for each viewpoint: (a) For each buildings i in the scene compute the building occlusion term boi by considering all the buildings that can block it for given viewpoint.
150
Neeharika Adabala (b) Sum up the values of boi to find the occlusion index for the scene.
5. Fix viewpoint to the direction at which the occlusion index assumed minimum value. When no efficient data structure is used in storing the building information, this is an O(n2 ) algorithm when there are n buildings in the scene and we are optimizing for all of them. If m buildings are selected as buildings of interest then the algorithm complexity is O(mn). This approach may not always result in a solution in which at least some part of every building is seen, especially in over crowded downtown areas. Also, modifying the viewpoint is not applicable in all cases, often the cartographer or user may decide which direction  north, south, east, or west  of the urban area should be displayed at the top of the map. Therefore there is a need to explore techniques to optimize the visibility of buildings after the orientation of the maps has been selected. The following three sections describe techniques that use funcational animation to optimize the visibility of buildings.
5.
Displacing Buildings
In this technique for handling occlusion we shift builds about their original position by small amounts when they occlude each other. This movement is dependent on the viewpoint, and it can make completely occluded buildings to become partially visible in the map. Constraints are applied to the extent to which the building can moved from its original position to maintain the usefulness of the map. Buildings cannot change its relative location with respect to roads or with respect to other buildings. The occlusion index is minimized to obtain the most suitable building locations for a given viewpoint. This approach cannot guarantee a solution in all cases, especially in over crowded downtown areas where the buildings have a limit region in which they can be displaced.
6.
Making Buildings Transparent
An alternative dynamic approach is to make the buildings that are blocking other buildings transparent. This is again a viewpoint dependent approach. When a building becomes transparent the contribution of the building to the occlusion index O is considered to be zero. This approach guarantees that all the buildings are visible from any given viewpoint. However it can happen that for certain viewpoints, several buildings become transparent for enabling the visibility of all the buildings. Alternative approaches of assigning a depth dependent translucency to the buildings can be adopted in this case. Here the contribution of a building to the occlusion index is weighted by the translucency associated with the P buildings. Therefore the occlusion index is given by O = i αi boi where αi is the transparency associated with the building i, and the values of boi are computed with the equation P boi = j (j6=i) αj BlkF racij where αj is the transparency associated with building j.
Dynamics for Managing Occlusion of Buildings in Panoramic Maps
151
Figure 2. Optimal view point for a simple urban scene.
7. Altering Heights of Buildings An alternative approach is to scale down the height of obstructing buildings. This approach is adopted in cases where the accuracy of the skyline and scale is not important, and in stylized maps where the information and aesthetics are more emphasized. This kind of distortion is often seen in tourist maps where some of the monuments of a city are draw to the scale of their fame rather than their physical size. Different ways of changing height of buildings can be explore. For example one can we exchange the height of buildings when a taller building obstructs a shorter one. This approach to change heights prevents an overly uniform appearance of buildings. Also it prevents excessive changing of building heights with changes in viewpoint. Alternatively one can assign ascending order of heights to buildings for a given viewpoint. This approach can result in a regular monotonous representation of the urban areas. In the approaches described here where we shorten the buildings in the foreground relative to the ones at the back, the occlusion index reduces as the values of BlkF racij decrease when shorter buildings are in front. Note that the approaches of manipulating properties of buildings present in a map are not completely novel. In fact these approaches have been used extensively in static visualizations of cities that are found in the Nuremberg Chronicle. Buildings were moved, and heights changed in the town plans printed in this fifteenth century incunabulum.
8.
Results and Discussion
The model of an urban area was synthesized, such that there are several multistory buildings in adjacent blocks, to demonstrate our results. In real world scenarios a subset of buildings from the map may be displayed in a separate window, similar to the panoramic map viewing tool provided at [8], and our techniques can work in real time in this window. We have implemented our algorithm in C++ and OpenGL on a Windows system. The results are illustrated in the figures 2 to 6. Figure 2 shows the synthesized simple urban area at an optimal viewpoint. In the case of results shown in figure 3 the model was created by considering an aerial image of a region in Las Vegas shown in the left. We marked out the footprint of some
152
Neeharika Adabala
Figure 3. Optimal view point for urban scene for irregular shape of building footprints. (Left) Aerial image of urban area  a region in Las Vegas. (Right above) Optimal viewpoint for the urban area model created from aerial image. (Right below) The two viewpoints when occlusion index reaches peak values. complex shaped buildings from this image and created a model by extruding these footprints to random heights. The variation of occlusion index is given for this model in figure 4. Notice that the occlusion index peaks at two points. These are the two viewpoints in which the buildings line up one behind the other in two rows, shown in the right bottom in figure 3. There are two points at which the occlusion index reaches a minimal value, one such viewpoint is shown in right top of figure 3. A limitation of this approach is that it cannot guarantee that all buildings are visible at optimal viewpoints. At some elevations of viewing the urban region some of the buildings are completely occluded even at optimal viewpoints. The other three approaches make use of functional dynamics to enhance the information visible to the user. Frames captured during interaction with the synthesized urban block are presented in figures 5 to 8. Observe the occlusion of the pink building in each case. The figure 5 shows the completely occluded building. In figure 6 the green building in the front is displaced suitably, and the pink building becomes partially visible. In the figure 7 the buildings obstructing a large portion of buildings behind them become transparent. A threshold value of the extant of obstruction is used to decide when a building should be made transparent, this value is provided as a user control. A low value of threshold makes the buildings become transparent even when they occlude a small fraction of the building behind them. In the implementation shown in figure 7 we made the building faces fully transparent and only retained the edges to indicate that a building is currently transparent in this viewpoint to improve visibility. In figure 8 shows the results when the buildings change height. Such a modification of the ground truth can be disorienting to a user. However this can be mitigated if the buildings are labeled and the user is aware that the map is stylized and not to scale with respect to the
Dynamics for Managing Occlusion of Buildings in Panoramic Maps
153
25
20
Occlusion Index
15
10
5
0
Figure 4. Graph of variation of occlusion index with change in view point for scene given in following figure. Notice the to peaks when the view point coincides with the two directions in which the buildings are apparently lined up one behind the other as shown in Figure 3. building heights. Our initial experimentation with users showed that they find the approach interesting, however more detailed user studies are required for wider use of this approach. These approaches enable viewing of more information in a single frame, therefore a better picture of the relative locations of buildings in the urban region is obtained by lesser interaction. A more thorough manipulation of the region would have been required without the support from functional dynamics.
9.
Conclusions
In this work we addressed the issue of information loss in oblique view maps due to occlusion among closely located multistory buildings. Maps are abstract representations of
Figure 5. Handling occlusion of buildings in panoramic maps observe the pink building in all cases. The pink building is completely occluded for this viewpoint.
154
Neeharika Adabala
Figure 6. Moving position of buildings slightly.
Figure 7. Making buildings transparent.
Figure 8. Changing height of buildings  exchange of height between obstructing building and building at the back.
Dynamics for Managing Occlusion of Buildings in Panoramic Maps
155
spatial information, therefore we manipulated the visualizations scheme to maximize the information presented to the user. We defined a parameter called occlusion index in the context of urban areas with multistory buildings, and determined optimal viewpoints for the region of interest. This solution did not always guarantee the visibility of all buildings in the urban region of interest. We therefore proposed alternative solutions to visualize overcrowded threedimensional spaces by taking recourse to the fourth ‘time’ dimension; we introduced the concept of functional dynamics. We applied this approach and suggested techniques that displace buildings slightly from original positions, make buildings transparent and change heights of buildings to improve the visibility of information in a region. It should be noted that while our work demonstrated the results individually for each of the approaches, they can be combined together to give more effective solutions. That is, buildings can be displaces, made transparent, and scaled, simultaneously to create optimal visualizations of the urban regions. The concept of introducing dynamics to improve visualization is relatively new to cartography, and it is important to conduct user studies to learn which approaches are more attractive and effective for users. The initial reaction of users to the proposed techniques has been encouraging.
Acknowledgment The author would like to acknowledge the discussions with Dr. Kentaro Toyama that led to the development of this work.
References [1] N. Adabala. A technique for building representation in obliqueview maps of modern urban areas. To appear in The Cartographic Journal, 2009. [2] H. Buchholz, J. D¨ollner, M. Nienhaus, and F. Kirsch. Realtime nonphotorealistic rendering of 3d city models. In Proceedings of the 1st International Workshop on Next Generation 3D City Models, 2005. [3] D. CohenOr, Y. Chrysanthou, C. Silva, and F. Durand. A survey of visibility for walkthrough applications. IEEE Transactions on Visualization and Computer Graphics, 9(3):412–431, JulySept. 2003. [4] J. D¨ollner and M. Walther. Realtime expressive rendering of city models. In Seventh International Conference on Information Visualization, Proceedings IEEE 2003 Information Visualization, pages 245–250, 2003. [5] F. Durand. A multidisciplinary survey of visibility. in ACM Siggraph course notes Visibility, Problems, Techniques, and Applications, July 2000. [6] F. Durand. 3D Visibility: analytical study and applications. PhD thesis, Universit´e Joseph Fourier, Grenoble I, July 1999. http://wwwimagis.imag.fr.
156
Neeharika Adabala
[7] F. Durand, G. Drettakis, and C. Puech. Fast and accurate hierarchical radiosity using global visibility. ACM Transactions on Graphics, 18(2):128–170, 1999. [8] Geography and MapsDivision. The library of congress: Panoramic maps collection. http://memory.loc.gov/ammem/pmhtml/panhome.html. [9] F. Grabler, M. Agrawala, R. W. Sumner, and M. Pauly. Automatic generation of tourist maps. ACM Trans. Graph., 27(3):1–11, 2008. [10] J. Hu, S. You, and U. Neumann. Approaches to largescale urban modeling. IEEE Comput. Graph. Appl., 23(6):62–69, 2003. [11] D. T. Lee and A. K. Lin. Computational complexity of art gallery problems. IEEE Trans. Inf. Theor., 32(2):276–282, 1986. [12] J. Lengyel, M. Reichert, B. R. Donald, and D. P. Greenberg. Realtime robot motion planning using rasterizing computer graphics hardware. In SIGGRAPH ’90: Proceedings of the 17th annual conference on Computer graphics and interactive techniques, pages 327–335, New York, NY, USA, 1990. ACM Press. [13] P. M¨uller, P. Wonka, S. Haegler, A. Ulmer, and L. V. Gool. Procedural modeling of buildings. In SIGGRAPH ’06: ACM SIGGRAPH 2006 Papers, pages 614–623, New York, NY, USA, 2006. ACM. [14] P. M¨uller, G. Zeng, P. Wonka, and L. V. Gool. Imagebased procedural modeling of facades. ACM Trans. Graph., 26(3):85, 2007. [15] Y. I. H. Parish and P. M¨uller. Procedural modeling of cities. In SIGGRAPH ’01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 301–308, New York, NY, USA, 2001. ACM. [16] Y. I. H. Parish and P. M¨uller. Procedural modeling of cities. In SIGGRAPH ’01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 301–308, New York, NY, USA, 2001. ACM Press. [17] P.P. V´azquez, M. Feixas, M. Sbert, and W. Heidrich. Viewpoint selection using viewpoint entropy. In VMV ’01: Proceedings of the Vision Modeling and Visualization Conference 2001, pages 273–280, 2001. [18] P. Wonka and D. Schmalstieg. Occluder shadows for fast walkthroughs of urban environments. In P. Brunet and R. Scopigno, editors, Computer Graphics Forum (Eurographics ’99), volume 18(3), pages 51–60. The Eurographics Association and Blackwell Publishers, 1999. [19] S. Yi, M. Haralick, and L. G. Shapiro. Automatic sensor and light source positioning for machine vision. In In Proceedings of the 10th International Conference on Pattern Recognition, pages 55–59, Piscataway, NJ, USA, 1990. IEEE Press.
In: Computer Animation Editors: J.S. Wright and L.M. Hughes, pp. 157175
ISBN: 9781607415596 © 2010 Nova Science Publishers, Inc.
Chapter 7
CONSTRAINTBASED AND FEATUREBASED CAD SYSTEMS AND APPLICATIONS Ioannis Fudosa and Vasiliki Stamatib Department of Computer Science, University of Ioannina, Greece
Abstract A new generation of Computer Aided Design systems has become available in which geometric constraints can be defined to determine properties of large designs. The new design concept, often called constraintbased design or design by features offers users the capability of easily defining and modifying a design, but introduces the problem of solving complicated, not always well defined, constraint problems. Traditional parametric models can also be enhanced to partially support declarative constraintbased descriptions. We provide an overview of representation schemes for CAD applications. Then we present a survey of methods for geometric constraint solving appropriate for Computer Aided Design. We demonstrate how these representations and constraint solving methods can be combined or adapted to support a broad range of CAD applications by presenting two example cases of successfully using a featurebased constraintbased representation scheme to support two different CAD applications.
1. Introduction Computer Aided Design has been the motivation for major breakthroughs in various fields of computer science such as computer graphics, visualization, and computer architecture. On the other hand, advancements in graph theory, geometric constraint solving, algorithms and data structures have enabled the use of computers in various fields such as manufacturing, VLSI design, reverse engineering, and restoration of artifacts. This new setting has established Computer Aided Design as a major framework for designing and editing machine parts, jewelry, archaeological findings, buildings, electronics and computers. We present a
a b
Email addresses:
[email protected] Email addresses:
[email protected]
158
Ioannis Fudos and Vasiliki Stamati
framework for representing and editing CAD models for various applications based on two main concepts: Local characteristics which are commonly called features. These characteristics may determine structural properties such as connectivity and hierarchy of objects, geometric properties such as dimensions, distances, angles, smoothness, inclusion, and other topological relations and finally functionality properties that usually are application dependent and determine the behavior of the system when performing its required tasks. Local or global constraints imposed on the model to enforce complex geometric structures and advanced functionality. Such constraints may be part of a feature or span a number of different features. Constraints are clearly declarative, meaning that there is no suggested enforcement procedure. The systems determine based of the application and the user interaction how to enforce the system of geometric constraints. The rest of this is structured as follows: Section 2 presents a comparative survey of representation models for CAD applications. The hybrid featurebased and constraintbased scheme is argued to be the most appropriate for a diverse collection of CAD applications. Section 3 presents methods to tackle the bottleneck problem of geometric constraint solving. Traditional methods are described and modern approaches that are not domain sensitive are presented. Section 4 presents two example cases of applying the framework to two different CAD applications. Section 5 offers conclusions.
2. Computer Aided Design Representation Schemes There is a variety of geometric representations that can be used at different levels of CAD applications. The suitable representation scheme for each application depends on the scope of the application and its peculiarities. Some modeling types are simple and aim at providing only an external representation of the object, whereas others aim at encapsulating and providing additional knowledge and data, such as design intent, functionality, and editability. In the following we study common modeling schemes used in CAD applications. An object can be represented in the simple form of raw data, such as a point cloud corresponding to points on the surface of the object. A widespread scheme in solid modeling is the Boundary Representation (Brep) model where the facets and edges that describe the boundary of a solid are modeled using a connectivity graph and a collection of surface and edge patches. On the other hand, Constructive Solid Geometry (CSG) and volume models handle objects as 3D solids. There are also higherlevel representation schemes that capture not only the shape of the object but also provide information pertaining to design intent and functionality, which can be used later on for reparameterization and modification. We briefly describe each scheme and evaluate its suitability for various CAD applications.
2.1. Raw Data The most basic and simple way to represent a 3D object is as raw data. By raw data we mean an unstructured collection of geometric primitives such as a point cloud or a range image. Such data are usually produced directly from a 3D object scanning or 3D reconstruction setting. The density of the data sets produced by these methods depends on the sampling rate
ConstraintBased and FeatureBased CAD Systems and Applications
159
used to acquire information from the object’s surface. Also, very often the point clouds obtained contain noisy data due to physical characteristics of the object or limitations and regulations of the acquisition method used. However, this problem has been dealt with and processing methods have been suggested that overcome this problem. The characteristic of this representation model is that it describes the object as discrete data, i.e. points, without providing any information about the connectivity, the topological relation among geometric primitives or the design intent. This type of representation is mainly used in pointbased modeling, i.e. [1], [2] and reverse engineering applications [3].
2.2. Boundary Representation (Brep) A more common representation model in CAD applications is the boundary representation model (Brep), which describes the edges and facets of the boundary of the object. This type of model consists of a collection of surface patches. Surfaces can capture objects of complex and freeform design. Thanks to advancements in computer graphics hardware we are able to handle efficiently the CPUintensive processing required by Brep. These factors have resulted in the increased usage of this representation in a wide spectrum of applications. A Brep model is often realized as a mesh of triangular or quadrilateral (and in general polygonal) planar or higher degree surface facets. Planar polygonal meshes (called polyhedral representations) are mostly suited for rendering and virtual reality and not for CAD applications since they do not provide sufficient detail. Often, other representation schemes are converted to polygonal representations for the purpose of rendering. Polyhedral representations such as triangulations are also used in reverse engineering applications, usually as intermediate representations during the reengineering process. A drawback of representing a 3D object with a polygonal mesh is that it cannot capture design semantics, such as design intent, inter part relations and overall behavior. Also model editing is only feasible in a local corrective sense. Smooth object surfaces cannot efficiently and accurately be represented by a polygonal mesh, even when a large number of polygons are used, since the polyhedral representation by definition cannot accommodate for G1 continuity. For example, to render areas of high curvature quite accurately we need to increase the number of polygons and decrease significantly the facet size. Overall, polyhedral representation is not suitable for describing objects with specific design characteristics and functionality, such as mechanical and industrial parts. Also it is not appropriate for describing complex and detailed objects since then the large number of polygons needed to sufficiently approximate the initial object makes the method unaffordable both timewise and spacewise. Applications such as aesthetic and industrial engineering, reverse engineering and jewellery design use commonly nonplanar surfaces to capture the boundaries of complex objects [4]. A Brep model may be constructed using NURBS (NonUniform Rational BSplines) or other parametric surface patches. This type of representation is useful in applications where freeform surfaces are part of the repertoire of primitive geometric entities. Brep can capture almost any type of object, such as mechanical parts and objects of aesthetic design. Surfaces can be described using appropriate parametric representations. Brep models make editing of local features feasible by interactively placing control points,
160
Ioannis Fudos and Vasiliki Stamati
therefore modifying the shape or curvature of the object’s feature. However, Brep models on their own do not capture higher design characteristics of the object such as functionality and part relationships. The information provided through this type of model is limited and does not provide tools for modifying parts of the model that affect the whole design. Therefore, Brep models are used in combination with other techniques (e.g. features, constraints) to obtain higherlevel descriptions that correspond to more flexible and useful models that are suitable for CAD applications. For instance, in [5], the authors present a beautification process based on constraints which is performed on Brep models constructed from reverse engineering range data. Brep models acquired by reengineering can present various inaccuracies and errors, therefore the authors suggest the beautification of the models by describing topological regularities in terms of geometric constraints.
2.3. Volume Modeling While surface raw data and Brep modeling schemes provide data concerning the boundary of a model, constructive solid geometry (CSG) and volumetric models represent the objects as a volume. This type of representation can be used for objects that Brep cannot sufficiently describe. For example, a Brep model cannot represent unambiguously a sphere containing a hollow, whereas a volume model can easily capture such solids. Constructive solid geometry (CSG) models are created by performing Boolean operations on solid primitives e.g. spheres, cones, cylinders and cubes. We perceive that CSG models represent objects that can be created from solid primitives. Unless we use a very large number of primitives we cannot use CSG to model higher degree freeform objects. In general, the CSG representation scheme is well suited for mechanical part design and for all applications where the design history can be expressed as a tree of Boolean operations on geometric primitives. Also editing and local shape modification is performed by intervening in the appropriate operation (internal tree node). Converting CSG models to renderable ones is extremely difficult and therefore CSG is commonly used in conjunction to Brep. In this case a Brep model is always maintained and every modification is transformed to an incremental Brep editing operation. Constraints may also be used in conjunction to CSG for performing multiple internal node modifications at a single step. Volume pixels (voxels) are used in a volumetric approach to 3D object representation. A voxel is a geometric primitive and represents the smallest discrete volume used in this representation scheme. Voxelbased representations are commonly used for visualizing unstructured 3D volume date such as data from scientific computing, medical imaging etc. Although used in early CAD/CAM settings, volumetric representations have been proven to be very inefficient for computer aided editing, rendering and manufacturing. This representation scheme may be used as redundant auxiliary information in CAD applications [6] such as solid modeling, reverse engineering and featurebased and constraintbased modeling for the purposes of physical modeling and simulation.
ConstraintBased and FeatureBased CAD Systems and Applications
161
2.4. HigherLevel Representations in CAD A current promising trend in computeraided design is to use higherlevel structures for model representation. These structures are based on one of the former representation types in combination with additional structural, topological or other information. A featurebased representation scheme describes the object as a combination of features, which are surfaces or solid parts with specific characteristics. A constraintbased representation scheme uses geometric constraints enforced on the model and its features to obtain a more accurate representation that captures designer requirements. The skeleton of a model can also be considered as a higherlevel CAD representation that can be used for specific operations such as feature detection and extraction. More specifically, the featurebased model is a representation scheme that is growing more and more popular. The model is described by defining collections of feature elements and relationships among them. The features are collections of points, surfaces or other features. For example a commonly used feature type is a crosssection of a solid. Constraints are applied to the features to create more accurate and robust models, but also for enforcing global criteria such as tolerance and beautification. This type of model representation has been established initially for manufacturing mechanical parts, where a library of features is created and then relationships among feature elements are enforced. The featurebased scheme is well suited to industrial design in general since it provides for advanced editability. This is due to the knowledge encapsulated by the model concerning tolerances, constraints, relationships and connectivity. For this reason, featurebased methods are often characterized as knowledgebased. Their main objective is to exploit any knowledge and information pertaining to design intent, functionality and construction process. Besides, this representation scheme supports collaborative CAD, reverse engineering and VLSI applications. This type of model also provides the userdesigner with the capability of editing, redesigning and reconstructing the original design, depending on her preferences and needs by tailoring the model features [7]. A powerful higherlevel structure for representing objects is the constraintbased scheme, which is often used in combination with features [8]. This representation scheme is particularly preferred in CAD applications where the objects being modeled, modified and manufactured are of geometric or freeform design and must conform to constraints determined locally on specific components or globally on the whole model [9]. Constraints defined on a model or its individual components can refer to almost any characteristic, i.e. geometric attributes, such as size and shape, topological characteristics, such as placement and connectivity, functionality and behavior. Constraintbased models are widely used in architecture, mechanical engineering, electronic design, aesthetic and industrial design, for design, modeling or reengineering. The types of constraints defined depend on the nature of the CAD application. For example, in VLSI CAD a geometric constraint scheme may be used in conjunction to featurebased or other graphbased connectivity modeling. Constraints are imposed on each design feature used in the VLSI circuit referring to the feature’s intraconnectivity and its local characteristics (i.e. area, size, geometry). Contraints may also be imposed to express interfeature connectivity requirements. Finally, constraints are also enforced globally on the circuit, and are targeted to optimize the overall placement and routing of the features on the chip.
162
Ioannis Fudos and Vasiliki Stamati
An object can also be represented by its skeleton. By skeleton we mean the closure of all points that have more than one closest point on the shape boundary (for example the medial axis transform). This representation provides the topology and shapes that exist in the object and also reflects the symmetries of an object. Depending on the type of application the skeleton is used for, it may be a 2D or 3D representation. For instance, in 3D the medial axis transform produces a medial surface. The exact computation of the 3D skeleton is a computationally intensive problem that returns a skeleton as complex as the object itself. Therefore we usually seek for an approximation. A skeleton representation scheme is used in various CAD applications for object recognition and retrieval [10], animation [11] and other solid modeling operations ([12], [13]). It is widely used in featurebased modeling, where it can be employed to describe the shape of features, in feature detection and extraction applications and shape deformation, for instance refer to [14] and [15].
3. Geometric Constraint Solving Higher level representations are powerful, accurate and user friendly. However they have a major bottleneck; all complicated functionality has been shifted to the solution of a large nonlinear system of geometric constraints with multiple valid solutions. In this section we present an overview of approaches to geometric constraint solving. We outline the most representative methods and evaluate their behavior in terms of the major concerns faced in CAD/CAM systems: solution selection, interactive speed, editability, handling of over and underconstrained configurations and scope [16].
3.1. Numerical Constraint Solvers In numerical constraint solvers, the constraints are translated into a system of algebraic equations and are solved using iterative methods. To handle the exponential number of solutions and the large number of parameters, iterative methods require sharp initial guesses. Also, most iterative methods have difficulties handling overconstrained or underconstrained instances. The advantage of these methods is that they have the potential to solve large nonlinear system that may not be solvable using any of the other methods. All existing solvers more or less switch to iterative methods when the given configuration is not solvable by the native method. This fact emphasizes the need for further research in the area of numerical constraint solving. Sketchpad [17] was the first system to use the method of relaxation as an alternative to propagation. Relaxation is a slow but quite general method. The NewtonRaphson method has been used in various systems [18] [19], and it proved to be faster than relaxation but it has the problem that it may not converge or it may converge to an unwanted solution after a chaotic behavior. For that reason, Juno [18] uses as initial state the sketch interactively drafted by the user. However, NewtonRaphson is so sensitive to the initial guess [20], that the sketch drafted must almost satisfy all constraints prior to constraint solving. A sophisticated use of the NewtonRaphson method was developed in [21], where an improved way for finding the inverse Jacobian matrix is presented. Furthermore, the idea of dividing the matrix of constraints into submatrices as presented in the same work has the potential of providing the
ConstraintBased and FeatureBased CAD Systems and Applications
163
user with useful information regarding the constraint structure of the sketch. Though this information is usually quantitative and nonspecific, it may help the user in basic modifications. To check whether a constraint problem is wellconstrained, Chyz [22] proposes a preprocessing phase where the graph of constraints is analyzed to check whether a necessary condition is satisfied. The method is however quite expensive in time and it cannot detect all the cases of singularity. An alternative method to NewtonRaphson for geometric constraint solving is homotopy or continuation [23], that is argued in [24] to be more satisfactory in typical situations where NewtonRaphson fails. Homotopy, is global, exhaustive and thus slow when compared to the local and fast Newton's method [25], however it may be more appropriate for CAD/CAM systems when constructive methods fail, since it may return all solutions if designed carefully.
3.2. Constructive Constraint Solvers This class of constraint solvers is based on the fact that most configurations in an engineering drawing are solvable by ruler, compass and protractor or using other less classical repertoires of construction steps. In these methods the constraints are satisfied in a constructive fashion, which makes the constraint solving process natural for the user and suitable for interactive debugging. There are two main approaches in this direction.
RuleConstructive Solvers Ruleconstructive solvers use rewrite rules for the discovery and execution of the construction steps. In this approach, complex constraints can be easily handled, and extensions to the scope of the method are straightforward to incorporate [26]. Although it is a good approach for prototyping and experimentation, the extensive computations involved in the exhaustive searching and matching make it inappropriate for real world applications. A method that guarantees termination, ruler and compass completeness and uniqueness using the KnuthBendix critical pair algorithm is presented in ([27], [28]). This method can be proved to confirm theorems that are provable under a given system of axioms [29]. A system based on this method was implemented in Prolog. Aldefeld in [30] uses a forward chaining inference mechanism, where the notion of direction of lines is imposed by introducing additional rules, and thus restricting the solution space. A similar method is presented in [31], where handling of overconstrained and underconstrained problems is given special consideration. Sunde in [32] uses a ruleconstructive method but adopts different rules for representing directed and nondirected distances, giving flexibility for dealing with the solution selection problem. In [33], the problem of nonunique solutions is handled by imposing a topological order on three geometric objects. An elaborate description of a complete set of rules for 2D geometric constraint solving can be found in [34]. In their work, the scope of the particular set of rules is characterized. [35] presents an extension of the set of rules of [34], and provides a correctness proof based on the techniques of [36].
164
Ioannis Fudos and Vasiliki Stamati
GraphConstructive Solvers The graphconstructive approach has two phases. During the first phase the graph of constraints is analyzed and a sequence of construction steps is derived. During the second phase these construction steps are followed to place the geometric elements. These approaches are fast and more methodical. In addition, conclusions characterizing the scope of the method can be easily derived. A major drawback is that as the repertoire of constraints increases the graphanalysis algorithm needs to be modified. Fitzgerald [37] follows the method of dimensioned trees introduced by Requicha [38]. This method allows only horizontal and vertical distances and it is useful for simple engineering drawings. Todd in [39] first generalized the dimension trees of Requicha. Owen in [40] presents an extension of this principle that includes circularly dimensioned sketches. DCM [41] is a system that uses some extension of Owen's method. [42] presents an elaborative graphconstructive method, with fast analysis and construction algorithms, and extensions for handling classes of nonsolvable, underconstrained and consistently overconstrained configuration
3.3. Propagation Methods Propagation methods follow the approach met in traditional constraint solving systems. In this approach, the constraints are first translated into a system of equations involving variables and constants. The equations are then represented by an undirected graph which has as nodes the equations, the variables and the constants, and whose edges represent whether a variable or a constant appears in an equation. Subsequently, we try to direct the graph so as to satisfy all the equations starting from the constants. To accomplish this, various propagation techniques have been used but none of them guarantees to derive a solution and at the same time have a reasonable worst case running time. For a review of these methods see [28]. In a sense, the constructive constraint solvers can be thought of as a sub case of the propagation method (fixed geometric elements for constants and variable geometric elements for variables). However, constructive constraint solvers utilize domain specific information to derive more powerful and efficient algorithms.
3.4. Symbolic Constraint Solvers In symbolic solvers, the constraints are transformed to a system of algebraic equations which is solved using methods from algebraic manipulation, such as Grobner basis calculation [43] or Wu's method [44]. Although, these methods are interesting from a theoretical viewpoint, their practical significance is limited, since their time and space complexity is typically exponential or even hyperexponential.
3.5. Hierarchical and Hybrid Approaches A major result in analysis of constraint graphs by [45] in which an efficient method for detecting dense constraint subgraphs is described has enabled the solution of large systems of
ConstraintBased and FeatureBased CAD Systems and Applications
165
geometric constraints in 2, 3 or more dimensions. By using this result we can build efficient algorithm for solving arbitrary systems of geometric constraints. We first find a set of minimal disjoint dense constraint subgraphs. Each subgraph is then reduced in a supernode of high dimension and the method is applied recursively to the resulting graph. In this way we build a hierarchy of constraint graphs that is treated bottom up or top down based on the application. Interfeature 3D constraints result in systems of 3D constraints. Such systems are very hard to solve with graph constructive methods since there is not even a necessary and sufficient condition for wellconstrainedness in 3D. By using the decomposition suggested by this approach we may breakdown the large geometric constraint system in a multitude of small systems with few variables. Such systems are usually easy to solve using global optimization with topological constraints to narrow down the root selection process. In this direction [46] has developed a novel method for placement and routing in VLSI by constructing a circuit hierarchy by detecting dense connectivity graphs and then employing global optimization algorithms for each subproblem.
4. CAD Applications from a FeatureBased/ConstraintBased Point of View Since the models constructed by CAD applications are most often meant for manufacturing or production in general, it is necessary that they are robust and accurate. Also, most applications require that the model can be modified and reengineered. The use of constraints and features in such CAD applications is essential and is the only representation scheme that sufficiently supports these requirements. In this section we will present two example approaches that adopt this modeling scheme.
4.1. Parametric Feature Based Design in Manufacturing Systems An example of CAD application where the feature and constraintbased representation model is most appropriate is parametric featurebased manufacturing [47]. Parametric modeling was commonly used for the construction of complex models on which parameters were used to provide for subsequent customization. The parameters defined during the design and modeling process are relative to the individual geometric characteristics of the model or to the model as a whole. For example, the parameters can control characteristics such as length, height, width and hole radius. On the other hand, featurebased modeling is a representation scheme based on the combination of individual feature components. In this context a feature is a unit that can be defined as a connected set of geometric elements (i.e. a subpart) associated with attributes that describe its shape and behavior, such as geometry, topology, functionality and connectivity with other features. In traditional approaches each feature is linked to a set of local parameters that control its attribute values. Here, the featurebased model is complemented through the use of local and global constraints. The constraints are applied locally, in reference to the parameter values or the geometric characteristics of the primitives of the features to impose design or userdefined specifics such as hole size, pocket depth, and globally, in reference to the connectivity and the interfeature relations of the model.
166
Ioannis Fudos and Vasiliki Stamati
Much work has been performed on the definition of features in relation to various CAD applications. Features are often perceived as 3D solid components that can be classified into feature libraries depending on their shape or geometry. This point of view is described for instance in [48], where the authors present a library of features fit for manufacturing applications, and in [49], where design features for machining are examined. In [50] features are defined as pierced voxels that are used to create traditional pierced jewellery. However, features can also be defined from surfaces, which is especially common in freeform design applications. For instance, [51] examines freeform surface features, whereas [52] presents a taxonomy of freeform features. Other work uses the notion of feature points and feature lines ([53], [54]) for applications usually related to data segmentation for reverse engineering, or shape deformation and manipulation. Since the definition of a feature is not strict, Hoffmann and JoanArinyo in [55] suggest the use of userdefined features in feature based modeling. Parametric and feature based modeling is an essential component of current CAD design systems. In traditional CAD systems, CSG and Brep models are created by adding and subtracting parts in the model and by applying transformations and various design operations. Design intent was not a concern in these systems and therefore precise editing that involved structural and arbitrary topological modifications of parts of the model was almost impossible without rebuilding the model from scratch. Editing a part of the model is feasible if the design steps are undone until the model returns to the previous state, when the part was created. This of course is possible if the design history of the model is recorded and it is obvious that even though editing theoretically concerns a part of the model, ultimately the whole design process is affected. Feature based CAD systems overcome this limitation by capturing design intent. Since the models are constructed using parameters and features, local editing is possible without necessarily affecting the whole model. Changes are propagated through the model based on the parameters and constraints defined in the system and based on the attributes and connectivity of the features. Featurebased constraintbased modeling systems provide libraries of feature components to be used in the design process and some support userdefined features. Applications such as custom design are feasible since components of models can be combined or redesigned to satisfy user defined preferences or requirements. Many commercial CAD modeling systems support parametric and/or featurebased modeling. Systems such as PRO/Engineer [56], AUTOCAD [57], IRONCAD [58], CATIA [59], Solidworks [60], SolidEdge [61] and Alibre design [62], which have been developed mainly for mechanical engineering, manufacturing and industrial design applications, have been integrated with parametric and/or feature based modeling capabilities. Architectural Desktop and AUTOCAD are CAD systems used in architectural applications that support parametric modeling. 3D Studio Max [63] and Maya [64] are parametric featurebased modeling systems used for artwork and animation. There are also systems that have been developed for specific CAD applications, such as jewelry, clothing and textile design, marine applications and furniture. The above modeling systems are very efficient for manufacturing and production applications. However more freeform applications, such as aesthetic and custom design, are still challenging even with these systems. An interesting case is jewellery design. A large number of CAD systems for jewellery design are parametric featurebased. They provide graphical interfaces with excellent rendering capabilities. The majority of these systems provide builtin libraries of settings and cut gems and stones and advanced featurebased design tools. Some systems provide advanced functionality that provides the use of builders
ConstraintBased and FeatureBased CAD Systems and Applications
167
for recording design steps and for defining parameter values for parts to be used in the process. Also, the majority of these systems have the capability of exporting models to rapid prototyping machines. However, in most CAD systems for jewellery, designing is performed manually using various tools and usually the design steps cannot be programmed to be executed automatically and accurately. This means that each different piece of jewellery has to be created basically from the beginning by hand, making custom design applications difficult and timeconsuming. Also these systems require that the user has designing skills or knowledge of using CAD systems. In the following we will present an interesting example of jewellery application that is difficult to carry out with existing CAD systems: the construction of traditional pierced jewellery, .
Figure 1. Using a chisel to create carvings around a hole.
Figure 2. A structural element (feature).
In [50] ByzantineCAD, a featurebased CAD system suitable for the design of pierced Byzantine jewellery is presented. The system is automated and parametric meaning that the userdesigner sets some parameter values and ByzantineCAD creates the jewellery model that corresponds to the specified values. This provides the designer with the ability to rapidly create customdesigned jewellery, based on the preferences of the customers such as including their initials on a ring. ByzantineCAD introduces a featurebased and voxelbased approach to designing jewellery, through the definition of elementary structural elements with
168
Ioannis Fudos and Vasiliki Stamati
specific attributes and properties that are used as building blocks to construct complex pierced designs. More specifically, pierced Byzantine jewellery are gold jewels with pierced designs that were made along the coastlines of the eastern Mediterranean Sea during the period 3rd –7th century A.D. Their originality is due to the particular processing technique that is used for their creation resulting in a special aesthetic effect. Pierced jewellery was created from thin sheets of gold. The designs were engraved on these sheets of gold with a thin sharp tool. After the outlining of the designs, holes following their shape were created and these were decorated with triangular carvings, using an iron chisel.
(a)
Figure 3. Pierced voxel elements such as (a) are used as features to create complex solid plaques representing designs, i.e. letters or words, that are sized and modified appropriately to construct customdesigned jewellery (i.e. ring).
In ByzantineCAD a feature library of carved, pierced voxel elements is defined in accordance to the craftsmanship used in traditional Byzantine jewellery. The design of pierced jewellery is made up of cylindrical holes that have carvings around them. Each hole with the corresponding carvings around it is considered for the purposes of reconstruction as a structural element (feature). Each feature is a solid made of a rectangular parallelepiped with a cylindrical hole and the corresponding carvings around the hole (figures 1,2) . According to the aesthetic rules that characterize traditional pierced jewellery, all structural elements have the same size but differ in the position of the hole and the carvings around it. The hole can be located either in the center of the parallelepiped or in the center of any of the four quarters.
ConstraintBased and FeatureBased CAD Systems and Applications
169
Note that, in terms of computer aided design and manufacturing, the cylindrical hole can be positioned anywhere in the rectangular parallelepiped; the above restriction follows from careful interpretation of the traditional artistic patterns used. Attributes of these feature elements are characteristics such as the number of carvings around the cylindrical hole, the position of the hole in the parallelepiped, the directions of the carvings and more. A large number of different structural elements can be created by a hole and various carvings and, since not all of these feasible feature elements are valid for use in creating pierced designs, restrictions concerning the carving directions are defined based on aesthetic and artistic rules. These feature elements are combined like 3D building blocks to create complex carved plaques representing pierced designs (Figure 3). The structural elements are placed side by side, either on top, bottom, right or left of each other, and unioned into a new object. The rules determining how the different features can be combined are defined by the designs to be recreated. The construction of these plaques is constrained by the parameter values defined by the userdesigner in reference to characteristics of the plaque such as length and width. The plaques are then used to create jewellery such as rings and necklace pendants. By parameterizing the process of creating pierced jewellery, it is very easy to modify characteristics of the jewellery such as the size and the designs represented.
4.2. FeatureBased Modeling for Reverse Engineering Reverse engineering aims to analyze a real object and to determine its characteristics and mechanisms, with further aim to reconstruct and remanufacture it. The data concerning the physical object can be obtained by various methods. A common method is using a 3D laser scanner to obtain a point cloud corresponding to points on the surface of the scanned object. Given this, a more specific definition of reverse engineering would be to define it as the process of obtaining a geometric CAD model from measurements acquired by scanning an existing physical model [65]. Reverse engineering is vital for various industries because the computer models acquired help improve the quality and efficiency of designs and also speed up the manufacturing and analysis process. In mechanical part engineering and manufacturing, reverse engineering aims to replicate existing parts for which no CAD models exist. Also, it is possible to manufacture objects for which the original CAD model no longer corresponds to the physical part that was manufactured due to subsequent undocumented modifications made after the initial design stage. Reverse engineering is applied in industrial design, such as automobile exterior parts design. Stylists and artists very often create physical models of their concepts by using clay, plaster or wood. These realscale models are then used to create CAD models for manufacturing the objects on an industrial scale. Also the CAD models provide the artists and stylists with the ability to reevaluate their designs, especially when they can easily redesign or modify them as needed. Reverse engineering encourages conceptual design because the designer creates an initial prototype, scans it and manipulates it as desired. Reengineering objects of freeform design is relatively more difficult and complex than reconstructing mechanical parts. Mechanical parts usually have specific geometric characteristics, such as symmetries and swept profiles that are fairly easy to detect and parameterize. However, in the case of freeform objects, there are often features that are difficult to extract. In the case of restricted mechanical parts, features libraries can be defined
170
Ioannis Fudos and Vasiliki Stamati
and used for detecting features in the point cloud, whereas for freeform objects this is not feasible. Since reverse engineering applications aim at reproduction and manufacturing, the reengineering process must create highly accurate and robust CAD models. A characteristic of the objects reengineered is that they are usually parts of larger objects and therefore have to fit and connect exactly with other parts, like pieces in a puzzle. For this reason, the models created through the reverse engineering process must be well defined and constrained [66]. Also, in some cases, the model created is to be edited in i.e. custom designing, therefore a simple Brep model is not appropriate. Given this, it is natural that featurebased and constrained based models are used more and more in reengineering applications. A featurebased reverse engineering method was also used in [67] for reverse engineering a mannequin for garment design. The basic concept in this method is to create a generic mannequin model of a human torso, which is appropriately aligned with the 3D point cloud of the desired human torso model, and the generic model is “fitted” to the point cloud by matching up characteristic points of the models e.g. peaks. This method creates parameterized models by exploiting the features of the object and by using them to constrain the fitting process. It is an automated approach to reverse engineering human torsos that creates parameterized models with good accuracy. Constraint definition and application has been used in building reconstruction and reverse engineering objects of aesthetic design. Specifically, [68] examines how a priori knowledge can be used to derive constraints to create more accurate models in architectural applications. Relevant work is also found in [69]. In [70] the authors suggest a constraintbased approach in reverse engineering for model beautification.
Figure 4. A point cloud of a screwdriver [74] (left) and its concavity intensity map (right).
A paradigm of featurebased/constraintbased reengineering is the REFAB ([71], [48]) project, which uses a featurebased and constraintbased method to reverse engineer mechanical parts. REFAB is a human interactive system where the 3D point cloud is presented to the user, and the user selects from a list a feature that exists in the cloud, specifies with the mouse the approximate location of the feature in the point cloud, and the system then fits the specified feature to the actual point cloud data using a least square means method iteratively. The authors give emphasis on the fitting of pockets, where the user draws a profile of the pocket on the point cloud and the system then fits the profile to the data and the profile is then extruded to create the pocket. This featurefitting process is made more
ConstraintBased and FeatureBased CAD Systems and Applications
171
accurate by using constraints that are detected by the system, verified by the user and then exploited to achieve a better fitting of the features according to the data. The system supports constraints [72] such as parallelism, concentricity, perpendicularity and symmetry. The constraints defined and used in REFAB seek to reduce the degrees of freedom associated with the object as much as possible, so as to achieve high precision models in less time. An interesting featurebased approach to reengineering objects of freeform design is presented in [73]. Reengineering objects of freeform design is essential for supporting custom design in a CAD model reconstruction system. It provides userdesigners with the capability to modify reengineered CAD models according to their preferences and to incorporate in novel designs. For instance, in the case of jewellery reengineering, the userdesigner might like to be able to modify the dimensions of a ring to produce one of larger size, or be able to choose certain parts of the object to use them to create other pieces of jewellery. To this end, one needs to exploit the features of the original model and the relationships and constraints that hold among them. In [73] a generic and global feature detection approach to reverse engineering point clouds of objects of freeform design is presented. A method is presented for detecting and segmenting a point cloud into individual subsets that correspond to features. This is achieved by using a point characteristic defined as “concavity intensity” to decompose the point cloud into subsets (components) that correspond to features of the physical object. The concavity intensity of a point corresponds to the smallest distance from the point to its convex hull that does not pass through the point cloud. This feature basically detects concave features in the object being reengineered. A concavity intensity map is shown in figure 4. In the concavity intensity map the values are rendered using greyscale, where black corresponds to points belonging to the convex hull, whereas white corresponds to points that are farthest away from the convex hull. We can observe that edges (rapid variations in concavity intensity), saddle points and extrema can be used to partition the point cloud into components that can later be refined as the features of the object.
Figure 5. Feature extraction is performed by region growing.
The point concavity intensity values calculated are used to segment the point cloud into feature components. A feature component is bounded by areas where abrupt changes in the direction of the normal and/or rapid concavity intensity variations are observed. After calculating the concavity intensities of the all the points that form the point cloud, a region
172
Ioannis Fudos and Vasiliki Stamati
growing method to divide the point cloud into its components is applied (figure 5). The region growing method is based on two criteria: i)
the normal vectors of neighboring points belonging to the same region should form an angle smaller than a threshold t and ii) the approximate gradients of the concavity intensity function in directions x, y and z for neighboring points of the same region should maintain the same sign value, meaning that there are no zero crossings observed between them.
5. Conclusion We presented a survey of representation schemes for CAD applications. Then we introduced a framework for representing and performing complex design operations in CAD systems. This framework has evolved from the featurebased design paradigm augmented with arbitrary intrafeature and interfeature constraints and a hierarchical constraint analysis approach for placing geometric elements. To demonstrate the employment of this framework we presented two example applications that we have developed using this new design concept.
References [1] Cripps R.J., Algorithms to support pointbased cadcam, International Journal of Machine Tools and Manufacture 2003, 43(4), 425432. [2] Kobbelt L. and Botsch M., A survey of pointbased techniques in computer graphic, Computers and Graphics 2004, 28(6), 801814. [3] Hoppe H., Derose T., Duchamp T., McDonald J. and Stuetzle W., Surface reconstruction from unorganized points, Computer Graphics 1992, 26(2), 7178. [4] Benko P., Martin R.R. and Varady T., Algorithms for reverse engineering boundary representation models, ComputerAided Design 2001, 33(11), 839851. [5] Langbein F.C., Marshall A.D. and Martin R.R., Choosing consistent constraints for beautification of reverse engineered geometric models, Computeraided Design 2004, 36(3), 261278. [6] Jense G.J., Voxelbased methods for CAD, Computer Aided Design 1989, 21(10), 528533. [7] Hoffmann C.M. and JoanArinyo R., Erep An editable highlevel representation for geometric design, Geometric Modeling for Product Realization, P. Wilson, M. Wozny , M. Pratt, eds., North Holland, 1993, 129164. [8] Benko P., Kos G., Varady T., Andor L. and Martin R.R., Constrained fitting in reverse engineering, ComputerAided Design 2002, 19(3), 173205. [9] Anderl R. and Mendgen R., Modelling with constraints: theoretical foundation and application, Artificial Intelligence in ComputerAided Design 1996, 28(3), 155168. [10] Cornea N.D., Demirci M.F., Silver D., Shokoufandeh A., Dickinson S.J. and Kantor P.B., 3D Object Retrieval using Manytomany Matching of Curve Skeletons, IEEE
ConstraintBased and FeatureBased CAD Systems and Applications
[11]
[12]
[13] [14]
[15]
[16] [17] [18] [19] [20]
[21] [22] [23] [24] [25] [26] [27] [28] [29] [30]
173
International Conference on Shape Modeling and Applications (SMI) 2005, Boston USA, 368373. Bloomenthal J., Medialbased Vertex Deformation, Proceedings of the 2002 ACM SIGGRAPH/Eurographics Symposium on Computer Animation 2002, San Antonio, TX, USA, . Sheehy D., Armstrong C. and Robinson D., Shape description by medial axis construction, IEEE Transactions on Visualization and Computer Graphics 1996, 2(1), 6272. Storti D.W., Turkiyyah G.M., Ganter M.A., Lim C.T. and Stal D.M., Skeletonbased modeling operations on solids, Solid Modeling '97 1997, Atlanta GA USA. Lien J., Keyser J. and Amato N.M., Simultaneous Shape Decomposition and Skeletonization, Proceedings of the ACM Syposium on Solid and Physical Modeling 2006. Cardiff, Wales, United Kingdom. Yoshizawa S., Belyaev A.G. and Seidel HP., Freeform skeletondriven mesh deformation, Proceedings of the eighth ACM Symposium on Solid Modeling and applications 2003, Seattle, Washington U.S.A. Fudos I., Constraint Solving for Computer Aided Design, Dept of Computer Sciences, Purdue University, 1995, PhD Thesis. Sutherland I., Sketchpad, a manmachine graphical communication system. Proceedings of the spring Joint Compo Conference, 1963. Nelson, G., Juno, a constraintbased graphics system, SIGGRAPH 1985, San Francisco USA. Serrano D. and Gossard D., Combining mathematical models and geometric models in CAE systems, Proc. ASME Computers in Eng. Conf 1986, Chicago USA. Beaty P.L., Fitzhorn P.A. and Herron G.J., Extensions in variational geometry that generate and modify object edges composed of rational Bezier curves, ComputerAided Design 1994, 26(2), 98108. Light R. and Gossard D., Modification of geometric models through variational geometry, Computer Aided Design 1982, 14(4), 209214. Chyz W., Constraint management for CSG, MIT, 1985, Master's Thesis. Allgower E.L. and Georg K., Continuation and path following, Acta Numerica 1993, 164. Lamure H. and Michelucci D., Solving geometric constraints by homotopy, Proc. Third Symposium on Solid Modeling and Applications 1995, Salt Lake City, USA. Morgan A., Solving polynomial systems using continuation for engineering and scientific problems 1987, Prentice Hall Inc. Bruderlin B. and Roller D., Geometric constraint solving and applications 1998, Springer Verlag. Bruderlin B., Constructing threedimensional geometric objects defined by constraints, In Workshop on Interactive 3D Graphics 1986. Sohrt W., Interaction with constraints in threedimensional modeling, Dept of Computer Science. The University of Utah, 1991, Master's Thesis. Bruderlin B., Using geometric rewrite rules for solving geometric problems symbolically, Theoretical Computer Science 1993, 116, 291303. Aldefeld, B., Variation of geometries based on a geometricreasoning method, ComputerAided Design 1988, 20(3), 117126.
174
Ioannis Fudos and Vasiliki Stamati
[31] Suzuki H., Ando H. and Kimura F., Variation of geometries based on a geometricreasoning method, Computers & Graphics 1990, 14(2), 211224. [32] Sunde G., Specification of shape by dimensions' and other geometric constraints, in Geometric modeling for CAD applications 1988, M. J. Wozny, H. W. McLaughlin, and J. L. Encarnacao, eds., North Holland, IFIP, 199213. [33] Yamaguchi Y. and Kimura F., A constraint modeling system for variational geometry, in Geometric Modeling for Product Engineering, M. J. Wozny, J.U. Turner and K. Preiss, eds., 1990, Elsevier Science Publishers B.V. (North Holland), 221233. [34] Verroust A., Schonek F. and Roller D., Ruleoriented method for parameterized computeraided design, Computer Aided Design 1992, 24(10), 531540. [35] JoanArinyo R. and Soto A., A ruleconstructive geometric constraint solver, Technical Report LSI9525R, 1995, Universitat Politecnica de Catalunya. [36] Fudos I. and Hoffmann C.M., Correctness proof of a geometric constraint solver, International Journal of Computational Geometry & Applications, 1996, 405420 [37] Fitzgerald W.J., Using axial dimensions to determine the proportions of line drawings in computer graphics, Computer Aided Design 1981, 13(6), 377382. [38] Requicha, A., Dimensionining and tolerancing, Technical report PADL TM19, Production Automation Project 1977, University of Rochester. [39] Todd P., A ktree generalization that characterizes consistency of dimensioned engineering drawings, SIAM J DISC MATH 1989, 2(2), 255261. [40] Owen J.C., Algebraic solution for geometry from dimensional constraints, ACM Symp. Found. of Solid Modeling 1991, Austin, TX. [41] DCubed, Ltd., 68 Castle Street, Cambridge, CB3 OAJ, England. The Dimensional Constraint Manager, June 1994, Version 2.7. [42] Fudos I. and Hoffmann C.M., A graphconstructive method to solving systems of geometric constraints, ACM Transactions on Graphics 1997, 16(2), 179216. [43] Buchberger B., Grobner Bases: An algorithmic method in polynomial ideal theory, in Multidimensional Systems Theory, N.K. Bose, Editor, 1985, D. Reidel Publishing Company, 184232. [44] Wu W.T., Basic principles of mechanical theorem proving in elementary geometries, Journal of Automated Reasoning 1986, 2, 221252. [45] Hoffmann C.M., Lomonosov A., and Sitharam M., Finding solvable subsets of constraint graphs, L. G. Smolka, editor, 1997, Springer Verlag. 463477. [46] Fudos I. and Markouzis D., A hierarchical featurebased approach to computer aided placement and routing for VLSI, Technical Report TR200709, 2007, Computer Science Department, University of Ioannina. [47] Shah J.J. and Mantyla M., Parametric and Featurebased CAD/CAM 1995, John Wiley & Sons Inc. [48] Thompson W.B., Owen J.C., James de St Germain H., Stark S.R. and Henderson T.C., Featurebased reverse engineering of mechanical parts, IEEE Transactions on Robotics and Automation 1999, 12(1), 5766, . [49] Lee J.Y. and Kim K., A featurebased approach to extracting machining features, ComputerAided Design 1998, 30(13), 10191035. [50] Stamati V. and Fudos I., A parametric featurebased CAD system for reproducing traditional pierced jewellery, Computer Aided Design 2005, 37(4), 431449.
ConstraintBased and FeatureBased CAD Systems and Applications
175
[51] Nyirenda P. J., Mulbagal M. and Bronsvoort W. F., Definition of freeform surface feature classes, Computer Aided Design & Applications 2006, 3(5), 665674. [52] Fontana M., Giannini F. and Meirana M., A free form feature taxonomy, In Proceedings of EUROGRAPHICS '99, 1999. [53] Dobson G.T., Waggenspack Jr W.N. and Lamousin H.J., Feature based models for anatomical data fitting, Computer Aided Design 1995, 27(2), 139146. [54] Gumhold S., Wang X. and MacLeod R., Feature extraction from point clouds, Proceedings of the 10th International Meshing Roundtable, 2001. [55] Hoffmann C.M. and JoanArinyo R., On userdefined features, Computer Aided Design 1998, 30(5), 321332. [56] PTC, PRO/ENGINEER, http://www.ptc.com. [57] Autodesk, AUTOCAD, http://www.autodesk.com/autocad. [58] IRONCAD, IRONCAD, http://www.ironcad.com. [59] Dassault Systems, CATIA, http://www.3ds.com/productssolutions/plmsolutions/catia/ overview. [60] Dassault Systems, Solidworks, http://www.3ds.com/productssolutions/ solidworks. [61] Siemens  UGS PLM Software, Solid Edge, http://www.solidedge.com. [62] Alibre Inc., Alibre Design, http://www.alibre.com. [63] Autodesk, 3D Studio Max, http://www.autodesk.com/3dsmax. [64] Autodesk, Maya, http://www.autodesk.com/maya. [65] Varady T., Martin R.R. and Cox J., Reverse engineering of geometric models  an introduction, Computer Aided Design 1997, 29(4), 253330. [66] Werghi N., Fisher R., Robertson C. and Ashbrook A., Object reconstruction by incorporating geometric constraints in reverse engineering, Computer Aided Design 1999, 31(6), 363399. [67] Au C.K. and Yuen M.M.F., Featurebased reverse engineering of mannequin for garment design, ComputerAided Design 1999, 31(12), 751759. [68] Fisher R.B., Applying knowledge to reverse engineering problems, ComputerAided Design 2004, 36, 501510. [69] Cantzler, H., Improving architectural 3D reconstruction by constrained modeling, Institute of Perception, Action and Behaviour, School of Informatics, University of Edinburgh, 2003, PhD Thesis. [70] Gao C.H., Langbein F.C., Marshall A.D. and Martin R.R., Local topological beautification of reverse engineered models, ComputerAided Design 2004, 36(13), 13371355. [71] Thompson W.B., James de St Germain H., Henderson T.C. and Owen J.C., Constructing highprecision geometric models from sensed position data, Proceedings of the 1996 ARPA Image Understanding Workshop, 1996. [72] James de St. Germain H., Stark S.R., Thompson W.B. and Henderson T.C., Constraint optimization and featurebased model construction for reverse engineering, ARPA Image Understanding Workshop, 1997. [73] Stamati V. and Fudos I., A feature based approach to reengineering objects of freeform design by exploiting point cloud morphology. Proceedings of the 2007 ACM symposium on Solid and physical modeling 2007, Beijing China. [74] Cyberware, Cyberware Rapid 3D Scanners  Desktop 3D Scanner Samples, 1999, http://www.cyberware.com/products/ scanners/desktopSamples.html
In: Computer Animation ISBN 9781607415596 c 2010 Nova Science Publishers, Inc. Editors: J.S. Wright and L.M. Hughes, pp. 177208
Chapter 8
C OMPUTER A IDED G EOMETRIC D ESIGN WITH P OWELL S ABIN S PLINES Hendrik Speleers1,2 , Paul Dierckx1 and Stefan Vandewalle1 1
Katholieke Universiteit Leuven, Department of Computer Science, Belgium 2 Research Assistant of the Fund for Scientific Research Flanders, Belgium
Abstract PowellSabin splines are bivariate C 1 continuous quadratic splines defined on an arbitrary triangulation. Their construction is based on a particular split of each triangle in the triangulation into six smaller triangles. In this article we give an overview of the properties of PowellSabin splines in the context of computer aided geometric design. These splines can be represented in a compact normalized Bspline basis with an intuitive geometric interpretation involving control triangles. Using these triangles one can interactively change the shape of the splines in a predictable way. We describe the simple subdivision rules for PowellSabin splines, and discuss some applications. We consider a new efficient spline visualization technique based on subdivision. We also look at two useful generalizations of the PowellSabin splines, i.e., QHPS splines and NURPS surfaces. The QHPS splines are a hierarchical variant of PowellSabin splines. They have very similar properties as the PowellSabin splines, and their hierarchical nature allows a local refinement of the spline in a very straightforward way. The NURPS surface is the rational extension of the PowellSabin spline. By means of weights they give extra degrees of freedom to the designer for the modelling of surfaces.
1.
Introduction
The ability to represent complex surfaces on the computer is important in a broad range of applications [10]. Tensor product Bsplines are today’s most commonly used surface splines in computer aided geometric design (CAGD) packages. They are very attractive because of their compact representation, the ease of implementation and their efficiency. The Bspline control net of such a surface enables local adaptations in a geometrically intuitive way. A definite drawback, however, is that they are restricted to regular meshes
178
Hendrik Speleers, Paul Dierckx and Stefan Vandewalle
on rectangular domains. Therefore, they are not well suited to represent strongly irregular objects defined on arbitrary domains. Instead of using the tensor product representation on rectangular domains, one can also describe piecewise bivariate polynomial surfaces in terms of barycentric coordinates with respect to a triangular domain. These triangular patches in Bernstein form are called B´ezier triangles [9]. In spite of the flexibility of a triangular mesh, the representation of complex smooth surfaces can require a large number of B´ezier triangles. Imposing smoothness conditions between these patches results in a large number of nontrivial continuity relations between the coefficients. A number of authors studied the construction of smooth bivariate spline functions on arbitrary triangulations. A major difficulty is to determine the dimension of such spline spaces. In general, it is not possible to express the dimension in terms of the number of vertices and triangles in the triangulation. There are some results for particular choices of polynomial degree and smoothness [1, 3, 13], and for particular constrained triangulations [11]. Yet, in general and especially for low degree polynomials the problem remains open. One can overcome this problem by using socalled macroelements, where each triangle in the triangulation is split in a particular way. Wellknown in the finite element literature is the C 1 continuous cubic CloughTocher spline space [5]. For C 1 continuous quadratic splines, Powell and Sabin [21] constructed an element by splitting all triangles into six subtriangles. PowellSabin splines can be compactly represented in a normalized basis. Dierckx [6] discovered that the basis splines have an intuitive geometric interpretation by means of control triangles. Windmolders [34] was the first to investigate the use of PowellSabin splines in CAGD applications. Many new interesting properties and techniques have been developed since, e.g., a triadic subdivision scheme. We will overview them in the context of computer aided design and modelling. Section 2. is devoted to the PowellSabin spline space. It recalls some basic concepts of PowellSabin splines and the construction of a suitable normalized basis. In section 3. we describe a subdivision scheme for the splines, and we discuss some applications. The next sections cover two generalizations of PowellSabin splines. Section 4. discusses a hierarchical variant of the splines, called QHPS splines. They retain similar properties as the PowellSabin splines, but they are defined on a hierarchical triangulation. These splines are very useful in an adaptive local refinement strategy. In section 5. we consider NURPS surfaces, a rational extension of PowellSabin splines. They give the designer more degrees of freedom by means of extra weights.
2.
PowellSabin Splines
In this section we detail the theory of PowellSabin splines. We first recall some general concepts of polynomials on triangles in their BernsteinB´ezier representation. PowellSabin splines are defined on arbitrary triangulations refined with a particular split. We discuss a normalized basis for this spline space, and we give an overview of their properties.
Computer Aided Geometric Design with PowellSabin Splines
2.1.
179
Polynomials on Triangles
Barycentric coordinates provide an elegant tool for defining points inside a triangle. Let T (V1 , V2 , V3 ) be a nondegenerated triangle. An arbitrary point P in the plane of the triangle can be uniquely expressed in terms of the barycentric coordinates τ = (τ1 , τ2 , τ3 ) with respect to T , such that P =
3 X
τi Vi ,
and τ1 + τ2 + τ3 = 1.
(2.1)
i=1
If the point P lies inside the triangle T , then its barycentric coordinates are all positive. Consider two points in the plane of the triangle, i.e., P1 and P2 . The barycentric direction δ = (δ1 , δ2 , δ3 ) of the vector P2 − P1 with respect to T is defined as the difference of the barycentric coordinates of both points. If the Euclidian distance kP2 − P1 k = 1, then δ is called a unit barycentric direction. Let Πd denote the linear space of bivariate polynomials of total degree less than or equal to d. Any polynomial pd ∈ Πd on triangle T has a unique BernsteinB´ezier representation [9], X d pd (τ ) = bijk Bijk (τ ), (2.2) i+j+k=d
with d Bijk (τ ) =
d! τ1 i τ2 j τ3 k i!j!k!
(2.3)
the Bernstein polynomials on the domain triangle T . The coefficients bijk are called B´ezier ordinates, and theB´ezier domain points ξijk are defined as the points with barycentric coordinates di , dj , kd . By associating each B´ezier ordinate bijk with the B´ezier domain points ξijk , we can display the BernsteinB´ezier representation schematically as in Figure 1(a) for the case d = 2. The piecewise linear interpolant of the B´ezier control points, defined as (ξijk , bijk ), is called the B´ezier control net. This control net is tangent to the polynomial surface z = pd (τ ) at the three vertices of the triangle, and it mimics the shape of the BernsteinB´ezier surface. Figure 1(b) shows such a surface together with its control points and control net. Polynomials in their BernsteinB´ezier representation (2.2) can be evaluated using the recursive de Casteljau algorithm [4], i.e., pd (τ ) = bd0,0,0 (τ ),
(2.4a)
where b0i,j,k (τ ) = bijk , bri,j,k (τ )
=
i + j + k = d,
τ1 br−1 i+1,j,k (τ )
+
τ2 br−1 i,j+1,k (τ )
i + j + k = d − r,
(2.4b) +
τ3 br−1 i,j,k+1 (τ ), and r = 1, . . . , d.
(2.4c)
This algorithm is numerically stable and has many interesting properties [9]. Besides for evaluation, the algorithm can also be used for computing derivatives and for obtaining continuity conditions on neighbouring triangular patches.
180
Hendrik Speleers, Paul Dierckx and Stefan Vandewalle
b002 V3 b011
b101
V2
V1 b200
b110
b020
(a)
(b)
Figure 1. (a) Schematic representation of a quadratic bivariate polynomial by means of its B´ezier ordinates bijk . (b) A quadratic BernsteinB´ezier polynomial with its control points and control net.
Representing complex surfaces requires the use of a large number of B´ezier triangles. Preserving a certain degree of continuity between all patches results then in a large set of nontrivial relations between their B´ezier ordinates. Therefore, one has looked for piecewise polynomials with inherent global continuity conditions.
2.2.
The PowellSabin Spline Space
Consider a simply connected subset Ω ⊂ R2 with polygonal boundary ∂Ω. Assume a conforming triangulation ∆ of Ω is given, consisting of t triangles Tj , with j = 1, . . . , t, and having n vertices Vk , with k = 1, . . . , n. A triangulation is conforming if no triangle contains a vertex different from its own three vertices. The PowellSabin (PS) refinement ∆∗ of ∆ partitions each triangle Tj into six smaller triangles in the following way: 1. Choose an interior point Zj in each triangle Tj , so that if two triangles Ti and Tj have a common edge, then the line joining Zi and Zj intersects the common edge at a point Rij between its vertices. 2. Join each point Zj to the vertices of Tj . 3. For each edge of the triangle Tj (a) which belongs to the boundary ∂Ω: join Zj to an arbitrary point on that edge; (b) which is common to a triangle Ti : join Zj to Rij . Figure 2(a) displays a triangulation with 8 triangles, and a corresponding PS refinement containing 48 triangles. The space of piecewise quadratic polynomials on ∆∗ with global C 1 continuity is called the PowellSabin spline space: n o S21 (∆∗ ) := s ∈ C 1 (Ω) : sTj∗ ∈ Π2 , Tj∗ ∈ ∆∗ . (2.5)
Computer Aided Geometric Design with PowellSabin Splines
(a)
181
(b)
Figure 2. (a) A PS refinement ∆∗ (in dashed lines) of a given triangulation ∆ (in solid lines). (b) The PS points (bullets) and a set of suitable PS triangles (shaded).
Each of the 6t triangles resulting from the PS refinement is the domain triangle of a quadratic BernsteinB´ezier polynomial. Powell and Sabin [21] proved that the following interpolation problem s(Vk ) = fk ,
∂s (Vk ) = fx,k , ∂x
∂s (Vk ) = fy,k , ∂y
k = 1, . . . , n,
(2.6)
has a unique solution s(x, y) ∈ S21 (∆∗ ) for any given set of n (fk , fx,k , fy,k )triplets. Hence, the dimension of the PowellSabin spline space S21 (∆∗ ) equals 3n.
2.3.
A Bspline Representation
Dierckx [6] presented a geometric method to construct a normalized basis for the spline space S21 (∆∗ ). Every PowellSabin spline can then be represented as s(x, y) =
3 n X X
ci,j Bij (x, y).
(2.7)
i=1 j=1
To obtain the basis functions Bij (x, y), we associate with each vertex Vi three linearly independent triplets (αi,j , βi,j , γi,j ), j = 1, 2, 3. These triplets are determined as follows: 1. For each vertex Vi ∈ ∆ with Cartesian coordinates (xi , yi ), find the corresponding PS points. These points are the immediately surrounding B´ezier domain points of Vi in the PS refinement ∆∗ . The vertex Vi itself is also a PS point. In Figure 2(b) the PS points are indicated as bullets. 2. For each vertex Vi , find a triangle ti (Qi,1 , Qi,2 , Qi,3 ) that contains all the PS points of Vi . These triangles ti , i = 1, . . . , n, are called PS triangles, and we denote their vertices as Qi,j = (Xi,j , Yi,j ). Figure 2(b) shows a possible set of PS triangles.
182
Hendrik Speleers, Paul Dierckx and Stefan Vandewalle
3. The three linearly independent triplets (αi,j , βi,j , γi,j ), j = 1, 2, 3, are derived from the PS triangle ti of vertex Vi as follows: • αi = (αi,1 , αi,2 , αi,3 ) are the barycentric coordinates of Vi with respect to ti , • βi = (βi,1 , βi,2 , βi,3 ) and γi = (γi,1 , γi,2 , γi,3 ) are the unit barycentric directions with respect to ti , in the x and ydirection respectively. Practically, they can be computed as xi X yi 1 Yi,1 1 1 1 i,1 Xi,2 Yi,2 1 , xi yi 1 αi = E E Xi,3 Yi,3 1 Xi,3 Yi,3 1 Yi,2 − Yi,3 Yi,3 − Yi,1 Yi,1 − Yi,2 , , , βi = E E E Xi,3 − Xi,2 Xi,1 − Xi,3 Xi,2 − Xi,1 γi = , , , E E E
with
1 , E
Xi,1 Yi,1 1 Xi,2 Yi,2 1 , xi yi 1
Xi,1 Yi,1 1 E = Xi,2 Yi,2 1 . Xi,3 Yi,3 1
The PowellSabin Bspline Bij (x, y) is defined as the unique solution of the interpolation problem (2.6) with all (fk , fx,k , fy,k ) = (0, 0, 0) except for k = i, where (fi , fx,i , fy,i ) = (αi,j , βi,j , γi,j ) 6= (0, 0, 0). Figure 3 shows an example of three linearly independent PowellSabin Bsplines corresponding to the same vertex. Choice of PS triangles. The set of PS triangles is not uniquely defined for a given PS refinement. One possibility for their construction is to calculate triangles of minimal area, the socalled optimal PS triangles [6]. Computationally, this problem leads to a quadratic programming problem. An alternative (and easier to implement) solution is given in [31], where the sides of the PS triangle are found by connecting two neighbouring PS points. From a practical point of view, other choices may be more appropriate. A particular choice of the PS triangles can, e.g., simplify the treatment of boundary conditions [28]. In such a case it is better to construct PS triangles at the boundary vertices with one side tangential and another normal to the boundary curve. For quasiinterpolation [20] the corners of each PS triangle are preferred to be chosen on edges of the triangulation.
2.4.
Properties of the Bspline Basis
The PowellSabin Bsplines have some nice properties, which are very useful in CAGD and approximation applications. In this section we review some of them. Later on, in section 3., we will discuss in more detail subdivision rules for PowellSabin splines in the representation (2.7).
Computer Aided Geometric Design with PowellSabin Splines
(a)
(b)
(c)
(d)
183
Figure 3. (a) A given triangulation with PS refinement. (b)(d) The three PowellSabin Bsplines Bij (x, y) corresponding to the central vertex Vi and its PS triangle. The contour lines of the basis functions are depicted.
Local support. It is easy to see that each PowellSabin Bspline Bij (x, y) has a local support, because the basis function is zero outside the molecule Mi of vertex Vi . The molecule (also called 1ring) is defined as the union of all triangles in the triangulation that contain Vi . Convex partition of unity. Dierckx showed in [6] that the proposed basis forms a convex partition of unity on the domain Ω, i.e., Bij (x, y) ≥ 0,
and
n X 3 X
Bij (x, y) = 1,
for all (x, y) ∈ Ω.
(2.8)
i=1 j=1
The fact that each PS triangle ti contains all PS points of vertex Vi guarantees the positivity of the basis functions. Stability. Maes et al. [19] proved that the basis is L∞ stable. For the maxnorms kck∞ = max ci,j , i,j
and ks(x, y)k∞ = max s(x, y), Ω
184
Hendrik Speleers, Paul Dierckx and Stefan Vandewalle
Figure 4. A PowellSabin spline and its PS control triangles.
they showed that for all choices of the coefficient vector c, one has that K1 kck∞ ≤ ks(x, y)k∞ ≤ K2 kck∞ ,
(2.9)
where K2 = 1, and K1 depends only on the smallest angle θ∆ in the triangulation ∆ and on the size of the PS triangles. Moreover, the smaller the PS triangles the better (the larger) the stability constant. In [25] an adapted version of the proof is given, resulting in a sharper stability bound. The ratio K2 /K1 yields an upper bound for the condition number of the basis. It reflects the influence of a change in the coefficients on the magnitude of the corresponding spline with respect to the L∞ norm. In [26, 32] the stability of the PowellSabin spline basis is proven in the more general Lp norm with 1 ≤ p ≤ ∞. A stable local basis provides spline approximations of smooth functions with an optimal order [16]. PS control triangles. Referring to the representation (2.7) for PowellSabin splines, we define control points as ci,j = (Qi,j , ci,j ). (2.10) These points lead to PS control triangles Ti (ci,1 , ci,2 , ci,3 ), which are tangent to the spline surface z = s(x, y) at the vertices Vi . The projection of the control triangles Ti in the (x, y)plane are simply the PS triangles ti . Using these control triangles a designer can interactively change the shape of a given PowellSabin spline locally in a predictable way. From property (2.8) it follows that the graph of the spline (2.7) lies inside the convex hull of its control points ci,j . Figure 4 shows a PowellSabin spline surface together with the corresponding control triangles. The spline is taken from [7] and represents a smooth approximation of the func−1 tion exp((x − 0.52)2 + (y − 0.48)2 ) − 0.95 on the domain [−1, 1] × [−1, 1].
Computer Aided Geometric Design with PowellSabin Splines Vk
sk
Qi,3
uk Rki
rj Zijk
Si′
Rjk
Si
ui
si
Rij
ω
wi
Vj
Vi
wk θj
vi
S˜i Qi,1
185
vk ri
θi
wj
θk rk
vj
uj sj
Qi,2 (a)
(b)
Figure 5. (a) PS refinement of a triangle T (Vi , Vj , Vk ), together with the PS triangle ti (Qi,1 , Qi,2 , Qi,3 ) associated with vertex Vi . (b) Schematic representation of the B´ezier ordinates of a PowellSabin spline.
Other bases for the PowellSabin spline space can be found in the literature [2, 17]. Their construction is based on socalled minimal determining sets. They are stable, but they do not form a convex partition of unity and they have no geometric interpretation via control triangles.
2.5.
A BernsteinB´ezier Representation
For further manipulation (e.g. evaluation and differentiation) of a PowellSabin spline in the form (2.7), we can write the spline in a BernsteinB´ezier representation. We consider a single domain triangle T (Vi , Vj , Vk ) ∈ ∆ with its PS refinement ∈ ∆∗ . The other triangles in the triangulation can be treated in the same way. We assume that the points indicated in Figure 5(a) have the following barycentric coordinates: Vi (1, 0, 0),
Vj (0, 1, 0),
Rij (λij , λji , 0),
Vk (0, 0, 1),
Rjk (0, λjk , λkj ),
Zijk (zi , zj , zk ), Rki (λik , 0, λki ).
On each of the six triangles in ∆∗ the PowellSabin spline is a quadratic polynomial, that can be represented in its BernsteinB´ezier formulation, i.e., with d = 2 in equations (2.2) and (2.3). The value of the corresponding B´ezier ordinates is derived in [6]. The outcome is schematically represented in Figure 5(b), with si = αi,1 ci,1 + αi,2 ci,2 + αi,3 ci,3 ,
(2.11a)
ui = Li,1 ci,1 + Li,2 ci,2 + Li,3 ci,3 ,
(2.11b)
L′i,3 ci,3 ,
(2.11c)
˜ i,1 ci,1 + L ˜ i,2 ci,2 + L ˜ i,3 ci,3 . wi = L
(2.11d)
vi =
L′i,1 ci,1
+
L′i,2 ci,2
+
˜ i,1 , L ˜ i,2 , L ˜ i,3 ) are The values of (αi,1 , αi,2 , αi,3 ), (Li,1 , Li,2 , Li,3 ), (L′i,1 , L′i,2 , L′i,3 ) and (L found as the barycentric coordinates of the PS points Vi , Si , Si′ and S˜i , respectively, with
186
Hendrik Speleers, Paul Dierckx and Stefan Vandewalle
respect to the PS triangle ti (Qi,1 , Qi,2 , Qi,3 ). These points are depicted in Figure 5(a). Analogously, we can compute the values of (sj , uj , vj , wj ) and (sk , uk , vk , wk ). The other B´ezier ordinates are derived from the inherent continuity conditions of the PowellSabin spline, e.g., rk = λij ui + λji vj ,
(2.11e)
θk = λij wi + λji wj ,
(2.11f)
ω = z i wi + z j wj + z k wk .
(2.11g)
In this BernsteinB´ezier representation the PowellSabin splines can be easily manipulated using the de Casteljau algorithm (2.4).
2.6.
Parametric PowellSabin Surfaces
A parametric PowellSabin surface is defined as Pn P3 j x x = P i=1 P j=1 ci,j Bi (u, v) y = ni=1 3j=1 cyi,j Bij (u, v) , z = Pn P3 cz B j (u, v) i=1 j=1 i,j i
(u, v) ∈ Ω,
(2.12)
or, compactly, s(u, v) =
n X 3 X
ci,j Bij (u, v),
(2.13)
i=1 j=1
(cxi,j , cyi,j , czi,j )
where the ci,j = are again called control points. Referring to (2.7), the graph of a PowellSabin spline is a particular case of the parametric PowellSabin surface, notably with x = u and y = v. A parametric surface s(u, v) lies within the convex hull of its control points. We can associate a control triangle Ti (ci,1 , ci,2 , ci,3 ) with each vertex Vi in the parameter domain. This triangle is tangent to the surface at s(Vi ). Note that in the parametric setting the choice of the control points ci,j is completely free, whereas for PowellSabin splines only the zcomponent of the control points can be chosen. Figure 6 depicts a torus modelled using a parametric PowellSabin surface with 16 control triangles. Parametric PowellSabin surfaces can be easily manipulated by applying the algorithms for PowellSabin splines separately on the three components in (2.12).
3.
Spline Subdivision
A natural question that comes up in many applications is how to represent a spline function on a refinement ∆1 of the given triangulation ∆0 . A procedure to do that is called a subdivision scheme. Windmolders and Dierckx [35] considered the subdivision problem for uniform PowellSabin splines with a dyadic scheme. Recently, this was used to construct a tangent subdivision scheme for parameter free surfaces [30]. However, the dyadic refinement scheme is generally not applicable for nonuniform PowellSabin splines. Vanraes et al. [33] presented a global triadic subdivision scheme for
Computer Aided Geometric Design with PowellSabin Splines
187
Figure 6. A torus modelled by a parametric PowellSabin surface. The control triangles and the triangular mesh lines are shown.
general PowellSabin splines, which was extended to a local adaptive scheme in [23]. First, we explain the refinement strategy for a given triangulation, and then we determine the corresponding PS control triangles of the new spline representation such that the original spline surface is preserved.
3.1.
Refinement Rules of the Triangulation
General subdivision for PowellSabin splines is based on the socalled scheme [14, 15]. Such a refinement proceeds as follows:
√
3 refinement
1. Split every triangle into three subtriangles by inserting a new vertex Vijk inside the old triangle T (Vi , Vj , Vk ), and connect it to the surrounding old vertices. For example, in the PowellSabin case the new vertex will be located at the interior point Zijk . 2. Flip each edge adjacent to two refined triangles of the original triangulation in order to rebalance the new triangulation. These edges connect now two new vertices instead of two original vertices. These two steps√are illustrated in Figure 7. From the construction it follows that the refined triangulation ∆ 3 does not preserve the original edges except at the boundary. However, if the new vertex√Vijk is chosen as the interior point Zijk of the original PS refinement, the new edges of ∆ 3 still coincide with the edges of that PS refinement. When we apply the √ 3 refinement scheme twice, we obtain a triadic split, as shown in Figure 8. Every original edge is trisected and each original triangle is split into nine subtriangles. √ 3,∗ In order to make subdivision possible, the interior points of the PS refinement ∆ √ √ of the new triangulation ∆ 3 must be chosen such that ∆ 3,∗ contains the edges √of the original PS refinement ∆∗ . As can be seen in Figure 7(c), the new triangles in ∆ 3 are bisected by an edge of√∆∗ . If their interior points are chosen on such an edge, we obtain a √ valid PS refinement ∆ 3,∗ . Figure 9 depicts a 3 refined triangle where the corresponding PS refinement is indicated with dashed lines.
188
Hendrik Speleers, Paul Dierckx and Stefan Vandewalle
(a)
(b)
(c)
√ Figure 7. Principle of 3 refinement. (a) PS refinement of two neighbouring triangles. (b) Place a new vertex at the position of the interior points and connect with the triangle corners. (c) Flip the edge adjacent to the two refined triangles. The dashed lines are part of the PS refinement.
(a)
(b)
Figure 8. Applying the
3.2.
√
(c)
3 refinement scheme twice results in a triadic refinement.
The Construction of Refined Control Triangles
In this section we explain how to derive the Bspline coefficients of the new PowellSabin spline on the (locally) refined triangulation, when the Bspline coefficients were given on the original triangulation. For the vertices Vi in the original triangulation one can reuse the old PS triangles defined by their corner points Qi,m , with m = 1, 2, 3. This is valid because any new PS point in the refined triangulation lies closer to the considered original vertex. However, it is better to determine a smaller PS triangle for improving the stability of the new PowellSabin spline. We will rescale the original PS triangle with an appropriate scalar ωi . The value of ωi can be found by comparing √the positions of the old and new PS points (see [33]). The corners of the new PS triangle ti 3 (see Figure 9) are then given by √
3 Qi,m = ωi Vi + (1 − ωi ) Qi,m ,
m = 1, 2, 3,
Computer Aided Geometric Design with PowellSabin Splines
189
Vk Qi,3
√
3 Qijk,3 √
Qi,33
Vijk √
3 Qijk,1
Qi,1
√
√
3 Qijk,2
Vi
Qi,13
Vj
√
Qi,23
Qi,2 √
√
√
√
√
√
√
√
3 3 3 , Qijk,2 , Qijk,3 ) associated Figure 9. The PS triangles ti 3 (Qi,13 , Qi,23 , Qi,33 ) and tijk3 (Qijk,1 √ with the vertices Vi and Vijk in a 3 refined triangulation. The PS refinement is indicated with dashed lines.
√
3 and the corresponding coefficients ci,m are calculated via the old coefficients ci,m as √
ci,13 = (ωi αi,1 + 1 − ωi ) ci,1 + ωi αi,2 ci,2 + ωi αi,3 ci,3 ,
(3.14a)
ci,23 = ωi αi,1 ci,1 + (ωi αi,2 + 1 − ωi ) ci,2 + ωi αi,3 ci,3 ,
(3.14b)
ci,33 = ωi αi,1 ci,1 + ωi αi,2 ci,2 + (ωi αi,3 + 1 − ωi ) ci,3 .
(3.14c)
√ √
√
The PS triangles tijk3 associated with the new vertices Vijk in the tion are defined by √
3 = (Vijk + Vi )/2, Qijk,1
√
3 Qijk,2 = (Vijk + Vj )/2,
√
3 refined triangula
√
3 and, Qijk,3 = (Vijk + Vk )/2,
as shown in Figure 9. The corresponding coefficients are computed as √
3 ˜ i,1 ci,1 + L ˜ i,2 ci,2 + L ˜ i,3 ci,3 , cijk,1 =L
(3.15a)
3 ˜ j,1 cj,1 + L ˜ j,2 cj,2 + L ˜ j,3 cj,3 , cijk,2 =L
(3.15b)
3 ˜ k,1 ck,1 + L ˜ k,2 ck,2 + L ˜ k,3 ck,3 . cijk,3 =L
(3.15c)
√
√
√
3 Remark that Qijk,1 simply corresponds to the point S˜i in Figure 5(a), and equation (3.15a) ˜ j,1 , L ˜ j,2 , L ˜ j,3 ) and (L ˜ k,1 , L ˜ k,2 , L ˜ k,3 ) can refers to equation (2.11d). Likewise, the triplets (√ L √
3 3 be computed as the barycentric coordinates of Qijk,2 and Qijk,3 , respectively, with respect to the PS triangles of the vertices Vj and Vk . The formulas (3.14)(3.15) use only convex combinations of the old coefficients. As a consequence, this subdivision scheme is numerically stable.
190
Hendrik Speleers, Paul Dierckx and Stefan Vandewalle
(a)
(b)
Figure 10. (a) A PowellSabin spline with 7 control triangles. (b) The equivalent triadically subdivided spline.
Applying those rules twice, one obtains the control triangles of the new vertices in a triadically refined triangulation. Figure 10 illustrates the triadic subdivision scheme for a given PowellSabin spline.
3.3.
Applications
In a broad range of application domains spline subdivision is of interest. In this section we discuss some applications of the PS subdivision scheme. Visualization. A common application of spline subdivision is visualization. After a few subdivision steps, the PS control triangles mimic the shape of the PowellSabin spline quite well. These control triangles can be used to construct a wireframe that approximates the spline surface. The question is how to connect the PS control triangles efficiently into a single wireframe. We consider a few approaches. The wireframe can be constructed by connecting the spline interpolation points s(Vi ), in the same way as the vertices Vi are connected in the domain triangulation. These points are the tangent points of the PS control triangles to the spline surface. This approach was suggested in [35]. Figure 11(a) shows such triangular meshes for the splines in Figure 10. Another strategy is to use the B´ezier control net, see Figure 1(b), of the PowellSabin spline in its BernsteinB´ezier representation [6]. This control net has the advantage that it forms a convex hull for the spline surface. It also converges more rapidly to the surface than the previous wireframe. However, many triangles are needed in this representation, as can be seen in Figure 11(b). A fair compromise is the wireframe in Figure 11(c), constructed by connecting projections of some PS points into the PS control triangles in a particular way, as described below. We first build the mesh in the parameter domain, and then we project the points in this mesh into the corresponding PS control triangles. There are three types of patches in such a mesh, as illustrated in Figure 12. The first type is obtained by constructing for each vertex Vi the smallest envelope polygon that contains all PS points associated with Vi . Note
Computer Aided Geometric Design with PowellSabin Splines
191
(a)
(b)
(c)
Figure 11. Different wireframes for visualizing the PowellSabin splines in Figure 10. The wireframe can be obtained (a) by connecting the spline interpolation points s(Vi ), (b) by using the B´ezier control net, (c) by connecting the projections of certain PS points into the corresponding PS control triangles, as explained in section 3.3..
that the corners of such a polygon are particular PS points. Then, these corner points are connected into triangular and quadrilateral patches in the following way. For each triangle in the domain triangulation ∆ we construct a triangular patch by connecting the three PS points in the interior of the considered triangle. We connect the adjacent PS points along each edge in ∆ to form a quadrilateral patch. The wireframe is then defined by the projections of the corners of these patches into the corresponding PS control triangles. Note that these projections are just particular B´ezier ordinates in the BernsteinB´ezier representation of the PowellSabin spline. For instance, the B´ezier ordinate wi is the projection of the PS point S˜i into PS control triangle Ti , as shown in Figure 5.
192
Hendrik Speleers, Paul Dierckx and Stefan Vandewalle
(a)
(b)
Figure 12. (a) A triangulation with PS refinement and PS points. (b) The mesh used for constructing a wireframe.
As can be seen in the Figures 11(b)(c), the edges of this wireframe coincide with particular edges in the B´ezier control net. The number of used patches is about 1/8 of the number needed for the B´ezier control net. Let n, t and e be the number of vertices, triangles and edges, respectively, in the triangulation ∆. Then, the B´ezier control net needs 24t patches. The wireframe of Figure 11(c) only needs n + t + e patches, which amounts to about 3t using Euler’s formulas. Because of the reduced number of patches, this wireframe is more visually pleasant than the B´ezier control net. Note that at most four patches come together at each vertex in this wireframe. It could be disadvantageous that the patches in this wireframe are constructed of polygons with a different number of edges. Nevertheless, many graphical libraries, as OpenGL, can efficiently handle such set of polygons. Modelling. The basis functions have a smaller support after subdivision. This gives the designer more local control for manipulating the spline surface. By the local nature of the √ 3 subdivision scheme, complex surfaces can be efficiently represented with a reasonable amount of memory [23]. Approximation. Subdivision is useful in data fitting and finite element applications. The original spline space is a subspace of the space obtained after subdivision. Hence, we are guaranteed of a better spline approximation in the refined space when a least squares data fitting strategy is applied [7], or a RitzGalerkin finite element method is used for the numerical solution of partial differential equations [24]. When we use an iterative method to solve the linear systems that arise in these methods, the subdivided version of the optimal spline approximation in the coarser spline space provides a good initial guess for the optimal solution in the finer space. Geometric multigrid methods can be used to solve partial differential equations in a very efficient way. By means of a hierarchy of meshes, one can accelerate the convergence of a basic iterative method. Using the PowellSabin subdivision scheme, a nested sequence of triangulations can be easily created with natural intergrid transfer operators [26].
Computer Aided Geometric Design with PowellSabin Splines
193
Multiresolution. Multiresolution techniques work with different levels of resolution. Wavelets are functions that split data into different frequency components. Each component is studied with a resolution matched to its scale. A common tool to develop wavelets is the lifting scheme, where subdivision can be used as the prediction step. Such wavelets, called PowellSabin spline wavelets, have been developed in [32, 38]. They are particularly suitable for image/surface compression [18].
4.
QHPS Splines
When an increased resolution is only required in a small part of the surface, the use of global subdivision may lead to excessive computational and storage √ costs. In such a case, a local (adaptive) subdivision scheme is recommended. Although the 3 refinement scheme described in section 3.1. can be applied locally, a naive use may introduce poorly shaped triangles at the boundary of the locally refined region. This problem can be dealt with by using a refinement propagation strategy [23]. When a triangle fails to satisfy a certain quality requirement, extra neighbouring triangles are √ refined. This results in an expansion of the refined region. By the edge flipping step in the 3 refinement method, a narrow triangle (with small angles) will be replaced by two better shaped triangles. At the boundary of the domain, artificial vertices outside the domain can be inserted into the triangulation. The proposed strategy is driven by one parameter that manages the tradeoff between the mesh quality and the refinement localization. Such a specialized shapeimprovement strategy could be avoided if nonconforming triangulations were allowed. In this section we review the idea of [25], where PowellSabin splines are adapted towards certain nonconforming triangulations. In particular, we consider hierarchical triangulations which are obtained by partitioning an initial conforming triangulation with a triadic split. A hierarchical triangulation gives rise to a set of nested spline spaces. A hierarchical basis for such a space is constructed in a way that the basis functions of the coarser spaces are retained in the basis of the finer space. In a socalled quasihierarchical basis some of the coarselevel basis functions are replaced by finerlevel basis functions [12]. This section discusses QHPS splines, which are a hierarchical variant of PowellSabin splines in a quasihierarchical basis representation [25].
4.1.
The Hierarchical PowellSabin Spline Space
Consider a simply connected subset Ω ⊂ R2 with polygonal boundary, and assume a conforming triangulation ∆0 of Ω is given. We construct a hierarchical triangulation ∆H on Ω by partitioning successively subsets of triangles with a triadic split, starting from the initial triangulation ∆0 . An example of such a triangulation is drawn in Figure 13(a) with solid lines. Here, ∆0 is the triangulation of Figure 2(a). The hierarchical triangulation has a total of n vertices. Of these vertices, nnc are nonconforming (or hanging) vertices. They are located on interiors of triangle edges. The remaining ones, i.e. nc = n − nnc , are called conforming vertices. In Figure 13(a), ∆0 consists of 8 triangles and 8 (conforming) vertices. The hierarchical triangulation in the figure consists of 16 triangles and 15 vertices (nc = 9, nnc = 6).
194
Hendrik Speleers, Paul Dierckx and Stefan Vandewalle
(a)
(b)
Figure 13. (a) A HPS refinement ∆∗H (in dashed lines) of a given hierarchical triangulation ∆H (in solid lines). (b) The QHPS points (bullets) and a set of suitable QHPS triangles (shaded).
To each hierarchical triangulation ∆H we can associate a hierarchical mesh structure ∆H . It is the set of triangles T k that are generated during the construction of ∆H . The superscript k of a triangle T k refers to the refinement level of that triangle, i.e., the minimal number of triadic refinement steps needed to construct the triangle. The triangles in ∆H that are part of ∆H are called leaf triangles. We will denote ∆lH as the subset of ∆H containing all triangles whose level is l or lower, and let ∆lH be its corresponding hierarchical triangulation. Note that these mesh structures are nested ∆0 ⊂ ∆1H ⊂ ∆2H ⊂ . . . ⊂ ∆H .
(4.16)
We will use PowellSabin triadic splits, as in Figure 8(c), in the construction of a hierarchical triangulation ∆H . The PS refinements needed in the splitting process generate a particular refinement of ∆H which partitions each triangle in ∆H into six subtriangles. This refinement is called the hierarchical PowellSabin (HPS) refinement ∆∗H of ∆H . Analogous to (4.16), the HPS refinement yields a nested structure of sets of triangles 2,∗ ∗ ∆0,∗ ⊂ ∆1,∗ H ⊂ ∆H ⊂ . . . ⊂ ∆H .
(4.17)
In Figure 13(a) such a HPS refinement is drawn in dashed lines. The space of piecewise quadratic polynomials on ∆∗H with global C 1 continuity is called the hierarchical PowellSabin spline space: n o 1 S2,H (∆∗H ) = sH ∈ C 1 (Ω) : sH Tj∗ ∈ Π2 , Tj∗ ∈ ∆∗H . (4.18) For this hierarchical spline space we considered a similar interpolation problem as (2.6) for PowellSabin splines [25] . Given a triplet (fk , fx,k , fy,k ) at each conforming vertex Vk in the hierarchical triangulation ∆H , the interpolation problem sH (Vk ) = fk ,
∂sH (Vk ) = fx,k , ∂x
∂sH (Vk ) = fy,k , ∂y
k = 1, . . . , nc ,
(4.19)
Computer Aided Geometric Design with PowellSabin Splines
195
1 (∆∗ ). It follows that the dimension of the hierarchihas a unique solution sH (x, y) ∈ S2,H H cal PowellSabin spline space is equal to 3nc .
4.2.
A Quasihierarchical PowellSabin Spline Basis
1 (∆∗ ) is very similar to the The construction of a normalized basis for the spline space S2,H H geometric approach for the PowellSabin basis, described in section 2.3.. A hierarchical PowellSabin spline in its quasihierarchical representation is called a QHPS spline, and is denoted as nc X 3 X j sQH (x, y) = ci,j Bi,QH (x, y). (4.20) i=1 j=1
We associate with each conforming vertex Vi in the hierarchical triangulation three linearly j independent triplets (αi,j , βi,j , γi,j ), j = 1, 2, 3. The QHPS Bspline Bi,QH (x, y) can then be determined as the solution of the interpolation problem (4.19) with all (fk , fx,k , fy,k ) = (0, 0, 0) except for k = i, where (fi , fx,i , fy,i ) = (αi,j , βi,j , γi,j ) 6= (0, 0, 0). The triplets (αi,j , βi,j , γi,j ) are computed as follows: 1. For each conforming vertex Vi in the hierarchical triangulation ∆H , identify the corresponding QHPS points. Let k be the smallest level of all triangles in ∆H that contain vertex Vi . Denote ∆kH as the triangulation, consisting of triangles of at most level k, that appears during the construction of ∆H . The QHPS points of Vi are defined as the midpoints of all edges in the HPS refinement ∆k,∗ H ending in Vi . The vertex Vi is also a QHPS point. In Figure 13(b) the QHPS points are indicated as bullets. 2. For each conforming vertex Vi , construct a triangle ti (Qi,1 , Qi,2 , Qi,3 ) containing all the QHPS points of Vi . The triangles ti , i = 1, . . . , nc , are called QHPS triangles. Figure 13(b) shows a possible set of QHPS triangles. 3. The three linearly independent triplets (αi,j , βi,j , γi,j ), j = 1, 2, 3, are derived from the QHPS triangle ti of a vertex Vi as follows: • αi = (αi,1 , αi,2 , αi,3 ) are the barycentric coordinates of Vi with respect to ti , • βi = (βi,1 , βi,2 , βi,3 ) and γi = (γi,1 , γi,2 , γi,3 ) are the coordinates of the unit barycentric directions, in x and ydirection respectively, with respect to ti . Figure 14 illustrates how a QHPS Bspline associated with the central vertex in a given triangulation changes when some of the triangles are triadically refined. The same QHPS triangle is used in the three cases. Note that if the hierarchical triangulation is obtained by global triadic splits, i.e., the final triangulation is conforming, then the QHPS Bsplines are just the classical PowellSabin Bsplines.
4.3.
Properties of the QHPS Bspline Basis
The quasihierarchical Bsplines have similar attractive properties as the classical PowellSabin Bsplines. Unless stated otherwise, the proofs of the properties can be found in [25].
196
Hendrik Speleers, Paul Dierckx and Stefan Vandewalle
(a)
(b)
Figure 14. (a) Several triadically refined hierarchical triangulations. The HPS refinement is indicated with dashed lines. (b) Contour plots of a QHPS basis function associated with the central vertex for the corresponding meshes in (a). The first Bspline is the same as in Figure 3(b).
j Local support. Each QHPS Bspline Bi,QH (x, y) is zero outside the molecule Mik of the corresponding vertex Vi in ∆kH , with k the smallest level of any triangle in ∆H containing Vi .
Convex partition of unity.
The QHPS basis splines are positive, and they sum up to one.
Stability. The quasihierarchical basis is strongly L∞ stable, i.e., the stability constants K1 and K2 in inequality (2.9) are both independent of the number of levels in the hierarchical triangulation. In [27] we investigated the Lp stability of the basis. It turns out that the QHPS basis is, in general, weakly Lp stable, i.e., the constants K1 −1 and K2 have at most a polynomial growth in the number of levels. However, this result can be improved for a broad class of hierarchical triangulations ∆H . Suppose there exists an upper bound on the difference between the level numbers of any two triangles in ∆H that lie inside the support of the same QHPS Bspline. If this bound is independent of the number of refinement levels in ∆H , then the QHPS basis is strongly Lp stable. More details can be found in [27]. Note that the classical PowellSabin Bsplines are always strongly Lp stable on a globally refined hierarchical triangulation.
Computer Aided Geometric Design with PowellSabin Splines
197
QHPS control triangles. We can define control points as ci,j = (Qi,j , ci,j ) and control triangles as Ti (ci,1 , ci,2 , ci,3 ). These triangles are tangent to the spline surface z = s(x, y) at the vertices Vi , and the graph of the QHPS spline lies inside the convex hull of these control points. Subdivision. The nonconformity of the hierarchical triangulation allows a local triadic refinement in a natural way. In addition, a QHPS spline can be locally subdivided on a given set of triangles, using the same triadic PS subdivision rules as explained in section 3.. The locality of the subdivision scheme ensures that a QHPS spline surface can be adaptively manipulated with a reasonable increase of dimension of the space. In Figure 15(a) we subdivided the torus, shown in Figure 6, locally on two triangles, and we adapted the QHPS control triangles of the newly introduced vertices in Figure 15(b). Note that the support of each QHPS Bspline associated with one of these new vertices stays within its original triangle. This is a very interesting property for surface editing. The designer selects a triangle where the surface must be locally subdivided. Then, he/she can freely morph the surface while only the part within the selected triangle will be affected. The new QHPS surface in Figure 15(b) only consists of 18 control triangles, whereas a globally subdivided PowellSabin spline would need 144 control triangles to represent the same surface.
4.4.
A Practical Implementation
In this section we show how the efficient algorithms for classical PowellSabin splines can be used for working with quasihierarchical PowellSabin splines. A QHPS spline can be represented on each leaf triangle by a particular PowellSabin spline. To construct this PowellSabin spline, we have to determine the PS control triangles corresponding to the three corner vertices of the considered triangle. The PS control triangles associated with conforming vertices can be taken identical to the QHPS control triangles of the QHPS spline. We can use subdivision to compute the control triangles for the nonconforming vertices. The correct PS control triangles can be recursively obtained while moving through the hierarchical mesh structure. When the leaf triangle is reached, the corresponding PowellSabin spline on the considered triangle will then be fully defined. In this way, an algorithm for QHPS splines can be straightforwardly reduced to the equivalent algorithm for PowellSabin splines. We now present a generic QHPS algorithm [25]. It only needs an equivalent algorithm for classical PowellSabin splines. Generic QHPS algorithm. Let ∆H be a hierarchical mesh structure, and ∆ be a conforming triangulation. The generic algorithm qhps algorithm(∆H ) for QHPS splines is based on the equivalent algorithm ps algorithm(∆) for classical PowellSabin splines. We suppose that each triangle in ∆H and ∆ has access to the control triangles associated with its three corner vertices. For conforming vertices, their control triangles are already known in advance, i.e., the QHPS control triangles. For nonconforming vertices, their control triangles are computed during the algorithm. function qhps algorithm(hierarchical structure ∆H )
198
Hendrik Speleers, Paul Dierckx and Stefan Vandewalle for all triangles T in the initial triangulation ∆0 of ∆H : qhps local algorithm(T , ∆H ) endfor end function qhps local algorithm(triangle T , hierarchical structure ∆H ) if T is not a leaf triangle: 1. Let l be the level of T in the hierarchical structure. 2. for all nonconforming vertices Vi of level (l + 1) situated at the interior of an edge of T : Calculate the control triangle Ti by triadic PS subdivision, using the control triangles of two corner vertices of T . endfor 3. for all 9 subtriangles Ti of level (l + 1) in T : qhps local algorithm(Ti , ∆H ) endfor else ps algorithm(T ) endif end
The algorithm requires that all triangles in the hierarchical mesh structure ∆H are stored in memory. It is recommended to manage these triangles in the following way. Use an array that contains the triangles in the initial triangulation ∆0 . A triangle that is refined keeps references to each of its nine subtriangles. That ensures that all triangles in ∆H are reachable. In order to navigate easily through the mesh structure it is also advisable that the triangles keep the references to their (at most three) neighbouring triangles. Using a different PowellSabin spline on each leaf triangle in the hierarchical triangulation is usually not more timeconsuming than considering a single PowellSabin spline on a larger set of triangles. Indeed, most of the algorithms for PowellSabin splines run over all triangles in the triangulation separately. The QHPS splines can be evaluated and manipulated in a stable way. Only convex combinations are needed to convert the QHPS spline on each triangle to a PowellSabin spline, to represent these PowellSabin splines with BernsteinB´ezier polynomials, and to evaluate these polynomials via the de Casteljau algorithm.
5.
NURPS Surfaces
Rational surfaces, such as rational B´ezier and NURBS surfaces, are commonly used tools in commercial computer aided design and computer graphic packages. Rational surface representations give a designer extra degrees of freedom compared to their nonrational counterparts through the weights that are associated with the control points. These rational
Computer Aided Geometric Design with PowellSabin Splines
199
(a)
(b)
Figure 15. (a) Local QHPS subdivision applied on two triangles of the surface in Figure 6. The control triangles and the triangular mesh lines are shown. (b) Effect of moving the two new control triangles of the QHPS surface.
surfaces are able to exactly represent patches on quadric surfaces, e.g., patches on the cone and the sphere. In this section we discuss the rational extension of PowellSabin splines, called NURPS surfaces. For more details we refer to the papers [29, 36, 37].
5.1.
Rational PowellSabin Surfaces
Consider a conforming triangulation ∆ on a given domain Ω ⊂ R2 , with PS refinement ∆∗ . A nonuniform rational PowellSabin (NURPS) surface s(u, v) is defined as
s(u, v) =
n X 3 X i=1 j=1
ci,j φji (u, v),
(u, v) ∈ Ω,
(5.21)
200
Hendrik Speleers, Paul Dierckx and Stefan Vandewalle
with ci,j = (cxi,j , cyi,j , czi,j ) the NURPS control points, and the blending functions wi,j B j (u, v) φji (u, v) = Pn P3 i . j i=1 j=1 wi,j Bi (u, v)
(5.22)
Here, Bij (u, v) are the normalized PowellSabin Bsplines on ∆∗ , and wi,j are positive weights. When all weights are constant, then (5.21) reduces to (2.13). The blending functions φji (u, v) in (5.22) have a local support and they form a convex partition of unity. A NURPS surface in representation (5.21) can be seen as the 3D projection onto the Euclidian space of a 4D PowellSabin spline in the homogeneous space, i.e.,
s(u, v) =
n X 3 X
chi,j Bij (u, v),
(5.23)
i=1 j=1
where the 4D homogeneous control points are given by hy hz hw y x z chi,j = (chx i,j , ci,j , ci,j , ci,j ) = (wi,j ci,j , wi,j ci,j , wi,j ci,j , wi,j ).
(5.24)
The next sections are devoted to some particular features of the NURPS surfaces. First, we discuss the use of the control points and the weights in shape modelling with NURPS surfaces. Then, we describe a subdivision scheme. We end with the NURPS representation of some quadric surfaces.
5.2.
Modelling with NURPS Surfaces
A designer has two types of freedom in the construction of a NURPS surface: the coefficients ci,j and the weights wi,j . They can be determined in a geometrically intuitive way by means of control triangles and weight points.
Control triangles. Similar to the PowellSabin splines, the influence of the coefficients in (5.21) on the NURPS surface can be intuitively interpreted via control triangles [36]. With each vertex Vi in the triangulation ∆, one can define the control triangle Ti (ci,1 , ci,2 , ci,3 ). This triangle is tangent to the NURPS surface at the point
where
s(Vi ) = α ˆ i,1 ci,1 + α ˆ i,2 ci,2 + α ˆ i,3 ci,3 ,
(5.25)
wi,j αi,j α ˆ i,j = P3 . k=1 wi,k αi,k
(5.26)
Here, (αi,1 , αi,2 , αi,3 ) are the barycentric coordinates of vertex Vi with respect to the PS triangle ti .
Computer Aided Geometric Design with PowellSabin Splines
201
Weight points. In [29] it is described how one can use socalled weight points as a design tool for handling the weights. Such a point is characterized by its position pi and by a scaling factor Ki . The position is chosen as the tangent point of the control triangle Ti to the NURPS surface, i.e., pi = s(Vi ). The three weights wi,j , with j = 1, 2, 3, are then uniquely defined by means of the barycentric coordinates (α ˆ i,1 , α ˆ i,2 , α ˆ i,3 ) of pi with respect to Ti , up to a positive factor Ki , via the formula α ˆ i,j wi,j = Ki . (5.27) αi,j A designer can freely move the weight point within the control triangle Ti . Since its position is the tangent point of Ti to the NURPS surface, the effect of the movement will be intuitive to the designer. This is illustrated in the first two pictures of Figure 16. One can use the scaling factor Ki to determine the relative importance of the considered three weights with respect to the other weights. Indeed, Ki can be interpreted as a weighted mean of these weights, i.e., Ki = αi,1 wi,1 + αi,2 wi,2 + αi,3 wi,3 . (5.28) The larger the value of Ki , the more the NURPS surface will be attracted to the control triangle. A cusp can be simulated by reducing the scaling factor. The effect of changing the scaling factor is illustrated in the bottom two pictures of Figure 16. Similar effects can be obtained by rescaling (enlarging/reducing) the control triangles [37]. However, changing the factor Ki has the advantage that the designer can continue to work with the same control triangles. Let ri,j be the intersection point of the line pi − ci,j and the edge of control triangle Ti opposite to ci,j , then α ˆ i,j ′ ci,j ′ + α ˆ i,j ′′ ci,j ′′ ri,j = , (5.29) α ˆ i,j ′ + α ˆ i,j ′′ with j ′ = 1 + (j mod 3) and j ′′ = 1 + (j ′ mod 3). Using these points, the ratio of two weights can be geometrically interpreted as the ratio of two lengths, i.e., wi,j ′′ αi,j ′ kri,j − ci,j ′ k = , wi,j ′ αi,j ′′ kri,j − ci,j ′′ k
(5.30)
and, using formula (5.27), as a ratio of two triangular areas, i.e., wi,j ′′ αi,j ′ A(pi , ci,j , ci,j ′ ) = . wi,j ′ αi,j ′′ A(pi , ci,j ′′ , ci,j )
5.3.
(5.31)
NURPS Subdivision
We now adapt the subdivision scheme for PowellSabin splines towards NURPS surfaces. √ 3 The construction of the refined triangulation ∆ and the choice of the PS triangles remain identical as described in section 3.. We can compute the control points of the subdivided NURPS surface via its homo√ h, 3 geneous representation (5.23). For instance, the homogeneous control points cijk,m , with
202
Hendrik Speleers, Paul Dierckx and Stefan Vandewalle
(a)
(b)
(c)
(d)
Figure 16. (a) Original NURPS surface. (b) The effect of moving the weight point pi inside the control triangle. (c)(d) Enlarging and reducing the scaling factor Ki . The considered weight point and control points are indicated with bullets. √
m = 1, 2, 3, corresponding to the PS triangle tijk3 , are then calculated as √
3 h h h ˜ ˜ ˜ ch, ijk,1 = Li,1 ci,1 + Li,2 ci,2 + Li,3 ci,3 , √ h, 3 cijk,2 √ 3 ch, ijk,3
(5.32a)
˜ j,1 chj,1 + L ˜ j,2 chj,2 + L ˜ j,3 chj,3 , =L
(5.32b)
˜ k,1 ch + L ˜ k,2 ch + L ˜ k,3 ch . =L k,1 k,2 k,3
(5.32c)
The convex combinations are identical to the ones in formulas (3.15). Projecting the new control points (5.32) back to the Euclidian space yields √ √ √ hx, 3 hy, 3 hz, 3 √ c c c ijk,m ijk,m ijk,m 3 √ , √ , √ cijk,m = . (5.33) hw, 3 hw, 3 hw, 3 cijk,m cijk,m cijk,m It is well known that working in the homogeneous space can lead to numerical instabilities. √ hr, 3 When the weights vary greatly in magnitude, the coordinates cijk,m , with r = x, y, z, can become extremely large. Then, the calculations do not operate in the convex hull of the rational control points anymore, and numerical stability is endangered. Inspired by the idea behind the rational variant of the de Casteljau algorithm from Farin [8], we can improve the numerical stability by rearranging the calculations in or
Computer Aided Geometric Design with PowellSabin Splines
203
Figure 17. The domain triangle of a NURPS patch, together with its PS refinement and PS triangles.
√
√
3 3 and its weight wijk,1 der to avoid working in the homogeneous space. For instance, cijk,1 are computed in a stable way as follows: √
√
hw, 3 3 ˜ i,1 wi,1 + L ˜ i,2 wi,2 + L ˜ i,3 wi,3 , wijk,1 = cijk,1 =L ˜ ˆ i,m = wi,m√Li,m , L m = 1, 2, 3, 3 wijk,1
(5.34a) (5.34b)
√
3 ˆ i,1 ci,1 + L ˆ i,2 ci,2 + L ˆ i,3 ci,3 . cijk,1 =L
(5.34c)
Analogously, one can calculate the other control points in a numerically stable way.
5.4.
Quadrics as NURPS Surfaces
In [37] closed formulas are derived for the control points of NURPS patches on a cylinder, a cone and a sphere. In this section we give another representation of such patches, resulting in a simpler choice of the control points. Further on, each triangular NURPS patch is defined on the equilateral domain triangle T (V1 , V2 , V3 ) depicted in Figure 17, with the given PS refinement and PS triangles, defined by the points Q1,1 = V1 ,
Q1,2 = (V1 + V2 )/2,
Q1,3 = (V1 + V3 )/2,
Q2,1 = V2 ,
Q2,2 = (V2 + V3 )/2,
Q2,3 = (V2 + V1 )/2,
Q3,1 = V3 ,
Q3,2 = (V3 + V1 )/2,
Q3,3 = (V3 + V2 )/2.
The derivation of the corresponding control points is analogous to the one described in [37]. Cylinder. The cylinder x2 + y 2 = r2 ,
0 ≤ z ≤ h,
(5.35)
can be split into eight isometrical triangular segments. The control points and weights of the NURPS representation of such a patch is given in the following table.
204
Hendrik Speleers, Paul Dierckx and Stefan Vandewalle
i 1 2 3
ci,1 (r, 0, 0) (0, r, 0) (r, 0, h)
wi,1 1 1 1
ci,2 (r, r, 0) (r, r, h2 ) (r, 0, h2 )
w√i,2 2 √2 2 2
1
ci,3 (r, 0, h2 ) (r, r, 0) (r, r, h2 )
wi,3 1 √ 2 √2 2 2
The entire cylinder, constructed with eight NURPS patches, is shown in Figure 18(a). Cone. The cone
x2 + y 2 =
h−z r h
2
,
0 ≤ z ≤ h,
(5.36)
can be split into four isometrical patches. The NURPS representation of such a patch is given in the following table. i 1 2 3
ci,1 (r, 0, 0) (0, r, 0) (0, 0, h)
wi,1 1 1 1
ci,2 (r, r, 0) (0, 0, h) (0, 0, h)
w√i,2 2 2
1 1
ci,3 (0, 0, h) (r, r, 0) (0, 0, h)
wi,3 1 √ 2 2
1
The complete cone is depicted in Figure 18(b). Sphere.
Using only NURPS patches, it is not possible to describe the entire sphere, x2 + y 2 + z 2 = 1.
(5.37)
Nevertheless, with 2n NURPS patches we can represent the sphere up to some small gaps. The maximal height of these gaps is equal to hg =
2 sin2 θ , cos2 θ + 1
(5.38)
where θ = π/n. Small gaps can be achieved by a small angle θ, but then a large number of patches has to be used. The control points of such a patch are shown in the table below. i 1 2 3
ci,1 (cos θ, − sin θ, 0) (cos θ, sin θ, 0) (0, 0, 1)
wi,1 1 1 1
ci,2 1 2 cos θ , 0, tan θ (cos θ, sin θ, 1) (cos θ, − sin θ, 1)
wi,2 2θ cos √ 2 √2 2 2
ci,3 (cos θ, − sin θ, 1) 1 2 cos θ , 0, tan θ (cos θ, sin θ, 1)
w√i,3
2 2 2θ cos √ 2 2
An extension to a sphere with radius r is straightforward, i.e., by multiplying the coordinates of each control point with r. An incomplete NURPS representation of the sphere with twelve patches is given in Figure 18(c). The hole filling strategy in [34] can be used to close the gaps in the sphere approximately.
Computer Aided Geometric Design with PowellSabin Splines
(a)
(b)
205
(c)
Figure 18. A NURPS representation of (a) a cylinder, (b) a cone, and (c) a sphere. The patches are defined on the domain triangle in Figure 17. Neighbouring patches are shaded in different colours.
6.
Conclusion
PowellSabin splines are C 1 continuous piecewise quadratic polynomials defined on an arbitrary conforming triangulation. These splines can be compactly represented in a stable normalized basis. The basis functions can be chosen in a flexible way by means of PS triangles. This leads to a natural definition of PS control triangles, that allow an interactive change of the shape of PowellSabin splines in a predictable way. √ PowellSabin splines can be refined using the 3 subdivision scheme. The control points of the subdivided spline can be easily calculated in a stable way. Applying the scheme twice results in a triadic refinement. Subdivision has many applications. The subdivision scheme can be applied for an efficient visualization of the PowellSabin splines. The increased resolution can also be used for obtaining a more detailed approximation or for a local manipulation of the spline shape. QHPS splines are a hierarchical variant of PowellSabin splines in a quasihierarchical basis representation. They are defined on a hierarchical triangulation. Such a mesh is obtained, starting from an initial conforming triangulation, by partitioning successively a subset of triangles with a triadic split. The QHPS basis retains all advantages of the PowellSabin Bsplines. In addition, a local refinement of the QHPS spline can be performed in a very natural way. The rational extension of a PowellSabin spline surface is called a NURPS surface. In the rational representation a weight is associated with each control point. These weights can be used as extra degrees of freedom in the modelling of shapes by means of weight points. The position of the tangent points within the control triangles leads to an intuitive and graphical interpretation of the weights. NURPS surfaces are able to exactly represent patches of quadric surfaces, as the cylinder, the cone and the sphere. Note that the quasihierarchical setting is also applicable to the NURPS surfaces. A generalization of PowellSabin splines to higher degrees and higher dimensions is not a trivial task. Some higher degree spline extensions on the PowellSabin split are con
206
Hendrik Speleers, Paul Dierckx and Stefan Vandewalle
sidered in [2, 17]. Unfortunately, the proposed Bsplines have no immediate geometric interpretation anymore, similar to PS triangles, and they are more difficult to implement. A generalization of the PowellSabin split in more dimensions is discussed in [22, 39]. Each simplex in a tessellation has to be split into a lot of smaller simplices. They must satisfy a particular set of geometric constraints in order to achieve C 1 continuity. At the moment it is not clear whether these constraints can always be satisfied for an arbitrary tessellation. Nevertheless, we can conclude that PowellSabin splines exhibit many nice properties that make them suitable for many CAGD applications.
References [1] P. Alfeld and L.L. Schumaker. The dimension of bivariate spline spaces of smoothness r for degree d ≥ 4r + 1. Constr. Approx., 3:189–197, 1987. [2] P. Alfeld and L.L. Schumaker. Smooth macroelements based on PowellSabin triangle splits. Adv. Comp. Math., 16:29–46, 2002. [3] L.J. Billera. Homology of smooth splines: generic triangulations and a conjecture of Strang. Trans. Amer. Math. Soc., 310:325–340, 1988. [4] W. Boehm and A. M¨uller. On de Casteljau’s algorithm. Comput. Aided Geom. Design, 16:587–605, 1999. [5] R.W. Clough and J.L. Tocher. Finite element stiffness matrices for analysis of plates in bending. In Proc. 1st Conference on Matrix Methods in Structural Mechanics, pages 515–545, Wright Patterson Air Force Base, Ohio, 1965. [6] P. Dierckx. On calculating normalized PowellSabin Bsplines. Comput. Aided Geom. Design, 15(1):61–78, 1997. [7] P. Dierckx, S. Van Leemput, and T. Vermeire. Algorithms for surface fitting using PowellSabin splines. IMA J. Numer. Anal., 12:271–299, 1992. [8] G. Farin. Algorithms for rational B´ezier curves. Comput. Aided Design, 15:73–77, 1983. [9] G. Farin. Triangular BernsteinB´ezier patches. Comput. Aided Geom. Design, 3:83– 127, 1986. [10] G. Farin. Curves and Surfaces for CAGD: A Practical Guide. Morgan Kaufmann Publishers, San Francisco, fifth edition, 2002. [11] G. Farin. Dimensions of spline spaces over unconstricted triangulations. J. Comput. Appl. Math., 192:320–327, 2006. [12] E. Grinspun, P. Krysl, and P. Schr¨oder. CHARMS: a simple framework for adaptive simulation. ACM Trans. Graphics, 21(3):281–290, 2002.
Computer Aided Geometric Design with PowellSabin Splines
207
[13] D. Hong. Spaces of bivariate spline functions over triangulation. Approx. Theory Appl., 7:56–75, 1991. √ [14] L. Kobbelt. 3Subdivision. In Computer Graphics Proceedings, Annual Conference Series, pages 103–112. ACM SIGGRAPH, 2000. √ [15] U. Labsik and G. Greiner. Interpolatory 3subdivision. In Proc. 21th European Conference on Computer Graphics, volume 19 of Computer Graphics Forum, pages 131–138, Cambridge, 2000. [16] M.J. Lai and L.L. Schumaker. On the approximation power of bivariate splines. Adv. Comp. Math., 9:251–279, 1998. [17] M.J. Lai and L.L. Schumaker. Macroelements and stable local bases for spaces of splines on PowellSabin triangulations. Math. Comp., 72:335–354, 2003. [18] J. Maes and A. Bultheel. Stable multiresolution analysis on triangles for surface compression. Electr. Trans. Numer. Anal., 25:224–258, 2006. [19] J. Maes, E. Vanraes, P. Dierckx, and A. Bultheel. On the Stability of normalized PowellSabin Bsplines. J. Comput. Appl. Math., 170(1):181–196, 2004. [20] C. Manni and P. Sablonni`ere. Quadratic spline quasiinterpolants on PowellSabin partitions. Adv. Comput. Math., 26:283–304, 2007. [21] M.J.D. Powell and M.A. Sabin. Piecewise quadratic approximations on triangles. ACM Trans. Math. Softw., 3:316–325, 1977. [22] T. Sorokina and A.J. Worsey. A multivariate PowellSabin interpolant. Adv. Comput. Math., in press, 2007. [23] H. Speleers, P. Dierckx, and S. Vandewalle. Local subdivision of PowellSabin splines. Comput. Aided Geom. Design, 23(5):446–462, 2006. [24] H. Speleers, P. Dierckx, and S. Vandewalle. Numerical solution of partial differential equations with PowellSabin splines. J. Comput. Appl. Math., 189(12):643–659, 2006. [25] H. Speleers, P. Dierckx, and S. Vandewalle. Quasihierarchical PowellSabin Bsplines. Technical Report 472, Dept. Computer Science, K.U. Leuven, 2006. [26] H. Speleers, P. Dierckx, and S. Vandewalle. Multigrid methods with PowellSabin splines. Technical Report 488, Dept. Computer Science, K.U. Leuven, 2007. [27] H. Speleers, P. Dierckx, and S. Vandewalle. On the Lp stability of quasihierarchical PowellSabin splines. Technical Report 492, Dept. Computer Science, K.U. Leuven, 2007. [28] H. Speleers, P. Dierckx, and S. Vandewalle. PowellSabin splines with boundary conditions for polygonal and nonpolygonal domains. J. Comput. Appl. Math., in press, 2007.
208
Hendrik Speleers, Paul Dierckx and Stefan Vandewalle
[29] H. Speleers, P. Dierckx, and S. Vandewalle. Weight control for modelling with NURPS surfaces. Comput. Aided Geom. Design, 24(3):179–186, 2007. [30] E. Vanraes and A. Bultheel. A tangent subdivision scheme. ACM Trans. Graphics, 25:340–355, 2006. [31] E. Vanraes, P. Dierckx, and A. Bultheel. On the choice of the PStriangles. Technical Report 353, Dept. Computer Science, K.U. Leuven, 2003. [32] E. Vanraes, J. Maes, and A. Bultheel. PowellSabin spline wavelets. Int. J. Wav. Multires. Inf. Proc., 2(1):23–42, 2004. [33] E. Vanraes, J. Windmolders, A. Bultheel, and P. Dierckx. Automatic construction of control triangles for subdivided PowelSabin splines. Comput. Aided Geom. Design, 21(7):671–682, 2004. [34] J. Windmolders. PowellSabin splines for computer aided geometric design. PhD thesis, Dept. Computer Science, K.U. Leuven, 2003. [35] J. Windmolders and P. Dierckx. Subdivision of uniform PowellSabin splines. Comput. Aided Geom. Design, 16:301–315, 1999. [36] J. Windmolders and P. Dierckx. From PSsplines to NURPS. In A. Cohen, C. Rabut, and L.L. Schumaker, editors, Proc. of Curve and Surface Fitting, SaintMalo 1999, pages 45–54. Vanderbilt University Press, 2000. [37] J. Windmolders and P. Dierckx. NURPS for special effects and quadrics. In T. Lyche and L.L. Schumaker, editors, Proc. of Mathematical Methods for Curves and Surfaces, Oslo 2000, pages 527 – 534. Vanderbilt University Press, 2001. [38] J. Windmolders, E. Vanraes, P. Dierckx, and A. Bultheel. Uniform PowellSabin spline wavelets. J. Comput. Appl. Math., 154(1):125–142, 2003. [39] A.J. Worsey and B. Piper. A trivariate PowellSabin interpolant. Comput. Aided Geom. Design, 5:177–186, 1988.
In: Computer Animation Editors: J.S. Wright and L.M. Hughes, pp. 209234
ISBN: 9781607415596 © 2010 Nova Science Publishers, Inc.
Chapter 9
AN ONTOLOGY OF COMPUTERAIDED DESIGN Udo Kannengiesser NICTA, Australia
John S. Gero Krasnow Institute for Advanced Study and Volgenau School of Information Technology and Engineering, George Mason University, USA, and University of Technology, Sydney, Australia
Abstract This chapter develops an ontology of computeraided design, based on the functionbehaviourstructure (FBS) ontology. It proposes two complementary views of the process of design. The objectcentred view applies the FBS ontology to the artefact being designed. Integrating an ontology of three “design worlds”, this view establishes a framework of designing as a set of transformations between the function, behaviour and structure of the design object, driven by interactions between the three design worlds. Building on this framework, the processcentred view applies the FBS ontology to the activities defined by the objectcentred view. This increases the level of detail and provides a more welldefined set of representations of these activities. Our ontological framework can be used to provide a better understanding of the functionalities required of existing and future computeraided design support.
1. Introduction The notion of computeraided design can be understood as an umbrella term for approaches to using computational tools to support human design activities. Its principal innovations to date include tools for computeraided drafting (CAD), engineering (CAE) and manufacturing (CAM), which have been recognised as a significant technological achievement of the past century (Weisberg 2000). These tools are now indispensable for practitioners in many design domains.
210
Udo Kannengiesser and John S. Gero
Some research has focused on expanding computer support to activities carried out in the early, conceptual stages of design. However, its impact on design practices and tool development in industry has generally been rather low. We believe that one of the reasons is that many of these approaches are based on an insufficient understanding of design. This concerns activities that are carried out by human designers, which include producing and reinterpreting drawings or sketches, and reflecting on current and previous design tasks. It has been shown that these activities are important drivers of designing (Schön and Wiggins 1992; Suwa et al. 1999; Suwa and Tversky 2002). Most traditional models of design are inadequate because they do not explicitly account for these findings. Progress on a more comprehensive understanding of designing has been made only recently. Our situated functionbehaviourstructure (FBS) framework (Gero and Kannengiesser 2004) represents designing as a situated act that is driven by the interactions between the designer and their environment. It uses a perspective that is oriented to the object being designed, so that designing can be shown as a set of transformations between the function, behaviour and structure of the artefact. The situated FBS framework has shown its potential to enhance human understanding of designing. This chapter develops extensions to this framework that provide a more detailed ontological basis on which computeraided design support can be built. Section 2 presents the situated FBS framework and shows how it derives from an objectcentred view of designing driven by the interactions between three “design worlds”. Section 3, adopting a processcentred view of designing, applies the FBS ontology to the design activities defined in Section 2. This adds a significant amount of detail and rigour to the representation of each activity. Section 4 uses this view to derive a framework that specifies the functions required of computational tools to support designing. Section 5 concludes the chapter.
2. An ObjectCentred Ontology of Design 2.1. An Ontology of Design Objects Most design models and design ontologies focus on the artefact or object of design. The FBS ontology distinguishes between three aspects of a design object (Gero 1990; Gero and Kannengiesser 2004): function (F), behaviour (B) and structure (S).
2.1.1. Object Function Function (F) of an object is defined as its teleology (“what the object is for”). For example, some of the functions of a window include “to provide view”, “to provide daylight” and “to provide rain protection”. Function represents the usefulness of the object for another system.
2.1.2. Object Behaviour Behaviour (B) of an object is defined as the attributes that can be derived from its structure (“what the object does”). Using the window example, behaviours include “thermal
An Ontology of ComputerAided Design
211
conduction”, “light transmission” and “direct solar gain”. Behaviour provides operational, measurable performance criteria for comparing different objects.
2.1.3. Object Structure Structure (S) of an object is defined as its components and their relationships (“what the object consists of”). The structure of physical objects includes their form (i.e., geometry and topology) and material. More generally, form can be viewed as a description of an object’s macrostructure, and material can be viewed as a shorthand description of the microstructure. In the window example, macrostructure (form) includes “glazing length” and “glazing height”, and microstructure (material) includes “type of glass”.
2.1.4. Relationships between Object Function, Behaviour and Structure Humans construct relationships between function, behaviour and structure through experience and through the development of causal models based on interactions with the object. Specifically, function is ascribed to behaviour by establishing a teleological connection between the human’s goals and observable or measurable effects of the object. There is no direct relationship between function and structure. Behaviour is causally related to structure, i.e. it can be derived from structure using physical laws or heuristics. This may require knowledge about external effects (exogenous variables) and their interaction with the artefact’s structure. In the window example, deriving the behaviour “light transmission” requires considering external light sources.
2.2. An Ontology of Design Worlds An aspect that has been ignored in most models of design relates to the interactions of the designer and their environment. Designers perform actions in order to change their environment. By observing and interpreting the results of their actions, they then decide on new actions to be executed on the environment. The designers’ concepts may change according to what they are “seeing”, which itself is a function of what they have done. One may speak of an “interaction of making and seeing” (Schön and Wiggins 1992). This interaction between the designer and the environment strongly determines the course of designing. This idea is called situatedness, whose foundational concepts go back to the work of Dewey (1896) and Bartlett (1932). Gero and Kannengiesser (2004) have modelled situatedness by specifying three interacting worlds: the external world, interpreted world and expected world, Figure 1(a).
2.2.1. The External World The external world is the world that is composed of representations outside the designer or design agent. The notion of “external” is meant in a conceptual sense rather than a physical one. It denotes an environment that contains design artefacts made available for interpretation.
212
Udo Kannengiesser and John S. Gero
Figure 1. Situatedness as the interaction of three worlds: (a) general model, (b) specialised model for design representations
2.2.2. The Interpreted World The interpreted world is the world that is built up inside the design agent in terms of sensory experiences, percepts and concepts. It is the internal representation of that part of the external world that the design agent interacts with. The interpreted world provides an environment for analytic activities and discovery during designing.
2.2.3. The Expected World The expected world is the world imagined actions of the design agent will produce. It is the environment in which the effects of actions are predicted according to current goals and interpretations of the state of the world.
2.2.4. Relationships between the Three Worlds These three worlds are related through three classes of interaction. Interpretation transforms variables that are sensed in the external world into sensory experiences, percepts and concepts that compose the interpreted world. Focussing takes some aspects of the interpreted world and uses them as goals for the expected world. Action is an effect which brings about a change in the external world according to the goals in the expected world.
An Ontology of ComputerAided Design
213
2.2.5. A More Detailed Framework of Design Interactions Figure 1(b) presents a specialised view of the ontology of design worlds, with the design agent (described by the interpreted and expected world) located within the external world, and with general classes of design representations placed into this nested model. The set of expected design representations (Xei) corresponds to the notion of a design state space, i.e. the state space of all possible designs that satisfy the set of requirements. This state space can be modified during the process of designing by transferring new interpreted design representations (Xi) into the expected world and/or transferring some of the expected design representations (Xei) out of the expected world. This leads to changes in external design representations (Xe), which may then be used as a basis for reinterpretation changing the interpreted world. Novel interpreted design representations (Xi) may also be the result of memory (here called constructive memory), which can be viewed as a process of interaction among design representations within the interpreted world rather than across the interpreted and the external world. Both interpretation and constructive memory are viewed as “pushpull” processes, i.e. the results of these processes are driven both by the original experience (“push”) and by some of the agent’s current interpretations and expectations (“pull”) (Gero and Fujii 2000). This notion captures two ideas. First, interpretation and constructive memory have a subjective nature, using firstperson knowledge grounded in the designer’s interactions with their environment (Bickhard and Campbell 1996; Clancey 1997; Ziemke 1999; Smith and Gero 2005). This is in contrast to static approaches that attempt to encode all relevant design knowledge prior to its use. Anecdotal evidence in support of firstperson knowledge is provided by the common observation that different designers perceive the same set of requirements differently (and thus produce different designs). And the same designer is likely to produce different designs at later times for the same requirements. This is a result of the designer acquiring new knowledge while interacting with their environment between the two times. Second, the interplay between “push” and “pull” has the potential to produce emergent effects, leading to novel and often surprising interpretations of the same internal or external representation. This idea extends the notion of biases that simply reproduce the agent’s current expectations. Examples have been provided from experimental studies of designers interacting with their sketches of the design object. Schön and Wiggins (1992) found that designers use their sketches not only as an external memory, but also as a means to reinterpret what they have drawn, thus leading the design in a surprising, new direction. Suwa et al. (1999) noted, in studying designers, a correlation of unexpected discoveries in sketches with the invention of new issues or requirements during the design process. They concluded that “sketches serve as a physical setting in which design thoughts are constructed on the fly in a situated way”. Guindon’s (1990) protocol analyses of software engineers, designing control software for a lift, revealed that designing is characterised by frequent discoveries of new requirements interleaved with the development of new partial design solutions. As Guindon puts it, “designers try to make the most effective use of newly inferred requirements, or the sudden discovery of partial solutions, and modify their goals and plans accordingly”.
214
Udo Kannengiesser and John S. Gero
2.3. The Situated FunctionBehaviourStructure Framework Gero and Kannengiesser (2004) have combined the ontology of design artefacts (Section 2.1) with the ontology of design worlds (Section 2.2), by specialising the model of situatedness shown in Figure 1(b). In particular, the variable X, which stands for design representations in general, is replaced with the more specific representations F, B and S. This provides the basis of the situated FBS framework, Figure 2 (Gero and Kannengiesser 2004). In addition to using external, interpreted and expected F, B and S, this framework uses explicit representations of external requirements given to the designer by another agent (usually the customer). Specifically, there may be external requirements on function (FRe), external requirements on behaviour (BRe), and external requirements on structure (SRe). The situated FBS framework also introduces the process of comparison between interpreted behaviour (Bi) and expected behaviour (Bei), and a number of processes that transform interpreted structure (Si) into interpreted behaviour (Bi), interpreted behaviour (Bi) into interpreted function (Fi), expected function (Fei) into expected behaviour (Bei), and expected behaviour (Bei) into expected structure (Sei). Figure 2 uses the numerals 1 to 20 to label the resultant set of processes; however, it should be noted that they do not represent any order of execution. The 20 processes can be mapped onto eight fundamental design steps (Gero 1990; Gero and Kannengiesser 2004). 1. Formulation: consists of processes 1 – 10. It includes interpretation of external requirements, given to the designer by a customer, as function, behaviour and structure, via processes 1, 2 and 3. Requirements are also constructed as implicit requirements generated from within the designer, using constructive memory (processes 4, 5 and 6). Focussing transfers a subset of the (explicitly and implicitly) required function, behaviour and structure into the expected world (processes 7, 8 and 9). In summary, processes 1 – 9 represent activities that populate the interpreted and expected worlds with design concepts, providing the basis for subsequent transformations of these concepts. Process 10 transforms expected function into additional expected behaviour. The set of expected function, behaviour and structure, resulting from the formulation step, represents the design state space. It includes all the variables and their ranges of values that are relevant for the design task. 2. Synthesis: consists of process 11 to generate an instance of structure that is expected to meet the required behaviour, and the externalisation of that structure via process 12. This design step can be viewed as part of a search process through the (previously formulated) state space of all possible instances of structure. 3. Analysis: consists of interpretation of externalised structure (process 13) and the derivation of behaviour from that structure (process 14). 4. Evaluation: consists of a comparison of expected behaviour and behaviour derived through analysis (process 15). 5. Documentation: produces an external representation of the final design solution for purposes of communicating that solution in terms of structure (process 12), and, optionally, behaviour (process 17) and function (process 18).
An Ontology of ComputerAided Design
215
6. Reformulation type 1: consists of focussing on different structures than previously expected (process 9). Precursors of this process are the interpretation of external structure (process 13), constructive memory of structure (process 6) or the interpretation of new requirements on structure (process 3). 7. Reformulation type 2: consists of focussing on different behaviours than previously expected (process 8). Precursors of this process are the derivation of behaviour from structure (process 14), the interpretation of external behaviour (process 19), constructive memory of behaviour (process 5) or the interpretation of new requirements on behaviour (process 2). 8. Reformulation type 3: consists of focussing on different functions than previously expected (process 7). Precursors of this process are the ascription of function to behaviour (process 16), the interpretation of external function (process 20), constructive memory of function (process 4) or the interpretation of new requirements on function (process 1). The numbering of the eight design steps, similar to the 20 processes, does not prescribe any order of execution. While it may be expected for some routine design tasks to follow a sequential execution of only the first five steps, it has been found that all three types of reformulation frequently occur throughout the process of designing (McNeill et al. 1998).
Figure 2. The situated FBS framework.
216
Udo Kannengiesser and John S. Gero
The situated FBS framework represents designing independently of the domain of the design and the specific methods used, and of the subject carrying out the process of designing. What we have referred to as the “design agent” in the definition of the three design worlds can be embodied by a human designer (or team of human designers), a computational tool, or a combination of both.
3. A ProcessCentred Ontology of Design The objectcentred ontology of design presented in Section 2 has been helpful for establishing a basic understanding of design. Its emphasis on artefacts provides an intuitive, tangible perspective, representing the process of designing as a gradual evolution of the design object across three levels. The threeworld model of design interactions, in which this representation is embedded, is sufficiently rich to account for the phenomena of situatedness. However, the objectcentred ontology lacks sufficient detail and rigour to be useful for comparing or developing different methods and computer support for designers. The key ideas and semantics conveyed by the situated FBS framework are only informally expressed using textual, naturallanguage descriptions such as in Sections 2.2 and 2.3. The graphical model in Figure 2 does not fully capture these semantics. The mapping of the 20 processes onto Gero’s (1990) eight fundamental design steps has added some more meaning by locating these processes within typical phases of a design project. However, this mapping does not completely capture all the semantics and is too informal to be used as an ontological framework for computeraided design. What is needed is an ontology that is processcentred, treating design processes as firstclass entities rather than derivatives of objectcentred constructs. This Section will present such an ontology, extending our recent work on an FBS ontology of processes (Gero and Kannengiesser 2007).
3.1. An Ontology of Processes Processes are usually understood as entities that are less tangible than (physical) objects. Nonetheless, they can be represented using the same set of ontological constructs as used for describing objects: function, behaviour and structure. To clearly distinguish between the notations of the processcentred and the objectcentred FBS ontology, we will use the indices “p” for “process” and “o” for “object”.
3.1.1. Process Function Function (Fp) of a process is ontologically no different to object function, as it is based on the observer’s goals rather than on embodiment as an object or as a process. Instances of process functions are largely domaindependent. However, most processes that we design and execute through actions have the general function of replacing an existing state of the world with a desired one.
An Ontology of ComputerAided Design
217
3.1.2. Process Behaviour Behaviour (Bp) of a process relates to attributes that allow comparison on a performance level as a basis for process evaluation. Typical process behaviours are speed, cost, amount of space required and accuracy. These behaviours can be specialised and/or quantified for instances of processes in particular domains.
3.1.3. Process Structure Through an analogy with the structure of physical objects, we can distinguish between a macro and a microstructure (Sp) of processes.
Figure 3. The macrostructure of a process (i = input; t = transformation; o = output).
The macrostructure of a process includes three components and two relationships, Figure 3. The components are • • •
an input (i), a transformation (t) and an output (o).
The relationships connect • •
the input and the transformation (i – t) and the transformation and the output (t – o).
Input (i) and output (o) represent properties of entities being transformed in terms of their variables and/or their values. For example, the process of transportation changes the values for the location of a (physical) object (e.g. the values of its x, y and zcoordinates). The process of electricity generation takes mechanical motion as input and produces electrical energy as output. A common way to describe the transformation (t) of a process is in terms of a plan, a set of rules or other procedural descriptions. A typical example is a software procedure that is expressed in source code or as an activity diagram in the Unified Modeling Language (UML). Such descriptions are often used to specify subcomponents of the transformation. The relationships between the three components of a process are usually unidirectional from the input to the transformation and from the transformation to the output. For iterative processes the t – o relationship is bidirectional to represent the feedback loop between the output and the transformation. The microstructure or “material” of a process differs from the macrostructure because its components and relationships cannot be distinguished (or are not relevant) at the same level of abstraction. For example, it is not common to specify the (business process)
218
Udo Kannengiesser and John S. Gero
transformation “pay the supplier” in terms of more finegrained activities (subcomponents) such as “log in to online banking system”, “fill out funds transfer form” and “click the submit button”. This set of activities is best viewed as a microstructure specified only through a shorthand qualifier such as “using internet banking”. Microstructure can also be associated with the input and output components of a process. For example, a set of measuring data that is the input of a statistical analysis process may be “materialised” through either digital or paperbased media. While microstructure is clearly needed to carry out (“materialise”) a process, the components and relationships of that microstructure are not explicitly represented. This fits with one of MerriamWebster’s definitions of material as “the formless substratum of all things which exists only potentially and upon which form acts to produce realities”. The “formless substratum” of a transformation may reference not only processes but also objects. It then denotes the entity or agent executing the transformation. In the “pay the supplier” example, it is possible to specify “finance officer” or “purchasing department” as a general descriptor for the executing agent. Including such references to agents (as “actors” or “roles”) has become wellestablished in process modelling (Curtis et al. 1992). Since microstructure does not specify components and relationships, it can be embodied by either (micro) objects or (micro) processes. In some instances, the microstructure of objects can refer to (micro) processes rather than (micro) objects. For example, the chemical bonds (macrorelationships) between the atoms (macrocomponents) of a molecule are realised by physical processes, according to the laws of quantum electrodynamics. A view of the world as being based on processes rather than objects has generally been suggested in process philosophy (Rescher 2006).
3.1.4. Relationships between Process Function, Behaviour and Structure Relationships among Fp, Bp and Sp are constructed according to the same principles as described for Fo, Bo and So (see Section 2.1.4). Function is ascribed to behaviour based on associations of process performance with human goals. Behaviour can be derived from structure either directly or indirectly based on external effects. An example of directly derived behaviour is the speed of a process, as this depends exclusively on the macrostructure (“what kind of transformation is used on what input to produce what output?”) and the microstructure (“how and by whom is the transformation carried out using what input/output media?”). An example of indirectly derived behaviour is accuracy, which needs an external benchmark against which the output of the process is compared.
3.2. An Ontology of Design Processes The FBS ontology of processes can be used to rerepresent the objectcentred description of the 20 design processes (presented in Section 2.3) as a processcentred one. Most parts of the objectcentred model depicted in Figure 2 directly map onto the input and output components of process macrostructure (Sp). For example, Sp of process 14 in Figure 2 includes (interpreted) object structure (So) as input (i) and (interpreted) object behaviour (Bo) as output
An Ontology of ComputerAided Design
219
(o).1 No specific information is given about transformation components, as this is available only at an instance level. Most of the semantics of the situated FBS framework can be captured by process function (Fp). Table 1 gives an overview of the structure and functions of each of the 20 design processes. Table 1. Function (Fp) and macrostructure (Sp) of the 20 design processes ID
Process class
(macro) Sp
1 2 3 4 5 6 7 8 9 10 11 12
Action
FRe → Fi BRe → Bi SRe → Si Fi → Fi Bi → Bi Si → Si Fi → Fei Bi → Bei Si → Sei Fei → Bei Bei → Sei Sei → Se
13
Interpretation
Se → Si
14
Transformation
Si → Bi
15 16 17 18 19 20
Comparison Transformation Action
{Bei, Bi} → decision Bi → Fi Bei → Be Fei → Fe Be → Bi Fe → Fi
Interpretation
Constructive memory Focussing
Transformation
Interpretation
Fp 1. transfer design concepts as intended 2. reinterpret design concepts 1. retrieve design concepts as stored 2. reconstruct design concepts construct function state space construct behaviour state space construct structure state space construct behaviour state space generate values for design structure 1. communicate the design to others 2. initiate reflective conversation 1. transfer design concepts as intended 2. reinterpret design concepts 1. analyse for performance expectations 2. generate new design issues evaluate the design generate new design issues 1. communicate the design to others 2. initiate reflective conversation 1. transfer design concepts as intended 2. re interpret design concepts
Interpretation processes (1, 2, 3, 13, 19 and 20) can have two different functions. One function is to transfer existing design concepts from one agent to another or the same agent without a change of the initial meaning of these concepts. This involves bringing external representations into a form that allows processing of these representations by the individual design agent. The other function of interpretation is to reinterpret design concepts based on existing ones. This generates design concepts and issues that are novel with respect to the ones initially intended. Constructive memory processes (4, 5 and 6) have a similar set of functions. One function is to retrieve design concepts from some storage space in the same way as they were experienced at the time of storage. While this may include some computation or transformation, such as refinement or decomposition of design concepts, the results of this 1
Indices for “interpreted” have been omitted here to improve notational clarity.
220
Udo Kannengiesser and John S. Gero
process will all have a predefined relationship with the initial concepts. The other function of the constructive memory processes is to reconstruct and thereby modify existing design concepts, which corresponds to the notion of reflection (Schön 1983). Focussing processes (7, 8 and 9) have the function to construct the design state space. This includes the construction of the initial design state space (maps onto the formulation step) and subsequent modifications of that space (maps onto the reformulation steps). Action processes (12, 17 and 18) can have two different functions. One function is to communicate aspects of the design to other stakeholders (agents). Here, the notion of communication is used in its traditional sense of sharing information, based on unambiguous transfer of design concepts. The other function is to initiate reflective conversation, either with other stakeholders (agents) or the initiator of the action process itself. In other words, external representations are produced to be reinterpreted in new ways. Processes 10, 11, 14 and 16 may be called “FBSo transformations” based on their role as transformers between Fo, Bo and So. Process 10 has the function to construct the behaviour state space, and process 11 has the function to generate values within the (previously constructed) structure state space. Process 14 has two functions. One function is to analyse the design with respect to current performance expectations. The other function is to generate new design concepts that can be included as new issues in the current design task. This is also the function of process 16. The comparison process (15) has the function to evaluate the design, based on decision making informed by comparison of expected and interpreted design performance. It can be seen that some of the functions (Fp) – loosely speaking – relate to nonsituated and others to situated aspects of designing. Nonsituated aspects are captured by those functions that do not address the potential for change during designing. These are the functions that involve “transfer” (in interpretation processes), “retrieval” (in constructive memory processes) and “communication” (in action processes). Situated aspects of designing describing the potential for change are captured by functions that involve “reinterpretation” (in interpretation processes), “reconstruction” (in constructive memory processes) and “reflective conversation” (in action processes). Table 1 does not include the behaviours (Bp) of the 20 processes. This is because, at the current level of abstraction, they are no different from the general process behaviours described in Section 3.1.2. This is based on the independence of our ontology of specific methods or design domains. No detailed information about structure (Sp) and exogenous effects is available to be able to specialise or quantify general process behaviours (Bp) such as speed, accuracy and cost. An example for such detailed information would be when process structures (Sp) were considered that contain iterations (e.g., when using genetic algorithms (GAs) in design synthesis). In this case, the behaviour (Bp) “rate of convergence” could be derived that is a specialisation of the behaviour (Bp) “speed”. However, as our aim here is to provide a general rather than an instancespecific ontology, different classes of design processes are distinguished only at the level of function (Fp) and structure (Sp).
4. An Ontological Framework for ComputerAided Design Support The ontological view presented in Sections 2 and 3 has provided a detailed description of 20 distinct processes in designing. This is useful for enhancing our understanding of designing as
An Ontology of ComputerAided Design
221
a human activity. However, the ultimate aim of most research in design is to enhance the performance of this activity, both in terms of higher effectiveness and efficiency. The key to improving performance or behaviour (Bp) of designing is in the structure (Sp) it is derived from. This requires more detailed representations of structure (Sp) than presented in Table 1, and mainly concerns microstructure. Exploring the microlevel of process structure is a general research theme that has been recognised in a number of other disciplines (Osterweil 2005). Research in the microstructure (Sp) of designing can be characterised loosely as either method or tooloriented. Methodoriented approaches focus on processcentred representations of microstructure. These representations can be viewed as composing a new macrostructure to be “materialised” by humans or tools. Tooloriented design research focuses on objectcentred representations of microstructure in terms of new design tools. Computeraided design research and development is clearly located in this field. Both method and tooloriented research streams are complementary, as each of them often uses results from the other. Computeraided design tools can themselves be regarded as design objects. Applying the FBS ontology to these tools provides a schema for the characteristics that the tools must exhibit to be useful in the process of designing. We will use the index “t” for “tool” to distinguish the FBS view of tools from the FBS view of design objects and design processes. The most essential characteristics of a tool relate to function (Ft) as they orient the specification of a tool’s behaviour (Bt) and structure (St) towards the required goals and context of use. Many of the functions of computeraided design tools do not differ from any other software product. They include such general characteristics as usability, reliability, maintainability and others (ISO 2001). However, there are a number of functions that are specific to computeraided design tools. These functions relate to the tools’ role as the “material” of design processes, and can generally be described as “to support design processes of class X”. For example, a general function (Ft) of a commercial CAD tool is “to support design processes of class X = documentation” (one of the fundamental design steps presented in Section 2.3). These functions can be further specialised using particular combinations of the FBSp properties of the 20 design processes presented in Table 1. An example of a more specific function (Ft) of a CAD tool is “to support the process structure (Sp) Sei → Se in a way to achieve the process function (Fp) of communicating the design to others”. The set of functions (Ft) derivable in this way can serve as highlevel requirements for the development of new design tools. This approach makes research and development in computeraided design look like a design process, generating computational models and architectures as the structure (St) of tools exhibiting certain behaviours (Bt) to achieve the required functions (Ft). The remainder of this Section will cast existing work on computeraided design systems in this ontology, classifying that work based on tool functions (Ft) derived from combinations of Fp and Sp shown in Table 1. This aims to provide an overview of the current range of both commercial software and academic proofofconcept demonstrators. For this purpose, detailed descriptions of their behaviour (Bt) and structure (St) are not required. Readers may consult our references to the literature for more specific information.
222
Udo Kannengiesser and John S. Gero
Figure 4. Action in the situated FBS framework.
4.1. Computeraided Design Support for Action The notion of a “tool” has traditionally been viewed as a mechanism for humans to perform actions. Computeraided design tools can serve two possible functions (Ft) in their support of action (see Table 1): • •
to support communicating the design to support initiating reflective conversation
Figure 4 highlights processes 12, 17 and 18 in the situated FBS framework to represent actions related to these two functions.
4.1.1. Support for Communicating the Design •
Sei → Se (process 12): The ability to generate representations of external object structure (So) is provided by commercial CAD systems. These tools produce 2D or 3D models and offer functionalities such as scaling, rotating and rendering to communicate different aspects of the object. The models generated by CAD systems
An Ontology of ComputerAided Design
•
•
223
are primarily used for data exchange with other designers, manufacturers or other stakeholders, or for providing input for tools that perform analyses of the designed objects. Communication across different tools has been recognised as an area of growing concern, as the tools generally use different languages (data formats) for representing object structure. A number of approaches address this problem by defining standardised product models, the best known of which are STEP and IFCs (Eastman 1999). Many CAD tools now have translators (called preprocessors) that map object structure onto a neutral format based on these standards. Some of our previous work was concerned with developing an agentbased approach to communicating product data in situations where no standard formats are available (Kannengiesser and Gero 2006; Kannengiesser and Gero 2007). Bei → Be (process 17): Virtual reality (VR) systems are increasingly used to generate 3D objects in a placelike context that usually include avatars representing potential users or stakeholders of the design. These tools support modelling not only the structure (So) but also the behaviour (Bo) of the designed object based on simulated interactions with avatars or other objects. Digital mockups (DMUs) are based on a similar concept, and are commonly used for the simulation of assembly operations or kinematics. Other tools that focus mainly on the communication of object behaviour (Bo) are those specialised in performing particular engineering analyses. Typical examples here include the representation of stresses and temperatures. Fei → Fe (process 18): There are currently no commercial tools specialised in generating formal representations of object function (Fo). This is mainly due to the lack of a commonly agreed representation language. In most cases, function is described informally using natural language expressions, usually based on verbnoun pairs (Jacobsen et al. 1991) that are also used in this chapter. These descriptions can be produced by generalpurpose word processors and annotation mechanisms provided by CAD systems. Future tool support may result from recent work on more formal representations of function (Chandrasekaran and Josephson 2000; Stone and Wood 2000; Szykman et al. 2001; Deng 2002).
4.1.2. Support for Initiating Reflective Conversation •
•
Sei → Se (process 12): There are no commercial design tools that explicitly aim at supporting reflective conversation. However, there are some methodoriented approaches that may inform the development of such tools. For example, Jun and Gero (1997) have demonstrated how shapes can emerge by representing the same geometrical structure in different ways. Current CAD systems do not have this ability, as their representations are fixed through the way they store a design’s geometry in their database. An approach by Reymen et al. (2006) uses checklists and forms for designers to stimulate the creation of textual descriptions of designs from multiple perspectives, at regular intervals during the process of design. Bei → Be (process 17): Reflective conversation at the behaviour level has not been well understood. However, the models of generating multiple representations described for the structure level can be applied when behaviour is represented using
224
Udo Kannengiesser and John S. Gero
•
shapes. The notion of space, for example, can be viewed as a behaviour (derived from a wallsandfloor structure) that can be described geometrically. Fei → Fe (process 18): Apart from cases in which functions represent references to shapes, reflective conversation at the function level has not been well understood. Tool support for generating multiple, textual representations of function may be developed based on research in natural language semantics. For example, de Vries et al. (2005) explore the use of the WordNet lexicon (Miller 1995) to generate a graph of synonyms and other semantic relations from a given set of words.
4.2. Computeraided Design Support for FBSo Transformations and Evaluation A number of research efforts have concentrated on tool support for performing those transformations and evaluations that have been viewed as fundamental in most traditional models of designing (e.g., Asimov (1962)). These include the transformations between the function, behaviour and structure of the design object, and evaluation based on comparing expected with “actual” behaviour. Figure 5 highlights processes 10, 11, 14, 15 and 16 to represent these activities.
Figure 5. FBSo transformations and evaluation in the situated FBS framework.
An Ontology of ComputerAided Design •
•
•
•
•
225
Si → Bi (process 14): There is a wide range of commercial tools that support the derivation of object behaviour from design structure. These are commonly referred to as analysis tools or simulation tools. Most of them are based on the physical laws and principles established in the engineering sciences. Examples of design analyses for which there is automated support include finite element analysis, thermal analysis, energy analysis and kinematic analysis. Some tools, such as design optimisation tools and parametric CAD systems, provide automated support for the Si → Bi transformation as part of a collection of transformations that also include evaluation (process 15) and the generation of object structure (process 11). These tools will be presented in more detail under the bullet points for processes 11 and 15 (below). The function of generating new design issues (see Table 1) is addressed by some CAD systems performing runtime analyses of the design, such Design for X (DFX) analyses. Gero and Kazakov (1998) have developed a computational model of behaviour analogy where new behaviour variables are introduced into the target design based on structure similarity with the source design. Bei → Sei (process 11): Parametric CAD systems have shown to significantly facilitate the creation of solid models (Shah and Mäntylä 1995), and many CAD vendors now offer parametric modelling features. These systems can be viewed as automating the process of computing an object structure once a set of parameters have been formulated for both structure and behaviour. Parametric CAD systems also allow for automated maintenance of parametric constraints (Sacks et al. 2004). This requires additional automation for analysing and evaluating the design for constraint violations, which can be mapped onto the transformation process Si → Bi (process 14) and the evaluation process {Bei, Bi} → decision (process 15). Design optimisation tools provide similar integrated functionalities supporting the same set of processes. They provide an extensive range of mechanisms to evolve object structure, including various deterministic and stochastic search methods (Papalambros and Wilde 2000). {Bei, Bi} → decision (process 15): Automated support for this process is provided in a number of computeraided design systems, as indicated above. Optimisation tools, in particular, incorporate sophisticated strategies for controlling the execution of alternative search paths, based on the performance of the current design candidate. Research on agentbased design systems addresses evaluation using conflict resolution mechanisms, which have been applied to instances of multiobjective design optimisation (Grecu and Brown 1996; Campbell et al. 1999). Fei → Bei (process 10): Few systems have been developed that support the generation of object behaviours based on object function (Maiden and Sutcliffe 1992; Bhatta et al. 1994; Umeda et al. 1996). This is mainly due to the lack of a formal language to represent function. Bi → Fi (process 16): There has been no work to date on tool support for this process.
226
Udo Kannengiesser and John S. Gero
Figure 6. Focussing in the situated FBS framework.
4.3. Computeraided Design Support for Focussing There has been some work on tools to support focussing, the processes involved in the formulation of a design state space. These tools are mainly based on decisionmaking mechanisms that use various kinds of information. Figure 6 highlights processes 7, 8 and 9 to represent focussing. •
Si → Sei (process 9): A number of computational approaches to focussing on object structure have been developed in the area of design optimisation. Some of this work uses information extracted from the current design. For example, Parmee’s (1996) clusteroriented genetic algorithms (COGAs) identify highperformance regions within the current structure state space. These features are then used for focussing on different structure variables and constraints, to concentrate the search for an optimum design on particular areas within the original structure state space. Other work uses information learnt from previous design tasks. A tool developed by Schwabacher et al. (1998) extracts characteristics of previous optimisation results and uses them to formulate new optimization problems. These characteristics include information such
An Ontology of ComputerAided Design
•
•
227
as optimal structure, mappings between structure and behaviour, infeasible behaviour and active constraints. This information is used to improve the problem formulation by reducing the structure state space. Bi → Bei (process 8): Some work has been done on focussing at the level of object behaviour, again mostly in the context of optimisation. Mackenzie and Gero (1987) have induced rules to detect certain features of Pareto optimal sets relating to curvature, sensitivity and other information. The rules use this information to reformulate the problem by carrying out focussing in a way that reduces the behaviour state space. Jozwiak’s (1987) approach uses learning to acquire knowledge of inactive constraints, which is then used to predict whether or not the constraints of the current optimisation task may be neglected. Fi → Fei (process 7): There has been no work to date on tool support for this process.
Figure 7. Interpretation in the situated FBS framework.
228
Udo Kannengiesser and John S. Gero
4.4. Computeraided Design Support for Interpretation Tools usually have some form of interface to receive and utilise input provided externally either by humans or other tools. In computeraided design, there are two possible functions (Ft) for interpretation by tools: • •
to support transferring design concepts as intended to support reinterpreting design concepts
Figure 7 highlights processes 1, 2, 3, 13, 19 and 20 to represent interpretation.
4.4.1. Support for Transfer of Design Concepts as Intended •
•
•
SRe → Si (process 3) and Se → Si (process 13): There has been considerable research in the computational interpretation of external object structure. The standardisation approaches to product modelling, mentioned in Section 4.1.1, provide the basis for the development of import mechanisms (called postprocessors) that translate the standard models into the tool’s native format. Postprocessors for STEP and IFC models are available in a number of commercial CAD/CAE/CAM systems. Another area of research is concerned with the interpretation of human sketches and freehand drawings by tools converting them into more exact graphical models or performing early design analyses (Taggart 1975; Gross 1996; Leclercq 2001). BRe → Bi (process 2) and Be → Bi (process 19): Most design tools dealing with object behaviour directly derive that behaviour from structure (process 14) rather than interpreting it externally (e.g., from other tools). As a result, not much work exists on tool support for the interpretation of object behaviour. However, recent approaches to interoperability aiming to standardise the representation of function and behaviour besides structure (Szykman et al. 2001) may lead to the development of tool translators that automate this process. FRe → Fi (process 1) and Fe → Fi (process 20): Most tool support for the interpretation of object function is based on mechanisms of word recognition, given that many representations of function are described using natural language annotations. While there is a large number of generalpurpose tools that provide interfaces for textual input (such as Word processors or electronic whiteboards), only few of them (e.g., the word generation system developed by de Vries et al. (2005)) offer more wordprocessing features than just editing. Future work on the interpretation of function can be expected to be driven by advances in both representing and reasoning about function, particularly in the area of design interoperability (Szykman et al. 2001).
4.4.2. Support for ReInterpretation of Design Concepts •
SRe → Si (process 3) and Se → Si (process 13): Most research in reinterpretation has been done at the level of object structure. A system presented by Saund and Moran (1994) supports the creation of multiple interpretations of line drawings, by first decomposing and then reassembling elements of freehand drawings. The
An Ontology of ComputerAided Design
• •
229
emerging shapes are then presented to the user for selection. A design agent capable of reinterpretation has been developed by Smith and Gero (2001) on the basis of Gero and Fujii’s (2000) “pushpull” model of situated cognition. This system has been able to learn new shapes over sequences of action and (re)interpretation that are themselves the result of the agent’s modified experience. BRe → Bi (process 2) and Be → Bi (process 19): There has been no work to date on tool support for this process. FRe → Fi (process 1) and Fe → Fi (process 20): There has been no work to date on tool support for this process.
4.5. Computeraided Design Support for Constructive Memory Most work on computeraided design tools includes support for memory in some way. There are two possible functions (Ft) related to this notion: • •
to support retrieval of design concepts as stored to support reconstruction of design concepts
Figure 8 highlights processes 4, 5 and 6 to represent constructive memory.
Figure 8. Constructive memory in the situated FBS framework.
230
Udo Kannengiesser and John S. Gero
4.5.1. Support for Retrieval of Design Concepts as Stored •
•
•
Si → Si (process 6): Research in using memory of object structure includes work on featurebased modelling. A number of CAD systems provide design databases, repositories or libraries to store design features, such as pockets, holes and slots. Their reuse can lead to significant gains of productivity in designing. Techniques of feature extraction from geometrical CAD models can be viewed as another example of retrieving design concepts, although they require some additional computation. Here, features are implicitly stored in the predefined mappings underpinning common extraction techniques such as graph matching, syntactic pattern recognition and shape grammars (Shah 1991). Bi → Bi (process 5): Some recent work on design repositories has concentrated on including properties related to behaviour (Bo) and function (Fo) of the design object (Szykman et al. 2001; Mocko et al. 2004). In addition, approaches to capturing and reusing design rationale have focused on appropriate representations of previous object behaviour to be accessible for guiding the generation of new object structure (Chandrasekaran et al. 1993). Fi → Fi (process 4): Simple retrieval of object function is best exemplified by work on storing and reusing function (Fo) hierarchies in design repositories (Szykman et al. 2001) or case bases (Navinchandra et al. 1991). Approaches to retrieving implicitly stored functions include work, mentioned earlier, on inferring word relations based on WordNet (de Vries et al. 2005). Other work focuses on the construction of subfunctions using decomposition knowledge encoded in grammars (Sridharan and Campbell 2005).
4.5.2. Support for ReConstruction of Design Concepts •
Si → Si (process 6), Bi → Bi (process 5) and Fi → Fi (process 4): The idea of generating design concepts by situated reconstruction rather than static retrieval from previous experience is quite new in design research. As a result, very little work has been done towards the development of computational models and tools that support this process. However, a number of research demonstrators have shown both the feasibility and the potential benefits of future constructive memory tools. Examples include neural network implementations used for the design of mechanical assemblies (Liew and Gero 2004), design optimisation (Peng and Gero 2006) and the exchange of product data between design tools (Kannengiesser and Gero 2007). The majority of this work provides support for reconstruction of design concepts at all three levels, comprising function (Fo), behaviour (Bo) and structure (So).
5. Conclusion Designing comprises a rich set of activities that is only beginning to be completely understood. Capturing these activities and defining them in a detailed framework is necessary to advance our understanding of design. The ontological framework presented in this chapter
An Ontology of ComputerAided Design
231
is a contribution to this aim. It extends our previous, objectcentred work on representing the process of designing by adding a processcentred view. This view is based on the direct application of the FBS ontology to design activities, treating them as firstclass entities with their own function, behaviour and structure, and no longer as mere derivatives of objectcentred constructs. This provides a more structured description at a higher level of detail, which has the potential to make our framework of situated designing more amenable to other researchers. We have shown that the processcentred ontology of designing also allows specifying a set of requirements for tool support. This is based on the connection we established between the FBS view of design processes and the FBS view of design tools. Specifically, tools are viewed as artefacts whose functions (Ft) are specialised to supporting particular aspects of design processes, which themselves consist of combinations of function (Fp), behaviour (Bp) and structure (Sp). We have demonstrated how some of the outcomes of existing computeraided design research and development can be mapped onto 20 classes of design processes represented in this way. One result of our mappings is that a lack of tool support can be identified for a number of design activities. At the level of granularity presented in this chapter, this concerns activities of reinterpretation and reconstruction of design concepts, and reasoning and focussing on object function (Fo). Our ontology allows understanding the research field of computeraided design as the “materials science” of designing, concerned with creating and analysing tools to form appropriate “materials” of design processes at different levels of granularity. This is possible because the FBS ontology represents all design objects, tools and processes uniformly. Future research may use this ontology to create more finegrained specifications of design tools. For example, different classes of feature extraction processes can be defined based on different classes of inputs (e.g., cubic, cylindrical or freeform shapes) and on different classes of transformations (e.g., graph matching, shape grammars, neural networks, etc.), and consequently different functions (Ft) of feature extraction tools can be derived. Information about specific process behaviour (Bp) and process function (Fp), on an instance level, can be added to derive more refined tool functions (Ft). Researchers and developers in computeraided design can then identify specific gaps in the functions (Ft) of existing tools and generate the behaviour (Bt) and ultimately the structure (St) of new tools to close these gaps.
Acknowledgments NICTA is funded by the Australian Government's Department of Communications, Information Technology and the Arts, and the Australian Research Council through Backing Australia's Ability and the ICT Research Centre of Excellence program.
References Asimov, M: 1962, Introduction to Design, PrenticeHall, Englewood Cliffs. Bartlett, FC: 1932 reprinted in 1977, Remembering: A Study in Experimental and Social Psychology, Cambridge University Press, Cambridge.
232
Udo Kannengiesser and John S. Gero
Bhatta, S, Goel, A and Prabhakar, S: 1994, Innovation in analogical design: A modelbased approach, in JS Gero and F Sudweeks (eds) Artificial Intelligence in Design ’94, Kluwer, Dordrecht, pp. 5774. Bickhard, MH and Campbell, RL: 1996, Topologies of learning, New Ideas in Psychology 14(2): 111156. Campbell, MI, Cagan, J and Kotovsky, K: 1999, ADesign: An agentbased approach to conceptual design in a dynamic environment, Research in Engineering Design 11(3): 172192. Chandrasekaran, B, Goel, AK and Iwasaki, Y: 1993, Functional representation as design rationale, IEEE Computer 26(1): 4856. Chandrasekaran, B and Josephson, JR: 2000, Function in device representation, Engineering with Computers 16(34): 162177. Clancey, WJ: 1997, Situated Cognition: On Human Knowledge and Computer Representations, Cambridge University Press, Cambridge. Curtis, B, Kellner, MI and Over, J: 1992, Process modeling, Communications of the ACM 35(9): 7590. Deng, YM: 2002, Function and behavior representation in conceptual mechanical design, Artificial Intelligence for Engineering Design, Analysis and Manufacturing 16(5): 343362. Dewey, J: 1896 reprinted in 1981, The reflex arc concept in psychology, Psychological Review 3: 357370. Eastman, CM: 1999, Building Product Models: Computer Environments Supporting Design and Construction, CRC Press, Boca Raton. Gero, JS: 1990, Design prototypes: A knowledge representation schema for design, AI Magazine 11(4): 2636. Gero, JS and Fujii, H: 2000, A computational framework for concept formation for a situated design agent, KnowledgeBased Systems 13(6): 361368. Gero, JS and Kannengiesser, U: 2004, The situated functionbehaviourstructure framework, Design Studies 25(4): 373391. Gero, JS and Kannengiesser, U: 2007, A functionbehaviorstructure ontology of processes, Artificial Intelligence for Engineering Design, Analysis and Manufacturing 21(4), in press Gero, JS and Kazakov, V: 1998, Using analogy to extend the behaviour state space in creative design, in JS Gero and ML Maher (eds) Computational Models of Creative Design IV, Key Centre of Design Computing and Cognition, University of Sydney, Australia, pp. 113143. Grecu, DL and Brown, DC: 1996, Learning by single function agents during spring design, in JS Gero and F Sudweeks (eds) Artificial Intelligence in Design ’96, Kluwer, Dordrecht, pp. 409428. Gross, MD: 1996, The electronic cocktail napkin – a computational environment for working with design diagrams, Design Studies 17(1): 5369. Guindon, R: 1990, Designing the design process: Exploiting opportunistic thoughts, HumanComputer Interaction 5: 305344. ISO: 2001, Software Engineering – Product Quality – Part 1: Quality Model, ISO/IEC 91261, International Organization for Standardization, Geneva, www.iso.ch
An Ontology of ComputerAided Design
233
Jacobsen, K, Sigurjonsson, J and Jacobsen, O: 1991, Formalized specification of functional requirements, Design Studies 12(4): 221224. Jozwiak, SF: 1987, Improving structural optimization programs using artificial intelligence concepts, Engineering Optimization 12: 155162. Jun, HJ and Gero, JS: 1997, Representation, rerepresentation and emergence in collaborative computeraided design, in ML Maher, JS Gero and F Sudweeks (eds) Preprints Formal Aspects of Collaborative ComputerAided Design, Key Centre of Design Computing and Cognition, University of Sydney, Australia, pp. 303320. Kannengiesser, U and Gero, JS: 2006, Towards mass customized interoperability, ComputerAided Design 38(8): 920936. Kannengiesser, U and Gero, JS: 2007, Agentbased interoperability without product model standards, ComputerAided Civil and Infrastructure Engineering 22(2): 8097. Leclercq, P: 2001, Programming and assisted sketching, in B de Vries, JP van Leeuwen and HH Achten (eds) CAAD Futures 2001, Kluwer Academic Publishers, Dordrecht, pp. 1532. Liew, P and Gero, JS: 2004, Constructive memory for situated design agents, Artificial Intelligence for Engineering Design, Analysis and Manufacturing 18(2): 163198. Mackenzie, CA and Gero, JS: 1987, Learning design rules from decisions and performances, Artificial Intelligence in Engineering 2(1): 210. Maiden, NA and Sutcliffe, AG: 1992, Exploiting reusable specifications through analogy, Communications of the ACM 35(4): 5563. McNeill, T, Gero, JS and Warren, J: 1998, Understanding conceptual electronic design using protocol analysis, Research in Engineering Design 10(3): 129140. Miller, GA: 1995, WordNet: A lexical database for English, Communications of the ACM 38(11): 3941. Mocko, G, Malak, R, Paredis, C and Peak, R: 2004, A knowledge repository for behavioral models in engineering design, Computers and Information Science in Engineering Conference ’04, Salt Lake City, UT. Navinchandra, D, Sycara, KP and Narasimhan, S: 1991, Behavioral synthesis in CADET, a casebased design tool, IEEE Conference on Artificial Intelligence Applications, Miami Beach, FL, pp. 217221. Osterweil, LJ: 2005, Unifying microprocess and macroprocess research, in M Li, B Boehm and LJ Osterweil (eds) Unifying the Software Process Spectrum, SpringerVerlag, Berlin, pp. 6874. Papalambros, P and Wilde, DJ: 2000, Principles of Optimal Design: Modeling and Computation, Cambridge University Press, Cambridge. Parmee I.C. (1996) Towards an optimal engineering design process using appropriate adaptive search strategies, Journal of Engineering Design 7(4): 341362. Peng, W and Gero, JS: 2006, Concept formation in a design optimization tool, in J van Leeuwen and H Timmermans (eds) Innovations in Design Decision Support Systems in Architecture and Urban Planning, SpringerVerlag, Berlin, pp. 293308. Rescher, N: 2006, Process Philosophical Deliberations, OntosVerlag, Frankfurt. Reymen, IMMJ, Hammer, DK, Kroes, PA, van Aken, JE, Dorst, CH, Bax, MFT and Basten, T: 2006, A domainindependent descriptive design model and its application to structured reflection on design processes, Research in Engineering Design 16(4): 147173.
234
Udo Kannengiesser and John S. Gero
Sacks, R, Eastman, CM and Lee, G: 2004, Parametric 3D modeling in building construction with examples from precast concrete, Automation in Construction 13(3): 291312. Saund, E and Moran, TP: 1994, A perceptuallysupported sketch editor, ACM Symposium on User Interface Software and Technology, ACM Press, New York. Schön, DA: 1983, The Reflective Practitioner: How Professionals Think in Action, Harper Collins, New York. Schön, DA and Wiggins, G: 1992, Kinds of seeing and their functions in designing, Design Studies 13(2): 135156. Schwabacher, M, Ellman, T and Hirsh, H: 1998, Learning to set up numerical optimizations of engineering designs, Artificial Intelligence for Engineering Design, Analysis and Manufacturing 12(2): 173192. Shah, JJ: 1991, Assessment of features technology, ComputerAided Design 23(5): 331343. Shah, JJ and Mäntylä, M: 1995, Parametric and FeatureBased CAD/CAM: Concepts, Techniques, and Applications, John Wiley & Sons, New York. Smith, GJ and Gero, JS: 2001, Interaction and experience: Situated agents and sketching, in JS Gero and FMT Brazier (eds) Agents in Design 2002, Key Centre of Design Computing and Cognition, University of Sydney, pp. 115132. Smith, GJ and Gero, JS: 2005, What does an artificial design agent mean by being ‘situated’?, Design Studies 26(5): 535561. Sridharan, P and Campbell, MI: 2005, A study on the grammatical construction of function structures, Artificial Intelligence for Engineering Design, Analysis and Manufacturing 19(3): 139160. Stone, RB and Wood, KL: 2000, Development of a functional basis for design, Journal of Mechanical Design 122(4): 359370. Suwa, M, Gero, JS and Purcell, T: 1999, Unexpected discoveries and sinventions of design requirements: A key to creative designs, in JS Gero and ML Maher (eds) Computational Models of Creative Design IV, Key Centre of Design Computing and Cognition, University of Sydney, Sydney, Australia, pp. 297320. Suwa, M and Tversky, B: 2002, External representations contribute to the dynamic construction of ideas, in M Hegarty, B Meyer and NH Narayanan (eds) Diagrams 2002, SpringerVerlag, Berlin, pp. 341343. Szykman, S, Fenves, SJ, Keirouz, W and Shooter, SB: 2001, A foundation for interoperability in nextgeneration product development systems, ComputerAided Design 33(7): 545559. Taggart, J: 1975, Sketching: An informal dialogue between designer and computer, in N Negroponte (ed.) Reflections on Computer Aids to Design and Architecture, Petrocelli Charter, New York, pp. 147162. Umeda, Y, Ishii, M, Yoshioka, M, Shimomura, Y and Tomiyama, T: 1996, Supporting conceptual design based on the functionbehaviorstate modeler, Artificial Intelligence for Engineering Design, Analysis and Manufacturing 10(4): 275288. de Vries, B, Jessurun, J, Segers, N and Achten, H: 2005, Word graphs in architectural design, Artificial Intelligence for Engineering Design, Analysis and Manufacturing 19(4): 277288. Weisberg, DE: 2000, The electronic push, Mechanical Engineering 122(4): 5259. Ziemke, T: 1999, Rethinking grounding, in A Riegler, M Peschl and A von Stein (eds) Understanding Representation in the Cognitive Sciences: Does Representation Need Reality?, Plenum Press, New York, pp. 177190.
INDEX A abstraction, 217, 220 accelerometers, 62 achievement, 209 actuation, viii, 129, 132, 133, 142 adaptation, 35 adaptations, 177 adjustment, 31 aesthetics, 151 age, 3 aggression, 119 Air Force, 206 algorithm, viii, 34, 71, 88, 89, 91, 92, 93, 96, 97, 99, 102, 104, 108, 109, 129, 130, 131, 138, 139, 141, 149, 150, 151, 163, 164, 165, 179, 186, 197, 198, 202, 206 amplitude, 70, 75, 77, 78, 123 anatomy, 132 anger, 115, 117 animations, vii, viii, ix, 55, 58, 85, 86, 87, 88, 90, 97, 103, 104, 105, 107, 108, 126, 129, 130, 145, 146 annotation, 223 anxiety, 119 articulation, 133 artificial intelligence, 233 assignment, 34 atoms, 218 Australia, 209, 231, 232, 233, 234 Austria, 60 automation, 225
B background, 118 bandwidth, 85, 89 banking, 218 beams, 9, 11, 25, 29, 33 beautification, 160, 161, 170, 172, 175 beauty, 145
behavior, viii, 24, 29, 58, 78, 113, 118, 158, 159, 161, 162, 165, 232 Beijing, 175 Belgium, 177 bending, 206 Bible, 83 blocks, 86, 87, 88, 91, 96, 148, 151 bonds, 218 bone, 130 boredom, 117 Brazil, 113, 127 breakdown, 165 Britain, 55 building blocks, 168, 169 buttons, 27
C calibration, 62 CAP, 4, 10 cast, 221 casting, 147 categorization, 133 Central Europe, 81 channels, 99 chicken, 92, 101, 102 China, 175 City, 143, 155, 173, 233 clarity, 41, 219 classes, 88, 164, 175, 212, 213, 220, 231 classification, 12, 114 cloning, 143 closure, 72, 74, 162 clustering, viii, 86, 90, 91, 92, 93, 94, 95, 96, 97, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109 clusters, 91, 92, 93, 94, 96, 97, 98, 101, 102, 104, 105, 106, 107 codes, 4, 130 coding, viii, 86, 87, 88, 89, 90, 94, 97, 98, 99, 102, 106, 107 cognition, 229 coherence, 88, 89, 99
236 communication, 59, 69, 78, 79, 173, 220, 223 compatibility, 122 compensation, 90 competition, 60 complement, 2 complexity, viii, 86, 96, 132, 150, 156, 164 components, 59, 85, 88, 89, 94, 96, 97, 101, 105, 106, 108, 136, 161, 165, 166, 171, 172, 186, 193, 211, 217, 218, 219 comprehension, 80 compression, viii, 41, 86, 88, 89, 90, 91, 92, 94, 96, 97, 98, 99, 100, 101, 102, 104, 105, 106, 107, 108, 109, 110, 111, 193, 207 computation, viii, 57, 59, 63, 64, 65, 68, 69, 75, 77, 78, 79, 80, 94, 96, 129, 130, 136, 141, 148, 149, 162, 219, 230 computer graphics, vii, viii, 57, 63, 110, 129, 131, 146, 156, 157, 159, 174 computing, 58, 94, 127, 131, 148, 179, 225 conception, vii, 58 concrete, 234 conditioning, 20 conduction, 211 configuration, 20, 41, 54, 93, 125, 126, 146, 162, 164 conflict, 120, 225 conflict resolution, 225 Congress, 82 conjecture, 206 connectivity, 85, 86, 87, 88, 89, 90, 91, 92, 94, 96, 97, 104, 108, 109, 158, 159, 161, 165, 166 consensus, 114, 115, 121 conservation, 3, 12, 54 constraintbased design, ix, 157 construction, vii, ix, 1, 16, 73, 82, 130, 161, 163, 164, 165, 167, 169, 173, 175, 177, 178, 182, 185, 187, 194, 195, 200, 201, 208, 220, 230, 234 consumption, 2 continuity, 136, 159, 178, 179, 180, 186 contour, 78, 134, 183 control, ix, 24, 27, 28, 29, 32, 33, 64, 66, 68, 122, 125, 130, 131, 132, 133, 141, 152, 159, 165, 177, 178, 179, 180, 184, 185, 186, 187, 190, 191, 192, 197, 198, 199, 200, 201, 202, 203, 204, 205, 208, 213 convergence, 93, 137, 192, 220 correlation, 88, 98, 213 costs, 193 creativity, 61, 63 cues, 105, 106, 107 cultural differences, 127 culture, 2, 3, 121 curiosity, 115, 119 customers, 80, 167 Czech Republic, 81
Index
D damping, 23 dance, 100 data set, 138, 158 data structure, 90, 94, 150 database, 149, 223, 233 decay, 121 decision making, 220 decisions, 233 decoding, 98 decomposition, 87, 90, 91, 94, 95, 96, 97, 101, 105, 106, 107, 109, 132, 142, 165, 219, 230 definition, viii, 58, 69, 99, 113, 115, 122, 159, 166, 167, 169, 170, 205, 216 deformation, viii, 67, 76, 77, 78, 97, 129, 130, 131, 133, 135, 138, 139, 162, 166, 173 degradation, 90, 105 density, 158 derivatives, 179, 216, 231 designers, vii, viii, 54, 57, 63, 80, 210, 211, 213, 216, 223 detection, 97, 107, 161, 162, 171 deviation, 130, 135 diet, 3 differentiation, 146, 185 diffusion, 12, 56 dimensionality, 142 disappointment, 119 discipline, vii, 1 discrete data, 159 displacement, 89, 99 disposition, 118 distress, 119 distribution, 2, 12, 133 divergence, 39 dominance, 119 drawing, 15, 163 DynaFeX, viii, 113
E ears, 123 ecstasy, 117 editors, 156, 208 Education, 56, 82 educational objective, 3 elastic deformation, 22, 28, 33 elearning, 61 electricity, 217 electromagnetic, 59, 62, 63, 64 emotion, viii, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 133 emotional state, 115, 121, 123, 126 emotions, viii, 113, 114, 115, 116, 117, 118, 120, 121, 122, 123, 124, 125, 126, 133 employment, 172
Index encoding, 86, 88, 89, 90, 97, 99, 100, 102, 103, 105, 106 energy, 93, 95, 115, 217, 225 England, 174 entropy, 88, 147, 156 environment, viii, 54, 55, 57, 59, 63, 65, 70, 71, 72, 77, 79, 81, 82, 83, 119, 149, 210, 211, 212, 213, 232 estimating, 136 Europe, 55 European Union, 2 evolution, vii, 1, 2, 3, 146, 216 excitation, 122 execution, 92, 163, 214, 215, 225 expressiveness, 114 extraction, 161, 162, 171, 175, 230, 231 extrapolation, 116, 117 extrusion, 66, 149
F Facial Action Coding, 130, 142 facial expression, viii, 87, 100, 113, 114, 115, 117, 120, 121, 122, 123, 124, 125, 126, 127, 129, 130, 131, 132, 133, 138, 139, 141, 142, 143, 144 facial muscles, 132, 133 family, 116, 117 FBS ontology, x, 209, 210, 216, 218, 221, 231 fear, 115, 117 feedback, 132, 217 FEM, 76, 77, 78, 80 finance, 218 finite element method, 192 flexibility, 163, 178 flight, 61, 62 floating, 85, 88 flour, vii, 1, 3, 4, 8, 9, 12, 22 fluid, viii, 57, 78, 79, 80 fluid dynamics, viii food, 39 freedom, x, 63, 72, 171, 177, 178, 198, 200, 205 friction, 11 functional dynamics, ix, 145, 146, 148, 152, 153, 155 funds, 218 furniture, 61, 166
G gene, ix, 177, 178 generalization, 126, 174, 205, 206 generation, viii, ix, 12, 57, 62, 63, 87, 107, 116, 117, 120, 122, 125, 126, 156, 157, 217, 225, 228, 230 goals, 211, 212, 213, 216, 218, 221 gold, 168
237 grains, 12, 22 graph, 91, 92, 95, 96, 97, 101, 105, 109, 147, 157, 158, 163, 164, 165, 184, 186, 197, 224, 230, 231 gravity, 22 Greece, 157 grief, 117 grounding, 234 grouping, 90, 91, 92, 95 groups, 17, 22, 98, 122, 123 growth, 118, 196 guidance, 81, 130 guidelines, 114, 126
H Hamiltonian, 131, 133, 135, 137, 139, 141, 143 happiness, 118 height, 4, 41, 122, 148, 151, 152, 154, 165, 204, 211 homogeneity, 94 human activity, 221 humidity, 62 hybrid, 158 hypercube, viii, 113, 114, 117, 118, 126
I ideal, 107, 146, 174 identification, 106 illumination, 13, 20, 34, 41, 135 illusion, 59 IMA, 206 image, viii, 3, 35, 36, 37, 38, 39, 41, 57, 60, 61, 81, 86, 91, 130, 134, 135, 136, 137, 138, 139, 141, 142, 143, 144, 147, 151, 152, 158, 193 image processing, viii, 57 imagery, 59 images, 3, 12, 15, 34, 37, 38, 41, 42, 54, 59, 60, 130, 132, 134, 135, 137, 138, 139, 141, 142, 143, 147 immersion, 59, 60 implementation, viii, 59, 63, 68, 73, 78, 80, 94, 96, 113, 123, 138, 146, 152, 177, 230 inclusion, 41, 158 independence, 220 India, 145 indices, 216 individual character, 118 industry, vii, 57, 210 inequality, 196 initial state, 20, 27, 162 insertion, 90 instability, 120 instruments, viii, 57, 63 integration, viii, 57, 63, 69, 80, 81
238
Index
interaction, viii, 57, 61, 62, 63, 69, 78, 80, 81, 146, 152, 153, 158, 211, 212, 213 interactions, x, 130, 209, 210, 211, 213, 216, 223 interface, 63, 81, 82, 228 interference, 62, 70 internet, 110, 218 interoperability, viii, 113, 228, 233, 234 interval, 117 intervention, 130 inventions, 3 iris, 122 iron, 10, 25, 168 Italy, 57 iteration, 126
J jaw, 123, 133 joints, 23, 25, 26, 27, 28, 68, 69, 70, 72, 73, 106
L language, 2, 64, 121, 223, 224, 225, 228 laws, 27, 211, 218, 225 layering, 16 learning, 54, 82, 114, 227, 232 learning environment, 82 Least squares, 109 legend, 78, 80 lens, 17, 19, 20, 39 lifetime, 119 light transmission, 211 likelihood, 95 limitation, 63, 146, 152, 166 line, 17, 39, 60, 78, 115, 122, 152, 174, 180, 201, 228 linear systems, 192 linkage, 62, 67 links, 24, 27, 28, 70, 73, 75 localization, 193 logistics, 61, 82 love, 119
M machinery, 4, 16, 54, 55 magnetic field, 62 maintenance, 61, 225 management, 173 manipulation, 81, 82, 153, 164, 166, 185, 205 manufacturing, 61, 82, 157, 160, 161, 165, 166, 169, 170, 209 mapping, 33, 34, 109, 126, 132, 146, 216 materials science, 231 matrix, 73, 75, 95, 96, 98, 100, 135, 136, 137, 138, 139, 142, 148, 162
measurement, 13, 16, 64 measures, 18, 99, 142, 148 mechanical properties, 130 mechanical stress, 76, 77 media, 218 Mediterranean, 168 memory, 3, 54, 85, 139, 156, 192, 198, 213, 214, 215, 219, 220, 229, 230, 233 memory processes, 219, 220 Miami, 233 Microsoft, 63, 64, 100, 145 microstructure, 218 military, 61 mixing, 62 model, viii, ix, 4, 13, 15, 16, 17, 24, 29, 34, 37, 39, 58, 63, 67, 73, 79, 80, 109, 113, 114, 115, 116, 117, 118, 120, 121, 122, 123, 126, 127, 129, 130, 131, 132, 133, 134, 135, 137, 138, 139, 140, 141, 142, 144, 146, 147, 151, 152, 158, 159, 160, 161, 165, 166, 167, 169, 170, 171, 175, 212, 213, 214, 216, 218, 225, 229, 233 modeling, vii, viii, 12, 16, 57, 63, 66, 69, 77, 78, 80, 82, 114, 118, 129, 131, 133, 142, 143, 144, 147, 156, 158, 159, 160, 161, 162, 165, 166, 173, 174, 175, 232, 234 models, viii, ix, 3, 12, 58, 68, 85, 108, 110, 114, 122, 129, 130, 131, 132, 133, 135, 142, 143, 144, 146, 147, 148, 155, 157, 158, 159, 160, 161, 165, 166, 167, 169, 170, 171, 172, 173, 175, 210, 211, 221, 222, 223, 224, 225, 228, 230, 233 money, 58 mood, 114, 115, 116, 118, 119 morphology, 175 motion, viii, ix, 57, 63, 69, 80, 85, 86, 87, 88, 89, 90, 91, 92, 94, 95, 96, 97, 98, 100, 101, 102, 105, 106, 107, 110, 111, 114, 121, 122, 129, 130, 131, 132, 133, 134, 135, 136, 137, 139, 141, 142, 144, 156, 217 motivation, 157 movement, 10, 13, 18, 20, 22, 24, 25, 27, 28, 29, 33, 62, 67, 68, 69, 71, 86, 121, 130, 131, 133, 150, 201 MRI, 81 multimedia, vii, 85 multiple interpretations, 228 muscles, viii, 114, 122, 129, 130, 133, 141
N navigation system, 82 neural network, 132, 230, 231 neural networks, 231 nodes, 94, 164 noise, 90, 121 nucleus, vii, 1
Index
O observations, viii, 86, 97 obstruction, 152 occlusion, ix, 145, 146, 148, 149, 150, 151, 152, 153, 155 oil, 3 olive oil, 55 operator, 138 optimism, 119 optimization, 132, 136, 137, 147, 165, 175, 226, 233 orientation, 70, 73, 150 originality, 168
P Pacific, 109 panoramic maps, 145, 146, 147, 153 parallel processing, 110 parallelism, 171 parameter, ix, 73, 74, 117, 122, 123, 125, 126, 130, 133, 134, 135, 136, 138, 139, 141, 143, 145, 146, 148, 155, 165, 167, 169, 186, 190, 193 parameter estimation, 141 parameter vectors, 138, 139, 141 parameters, viii, ix, 20, 34, 58, 70, 73, 74, 80, 86, 89, 118, 122, 123, 129, 130, 131, 132, 133, 135, 136, 137, 138, 139, 140, 141, 142, 162, 165, 166, 225 Pareto, 227 Pareto optimal, 227 partial differential equations, 192 particles, 12, 22 partition, 91, 92, 171, 183, 185, 196, 200 passive, 130, 133 path planning, 147 pattern recognition, 230 PCA, 89, 90, 104, 106, 135, 138 pelvis, 92 personality, 114, 115, 116, 119, 122 personality traits, 119 pessimism, 119 phonemes, 121, 122, 123 photographs, 4, 12, 15, 36, 39, 144 physical environment, 118, 119 physics, 27 planning, 156 plaque, 169 poor, 105 poor performance, 105 PowellSabin splines, ix, 177, 178, 182, 184, 186, 187, 191, 194, 197, 198, 199, 200, 201, 205, 206, 207, 208 power, 207
239 prediction, 86, 87, 89, 90, 91, 96, 97, 105, 107, 111, 193 pressure, 78, 79, 80 production, vii, 2, 3, 54, 55, 58, 165, 166 productivity, 230 program, 16, 18, 27, 54, 69, 231 programming, 64, 182 propagation, 162, 164, 193 protocol, 213, 233 prototype, ix, 63, 80, 145, 169 psychology, 232 pupil, 122
Q quality standards, vii, 57 quantization, 88, 109 quantum electrodynamics, 218
R radius, 165, 204 rain, 22, 210 range, ix, 24, 27, 29, 41, 54, 64, 95, 121, 141, 157, 158, 160, 177, 190, 221, 225 raytracing, 147 real time, 59, 62, 66, 70, 151 realism, viii, 15, 34, 39, 42, 57, 62, 63, 132 reality, vii, 1, 2, 54, 55, 59, 60, 61, 63, 65, 67, 69, 70, 71, 77, 78, 79, 80, 81, 82, 83, 159, 223 reason, vii, 1, 58, 114, 161, 162, 170 reasoning, 174, 228, 231 recall, 178 recognition, 70, 71, 123, 129, 143, 162 reconstruction, 2, 15, 16, 56, 88, 89, 90, 97, 99, 100, 101, 158, 168, 170, 171, 172, 175 recovery, vii, 1, 2, 12, 56 recreation, 13, 54 rectangular domains, 178 redundancy, 89 reference frame, 65, 69, 73, 77, 78, 79 reflection, 220, 233 region, viii, 92, 123, 125, 129, 130, 134, 136, 139, 141, 142, 143, 146, 148, 149, 150, 151, 152, 153, 155, 171, 172, 193 regulation, 54 regulations, 159 relationship, ix, 2, 101, 129, 130, 138, 211, 217, 220 relaxation, 162 reliability, 61, 221 relief, 9, 17 repair, 61 reproduction, 143, 170 Requirements, 214 residual error, 89, 97, 98, 142 residuals, 88
240
Index
resolution, 33, 41, 60, 62, 63, 64, 193, 205 resources, 59, 86 returns, 162, 166 rings, 25, 27, 169 robotics, 61 routines, 65 routing, 161, 165, 174
S sadness, 115, 117, 118 sampling, 108, 158 scaling, 201, 202, 222 schema, 221, 232 scientific computing, 160 search, 2, 214, 225, 226, 233 searching, 163 selecting, ix, 20, 66, 93, 145, 146 semantics, 147, 159, 216, 219, 224 sensation, 37 sensitivity, 227 sensors, 70, 80 sensory experience, 212 separation, 24, 122 shape, vii, ix, 57, 77, 78, 95, 114, 132, 133, 134, 142, 152, 158, 160, 161, 162, 165, 166, 168, 174, 177, 179, 184, 190, 200, 205, 230, 231 sharing, 220 simulation, viii, 23, 27, 57, 58, 63, 67, 68, 69, 70, 71, 72, 73, 77, 78, 79, 82, 83, 141, 160, 206, 223, 225 Singapore, 85, 129 skeleton, 161, 162 skills, 167 skin, viii, 114, 129, 130, 132, 133 smoothness, 90, 98, 99, 101, 158, 178, 206 snakes, 132 social development, vii, 1 software, vii, 20, 34, 54, 57, 63, 64, 68, 213, 217, 221 space, viii, ix, 23, 62, 64, 65, 66, 70, 73, 74, 80, 86, 88, 92, 95, 108, 113, 114, 116, 117, 118, 120, 121, 122, 123, 126, 136, 143, 145, 146, 148, 163, 164, 178, 179, 180, 181, 185, 192, 193, 194, 195, 197, 200, 202, 203, 213, 214, 217, 219, 220, 224, 226, 227, 232 Spain, 1, 3, 4, 55, 56 spatial information, 155 specialisation, 220 spectrum, 159 speech, viii, 113, 121, 123, 125, 126, 132, 144 speed, 12, 13, 18, 141, 142, 162, 169, 217, 218, 220 sports, 59 stability, 184, 188, 196, 202 stakeholders, 220, 223 standards, 223, 233 static geometry, 78
steel, 77 stimulus, 116, 117, 118, 121 stock, 6 storage, 12, 85, 193, 219 strategies, 68, 77, 78, 126, 225, 233 stress, 77, 78 structural modifications, 61 subgroups, 22, 23 surface area, 4 symbols, 98 symmetry, 171 synchronization, viii, 113, 125 synthesis, viii, 3, 39, 129, 131, 142, 144, 220, 233
T taxonomy, 118, 166, 175 teeth, 122 teleconferencing, 143 teleology, 210 television, 59 temperature, 62, 78, 79, 80 thermal analysis, 225 thoughts, 213, 232 threedimensional model, 147, 173 threedimensional space, 155 threshold, 89, 97, 98, 101, 102, 104, 152, 172 timing, 54 tissue, 133 tones, 41 topology, 86, 91, 94, 97, 101, 162, 165, 211 torus, 186, 187, 197 tracking, viii, 59, 61, 62, 63, 65, 129, 130, 131, 132, 134, 135, 136, 138, 139, 140, 141, 142, 143, 144 tradeoff, 90, 98, 193 tradition, 55 training, viii, 54, 60, 129, 130, 134, 135, 138, 139, 141, 142 transformation, 27, 33, 65, 73, 74, 75, 78, 79, 97, 133, 137, 148, 217, 218, 219, 225 transformation matrix, 73, 74, 97, 137, 148 transformations, x, 20, 69, 70, 75, 85, 90, 97, 98, 107, 166, 209, 210, 214, 220, 224, 225, 231 transition, 20, 54, 121, 126 transitions, 54 translation, 75, 78, 79, 133, 141 transmission, 25, 64, 85, 88, 99, 102 transmits, 25 transparency, 146, 150 transportation, 217 trees, 145, 164 triangulation, ix, 96, 147, 177, 178, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 205, 207 trust, 115, 117
Index
U UNESCO, 2 uniform, 151, 186, 208 United Kingdom, 2, 82, 173 updating, 12 urban areas, ix, 145, 146, 147, 151, 155
V validation, 126 variables, 164, 165, 211, 212, 214, 217, 225, 226 vector, 65, 73, 74, 75, 89, 104, 111, 117, 118, 120, 124, 125, 133, 134, 135, 136, 138, 139, 179, 184 velocity, 70, 79, 121 Venezuela, 56 vibration, 78 virtual actors, viii, 113 virtual reality, vii, 1, 54, 55, 59, 83, 159 vision, 17, 81, 146, 147, 156 visual system, 130
241 visualization, ix, 2, 35, 63, 67, 78, 80, 82, 121, 123, 148, 155, 157, 177, 190, 205 voicing, 121
W Wales, 173 wavelet, 89, 90 wear, 64, 80 websites, 2 wheat, 12, 22 wind, 4, 5, 6, 7 windmill, vii, 1, 3, 4, 5, 6, 7, 10, 11, 13, 14, 15, 16, 17, 18, 20, 28, 34, 37, 39, 41, 54 windows, 6, 17, 20, 22, 133 wood, 34, 64, 169 word recognition, 228 workers, 59 workload, 91
Z zinc, 4