edited by
C H Chen Electrical and Computer Engineering Department, University of Massachusetts Dartmouth, N. Dartmouth, MA, USA
L F Pau Ericsson, Sweden
P S P Wang College of Computer Science, Northeastern University, Boston, MA, USA
orld Scientific New Jersey. London Hong Kong
Published by World Scientific Publishing Co. Pte. Ltd. P 0 Box 128, Farrer Road, Singapore 912805
USA ofJice: Suite 1B, 1060 Main Street, River Edge, NJ 07661 UK oflice: 57 Shelton Street, Covent Garden, London WC2H 9HE
Library of Congress Cataloging-in-Publication Data Handbook of pattern recognition & computer vision /edited by C.H. Chen, L.F. Pau, P.S.P. Wang -- 2nd ed. p. cm. Includes bibliographical references and index. ISBN 9810230710 (alk. paper) 1. Pattern recognition systems. 2. Computer vision. I. Chen, C. 11. Pau, L.-F. (Louis Franpis), 1948H. (Chi-hau), 1937111. Wang, Patrick S.-P. (Shen-pai) IV. Title. Handbook of pattern recognition and computer vision. TK7882.P3H35 1999 006.4--dc21 98-51616 CIP
British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library
First published 1999 Reprinted 2001
Copyright 0 1999 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, orparts thereoJ may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher.
For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the Dublisher.
Printed i n Singapore by U t e P r i n t
PREFACE TO THE SECOND EDITION
The progress in pattern recognition and computer vision since the publication of the first edition in 1993 has been enormous. While the first edition is never out of date, t o incorporate new activities in the field, and to incorporate many valuable comments by the readers of the first edition, it is useful now to present a new edition. More than half of the pages of this volume are new, reporting a number of new activities. This edition also contains five parts: (1) Basic Methods in Pattern Recognition, (2) Basic Methods in Computer Vision, (3) Recognition Applications, (4) Inspection and Robotic Applications, and (5) Architectures and Technology, with a total of 34 chapters of original work on various topics. Part 1 starts with a chapter on cluster analysis which is followed by a chapter on statistical recognition and a chapter on syntactic pattern recognition. Until about twenty years ago, these three areas are what fundamental pattern recognition was about. The re-discovery of artificial neural networks in the mid-eighties has made it a n essential topic of modern pattern recognition. The fourth chapter addresses particularly neural network computing from the viewpoint of implementing pattern recognition algorithms. Additional basic pattern recognition methods are presented in the next two chapters on Gaussian Markov random field model and 3-D object pattern representation respectively. The first two chapters of Part 2 deal with texture analysis and segmentation. We agree with some readers’ comments that the first edition placed a lot emphasis on texture analysis and in fact Chapters 1.5; 2.1; 2.2; 4.2 of this edition are texture-based. This reflects that more progress has been made on this topic. The third chapter on color in computer vision has been fully rewritten from the first edition t o provide some latest progress. The next two chapters are on projective geometry and 3-D motion analysis. The last two chapters of Part 2 present aspects of 3-D shape representation and 3-D surface-based systems. Part 3 provides a comprehensive study of nine recognition applications including: nondestructive testing of materials, speech recognition, remote sensing application, fingerprint processing using an opto-electronic system, human chromosomes classification, document processing, biomedical signal recognition, geographic data analysis, and face recognition. Obviously it is not possible to present all application areas and the readers are kindly requested to refer t o the first edition and other publications for additional recognition applications. The first two chapters of Part 4 are in fish industry inspection and textured surface inspection with industrial objects. The next chapter on context related issues in image understanding is of concern t o all inspection and robotic applications. The chapter on computer vision in postal automation represents one aspect of robotic application. The last V
vi
Preface to the Second Edition
chapter on vision-based automatic vehicle guidance presents an area of emerging importance in intelligent transportation systems and again robotic vision can play an important role. Part 5 deals with several aspects of architecture and technological development. The first chapter on vision provides a tutorial discussion of technological issues in computer vision. The next chapter presents highlights of optical pattern recognition. Infrared imagery technology has been around for some time and its application has not been limited t o military and medical areas. The third chapter deals with this technology and its use in classification problems. Another major technological development is the video content analysis and retrieval which is discussed in the fourth chapter. The VLSI architecture for computing central moments for pattern recognition and image processing is presented in the final chapter with applications to breast cancer detection and road pavement distress detection. We understand that even with all chapters in this volume, there are still a lot of activities in pattern recognition and computer vision not presented here. Our view is that a handbook is not a dictionary and it is not desirable to cover all topics superficially. The in-depth treatment of a topic in each chapter can provide readers with a good understanding on the subject while collectively the chapters capture well all major developments in this field. The second edition is specially dedicated to the memory of Professor King Sun Fu. There are five chapters written by Professor Fu’s former students. During his career, Professor Fu has approached almost every area of recognition application in his time. By following his emphasis on applications, we have expanded the application part considerably. We are fortunate to have Professor Azriel Rosenfeld, who knew Professor Fu well, to write on the topic of vision with some speculations. Professor Rosenfeld’s words of wisdom will always be helpful to the research and education community. We are also fortunate to have Professor C. C. Li, who was Professor Fu’s classmate at the National Taiwan University, to prepare a memorial article. We thank both of them for their contributions. We like to take this opportunity to express our deep gratitude to all new and old contributors to this handbook series. This volume continues to represent the most comprehensive and most up to date handbook publication in pattern recognition and computer vision. The book will certainly help us to prepare for a new century of dynamic development in this field.
The co-editors June 1998
PREFACE
The area of pattern recognition and computer vision, after over 35 years of continued development, has now reached its maturity. The theories, techniques and algorithms are mostly well developed. There are a number of applications which are still being explored. New approaches motivated by applications and new computer architectures available are still being studied. Also the recently renewed and intensive efforts on neural networks have had great and positive impact on pattern recognition and computer vision development. Pattern recognition and computer vision will definitely play a very major role in advanced automation as we enter the 21st century. Amid all of these activities now going on, this new Handbook of Pattern Recognition and Computer Vision is much needed to cover what has been well developed in theory, techniques and algorithms and the major applications of pattern recognition and computer vision as well as the new hardwarelarchitecture aspects of computer vision and the related development in pattern recognition. The previous Handbook of Pattern Recognition and Image Processing, edited by T. Y. Young and the late K. S. Fu (Academic Press, 1986) was well received. The progress in pattern recognition and computer vision has been particularly significant in the recent past. We believe this new handbook that reflects more recent developments especially in computer vision will serve well the increasingly larger community of readers in the area. As students and friends of Prof. Fu, we remember well his vigorous efforts to broaden the frontiers of pattern recognition and computer vision in both theories and applications, to build it as an interdisciplinary area, and to lay down the foundation of intelligent and automated systems based on pattern recognition and computer vision. The book, in keeping up with his vision for the area, provides an extensive coverage of major research progress since the publication of Young and Fu’s book. The book is organized into five parts. Part 1 presents a thorough coverage of the basic methods in pattern recognition including clustering techniques, statistical pattern recognition, neural network computing, feature selection, and syntactic, structural and grammatical pattern recognition. Part 2 presents comprehensively the basic methods in computer vision including texture image analysis and model based segmentation, color and geometrical tools, 3-D motion analysis, matheniatical morphology, and parallel thinning algorithms. Part 3 presents several major pattern recognition applications in nondestructive evaluation, geophysical signal interpretation, economics and business, underwater signals, character recognition and document understanding, biomedical image recognition and medical image vii
viii
Preface
understanding. Part 4 focuses on unique applications in inspection and robotics with topics on computer vision in the food processing industry, context modeling and position estimation for robots, and related issues. Part 5, on the other hand, examines the broader system aspects, including designing computer vision systems, optical pattern recognition, spatial knowledge representation, neural network architecture for image segmentation, architectures for computer vision and image information systems. More than 85 per cent of the chapters are original and unpublished work while the remaining reprint chapters provide complementary coverage. There is no doubt that a single volume handbook like this cannot examine every aspect of pattern recognition and computer vision, nor can it present the contributions of all leading researchers. However, we believe the book has captured both the scope and depth of progress in this highly dynamic and multidisciplinary area. In preparing the book, we are most fortunate to bring together all contributors who are among the leaders in the area. We would like t o take this opportunity to express our deep gratitude to their unselfish and timely efforts to share their expertise with the readers. We also like to thank Dr. K. K. Phua and Ms. Jennifer Gan of World Scientific Publishing for their help and encouragement throughout the preparation of this volume. C. H. Chen L. F. Pau P. S. P. Wang September 1992
FOREWORD VISION: SOME SPECULATIONS
AZRIEL ROSENFELD C e n t e r for A u t o m a t i o n Research University of Maryland, College Park, M D 20742-3275, USA
1. Introduction The purpose of vision is t o extract useful information about the world from images. Computer vision attempts t o do this by computer analysis of digitized images. Efforts along these lines have been under way for over 40 years; but even the domains first studied in the 1950s (which included handwriting, photomicrographs of biological specimens, and aerial photographs of built-up areas) still present many unsolved problems. This very brief note presents some comments, many of them intended t o be provocative, about the reasons for the slow progress in the field and the ways in which successes are likely to be achieved. 2. Vision and Mathematics
As an hommage t o Wigner [l],this section might have been entitled “On the unsurprising ineffectiveness of mathematics in the visual sciences”. Most vision problems, even those that were first tackled in the 1950s, are mathematically ill-defined (reading handwritten words, counting cells, recognizing buildings). Real-world visual domains do not satisfy simple mathematical (even probabilistic) models. Even when such models are assumed, problems that involve inferring information about a scene from images are often mathematically ill-posed or computationally intractable; but a more serious difficulty is that the models themselves are unrealistic, and are likely to remain so for a long time to come [2]. On a more positive note, mathematical and statistical tools do have their uses in formulating vision problems; in particular, they provide methods of describing image formation processes and image analysis algorithms. There also exist domains (typically involving machine-made scenes: printed documents, integrated circuits, mechanical arts, all with controlled lighting) that can be mathematically modelled quite accurately (as long as the scene is not too dirty!). In these sorts of domains, vision systems can be designed t o perform quite successfully, provided they take the model seriously and bring adequate computer power to bear on the task. ix
x
Foreword
3. Vision and Biology In this section we put mathematics aside and consider what might be called the “multilegged existence theorems” for vision [3]. Animals use vision quite effectively in the real world. Apparently, simple algorithms can be quite useful for extracting useful information about natural environments (for example [4],insects rarely collide with the underbrush even when flying through it at many body lengths per second); though an organism may have trouble coping if its environment changes significantly (for example, a frog surrounded by unmoving flies will probably starve to death [ 5 ] ) . Simple computer vision algorithms can in fact be designed, sometimes with the aid of learning techniques, that will usually perform usefully in real-world environments; but they may fail disastrously if they encounter unusual situations. For simple organisms, nature overcomes their individual limitations by providing large populations in which there are variations among the individuals. Thus even if many individuals fail, others survive (or learn to survive) and may pass their successful characteristics on to their descendants. (It has been suggested, halfseriously, that this might provide a biological justification for the hundreds of “yet another ...” computer vision algorithms in the literature; but note that the analogy is valid only if these algorithms are compared on a variety of tasks, so their potential advantages can be discovered!) Only a few pieces of research are truly innovative and can be regarded as major mutations; as in nature, such breakthroughs are sometimes lethal, but they may sometimes succeed in producing new successful species of vision systems. This “biological” approach could be used to discover useful computer vision algorithms through extensive experiments with real data; but it does not seem practical to develop robot vision systems in this way, since the cost of the failures would probably be unacceptable. A possible strategy for developing useful vision algorithms, without incurring many costly failures, might be t o initially use large sets of recorded real-world data t o design and test the algorithms. Higher-level organisms are more flexible in their uses of vision. Their flexibility may be based on an ability to evaluate and combine the information extracted from the images in multiple visual “areas” or “pathways”. This suggests that computer vision systems could benefit from the deliberate use of multiple techniques an approach that has been generally avoided in the past because of its computational cost. Another possible basis for the successful visual performance(s) of organisms is that they make use of redundant visual data. Computer vision systems often t,ry to reduce computational cost by analyzing only single (sets of) frames, and (for dynamic scenes) using results obtained from earlier frames as predictors t o simplify the analysis of later frames. Biological organisms, on the other hand, process all the data that their visual systems provide (though results obtained from earlier processing may serve to direct attention to subsets of the results obtained from later processing). The availability of redundant data allows an organism to discover
Foreword
xi
processing errors, since they give rise to results that are not persistent. Computer vision systems are just reaching; the levels of processing power that will allow them to handle, in real time, amounts of input data comparable to those handled by biological visual systems, and t o apply multiple processing techniques to the data. As these levels are reached, vision system performance may significantly improve. 4. Concluding Remarks Over the past four decades, the performance of computer vision systems has largely kept pace with available computer power. Many domains remain intractable from a theoretical standpoint, but systems that take advantage of data and algorithm redundancy may eventually achieve performances comparable t o those of biological organisms, which serve as living demonstrations of the effectiveness of vision in the real world. References [l] E. Wigner, The unreasonable effectiveness of mathematics in the natural sciences, Comm. Pure Appl. Math. 13 (1960). [2] A. Rosenfeld, Some thoughts about image modeling, In K. V. Mardia and G. K. Kanji, (eds.), Statistics and Images 1 (Advances in Applied Statistics, Supplement to J. Appl. Stat. 20 (5/6), Carfax Pub. Co., Abingdon, Oxfordshire, UK, 1993) 19-22. [3] R. A. Kirsch (personal communication, June 1967) used the phrase “two-legged existence theorem” as an argument for the feasibility of computer vision (or, as we were calling it then, “pictorial pattern recognition”). He probably had in mind only featherless bipeds, but an even earlier example (an existence proof for the air-to-ground ATR problem) is Skinner’s 1945 demonstration that pigeons could be trained to serve as effective bombsights. [4] J. A. Albus, personal communication, about 1990. [5] J. Y. Lettvin, H. R. Maturana, W. S. McCulloch and W. H. Pitts, What the frog’s eye tells the frog’s brain, Proc. Ire 47 (1959) 1940-1957.
A MEMORIAL TO T H E LATE PROFESSOR KING-SUN FU
Twelve years have gone by since the passing away of Professor King-Sun Fu, an eminent scholar and a pioneer in the field of pattern recognition, computer vision, and machine intelligence. This volume is dedicated to his memory by his friends and students as a testimonial to his profound and enduring influence. I am honored to be asked to write a memorial article. Here I will give a biography* of King-Sun as well as a short addendum t o the bibliography of his published works compiled in 1985. King-Sun was born on October 2, 1930, in Nanking, China, the second son of his parents, General and Mrs. Tzao-Jen Fu. The Fu family’s native city, however, is Hangzhou, Zhejiang. He pursued his middle school education at the Chinese Air Force Youth Preparatory School. After his father died in 1949, his mother took him and two younger brothers to Taiwan, where he matriculated in the National Taiwan University in the Fall of 1949. He was one of the top students in the Electrical Engineering Department. Not only did he excel in mathematics and engineering subjects, but he also cultivated interests in classical music and literature. Perhaps the latter pursuit laid the foundation for his own prolific writing in serial journals and scientific books. His writing was distinct: clearly expressive and succinct. The demands of academia did not serve to stifle his personal exuberance and vitality in life, however, and, in finding a balance to his studies, King-Sun was also active on the basketball and volleyball teams in the school. He was indeed a versatile student on the campus. After graduating from the National Taiwan University with a B.S.E.E. degree in 1953 and completing one year of ROTC training, King-Sun received a graduate assistantship from the Electrical Engineering Department of the University of Toronto, Canada, and went there in September 1954 to start his graduate study. He wrote his master’s degree thesis on dynamic analysis of large electric machines and received his M.A.Sc. degree in the summer of 1955. In September 1955 he transferred to the University of Illinois, Urbana, IL, USA, for his doctoral study in Electrical Engineering, and completed his Ph.D. dissertation in network theory on “An Approximation Method for Both Magnitude and Phase by Rational Functions” in February 1959 under Professor M. E. Van Valkenburg. During his three and a half years at the University of Illinois, King-Sun became immensely interested in statistical methods, information theory, abstract algebra, and modern analysis. He was a top student in Professor J . L. Doob’s course *Portions of the biography are reprinted, with permission, from IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. PAMI-8, No. 3, pp. 291-294, May 1996. @ 1986 IEEE.
Xiii
xiv
A Memorial to the Late Professor King-Sun Fu
on stochastic processes which was intended for mathematics graduate students only. This extraordinary background paved the way for his later research in sequential methods of statistical pattern recognition and machine learning. On the Urbana campus, King-Sun fell in love with Miss Viola Ou, then a graduate student in library science. They were married in Urbana, Illinois, on April 7, 1958. After receiving his Ph.D. degree from the University of Illinois in 1959, King-Sun worked for a year and a half as a research engineer at Boeing Airplane Company, Seattle, Washington, from February 1959 to August 1960. He also taught as a special lecturer at the Seattle University during the Spring Semester of 1960. In September 1960 he accepted a faculty position a t Purdue University, West Lafayette, Indiana, as an assistant professor in the School of Electrical Engineering. The following semester he was selected by Purdue to be a visiting scientist with the Research Laboratory of Electronics at Massachusetts Institute of Technology from February to June 1961. During that summer he was with the IBM Thomas J. Watson Research Center at Yorktown Heights, NY. After he returned to Purdue University in September 1961, he began to pursue his research in pattern recognition and machine intelligence - a field in which he played a prominent role in its development during the next quarter of a century. He became a n associate professor in September 1963 and was promoted to the rank of professor of electrical engineering in September 1966. In 1967 he was a visiting professor of electrical engineering and computer science at the University of California, Berkeley. He was the assistant head for research a t the School of Electrical Engineering a t Purdue from 1969 to 1972. In 1972 he was awarded a prestigious Guggenheim fellowship and was a visiting professor of electrical engineering a t both Stanford University and the University of California, Berkeley. After returning to Purdue, he established the Advanced Automation Research Laboratory in the School. He was named the Goss Distinguished Professor of Engineering at Purdue University in 1975. During the Fall of 1984 he, along with other colleagues at Purdue, initiated the highly innovative program of research in intelligent manufacturing. This program resulted in the startup of the National Science Foundation Engineering Research Center for Intelligent Manufacturing Systems during early 1985. King-Sun was the founding director of the Center. At the peak of his career, his sudden death in April 1985 was a tremendous loss to our scientific and engineering community. During the earlier years he first focused his study on statistical pattern recognition and learning systems. From 1961 t o 1970 he and his students developed sequential methods for feature selection and pattern recognition, non-parametric procedures for pattern classification, a stochastic approximation approach to learning control systems, and stochastic and learning automata. His first research monograph entitled “Sequential Methods in Pattern Recognition and Machine Learning” was published in 1968. By the late 1960’s he began his unique research on syntactic pattern recognition, which was introduced by the earlier efforts of Murray Eden, R. Narasimhan, R. A. Kirsch, Robert S. Ledley, and Alan Shaw. King-Sun initiated and launched in-depth studies on stochastic context-free
A Memorial to the Late Professor King-Sun Fu
xv
programmed languages and stochastic syntax analysis for pattern recognition and image analysis. His book, “Syntactic Methods in Pattern Recognition”, was published in 1974. In the ensuing years he and his students made the greatest and foremost impact on syntactic pattern recognition research. His school developed fundamental methodologies of stochastic error-correcting syntax analysis, error-correcting parsers for formal languages and, in particular, for attributed and stochastic tree grammars, and error-correcting isomorphisms of attributed relational graphs for pattern recognition. The syntactic methods for texture analysis, shape recognition, and image modeling were introduced in the late 1970’s, and the three-dimensional plex grammar in 1984. Attributed grammars were developed from the viewpoint of combining syntactic and statistical pattern recognition. In the meantime, the contextual information was also introduced into statistical pattern recognition. The unification of both syntactic and statistical approaches was always in his thoughts. Inference procedures of context-free programmed grammars, multi-dimensional grammars, transition network grammars, and stochastic tree grammars were developed one after another in the late 1970’s through early 1980’s. It is probably appropriate to say that all these constitute what we may call Fu’s theory of syntactic pattern recognition. His treatise, “Syntactic Pattern Recognition and Applications”, published in 1982, made this subject material more easily understandable to researchers and practitioners in various disciplines. King-Sun and his colleagues also made important contributions to pattern recognition applications. His work on pattern classification of remotely sensed agriculture data (1969) and earth resources (1976) is considered classic in the field. During the mid 1970’s through the early 1980’s, his biomedical pattern recognition research extended to chest radiographic image analysis, automatic recognition of irradiated chromosomes, nucleated blood cell classification, and Pap smear and cervical cell image analysis and classification. The Moayer-Fu paper on fingerprint pattern recognition based on the syntactic approach received the 1976 outstanding paper award of the IEEE Transactions on Computers. His work on seismic signal discrimination and bright spots detection appeared in 1982 and 1985. His research on industrial automatic inspection and computer vision included IC chip inspection (1980), metal surface inspection (1984), and inspection of industrial assemblies (1985). An expert system was developed by his group for the assessment of structure damages caused by earthquakes (1983). Since the late 1970’she envisioned the importance of integrated and special computer architectures and parallel algorithms for pattern recognition, image processing, and database management. This led t o his works in 1980’son parallel parsing of tree languages, query languages for image database systems, and VLSI implementation of parallel parsing algorithm and hierarchical scene matching. In the meantime, his research on three-dimensional object representation and shape description, orientation estimation, overlapping workpiece identification, knowledge organization, and robotic vision for path planning laid the foundation for the establishment of
xvi
A Memorial to the Late Professor King-Sun Fu
the Engineering Research Center on Intelligent Manufacturing Systems by Purdue University and the National Science Foundation in 1985. As mentioned earlier, King-Sun was the chief architect and the first director of the research center. He wrote six books, edited or co-edited eighteen books, authored or co-authored forty-four book chapters, and one hundred sixty-two serial journal papers. In addition, he authored and co-authored two-hundred forty-eight conference papers. Seventy-two Ph.D. dissertations were completed under his supervision. His activities in professional societies started in 1965-67 as the chairman of the Institute of Electrical and Electronic Engineers (IEEE) Discrete Systems Committee and the chairman of the Fifth Symposium on Discrete Adaptive Processes. Under his leadership he organized and served as the first chairman (1967-69) of the IEEE Automatic Control Group’s Learning and Adaptive Systems and Pattern Recognition Technical Committee. He was on the official American delegation to the International Conference on Artificial Intelligence, Moscow, USSR, in 1967, and an official American delegate to the 1969 International Federation of Automatic Control (IFAC) International Congress held in Warsaw, Poland. He served on the administrative committee of the IEEE Automatic Control Group (1969-71) and later of the IEEE Control System Society (1974-76), was the chairman of the 1969 IEEE International Convention, a director (for IEEE) of the American Automatic Control Council in 1972, and the general chairman of the 1977 IEEE Conference on Decision and Control. He took part in the IEEE Systems, Man, and Cybernetics Society activities, beginning in 1969 when he served as the chairman of the Adaptive Systems Technical Committee of its predecessor, the IEEE Systems Science and Cybernetics Group, and was on the administrative committee of the Group (1970-72). He became the Cybernetics Technical Committee chairman (1972-76) and then the Society’s Vice President for Technical Committees (1978-79). He was an associate editor of IEEE Transactions on Systems, Man, and Cybernetics (1969-1985). In order to provide an international forum to promote advances in pattern recognition, he and the contemporary leaders in the field organized the first International Conference on Pattern Recognition in Washington, DC, in 1973, for which he served as chairman. The biannual conferences evolved into the formation of the International Association for Pattern Recognition (IAPR) by 1976. He was elected to be its president for 1976-78, a member of its executive committee (1976-80), chairman of its long range planning committee (1979-81), and a member of its governing board (1976-85). (In memory of his distinctive contributions, the IAPR has established since 1986 the “K. S. Fu Award, to be given to a distinguished contributor in the field once every two years.) In the meantime, he reorganized the Pattern Recognition Committee into the Machine Intelligence and Pattern Analysis Technical Committee (later renamed as the Pattern Analysis and Machine Intelligence Technical Committee) of the IEEE Computer Society and was its first chairman (1974-77). He was an associate editor of IEEE Transactions on Computers during 1977-78. His initiative led to the founding of the IEEE Transactions on Pattern
A Memorial to the Late Professor King-Sun Fu xvii
Analysis and Machine Intelligence, and he served as its first editor-in-chief (1978-81) as well as a member of the editorial committee (1981-85). In addition, he served on editorial boards (editor, associate editor, and advisory board) of many other scientific journals. These include Pattern Recognition (associate editor, 1971-85); International Journal on Information Sciences (associate editor, 1970-82; editor 1982-85); Journal of Cybernetics of the American Society of Cybernetics (editorial board, 1970-85); International Journal on Computer and Information Sciences (advisory editor, 1971-85); Journal of Information Processing (editorial advisory committee, 1978-81); Journal of Analytical and Quantitative Cytology (editor, 1978-85); International Journal of Fuzzy Sets and Systems (advisory editor, 1979-1985); International Journal of Cybernetics and Systems (advisory board, 1980-85); Computer Vision, Graphics and Image Processing (associate editor, 1981-85); Pattern Recognition Letters (advisory editor, 1982-85); IEEE Transactions on Geoscience and Remote Sensing (associate editor, 1984-85); IEEE Computer (editorial board, 1983-85); and Journal of Parallel and Distributed Computing (editorial board, 1984-85). He chaired and co-chaired the Engineering Foundation Conference on Pattern Information Processing in 1972, on Algorithms for Image Processing in 1976, and on Algorithms for Image and Scene Analysis in 1978. He was Program Chairman of the 1975 IEEE-ACM Conference on Computer Graphics, Pattern Recognition and Data Structure, of the 1978 IEEE Computer Society Conference on Pattern Recognition and Image Processing, and of the 1979 IEEE Computer Society COMPSAC Conference. He chaired the 1980 IEEE Picture Data Description and Management Workshop, and initiated and chaired the IEEE Computer Society Workshop on Computer Architecture for Pattern Analysis and Image Database Management in 1981 and 1983. He was the general chairman of the 1984 IEEE Workshop on Language for Automation; Honorary Chairman of the 1984 IEEE Workshop on Visual Languages at Hiroshima, Japan; and general chairman of the 1985 IEEE International Conference on Robotics and Automation. His leadership was well recognized for organizing many international joint seminars and workshops. He served as the coordinator of the NSF supported U.S.-Japan Seminar on Learning Processes in Control Systems at Nagoya, Japan, in 1970 and the Second U.S.-Japan Seminar on Learning Control and Intelligent Control at Gainesville, Florida, in 1973; as the vice-coordinator of the US-Japan Seminar on Fuzzy Sets and Their Applications at Berkeley, California, in 1974; as a co-director of the NATO Advanced Study Institute on Pattern Recognition and Applications in 1975; as the co-chairman of the Dahlem Konferenzen on Biomedical Pattern Recognition and Image Processing at Berlin, West Germany, in 1979; and the coordinator of the NSF sponsored U.S.-France Seminar on the Applications of Pattern Recognition and Machine Intelligence to Automatic Testing at Alexandria, Virginia, in 1983. As a guest editor, he helped put together the special issue on “Feature Extractions and Selection in Pattern Recognition” of IEEE Transactions on Computers, September 1971; the special issue on “Syntactic Pattern Recognition” of Pattern Recognition, Part One,
xviii
A Memorial to the Late Professor King-Sun Fu
November 1971, and Part Two, January 1972; special issue on “Pattern Recognition” of IEEE Computer, May 1976; and on “Robotics and Automation” of IEEE Computer, December 1982. He was on the IEEE Computer Society Governing Board (1978-81), and served as the Society’s Vice President for Publications and a member of the Executive Committee (1982-83) and Fellow Committee (1972-76, 1984-85). He served as the president of the Chinese Language Computer Society (1983-85). He was the Vice President (1984-85) and President-elect of the then newly formed IEEE Robotics and Automation Council. He was on the IEEE Fellow Committee (1977-79), IEEE TAB Awards and Recognition Committee (1979-78), American Federation of Information Processing Societies (AFIPS) Harry Goode Memorial Award Committee chairman (1982-85), American Society of Engineering Education (ASEE) Award Committee (1983-85), and IEEE Award Board, Education Medal Committee (1983-85). King-Sun was literally showered with honor in recognition of his monumental research contributions and contributions to the profession. He was elected a Fellow of the Institute of Electrical and Electronic Engineers in 1971. He was elected a member of the National Academy of Engineering in 1976 and a member of the Academia Sinica in 1978. He served on the National Science Foundation’s Advanced Automation Panel in 1973, Automation Research Council for 1972-78, and Committee on Cytology Automation of the National Institutes of Health, 197881. Among the many awards which he received are the Herbert N. McCoy Award in 1976 for Contributions to Science; the American Society of Engineering Education Senior Research Award in 1981 for outstanding loyalty and contributions as a pioneer in the contemporary engineering disciplines of pattern recognition, image processing and machine intelligence; IEEE Education Medal in 1982 for contributions to engineering education through inspired teaching and research in computer engineering, system theory and pattern recognition; American Federation of Information Processing Societies Harry Goode Memorial Award in 1982 in recognition of his contributions in pattern recognition and its applications and his leadership in education in information processing; Chinese Institute of Engineers - USA (CIE-USA) Achievement Award in 1983 for leadership in engineering education and contribution to pattern recognition; and the IEEE Centennial Medal in 1984. King-Sun helped Taiwan, China, with his scientific advice in various ways. Over a period of fifteen years (1970-1985), he gave invited lectures there almost every year. He was the Program Chairman of the Academia Sincia International Computer Symposium at Taipei in 1978. He helped found the Institute of Information Science, Academia Sinica and was instrumental in establishing the Microelectronics and Information Science and Technology Research Center at the National Chiao Tung University, Taiwan, in 1984. He nurtured a number of young scholars who have become the principal researchers and engineers for the vital development of computer engineering and information science in Taiwan. Likewise he educated a number of scholars from mainland China during 1979 through 1985. He was invited
A Memorial to the Late Professor King-Sun Fu xix
to give lectures to the Institute of Automation, Chinese Academy of Sciences in 1979, and was honored as a Distinguished Visiting Professor of Beijing University, an Honorary Professor of Tainghua University, Beijing, and an Honorary Professor of Fudan University, Shanghai. King-Sun took great pride in his two sons, Francis and Thomas, and one daughter, June. When they were young he always spent his leisure time playing ball or other sports with them. Together with Mrs. Fu, he provided their children with the best education in the home and at school. They are all grown up now: Francis is a computer engineer; Thomas, an oceanographer; and June, a biochemist. They have their own accomplishments in their respective professions. In spite of his overwhelming achievements, King-Sun was a modest man with great sensitivity. Considerate and generous to his friends and students, he exemplified the notion of greatness both professionally and in his personal relationships with others. Standing alongside his outstanding contributions to the scientific world, King-Sun’s great wisdom and human warmth will always be remembered. Ching-Chung Li Department of Electrical Engineering University of Pittsburgh Pittsburgh, PA 15261, USA January 1997
Addendum to “A Bibliography of Published Works of the Late Professor King-Sun Fu,” I E E E Trans. Pattern Analysis and Machine Intelligence, Vol. PAMI-8, No. 3, pp. 295-300, May 1986. At the time of compiling the bibliography, eight items were missed (seven of which appeared later in 1986 and 1987). To make his bibliography complete, these are listed below.
Books: K. S. Fu, T-X Cai and G-Y Xu, Artificial Intelligence and Its Applications, (in Chinese) (Beijing: Tsinghua University Press, 1987).
Book Chapters: K. S. Fu, M. Ishizuka and J. T. P. Yao, “Application of Fuzzy Sets in Earthquake Engineering,” in Fuzzy Set and Possibility Theory, R.R. Yager, Ed. (New York: Pergamon Press, 1982) 504-523. Serial Journal Articles E. K. Wong and K. S. Fu, “A Hierarchical Orthogonal Space Approach t o ThreeDimensional Path Planning,” I E E E Journal of Robotics and Automation, Vol. RA-2, March 1986, 42-53.
xx A Memorial to the Late Professor King-Sun Fu
W. C. Lin and K. S. Fu, “A Syntactic Approach to Three-Dimensional Object Recognition,” IEEE Trans. Syst., Man, Cybern., SMC-16, May/June 1986, 405-422.
H-S Don and K. S. Fu, “A Parallel Algorithm for Stochastic Image Segmentation,” IEEE Trans. Pattern Anal. Machine Intell., PAMI-8, Sept. 1986, 594-603.
M. A. Eshera and K. S. Fu, ‘&AnImage Understanding System Using Attributed Symbolic Representation in Inexact Graph-Matching,” IEEE Trans. Pattern Anal. Machine Intell., PAMI-8, Sept. 1986, 604-618. K-Y Huang and K. S. Fu, “Decision-Theoretic Approach for Classification of Ricker Wavelets and Detection of Seismic Anomalies,” IEEE Trans. Geoscience and Remote Sensing, GE-25, March 1987, 118-123. S. Basu and K. S. Fu, “Image Segmentation by Syntactic Method,” Pattern Recognition, 20, No. 1 (1987), 33-44.
CONTENTS
Preface to the Second Edition
V
Preface
vii
Foreword
ix
A Memorial to the Late Professor King-Sun FU
xiii
Contents
xxi
PART 1. BASIC METHODS IN PATTERN RECOGNITION 1.1 Cluster Analysis and Related Issues Richard C. Dubes
3 33
1.2 Statistical Pattern Recognition
Keinosuke Fukunaga 61
1.3 Syntactic Pattern Recognition
Kou- Yuan Huang 1.4 Neural Net Computing for Pattern Recognition
105
Yoh-Han Pao 1.5 On Multiresolution Wavelet Analysis using Gaussian Markov Random
Field Models C. H. Chen and G. G. Lee
143
1.6 A Formal Parallel Model for Three-Dimensional Object Pattern Representat ion
183
P. S. P. Wang PART 2. BASIC METHODS IN COMPUTER VISION 2.1 Texture Analysis
207
Mihran Tuceryan and Anil K. Jain 2.2 Model-Based Texture Segmentation and Classification
249
R. Chellappa, R. L. Kashyap and B. S. Manjunath 2.3 Color in Computer Vision: Recent Progress
283
Glenn Healey and Quang- Tuan Luong 2.4 Projective Geometry and Computer Vision
Roger Mohr xxi
313
xxii
Contents
2.5 3-D Motion Analysis from Image Sequences using Point Correspondences
339
John. J. Weng and Thomas S. Huang 2.6 Signal-to-Symbol Mapping for Laser Rangefinders Kenong W u and Martin D. Levine
387
2.7 3-D Vision of Dynamic Objects S.-Y. Lu and Chandra Kambhamettu
425
PART 3. RECOGNITION APPLICATIONS 3.1 Pattern Recognition in Nondestructive Evaluation of Materials
455
C. H. Chen 3.2 Discriminative Training - Recent Progress in Speech Recognition
473
Shigeru Katagiri and Erik Mcdermott 3.3 Statistical and Neural Network Pattern Recognition Methods for Remote Sensing Applications
507
Jdn Atli Benediktsson 3.4 Multi-Sensory Opto-Electronic Feature Extraction Neural Associative Retriever H.-K. Liu, Y.-H. Jan, Neville I. Marzwell and Shaomin Zhou 3.5 Classification of Human Chromosomes - A Study of Correlated Behavior in Majority Vote
535
567
Louisa Lam and Ching Y. Suen 3.6 Document Analysis and Recognition by Computers Yuan Y. Tang, M. Cheriet, Jiming Liu, J. N . Said
579
and Ching Y. Suen 3.7 Pattern Recognition and Visualization of Sparsely Sampled Biomedical Signals
613
Ching-Chung Li, T. P. Wang and A. H. Vagnucci, M.D. 3.8 Pattern Recognition and Computer Vision for Geographic Data Analysis
625
F. Cavayas and Y. Baudouin 3.9 Face Recognition Technology
667
Martin Lades PART 4. INSPECTION AND ROBOTIC APPLICATIONS 4.1 Computer Vision in Food Handling and Sorting Hordur Arnarson and Magnth Asmundsson
687
Contents
4.2 Approaches to Texture-Based Classification, Segmentation and Surface Inspection Matti Pietakainen, Timo Ojala and Olli Silven 4.3 Context Related Issues in Image Understanding L. F. Pau 4.4 Position Estimation Techniques for An Autonomous Mobile Robot-A Review Raj Talluri and J. K. Aggarwal 4.5
Computer Vision in Postal Automation G. Garibotto and C. Scagliola
4.6 Vision-Based Automatic Road Vehicle Guidance Dieter Koller, Quang-Tuan Luong, Joseph Weber and Jitendra Malilc
xxiii
71 1 737
765 797 817
PART 5. ARCHITECTURE AND TECHNOLOGY 5.1 Vision Engineering: Designing Computer Vision Systems Rama Chellappa and Azriel Rosenfeld
857
5.2 Optical Pattern Recognition for Computer Vision David Casasent
869
5.3 Infra-Red Thermography: Techniques and Applications M. J . Varga and P. G. Duclcsbury
891
5.4 Viewer-Centered Representations in Object Recognition: A Computational Approach Ronen Basri 5.5 Video Content Analysis and Retrieval Hongjiang Zhang 5.6 VLSI Architectures for Moments and Their Applications to Pattern Recognit ion Heng-da Cheng, Chen-Yuan Wu and Jaguang Li
Index
925 945
979
1003
PART 1
BASIC METHODS IN PATTERN RECOGNITION
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 3-32 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
I CHAPTER 1.1I CLUSTER ANALYSIS AND RELATED ISSUES
RICHARD C. DUBES Department of Computer Science, Michigan State University, East Lansing, M I 48824-1087, U S A This chapter explains how cluster analysis organizes information in applications such as Computer Vision and Pattern Recognition. Information is represented as points in multidimensional feature spaces where each coordinate represents a measurement. Some tools from exploratory data analysis are discussed, with an emphasis on linear projections derived from the covariance matrix. Two types of clustering are reviewed - hierarchical and partitional. Hierarchical clustering leads to nested partitions of the data. SAHN algorithms for hierarchical clustering are defined and some of the common characteristics are explained. Partitional clustering arranges data in separate clusters, as with the K-Means algorithm. The chapter ends with a discussion of validation that centers on external and internal tests of validity and tests for the number of clusters. A bibliography is provided for further reading. Keywords: Proximity, exploratory data analysis, projection, hierarchies, dendrograms, K-means, cluster validity, algorithms.
1. Introduction
Organizing information is an essential part of any learning task. Cluster analysis is the formal study of methods and algorithms for objectively organizing numerical data. One finds cluster analysis in the literature of almost all disciplines, including engineering, statistics, psychology, sociology, biology, astronomy, business, medicine, archeology, psychiatry, geography, anthropology, economics, and computer science, to name a few. No single definition of “cluster” is universally accepted. Cluster analysis includes the process of “looking” at data, known as exploratory data analysis, which is a tool for igniting creativity and suggesting alternative models for the data. This chapter views cluster analysis as the initial step in organizing numerical data so as to abstract the essence of the data and describe the data as simply as possible. The discussion is informal and omits mathematical proofs. Computational issues are mentioned only briefly. Cluster analysis is sometimes called “unsupervised learning” because only actual observations affect the data organization. By contrast, pattern recognition uses a priori labels to “learn” the parameters of models for the categories, or pattern classes, present in the data. A pattern recognition algorithm seeks to define a good 3
4
R. C. Dubes
decision rule for labeling patterns of unknown origin, based on information gleaned from labeled patterns. The algorithms of cluster analysis make no decisions but fit various structures, such as partitions and hierarchies, to the data. Although the literature in several fields of application carry papers on clustering, the only journal exclusively devoted to the methodology of clustering is the Journal of CZasszfication, published by the Classification Society of North America since 1984. Some general books on the topic are [1,2,3,4,5]. One application, image segmentation, will help explain the context of this chapter. Each pixel, or each small sub-image, is characterized by a set of numbers [6]. Candidates for the numbers are co-occurrence features?gray-level intensities, measures of fractal dimension? estimates of Markov random field-parameters?and other indices popular in the computer vision community. Cluster analysis labels each pixel or sub-image so that regions from the same underlying class, such as land-use category in remote sensing, have the same label and regions from different classes have different labels. The organization is done in the feature space, in which each axis represents one of the measurements. One must develop faith in the clustering algorithm and must formally validate the results. The cluster labels are then transferred to the image for interpretation. This chapter will concentrate on the process of assigning the cluster labels, and not on the choice of features. Figure 1 is an overview of the most important aspects of a typical cluster analysis. Once data have been gathered and some type of exploratory data analysis has been applied to evaluate the data representation, one can apply clustering tendency algorithms to ensure that the data are not random. This avoids the embarrassment and futility of imposing sophisticated procedures for analyzing data that contain no clusters [4]. This chapter covers two types of clustering: hierarchical clustering creates a complete hierarchy, or nested sequence of partitions; partitional clustering creates one partition of the data. Omitted due to lack of space are treatments of fuzzy clustering [7,8], conceptual clustering [9,10], and any mention of neural nets [ l l ] . The validation step, in which one applies statistical tests to ensure that the structure recovered from the clustering algorithm is “real” in some sense, is the most difficult step of the entire process. The interpretation of the results requires experience and interaction with the expert in the field of application. The entire process, or any part of it, may need to be repeated until one is satisfied with the result. All this effort should reveal the underlying structure of the data so that sharper and more definitive studies can be planned. 2. Data
The procedures and algorithms of cluster analysis are geared towards the type and scale of the data, so this section begins by reviewing some basic definitions about data in Section 2.1 and about normalization in Section 2.2. The most important
1.1 Cluster Analysis and Related Issues 5
I
Xrtrix
Clurtrring
Birrrrchicil
Prrtitionrl Clurt-ring
Clurtrring
I A
Partition
Vilidrtion
Intrrprrtrtion
Fig. 1. Methodology of cluster analysis.
characteristic of a set of data is its dimensionality, which is briefly explained in Section 2.3. 2.1. Representing Numerical Information
2.1.1. Scale and type Data occur in several types and scales. The simplest unit of data is a number. Vectors and matrices are built from numbers, but all numbers should be on the same scale and have the same type. The scale of a number refers to its relative significance. Usually recognized are nominal, ordinal, interval, and ratio scales. A number on a nominal scale is simply a numerical tag, such as “1” for “Ford”, “2” for “Chevy”, and “3” for “Toyota”. Numbers on an ordinal scale have significance only in their relative positions. Numbers on nominal and ordinal scales are sometimes called qualitative, whereas numbers on the interval and ratio scales are called quantitative. The gap between numbers has significance when the numbers are on an interval scale. If, for example, a person were asked to state his preference for soda on a scale of “1” to “10” with “10” being most preferred, then responses 1,5,9 and 1,2,9 would have different meanings on an interval scale, but not on an ordinal scale.
6
R. C. Dubes
The most important data scale in engineering work is the ratio scale, which is the interval scale with a natural zero. Data from sensors, and numbers which can be placed on the real line are examples. For example, distance is measured on a ratio scale. Doubling the distance between two towns means using twice as much gas to get between them, whatever the unit of distance. Temperature, on the other hand, is an interval measurement because its significance depends on the unit. Measuring in degrees Kelvin is a ratio-scale measurement, while temperature in degrees Celsius is an interval-scale measurement. Data type refers t o degree of quantization. The three types recognized here are binary, or two-valued, discrete, or multi-valued, and continuous, or data taken from the mathematical real line. Binary data, also called dichotomous data, are for situations where the possible responses are (“yes”, “no”), or (“on”, “off”). A discrete type has a small number of values, where “small” depends on the situation. Since instruments have finite resolution, all data measured in the real, as opposed to the mathematical, world are discrete. Calculus ordinarily requires that data be continuous. Thus, we often assume data are continuous and ignore the unpleasant reality. 2.1.2. Patterns and proximity Whatever the scale and type, data are collected in one of two basic formats, called a pattern matrix and a proximity matrix. A pattern matrix represents each object under examination as a set of measurements. Each measurement is called a feature and a pattern is a set of feature values measured on an object. The set of d measurements form the feature space, each feature corresponding to one orthogonal axis. A pattern matrix is thus an n x d matrix, where n is the number of patterns and d is the number of features. The notation for the j t h feature of pattern i will be xij and the ith pattern itself will be denoted by the column vector xi. Letting superscript T denote matrix transpose, xi = [ZZl xi2
. . . ZZd]T .
We require n >> d and think of the patterns as a swarm of n points floating in a d-dimensional space. The transpose of the n x d pattern matrix X is written as:
A pattern matrix, called the speaker data, that consists of 40 rows (patterns) and 5 columns (features) will be used to demonstrate several procedures. A listing is given in Table 1. Each pattern represents a spoken phrase and each feature represents a measurement taken on the spectrum derived from the phrase. There are eight categories, one for each person involved in the study. Eight male speakers spoke the same phrase five times; patterns 1-5 are from speaker number 1, patterns 6-10 from speaker number 2 , and so forth. The category labels are on a nominal scale
1 . 1 Cluster Analysis and Related Issues 7 Table 1. Speaker data. Features
Category
Pattern
1
1 2
107.0000
1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4 5
105.5000
18.0000 15.0000
87.0000 90.0000
94.6500 96.4750
89.0000 91.0000
3
106.5000
13.0000
88.0000
94.9500
93.0000
4 5
102.5000
13.0000 17.0000
85.0000 85.0000
91.1250 93.1250
90.0000 87.0000
17.0000 13.0000
97.0000 97.0000
103.3000 103.3500
100.0000 103.OOOO
6 7
106.0000 118.0000 116.5000
8
123.0000
16.0000
100.0000
105.7250
104.0000
9 10
119.0000
15.0000 16.0000
98.0000 101.oooo
103.5000 105.1750
104.0000 104.0000
11
103.5000
83.9000
81.0000
104.OOOO 104.5000
10.0000 1 2.oooo
78.0000
12
81.0000
86.9750
81.0000
10.0000
77.0000
91.0000
11.0000 11.0000
84.5250 86.3000
13 14
122.0000
104.5000 100.0000
15 16
109.0000
16.0000
78.0000 79.0000 82.0000
17 18
113.0000 111.0000
22.0000 16.0000
85.0000 85.0000
19
113.0000
15.0000
78.0000
94.0250 92.6750 94.3250
20 21
112.0000 122.5000
20.0000 30.0000
84.0000 87.0000
93.5500 89.6500
5
22
119.0000
81.0000
87.0750
93.0000 90.0000
5
23
87.0000
92.1750
97.0000
5 5 6 6 6 6 6 7 7 7 7 7 8 8 8 8 8
24 25
122.5000 125.5000
28.0000 24.0000 32.0000
87.0000 83.0000
94.3000 89.7000
90.0000 96.0000
72.0000
78.9000 79.0250
74.0000
87.0000 91.6000
83.0000 84.0000 87.0000 90.0000 90.0000 100.0000 90.0000
26
119.0000 95.5000
23.0000 23.0000
27
94.0000
28 29
103.0000
15.0000 24.0000
76.0000 77.0000
82.7500
78.0000 78.0000
106.5000 102.5000
28.0000
76.0000
84.2750
78.0000
19.0000
75.0000
79.9750
84.0000
31 32
111.oooo 110.0000
25.0000 26.0000
77.0000 78.0000
86.7750 89.7500
82.0000 78.0000
33
109.0000 109.0000
22.0000 23.0000
78.0000
87.5250
84.0000
34
110.5000 107.0000
22.0000 16.0000
89.3250 88.7750
83.0000
35 36
78.0000 81.0000 79.0000
87.3250
82.0000 81.0000
37
107.0000
16.0000
77.0000
86.3500
81.0000
38
105.0000
20.0000
78.0000
80.0000
39 40
111.0000 110.0000
18.0000 15.OOOO
80.0000 85.0000
87.6500 87.5000
30
86.2000
84.0000 88.0000
8
R. C.Dubes
while the features are on a ratio scale since all have natural zeros. The features are assumed to be continuous. A proximity matrix is a square, symmetric matrix. Its rows and columns both correspond to patterns, or t o features. The (2, j) entry contains an index of proximity that denotes the degree of closeness or alikeness between the objects corresponding to row i and column j . If the proximity matrix is a dissimilarity matrix, then the larger the entry (i, j ) , the less items i and j resemble one another, as when Euclidean distance measures the proximity between two patterns. In a similarity matrix, a large value indicates a close resemblance between the two objects, as when a correlation coefficient measures the proximity between two features. 2.1.3. Indices of proximity Clustering algorithms require that an index of proximity he established between all pairs of items. Anderberg [l]defines several such measures of proximity. This chapter covers only the case when proximities are computed from a pattern matrix. An index of dissimilarity d(q, r ) between patterns xq and x, is a real-valued function satisfying the following for all q and r .
An index of similarity satisfies the first two conditions, hut replaces the third with 4%Q ) 2 my{d(q, .)I. The most common index of dissimilarity in engineering work is the Minkowski metric (2.4). (llm)
.>(jcr,
4%
=
b q j - Z,jIrn
)
(2.4)
See [12] for a full discussion of this and related indices of dissimilarity. Common parameter values are m = 2, or Euclidean distance, m = 1, or Manhattan distance, also called taxicab and city-block distance, and m + 00, or “sup” distance. The Euclidean distance must be carefully separated from the squared Euclidean distance. The Minkowski metric is for continuous data on a ratio scale and is itself on a ratio scale. The Minkowski metric satisfies the triangle inequality.
d ( q , r ) 5 d ( q , s)
+ d(s, r ) for all ( 4 , r, s)
(2-5)
A common index of similarity between features u and w is the sample correlation coefficient.
1 . 1 Cluster Analysis and Related Issues 9
The sample means, mu and m,, and the sample standard deviations, s, and s,, are defined in (2.7). The sample correlation coefficient indicates the degree of linear dependence between two features, with a value of 0 indicating linear independence. The absolute value measures the degree of resemblance between two features since negative and positive correlations with the same magnitude have the same interpretation. A number of indices of proximity for binary data, such as the simple matching coefficient and the Jaccard coefficient, have been proposed [1,4]but are not discussed here. 2.2. Normalization and Standardization
A pattern matrix is normalized t o equalize the contributions of the features to a projection or a clustering. Milligan and Cooper [13] noted that some normalizations that appeal to our intuition may affect performance in unexpected ways. In this section, a “*’, superscript will denote the raw, or un-normalized, data, as in x:~. Throughout the remainder of the chapter, the context must make clear which normalization has been applied. One common normalization is t o move the origin of the feature space t o the grand mean vector, which is the vector of sample means for the d features. The sample mean m3,and the sample standard variance 532, for feature j are:
The origin is shifted to the grand mean vector by defining, for all i and j = 1, 2, . . . , d: 2 . . - xt. - m j . 23 - a j
=
1, 2, . . . , n (2.8)
This shifting of the origin does not affect Euclidean distance and merely simplifies equations. The z-score normalization divides each feature by its standard deviation s j in (2.9). x*. - m x . . - 23 3 (2.9) 23 sj
This normalization stretches or squeezes each of the coordinate axes t o equalize the spreads along all axes. A third normalization, called a range method, reduces all features to the range [0,1] by subtracting the smallest value in each column and dividing by the range of each column. x.. ” -
xij - min{xij} max{xij} - min{xij}
The min and max are taken over the j t h column of the pattern matrix. Miligan and Cooper [13] found that the range normalization outperformed the z-score normalization when extracting the structure of the data.
10 R. C. Dubes
2.3. Dimensionality
The dimensionality of data refers to t,he number of independent parameters required t o describe the data. These parameters for each pattern are often taken to be the feature values on the d coordinate axes. However, data can sometimes be projected to fewer than d dimensions, as discussed in Sections 3.1 and 3.2. The number of dimensions in the target space is called the intrinsic dimensionality [14,15] since it represents the dimensionality suggested by the data themselves and provides a parsimonious characterization of the data. Dimensionality has taken on new importance with the emergence of fractal geometry as a cross-disciplinary field of study [16]. The true dimensionality of the data, whether fractional or integer, may become an important characteristic of the data in exploratory data analysis. The estimation of fractal dimensionality [17] is beyond the scope of this paper. 3. Exploratory Data Analysis
Cluster analysis includes a variety of heuristic procedures for getting to know data. Whereas clustering algorithms strive to be objective and quantitative, exploratory data analysis is purposefully subjective. The goal is to use whatever tools are available t o look a t the data, with emphasis on graphs, charts, projections, and any graphical representation that assists the visual system and nudges intuition. Everitt [18] describes several techniques that go beyond simple graphing of functions. Recent advances in graphical computer displays have made the techniques of exploratory data analysis widely available. Two that I have found useful are the S program [19] originally developed at AT&T and the MacSpin program for the Macintosh.” A few standard techniques are explained in this section to indicate the flavor of exploratory data analysis. 3.1. Linear Projections
Why represent d-dimensional data in two or three dimensions? One reason is to be able to see the data. A second reason is to simplify the data by eliminating redundancy and isolating the important characteristics of the data. The process of representing data in a new space is sometimes called ordination. No two- or three-dimensional representation can fully capture the intricacies and complexity of, say, ten-dimensional data. A linear projection does little “violence” to the data since relative distances are preserved and certain geometrical characteristics are maintained. Several schemes for representing data have been proposed [20], including discriminant analysis [4]. The transformation discussed here has been called the princzpal component, the Karhunen-Loeve, and, simply, the eigenvector transformation. It is based on the eigenvalues and eigenvectors of the d x d sample covariance matrix R computed aThe S software is available from Statistical Sciences, Inc., 1700 Westlake, Seattle, WA 98109, USA. The MacSpin program is marketed by Abacus Concepts.
I . 1 Cluster Analysis and Related Issues
11
from the given n x d pattern matrix, X.
Equation (3.1) assumes that either (2.8) or (2.9) has been applied to the raw data. If (2.8) has been applied, R is a covariance matrix whose diagonal entries are the variances of the columns in X. If (2.9) has been applied, R is a correlation matrix meaning that all diagonal entries are unity and off-diagonal entries are correlation coefficients between -1 and 1. Most linear algebra texts show that the eigenvalues of R are solutions for the scalar X to the determinant equation (3.2).
Here, I is a unit matrix of order d x d and 0 is a vector of zeros. If R has full rank of d , and we denote its (necessarily real and non-negative) eigenvalues by 2 2 . . . 2 A d , then the eigenvectors c1, cz, . . . , Cd are orthonormal (column) vectors satisfying, for each j from 1 to d:
(R - X j ) c j
=0
C T C =~ 0 i f j # IC =
1 if j = k .
(3.3) (3.4) (3.5)
Eigenvectors are not unique. The eigenvector transformation from d to m 5 d dimensions is defined by a coefficient matrix whose rows are eigenvectors corresponding to the m largest eigenvalues of R. If xi is a pattern in the original d-dimensional space, its image yz is the m-vector defined in (3.6).
Yi =
If the rank of R is full and m = d, then (3.6) rotates the coordinate axes and decorrelates the features. That is, the covariance matrix computed from the new vectors { y i } is a diagonal matrix whose diagonal entries are the eigenvalues XI, X2, . . . , A d . Thus, the eigenvalues of R can be interpreted as sample variances in the rotated space. The d-dimensional patterns can be projected to two or three dimensions for viewing by setting m to 2 or 3 in (3.6). Why the top rows? A reasonable criterion for projecting data is square-error, meaning that the target space saves as much of
12
R. C. Dubes
the variance as possible. The total variance is the sum of the diagonal elements of R . d
d
j=1
j=1
Since the eigenvalues are ordered by size, it makes sense to save eigenvectors corresponding to the largest eigenvalues. Tou and Heydorn [21] phrase this problem nicely and prove these facts. Figure 2 shows the eigenvector transformation for the speaker data both with pattern numbers and with category labels. Some clustering of the patterns is suggested, especially in Fig. 2(b).
u 7
2
' >
r)
0
4
2
4
:
e8
rq
I
P29
*'
25
.
*
' 6
1
17
s 6
7'
23
u
6
6 u
I
7 -
n
5 21
5
s
s24
5 160
110
110
190
?m
210
fir.,
Fig. 2. Eigenvector projections of speaker data; (a) pattern labels; (b) category labels.
3.2. Nonlinear Projections Nonlinear projections share the goals of linear projections but the details differ. The projection introduced by Sammon [22] illustrates these differences. Sammon's projection begins with a n n x n dissimilarity matrix whose entries are Euclidean distances between all pairs of patterns in the &dimensional feature space. A set of n points is scattered randomly in a portion of the two-dimensional target space, with each point representing one pattern. Given such a configuration of n points, the following stress criterion can be computed.
1.1 Cluster Analysis and Related Issues 13 Here, d(q, r ) is the Euclidean distance between patterns q and r in the ddimensional feature space and D(q, r ) is the Euclidean distance between the points representing patterns q and T in the two-dimensional target space. The sums are over all pairs of patterns 1 5 q 5 r 5 n. Stress is a function of the 2n coordinates of the points in the target space. The idea is to move the points around so as to minimize E . The configuration a t which E is minimum is taken t o be the Sammon projection. A configuration for which D(q, r ) = d(q, T ) for all q and r would be ideal, but it is seldom possible to match all d-dimensional distances in two dimensions. The stress function resembles the criterion function in multidimensional scaling [23]. However, multidimensional scaling begins with ordinal data and Sammon’s method begins with ratio data. Since E uses the difference between two distances, one must have ratio-scale dissimilarities between all pairs of patterns. The denominators normalize the stress. The term d ( r , q ) in the sum on the right weights small distances more heavily than large ones and tends to preserve local structure. The multiplier on the left, which is fixed throughout the minimization procedure, tends to make E insensitive to changes in scale and sample size. Several algorithms exist for minimizing functions of many variables. Sammon [22] proposed a gradient descent algorithm. Simulated annealing has also been tried [24] but the results were disappointing. Whatever the minimization technique, one encounters the usual problems of stopping at local, rather than global, minima and of dependence on the starting configuration. Any minimization should be run several times and the one achieving the smallest stress should be retained. Our implementation requires that the user supply the maximum number of iterations allowed in the gradient descent algorithm and a “magic factor”, which influences the internal stopping criterion. An advantage of a nonlinear projection over a linear projection is its greater flexibility and ability to “see” complex structures. A disadvantage is that a nonlinear method can distort the data unduly and paint a misleading picture. In addition, extra points cannot be easily located in the target space of a nonlinear projection whereas the entire feature space can be projected to two dimensions with (3.6). Figure 3 pictures the speaker data with Sammon’s projection. The gradient descent algorithm was run ten times and the best of the ten runs is exhibited. The stress values for the ten runs ranged from 0.02512 to 0.0009157. The algorithm always stopped because it had achieved a minimum, not because it had reached the maximum number of iterations. Figures 2 and 3 agree on the structure of the data and provide evidence for the conclusion that the clustering by category is real, and not an artifact of an algorithm. One can try to name the variables on the Sammon plot in Fig. 3 from characteristics of the pattern, but no conclusions about correlation between these new variables can be drawn. The last step in the Sammon projection is to apply the eigenvector rotation to decorrelate the variables.
14
R. C. Dubes 8 10
8 -
2 2 8
1 7
e
2
2
a-
0
Ji
-
n
24 z1
n
26
11
2
11
20
in
7
I
R4
0 -
*
o
1
3
0
3
' 3
6 6
1
B
3 6
6 6
Fig. 3. Sammon projection of speaker data; (a) pattern labels; (b) category labels.
3.3. Graphical Procedures Everyone has their favorite graphical representation of data. A few possibilities are mentioned in this section. Both the columns (features) and the rows (patterns) of the pattern matrix can be depicted by graphical methods. Box plots and histograms provide quick looks at the distribution of a feature. Figure 4(a) shows box plots for the first three features of the speaker data. Each box identifies the 25th and 75th percentiles of the data. The horizontal line inside the box is at the median value. The dashed vertical lines are drawn from the minimum value to the 25th percentile and from the 75th percentile to the maximum. A star is drawn to signal that an outlier has occurred according to a particular criterion. Figure 4b gives histograms for the same three features. The number of bins is about filor 7 in this case. The ordinate is the number of feature values in each bin. Box plots are best for quick comparisons among features. Differences in ranges and locations are obvious a t a glance. The fact that feature 3 has a larger right tail than left is also obvious. Histograms provide more detailed information and exhibit the modes in multi-modal data. Graphical representations of the patterns can also be useful. Figure 5 exhibits Chernoff faces [25] for the first 20 patterns from the speaker data. Each face depicts one row in the pattern matrix by associating one characteristic of the face with each feature. Feature 1 is proportional to the area of the face, feature 2, to the shape
1.1 Cluster Analysis and Related Issues 15
-0
T
I
88 -
8 -
0
.
T
0
-
_-(4
-.
1 :,
(b)
Fig. 4. Graphical views of the first three features of the speaker data. (a) Box plots; (b) histograms.
Fig. 5. Chernoff faces of the speaker data.
of the face, feature 3, to the length of the nose, feature 4, to the location of the mouth, and feature 5 is related to the curve of the smile. The faces are arranged in Fig. 5 so that each column contains faces from a different category. Associations
16 R. C. Dubes
among the patterns can be recognized by mentally clustering the faces. Outliers can sometimes be quickly identified. For example, pattern 19 in Fig. 15 might be considered different from other faces. All features were normalized t o fall between 0 and 1 before the faces were drawn. 4. Hierarchical Clustering Clustering data means grouping either patterns or features such that items in the same group are more alike than are items in different groups. The objective is to abstract the essence of the data by isolating groups, or clusters, which explain the data. A hierarchical clustering is a sequence of nested groupings. In some biological applications, the hierarchical structure itself is fitted to the data. In many engineering problems, one searches the hierarchy for a single significant grouping. This section explains the basic mathematical structure for describing hierarchical clustering and presents some standard clustering algorithms. For more complete treatments, see [1,4,2,26]and Chapter 5 of [27].
4.1. Hierarchies and Dendrograms
A hierarchical clustering method is a mathematical procedure for creating a sequence of nested partitions from a proximity matrix, assumed to be a dissimilarity matrix for patterns. Let X denote the set of n patterns. A partition C = {Cl, C2, . . . , Cm} of X is a set of disjoint, non-empty subsets of X which, taken together, constitutes X. That is, if i # j ,
c,n cj = a ; c1u c2u . . . u cm= x.
(4.1)
where lLn” denotes set intersection, “U” denotes set union and W7’denotes the empty set. Each set Ci is called a component of the partition. A clustering is a partition of X ;its components are formally called clusters. A hierarchical clustering is a sequence of nested partitions starting with the trivial clustering in which each pattern is in a unique cluster and ending with the trivial clustering in which all patterns are in the same cluster. Partition B is nested into partition C if every component of B is a subset of a component of C, so C is formed by merging components of B. A dendrogram is a binary tree that depicts a hierarchical clustering. Each node in the tree represents a cluster. Cutting the dendrogram horizontally creates a clustering. A dendrogram represents n clusterings of n patterns, including the two trivial clusterings. Figure 6 is a simple dendrogram for five patterns. The five pattern labels can be permuted in several ways without altering the information in the dendrogram. Murtagh [28]enumerated several types of dendrograms. Dendrograms can simply picture the clusterings, as in Fig. 6, or can have a scale showing the level of dissimilarity at which each clustering is formed.
1 . 1 Cluster Analysis and Related Issues 17
Fig. 6 . Dendrogram for five patterns.
4.2. Recovered Structure and Ultrametricity
A hierarchical clustering method tries to fit a dendrogram to the given dissimilarity matrix. The dendrogram imposes a measure of dissimilarity called the cophenetic dissimilarity on the patterns as follows. Let {CO,C1, . . . , Cn-l} be the sequence of partitions in a dendrogram where Co is the trivial clustering that puts each pattern in its own cluster and C,-1 places all patterns in the same cluster. The clusters in the mth clustering are denoted {Cml,Cmz, ... , Cm(n-m)}.A level function L ( m ) is defined on the partitions as the dissimilarity level a t which clustering m first forms. Specific level functions are defined by each clustering method. The cophenetic dissimilarity d c between patterns x q and x, is defined in (4.2).
where k q , , = min[m : ( x q ,x,) E Cmt, some t].
The cophenetic dissimilarity is a dissimilarity index in the sense of (2.1). However, it also satisfies the ultrametric inequality.
The nesting of partitions to form the hierarchy and the monotonicity of the level function ensure that (4.3) is satisfied. This inequality is stricter than the triangle inequality (2.5). For the distances between patterns in a feature space t o satisfy the ultrametric inequality, for example, all triples of patterns must form isosceles triangles with the one side being shorter than the two sides of equal length. This demands that many ties in proximity occur at just the right places.
4.3. Hierarchical Clustering Algorithms A hierarchical clustering algorithm is a process for constructing a sequence of partitions from a proximity matrix. A particular clustering method can be implemented
18 R. C. Dubes
by several algorithms. Some hierarchical clustering algorithms for constructing dendrograms by hand are based on graph theory [4]. This section is limited to a class of algorithms commonly known as “SAHN” (Sequential, Agglomerative, Hierarchical, Nonoverlapping) algorithms. A SAHN algorithm begins with a n n x n dissimilarity matrix [d(q, r ) ] between patterns. All SAHN algorithms are appropriate when the dissimilarities are on a ratio scale, as with distances in a feature space. In addition to a sequence of clusterings, a SAHN algorithm creates a level function. The algorithm begins with L(0) = 0 and each pattern in a unique cluster. A cluster is denoted (s).
SAHN Algorithm for Hierarchical Clustering 1. Set the sequence number of the clustering: m = 0. Repeat the following steps until m = n. 2. Find the pair of clusters [(s), (t)] for which
4 ( s ) , (t)l = min{d[(q),
(.)I)
(4.4)
where the minimum is taken over all pairs of clusters. 3. Increment m by 1. Merge clusters (s) and ( t ) into a single cluster to define clustering m. Define the level of this clustering as:
L ( m ) = d [ ( s ) , (t)I.
(4.5)
4. Update the dissimilarity matrix by deleting the rows and columns corresponding to clusters ( s ) and ( t ) and adding a row and column for the newly formed cluster (s, t ) . The dissimilarity between this new cluster and an existing cluster, cluster ( k ) ,depends on the clustering method being employed and is given by (4.6).
4.4. Characteristics of Hierarchies and Algorithms
Day [29] explains the computational complexities of hierarchical and partitional clustering algorithms. Some comments on practical aspects of hierarchical clustering are given below without justification. The primary application of hierarchical clustering in engineering work is to create a representative sequence of clusterings while searching for a single “good” clustering. It is seldom the case that data are hierarchically related on more than a few levels.
1.1 Cluster Analysis and Related Issues 19 Clustering method Single-link Complete-link UPGMA WPGMA
fft
P
Y
112
112
0
-112
112
112
0
112
ns ns f nt
ns ns nt
+
0
0
112
112
0
0
1J2
-114
0
ffS
UPGMC WPGMC Ward’s Flexible
112
ns
+ nk
ns f n t ff
+nk
nt +nk ns +nt ff
+n k
-nk
ns f n t
+nk
1 - 2ff
0 0
Fig. 7 . Coefficients for SAHN hierarchical clustering algorithms.
The single-link method is also called the connectedness method, the minimum method, and the nearest neighbor method. The updating equation uses the minimum of the two dissimilarities. Single-link clusters can be “straggly” since the smallest of the pairwise dissimilarities between two clusters determines when the clusters join. Other algorithms for the single-link method involve the minimum spanning tree [30] and graph theory [31]. By contrast, the complete-link method is also called the completeness method and the maximum method. The updating equation translates into taking the maximum of the dissimilarities between pairs of patterns in different clusters. Complete-link clusters often form in small clumps because the largest of the pairwise distances determines when two clusters are merged. Figure 8 shows the complete-link dendrogram for the speaker data. The visual impact of the dendrograms is obvious. It should also be clear why dendrograms are not very useful for more than 200 patterns. The scale on the left is on the same scale as the dissimilarity matrix. The patterns were normalized by (2.9). Some of the clusters in Fig. 8 can be identified in Figs. 2 and 3. With the single-link and complete-link methods, hierarchies merge only at dissimilarities that occur in the given dissimilarity matrix. Other clustering methods do not have this property. Ward’s method 1321 is also called the minimum square error method because it merges, at each level, the two clusters which will minimize the square error from among all mergers of pairs of existing clusters. Ward’s method does not create the clustering which minimizes square error among all clusterings with that number of clusters, however. Two characteristics of Ward’s method should be noted. First, it minimizes square error as described above only when the dissimilarity index is the squared Euclidean distance. Second, the scale on the dendogram is not the same as the dissimilarity scale between patterns. Hierarchies can merge at levels several hundred times the largest dissimilarity between patterns. The visual effect on dendrograms is to accentuate the “lifetime” of individual clusters.
20
R. C. Dubes
Fig. 8. Complete-link dendrogram for speaker data.
The acronym “PGM’ refers to “pair group method” since clusters are merged in pairs. The prefixes “U” and “W’ refer t o “unweighted” and “weighted” methods, repectively. A weighted method treats all clusters the same in (4.6), so patterns in small clusters achieve more individual importance than do patterns in large clusters. An unweighted method takes the size of the clusters into account, so the patterns are treated equally. The suffixes “A” and “C” refer to “arithmetic averages” and “centroids”, respectively. Thus, the full name for the UPGMA method is “unweighted pair group method using arithmetic averages” , sometimes called the group average method. The UPGMC and WPGMC methods have direct geometric interpretations with patterns in a feature space [27]. The UPGMA and WPGMA methods have no simple geometric interpretation. An implicit assumption underlying this entire discussion is that no two entries in any of the matrices encountered in matrix updating are the same. Ties in dissimilarity can produce unexpected and baffling dendrograms for all methods except the single-link method. Jardine and Sibson [33] show that the single-link method is the only method having a continuity property, whereby dendrograms merge smoothly as dissimilarities approach one another. Unfortunately, the single-link method has performed much worse that other methods in extracting the true structure of the data. If the dissimilarity matrix contains ties, either break the ties or use the single-link method. A clustering method is monotone if, when merging clusters (s) and ( t ) into cluster (s, t ) , then for all clusters (Ic) distinct from ( s ) and ( t ) ,
1.1 Cluster Analysis and Related Issues 21 Monotonicity permits dendrograms to be drawn as binary trees, as in Figures 6 and 8. The cophenetic dissimilarity is satisfied only for monotone clustering methods. Without monotonicity, crossovers or reversals can occur in which two clusters merge at a level lower than the level a t which one of the two clusters was formed. This counter-intuitive phenomenon has nothing to do with the data, but is a characteristic of the clustering method. Interpreting a non-monotone dendrogram is extremely difficult. A simple condition for monotonicity, proved by Milligan [34], is that the coefficients in (4.6) satisfy (4.7).
Which clustering method is best? No clear answers exits, even though several comparative studies have been conducted [35]. If the input dissimilarity matrix satisfies the ultrametric inequality then the single-link and complete-link dendrograms will be exacty the same. This is usually taken to mean that the data are organized in a perfect hierarchy. The degree to which the two dendrograms resemble one another is an indication of how appropriate a hierarchical structure is for the data. Defining a quantitative measure of similarity between dendrograms is a very difficult task. 5. Partitional Clustering
A dendrogram displays n clusterings, or partitions, of n patterns, but how do we find a single good clustering of the data? A partitional clustering method handles a large number of patterns and applies an objective criterion in an attempt to achieve the “best” clustering. A solution to the clustering problem is easy to state. Select a criterion and evaluate it for all clusterings, saving the optimal result. This solution is impractical for two main reasons. First, the number of possible clusterings is astronomical [4]. For example, there are about 11,259,666,000 clusterings of 19 objects into 4 clusters. Evaluating a criterion for all clusterings is out of the question. Even if one could enumerate all clusterings, the choice of a single criterion function raises severe difficulties. Clusters of patterns in d dimensions can have a variety of shapes, from spherical to line-like [36]. No single criterion can search for shapes simultaneously. Before choosing a clustering criterion, one must determine what is meant by “cluster” in the application at hand. One general definition is that a cluster is a set of patterns whose inter-pattern distances are smaller than interpattern distances for patterns not in the same cluster. This idea of “cluster” leads to accepting the square-error criterion stated below. One can also reasonably decide that a cluster is a region of high density in the feature space, surrounded by regions of low density. Square-error is not the only criterion. However, square-error is relatively easy to compute, makes good intuitive sense, and the clusters can be interpreted as hyperspheres in the feature space.
22
R. C.Dubes
Given n patterns {XI,x2, . . . , x,} in the d-dimensional feature space, define an indicator function as follows. Let z i k = 1 if pattern xi is a member of the kth cluster and 0 if not. Every pattern must belong to one cluster and the clusters are labeled sequentially. The center of the kth cluster is the centroid of the patterns belonging to the cluster.
Here, n k = C:==, &k is the number of patterns in cluster k . The square-error e$ for cluster k is the sum of the squared Euclidean distances between patterns in cluster k and mk. n
The square-error for the entire clustering is the sum of the square-errors for the individual clusters. K
(5.3) k=l
The clustering problem can be stated as the problem of minimizing E L , for K fixed, by selecting the binary weights { z i k } in such a way that each row of the n x K matrix of weights has exactly one “1” in each row and a t least one “1” in each column. That is, each pattern can belong to only one cluster and no cluster can be empty. In fuzzy clustering [7] zik is taken to be the degree of belonging for pattern i in cluster k and z i k in (5.2) is replaced by z:k where q is chosen empirically, and is usually 2. One can approach this optimization in many ways, including simulated annealing [24]. Gordon and Henderson [37] translate the constrained optimization problem into an unconstrained problem to simplify the solution. The computational demands of formal minimization techniques have encouraged the development of simple, heuristic algorithms [1,4]. The most popular of these is the K - m e a n s method.
K-Means Algorithm for Partitional Clustering 1. Select an initial clustering of the n patterns with K clusters and initialize the cluster centers. Repeat until the clustering stablizes: 2. Assign cluster labels to all patterns by finding the closest cluster centers. 3 . Compute cluster centers for all clusters with (5.1). Repeat Steps 2 and 3 until no cluster labels change. 4. Apply heuristic splitting and lumping criteria. End the repeat. 5 . Compute statistics of the final clustering.
1.1 Cluster Analysis and Related Issues 23 The clustering with the smallest square-error is retained as the solution. Although the convergence of the algorithm can be proven [38] no assurances that the solution is optimal can ever be given. It is easy to propose data that will defeat the algorithm [4]. One hopes that running the algorithm several times for different starting configurations leads to a reasonable solution. The problem of choosing the number of clusters and of formally validating the results is discussed in Section 6. Steps 2 and 3 constitute a K - m e a n s pass. An alternative is to recompute the cluster centers after each cluster label is changed. Step 4 changes the number of clusters by merging small clusters, removing outliers, or splitting large clusters. Criteria for these operations are chosen heuristically. The initial partition can be chosen in several ways [l].For example, one can choose k patterns at random as seed points. One can also choose k patterns that are reasonably separated from one another as seed points. In any event, the algorithm should be run with different seed points to seek the best clustering. Any program for implementing a partitional clustering method involves several parameters. The primary ones are K and any parameters associated with splitting clusters, lumping clusters, and identifying outliers. An exception is the CLUSTER program [4] which creates a sequence of clusterings and uses no parameters. Only experience can dictate selection of these parameters. The manner in which the data are normalized must also be considered. Normalization (2.9) can reduce the effects of spreads in the individual variables and equalize the contributions of all variables to Euclidean distance. The results of any partitional clustering algorithm is a set of tables of numbers showing the cluster centers, the square-errors for the individual clusters, the covariance matrices for the clusters, and various statistics which try to quantify the characteristics of the clustering [4]. Clustering algorithms are often run to see if patterns are clustered according to some a priori category information and a clusterby-category table is displayed to see if the clusters correspond to categories in any way. The cluster centers can be taken as a sampling of the original data and one can represent the data by projecting the cluster centers t o two dimensions. Figure 9 shows the cluster-by-category table from the CLUSTER program with the speaker data, after normalizing by (2.9). Also shown are the square-errors for all clusters. Comparing the table entries to Fig. 2, 3, and 8 may justify some of the entries in the table. For example, categories 1, 2, 3, and 5 are in unique clusters. However, cluster 3 has a much larger square-error than clusters 1, 2, and 5. This suggests that clusters 1, 2, and 5 have smaller dispersions than cluster 3 . 6. Validation and Interpretation
Validation refers to the objective assessment of a clustering structure so as to determine whether a structure is meaningful. The structures under consideration are hierarchies, clusterings, and individual clusters. The sense of validity explained
24
R. C. Dubes
here calls a structure valid if it is unusual under the circumstances of the study. That is, a structure is valid if it cannot reasonably be assumed to have occurred by chance or to be an artifact of a clustering algorithm. Validation is accomplished by carefully applying statistical methods and testing hypotheses.
Cluster 1 2
1 0 0
2
3
4
5
0
0
0
0
5
0
0
3 4
0
0
0
0
0 5 0 0 0
0
5
0 0
5
6 7 8
7
8
0
0
0
6 3 0
0
0
5
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
2
5
1
0
0
0
0
0
0
0
0
4 1
0
0
0
4
e: 274.1 139.5 441.4 177.4 192.4 438.3 296.9 221.5
6.1. A n Attitude Towards Validation
One might argue that a structure is “valid” if it makes sense or is useful or can be interpreted or provides new insight. The list of ways t o justify a result is endless. This section considers only objective measures of validity that can be tested statistically. Formal testing may not be required in every application. One might use experience and judgement t o interpret a clustering structure or use clustering in an exploratory manner. In addition to the three types of structure (hierarchy, clustering, cluster), three types of validation studies called external, internal, and relative can be defined [4]. An index must be chosen to reflect the sense of validity being examined, with the structure and type of study in mind. An external assessment of validity objectively compares the recovered structure t o an a praora structure and tries t o quantify the match between the two. For example, one might test how closely cluster labels match category labels. An internal examination of validity uses no a priori information but tries t o determine if the structure is intrinsically appropriate for the data. For example, one might try to determine if a cluster derived from the single-link method is unusually compact or isolated, as compared t o other single-link clusters of the same size in random data. A relative test compares two structures and measures their relative merit. For example, one might compare a 4-cluster clustering to a 5-cluster clustering without using any a prior2 information. Several indices have been proposed for this purpose and some are explained in Section 6.4. The
1.1 Cluster Analysis and Related Issues 25 paradigm for testing the validity of a clustering structure is summarized below.
Validity Paradigm 1. Identify the clustering structure (hierarchy, clustering, cluster) and the type of validation (external, internal, relative). 2. Select an index. 3. Select an hypothesis of “no structure”. 4. Obtain (by theory or simulation) the baseline distribution of the index under the “no structure’’ hypothesis. 5. Compute the index for the structure being tested. 6. Formally test the hypothesis of “no structure’’ by determining whether the observed index is LLunusual”.
Establishing the baseline distribution might require extensive statistical sampling such as Monte Carlo estimation or bootstrapping. Although the same index can apply to all three types of tests, the circumstances of the application are very different. External tests of validity are generally much easier to apply than internal tests because hypotheses of randomness are easier to propose and baseline distributions are easier to derive than for internal tests. One example of a n external test is the Bailey profile [39,4] which measures the validity of an individual cluster that is defined before the analysis begins. The natural indices of validity for an individual cluster are simple measures of compactness and isolation derived from graph theory. Bailey profiles are restricted to ordinal data and use the hypergeometric distribution to derive baseline distributions. Baseline distributions of compactness and isolation indices cannot be derived when the cluster is obtained by applying a clustering method because the cluster depends on the method itself, among other things. Even though data are purely random, a diligent clustering method might uncover an unusual cluster. An internal test of validity should recognize such clusters as being artifacts. Three validation problems of practical interest are discussed in this section. 6.2. External Tests of Validity f o r a Partition
Suppose two partitions of n patterns are to be compared. One is from category information, obtained before the analysis is begun. The other is obtained from some clustering method. How well does the clustering match the categories? The labels themselves are not important, so renaming categories or clusters cannot affect the degree of match. Hubert and Arabie [40] studied the Rand index as a means for assessing the match between two such clusterings. Let a denote the number of pairs of objects that are in the same cluster in both clusterings and let d denote the number of pairs in different groups. There are
26
R. C. Dubes
n(n - 1)/2 pairs of objects to check. The Rand index [41] measures the degree of match.
Other statistics have been suggested for this purpose but they are linear functions of one another [4]. A clustering is termed “valid” if R is unusually high, as measured with respect to some baseline distribution. To make R less sensitive to problem parameters. Hubert and Arabie [40] corrected it for chance. The hypergeometric distribution was applied to find E ( R ) ,the expected value of R under the baseline distribution. The maximum possible value of R is 1, so the corrected Rand index is :
R’
=
R - &(R) 1 - & ( R ).
Detailed formulas are given elsewhere [40,4]. Using the Rand index, corrected or not, in a test for external validity requries that a baseline distribution be developed. The variance of R‘ is known when one of the partitions is assumed to be assigned a t random. However, the full baseline distribution of R’ is required t o formally test this hypothesis of randomness. 6 . 3 . Internal Tests of Validity for a Partition
The paradigm for external tests of validity is the same as for internal tests. The hypothesis of randomness and the baseline distributions differ. This has led t o some confusion in the literature. For example, the distribution of certain Fstatistics and x2 statistics are listed in standard books on multivariable statistics. These distributions assume that the groups have been chosen without reference to the data, as when one assigns a priori category labels. These distributions are not applicable to the internal validation of clusterings found by sifting through the data. Using them can create misleading results [35]. Milligan [42] compared the performances of 30 internal indices of validity. Three primary difficulties arise in obtaining the baseline distribution needed for an internal test of validity. The first difficulty lies in choosing a hypothesis of “no structure” or “randomness”. To create purely random data, one must choose the region of space in which the random data are to be generated. To be fair, this region must match the characteristics of the data. The second difficulty is the necessity to match all the data parameters. This implies that one must estimate the baseline distribution anew in every application, usually by Monte Carlo sampling. Bock [43] has derived asymptotic distributions for some indices, but it is seldom clear when the assumptions of the derivation are satisfied. The third difficulty is a bit subtle. It is not fair to compare a clustering obtained from a clustering algorithm t o just any clustering of random data. One should compare it to the best clustering of random data. That is, before calling a result valid, one should be reasonably certain that the same result could not be obtained
1.1 Cluster Analysis and Related Issues 27 from any random data, not simply that the result could not be obtained from some random data. Engelman and Hartigan [44] demonstrated this point in estimating the distribution of the ratio of between t o within scatter for one-dimensional data. Given a set of random data (from a normal distribution), they found the ratio for the best separation of the data into two clusters. They published the percentage points for the distribution of the best value of the ratio. However, the result is only applicable t o one-dimensional data. The methodology cannot be extended to more than one dimension easily because the number of clusterings to be examined increases exponentially with dimension. These difficulties may explain why more internal validation is not performed. 6.4. Relative Tests of Validity -How
M a n y Clusters?
The problem of determining the “true” number of clusters has been called the fundamental problem of cluster validity [45,46]. This question is particularly important in image segmentation, where the number of categories, such as land-use categories, is not know in advance. The question “HOWmany clusters are there in the data?” can be asked in at least three meaningful ways. 0
0
0
Do the data contain the number of clusters I expect? This clearly calls for an external test of validation and one might use the procedures of Section 6.2. Is it unusual to find this many clusters with data of this sort? This somewhat vague statement can best be answered with an internal criterion, as in Section 6.3. The basis for comparison must be defined and a baseline distribution must be derived- two difficult tasks. Which of a few clusterings is best? This is more difficult to answer than the first question, but is more specific than the second. The question implies that several clusterings, such as those derived by cutting a dendrogram at several levels, or running a partitional clustering algorithm several times, be considered as candidates. The number of clusters in the best of these clustering is taken to be the “correct” number.
This section considers the third question by examining a sequence of clusters as the number of clusters changes monotonically. For example, one might seek a stopping rule for choosing the best level for cutting a dendrogram. What is a good index? One possibility is to pick the clustering that minimizes square-error (5.3). Square-error is a strong function of the dimensionality, the number of patterns, and the number of clusters [4]. As the number of clusters increases, the square-error has a tendency to decrease whether or not one clustering is better than another. Milligan and Cooper [47]compared 30 indices as stopping rules by applying each to a wide variety of data and ranking them according to the one which found the correct answer most frequently. The correct answer was known because they generated their own data. Any index that performed well should, logically, be trusted with real data. Dubes [48] made more detailed comparisons of two other indices. Zhang
28
R. C.Dubes
and Modestino [49] integrate formal estimation of the number of clusters into image segmentat ion. Three representative indices are defined below; n is the number of patterns, K is the number of clusters in the clustering being evaluated, and E$ is the square-error of the clustering (5.3). To estimate the number of clusters, plot the chosen index as K varies and look for either a peak, a valley, or a knee in the curve, depending on the index. Two underlying assumptions are that the data are not random and at least two clusters exist. The Calinski-Harabasz index, CH(lc) was the best of the 30 indices tested by Milligan and Cooper [13].
The index will always be positive and will be zero for K = 0. Its upper bound depends on problem parameters. The value of K that maximizes C H ( K )is chosen to estimate K . This index normalizes the square-error and tends t o depend less on problem parameters than does the square-error itself. The C index is a normalized form of the r statistic [50] proposed to measure the correlation between spatial observations and time. Let c ( q , r ) be 1 if patterns xq and x, are in the same cluster and 0 if not. Let d(q, r ) denote the dissimilarity, or Euclidean distance, between the two patterns. The “raw” r statistic is: n-1
n
g=l
r=q+l
The dissimilarities need not be distances. Let a K be the number of pairs of patterns in which both patterns are in the same cluster. Define the following two statistics as the smallest and largest possible values of I? for a clustering of the kind being examined. min(r) = sum of
UK
smallest dissimilarities
m a x ( r ) = sum of
UK
largest dissimilarities
The C index is defined as:
C ( K )=
r - min(r) max(r) - min(r) .
The range of the C index is limited to [0,1]. The value of K that minimizes C(K) estimates the number of clusters. 0 The Goodman-Kruskal y statistic [51,4] measures the rank correlation between Euclidean distances d ( q , r ) and function f ( q , r ) for the clustering being evaluated, where f ( q , r ) = 1 - c(q, r ) is 1 if xq and x, are in different clusters and 0,
1 . 1 Cluster Analysis and Related Issues 29 if in the same cluster. In standard notation, S(+) denotes the number of concordant quartets and S ( - ) , the number of discordant quartets. This requires some explanation. A “quartet” is two pairs of numbers. One is a pair of dissimilarities, say [d(q, r ) , d(s, t ) ] ,and the other is the corresponding pair of indicator values, [ f ( q , r ) , f(s, t ) ] . A quartet is concordant either if d ( q , r ) < d(s, t ) and f ( q , T ) < f ( s , t ) or if d(q, r ) > d(s, t ) and f ( q , r ) > f(s, t ) . A quartet is discordant either if d(q, r ) < d(s, t ) and f ( q , r ) > f(s, t ) or if d(q, r ) > d(s, t ) and f ( q , r ) < f(s, t ) . If either pair is tied, the quartet is neither concordant nor discordant. Then,
This index is limited t o the range [-1,1]. The value of K then maximizes r ( K ) estimates of the number of clusters. Studies [13] have shown that some indices proposed in the literature perform very poorly, while some perform very well. It is impossible to claim optimality for any of them, because the characteristics of the data can affect performance in unknown ways.
7. Final Comments Cluster analysis is a valuable tool for organizing, summarizing, and exploring multivariate data. Among the problems that appear in applications are the choice of a clustering criterion that recognizes only clusters of interest, and the choice of a clustering algorithm. Validating the results objectively is the most difficult problem of all. Notwithstanding these real problems, cluster analysis has proved enlightening, especially when invoked by a careful practioner who is aware of inherent limitations and has the proper computer tools.
Acknowledgements
I acknowledge the support of the National Science Foundation, most recently through grant IRI-8901513 and grant CDA-8806599. References [I] M. R. Anderberg, Cluster Analysis for Applications (Academic Press, New York, NY, 1973). [2] A. D. Gordon, Classification (Chapman and Hall, London, 1981). [3] J. A. Hartigan, Clustering Algorithms (John Wiley & Sons, New York, NY, 1975). [4] A. K. Jain and R. C. Dubes, Algorithms for Clustering Data (Prentice-Hall, Engelwood Cliffs, NJ, 1988). [5] L. Legendre and P. Legendre, Numerical Ecology (Elsevier Scientific, Amsterdam, 1983).
30
R . C. Dubes
[6] 3. M. Jolion, P. Meer and S. Bataouche, Robust clustering with applications in computer vision, ZEEE Trans. Pattern Anal. Mach. Intell. 13 (1991) 791-802. [7] J. C. Bezdek, Pattern Recognition with Fuzzy Objective Function Algorithms (Plenum Press, New York, NY, 1981). [8] X. L. Xie and G. Beni, A validity measure for fuzzy clustering, ZEEE Trans. Pattern Anal. Mach. Intell. 13 (1991) 841-847. [9] R. S. Michalski and R. E. Stepp, Automated construction of classifications: Conceptual clustering versus numerical taxonomy, IEEE Trans. Pattern Anal. Mach. Intell. 5 (1983) 396-410. [lo] G. Matthews and J. Hearne, Clustering without a metric, ZEEE Trans. Pattern Anal. Mach. Intell. 13 (1991) 175-184. [ll]Institute of Electrical and Electronics Engineers, Special Issue of Proceedings o n Neural Networks, I: Theory and Modeling, Sept. 1990. [12] J. C. Gower and P. Legendre, Metric and Euclidean properties of dissimilarity coefficients, J . Classification 3 (1986) 5-48. [13] G. W. Milligan and M. C. Cooper, A study of standardization of variables in cluster analysis, J. Classification 5 (1988) 181-204. [14] K. Pettis, T. Bailey, A. K. Jain and R. Dubes, An intrinsic dimensionality estimator from near-neighbour information, ZEEE Trans. Pattern Anal. Much. Intell. 1 (1979) 25-37. [15] N. Wyse, R. Dubes and A. K. Jain, A critical evaluation of intrinsic dimensionality algorithms, in E. S. Gelsema and L. N. Kana1 (eds.), Pattern Recognition in Practzce (North-Holland Amsterdam, 1980) p. 415-425. [16] K. Falconer, Fractal Geometry (John Wiley & Sons, New York, NY, 1990). [17] J. Theiler, Estimating fractal dimension, J. Opt. SOC.Am. A 7 (1990) 1055-1073. [18] B. S. Everitt, Graphical Techniques for Multivariate Data (Elsevier North-Holland, New York, NY, 1978). [19] R. A. Becker, J. M. Chambers and A. R. Wilke, The New S Language (Wadsworth & Brooks/Cole, Pacific Grove, CA, 1988). [20] T. Okada and S. Tomita, An optimal orthonormal system for discriminant analysis, Pattern Recogn. 18 (1985) 139-144. [21] J. T. Tou and R. P. Heydorn, Some approaches to optimum feature extraction, in J. T. Tou (ed.), Computer and Information Sciences I 1 (Academic Press, New York, NY, 1967) 57-89. [22] J. W. Sammon, A nonlinear mapping for data structure analysis, IEEE Trans. Comput. 18 (1969) 401-409. [23] J. B. Kruskal, Multidimensional scaling and other methods for discovering structure, in K. Enslein, A. Ralston, and H. S. Wilf (eds.), Statistical Methods f o r Digital Computers (John Wiley & Sons, New York, NY, 1977) 296-339. [24] R. W. Klein and R. C. Dubes, Experiments in projection and clustering by simulated annealing, Pattern Recogn. 22 (1989) 213-220. [25] H. Chernoff, The use of faces to represent points in k-dimensional space graphically, J. Am. Stat. Assoc. 68 (1973) 361-368. 126) A. D. Gordon, Hierarchical classification, in P. Arabie and L. Hubert (eds.), Clustering and Classification (World Scientific, Singapore, 1992).
1.1 Cluster Analysis and Related Issues 31 [27] P. H. A. Sneath and R. R. Sokal, Numerical Taxonomy (W. H. Freeman and Company, San Francisco, CA, 1973). [28] F. Murtagh, Counting dendrograms: A survey, Disc. Appl. Math. 7 (1984) 191-199. [29] W. H. E. Day, Complexity theory: An introduction for practitioners of classification, in P. Arabie and L. Hubert (eds.), Clustering and Classification (World Scientific, Singapore, 1992). [30] J. C. Gower and G. J. S. Ross, Minimum spanning trees and single-linkage cluster analysis, Appl. Stat. 18 (1969) 54-64. [31] L. J. Hubert, Some applications of graph theory to clustering, Psychometrika 39 (1974) 283-309. [32] J. H. Ward, Hierarchical grouping to optimize an objective function, J. Am. Stat. ASSOC. 58 (1963) 236-244. [33] N. Jardine and R. Sibson, Mathematical Taxonomy (John Wiley & Sons, New York, NY, 1971). [34] G. W. Milligan, Ultrametric hierarchical clustering algorithms, Psychometrika 44 (1979) 343-346. [35] G. W. Milligan, A review of Monte Carlo tests of cluster analysis, Multivar. Behav. Res. 16 (1981) 379-407. [36] C. T. Zahn, Graph-theoretical methods for detecting and describing gestalt clusters, I E E E Trans. Comput. 20 (1971) 68-86. [37] A. D. Gordon and J. T. Henderson, Algorithm for Euclidean sum of squares classification, Biometrics 33 (1977) 355-362. [38] S. Z. Selim and M. A. Ismail, K-means type algorithms: A generalized convergence theorem and characterization of local optimality, I E E E Trans. Pattern Anal. Mach. Intell. 6 (1984) 81-87. [39] T. A. Bailey and R. C. Dubes, Cluster validity profiles, Pattern Recogn. 15 (1982) 61-83. [40] L. J. Hubert and P. Arabie, Comparing partitions, J . Classification 2 (1985) 193-218. [41] W. M. Rand, Objective criteria for the evaluation of clustering methods, J. Am. Stat. ASSOC. 66 (1971) 846-850. [42] G . W. Milligan, A Monte Carlo study of 30 internal criterion measures for clusteranalysis, Psychometnka 46 (1981) 187-195. [43] H. H. Bock, On some significance tests in cluster analysis, J . Classification 2 (1985) 77-108. (441 L. Engelman and J. A. Hartigan, Percentage points of a test for clusters, J. Am. Stat. ASSOC. 64 (1969) 1647-1648. [45] R. 0 . Duda and P. E. Hart, Pattern Classification and Scene Analysis (John Wiley & Sons, New York, NY, 1973). [46] B. S. Everitt, Unresolved problems in cluster analysis, Biometrics 35 (1979) 169-181. [47] G. W. Milligan and M. C. Cooper, An examination of procedures for determining the number of clusters in a data set, Psychometrika 50 (1985) 159-179. [48] R. C. Dubes, How many clusters are best?-An experiment, Pattern Recogn. 20 (1987) 645-663. [49] J. Zhang and J . W. Modestino, A model-fitting approach to cluster validation with application to stochastic model-based image segmentation, I E E E Trans. Pattern Anal. Mach. Intell. 12 (1990) 1007-1009.
32
R. C.Dubes
[50] L. J. Hubert and J. Schultz, Quadratic assignment as a general data-analysis strategy, British J . Math. Stat. Psychol. 29 (1976) 19G-241. [51] L. A. Goodman and W. H. Kruskal, Measures of association for cross-classifications, J . Am. Stat. ASSOC. 49 (1954) 732-764.
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 33-60 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
[ CHAPTER 1.2 I STATISTICAL PATTERN RECOGNITION
KEINOSUKE FUKUNAGA School of Electrical Engineering, Purdue University West Lafayette, IN 47907, USA In the introductory Section 1, the problems of statistical pattern recognition are defined, and a flow chart is presented to show how a classifier ought to be designed. In Section 2, the theoretically optimal (Bayes) classifier and its variations are introduced. The discussion is extended to show how the resulting classification (Bayes) error can be computed for some limited cases. Also, the upper and lower bounds of the Bayes error are shown. Section 3 discusses how a classifier is designed in practice. Linear, quadratic and piecewise classifiers are included. These are based on the expected vectors and covariance matrices of underlying probability distributions. In practice, these vectors and matrices are not given, and must be estimated from an available set of samples. Consequently, the designed classifier variates and the classification error becomes a random variable. Section 4 discusses how the number of available samples affects the classification performance, and also how to allocate the samples into design and test. In Section 5, nonparametric techniques are presented. They are needed in the estimation of the Bayes error and the structure analysis of data, where a mathematical formula such as Gaussianness cannot be applied. Both the Parzen and Ic nearest neighbor approaches are discussed. Feature extraction and clustering are discussed in other chapters. Keywords: Statistical pattern recognition, classifier, probability of error, hypothesis tests, effect of sample size, nonparametric.
1. Introduction The purpose of statistical pattern recognition is to determine to which category or class a given sample belongs. Through observation and measurement processes, we obtain a set of numbers which make up the measurement vector. The vector is a random vector and its density function depends on its class. The design of a classifier consists of two parts. One is to collect data samples from various classes and to find the boundaries which separate the classes. This process is called classifier design, training, or learning. The other is to test the designed classifier by feeding the samples whose class identities are known. Figure 1 shows a flow chart of how a classifier is designed [I]. After data is gathered, samples are normalized and registered. Normalization and registration are very important processes for a successful classifier design. However, different data require different normalization and registration, and it is difficult to discuss 33
34
K. Fukunaga Y
DATA GATHERING
---
-I-
REGISTRATION
.-----
1 SEARCH FOR NEW MEASUREMENTS
ERROR ESTIMATION
NONPARAMETRIC PROCESS
ERROR ESTIMATION (NONPARAMRRICI DATA STRUCTURE ANALYSIS
7
$. FEATURE EXTRACTION CLUSTERING STATISTICAL TESTS MODELING LINEAR CLASSIFIER QUADRATIC CLASSIFIER PlECEWlSE CLASSIFIER NONPARAMETRICCLASSIFIER
PARAMETRIZATION PROCESS ERROR ESTIMATION
Fig. 1. Process of designing a classifier. (From [l],reprinted with permission.)
these subjects in a generalized way. Therefore, these subjects are not included in this chapter. After normalization and registration, the class separability of the data is measured. This is done by estimating the Bayes error, the overlap among different class densities, in the measurement space. Since it is not appropriate a t this stage to assume a mathematical form for the data structure, the estimation procedure must be nonparametric. If the Bayes error is larger than the final classifier error we wish to achieve (denoted by E O ) , it means the data does not carry enough classification information to meet the specification. Selecting features and designing a classifier in the later stages merely increase the classification error. Therefore] we must go back to data gathering and seek better measurements. Only when the estimate of the Bayes error is less than E O , may we proceed to the next stage of data structure analysis in which we study the characteristics of the data. All kinds of data analysis techniques are used here. They include feature extraction, clustering] statistical tests, modeling, and so on. Note that each time a
1.2 Statistical Pattern Recognition 35 feature set is chosen, the Bayes error in the feature space is estimated and compared with the one in the measurement space. The difference between them indicates how much classification information is lost in the feature selection process. Once the structure of the data is thoroughly understood, the data dictate which classifier must be adopted. Our choice is normally either a linear, quadratic, or piecewise classifier, and rarely a nonparametric classifier. Nonparametric techniques are required in off-line analyses to carry out many important operations such as the estimation of the Bayes error and data structure analysis. However, they are often too complex for any on-line operation. After a classifier is designed, the classifier must be evaluated. The resulting error is compared with the Bayes error in the feature space. The difference between these two errors indicates how much the error is increased by adopting the classifier. If the difference is unacceptably high, we must re-evaluate the design of the classifier. At last, the classifier is tested in the field. If the classifier does not perform as was expected, the database used for designing the classifier is different from the test data in the field. Therefore, we must expand the database and design a new classifier. In this chapter, only the boldfaced portions of Fig. 1 will be discussed briefly. More details are found in [I]. Clustering and feature extraction are discussed in other chapters. Also, unless otherwise stated, only the two-class problem is discussed in this chapter, although the results for a more general multi-class problem are listed whenever available. The notations, which are frequently used in this chapter, are summarized as
Dimensionality Number of classes Number of total samples Number of class i samples Class i A priori probability of wi Vector Random vector Conditional density function of wi Density function A posteriori probability of wi given X Expected vector of wi Expected vector Covariance matrix of wi Covariance matrix
36
K. Fukunaga
2. The Bayes Classification In this section, the classification algorithms and the resulting errors are presented, based on the assumption that pi(X) and Pi are known. The classification algorithms are also known as hypothesis tests. 2.1. Likelihood Ratio Classifier
The probability of the classification error can be minimized by classifying X into either w1 or w2, depending on whether q1(X) > q2(X) or q1(X) < q2(X) is satisfied. That is, qi(X)
qz(X)
(Bayes classifier) .
W2
The resulting risk at X is
r * ( X ) = min[ql(X), q2(X)]
(Bayes risk).
(2.2)
The overall error is obtained by taking the expectation of (2.2) over X: E* = E{r*(X)} =
Pi
p2(X)dX
sr2
(Bayes error)
s,
where ~1 = p l ( X ) d X and ~2 = pz(X)dX are called the w1 and respectively. ri is the region where X is classified to wi. For the multiclass problem, qk(X) = max 2 q,(X) + X E E* =
w2
(2.3) errors,
(2.4)
Wk
-L
E 1 - max q,(x)}.
(2.5)
A more convenient form of the Bayes classifier is obtained by applying the Bayes theorem, q2(X)= P$pp(X)/p(X),and taking negative logarithm: h(X) = - ln[pi(X)/pz(X)]
3 ln[Pi/P2]
(2.6)
W2
The h ( X ) combined with a threshold is called the likelihood ratio classifier. When X is distributed Gaussianly with M , and C, for w,, 1 -In pi(X) = - ( X 2
-
1 Mi)TC;l(X - M i ) + - In lCil 2
+
(:)
1n27-r.
(2.7)
The threshold of the classifier could be changed according to various requirements as follows: Bayes classifier for minimum cost. Let c j i be the cost of classifying a wi sample into wj. The expected cost of classifying X into wi is
1.2 Statistical Pattern Recognition 37 The classification rule and the resulting cost are ck(X) = min z ci(X) + X E wk
(2.9) (2.10)
For the two-class problem, w1
h(X)
= - ln[pl(X)/p2(X)]
5 ln[(c12 - C I I ) ~ ' I / ( C ~ I - c22)P2].
(2.11)
w2
This is a likelihood ratio classifier with a new threshold. Neyman-Pearson test. Let ~1 and ~2 be the error probabilities from w1 and w2, as shown in (2.3). The likelihood ratio classifier minimizes ~1 subject to ~2 being equal to a given constant, say, EO. The threshold value must be selected t o satisfy ~2 = EO and is normally determined empirically. A plot of ~1 vs. E Z for the likelihood ratio classifier with varying threshold is called the operating characteristics and is used frequently as a visual aid to see how two errors are traded by changing the threshold. In the Neyman-Pearson test, ~2 = EO is the operating point and the corresponding threshold value is chosen. When w2 represents a target t o be identified against the other class ( w l ) , E I , E Z , and 1 - ~2 are called the false alarm, the leakage, and the detection probability, respectively. Minimax test. We can make t,he expected cost invariant even when Pa varies unexpectedly after the classifier has been implemented. This is done by selecting the threshold of the likelihood ratio classifier to satisfy (c11 - c22) i- (c12 - C 1 1 ) E l
- (C2l -
C22)&2
=0
f
(2.12)
Particularly, when c11 = c22 and c12 - c11 = c21 - c22, the threshold is chosen t o satisfy ~1 = ~ 2 .This classifier eliminates the possibility of having an unexpected large error due t o the unexpected variation of Pi, In all of these three cases, the likelihood ratio classifier is commonly used, and only the threshold varies. This may be interpreted as replacing the true Pi's of (2.6) by artificial Pi's. Therefore, theoretically, all these cases may be treated as the Bayes classifier, assigning the different meaning to Pi for each application. Some other subjects related to hypothesis tests are as follows. Independent measurement set. When X consists of statistically independent measurement sets as X T = [XTXr.. . XL], the Bayes classifier becomes
This suggests how to combine, for classification, seemingly unrelated information such as radar and infrared signatures.
38
K. Fukunaga
One-class classifier. When one clearly defined class is classified against all other (sometimes not well-defined) possibilities, the boundary may be determined from the knowledge of one class only. A typical example is a hyperspherical boundary around a Gaussian distribution with M = 0 and C = I. This technique could work when the dimensionality of the data, n, is very low (such as 1 or 2). However, as n increases, the error of this technique increases significantly. The mapping from the original n-dimensional space to a one-dimensional distance space destroys valuable classification information which existed in the original space. For an example with n = 64, the error increases from 0.1% t o 8.4%. Reject. When X falls in the region where the Bayes risk r * ( X ) is high, we may decide not to classify the sample. This concept is called reject. The reject region r,(t) and the resulting probability of rejection R ( t ) are specified by the threshold
tas
r,(t) = { X : r * ( X ) > t } R(t)= Pr{r*(X) > t } = 1 - Pr{r*(X) 5 t } .
(2.14) (2.15)
Note that P r {r*(X) 5 t } is the distribution function of a random variable r * ( X ) . The probability of error with reject is the integration of r * ( X ) p ( X )in outside r,, and thus depends on t. The error may be evaluated directly from the reject probability as
r,,
1 two. t
4t)= -
(2.16)
The error decreases as the reject probability increases, and vice versa. A plot of ~ ( tvs.) R ( t ) is called the error-reject curve, and is used as a visual aid to see how E ( t ) and R(t) are traded by changing the threshold t. With the largest possible t = 1 - 1/L for the L-class problem, R(1- 1/L) = 0 and ~ ( 1 -1/L) is equal t o the Bayes error,
E*.
Model validation. The distribution function of the random variable r* (X) is a simple and good parameter to characterize the classification environment, determining both E ( t ) and R ( t ) .Thus, when samples are drawn and a mathematical model is assumed, two distribution functions of r * ( X ) may be obtained: one empirically from the samples and the other theoretically from the model. The comparison of these two distribution functions could be used to test the validity of the mathematical model. 2.2. The Bayes Error The Bayes error of (2.3) is generally hard to compute, except for the following two cases. (1) Gaussian X with C1 = Cz = C. For this case, the Bayes classifier becomes a linear function of X as 1 h ( X ) = (Mz - M I ) ~ C - ~ X- ( M T C - ' M i - M,TC-'M2) 6 t . (2.17) 2
+
1.2 Statistical Pattern Recognition 39 Since X is Gaussianly distributed, h ( X ) is also a Gaussian random variable. Therefore,
where ml = E { h ( X ) l w l } =
m2
= E{h(X)Iwz} =
g: = Var{h(X)Iwi} =
-i1 (M2 - M l ) T C - 1 ( M 2
-
+-21 (M2
-
M1)
MI)~C-~ - M1) (M~
( M~ M ~ ) ~ c - ~- (M IM) ~ ( i = 1, 2 ) .
(2.19) (2.20) (2.21)
(2) Gaussian X with C1 # C2. The Bayes classifier for this case is 1 1 h ( X ) = - ( X - Ml)TC,l(X - M I ) - - ( X - M2)TC,1(X - M2) 2 2 (2.22) The distribution of h ( X ) is no longer Gaussian, but the errors for a general h ( X ) can be expressed as follows:
where the unspecified integral regions are the entire domain for X and [-m, +co] for w. Particularly, when h ( X ) is the quadratic function of (2.22) and p i ( X ) ’ s are Gaussian, we can integrate (2.23) and (2.24) explicitly with respect to X to obtain
(2.25)
(2.26) where (2.27)
L&(w)
=
1 1 tan-’(ujiw) - - w[cji 2 2
-
+ (ujib’$w2)/(1 + w2u’$)]
(2.28) (2.29) (2.30)
40
K. Fukunaga
The A’S are the diagonal components of A which are obtained by simultaneously diagonalizing C1 and Cz as
AT&A = I and AT&A
=A
(2.31)
and the (dzi - d l i ) is the i t h component of the vector AT(Mz - M I ) . Equations (2.25) and (2.26) must be integrated numerically, but they are one-dimensional integrations.
Upper and lower bounds. The computation of the Bayes error is very complex unless h ( X ) is linear. Furthermore, since a numerical integration is involved, the Bayes error cannot be expressed explicitly. An alternative is to use the upper and lower bounds of the Bayes error as a measure of class separability. Some of popular bounds are listed as follows: E{yl(X)yZ(X)} :
2 nearest-neighbor error
E{min[ql(X) , yz(X)]} : 2E{yl(X)yz(X)} :
Bayes error
Nearest-neighbor error
2k2E{yi(X)ln yl(X) + yz(X)lnyz(X)} :
--
E{}-/,
:
Bhattacharyya bound
(2.32) (2.33) (2.34)
Equivocation
(2.35)
(2.36)
The inequalities (2.32) 5 (2.33) 5 (2.34) 5 (2.35) 5 (2.36) hold regardless of the distributions. One of the popular bounds is the Bhattacharyya bound, which has an explicit expression for Gaussian distributions: (2.37)
The first term of (2.38) indicates the class separability due to the mean difference, and the second term gives that due to the covariance difference. When the distributions are non-Gaussian, (2.37) with (2.38) is no longer guaranteed to bound the Bayes error. Still, in order to use (2.38) as an effective class separability measure, one may transform each variable to a Gaussian-like one. For example, power transforms, yi = xy (i = 1, . . . , n ) , may be used for causal distributions. Such a variable transformation is useful not only for measuring the class separability, but also for designing a better classifier. Designing the Bayes classifier for Gaussian distributions, even with additional variable transformations, is often easier than designing the Bayes classifier for non-Gaussian distributions.
Scatter measures. The bounds discussed above are valid only for two class problems. There exist no well accepted extensions of the above bounds t o multiclass. Scatter measures, introduced here as an alternative, are intuitively derived, simple, but not
1.2 Statistical Pattern Recognition 41 directly related to the Bayes error. They are for multiclasses. Let us define for L-class: L
Pi(hfi- M ) ( M i - hf)T Between-class scatter matrix
Sb =
(2.39)
i=l
L
Pi&
S, =
Within-class scatter matrix
(2.40)
i= 1
s, = E{(X
-
M)(X - k f ) T= } s b f sw Mixture scatter matrix
(2.41)
The class separability can be measured by the combinations of these matrices: trSLlSb,
t r SL'S,,
In ~S~'swl etc.
(2.42)
where trace and log-determinant are for converting a matrix to a number. The first and second ones are often used for feature extraction and clustering respectively. All combinations, SG'Sb, s;'Sw, sk'sb etc., share the same eigenvectors, and their eigenvalues are well related. The first one of (2.42) measures class separability based on the scatter of class means, normalized by S,, and the others are the variations. Therefore, the scatter measures can be applied only for the cases where classes are separated mainly by mean-difference. There are no measures available for multiclasses mainly separated by covariance-difference.
3. Classifier Design Once the structure of data is studied thoroughly, it is easy to select a proper classifier for the data. This section presents how several typical classifiers can be designed. 3.1. Linear Classifiers
The Bayes classifier becomes linear for the following two cases. (1) Gaussian X with C1 = Cz = C: For this case, the Bayes classifier is expressed by (2.17), which is linear. In particular, when C = I ,
h ( X ) = (MZ - M1)TX
+ 51 ( M T M 1 -
M,TM2) 5 t .
(3.1)
This classifier is also known as the distance classifier in which ( X - M 1 ) T ( Xw1
w1
w2
W2
M I ) g ( X - M z ) ~ ( -XM z ) , or the correlation classifier in which M F X 2 M T X with an energy constant condition MFM1 = MFMz. In both cases, PI = Pz and thus t = 0 is assumed. When C # I , the distance or correlation classifier may still be applied, but only after X is linearly transformed to Y = ATX in order to make C y = ATCA = I . (2) Binary independent x,'s: For independent xJ7staking either +l or -1, n
(1+2,)/2
p z ( X )=
wy j=1
(1 - wz3)(1-z')/2[s(s, - 1)
+ b(z, + l)]
(3.2)
42
K. Fulcunaga
where wiJ
= Pr{zj = fllwi}.
Substituting (3.2) into (2.6),
For Gaussian X with C1 # C2 and more general non-Gaussian X, a linear classifier is not the best one. However, because of its simplicity and robustness, a linear classifier is frequently adopted. The design procedure is as follows. W1
h ( X )= V T X 5 t
(3.4)
w2
Equation (3.4) indicates that X is linearly mapped down t o a variable h and the distributions of h for w1 and w2 are separated by a threshold t. Thus, the optimum V is found by minimizing the probability of error in the h-space. Because of complexity in the error computation, simpler criteria such as f ( m l , 77x2, g f , a;) are often used, where mi = E{h(X)lwi} = VTMi and ~7: = Var{h(X)lwi} = VT&V. The typical examples are
f=
Pl(m1 - may + P2(m2 - m0l2 PI.-::
+P 2 4
between class scatter within class scatter
)
(3.6)
+
where mo = P1ml P2m2 is the mixture mean. These criteria measure the class separability of the distributions of h. The solution of d f /dV = 0 is
where
That is, the optimum V always takes the form of (3.7) regardless of the functional form of f . The effect of f is observed only in the averaging coefficient of covariance matrices, s. For example, s = 0.5 for (3.5) and s = PI for (3.6). V can be found even without specifying f . Since the form of V is known as in (3.7), we change s from 0 to 1 with a certain increment, say, 0.1, compute the empirical distribution functions of h = V T X for w1 and w2 from the given data set, select the value of the threshold, and count the number of misclassified samples. The optimum s is the one which gives the smallest error in this operation. The Bhattacharyya bound of (2.38) gives a simple test t o decide whether or not a linear classifier is appropriate. When the first term of (2.38) is dominant, the classifiability comes mainly from the mean difference. Therefore, a linear classifier is
1.2 Statistical Pattern Recognition 43
a proper choice. However, if the second term is significant, the covariance difference plays an important role, and a quadratic classifier is called for. 3.2. Quadratic Classifiers
For Gaussian X, the Bayes classifier becomes quadratic, as shown in (2.7) or (2.22). In practice, the quadratic classifier of (2.22) is widely adopted in many applications, even without checking the Gaussianness of X, and with much success. Probably, this is the classifier everyone may try first, even before conducting data structure analysis. However, it is not known how to design the optimum quadratic classifier for non-Gaussian distributions, as the linear classifier was designed. The optimization of f(m1,m2, c;,c;) for h = X T Q X + VTX with respect to a matrix Q and a vector V is too complex. If quadratic terms xJxk's are treated as new variables y,, h = CCqJkxJxk C v , x , becomes a linear equation as h = Ca,y, Cv,x,. However, for high-dimensional cases, the number of y z ' s becomes prohibitively large.
+
+
Two-dimenszonal display. One of the procedures used to improve the performance of the quadratic classifier is t o plot X in a two-dimensional display where d : ( X ) = ( X - Mz)TC,l ( X - M,) for i = 1, 2 are used as the J: and y axes. If X is Gaussian, the Bayes classifier becomes a 45" line with a proper y-crossing point. When the distribution is not perfectly Gaussian, we can observe in the display that the 45" line is not the best boundary to minimize the number of misclassified samples. Then, visually we can find a better line to classify samples by changing the slope and the y-crossing point of the line. It corresponds to adjusting (Y and p of the following quadratic classifier:
Once samples are plotted and examined, the boundary in the display need not be restricted to a line. Any curve could be drawn. This flexibility is the advantage of seeing the data on the display.
Fourier transform. When a stationary random process is time-sampled, the coefficients of the discrete Fourier transform are uncorrelated, and its covariance matrix becomes diagonal. Thus, the quadratic classifier of the Fourier coefficients y , is reduced to h = Cr=,(q,ly,12 v,y,) vo. This is the Bayes classifier, if y,'s are Gaussian.
+
+
Approximation of covariance matrices. If we can assume a structure for a covariance matrix, we can simplify the design of a quadratic classifier. In addition, the classifier becomes less sensitive to the parameter variation due to the estimation process by using a finite number of design samples. This will be discussed in the next section. One of the possible structures is the toeplitz f o r m for the correlation matrix, allowing each variable to have its distinct variance. In this case, parameters must be selected to assure that the toeplitz matrix be positive-definite. In particular, when
44
K. Fukunaga
the correlation coefficient between the ith and j t h variables, ~ i j is, approximated by pli-jl ( p : a constant), the entire correlation matrix is characterized by one parameter, p, and its determinant and inverse matrix may be expressed explicitly.
3.3. Piecewise Classifiers For the multiclass problem, the boundary must have a piecewise structure as follows.
Piecewise quadratic classifiers. If X is Gaussian, the classifier becomes, from (2.7), piecewise quadratic:
The first term of (3.10) is widely used even for non-Gaussian distributions. However, it must be noted that the normalized distance of X from each class mean, M i , must be adjusted by two constant terms, In [Cil and 1nPi.
Piecewise linear classifiers. When Xi's are similar, the quadratic term X T C i ' X is eliminated from (3.10) to get a piecewise linear classifier:
Or, replacing Ci by the averaged covariance, C = (Cl
+ . .. + C L ) / L , (3.12)
Another possibility is to design the optimal linear classifier for each pair of classes. In this case, L(L - 1)/2 classifiers must be designed, instead of L in (3.11) or (3.12).
Clustering. In some applications, each class distribution would be handled better by dividing it into several clusters. For example, take the signatures of a target viewed from the front and side. Since they are so different, it may be more appropriate t o separate them into several clusters rather than to treat all of them as one class. Considering each cluster as a class, we can form a new multiclass problem with a significantly increased number of classes. However, the details of designing such a classifier depend very much on how clusters are defined and obtained and how many classes are generated. Therefore, although important, the subject is not discussed in this chapter.
IC nearest neighbor (ICNN). The ICNN classifier forms a piecewise linear boundary, although it is very complex and data dependent. The simpler boundary could be obtained by merging samples into a smaller number of representatives and then applying the ICNN classifier t o these representatives.
1.2 Statistical Pattern Recognition 45
3.4. Sequential Classifiers When m consecutive observation vectors, XI, . . . , X,, are known as coming from the same class, we can use this additional information to reduce the classification error. That is, the number of variables is extended from n for one vector t o m x n for m vectors. Thus, we can form a new random vector with m x n components and design a classifier in the (m x n)-dimensional space. However, when these vectors are statistically independentl a simpler formula could be adopted:
That is, the likelihood ratio classifier is applied t o the incoming sample X i , and the output is accumulated. Rewriting the left-hand side of the inequality as (3.14)
the expected values and variances of s and h E{s(wi} = E{h(wi} and
= h(X)
are related by
1 Var{s(wi} = - Var{h(wi}. m
(3.15)
Thus, we can reduce the variances of s by increasing m, while maintaining the expected values of s. Furthermore, the density function of s becomes close to a Gaussian by the central limit theorem. Two important properties of the sequential classifier emerge from the above discussion. One is that we can make the error as small as we like by increasing m. The other is that the error is determined by a small number of parameters, E{hlwi}, Var{hlwi}, and m, and is little affected by the higher-order moments of h. In practice, the true p i ( X ) ’ s are never known, and h ( X ) = - l n [ p ~ ( X ) / p z ( X ) ] must be replaced by some function i ( X ) . A desired property for i ( X ) is E{k(X)(w1} 5 0 and
E{L(X)Iwz}
2 0.
(3.16)
regardless of the distribution of X . As long as (3.16) is satisfied, the random variable i ( X ) carries classification information, however small, regardless of the distribution of X. The classifiable information can be enhanced as much as we like by increasing m in the sequential operations. Two k ( X ) ’ s are known to satisfy (3.16) for all distributions of X, whose expected vectors and covariance matrices are A41 and C1 for w1 and A42 and Cz for w2: 1
- (M,TC-lA41-
2
- MI)
1 (X 2
- -
-
A4,Tc-’A42) A42)TC,1(X
-
(3.17) A42)
(3.18)
46
K. Fukunaga
Any positive-definite matrix C in (3.17) satisfies (3.16). But, the averaged covariance matrix such as the one in (3.7) would be a better C to achieve the same performance with a smaller m. Equation (3.17) could be used, if the first term is dominant in the Bhattacharyya bound (2.38), but (3.18) is more appropriate otherwise. Note that these equations are the same as (2.17) and (2.22), respect ively. One of the most important aspects in classifier design is t o make the classifier robust. That is, the performance of the classifier must be maintained, even if the distribution of test samples becomes somewhat different from the one used for design. The sequential technique can compensate the degradation of the performance of h ( X i ) 5 t by increasing m. 4. Estimation of Classification Errors
So far, we have discussed the design of parametric classifiers, assuming that Mi and Ci are given. In practice, these parameters are estimated by using a finite number of available samples, and the estimates are random variables. Consequently, the classifier designed with these estimates variates, and its performance is also random. Therefore, it is important to know how the sample size affects classifier design and performance. 4.1. Eflect of Sample Size o n Estimation
General formula. First let us consider the problem of estimating f = f (y1, . . . , yq) by f = f (yl, . . . , y q ) ,where f is a given function, the yi's are the true parameter values, and yi's are their estimates. When the deviation of yi from yi is small, f may be expanded by a Taylor series as follows:
where Ayi
= yi
-
yi. If the estimates are unbiased,
In most parametric cases, the yi's are the components of M , and C, ( r = 1, 2), and Y T = [ y l . . . yq] can be expressed as
Y T = [ml(1)
7
m(l) " ' 7
7L
7
(1) J1) (2) m,(2) 7 .. ., m,(2) 1 c11 > . .. , n7L7 c11, . . . ,
?.A]
(4.4)
where mi') and cii) (i 5 j ) are the components of M , and C, respectively. Their unbiased estimates are obtained by the sample m e a n and sample covariance matrix
1.2 Statistical Pattern Recognition 47 as
where X i T ) is the ith sample from w,. When X:')'s are drawn from Gaussian distributions, E{AyiAyj} for the y;'s of (4.4) are known, and (4.2) and (4.3) become
(4.7) where both C1 and Cz are assumed to be diagonal with ( r = 1, 2) as their diagonal components. Without loss of generality, any two covariance matrices can be simultaneously diagonalized. Also, N1 = N2 = N is assumed for simplicity. Note that both the bias of (4.6) and the variance of (4.7) are proportional to 1/N. The other terms are determined by the underlying distributions. This is true even when XiT)'sare drawn from non-Gaussian distributions. Estimation procedure o f f . Equation (4.6) can be rewritten as U
E{f}" f + - N .
(4.8)
This equation suggests the following procedure t o estimate f : (1) Change N to N1, . . . , Ne. For each Ni, compute MTand 2,and subsequently f . Repeat this T times independently, and approximate E{f} by the sample mean of the 7 results. (2) Plot the empirical points of E{f} vs. 1/N. Then, find the line best fitted to these points. The slope of the line is u , and the y-crossing point is the estimate off.
Bhattacharyya distance. The Bhattacharyya distance of (2.38) is a function of M , and C,. Thus, the bias of (4.6) can be further reduced by computing the partial derivatives of this function. Treating the first and second terms separately, r
48
K. Fukunaga
where M I = 0, M2 = [ml . . . m,IT, Xi1) = 1 (A1 = I) and Xi2) = X i (A2 = A) are used without losing generality. For example, when mi = 0 and X i = 1, E{Apl} %’ n/(4N) and E{Ap2} n(n 1)/(8N). This is the case where the Bhattacharyya distance is measured between two sample sets generated from the same Gaussian distribution. Although p1 = p2 = 0 for N = 00, a finite N creates the biases. Note that E{Ap1} is proportional to l / k ( k = N / n : ratio of sample size and dimensionality) while E(Ap2) depends on (n+l ) / k . In order to maintain the same amount of bias (E{Ap} = E{Apl} E{Ap2}), a larger k must be chosen as n increases. For example, t o meet E{Ap} 5 0.223, k must be larger than 6.2 and 39.6 for n = 8 and 64 respectively. The variances of (4.7) also can be computed similarly.
+
+
4.2. Estimation of Classification Errors
The classification errors of (2.23) and (2.24) are the members of the family of functions presented in (4.1) and (4.4), when h ( X ) and p , ( X ) are functions of M , and C,. However, in this case, the randomness comes from two sources: the finite design-sample set to make h ( X ) random and the finite test-sample set to make pi(X) random. Since these two affect the error differently, we need t o discuss their effects separately.
Effect of test samples. When a finite number of samples is available for testing a given classifier, a n error-counting procedure is the only feasible possibility in practice. That is, each sample is tested by the classifier and the number of misclassified samples is counted. Then,
Et(6,) = E ,
and
Vart{h,} =
Er(1 -
Nr
~ r )
(7- =
1, 2)
(4.11)
where Et and Vart indicate that the expectations are taken with respect to test samples, and N, is the number of test samples from w,. This is a n unbiased estimate. Furthermore, (4.11) is valid regardless of functional forms for h ( X ) and Pr(X).
Effect of design samples. When a finite number of design samples is used to compute M,, 9, and then h ( X ) , Ed{&.} and Vard{i,} can be obtained through (2.23) and (2.24) for given test distributions, p , ( X ) , where Ed and Vard indicate the expectations with respect to design samples. The resulting bias is
Ed{&} 2 E
+
v +N
(4.12)
where E = P1~1 P ~ E and z , v is determined by the underlying distributions of design samples, given test distributions and the functional form of h ( X ) . The number of design samples is denoted by N (= N1 = N2) and distinguished from the test sample size N . Although v is a complicated function and can be computed
1.2 Statistical Pattern Recognition 49 explicitly only for simple special cases, v can be obtained empirically by the estimation procedure of (4.8). When h ( X ) is the quadratic classifier of ( 2 . 2 2 ) with C1 = C2 = I and PI = Pz, v becomes 1
vq 2
4
J
2-1
m
(4.13) where M = M2 - M I . On the other hand, when h ( X ) is the linear classifier of (2.17) with C1 and PI = P2, v becomes
ve =
1 4JGimiT
e--MTM/8
[
(1
--+
+ MTM
- 1]
.
=
C2 = I
(4.14)
When C1 = C2, the quadratic h ( X ) of (2.22) becomes the same as the linear h ( X ) of (2.17). However, when the estimated covariance matrices, 21 # 2 2 , are used, h(X) of (2.22) differs from that of (2.17). As a result, E{AE} for the quadratic classifier is proportional to n 2 / N while E{&} for the linear classifier tends t o n / N when n gets large. This implies that many more samples are needed to properly design a quadratic classifier than a linear classifier. More generally, (4.6) suggests that the bias could be proportional to n2 because of the double summation of the last term. This is due to the fact that n2 correlations are estimated. This number could be significantly reduced, if we assume a structure of the covariance matrix and estimate a smaller number of parameters. As for Vard{i}, we have for the Bayes classifier Vard{d}
(4.15)
0:
N
otherwise
Effect of independent design a n d t e s t samples. When both design and test samples are finite and they are independent, the bias and variance of 2 are
E{d}
(4.16)
Z Ed{€}
(4.17) where E and Var indicate that the expectations are taken with respect t o both design and test samples. Note that Vart{d,} of (4.11) is obtained from the first term of (4.17) by replacing Ed{&} by E,. Since Ed{&,} Z E , v,/N, we can conclude as follows:
+
(1) the bias of the classification error comes entirely from the finite design set, and ( 2 ) the variance comes predominantly from the finite test set.
50
K. Fukunaga
4.3. Holdout, Leave- One- Out, and Resubstitution Methods When only one set of samples is available and the performance of the specified classifier is to be estimated, we need to decide how to divide the samples into two groups, design and test.
Upper and lower bounds of the Bayes error. In general, the classification error is a function of two sets of data, the design set PD and the test set PT, and may be expressed as E ( P DP , T ) where E is an operator to compute the Bayes error. We assume that both PD and PT are drawn from the same set of underlying distributions P = { p l ( X ) , p 2 ( X ) } . If P is used for design, the resulting classifier is the Bayes which produces the Bayes error by testing P . That is, the Bayes error is expressed by & ( P P , ) . Letting and +2 be two different sets of samples independently drawn from P , E ( P ,P ) can be bounded as (4.18)
The rightmost term indicates that, as long as design and test sample sets are independent, the resulting error is larger than the Bayes in expectation with respect to test samples. The leftmost term suggests that, if the same set is used for both design and test, the resulting error is smaller than the Bayes in expectation. These procedures are called the holdout (H) and resubstitute (R) methods respectively. The H method works well, if many data sets can be generated by a computer. However, in practice, with only one set of data, we need t o divide the available data into two independent groups. This reduces the number of samples available for each of design and test. Also, it must be assured that the distributions of design and test samples are close. Another problem is how to allocate samples to design and test. This is normally done by balancing the bias due t o the design sample size and the variance due to the test sample size. The leave-one-out (L) method alleviates the above difficulties of the H method. In this method, one sample is excluded, the classifier is designed on the remaining N - 1 samples, and the excluded sample is tested by the classifier. This is repeated N times to test all N samples. The number of misclassified samples is counted to obtain the estimate of the error. Since each test sample is excluded from the design sample set, the design and test sets are independent. Also, since all N samples are tested and N - 1 samples are used for design, the available samples are more effectively utilized. Furthermore, we do not need t o worry about dissimilarity between the design and test distributions. Although the L method requires N classifiers (one for each sample), these classifiers may be computed with little extra computer time as the perturbations from the classifier designed from N samples. This will be shown for the quadratic classifier next and for nonparametric cases later.
R and L methods for the quadratic classifiers. In the R method, all available samples, X J T )( r = 1, 2 ; i = 1, . . . , N T ) ,are used t o compute MT and 2, of (4.5).
1.2 Statistical Pattern Recognition 51
Then, the same samples are tested as to whether or not the following inequality is satisfied:
for testing
Xk
E { X (1) , , .. ., X N ( 11 ) ,XI ( 2 ), . . . , X N ( 2 )Z } .
(4.19)
Then, the number of misclassified samples is counted. This error is supposed t o be smaller than the true error of the classifier. On the other hand, in the L method, X i 1 ) E w1 is excluded from the computations of k 1 and 21, and the modified k l and 51 are used in (4.19) to test X:'). Similarly, k 2 and 2 2 are modified for testing X r ) E w2. The resulting quadratic equation ~ L ( X I becomes ,)
where
and
d $ (X,) = ( X ,
- k,)T2,1(XI,
-
k,) .
(4.22)
Equation (4.20) indicates that, for X I , E w1, h ~ ( X k is) larger than h ~ ( X k )and , the chance of X I , being misclassified is increased. The same is true for XI, E w2. Therefore, the L error is always larger than the R error. When the R method is used to count the error, h ~ ( X kand ) d z ( X ~ ,()r = 1, 2) must be computed for all (N1 N2)Xk's. The L method requires an additional computation of (4.21) for each 5. However, since (4.21) is a scalar function, the computation time for this part is negligibly small. Thus, when h ~ ( X kis) computed and tested for each XI,, h ~ ( X kis) also computed and tested at the same time with little additional computer time.
+
5. Nonparametric Procedures As Fig. 1 shows, nonparametric procedures are necessary for error estimation in both measurement and feature spaces and data structure analysis before a
52
K. Fukunaga
parametric structure of the data is determined. Nonparametric procedures are based on the estimation of a density function without assuming any mathematical form.
5.1. Estimation of a Density Function There are two approaches for density estimation: one is the Parzen approach and the other is the k-nearest neighbor (NN) approach. They have similar statistical properties with minor differences.
Parzen density estimate. When N samples X I , . . . , X N , are given, but no mathematical form can be assumed for the density function, the value of the density function at X may be estimated by p ( x )=
1
c N
Ic(X
-
Xi)
i=l
where K ( . ) is called a kernel function. In practice, selection of the kernel function is limited to either Gaussian or uniform particularly in a high-dimensional space. A more general form for K ( . ) is
(5.2) where I?(.) is the gamma function. When m = 1 and 00, K ( . ) of (5.2) are reduced to the Gaussian and uniform kernels respectively. The matrix A , which is called a metric, determines the shape of the hyperellipsoid, and r controls its size (both in the uniform case). Otherwise coefficients are selected to satisfy two conditions: tc(X)dX = 1 (which is required to satisfy p ( X ) d X = 1) and X X T l c ( X ) d X = r 2 A (the covariance matrix of .(X)). The bias and variance of (5.1) are
s
s
s
The control parameters of the Parzen density estimate are m, r , A and N and their optimal choices could be found by minimizing E{ [ p ( X )-p(X)]’} which is (Bias)2+ Var. However, the optimal selection of parameters for density estimation does not coincide with the one for classification.
kNN density estimate. In this approach, the kth N N sample of X is found, and the distance d k (or the corresponding volume U k ) is measured, where d i = (XkNN X ) T A - ’ ( X k ~-~X ) , uk = CIA^^/^^;, A is a metric, and c i s a constant. Then, the
1.2 Statistical Pattern Recognition 53 density estimate a t X is k-1 (5.5)
p(x) = NVk(X)
where vk is a random variable and the function of X . Defining u as the probability of a sample falling within ?&, the density function of u is known as (5.6) Since u g u k p ( X ) for a small ?&, the density function of vk can be computed from (5.6). Thus, the bias and variance of the k N N density estimate are also obtained, resulting in
[ +a
E{p(X)} 2 p(X) 1
Var{p(X)}
- tr
':rv;
{ -A }
(5.7)
(Ncl:l&x))2/n]
P2 (XI
E -.
k-2
The term (.)'//" of (5.7) is d; by (5.5). Also, Var {p(X)} of (5.4) becomes p2(X)/k for a uniform kernel in which w is l / w . That is, the biases and variances of the Parzen and k N N density estimates are very similar. The k N N density estimate could be considered as the Parzen one with a uniform kernel whose kernel size is adjusted by p(X). The control parameters of the k N N density estimate are k , A and N , and their optimal choices are found by minimizing E{p(X) - p(X)I2}.
Moments of the kNN distance. Since the density function of moments of d k can be computed, resulting in
Vk
is known, the
where the integrals for some distributions with the covariance matrix C are Gauss: Uniform:
(27r)m/21C1"/2"(1 -- m / n ) - n / 2
(zn-)m/2jCIm/2nr--m/n (1
+n/2)(1+ n / 2 1 ~ / ~ .
(5.10) (5.11)
When m / n is small in a high-dimensional space, and A is selected as A = C, E{dr} is determined predominantly by n and m. The effects of k and N are minimal as r ( k m / n ) 2 r ( k ) and r ( N 1 m / n ) 2 r ( N 1). Also, E{drlX} is computed by (5.9) without taking the integration, but is little affected by p(X) because of a small m/n in power. The variance of d k is very small and all dk's are close to the expected value.
+
+ +
+
Estimation of the local dimensionality. The ratio of two averaged k N N distances depends only on k and n, but not on N and p ( X ) as follows: (5.12)
54
K. Fukunaga
The n computed from d k ’ s and d k + l ’ s by (5.12) depends only on neighboring information, and thus indicates the local dimensionality (or intrinsic dimensionality). Generally, the dimensionality plays a dominant role in determining the statistical properties of any nonparametric estimation. For example, E{dy} of (5.9) with A = C is predominantly determined by n and m. However, it must be kept in mind that the n of (5.9) means the local dimensionality but not the global one.
Very large number of classes. When the number of classes is very large as in the hundreds, we may consider class expected vectors Mi (i = 1, . . . , L ) as random vectors drawn from a distribution, p ( M ) . The classification error between a pair of classes, wi and w j , is determined by the distance between Mi and Mj and the amounts of noises around these M’s. The overall error depends on how many neighboring classes contribute the error. On the other hand, the previous discussion indicates that, in a high-dimensional space, the k N N distance is not affected by k , L and p ( M ) . That is, each class is surrounded by many other neighboring classes with almost equal distances. Thus, almost-equal pairwise errors are added up to form the total error which could become large. Classification of a very large number of classes must be handled with special care. It is not enough to confirm that each pair of classes can be classified with a reasonably small error. 5.2. Classification
Parzen classifier. Substituting the Parzen density estimates into the likelihood ratio classifier of (2.6),
This is called the Parzen classifier and can be used to classify X , when a set of samples {Xi’), . . . , X N(1)1, X ,(2) , . . . , X N(2), } is given. Each class may have a distinct kernel function. In order t o find the upper and lower bounds of the Bayes error, we may adopt the resubstitution ( R ) and leave-one-out ( L ) methods for the Parzen classifier. In the R method, the same samples, Xi‘) ( r = 1, 2 ; i = 1, . . . , N‘), are tested by (5.13), and the number of misclassified samples is counted. On the other hand, when Xe(l) is tested in the L method, Xj’) is excluded t o form the Parzen density estimate of w1. Therefore, the numerator of (5.13) must be replaced by
The denominator stays the same as l;z(Xe( 1 )). Again, X,(’) (f? = 1, . . . , N1) are tested and the number of misclassified samples is counted. Note that the amount
1.2 Statistical Pattern Recognition 55 subtracted in (5.14), ~ 1 ( 0 )does , not depend on !. When an w2 sample is tested, the denominator of (5.13) is modified in the same way. It can be proved that P1L(Xj1))5 fil(Xjl))if K ( X )5 ~ ( 0 which ) is satisfied for the kernel functions of (5.2). Therefore, the tested sample has more of a chance to be misclassified in the L method than in the R method. Also, note that the L density estimate of (5.14) can be obtained from the R density estimate by simple scalar operations -subtracting ~ l ( 0 and ) dividing by N1 - 1. Therefore, the computation time needed to obtain both the L and R density estimates is almost the same as that needed for the R density estimate alone.
kNN classifier. Using the k N N density estimates of (5.5), the likelihood ratio classifier becomes
where us)= clA,I1/”dL)’” and dL)’(X) = ( X i : h N - X)TA-l(XL:,,N - X ) . That is, in order to test X , the k l N N from w1,X k(1)l N N ,and the k 2 N N from w2, X l(c2 )z N N , are found, the distances from X to these neighbors, dg’ and d t ) , are measured, and
d t ) and d g ) are inserted into (5.15) to test whether the left hand side is smaller or larger than t. For simplicity, kl = k2 = k is used in this chapter. Normally, the class covariance matrix is used for A, and therefore A1 # A2. The R and L methods of the k N N classifier are used to bound the Bayes error. In the R method, all samples are included in the list of design samples from which the k N N of the test sample is found, and the same samples are tested. When X,(l) is tested, X,(’) itself is the closest sample in the list. Therefore, dFll(Xjl)) is inserted into the denominator of (5.15) while df)(Xjl)) is inserted into the numerator. On the other hand, when Xi1) is tested in the L method, Xk’) must be excluded from the list of design samples. Therefore, d r ) ( X { ’ ) ) and d f ) ( X j ’ ) ) are compared. Since drjl(Xjl)) 5 d P ) ( X j l ) ) , Xk‘) has more of a chance to be misclassified in the L method than in the R method. Also, note that in order to find the k N N , the distances to all samples are computed and compared. When d r ) ( X j ’ ) ) is obtained, d,&(Xj’)) is also available. This means that the computation time needed to get both the L and R results is practically the same as the time needed for the R method alone. Similarly, for testing a w2 sample, X ,( 2 ), d f j l ( X j 2 ) )and d f ) ( X j 2 ) )are compared with d r ) ( X d ” ) in the R and L method respectively. Voting kNN classifier. Instead of selecting k N N from each class separately and comparing the distances, the k N N ’ s of a test sample are selected from the mixture of classes, and the number of neighbors from each class, ki, among the k N N is counted. The test sample is then classified to the class represented by a majority
56
K. Fukunaga
of the kNN’s. That is,
k,
= max(k1,
..., k L }
+
X E w,
(5.16)
In order to avoid confusion between these two k N N procedures, we will call (5.16) the voting k N N procedure and (5.15) the volumetric kNN procedure. For the voting kNN procedure, it is common practice t o use the same metric to measure the distances to samples from all classes, although each class could use its own metric. Since the kz’s are integers and a ranking procedure is used, it is hard to find a component of (5.16) analogous with the threshold of (5.15). It can be shown that, with t = 0 in (5.15), the volumetric k N N and voting (2k - 1)” procedures give identical classification results for the two-class problem using the same metric for both classes. For example, let k and (2k - 1) be 3 and 5 respectively. In the voting 5NN procedure, a test sample is classified to w1, if 3 , 4, or 5 of the 5”’s belong to w1. This is equivalent to saying that the third N N from w1 is closer to the test sample than the third N N from w2. In the voting k N N classification for the two-class problem, k must be odd. Otherwise, k l = kz could occur and we cannot decide which class the test sample is classified to. This problem may be alleviated by introducing the concept of rejection. That is, when kl = k2 occurs, the test sample is rejected, and not counted as misclassified one. As a result, the classification error becomes smaller than even the Bayes error. This happens because some of the to-be-misclassified samples are rejected and not counted as the error. The asymptotic ( N , = m) performance of the voting k N N is known. For the two-class problem, the risk of the voting k N N classification given X, r k ( X ) , is (5.17)
(5.18) where y 2 . When a syntactic deletion error has taken place, the corresponding semantic length is also deleted. A semantic deletion error can occur without a syntactic deletion error. For local deformations, a substitution error can take place. (5) A syntactic substitution error is defined as the replacement of primitive s by another primitive t , i.e. (s,
Y) + (tl Y) .
A semantic substitution error is not defined, because it may be counted as a semantic insertion or deletion error. 4.4. Inference of Attributed Grammar
An attributed context-free grammar is a 4-tuple G = (V,, Vt, P, S), where V, is the nonterminal set, Vt is the terminal set, P is a finite set of production rules, and S E V, is the starting symbol. In P, each production rule contains two parts: one is a syntactic rule, and the other is a semantic rule. Each symbol X E (V, U V,) is associated with a finite set of attributes A(X); A(X) is partitioned into two disjoint sets, the synthesized attribute set Ao(X) and the inherited attribute set Al(X). The syntactic rule has the following form:
where k means the kth production rule. The semantic rule maps values to the attributes of X ~ , O Xk,l, , X ~ J ,. . ,Xk,nk. The evaluation of synthesized attributes is based on the attributes of the descendants; therefore it proceeds in bottom-up fashion. On the other hand, the evaluation of inherited attributes is based on the attributes of the ancestors; therefore it proceeds in top-down fashion. To explain the inference procedure, let us consider the example of the previous string aaadgggggeeaaagg,
where each primitive has a unit length attribute 1. First it will be converted into the following string by merging identical primitives:
Let the upper case character be nonterminal and the lower case character be terminal. Then, we can infer the following attributed grammar.
1.3 Syntactic Pattern Recognition 75
Syntactic rules
Semantic rules
(1) S -+ ADGEAG (2) A + a A (3) A + a (4) D + dD (5) D +d (6) E + eE (7) E + e (8) G + 9G (9) G + 9 where L denotes the inherited length attribute, and y denotes the synthesized length attribute. The number right after the nonterminal symbol in the semantic rules is used to distinguish between occurrences of the same nonterminal. For example, in the production rule (2), A1 represents the nonterminal A on the left side; A2 represents the nonterminal A on the right side of the syntactic part. It is noted that the inherited length attribute, L, is not down to the descendents as it usually is; rather it is used t o maintain the semantic information of the training string and as a reference for comparison in parsing. For simplicity, let y(a) = 1 for all a E Vt. Consider the second input string
aakkdddf f eeeea. We convert it into
( a ,2)@, 2 x 4 3 ) ( f , 2)(e, 4)(a, 1) and add the following productions to the inference grammar. Syntactic rules
Semantic rules
S -+ AKDFEA
L(A1) = 2, L(K) = 2, L(D) = 3, L(F) = 2, L(E) = 4, L(A2) = 1 K + kK Y(K1) = Y(k) + Y(K2) K-+k Y(K) = Y(k) F + fF y(F1) = Y ( f ) + Y(F2) F-+f Y(F) = Y ( f ) For the new input string, there will be no need to add those production rules, A -+ a A , A + a , . . . , etc. One production rule is created for each input string, i.e. the first production rule in the above example. In fact, there are (2m+n) production rules for a set of n training strings, where m is the number of nonterminal symbols. We now formulate the inference algorithm of attributed grammar which uses the length attribute [59].
Algorithm 4.1: Inference of an attributed context-free grammar. Input: A set of training strings. Output: An inferred attributed context-free grammar. Method:
76
K.-Y. Huang
(1) Convert each input string t o the attributed string by merging identical primitives. (2) For each input attributed string a l a 2 a 3 . . ’ a k , add the production S + AlAzA3.. . A k t o the inference grammar, where Ai is the nonterminal corresponding to terminal ai, and the semantic rule L(Ai) = yi, 1 5 i 5 k, where yi is the length attribute of primitive ai. (3) For each primitive a,add the production rule A -+ aA, y(A1) = y(a)+y(A2) and A + a,y(A) = y ( a ) to the inference grammar, if they are the new production rules. This inferred grammar will generate excessive strings if we only apply syntactic rules. However, we can use semantic rules (inherited attributes) t o restrict the grammar so that no excessive strings are generated. 4.5. Minimum-Distance Error-Correcting Earley ’s Parsing f o r
Attributed Strings
A modified Earley’s parsing algorithm is developed here for attributed contextfree languages. Here, errors of insertion, deletion, and substitution transformation are all considered in the derivation of Earley’s item lists. Let the attributed grammar G = (V,, V t ,P, S) be a CFG (context-free grammar), and let z = b l b 2 . . . b, be an input string in Vt*. The form [A + a . p, 5 , y, i] is called a n item for z if A + ap is a production in P and 0 5 i 5 n [ 2 , 2 2 ] . The dot in a . ,L? between a and p (we use a . p in the program output) is a metasymbol not in V, or V t ,which represents the parsing position; 5 is a counter for local syntactic deformation which accumulates the total cost of substitution of terminal symbols. When A = S,y is used as a counter for global deformation which records the total cost of insertion and deletion errors. On the other hand, if A # S then y is used as the synthesized attribute of A. The meaning of index i is the starting parsing position of the string, and it is the same pointer as the conventional item of Earley’s parsing algorithm. The parsing algorithm for an input string z is shown in the following. Algorithm 4.2: Minimum-distance error-correcting Earley’s parsing for an attributed string. Input: An attributed grammar G = (V,, Vt, P, S) and a test string z = blb2 . . . b, in Vt*. Output: The parse lists 10,11,. . . ,I,, and the decision whether or not z is accepted by the grammar G together with the syntactic and semantic deformation distances. Method: (1) Set j = 0 and add [S + . a , 0, 0, 01 to I j if S + a is a production in P. (2) Repeat Steps (3), (4) and (5) until no new items can be added t o Ij. (3) If [B + ( . , x l , y l , z ] is in Ij, B # S, and
1.3 Syntactic Pattern Recognition 77 (a) if [A + a . BP, 2 2 , y2, Ic] is in Ii and A # S, then add item [A + a B . p, 21 2 2 , yl y2, k] to Ij. (b) if [S -+ a . BP, 2 2 , y2, k] is in Ii, then add item [S + a B . p, 21 2 2 , y2 IL(B) - yll, k] to Ij. (c) if [S + a . Cp, 2 2 , y2, Ic] is in Ii, C # B, then add item [S + a . Cp, 2 2 , yl y2, Ic] t o Ij. (d) if [S 4a .,2 2 , y2, Ic] is in Ii, then add item [S + a ., 2 2 , yl y2, Ic] to Ij. If B + [ is a production in P, and if [A + a . BP, 2,y, i ] is in Ij, then add item [B 4 .[, O , O , j] to Ij. If [S + a . BP, 2 , y, i ] is in I j , then add item [S -+a B . p, 2,y L(B), i ] t o Ij. If j = n, go to Step (8); otherwise increase j to j 1. For each item [A + a . a p , z , y , i ] in Ij-1, add item [A + a a . p , z + S ( a , b j ) , y + y(a),i]to Ij, where y ( a ) is the synthesized attribute of a. For simplicity, let y ( a ) = 1 for all a in Vt. S(a, b j ) is the substitution cost, and S ( a ,a ) = 0. Go to Step (2). If item [S -+ a ., 2 , y, 01 is in I,, then string z is accepted by grammar G where z is the local deformation distance, and y is the global deformation distance; otherwise, string z is not accepted by grammar G. Exit.
+ +
+ +
+
+
(4) (5) (6) (7)
(8)
+
+
In the above algorithm, Step (3b) handles the semantic insertion and deletion errors, Steps (3c) and (3d) handle the syntactic insertion errors, Step (5) handles the syntactic deletion errors, and step (7) handles the syntactic substitution errors. It is possible for collision t o occur in the process of developing a new item, i.e. the old item has already been in the list when a new item is to be put in the list. Under this situation, the one with the minimum distance (minimum of z + y) is selected for that item. Actually, collision may occur with only the items related to S-productions, because insertion and deletion transformations are allowed for those items only. Training string : ahhc The attributed grammar G(Vn,Vt,P,S)is as follows : Vn = ( S , A, B, C) V, = la. h, c )
s
=IS}
The production set P i s as follows : Syntactic rules Semantic rules
Fig. 7. Training string abbc and its inferred attributed grammar for Earley’s parsing.
78
K.-Y. Huang
‘est string aabc and its item lists :
I[O] contains [S->ABC. [C->.c [C->.cc [S->AB.C [B->.b [B->.bB [S->A.BC [A->.a [A->.aA [S->.ABC
I[3] contains [A->aA. [B->bB. [C->cC. [C->.c [C->.cC [B->.b [B->.bB [A->.a [A->aA [C->cc. [B->bB. [S->.ABC [A->aA. [S->AB.C [S->ABC. [S->A.BC [A->a.A [A->a. [B->b.B [B->b. [C->c.c [C->c.
,0,4,01 0 0 0I ,O,O,O] ,0,3,0l ,O,O,OI ,O,O,O] 7
2
7
,0,1,0]
,O,O,OI ,O,O,Ol ,O,O,O]
,1,3,01 ,2,3,01 ,3,3,01 ,0,0,31 ,0,0,3] ,0,0,31 ,0,0,3l .0,0,3l ,0,0,31 ,2,2,1] .1,2,1l ,0,3,0] ,1,2,1] ,l,O,OI , I ,1,0] ,0,2,0] ,1,1,2] > 1,1,2l ,0,1,2] ,0,1,21 ,1,1,21 ,1,1,21
I[1] contains [C->.c [C->.cc [B->.b [B->.bB [A->.a [A->.aA [S->.ABC [S->A.BC [S->AB.C [S->ABC. [A->a.A [A->a. [B->b.B [B->b. [C->c.c [C->c.
I[4] contains [C->cC. [B->bB. [A->aA. [A->aA. [B->bB. [C->cC. [C->.c [C->.cC [B->.b [B->.bB [A->.a [A->aA [C->cc. [B->bB. [S->.ABC [A->aA. [S->ABC. [S->AB.C [S->A.BC [A->a.A [A->a. [B->b.B [B->b. rc->c.c
,O70,1l ,0,0,1]
,0,0,11 ,0,0,1]
,o,o,11 ,0,0,1] ,0,1,0] ,O,O,O] .0,2,0] ,0,3,0] ,0,1,0] ,o,1,OI ,1,1,0] ,1,1 ,OI ,1,1,0] ,1,1 ,OI
I[2] contains [C->.c [C->.cc [B->.b [B->.bB [A->.a [A->.aA [C->cc. [B->bB. [S->.ABC [A->aA. [S->ABC. [S->AB.C IS->A.BC [A->a.A [A->a. [B->b.B [B->b. [C->c.c [C->c.
.0,0,21 ,0,0,21 ,0,0,2] ,0,0,2] ,0,0>21 ,O,O,21 ,2,2,0] ,2,2,0] ,0,2,0] ,0,2,0l .1,2,0] , I , I ,O] ,O,l ,O] ,0,1,1I 8 . 1 >1l ,1,1,1] .1,1,1l ,l,l,ll ,l,~?~l
.3,4,0] .3,4,0] .2,4,0] ,2,3,1] ,2,3,1l ,2,3,11 ,0,0,4l ,0,0,4] ,0,0,41 ,0,0,4l ,0,0,4l ,0,0,41 ,1,2,2l ,1,2,2] ,0,4,0] ,2,2,2] ,1,0,0] .1,1,0] ,0,3,0] ,1,1,3] ,I, 1 3 1 ,0,1,3] ,0,1,31 .I,M
Fig. 8. Item lists of t h e Earley’s attributed parssing on the test s t r i n g a u k .
1.3 Syntactic Pattern Recognition 79 Since the error-correcting grammar is ambiguous, the time complexity is O ( n 3 ) , and the space complexity is O ( n 2 ) ,where n is the length of the input string. The parsing is inefficient if the length of input string is large. 4.6. Experiment
Given a training string abbc and using Algorithm 4.1, the inferred attributed grammar is shown in Fig. 7. An input test string aabc is parsed by Algorithm 4.2. The corresponding item lists are shown in Fig. 8. As we can see from the derived item lists, all three kinds of errors are considered. The corresponding items are generated for each possible error transformation. Because the item [S + ABC ., 1 , 0 , 0 ] is in the 14 list, the string aabc is accepted with local syntactic deformation distance 1 and global deformation distance 0. 5. Parsing of Attributed Strings Using Match Primitive Measure (MPM) Although the modified Earley’s parsing algorithm considers all three types of errors, the parsing speed is inefficient. Here, the parsing of an attributed string using the match primitive measure (MPM) is proposed. The similarity measure of attributed strings is discussed in the following.
Fig. 9. The partial MPM f[i,j] computed from f [ i , j - 11 and f[i
-
1, j ]
5.1. Match Primitive Measure- Similarity Measure of Attributed String Matching The match primitive measure (MPM) is defined as the maximum number of matched primitives between two strings. Here, the similarity measure between two attributed strings is proposed. The computation of MPM between two lengthattributed strings can be implemented by the dynamic programming technique on grid nodes as shown in Fig. 9. For each node, three attributes are associated, i.e.
80
K.-Y. Huang
(f,h, w). Let a be an attributed string, where a[i] denotes the ith primitive in a; a[i].s and a[i].y denote the syntactic symbol and length attribute of a[i], respectively. Let ( i l j )indicate the position in the grid. f [i,j]represents the MPM value from point (0, 0) to ( i , j ) ,i.e. the MPM value between two attributed substrings (a[l] . s, a [ l ] . y)(a[2].s,a[2].y). . . (a[i].s,a[i].y)and (b[l].s,b[l]-y)(b[2].s,b[2].y).. . ( b [ j ] . sb[j].y) , of attributed strings a and b. h [ i , j ]and w [ i , j ] represent the residual length attributes of primitive a[i]and b [ j ] ,respectively, after the match primitive measure (MPM) between two attributed substrings (a[l] . s, a [ l ] . y)(a[2]. s, a[2]. y) . . . ( a [ i ]. s, a[i]. y) and (b[l]. s, b [ l ] . y)(b[2] . s , b[2] . y) . . . ( b [ j ]. s, b [ j ]. y) of attributed strings a and b. The partial MPM f [i,j]can be computed from the partial MPM’s f [i - 1,j] and f [i,j - 11 as shown in Fig. 9. The following algorithm is proposed t o compute the MPM between two attributed strings. Algorithm 5.1: Computation of the match primitive measure (MPM) between two attributed strings. Input: Two attributed strings a and b. Let a = (a[l] . s, a[I] . y)(a[2] . s, a[2] . y) . . . (a[rn] . s, a[m] . y) b = (b[l] . S, b[l] . y)(b[2] . S , b[2] . Y ) . . . ( b [ n ]. S , b[n]. Y), where m,n are the number of primitives of a and b, respectively. Outout: The maximum MPM S(a, b). Method: (1) f[O, 01 := 0; h[O,01 := 0; w[O,01 := 0; (2) for i := 1 to m do begin f [ i , O ] := 0; h[i,O]:= a[i]. y; w[i,O] := 0; end; (3) for j := 1 to n do begin f [O, j] := 0; h[O,j]:= 0; w [ O , j ] := b[j] . y; end; (4) for i := 1 to m do for j := 1 to n do begin nodl:= hmove ( i ,j ) nod2:= vmove ( i , j ) if nod1.f >nod2.f then node [Z,j]:= nod1 else node [Z,j]:= nod2; end; (5) output S ( a , b ) := f [m,n]/dGqE;where y1 = a[i] . y,y2 = cj b[jl . y Functions hmove and vmove are written as follows: function hmove (i, j ) : node-type; {node (i - 1,j)+ node ( i , j ) } begin
1.3 Syntactic Pattern Recognition 81 if a[i]. s # b[j] . s then d l := 0; else d l := min(v[i - I , j ] , a [ i ] .y); hm0ve.f := f[i - l , j ] d l ; hm0ve.h := a[i] . y - d l ; hm0ve.v := v[i - l , j ]- d l ; return(hmove) ;
+
end; function vmove (i,j): node-type; {node ( i , j - 1) + node ( i , j ) } begin if a[i] . s # b [ j ]. s then d l := 0; else dl: = min(h[i,j - I],b [ j ]. y); vmove-f := f[i,j - 11 d l ; vm0ve.h := h [ i , j- 11 - d l ; vmovew := b [ j ]. y - d l ; return (vmove); end;
+
J
\
0
f=O h=O v=o
f=O h=3 v=o
1
f=O h=O v=4
f=3 h=O v=l f=3
2
f=O h=O v=2
h=O v=2
3
f=O h=O v=2
f=3 h=O v=2
4
f=O h=O v=4
f=3 h=O v=4
5
f=O h=O v=1
f=3 h=O
6
f=O h=O v=3
f=3 h=O v=3
v=l
In the above algorithm, two functions are used. Function hmove is used t o compute the variation of attributes ( f , h , u ) from node (i - 1,j) to node ( i , j ) . Function vmove is used to get the value of (f,h, u)at point (i, j ) from point (i, j-1). An example of the MPM computation of two attributed strings is shown in Fig. 10. The normalized MPM value is calculated. 5.2. Inference of Attributed Grammar For the parsing of an attributed string using the property of the MPM, the attributed grammar for the training strings is inferred first.
Algorithm 5.2: Inference of attributed grammar. Input: A set of training strings. Output: An attributed grammar. Method: (1) Convert each input string to the attributed string by merging identical primitives. (2) For each input attributed string a1a2a3 . . . ak, add to the grammar the production S -+AlA2As . . . Ak, where Ai is the nonterminal corresponding to terminal a,; and the semantic rule L(Ai) = yi, 1 5 i 5 k , where yi is the length attribute of primitive ai. (3) For each primitive a , add the production rule A + a , y(A) = y ( a ) and y(a) = y, where y is the length attribute of primitive a. The example is shown in Fig. 11
Training string : abbc The attributed grammar G(Vn,Vt,P,S) is as follows Vn = {A,B, C, SJ Vt = {a, b, c ) s ={S) The production set P is as follows : Syntactic rules Semantic rules
Fig. 11. Training string abbc and its inferred attributed grammar for the M P M parsing.
1.3 Syntactic Pattern Recognition 83 5.3. Top-down Parsing Using M P M Given an attributed grammar G and input attributed string z , the value of MPM between z and L(G), the language generated by the grammar G, is calculated. Consider an S-production rule in the grammar, which has the form
For each nonterminal a t the right-hand side of S-production rule, two attributes are associated with it. f [ k ]denotes the MPM value calculated from the beginning up to the parse of kth nonterminal. h [ k ]is a kind of residual attribute used for the calculation later on. The proposed algorithm to compute the MPM between z and L(G) is described in the following.
Algorithm 5.3: Top-down parsing using the MPM. An attributed grammar G = (Vn,Vt, P, S) and an input string z . Let m = the number of primitives in z . n = the length of z = z [ i ]. y. Output: The maximum MPM between z and L(G). Method:
Input:
xi
(1) Set N = the number of S-production rules, and max-MPM = 0. (2) Set f [ O ] = 0 and h[O]= 0. (3) For all 1 5 k 5 N do Steps (4)to (10). (4) Apply the kth S-production rule with the form Sk + Ak,lAk,Z. . . Ak,,, , where m k is the number of nonterminals at the right-hand side of the kth S-production rule to do Steps (5) to (8). (5) For all 1 5 i 5 mk do
f [;I = 0; Wl = L(Ali,z); 1. (6) For all 1 5 j 5 m do Steps (7) and (8). (7) Set v0 = z [ j ]. y and v = v0. (8) For all 1 5 i 5 mk do Apply production rule Ak,i t Uk,i. (a) if z [ j ]. s = ak,i, then d l = min(y(ak,i),ZI) else d l = 0; f l = f [ i - 11 d l ; hl = Y ( a k , i ) - d l ; w l = Y - dl; (b) if z [ j ]. s = u k , i , then d l = min(h[i],v0) else d l = 0; f2 = f [ i ] dl; h2 = h[i]- d l ; v2 = vo - d l ;
+
+
84 K.-Y. Huang
(c) if f l
> f 2 then
{ f[iI = f l ; h[i]= h l ; v = vl; } else{ f[i] = fa; h[i] = h2; v = v2; } (9) MPM = f [ m k ] / d x , where l k = CzlL(Ak,i). (10) If MPM>max-MPM, then max-MPM = MPM. (11) Output max-MPM. Here the normalized MPM is calculated. Algorithm 5.3 is obtained from the comparison between the input string and the string generated by the S-production rule.
Example 5.1. The training string abbc and its inferred attributed grammar are shown in Fig. 11. One input string aabc has been tested, and the parsing result is shown in Fig. 12. The MPM value is 0.75 after normalization. Test string : aabc = (a,2)(b,l)(c,l) 1
2
3
0
0 1
1 1
0
1
0 1
1
2
1
1
3
1
1 2 2
2 3
Fig. 12. Parsing result using M P M for the test string aabc, the MPM value is 0.75.
6. Tree Grammar and Automaton 6.1. Introduction Tree grammars and the corresponding recognizers, tree automata, have been successfully used in many applications such as: English character recognition, LANDSAT data interpretation, fingerprint recognition, classification of bubble chamber photographs, and texture analysis [30,57,64,67,89]. Fu pointed out that “By the extension of one-dimensional concatenation to multidimensional concatenation strings are generalized to trees.” [30]. Comparing with other high dimensional pattern grammars: web grammar, array grammar, graph grammar, plex grammar, shape grammar,. . . , etc. [30], tree grammar is easy and convenient to describe a pattern using data structure of the tree, especially in the tree traversal and in the substitution, insertion, and deletion of a tree node.
1.3 Syntactic Pattern Recognition 85
Input testing seismogram
Recognition Training
-
Preprocessing: (1) Envelope (2) Thresholding (3) Compression (4) Thinning
~
1 Training seismic
Pattern representation: (1) Pattern extraction (2) Primitive recognition (3) Tree construction
Pattern representation: ( 1 ) Pattern extraction
Error-
conecting
Classification
tree automata
grammar
Fig. 13. A tree automaton system for seismic pattern recognition.
An example of applying tree grammar and automata t o recognize 2-D synthetic seismic patterns is presented. The system of a tree automaton is shown in Fig. 13. In the training part of the system, the training seismic patterns of known classes are constructed into their eorresponding tree representations. Trees can infer tree grammars [5,7,55,56,65].Several tree grammars are combined into one unified tree grammar. Tree grammars can be used t o generate the error-correcting tree automaton. In the recognition part of the system, each input testing seismogram passes through preprocessing [38,47],pattern extraction, and tree representation of the seismic pattern. Then each input tree is parsed and recognized by the errorcorrecting tree automaton into the correct class. Three kinds of tree automatons are adopted in the recognition: weighted minimum distance structure preserved errorcorrecting tree automaton (SPECTA), modified maximum-likelihood SPECTA, and minimum distance generalized error-correcting tree automaton (GECTA). We have some modifications on the methods of weighted minimum distance SPECTA and maximum-likelihood SPECTA. 6.2. Tree Grammar and Language
A tree domain (tree structure) is shown below [30]. Each node has its ordering position index and is filled with a terminal symbol. 0 is the root index of a tree. 0
0.1
/ I \ 0.2
/ \ 0.1.1
0.1.2
...
0.1.1.2
. ..
/ \ 0.1.1.1
0.3 . ..
86
K.-Y. Huang
Each node has its own children, except the bottom leaves of the tree. The number of children at each node is called the rank of the node. Although there are different kinds of tree grammars [30], we use the expansive tree grammar in the study because of the following theorem. Theorem 6.1. For each regular tree grammar Gt, one can effectively construct an equivalent expansive grammar Gt’, i.e. L(Gt’) = L(Gt) [30]. An expansive tree grammar is a four-tuple Gt = (V, T , P, S), where = V N u VT, VN = the set of nonterminal symbols, VT = the set of terminal symbols, S : the starting nonterminal symbol, T : the rank of terminal symbol, i.e. the number of children in the tree node, and each tree production rule in P is of the form
v
(11x0 -+
x
(2)Xo -+ x,
or
/I...\
x1 x2
&(z)
where x E VT and XO,X I , X2,. . . , XT(z)E VN. For convenience, the tree production rule (1) can be written as XO + XXlX2 . . . Xr(z) [30]. From the starting symbol S, a tree is derived by using the tree production rules of P, s -+(Yo =+ a1 =+ ... =+ (Ym = a. The tree language generated by Gt is defined as L(Gt) = { a is a tree IS (Y in G t } , where * represents several derivation steps using tree production rules in Gt.
>
6.3. B e e Automaton
The bottom-up replacement functions of the tree automaton are generated from the expansive tree production rules of the tree grammar. Expansive tree grammar is Gt = ( V , r , P , S ) , and tree automaton is Mt = ( Q , f , S ) , where Q is the set of states, f are the replacement functions, and S becomes the final state.
xo -+ If tree production rule
X
/ I
x1 xz
... \
is in P, then the bottom-up
x,
replacement function in the tree automaton can be written as
xo 4/ XI
x
I .. . \ x2
xn
, or
f ( 2 14x0 / I . .. \ , or fz(X1,X2,. .. , X n ) XI
x2
xn
-+
XO.
The tree automaton is an automatic machine to recognize the tree and has the tree bottom-up replacement functions which are the reverse direction of the tree production rules. Tree grammar uses forward and top-down derivation to derive the tree. The tree automaton uses a backward replacement of the states from the bottom to the root of the tree. If the final replacement state is in the set of the final
1.3 Syntactic Pattern Recognition 87 states, then the tree is accepted by the automaton of the class. Otherwise the tree is rejected.
Example 6.1. The following tree grammar Gt = (V, T , P, S) where V = { S , A , B , $ , a 1 b } , V ~= {*,$,.L a , + b } , r ( a ) = { 2 , 0 } , r ( b ) = {2,0}, r ( $ ) = 2, and P: (1) S + $
(2) A
+a
/ \ A
(3) B
+b
A
+a
(5) B
+b
/ \
/ \
B
(4) A
B
A
B
can generate the patterns, for example, using productions (l),(4), and (5),
or using productions ( l ) ,( 2 ) , ( 3 ) , (4), (5), (4),and (5). $
1
a
b I \ a b
b
The tree automaton which accepts the set of trees generated by Gt is
Mt = ( Q , f a , f b , f $ , F ) , w h e r e Q = { 4 A , q B , Q S } , F = { q ~ } ,a n d f : f a = q A , f b = qB, f a ( Q A , q B ) = ~ Af bI( q A , q B ) = q B , f$(qA,
qB) = 4%.
Example 6.2. The following tree grammar can be used to generate trees representating L-C networks.
Gt
=
(V, T , P, S), where V = {S, A, B, D, E, $, Vin, L, C, W}, VT = {$, Vin, L, C,W}, 2, r(Vin) = 1,r(L) = {1,2},r(C) = 1,T(W) = 0, and P:
T($) =
(1) S + $
(2) A
/ \
A
B
+ Vin I E
(3) B
+L
(4) B
/ \
D
B
+L I
D
(5) D
+C I
E
(6) E
+W
88
K.-Y. Huang
For example, after applying productions (l),(2), (3), ( 6 ) , (5), (4), (6), (5), and (6), the following tree is generated.
$
I\ i'n L I I\ w C L I
I
ni'
w c
The tree automaton which accepts the set of trees generated by Gt is Mt =
(Q, fw, fc, f ~fv,,, , f$,F), where Q = {QE, 4 D , QB, q.4, qs}F = { q s } , and f:
fW()=qE,fC(qE)=qD,
fL(qD)=qB,fL(qD,QB)=qB,
fV,,(qE)=qA,
f$(qA,qB)=qS-
Example 6.3. Tree representation and tree grammar of seismic bright spot pattern. (A) Bright spot pattern: The seismogram of the primary reflection of bright spot is generated from a geologic model [19,71]. The geologic model is shown in Fig. 14 and the seismogram Distance
0.0
Density=2 0 gm/cm**3 Velocity=2.0km/sec
D=2.3 V=2.3
-1.0
I
/
D12.270 V=2.225
D=2.8 V=2.8
.." Fig. 14. Geologic model.
\
1 . 3 Syntactic Pattern Recognition
89
Fig. 15. Bright spot seismogram.
is shown in Fig. 15. After preprocessing, thresholding, and compression in the vertical direction, the bright spot pattern can be shown below. We can scan the pattern from left to right, then top t o bottom. The segments (branches) can be extracted in the tracing. Eight directional Freeman’s chain codes [27] are used to assign primitives t o the segments. From expanding the segments (branches), the tree representation of the seismic bright spot pattern is constructed. And the tree can infer the production rules of the tree grammar. $0
X
xxxxx xxxxx I xxx xxx xxx xxxxxxxxxxxx xxx 5 xxx 0 xxx 7 xx xxxxxxxxxxxxxxxxyxxxxxx xx 5 xxxxx 0 xxxxx 7 xx xx 5
Primitives: eight directional Freeman’s chain codes [27] and terminal symbol 0 (the neighboring segment have already been expanded), 3
2
1
4 1% O
5
--
-6 7 Primitives
90
K.-Y. Huang
(B)(1) Tree representation of bright spot after scanning and tree construction: $
I \ 5 7 I\ I \ 5 0 0 7 I \ I \
5
0
@ 7
(2) Corresponding tree node’s position: 0
/ \ 0.1 I \ 0.1.1 0.1.2 I\ 0.1.1.10.1.1.2
0.2 I \
0.2.1 0.2.2 I\ 0.2.2.1 0.2.2.2
(C) Tree grammar: Gt = (V, r, P, S), where V = set of terminal and nonterminal symbols = {$, 0, 5, 7 , 0, S, A, B, C, D, E, F, GI HI I, J > , VT = the set of terminal symbols = {$, 0, 5, 7, Q},$: the starting point (root) of the tree, @:represents that the neighboring segment has already been expanded, S: the starting nonterminal symbol, T : r ( 5 ) = r(7) = { 2 , 0 } , r ( $ )= 2 , r ( @ ) = O,r(O) = 0, and P: (2)A -+ 5 (3)Bj 7 (4)C+ 5 I \ I \ I \ I \ G H C D E F A B (7)F-+ 7 (8)G-+5 (9)HjO (lO)I+@ I \
(1) S-+ $
I
(6)E-+@
(5)D+O
(11)J-+7
J
The tree derivation steps are as follows:
j
I\ A
B
(4,5 , 6, 7, 8, 9, 10, 11)
(3)
(2)
(1) S + $
$
I \ 5 B I \ C D
$
-+
I 5
_ _ - - . _ - - . -> -._ $- . . - - . -
\
7 I \ I \ C D E F
I
\
5 7 I \ I \ 5 0 @ 7 I \ I \ 5 0 @ 7
1.3 Syntactic Pattern Recognition 91 So following the steps as described in (A) and (B), each seismic pattern can be represented as a tree. From the steps (B) to (C), a tree can infer tree production rules. The tree production rules can derive trees. Each tree corresponds t o its pattern class. (D) Tree automaton from tree production rules of (C): A tree automaton generated by Gt is Mt = (Q, f$, fo, f5, f7, f ~S),, where Q = {S, A, B, C, D, E, F, G, H, I, J}, S: the final state, and the bottom-up replacement function f: (11) f7 + J (10) f~ + 1 (9) fo + H (8) f5 + G (7) f7(I,J) + F (6) f~ + E (5) fo + D (4) f5(G,H) -+ c (3) f 7 ( W + B (2) f5(C,D) + A (1) f$(A,B) + s.
The number on the left hand side of the bottom-up replacement function corresponds to the number of production rule of the tree grammar, and the bottom-up replacement function is the reverse of the corresponding production rule. The above tree in (B) can be replaced by replacement functions step by step from the bottom to the root of the tree and accepted by this tree automaton Mt as the seismic bright spot pattern. 6.4. Tree Representations of Patterns
In the tree automaton system, patterns must be extracted from image data and constructed as the tree representations. In order to construct the tree representation of a pattern automatically, a scanning algorithm is proposed. The following Algorithm 6.1 is the construction of tree representation from scanning an input pattern. The algorithm works for both four-neighbor and eight-neighbor connectivity. The scanning is from left t o right and top to bottom on the binary image. In the algorithm, breadth-first tree expansion is adopted to construct the tree representation such that the depth of the tree will be shorter and the parsing time of the input tree by tree automaton will also be shorter in parallel processing.
Algorithm 6.1. Construction of a tree representation from a pattern. Input: Image of a pattern after thinning. Output: Tree representation of a pattern. Method: (1) While scanning image from left to right and then top to bottom, (a) If the scanning reaches a point of the pattern, then the point is the root (node) of a tree. (b) Trace all following branches (segments) from a node. And assign the terminal symbol to each branch (segment) by the chain code. (c) If the lower end of each branch (segment) has sub-branches (sub-segments), then go to Step (b), trace the sub-branches (sub-segments) and expand all children nodes from the left-most child node in the same tree level. After the whole children nodes in the same level are expanded, then go t o Step (b) to expand the descendants from the left-most in the next down tree level.
92
K.-Y. Huang
Expand level by level until there is no node to be expanded. Then a pattern is extracted and its corresponding tree representation is constructed. There may exist several patterns in the image data. Algorithm 6.1 can extract a pattern. The following Algorithm 6.2 can extract all the entity patterns in the image and construct their tree representations of all the patterns in the image.
Algorithm 6.2. Extract all patterns and construct their tree representations from the binary image. Input: Image data after thinning. Output: Tree representations of all patterns. Method: (1) Scan image from left t o right, then top to bottom. (2) When the scan reaches a point of a pattern, follow Algorithm 6.1 to extract one pattern and construct its corresponding tree representation, then erase the position of current pattern from the image. (3) Go to Step (l),continue to scan, extract the next pattern, and construct the tree representation until there is no pattern to be extracted.
6.5. Inference of Expansive Free G r a m m a r In the training part of tree automaton system of Fig. 13, training patterns must be given in order t o infer the production rules. The following Algorithm 6.3 is presented to infer expansive tree grammar from tree representation of the pattern.
Algorithm 6.3. Inference of expansive tree grammar. Input: Tree representation of a pattern. Output: Expansive tree grammar. Method:
(1) From top to bottom of a tree, for every node of the tree, derive a tree production rule:
x+
a
/ I
x1
x2
... \ x n
where X: the nonterminal symbol assigned to the node, a: the primitive (terminal) of the node, and XI, Xa,. . . ,Xn: the nonterminals of the direct descendants (children) to cover the next level subtrees. (2) Go to Step (1) to handle the other nodes in the same level of the tree, until every node is reached. Handle each node level by level of the tree. This algorithm can be implemented in one recursive procedure.
1.3 Syntactic Pattern Recognition 93
6.6. Weighted Minimum-Distance SPECTA Due to noise and distortion there are some possibilities that terminals may be recognized as the neighboring terminals in the process of primitive recognition. The tree may have substitution error terminals, then tree automaton must be expanded to recognize error trees. Given a tree W below, node b at position 0.1 is substituted by node x of W'. But the trees W and W' have the same tree structure. This
'2W'.
substitution error is written as W
Tree W = b I d
a
0
a I \
Substitutenode b
c = 0.1 0.2 I 0.1.1
at position 0.1 by x
I
\
I d
For a given tree grammar Gt or tree automaton Mt, the minimum-distance SPECTA is formulated to accept the input tree and to generate a parse that consists of the minimum number of substitution error tree production rules or error replacement functions, while the tree structure is still preserved. Assume that W' is an input tree, the parsing of the input tree by minimum-distance SPECTA is a search for the minimum distance and is a backward procedure of constructing a tree-like transition table with all candidate states and their corresponding costs recorded from the leaves to the root of W'. For each tree node a (index position in tree), there is a corresponding transition box t, which consists of triplet items (X, # k , c) in the transition table, where X is the state, # k is the kth production rule or replacement function, c is the accumulated error cost. An example is shown as follows. Example 6.4. Parsing by minimum-distance SPECTA. Given tree production rules P: (1)s -+
(2)A
$
I \
A
B
-+
5
( 3 ) B -+ 7
$
I \
, and given an input tree,
5
7
we can generate the parsing of the input tree by minimum-distance SPECTA as below.
K.-Y. Huang
94
includes state A (nonterminal) in (A, 2, 0) and state B (nonterminal) in (B, 3, 1). Triplet item (A, 2, 0) represents that we can use A in production rule (2) A -+ 5 to derive terminal 5 with 0 substitution error because input terminal is 5, triplet item (B, 3, 1) represents that we can use state B in production rule (3) B + 7 to derive terminal 7 with 1 substitution error because input terminal is 5. t o . 2 has a similar explanation. t o has state S in triplet item (S, 1, 0) so that we can use S in
to.1
$
production rule (1)to derive
/ \ with 0 substitution error, 0 in (S, 1, 0) is the
A B sum of 0 from (A, 2, 0) and 0 from (B, 3, 0). Although there are other derivations from the combinations of the states (nonterminals) A and B of the triplets, i.e. $
$
$
$
/ \ , only S in production rule (1) can derive
/ \ , / \ , and
/ \
A
A B A B B A B with minimum error, others are counted as larger errors and neglected. If X is a candidate state of tree node at position index a , then each triplet item (X, #lc, c) is added to a box t,, #lc specifies the lcth production rule or bottom-up replacement function used, and c is the accumulated minimum number of substitution errors from the leaves to the node a in subtree of W' from node a when tree node a is represented by state X. The algorithm of minimum-distance SPECTA was in Fu [30]. For consideration of the weighted substitution error costs of pair terminals, Basu and Fu [6], presented the minimum weighted distance SPECTA. Here we have some modification. We can expand each production rule to cover the substitution error production rules and embed cost t o each production rule. So the expanded tree grammar with weighted error costs are generated. Production rule X 4z is expanded to substitution error production rule with error cost c, X 4 y, y # z; X 3 z, c = 0, and production rule C
x+ I
I
...
x,x*
\
X"
isexpandedto
/
XI
Y I ... \ x2
,y#x;c=Oify=x.
X"
Initially the expanded tree grammar with error costs must be generated. Then the algorithm of weighted minimum distance SPECTA is presented as follows.
Algorithm 6.4. Weighted minimum distance SPECTA. Input: An expanded tree grammar Gt (or a tree automaton Mt) with error costs, a tree W'. Output: Parsing transition table of W' and minimum distance. Met hod: (1) Replace each bottom leaf of the input tree W' to a state. If the bottom leaf (terminal) of the input tree is 2, i.e. r[W'(a)]= 0 (rank of bottom node a in tree W' is 0) and W ' ( a ) = z (terminal at bottom node a
1.3 Syntactic Pattern Recognition 95 in tree W’ is x), then for any terminal y and production rules ((#k)X % y ) , replace the leave (terminal) x of the input tree t o state X, and store (X, #k, c) to the box t a .
Do Steps (2) and (3) until there is no replacement to the root of the tree. Replace the subtree to a state using bottom-up replacement function. m
L .
If subtree is
/ 1 . .. \
x1 xz n) and W’(a)
, i.e. r[W’(a)] = n > 0 (rank of node a in tree W’ is
xn
(terminal at node a in tree W’ is x), then for any terminal y x% y X and production rules ( ( # k ) / I . . . \ ), replace subtree / I . . . \ =x
x1 xz
xn
x1
x2
+ + +
x,
to state X, and store (X, # k , c‘) to the table box t,, c’ = c1 c2 . . . c,, where ci is the cost in table box t,,j for state Xi, z = 1,.. . ,n. Whenever more than one item (X,#k,c) in t, has the same state X, keep the item with smaller error cost, and delete the items with larger error costs. If items (S, # k , c) are in t o , choose one item with minimum distance c, then the input tree is accepted with distance c. If no item is associated with the starting nonterminal S in t o of the form (S, # k , c), then the input tree is rejected.
Example 6.5. Parsing of error bright spot pattern by minimum distance SPECTA. (A) Bright spot pattern with primitive errors: $0
X
5
xxxxx xxxxx
7 xxx xxx xxxxxxxxxxx xxxxxxxxxxxxxxxx 0 xxx 5 xxx 0 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xx xx xxx 0 xxxxxxxxxx 0
xxx
5
xx
(B) Tree representation with primitive errors: Using eight directional F’reeman’s chain codes [27] and @, tree representation of bright spot with primitive errors is shown below. $
f 5
\
7 I \ 5 O @ 0 I \ I \ 5 0 0 0 I\
96
K.-Y. Huang
(C) Parsing by minimum distance SPECTA: Using the tree automaton inferred from bright spot pattern in Example 6.3, the transition table of parsing on tree of error bright spot pattern of Example 6.5 using minimum distance SPECTA can be generated as follows. Here the costs of terminal substitution errors are set and equal t o 1. The explanation of each box from the bottom-up replacement is the same as that of Example 6.4
6.7. Modified Maximum-Likelihood SPECTA When the probabilities of tree production rules and the substitution deformation probabilities on terminal pair symbols are available, maximum-likelihood SPECTA can be used for the recognition of patterns [30]. The stochastic expansive tree grammar G, = (V, T , P, S) has the form of production rules in P : (1) xo
3
or
z
/I
x1 x2
(2)
XO
3 z,
... \ &(z)
where p is the probability of production rule, z E VT and Xo,X1, Xz,. . . ,XT(z)E VN.
I . 3 Syntactic Pattern Recognition 97 The major steps of the maximum-likelihood SPECTA in Fu are described as follows [30]. Given stochastic expansive tree grammar G,, terminal substitution probabilities q(y/x) and input tree W’. (1) Replace each leaf of the input tree t o a state. If r[W‘(a)]= 0 (rank of node a in tree W’ is 0), W’(a) = y (symbol at node a in tree W’ is y), and X 3 x is the lcth production rule in P, then add (X, #lc,p’) to t, and p’ = p x q(y/x). (2) Replace the subtree to a state using bottom-up replacement function. If r[W’(a)]= n > 0 (rank of node a in tree W’ is n ) , W’(a) = y (symbol at node a in tree W’ is y), and
x 4
x
x1
x2
/
I . . .\
is the lcth production rule in P,
x n
then add (X, #lc,p’) t o table box t,, p’ = p i x p i x . . . x p; x p x q ( y / z ) , where p’, is the probability in table box t,.i for state Xi, i = 1,.. . , n. The calculation of probability p‘ was from the multiplication of the production rule probability p and the terminal substitution probability q(y/x). Instead of the calculation from the multiplication, p x q(y/x), here a modification is proposed. Similar to the previous expanded tree grammar with error costs, each production rule is expanded t o cover substitution error production rules with the probabilities,
x+
i.e. if
X S
x
/ 1 .. . \
x1 x2
is in P, then expand to
x n
X
/ I . ..\
x1 x2
for all
xn
terminals y, y = x or y # x. Summation of the probabilities from expansion of one tree production rule is 1. There are some possibilities that terminals may be recognized as the neighboring terminals in the process of primitive recognition. The value of probability of substitution pair terminals is inversely proportional to the angle of pair terminals. Each tree production rule of tree grammar Gt is expanded to cover the substitution error production rules with probabilities. Based on the expanded grammar, the maximum-likelihood SPECTA is modified here for the recognition of seismic patterns. The algorithm is presented as follows.
Algorithm 6.5: Modified maximum-likelihood SPECTA. Input: (1) Expanded tree grammar Gt (or tree automaton M,) with probability on each production rule, (2) input tree W’. Output: Parsing transition table of W’ and maximum probability. Method: (1) Replace each bottom leave of the input tree W’ t o a state. If r[W’(a)]= 0 (rank of node a in tree W’ is 0), W’(a) = x, and X lcth production rule in P, then add (X, #lc,p’) t o t, and p’ = p .
5 x is the
Do Steps (2) and (3) until there is no replacement at the root of the tree.
98
K.-Y. Huang
(2) Replace the subtree to a state using bottom-up replacement function. If r[W’(a)] = n > 0 (rank of node a in tree W’ is n),W’(a) = x, (terminal at X2; 5 node a in tree W’ is x), and / I . . . \ is the kth production rule
x1
x2
x,
in P, then add (X, # k , p ’ ) to table box t,, p’ = p i x p i x . . . x p k x p , where pi is the probability in table box t,.i for state Xi, i = 1,.. . ,n. ( 3 ) Whenever more than one item (X, # k , p ’ ) in t , has the same state X, keep the item with larger probability and delete the item with smaller probability. (4) If items (S, # k , p ’ ) are in t o , choose one item with maximum probability p’, then the input tree W’ is accepted with probability p’. If no item in t o is associated with the starting nonterminal S, then the input tree W‘ is rejected. 6.8. Minimum Distance GECTA
Due to noise, distortion, and interference of the wavelets, the tree has an error structure. The error may cause the tree to become the preserved or non-preserved tree structure. If the tree structure is preserved, then weighted minimum distance SPECTA and modified maximum-likelihood SPECTA can be applied in the recognition of patterns. If the tree structure is not preserved, then minimum distance GECTA [30] can be applied in the recognition of patterns. The syntax errors between two trees may include substitution, deletion, and insertion errors. The insertion error includes three types of errors: stretch, branch, and split errors. Totally there are five types of syntax errors on the tree. The distance between two trees is defined to be the least cost sequence of error transformations needed t o transform one t o the other [30,61,80].Because there are five possible error transformations to transform one tree into the other tree, each production rule of the tree grammar must be expanded to cover all five syntax errors on trees. Then the expanded grammar can generate a recognizer, i.e. the minimum distance generalized error-correcting tree automaton (GECTA). Similar to the weighted minimum distance SPECTA, the parsing of an input tree W’ using minimum distance GECTA is also to construct a tree-like transition table with all candidate states and their corresponding costs recorded. The procedure is a backward from the leaves to the root of W’ for the least cost solution [30].
7. Conclusions and Discussions Theoretical studies in syntactic pattern recognition were effective in handling abstract and artificial patterns [30,89]. We need simultaneous progress in both theoretical studies and real-data applications in the future. Combining the syntactic and the semantic approaches can expand the power of syntactic pattern recognition. Semantic information often provides spatial information, relation, and reasoning between primitives, subpatterns, and patterns, and can be expressed syntactically, for example, by attributed strings and attributed graphs. The attributed 2-D and
I . 3 Syntactic Pattern Recognition 99 3-D pattern grammars such as attributed tree, graph, and shape grammars, may be the subject of future study [9,31,76,86-881. The distance computation between two attributed patterns (attributed strings, attributed trees, attributed graphs,. . . , etc.) may also be studied in the future [44]. The error-correcting finite-state parsing, Earley’s parsing, tree automaton,. . . , etc. may also be expanded for attributed strings, trees,. . . , etc. [44,45,59]. The distance can be computed between input pattern y and language L(G), or between input pattern and training pattern. Using a distance or similarity measure, clustering methods, such as minimum-distance classification rule, nearest neighbor classification rule, K-nearest neighbor classification rule and method of hierarchical clustering, can be easily applied to syntactic patterns [30,34,61,62,89]. If the pattern has inherent structural property, globally we can use the syntactic approach t o recognize the pattern. Locally we can use neural network techniques in the segmentation and the recognition of primitives, such that the syntactic pattern recognition will improve and become more robust against the noise and distortions. In the study of certainty effect, besides the probability approach, fuzzy logic may be considered in grammar and automaton, for example, fuzzy tree automaton [53]. Parallel parsing algorithms can speed up the parsing time [12,18,20]. For example, the tree automaton can be parsed from the bottom leaves to the top root of the tree in parallel. Further, a syntactic approach to the time-varying pattern recognition may also become one of the research topics in the future [24].
References [l] A. V. Aho and T. G. Peterson, A minimum distance error-correcting parser for context-free languages, SIAM J. Comput. 1 (1972) 305-312. [a] A. V. Aho and J. D. Ullman, The Theory of Parsing, Translation, and Compiling, Vol. 1: Parsing (Prentice-Hall, Englewood Cliffs, NJ, 1972). [3] F. Ali and T. Pavlidis, Syntactic recognition of handwritten numerals, IEEE Trans. Syst. Man Cybern. 7 (1977) 537-541. [4] K. R. Anderson, Syntactic analysis of seismic waveforms using augmented transition network grammars, Geoexploration 20 (1982) 161-182. [51 A. Barrero, Inference of tree grammars using negative samples, Pattern Recogn. 24, (1991) 1-8. [6] S. Basu and K. S. Fu, Image segmentation by syntactic method, Pattern Recogn. 20, (1987) 33-44. [7] J. M. Brayer and K. S. Fu,A note on the K-tail method of tree grammar inference, IEEE Trans. Syst. Man Cybern. 7 (1977) 293-299. [8] I. Bruha and G. P. Madhavan, Use of attributed grammars for pattern recognition of evoked potentials, IEEE Trans. Syst. Man Cybern. 18 (1988) 1046-1089. [9] H. Bunke, Attributed programmed graph grammars and their application to schematic diagram interpretation, IEEE Trans. Pattern Anal. Mach. Intell. 4 (1982) 574-582. [lo] H. Bunke and A. Sanfeliu (eds.), Special Issue: Advances in Syntactic Pattern Recognition, Pattern Recogn. 19, 4 (1986). [ll] H. Bunke and A. Sanfeliu (eds.), Syntactic and Structural Pattern Recognition Theory and Applications (World Scientific, 1990).
100
K.-Y. Huang
[12] N. S. Chang and K. S. Fu, Parallel parsing of tree languages for syntactic pattern recognition, Pattern Recogn. 11 (1979) 213-222. (131 C. H. Chen (ed.), Special Issue: Seismic Signal Analysis and Discrimination, Geoexploration 20,1/2 (1982). [14] C. H. Chen (ed.), Special Issue: Seismic Signal Analysis and Discrimination 111, Geoexploration 23, 1 (1984). [15] C. H. Chen, L. F. Pau and P. S. Wang (eds.), Handbook of Pattern Recognition and Computer Vision (World Scientific, 1993). [16] C. H. Chen (ed.), Special Issue: Artificial Intelligence and Signal Processing in Underwater Acoustic and Geophysics Problems, Pattern Recogn. 18,6 (1985). [17] J. C. Cheng and H. S. Don, A graph matching approach to 3-D point correspondences, Znt. J. Pattern Recogn. A d i f . Intell. 5 (1991) 399-412. [18] Y . C. Cheng and S. Y. Lu, Waveform correlation by tree matching, IEEE Trans. Pattern Anal. Mach. Intell. 7 (1985) 299-305. [19] Y . T. Chiang and K. S. Fu, Parallel parsing algorithm and VLSI implementations for syntactic pattern recognition, IEEE Trans. Pattern Anal. Mach. Intell. 6 (1984) 302-314. [20] M. B. Dobrin and C. H. Savit, Introduction to Geophysical Prospecting, 4th edn. (McGraw-Hill, New York, 1988). (211 H. S. Don and K. S. Fu,A parallel algorithm for stochastic image segmentation, I E E E Trans. Pattern Anal. Mach. Intell. 8 (1986) 594-603. 1221 D. Dori, A syntactic/geometric approach to recognition of dimensions in engineering machine drawings, Comput. Vision Graph. Image Process. 47 (1989) 271-291. [23] J . Earley, An efficient context-free parsing algorithm, Commun. of A C M 13 (1970) 94-102. [24] M.A. Eshera and K. S. Fu, A graph distance measure for image analysis, IEEE Trans. Syst. Man Cybern. 14 (1984) 398-408. [25] T. I. Fan and K. S. Fu, A syntactic approach t o time-varying image analysis, Comput. Graph. Image Process. 11 (1979) 138-149. [26] T. Feder, Plex languages, I f . Sci. 3 (1971) 225-241. [27] G. Ferrate, T. Pavlidis, A. Sanfeliu and H. Bunke (eds.) Syntactic and Structural Pattern Recognitions (Springer-Verlag, 1988). [28] H. Freeman, On the encoding of arbitrary geometric configurations, I E E E Electron. Comput. 10 (1961) 260-268. [29] K. S . Fu, Syntactic Methods in Pattern Recognition (Academic Press, New York, 1974). [30] K. S. Fu, Syntactic image modeling using stochastic tree grammars, Comput. Graph. Image Process. 12 (1980) 136-152. [31] K. S . Fu, Syntactic Pattern Recognition and Applications (Prentice-Hall, Englewood Cliffs, NJ, 1982). [32] K. S. Fu, A step towards unification of syntactic and statistical pattern recognition, I E E E Trans. Pattern Anal. Mach. Intell. 5 (1983) 200-205. [33] K. S. Fu and B. K. Bhargava, Tree systems for syntactic pattern recognition, I E E E Tkans. Comput. 22 (1973) 1087-1099. [34] K. S . Fu and T. Huang, Stochastic grammars and languages, Int. J. Comput. In$ Sci. 1 (1972) 135-170. [35] K. S . Fu and S. Y . Lu, A clustering procedure for syntactic patterns, I E E E Trans. Syst. Man Cybern. 7 (1977) 734-742. [36] J. E. Gaby and K. R. Anderson, Hierarchical segmentation of seismic waveforms using affinity, Pattern Recogn. 23 (1984) 1-16.
1.3 Syntactic Pattern Recognition 101 [37] P. Garcia, E. Segarra, E. Vidal and I. Galiano, On the use of the morphic generator grammatical inference (MGG) methodology in automatic speech recognition, Int. J. Pattern Recogn. Artif. Intell. 4 (1990) 667-685. [38] R. C. Gonzalez and M. G. Thomason, Syntactic Pattern Recognition (Addison Wesley, Reading, MA, 1978). [39] K.-Y. Huang, Branch and bound search for automatic linking process of seismic horizons, Pattern Recogn. 23 (1990) 657-667. [40] K.-Y. Huang, Pattern recognition to seismic exploration, in Automated Pattern Analysis in Petroleum Exploration, eds. I. Palaz and S. K. Sengupta (Springer-Verlag, New York, 1992) 121-154. [41] K. Y. Huang, W . Bau and S. Y. Lin, Picture description language for recognition of seismic patterns, SOC. Exploration Geophysicists Int. 1987 Mtg., New Orleans, 326-330. [42] K. Y. Huang and K. S. Fu, Syntactic pattern recognition for the classification of Ricker wavelets, Geophysics 50 (1985) 1548-1555. [43] K. Y. Huang and K. S. Fu, Syntactic pattern recognition for the recognition of bright spots, Pattern Recogn. 18 (1985) 421-428. [44] K. Y. Huang, K. S. Fu,S. W. Cheng and Z. S. Lin, Syntactic pattern recognition and Hough transformation for reconstruction of seismic patterns, Geophysics 52 (1987) 1612-1620. [45] K. Y. Huang and D. R. Leu, Modified Earley parsing and MPM method for attributed grammar and seismic pattern recognition, J . Inf. Sci. and Eng. 8 (1992) 541-565. [46] K. Y. Huang and D. R. Leu, Recognition of Ricker wavelets by syntactic analysis, Geophysics 60 (1995) 1541-1549. [47] K. Y. Huang and T. H. Sheen, A tree automaton system of syntactic pattern recognition for the recognition of seismic patterns, 56th Annu. Int. Mtg., SOC.Expl. Geophys. (1986) 183-187. [48] K. Y. Huang, T. H. Sheen, S. W. Cheng, Z. S. Lin and K. S. Fu, Seismic image processing: (I) Hough transformation, (11) Thinning processing, (111) Linking processing, Handbook of Geophysical Exploration: Section I. Seismic Exploration, 20, Pattern Recognition €9 Image Processing, (ed.) F. Aminzadeh (1987) 79-109. [49] K. Y. Huang, J. J. Wang and V. M. Kouramajian, Matrix grammars for syntactic pattern recognition, 1990 Telecomm. Symp. Taiwan, 576-581. [50] J. W. Hunt and T. G. Szymansky, A fast algorithm for computing longest common subsequences, Commun. A C M 20 (1977) 350-353. [51] S. Kiram and C. Pandu, A linear space algorithm for the LCS problem, Acta Informatica 24 (1987) 353-362. [52] A. Koski, M. Juhola and M. Meriste, Syntactic recognition of ECG signals by attributed finite automata, Pattern Recogn. 28 (1995). [53] L. H. T. Le and E. Nyland, An application of syntactic pattern recognition to seismic interpretation, in Computer Vision and Shape Recognition, A. Krzyzak, T. Kasvand and C. Y. Suen (eds.) (World Scientific, 1988) 396-415. [54] E. T. Lee, Fuzzy tree automata and syntactic pattern recognition, I E E E Trans. Pattern Anal. Mach. Intell. 4 (1982) 445-449. [55] V. I. Levenshtein, Binary codes capable of correcting deletions, insertions and reversals, Sou. Phys. Dokl. 10 (1966) 707-710. [56] B. Levine, Derivatives of tree sets with applications to grammatical inference, IEEE Trans. Pattern Anal. Mach. Intell. 3 (1981) 285-293. [57] B. Levine, The use of tree derivatives and a sample support parameter for inferring tree systems, IEEE Trans. Pattern Anal. Mach. Intell. 4 (1982) 25-34.
102
K.-Y. Huang
[58] R. Y. Li and K. S. Fu, Tree system approach for LANDSAT data interpretation, Symp. Mach. Process. Remotely Sensed Data, West Lafayette, Ind., June 29-July 1, 1976. [59] W. C. Lin and K. S. Fu, A syntactic approach to 3-D object representation, I E E E Trans. Pattern Anal. Mach. Intell. 6 (1984) 351-364. [60] H. H. Liu and K. S. Fu,A syntactic approach to seismic discrimination, Geoexploration 20 (1982) 183-196. [61] S. W. Lu, Y . Reng and C. Y . Suen, Hierarchical attributed graph representation and recognition of handwritten Chinese characters, Pattern Recogn. 24 (1991) 617-632. [62] S. Y. Lu, A tree-to-tree distance and its application to cluster analysis, I E E E Trans. Pattern Anal. Mach. Intell. 1 (1979) 219-224. [63] S . Y. Lu and K. S. Fu, A sentence-to-sentence clustering procedure for pattern analysis, IEEE Trans. Syst. Man Cybern. 8 (1978) 381-389. [64] S. Y. Lu and K. S. Fu, Error-correcting tree automata for syntactic pattern recognition, I E E E Trans. Comput. 27 (1978) 1040-1053. [65] S. Y. Lu and K. S. Fu, A syntactic approach to texture analysis, Comput. Graph. Image Process. 7 (1978) 303-330. [66] S . Y . Lu and K. S. Fu,Stochastic tree grammar inference for texture synthesis and discrimination, Comput. Graph. Image Process. 9 (1979) 234-245. [67] W. Min, Z. Tang and L. Tang, Using web grammar to recognize dimensions in engineering drawings, Pattern Recogn. 26 (1993) 1407-1416. [68] B. Moayer and K. S. Fu, A tree system approach for fingerprint pattern recognition, I E E E Trans. Comput. 25 (1976) 262-274. [69] R. Mohr, T. Pavlidis and A. Sanfeliu (eds.) Structural Pattern Recognitions (World Scientific, 1990). [70] T. Pavlidis, Linear and context-free graph grammars, J. ACM 19 (1972) 11-12. [71] T. Pavlidis, Structural Pattern Recognition (Springer-Verlag, New York, 1977). [72] C. E. Payton (ed.), Seismic Stratigraphy - Applications to Hydrocarbon Exploration (AAPG Memoir 26, Tulsa, OK, Amer. Assn. Petroleum Geologists, 1977). [73] J. L. Pfaltz and A. Rosenfeld, Web grammars, Proc. 1st Int. Joint Conf. Artif. Intell., Washington, D.C., (1969) 609-619. [74] A. Rosenfeld, Picture Languages (Academic Press, New York, 1979). [75] A. Sanfeliu, K. S. Fu and J . Prewitt, An application of a graph distance measure to the classification of muscle tissue patterns, Int. J. Pattern Recogn. Artif. Intell. 1 (1987) 17-42. (761 A. C.Shaw, The formal picture description scheme as a basis for picture processing system, Znf. Control 14 (1969) 9-52. [77] Q. Y. Shi and K. S. Fu, Parsing and translation of (attributed) expansive graph languages for scene analysis, IEEE Trans. Pattern Anal. Mach. Intell. 5 (1983) 472485. [78] L. Stringa, A new set of constraint-free character recognition grammars, ZEEE Trans. Pattern Anal. Mach. Intell. 12 (1990) 1210-1217. (791 P. H. Swain and K. S. Fu, Stochastic programmed grammars for syntactic pattern recognition, Pattern Recogn. 4 (1972). [80] E. Tanaka and K. S. Fu,Error-correcting parsers for formal languages, I E E E Trans. Comput. 27 (1978) 605-616. [81] E. Tanaka and K. Tanaka, The tree-to-tree editing problem, Int. J. Pattern Recogn. Artif. Intell. 2 (1988) 221-240. [82] M. G. Thomason, Generating functions for stochastic context-free grammars, Int. J. Pattern Recogn. Artif. Intell. 4 (1990) 553-572.
1.3 Syntactic Pattern Recognition 103 [83] M. G. Thomason and R. C. Gonzalez, Error detection and classification in syntactic pattern structures, ZEEE Trans. Comput. 24 (1975) 93-95. [84] R. A. Wagner and M. 3. Fischer, The string to string correction problem, J. ACM 21 (1974) 168-173. 1851 P.S. P. Wang (ed.), Special issue on array grammars, patterns and recognizers, Znt. J. Pattern Recogn. Artif. Intell. 3,3&4 (1989). [86] G. Wolberg, A syntactic omni-font character recognition system, Znt. J. Pattern Recogn. Artif. Intell. 1 (1987) 303-322. [87] A. K. C. Wong, S. W. Lu and M. Rioux, Recognition and shape synthesis of 3-D objects based on attributed hypergraph, I E E E Trans. Pattern Anal. Mach. Intell. 11 (1989) 279-290. [88] K.C.You and K. S. Fu, A syntactic approach to shape recognition using attributed grammars, I E E E Trans. Syst. M a n Cybern. 9 (1979) 334-345. [89] K. C. You and K. S. Fu, Distorted shape recognition using attributed grammars and error-correcting techniques, Comput. Graph. Image Process. 13 (1980) 1-16. [go] T. Y . Young and K. S. Fu (eds.), Handbook of Pattern Recognition and Image Processing (Academic, New York, 1986).
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 105-142 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
I CHAPTER 1.4 I NEURAL NET COMPUTING FOR PATTERN RECOGNITION
YOH-HAN P A 0 Electrical Engineering and Computer Science, Case Western Reserve University 10900 Euclid, Cleveland, Ohio 44106-7221, USA In this chapter we discuss Artificial Neural Net computing from the viewpoint of its being an enabling methodology for pattern recognition research and practice. The four functionalities of clustering, learning functional mappings, classification through associative recall, and optimization are discussed in a comparative manner relative to other practices in pattern recognition and also relative to each other. In addition to references, two bibliographies, one for books and the other for journals, are provided as guides for further reading.
Keywords: Neural net computing, ART, Hopfield net, optimization, functional m a p ping, supervised learning, Boltzmann machine, simulated annealing, functional-link net, associative memory.
1. Introduction In this chapter, we address Artificial Neural Net (ANN) computing from the perspective of its being a tool for implementing pattern recognition algorithmic practices. The primary context of our discussion is that of pattern recognition, but the topic of specific interest is how neural net computing can be used for attaining pattern-based information processing objectives, especially those which have been established over the years t o be of central interest and importance to the pattern recognition research and practitioner communities. Researchers in information processing have long recognized the strikingly different information processing propensities of serial digital computers and of biological systems. The former rely on speed and accuracy and on ability to execute vast amounts of detailed programmed instructions precisely. But they are, nevertheless, easily overwhelmed by algorithmic tasks, which are of exponential or greater complexity. Unfortunately most real-world perception/cognition tasks, if approached in a direct manner, are of such a nature. In contrast, the nature of biological systems is that of distributed parallel processing systems, made up of large numbers of interconnected elemental processors of rather slow processing speed. In addition, information processing seems to depend on the ability to discern what is cogent and relevant, and to focus on that while
106
Y.-H. Pao
sustaining a minimal degree of maintenance on other matters. Situations, circumstances, and events seem to be evaluated on the basis of the “pattern-ness” of things and on similarities between patterns, and on associations between patterns. This is in marked contrast to the operational strategies of the high-speed, general-purpose, serial-digit a1 computers. At the risk of overstating the case, it almost seems that in approaching the performance of a task, serial-digital computer algorithms tend t o search all of the system space to find a reasonably good path from start state to goal state. We know such approaches are doomed to failure because of the combinatorial explosion in the number of paths to be tried. In contrast to the systematic, frontal-attack approach, biological systems seem to rely more on experience and education so that any good path or even a segment of a good path is remembered, and that knowledge is transmitted through generations, either genetically or through education. In this latter mode of information processing, individual operations are of limited significance but patterns, both spatial and temporal, are of central importance. The significance of patterns is established by associations between a pattern (or a set of patterns) and other patterns (or sets of patterns). Accordingly, the formation of such associations and the activation of such linkages are matters of critical importance. One of the practical objectives of pattern recognition researchers has always been the ability to design and implement machine systems, which are able to perform perception tasks competently t o degrees of proficiency comparable t o that of biological systems. To date it cannot be said that progress in that respect has been as substantial as desired or as expected. If we try to identify reasons for this relative lack of success, we might include the following. It would seem that detailed studies of information processing architectures and procedures in actual biological neuronal studies are so difficult that progress comes a t a very slow pace, indeed. Therefore guidance from that source, though much valued, is limited. In addition, tragically, one of the few initial attempts at artificial neural net computing was so thoroughly discredited at its onset that no academic research in that topic could be sustained for the past decades, until recently. For example, pattern recognition texts have always taught the WidrowHoff algorithm [l]as a procedure for learning a linear discriminant but never with any suggestion that it might also be considered to be a representation of a net capable of learning functional mappings. These matters and others contributed to the absence of a coherent body of commonly-shared knowledge of adaptive and associative pattern-based, information processing practice, even when it was clear that such knowledge and activity were critical to further progress in pattern recognition research. The most recent resurgence in artificial neural net computing is due to initiatives from the cognitive psychology sciences and from researchers interested in biological information processing matters.
1.4 Neural Net Computing for Pattern Recognition 107 It is a huge and high risk jump t o go from well-accepted, highly professional psychological or biological studies to the dubious practice of postulating some drastically simplified “neuronal” computational models and to try to establish some relevancy between the two types of endeavors. However, at any rate, as is well known, such initiatives were carried out over the past decade and have stimulated a powerful resurgence of interest and activity in artificial net computing [2]. Of primary significance to us, is the outcome that regardless of whether the artificial neural net computing paradigm models biology or not, it is of intrinsic value to information processing researchers especially pattern recognition researchers who are interested in the “pattern-ness” of matters and in the rapid distributed parallel processing of associated nets of such patterns [2]. Currently there is not only interest in basic matters in artificial neural net computing, but also extensive activity in the application of this technology to practical tasks, with reports of considerable success. This chapter is primarily in the nature of a n annotated guide to the knowledge which comprises the core of the state-of-the-art in this field at this time. The guide is, therefore, selective rather than comprehensive, and the notation reflects our personal biases and viewpoints, as indeed must be the case for the annotation to be meaningful. The organization of our presentation of materials is described in Section 2. The topical matters themselves are discussed in subsequent sections. These rather sparse schematic discussions are stressed in a section on comments and bibliographic remarks and by a list of titles for further reading. 2. Organization of Chapter In a manner consonant with accepted practice, we divide the architectures and algorithms of artificial neural net computing into four parts characterized by the headings of unsupervised learning, supervised learning, associative memory, and optimization. In addition we list a fifth area, which addresses systems level issues. In Table 1 we list for each such topical area some typical architectures, algorithms, and functionalities supported by the algorithms and corresponding activities and results in traditional pattern recognition research. We believe that Table 1 indicates that neural net computing does indeed address issues of interest to pattern recognition and might indeed provide effective means for realizing the computational objectives of pattern recognition. There are aspects of artificial neural net computing which have been well discussed in literature and even in books. There is no need for us to repeat such discussions in this brief chapter. For accepted background material, we refer the reader to the referenced works and also to the additional bibliographies. In the following sections, we discuss each of these areas.
Pao and Hafez algorithm for concept formation
0
System Level Issues
ART 1, 2 or 3
0
Hopfield and Tank approach
Hopfield net
Generalized delta rule/back propagation-of-error Functional-link net
0
0
Topologically correct mapping
LVQ
0
0
ART 1 & 2
0
Representation Algorithms
Optimization
Associative Memory
Supervised Learning
Unsupervised Learning
ANN Computing Area
Classification
Disconcerning regularities in data
Classification
0
Max nets
Forming clusters
Topologically correct
Classification
0
0 0 0
Feature extraction Concept formation
D D
Associative memory
Learning discriminant
Feature extraction
Underdeveloped in pattern recognition
Undeveloped in pattern recognition
Distributed matrix associative memories
No direct correspondence
This area is of great importance but underdeveloped in traditional pattern recognition
Non-parametric estimation (usually limited to estimation of density distribution functions)
Neural net computing does not deal with feature extraction
Modifying clusters
Vigilance factor
K-means and ISODATA
0
New concepts:
Comments
Data reduction
Traditional Pattern Recognition Issues
0
0
0
Inductive learning
Optimal activation to :omplex problems :radient search
Associative recall
0
Restoration of corrupted patterns
Learning a functional mapping from a set of examples
0
0
Functionality
Table 1. Neural net computing and pattern recognition.
cc
0
F
1.4
Neural Net Computing f o r Pattern Recognition
109
3. Unsupervised Learning We can distinguish between three types of unsupervised learning represented by the algorithms of the types of ART [3-51, LVQ [6], and topologically correct mapping [7,8]. To some this area of neural net computing contributes the least to pattern recognition, because in a sense nothing significantly new is added to the principal functionality of cluster formation. Indeed, it might be argued that existing methods, such as the K-means [9] algorithm or the ISODATA [lo] algorithm, can do just as well if not better than the corresponding neural net algorithms. However to others it is exactly this close correspondence which is satisfying and stimulating. Currently in neural computing, clustering is established on the basis of some metric defined in the actual pattern space in question. This means that we establish a rule for calculating the “distance” between two patterns, and decide whether they should be considered sufficiently similar t o be grouped within one and the same cluster or whether they should be in different clusters. This is illustrated in Fig. l ( a ) for some geometric but not necessarily isotropic metric. If the metric is isotropic, meaning that the rule for calculating distances is the same regardless of the direction in which we look from any one pattern, then the result is a partitioning of pattern space into distinctive nonoverlapping hyperspherical regions or clusters, as shown in Fig. l ( b ) , for a Euclidean distance metric, in two dimensions. Even in such a straightforward simple procedure, we can introduce variety by specifying different cluster radius thresholds for different regions of the pattern space. That can and in general does result in the need for special procedures for resolving conflict and for ensuring convergence. To date no neural net algorithm provides the capability of shaping clusters of the form shown in Fig. l(c) in a meaningful and adaptive manner. 3.1. ART
The well-accepted ART algorithms might seem to differ from the above Euclidean distance approach but actually deviate from it only slightly, being exactly the Euclidean distance approach if all of the vectors are of the same length. As shown in Fig. 2, in the ART algorithm each input pattern vector x is projected on each and all of the prototype vectors bj, and the cluster (prototype) node with the largest projection sum y j = C bjizi is identified with the use of the MAXNET. The proposition that the input vector x belongs to cluster j is then checked by forming the vigilance factor t j z z i . If that exceeds a threshold value (say) p, then the vector x is accepted as an additional new member of that j t h cluster, and the values bj are updated. The top down vigilance factor components are updated also. The ART algorithms are well explained in the literature [3-51, but we advocate and practice a slightly modified version of these especially in so far as updating is concerned [ 111.
Xi
110
Y.-H. Pa0
limitof duster exteni
0 ..... .
..
(c)
Fig. 1. Formation of clusters in unsupervised learning: (a) essentials of a cluster, (b) formation of distinctive non-overlapping clusters, and ( c ) more general cluster formation.
The projection procedure is adequate as long as both the b vectors and the input vectors are all of the same length. Under such circumstances, the scalar product of the two vectors b and x do, indeed, provide a measure of t-he similarity. Also we note that the square of the Euclidean distance between the bj vector and x vector is
= bgbj - 2bSx
+X ~ X .
(3.1)
Clearly the larger the value of (bjx),the smaller the Euclidean distance between bj and x or, in other words, the more similar they are. Also clearly all the previous remarks are also vaIid for the case of binary valued features as in ART 1.
1.4 Neural Net Computing for Pattern Recognition
111
In general, however, we advocate the practice described in Box 1, which is compatible with standard pattern recognition practice and with ART 2.
Box 1 Activate all output nodes j , j = 1, 2, . . . , J . Initialize weights b,, = E,, where E,, are random numbers (-1 < E,, < 1). Input pattern {z,} i = 1, . . . , N . Calculate the square of the Euclidean distance ED;, = 1,(b,, - y,)’. Determine that j for which ED:, < ED;, for all k = 1, 2, . . . , J ; K # j. Assign pattern {z,} as belonging to node j if ED:, also is equal to or less than ED;, (limit), where ED:, (limit) is a more or less arbitrary chosen limiting radius beyond which patterns are not considered to be of that cluster. n 1 z, (n = 0 at initialization). 7. Update b,,(n 1) = - b,z(n) + n+l Therefore, after the input of the first pattern b,,(1) = z,. 8. Input the next pattern, determine t o which unsupervised learning node it belongs, and update corresponding { b , , } .
1. 2. 3. 4. 5. 6.
+
We note that the cluster centers, the bj vectors, are only slightly perturbed by the inclusion of a new member, especially if the cluster already contains a number of members. The updating of bj is weighted so that
n 1 bj(n + 1) = -bj(n)+ n+l n+lX when the j t h cluster with n members in the cluster adds an additional member x to the cluster. The top-down vigilance vector tj is then taken to be equal to b, and is updated in the same manner. There are different ways of exercising this algorithm depending on whether one should activate all cluster prototype nodes initially or activate additional ones only as needed. In contrast to concerns which might dominate if we were endeavoring to build models of the brain, in artificial neural net computing it would seem that the latter practice, that of activating each additional new prototype cluster as needed, is more reasonable and usually convergence to stable cluster centers occurs in a straightforward manner. Also in the case of artificial neural net computing, there may be circumstances where determination of maximum similarly might be carried out more simply than with use of the MAXNET 1121. This type of algorithm corresponds closely to the K-means and ISODATA algorithms, and more to the former than to the latter. Our interest in the ART type of algorithm lies in the suggested net architecture and in the fact that the procedure corresponds to that of the K-means algorithm.
112
Y.-H. Pao y2
Y1
.................
YM
MAX-NET (0 detmine dustermembership. Not allcavrecbnsdwmn.
7oPdoW'
tF link.not shorm. used in verification.
f
f ...........
f ...............
XI
x2
xi
t
XN
Fig. 2. Some aspects of the ART algorithm. (a) Schematic illustration of the ART net, (b) two-dimensional illustration of the equivalence of the projection and distance measures when all pattern vectors are of the same length.
The ART algorithm has been extended to hierarchical ART structures [5] in work which addresses systems issues in the use of such algorithms. What is a little disappointing is the lack of opportunity to shape the clusters and to merge or split them as in the case of ISODATA, as illustrated in Fig. l(c). 3.2. Learning Vector Quantization
The Learning Vector Quantization (LVQ) algorithm builds [6] on the ART type of algorithm and mixes supervised learning with cluster formation. In a manner similar to that of the Widrow-Hoff [l]algorithm of pattern recognition, it refines the structure of a cluster by examining the class membership of each of the members in turn. The assumption is that nearly all of the cluster members belong t o one and the same class. Now as each member is examined in turn, the cluster prototype is modified to move closer to the current member under consideration or away from it depending on whether that member is or is not a member of the majority class.
1.4
Neural Net Computing for Pattern Recognition
113
That is m(n
+ 1) = m(n) + a(x
-
m(n)) if x is of the class of the cluster
(3.3)
+ 1) = m(n)
-
a(x - m(n)) if x is not of that class
(3.4)
or
m(n
where n is the number of cluster members already checked, and m is the vector denoting the cluster center. The parameter a is a fractional quantity which decreases with n so that there is convergence. This situation is depicted schematically in Fig. 3. X X
X
X
X
0 X
/Ax
X
x o
O X
x 0
cluster center adapts by moving A x
- patterns of class c (the majority)
o - patterns aof class C
A
- cluster "center of cluster of class c patterns (biased by presence of c patterns)
Fig. 3. The Learning Vector Quantization (LVQ) algorithm incorporating supervised classification into the cluster procedures.
3.3. Topologically- Correct Mapping The topologically correct mapping approach to clustering allows us t o investigate relationships between the following matters: 0 0 0
0
patterns defined in an N-dimensional positional space X , a process (or metric) defined on X and on the patterns described in X , nodes (or neurons) located spatially in an M-dimensional positional space Y , and an interactional process (or metric) defined for the neurons described in Y .
These entities are described in Fig. 4 where we show pattern vectors {x} defined in pattern space X . The ordering process we impose on top of the patterns in that space is not limited to a determination of the inter-pattern Euclidean distance, but can be quite general indeed. Given the patterns in X and the ordering process, we ask how the consequences of that ordering might be reflected in another space. In particular, for instance, for an array of fixed position neurons in space Y with
114
Y.-H. Pao
. .:. . .
. ..: . .
f
.
..
. .
.
.
. ., .. _
XJpace
.
... ... . .... .. . .
.
.
NdimendcndX space and an wdedng process which imposes a ‘meaning’ on the panern-ness 01 the members of X space
’. .
.
. . . .* . ... . . .
high trequency
sibilants
fricatives
.
/ \
A bwerdimensiondY soace suitable for representingthe essenw d the origin4 Order hhequencl
vdced
I
I
(eg.a 2-D mapping 01 utteranms)
Fig. 4. Schematic illustration of a hypothetical instance of a useful topologically correct mapping.
on-center off-surround interneuron interactions, how might the original X-space ordering process influence the correspondences between the pattern vectors in X space and their representation neurons in Y-space? The illustrations given by Kohonen [7,8] are for one- and two-dimensional Y spaces, the “exhibit” space, so to speak. Although the illustrations are striking and may provide insight for understanding how biological neuronal spatial structures are achieved in nature, our feeling is that in a sense those examples are perhaps too obvious. For example, in the two-dimensional case, the “process” or “metric” ordering the random vectors is determination of the inter-pattern Euclidean distance, and the underlying factor governing the positions of the representative neurons in Y space is also the Euclidean distance. Under such circumstances, it is not surprising that a good “topologically correct” mapping should have been achieved. This is not to say that demonstration of such a mapping is insignificant, but rather that we are not quite clear what other possibilities are implied. The one-dimensional acoustic signal spectral ordering example is slightly more intriguing, but it can be understood in about the same manner as a mapping from a one-dimensional (spectral) space to another one-dimensional (positional) space with corresponding Euclidean distance type of metrics in both cases. The finite Q filter banks are interesting but tend to obscure the situation slightly. However the intrinsic measure in both spaces is still that of inter-frequency distance. That and the additional requirement that neighboring neurons should resonate t o about the same frequency suffices t o establish a monotonic spectral ordering in neuron positional space.
1.4 Neural Net Computing for Pattern Recognition
115
Despite our attempts to rationalize our ready acceptance of the results of those illustrations, it is true that very little has been said about the theory of such mapping processes and work remains. In our fanciful illustration exhibited in Fig. 4, we suggest that in speech processing some weighting of formant-time values of utterances together with a n interneuron interaction in display space might result in a meaningful topologically correct mapping, in which the underlying order, always present, is now made manifest. Such mappings might provide some model of how biological systems organize themselves, but also would be interesting for neural net computing and pattern recognition purposes. 4. Supervised Learning In neural net computing, the notion of “supervised learning” corresponds to the inductive learning of a functional mapping from R” t o R”, given a set of examples of instances of that mapping. In other words, if we know that the vectors xi map into vectors yi for i = 1, 2, 3, . . . , I , can we construct a network computational structure which will accurately map all other x vectors in the N-dimensional X space into the corresponding correct image y vectors in the M-dimensional Y-space? This situation is illustrated schematically in Fig. 5.
Fig. 5 . The concept of learning a functional mapping from observation of examples of such mappings.
This type of activity corresponds most closely to the pattern recognition task of learning a discriminant function for the purposes of classification. It is interesting to note that the task of quantitative estimation is not addressed in pattern recognition except for estimation of density distribution functions, and even there the nature of the task is closer to synthesizing an analytical representation of known (measurable) densities rather than the inductive learning of functional mapping identified only through a set of examples. In this section, we comment on the backpropagation-of-error algorithm, briefly and schematically, because it is well known and the details of the algorithm have been widely disseminated [2].
116
Y.-H. Pao
In so far as learning procedures are concerned, we describe briefly two others in the following, these being the Boltzmann machine (with simulated annealing) [13] and the other being the functional-link net approach [11,14].
4.1. Backpropagation-of-Error Learning Algorithm The feedforward net is illustrated schematically in Fig. 6 , for a functional mapping of R" -+ R. The input to such a net is a vector in N-dimensional space and the output is a single real number. It is assumed that there is a functional mapping y = f(x), instances of which are known {y, = f(x,)}, and the learning task consists of determining the values of the weights { A j i } and { P j } and the thresholds { b j } so that the mean of the squares of the error, C,(f^(x, - f(~,))~, is minimized. There is no loss of generality in omitting a nonlinear transform a t the single output node. In the general case, there would be more than a single output and there could be more than one hidden layer. The weights and thresholds are determined on the basis of minimizing the overall system error averaged over all the training sets. That is, the quantity CkC,(Ok(x,) - Ok(xP))' is minimized where Ok(xp) is the desired (or target) output at the kth node for the pth pattern, and Ok(xP)is the actual computed value of the kth output for the same pattern.
t
Fig. 6. A feedforward neural net with hidden layer and no intra-layer node interactions, used with the backpropagation-of-error algorithm. Shown for Rn -+ R.
In the learning process, the weights Pj (or ,&j in the multi-output case) are readily learned because we have a direct measure of the error (&(xP) - Ok(x,)), at each and all outputs, for all of the training set patterns. However, for the hidden nodes, there is no direct measure of the relevant error ascribable to a particular hidden node and so the output pattern error has to be propagated backwards and interpreted appropriately to serve as a measure of guidance for improving the values of the weights leading into the hidden-layer node.
1.4 Neural Net Computing for Pattern Recognition 117 Although the overall learning procedure of the backpropagation-of-error algorithm is that of gradient search in weight space and that protocol is rapidly adhered to in all cases, there are, nevertheless, many variations on the adaptation scheme, primarily on how to improve the rate of convergence to the point of least error. There exist a number of papers which prove that a multilayer feedforward net can serve as a universal approximator, from a computational point of view, of quite general functional mappings. In other words, provided the spaces X and Y are measurable spaces and the known function is well behaved, then a net of the type shown in Fig. 6 can, indeed, reproduce the known mapping [15-171 and even the derivatives of the functional mapping [18]. Furthermore, even nets with only a single hidden layer can serve as a universal approximator provided the activation functions are of an appropriately constrained form. The multilayer feedforward net depicted in Fig. 6 has linear links and nonlinear activation functions at the nodes. The theoretical proofs of the adequacy of this computational model assure us that the known mapping as made evident by the act of examples {xi + yi} can, indeed, be computed by that type of net. However in pattern recognition and in applications of pattern recognition, interest in supervised learning goes beyond the question of whether known instances of mappings can be duplicated or not. In fact, the primary interest is whether the net can inductively learn a representation of the presumed functional mapping, which is valid for samples of x not included in the training set. In other words, as in other cases of pattern recognition, the interest is in whether the learned mapping is valid for the test set (of x vectors) also. The critical issue is the validity of the generalization. From a signal processing point of view, the generalized delta rule (GDR) multilayer feedforward net is a complex system. If we want to represent the functionality of such a net in terms of a transfer function, we would find that perhaps the best we could do would be to give instances of the effective small-signal transfer function at different signal regions. Even then there remain questions of the efficiency of learning and the quality of the learning achieved with use of different learning procedures. We will discuss these latter issues again briefly in the following in the context of the functional-link net. 4.2. The Boltzmann Machine and Simulated Annealing
An alternate approach to learning an optimal set of weight and threshold values is to “generate and test”. In this alternate approach [13,19,20], different states in weight space can be generated statistically and each new proposed state is evaluated as being accepted or not, depending on whether the LMS error is decreased or whether the increase in the magnitude of the LMS error is within a tolerable amount. In the simulated annealing approach to matters, we evaluate the change in the magnitude of error A&= ~ ( n1) - ~ ( nas) we generate the (n 1)th state. We also generate a random number p in the interval [1,0].
+
+
118
Y.-H. Pao
If A& < 0 then we accept the new set of weights as a better set and go on t o generate yet another (hopefully) even better set. In this way we let the system migrate to an optimum state in weight space. However if A& > 0, we do not necessarily reject the new state. Instead, we compare exp(-Ae/c) with the random number p .
If exp(-A&/c) increase in error.
> p , we accept the new set of weights even though there is an
However if exp(-Ae/c) another trial state.
5 p , we reject the new state and go on t o generate
In simulated annealing the “temperature” parameter c, at first, is taken to be quite large so that A&/c is liable to be quite small, and exp(-Ae/c) large, and the state of the system can wander quite a bit in weight space. As c + 0 large increases in error becomes less and less tolerated and the overall effect is to cause the state of the system to search for and to diffuse toward regions of lower and lower error. If and when carried out well, the simulated annealing procedure allows the system to explore, at first, wide regions of weight space and to avoid being trapped in narrow local minima. Use of the expression (exp -Ae/c) is inspired by an analogy t o the Boltzmann distribution of energy states in a classic (nonquantum-mechanic) system in thermodynamic equilibrium at some temperature. The gradual lowering of the “temperature” parameter corresponds to annealing, hence the term lLsimulated))annealing. It is amusing to note that in practice we often have simulated quenching working quite well also [2O]. The term Boltzmann machine generally refers t o network structures other than the feedforward net, but does not exclude the feedforward architecture. Indeed, it is used frequently for nets which have bidirectional excitatory and inhibitive internode interactions [13]. The procedure we have just described can also be considered to be an instance of the “generate and test” approach to learning in contrast t o the gradient search approach. In practice, use of the Boltzmann machine comprises two separate tasks, one being the choice of an appropriate structure and the other the learning of the values of the weights. To illustrate this and other points we have made, we discuss briefly the task of training a Boltzmann machine digit recognizer [ll]. The numerical digits are represented in terms of the segments of a seven-segment display as shown in Fig. 7 and the input/output relationships of the Boltzmann machine are shown in Fig. 8.
For this case no “hidden” nodes are needed, and the structure of the machine is that shown in Fig. 9. An important point is that there are extensive intra-layer node-to-node interactions. However it is not always true that “hidden” nodes can be avoided.
1.4 Neural Net Computing
~ O Pattern T
Recognition
119
BBRBR
H 5 u 6 7
Fig. 7 . A seven-segment display format for numerical digits 111).
digit -
0 1
2 3 4 5 6 7 8 9
input
1 1 101 1 1 001 001 0 I011101 101 101 1 0111010 1 1 01 01 1 0101 1 1 1 101 001 0 1111111 1111010
OUtDUt
1000000000 0100000000 0010000000 0001 000000 00001 00000 000001 0000 0000001000 00000001 00 0000000010 0000000001
Fig. 8. Fig. 8. Input/output relationship for a Boltzmann machine classification net [ll].
Input
Fig. 9. Structure of digit recognition Boltzmann machine [Ill.
4.3. The Functional-LinkNet
Experience with the backpropagation-of-error algorithm indicates that the algorithm is often slow and does not extrapolate well to high dimensions or t o large training sets. However users often find that ease of learning can be greatly enhanced by appropriate “preprocessing”. It is because of that type of experience that we
120
Y.-H. Pao
initially advocated a functional-link net approach to supervised learning. Instead of using a multilayer feedforward net with backpropagation of error, we advocated enhancing the input vector with functional-links g,(x) to yield a description of the input in an extended pattern space, with additional dimensions [11,14]. The functions g(x) are functions of the entire input pattern vector x and not just functions of any one component x,. In one version of that approach, the one which approaches the back-propagationof-error algorithm the closest, our approach consists in simply claiming that the first layer weights A,, and thresholds b, in the feedforward net of Fig. 6 need not be learned. Subject to rather general and easily satisfied constraints, only the output weights P, need to be learned. This is easily demonstrated. For illustration purposes we consider a functional mapping R + R. Namely, both the input and output spaces are one-dimensional. There is no loss of generality. We choose the one-dimensional case because of the ease of displaying results graphically. In other words we assume that there is a mapping y = f (x). Given instances {y, = f(x)}, can we learn the functional mapping sufficiently well so that we can interpolate and extrapolate to values of x not encountered in the training set? This is a question of utmost importance and interest. Both the B P net and the functional-link net are illustrated in Fig. 10 for this case. Let there be J hidden-layer nodes in a BP net. Then the value of the output is
where g ( ) is the activation function. Let there be N training set patterns so that the entire set of N simultaneous equations to be solved can be written as
GP=f or
C gw. P .-- f n
for n = l , 2 , 3 , . . . , N
j
Equation 4.1 can be expressed in component form as follow: P1 P2
P3
6 OJ
1.4 Neural Net Computing for Pattern Recognition 121
t
.
.
.
t Fig. 10. Comparison of backpropagation of functional-link nets.
+
where gj(xn) = gnj = g(Ajx, b j ) and x, is the value of the input for the n t h training set pattern. It is clear the nature of the solutions of Eq. (4.4) depends critically on the rank of the G matrix, and the values of the individual components are t o some extent immaterial. That is, instead of the feedforward net of Fig. lO(a), we advocate the net illustrated in Fig. 10(b) in which the initial input x is enhanced in dimensions and has the additional components gj(Ajz b j ) . If G is exactly of the correct rank, then a unique solution exists. If there are too many constraints, then there is only an LMS solution or possibly a degenerate set of such solutions. If there is an insufficient number of constraining equations, then there may be an infinite number of solutions for the weights {&}. The point we make is that the weights A3 (and thresholds b j ) can be randomly generated with no loss of generality.
+
122
Y.-H. Pao
For a mapping Rn + R, the input is a vector rather than a scalar, but the argument remains the same. The equations to be solved are then (4.5) Our point is that the vectors A, and thresholds b, may be generated randomly and only the weights @, need to be learned. The function g,() is a function of the entire input pattern vector x and not just a function of any one component x,. The function g() is not learned but is a “hardwired” functional transform of the input vector constituting a preprocessing set, so to speak. In view of these findings, we advocate regarding a supervised learning net t o be essentially a linear net with the input augmented with extra (non-linear) nodes. A large set of experiences indicate that our view is correct and that there are very large improvements in the fidelity of representation. The rate of learning can be achieved with use of the random vector version of the functional-link net. However there is a very important precaution t o be observed. The range of the amplitude of the “random” vectors A, need to be scaled so that the functional outputs g(A,x+ b,) are not all saturated, nor are all so small so that the additional components are linearly dependent. In general this precaution is not difficult to deal with. A normalizing scaling of the range of the input vectors and of the norms of random vectors would be sufficient. The situation is changed significantly if we insist that both the derivative of the function f(x) as well as the function itself be approximated well. Under such circumstances the vectors A, and thresholds b, do indeed need to be learned and the two sets of equations to be satisfied are
and (4.7) where
representing differentiation with respect to the ith component of the vector x and (4.9) It is difficult to solve Eqs. (4.6) and (4.7) simultaneously for a set of { P j } , {Aji}, and { b j } values which will approximate the derivative as well as the functions. But good mappings can be learned, nevertheless, by retreating to our simple functionallink net approach and taking sets of points near each training set input vector so
1.4 Neural Net Computing for Pattern Recognition
123
that in essence something is learned about how the functions varies in different direct ions. The result of the use of this “random vector” version of the functional-link net approach is that we can achieve the learning of rather complex functional mappings in moderate lengths of time. The nets used are linear nets with the input augmented with extra non-linear functional transforms of the input vector. We present and discuss some experimental results in the following subsection. These results are suggestive, but we refrain from generalizing too optimistically on the basis of these partial findings. 4.4. Experimental Results in Support of the Functional-Link N e t
For rather straightforward training tasks, the L‘randomvector” functional-link net outperforms the BP algorithm, principally, in terms of the rapidity with which learning is achieved. However we are also concerned about the ability to inductively learn a mapping of which we know a few instances. To explore the interpolation and extrapolation capabilities of such nets, we revert initially to the one-dimensional case and consider the task of learning a function y = f(z)where both z and y are scalar quantities. Of course sometimes correct interpolation cannot be achieved because not enough information was available in the first place. Given the training set of Fig. 11, there is simply no way for either the BP net or the functional-link net to guess what nature had in mind. Actually, in practice, both nets learned the smooth function f ~ ( z ) .
‘T
Fig. 11. An ambiguous training set of patterns for R
-+
R mapping.
However we might add a few (non-random) instances of the function to provide further information as shown in Fig. 12, in which case both the functional-link net and the backpropagation net interpolated well, but in different ways. The BP net
124
Y.-H. Pao
Fig. 12. An augmented training set.
took a very long time and a large number of iterations. The functional-link net used a large number of augmentations, but learned rapidly in about of the time required by the B P net. These matters are illustrated in Figs. 13 and 14. To explore comparable circumstances for higher dimensions, we also considered a two-input and two-output learning task. This can be visualized as learning two surfaces in a three-dimensional space. In every instance the input is a pair of coordinates (x,y) and the outputs are fi(z, y), and f2(z,y) representing the upper and lower surfaces of a bounded region. The net configurations are depicted in Fig. 15 and the surfaces to be learned are shown in Figs. 16 and 17. Given a reasonably uniform and representative sampling of the two surfaces, both the BP net and functional-link net do learn the two surfaces and interpolate reasonably well. However the functional-link net again learns much more rapidly. The interpolation achieved by the B P net and the FLN net for the upper surface are shown in Figs. 18 and 19, respectively. The B P net and the FLN net for the lower (and smoother surface) that are achieved are shown in Figs. 20 and 21, respectively. To the eye, the FLN results look more irregular, but actually the estimated results are more accurate. Again, the FLN net is faster by a factor of about lo3. We mention, in passing, that in the case of the B P net the hidden layer nodes serve both outputs. The hidden layer nodes are therefore constrained. There are advantages and disadvantages. One disadvantage is that any change in the input/output relationships at any single output will have widespread and severe repercussions throughout the entire net. One advantage is that such severe interactions might indeed force the hidden layer to take on the form of a meaningful internal representation. But that would be attained only by paying a high price in the form of the difficulty of learning!
1.4
Neural Net Computing for Pattern Recognition
1
'I
0.6
e
a
I 0
0
0
0
V
0
0
Training set
125
0
o Estimated output
(4
BP Net:
20 hidden layer nodes;
Number of iterations: 556245;
System error: 0.000 025 Training time (486 PC): Approx. 6 hm
(b) Fig. 13. Demonstration of interpolation achieved with a BP net for R
-+
R mapping.
In contrast, in the case of the functional-link net, each output would be served by its own net and learning is rapidly achieved. However this does not mean that interactions between components of the components of the input vector are neglected. 5. Associative Memories The term associative memories is used in different ways within neural net computing practice, as well as vis-8-vis pattern recognition.
126
Y.-H. Pao 2 -
--
1.5
._
1
8
8 0
0.6 - 0
4
-0.6 i. 1
--
.1.6
--
-2
0
0
0 2iO
0 0
0
0
0
b 260
270 0
2M)
280
300
8 310
320
,
350
I 340
3w
0
..
'T
FLN Net:
Auto expansions: 500;
System Error: 0.000025
Number of iterations: 1160;
Training time (486 PC): Approx. 5 min
(b) Fig. 14. Demonstration of interpolation achieved with a FLN net for R
+ R mapping.
For example, the matrix associative memory was studied by Nakano [21], Kohonen [22], Willshaw [23], Pao [24], and others primarily as models of distributed content-addressable memories, which were forgiving of errors or distortion in the cue and also forgiving of local damage to the memory. Hopfield [25] accentuated the perspective of viewing such distributed contentaddressable devices as nets. In the Hopfield net, there is no learning per se, just memorization, and the net computes a more nearly correct output pattern in response to a possibly distorted input pattern. These nets are fully and widely described in literature.
1.4 Neural Net Computing f o r Pattern Recognition I, 0
t
Fig. 15. Illustration of a two-input/two-output FLN configuration.
127
',U
t
net. (a) The BP net configuration and (b) the
Fig. 16. Output 1 of the two-output net.
128
Y.-H. Pao
Fig. 17. Output 2 of the two-output net.
d
Fig. 18. 2-D mapping learned with a BP net: the upper surface fi(x).
In a series of papers, Kosko [26] explored the question of whether bidirectional associative memories could be synthesized with the memory still in distributed matrix form, but with a nonlinear transforrnation at each end so that
MS(y) = x
(5.1)
1.4 Neural Net Computing for Pattern Recognition 129
Fig. 19. 2-D mapping learned with an F L net: the upper surface f l ( x ) .
Fig. 20. The surface f 2 ( x ) as learned with a BP net.
and
MtS(x) = y . Such memories can indeed be achieved, but often only with great difficulty. They are, nevertheless, noteworthy because they are not limited t o being auto-associative memories.
130
Y.-H. Pao
Fig. 21. The surface fi(x) as learned with a n FL net.
In contrast to such associative memories, we also have nets such as ART 1, ART 2 and ART 3 [3-51, which are also considered to be associative memories. These are probably closer in spirit to the associative memory models such as ACT 1271 and ACT* [28], models devised by psychologists in attempts to mirror the workings of human memory. As far as pattern recognition is concerned, a memory such as the Hopfield net might be considered to be a n excellent pattern recognizer capable of accepting partial or distorted cue patterns and returning a fully restored correct pattern. But, in practice, such fully connected nets are inefficient with the need for N 2 links for an N neuron net and with low storage capacity.
6. Optimization In pattern recognition research, interest in the topic of optimization manifests itself somewhat indirectly in the search for optimal values of decision functions, usually for classification purposes. This type of interest is different from the optimization concerns in systems or controls research where a typical task is to find that system state or control path for which an appropriately defined objective function has an optimal value, subject t o certain constraints on the system. Despite this large difference in the degree and mode of involvement, we discuss briefly a widely known but somewhat controversial approach t o optimization, which is part of neural net computing practice.
1.4 Neural Net Computing for Pattern Recognition 131 Energy Function
Direction of evolution of memory state
E
as distorted cue activates the correct stored pattern and is retrieved as such.
J
I I
I I
I Memory State Vectorx .--)
Stored Pattern
Distorted Cue
Fig. 22. Associative recall as optimization.
We start by going back to the Hopfield net auto-associative memory. For that memory, storage of a pattern may be likened to the creation of a (local) minimum in an energy function as indicated schematically in Fig. 2 2 . In associative retrieval the Hopfield net is activated by an input which might be a distorted version of the stored pattern. The algorithm of the net is such that the system evolves to that state (that pattern) which corresponds to the energy function being at the (local) minimum. In this manner retrieval with the Hopfield associative memory is equivalent to a set of optimization tasks. We know, from Hopfield and Tank [29], that given any initial state v the system evolves t o the state corresponding t o a minimum in the energy, if we let the system update itself iteratively in accordances with the equation
where ui is the input to the ith neuron and vi = g ( u i ) is the output of the ith neuron. For such a system, we have for the temporal evolution of the energy
And we see that E , indeed, evolves to a minimum if g(ui), the neuron activation function, is a nondecreasing function of ui. Hopfield and Tank suggested that the well-studied Traveling Salesman Problem (TSP) be encoded in the following manner. As illustrated in Fig. 23, a five city/five day planning task would be represented by the values of a set of 25 neurons. A
132
Y.-H. Pao B
Example Tour
ci A
B
-
C
D
E
1
2
Encoding used for Neural-net Computing 4
5
Fig. 23. Encoding the TSP Problem for neural net computing.
neuron would represent a specific city visited on a specific day and that neuron would have an output value of 1 if that combination were part of the salesman’s tour and it would have a value of 0 if it were not. Hopfield and Tank synthesized an energy function in analytical form, which represented not only the length of the salesman’s path, but also imposed penalties for nonvalid solutions, such as a city being visited twice or the salesman being a t two different places at the same time. In their approach, although the ultimate acceptable values for the neuron outputs were restricted to 1 or 0, they were treated as continuous variables in the processing. In this manner, a combinatorial optimization problem was converted into a gradient search task for neural net computing. In our opinion this constitutes both the strength and the weakness of this approach. A gradient search approach is advantageous because it obviates the necessity for devising some algorithm for generating new trial states. However the advantage is real only if one can be reasonably assured of a smooth descent into the minimum state or into one of the sets of acceptable minima.
1.4 Neural Net Computing f o r Pattern Recognition
133
The hypothetical advantage is more than offset by the real difficulties if there is a tremendous number of spurious local minima in the energy function so that the search procedure almost immediately comes to a halt in the nearest spurious local minimum. We believe that imposition of constraints in the energy function results in the creation of very large numbers of such minima. Some researchers, nevertheless, have employed this methodology to good effect for subclasses of optimization tasks. Takefuji [30], for example, has used “hillclimbing” terms in the energy function to eject the system state out of a local minimum if the state is not one which is acceptable as a solution. Other innovations, such as “maximum neuron”, have also been helpful for certain other circumstances. In other words Takefuji and his collaborators found that the neural net version of gradient search for optimization is viable if certain acceptability conditions are known for the solution. These should not be incorporated into the energy function but can be used t o activate “hill-climbing’’ terms if one or more of the validation conditions are not satisfied.
7. Representation, Feature Extraction and Concepts In our opinion the most valuable aspect of neural net computing lies in none of the above four computational capabilities. It is rather in the promises of its being a useful (perhaps even the correct) tool for research in a very murky area of information processing research. This is an area which is only dimly perceived by many, denied by others, and partitioned and vehemently defended in isolated parts by yet others, but, nevertheless, tantalizing and beguiling to almost all. We speak of some underlying mysteries of human behavior, perhaps ultimately attributable to the nature of our “hardware”, in this case the left and right halves of the brain, with different functionalities and propensities and not overly communicative with each other. In human behavior we have the dichotomy of perception and action on the one hand and language and reason on the other. In some human cultures, the importance of an internal “knowing” is elevated above all other considerations. Thus, one behaves well not necessarily through elaborate reasoning, but because we “know” that it is the correct behavior. One aims an arrow most accurately when one almost feels that no deliberate aiming is being done. In such behavior, perception and action are everything, and all matters can proceed smoothly and rapidly in an easy flow. To the modern day information processing researcher, it is interesting and entertaining to go back into the history of philosophy and psychology to see that other human cultures have glorified the “pure light of reason”. Language and reason are then supreme. There is even the hypothesis that humans are rational beings and always act to optimize attainment of their goals [28].
134
Y.-H. Pao
Most of us will admit that, in practice, our cultures and our behavior comprise admixtures of both aspects of behavior and both are important. The perception and action channel allows us to carry out intricate actions appropriately and rapidly in response to rapidly changing external conditions with no time for explicit cognitive deliberations. Thus, we can ride bicycles even on highly uneven roadways. Incidentally we note that some bears can also be trained to ride bicycles, but they do not articulate their skill at all, as far as we can tell. However humans can articulate some aspects of the bike-riding skill in language and help teach other humans that skill through use of the language and reasoning channel as well. Interactions between the two channels are, indeed, of great importance and value. One demonstration of the value and effectiveness of such interaction is provided by the example of a n athletics’ coach being able t o produce significant improvements in the performance of a star athelete even though the performance capabilities of the coach himself may be substantially below that of the athlete. In another instance it is found that improvements in foundry practice are obtained when experiences are carefully documented and the information shared with the foundry shop community through the language and symbols channel. It seems to us that one bottleneck to communications between the channels lies in the discovery and articulation of concepts. This same matter is also encountered under the guises of feature extraction, knowledge representation and so on. We believe that the perception and action channel draws a veil over its workings, so as to speak, and in that mode of information processing, both for humans and in algorithms, uniqueness in representation or in feature selection might not be critical. To use an analogy, it is as if matrix operations proceed equally well in general representations as in eigenfunction representation. To pursue the analogy, matters are very different when one wants to describe matters in terms of concepts expressible in terms of linguistic symbols. In artificial intelligence, researchers do strive to learn concepts and there is success when concepts are “learned” in the sense of being inferred from other sets of concepts. However the bridge between the linguistic symbolic world and that of perception is weak. These matters are very important not only because the subject matter is so interesting, but also because there may be the opportunity to fashion computer aids in ways t o compensate for inadequacies due t o the idiosyncrasies of human physiology. We believe that neural net computing provides a tool for capturing and manipulating the “pattern-ness” of things and also for encoding and articulating such matters into the linguistic symbol world. Some work by Pa0 and Hafez [31] address these matters. Hinton, McClelland, Rumelhart, Touretsky and others [32,33] have addressed the relationships between distributed associative processing and linguistic symbolic, logic-based, and information processing. But the emphasis has been to ask how the same type of rule-based processing, entirely with linguistic symbols on both the
1.4 Neural Net Computing for Pattern Recognition 135 antecedent and consequent sides of the scale, might be carried out in a “connectionist” representation. We believe what is just as interesting, perhaps more so, is to investigate what can be achieved if the distributed coarse-coding connectionist scheme is used as an interface between the pattern-ness of things and the extracted linguistic symbolic entities. In terms of human behavior, we would be striving to understand how we learn how to dance a fast waltz not only by example but also aided by spoken instructions. Or in a related matter, how do we “internalize” perceptual experiential knowledge so that we can verbalize that information and reason with it. Finally, it can be said that it is not that neural net computing is relevant to pattern recognition and to computer vision. It is rather that neural net computing might turn out to be a n essential tool for unifying our pieces of knowledge in the fragmented bastions of research endeavor known presently as artificial intelligence, pattern recognition, fuzzy logic, computer vision, and so on. References (11 R. 0. Duda and P. E. Hart, Pattern Classification and Scene Analysis (John Wiley, NY, 1973). [2] D. E. Rumelhart and J. L. McClelland, Parallel Distributed Processing: Explorations i n the Microstructure of Cognition, Vols. 1 & 2 (MIT Press, Cambridge, MA, 1986). [3] G. A. Carpenter and S. Grossberg, A massively parallel architecture for a selforganizing neural pattern recognition machine, Comput. Vision Graph. Image Process. 37 (1987) 54-115. [4] G. A. Carpenter and S. Grossberg, ART2: Self-organization of stable category recognition codes for analog input patterns, Appl. Opt. 26 (1987) 4919-4930. [5] G. A. Carpenter and S. Grossberg, ART3: Hierarchical search using chemical transmitters in self-organizing pattern recognition architectures, Neural Networks 3 (1990) 129-152. [6] T. Kohonen, An introduction to neural computing, Neural Networks 1 (1988) 3-16. [7] T. Kohonen, Self-organized formation of topologically correct feature maps, Biol. Cybern. 43 (1988) 59-69. [8] T. Kohonen, Clustering, taxonomy, and topological maps of patterns, in PTOC.Sizth Int. Conf. on Pattern Recognition, Silver Spring, MD (IEEE Computer Society Press, 1982) 114-128. [9] C. H. Chen, Statistical Pattern Recognition (Hayden, Washington, DC, 1973). [lo] G. H. Ball and D. J. Hall, ISODATA, an interative method of multivariant data analysis and pattern classification, in PTOC.I E E E Int. Communication Conf., Philadelphia, PA, Jun. 1966. [ll]Y. H. Pao, Adaptive Pattern Recognition and Neural Networks (Addison-Wesley, Reading, MA, 1988). [12] R. P. Lippman, B. Gold and M. L. Malpass, A comparison of Hamming and Hopfield neural nets for pattern classification, MIT Lincoln Laboratory Technical Report, TR769, Massachusetts Institute of Technology, Cambridge, MA, 1987.
136
Y.-H. Pao
[13] E. Aarts and J. Korst, Simulated Annealing and Boltzmann Machines (John Wiley, New York, 1989). [14] Y. H. Pa0 and Y. Takefuji, Functional-link net computing, I E E E Computer 3 (1992) 76-79. [15] G. Cybenko, Approximation by superpositions of a sigmoidal function, Mathematics of Control, Signal, and Systems 2 (1989) 303-314. [16] M. Hornik, M. Stinchcombe and H. White, Multilayer feedforward networks are universal approximators, Neural Networks 2 (1989) 359-366. [17] K. Funahashi, On the approximate realization of continuous mappings by neural networks, Neural Networks 2 (1989) 183-192. [18] K. Hornik, M. Stinchcombe and H. White, Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks, Neural Networks 3 (1990) 551-560. [19] G . E. Hinton and T. J. Sejnowski, Analyzing cooperative computation, in Proc. Fifth Annual Conf. of the Cognitive Science Society, Rochester, NY, May 1983. [20] D. S. Touretzky and G. E. Hinton, Pattern matching and variable binding in a stochastic neural network, in L. Davis (ed.), Genetic Algorithm and Simulated Annealing (Morgan Kaufmann, Inc., Los Altos, CA, 1987). [21] K. Nakano, Associatron-A model of associative memory, I E E E Trans. Syst. Man Cybern. 2 (1972) 380-388. [22] T. Kohonen, Associative Memory: A System- Theoretical Approach (Springer-Verlag, New York, 1977). [23] D. J. Willshaw, Model of distributed associative memory, unpublished doctoral dissertation, Department of Machine Intelligence, University of Edinburgh, Edinburgh, 1971. [24] Y. H. Pao and G. P. Hartoch, Fast memory access by similarity measure, in J. Hayes, D. Michie and Y . H. Pao (eds.), Machine Intelligence 10 (Wiley, New York, 1982). [25] J. J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, in Proc. Nat. Acad. Sci. 74 (1982) 2554-2558. [26] B. Kosko, Neural Networks and Fuzzy Systems (Prentice-Hall, Englewood Cliffs, NJ, 1992). [27] J. R. Anderson and G. H. Bower, Human Associate Memory (V. H. Winston, Washington, DC, 1973) (distributed by the Halsted Press, Division of Wiley, NY). [28] J. J. Hopfield and D. W. Tank, Neural computation of decisions in optimization problems, Biol. Cybern. 52 (1985) 144-152. [29] Y . Takefuji, Neural Network Parallel Computing (Kluwer Academic, Boston, MA, 1992). [30] J. R. Anderson, The Adaptive Character of Thought (Lawrence Erlbaum Associates, Hillsdale, NJ, 1990). [31] Y . H. Pao and W. Hafez, Analog computational models of concept formation, International Journal of Analog Integrated Devices and Signal Processing, Special Neural-Net Issue on Analog VLSI Neural Networks 2 (1992) 3-10. [32] G. E. Hinton, J. M. McClelland and D. E. Rumelhart, Distributed Representations, in D. E. Rumelhart and J. M. McClelland (eds.), Parallel Distributed Processing: Explorations i n the Microstructure of Cognition, Vol. 1 (Bradford Books, Cambridge, MA, 1986).
1.4 Neural Net Computing for Pattern Recognition 137 [33] D. S. Touretzky, BoltzCONS: Reconciling connectionism with the recursive nature of stacks and trees, in Proc. Eighth Annual Conf. of the Cognitive Science Society, Amherst, MA, Aug. 1986.
Appendix A A list of texts, monographs and edited volumes which tnight contain detailed information of interest to readers: Adaptive Pattern Recognition and Neural Networks. AUTHOR: Pao, Yoh-Han. PUBLISHER: Reading, MA: Addison-Wesley, 1989. ISN/OTHER No.: 0201125846.0 Advanced Neural Computers. EDITOR: Rolf Eckmiller. PUBLISHER: Amsterdam; New York: North-Holland, 1990. ISN/OTHER No.: 0444884009 (US.) Analog VLSI: Implementation of Neural Systems. EDITORS: Carver Mead and Mohammed Ismail. PUBLISHER: Boston: Kluwer Academic Publishers, 1989. SERIES: The Kluwer international series in engineering and computer science; SECS 80. ISN/OTHER No. 0792390407 Artificial Neural Networks for Computer Vision. AUTHORS: Yi-Tong Zhou and Rama Chellappa. PUBLISHER: New York: Springer-Verlag, 1992. SERIES: Research ISN/ OTHER No.: 0387976833 (New York), 3540976833 (Berlin) Artificial Neural Networks: Theoretical Concepts. AUTHOR: V. Vemuri. PUBLISHER: Washington, D.C.: IEEE Computer Society Press, 1988. SERIES: Neural networks. Computer Society Press technology series. ISN/OTHER No.: 0818608552 Artificial Neural Systems: Foundations, Paradigms, Applications, and Implementations. AUTHOR: Patrick K. Simpson. PUBLISHER: New York: Pergamon Press, 1990. SERIES: Neural networks, research and applications. ISN/OTHER No.: 0080378951, 0080378943 (pbk.) Code Recognition and Set Selection with Neural Networks. AUTHOR: Clark Jefiies. PUBLISHER: Boston: Birkhauser, 1991. SERIES: Mathematical modeling (Boston, MA); no. 7. ISN/OTHER No.: 0817635858 (acid-free paper), 3764335858 (acid-free paper) Cognizers: Neural Networks and Machines That Think. AUTHOR: R. Collin Johnson and Chappell Brown; illustrated by Lisa Metzger. PUBLISHER: New York: Wiley, 1988. SERIES: Wiley science editions. ISN/OTHER No.: 0471611611 Cognitive Psychology: A Neural-Network Approach. AUTHOR: Colin Martindale. PUBLISHER: Pacific Grove, CA: Brooks/Cole Pub. Co., 1991. ISN/OTHER No.: 23654900, 0534141307 Common LISP Modules: Artificial Intelligence in the Era of Neural Networks and Chaos Theory. AUTHOR: Mark Watson. PUBLISHER: New York: Springer-Verlag, 1991. ISN/OTHER No.: 0387976140, 3540976140 Competitively Inhibited Neural Networks for Adaptive Parameter Estimation. AUTHOR: Michael Lemmon; foreword by B. V. K. Vijaya Kumar. PUBLISHER: Boston: Kluwer Academic, 1991. SERIES: The Kluwer international series in engineering and computer science; SECS 111. Knowledge representation, learning, and expert systems. ISN/OTHER No.: 0792390865 Computer Systems that Learn: Classification and Prediction Methods from Statistics, Neural Nets, Machine Learning, and Expert Systems. AUTHORS: Sholom M. Weiss and Casimir Kulikowski. PUBLISHER: San Mateo, CA: M. Kaufmann Publishers, 1990. ISN/ OTHER No.: 1558600655
138
Y.-H. Pao
Connectionist Modeling and Brain Function: The Developing Interface. EDITORS: Stephen Jose Hanson and Carl R. Olson. PUBLISHER: Cambridge, MA: MIT Press, 1990. SERIES: Neural network modeling and connectionism. ISN/OTHER No.: 0262081938 DARPA Neural Network Study: October 1987-February 1988. AUTHOR: DARPA Neural Network Study (U.S.). PUBLISHER: Fairfax, VA: AFCEA International Press, 1988. LC Card Number: 88031655//r90 ISBN No.: 0-916159-17-5 Exploring the Geometry of Nature: Computer Modeling of Chaos, Fractals, Cellular Automata, and Neural Networks. AUTHOR: Ed Rietman. PUBLISHER: Blue Ridge Summit, PA: Windcrest, 1989. SERIES: The advanced programming technology series. ISN/OTHER No.: 0830691375, 0830631372 (pbk.) Hebbian Neural Network Simulation: Computer Program Documentation. AUTHORS: Robert G. Day and Lee J. White. PUBLISHER: Columbus, OH: Computer and Information Science Research Center, Ohio State University, 1969. SERIES: Ohio State University, Columbus, Computer and Information Science Research Center, Technical report series; OSU-CISRC-TR-69-19 Introduction to Artificial Neural Systems. AUTHOR: Jacek M. Zurada. PUBLISHER: St. Paul, New York, Los Angeles, San Francisco: West Publishing Company, 1992. ISN/ OTHER NO.: ISBN 0-314-93391-3 A n Introduction to Fuzzy Logic Applications i n Intelligent Systems. EDITORS: Ronald R. Yager and Lotfi A. Zadeh. PUBLISHER: Boston: Kluwer Academic, 1992. SERIES: The Kluwer International series in engineering and computer science; SECS 165. ISN/OTHER No.: 0792391918 A n Introduction to Neural Computing. AUTHOR: Igor Aleksander and Helen Morton. PUBLISHER: London: Chapman and Hall, 1990. ISN/OTHER No.: GB90-14110, 0412377802 (pbk) Introduction to Neural Networks. AUTHORS: Jeannette Stanley and Evan Bak. EDITOR: Sylvia Luedeking. PUBLISHER: Sierra Madre, CA 91024: California Science Software, 1988 The Metaphorical Brain 2 : Neural Networks and Beyond. AUTHOR: Michael A. Arbib. PUBLISHER: New York: Wiley, 1989. ISN/OTHER No.: 0471098531 Modeling Brain Function: The World of Attractor Neural Networks. AUTHOR: Daniel J. Amit. PUBLISHER: New York: Cambridge University Press, 1989. ISN/OTHER No.: 0521361001 Models of Neural Networks. EDITORS: E. Domany, J. L. van Hemmen and K. Schulten. PUBLISHER: Berlin, New York: Springer-Verlag, 1991. SERIES: Physics of neural networks. ISBN 0387511091 Nested Neural Networks [microform]. AUTHOR: Yoram Baram. PUBLISHER: Moffett Field, CA.: National Aeronautics and Space Administration, Ames Research Center (Springfield, VA: For sale by the National Technical Information Service, 1988). SERIES: NASA technical memorandum; 101032. ISN/OTHER No.: N 88-30373 NASA., 0830-d (MF), GOV DOC No.: NAS 1.15:101032 Neural and Automata Networks: Dynamical Behavior and Applications. AUTHOR: Eric Goles Servet Martinez. PUBLISHER: Dordrecht, Boston: Kluwer Academic 1990. SERIES: Mathematics and its applications (Kluwer Academic Publishers); Vol. 58. ISN/ OTHER No.: 0792306325 (alk. paper) Neural and Intelligent Systems Integration: Fifth and Sixth Generation Integrated Reasoning Information Systems. AUTHORS: Branko Soucek and the IRIS Group. PUB-
1.4 Neural Net Computing f o r Pattern Recognition
139
LISHER: New York: Wiley, 1991. SERIES: Sixth-generation computer technology series. ISN/OTHER No.: 0471536768 Neural and Massively Parallel Computers: The Sixth Generation. AUTHORS: Branko Soucek and Marina Soucek. PUBLISHER: New York: Wiley, 1988. ISN/OTHER No.: 0471635332 Neural Computation and Self-organizing Maps: A n Introduction. AUTHORS: Helge Ritter, Thomas Martinez and Klaus Schulten. PUBLISHER: Addison-Wesley Publishing CO., 1992. ISN/OTHER NO.: ISBN 0-201-55443-7 (hbk.), 0-201-55442-9 (pbk.) Neural Computers. EDITORS: Rolf Eckmiller and Christoph v.d. Malsburg. CONFERENCE: NATO Advanced Research Workshop on Neural Computers (1987: Neuss, Germany) PUBLISHER: Berlin, New York: Springer-Verlag, 1989. SERIES: NATO AS1 Series (Advanced Science Institute Series) F, Computer and systems sciences; vol. 41. ISN/OTHER No.: 0387508929 (U.S.) Neural Computing: An Introduction. AUTHORS: R. Beale and T. Jackson. PUBLISHER: Bristol: Hilger, 1990. ISN/OTHER No.: GB90-35434, 0852742622 Neural Computing: Theory and Practice. AUTHOR: Philip D. Wasserman. PUBLISHER: New York: Van Nostrand Reinhold, 1989. ISN/OTHER No.: 0442207433 Neural Dynamics of Adaptive Sensory-motor Control. AUTHORS: Stephen Grossberg and Michael Kuperstein. EDITION: Expanded ed. PUBLISHER: New York: Pergamon Press, 1989. SERIES: Neural networks, research and applications. ISN/OTHER No.: 008036828X, 0080368271 (pbk.) Neural Models and Algorithms for Digital Testing. AUTHORS: Srimat T. Chakradhar, Vishwani D. Agrawal and Michael L. Bushnell. PUBLISHER: Boston: Kluwer Academic Publishers, 1991. SERIES: The Kluwer international series in engineering and computer science; SECS 140. VLSI, computer architecture, and digital signal processing. ISN/OTHER No.: 0792391659 (acid-free paper) Neural Network Application to Aircraft Control System Design [microform]. AUTHORS: Terry Troudet, Sanjay Garg and Walker C. Merrill. PUBLISHER. Washington, DC: National Aeronautics and Space Administration; [Springfield, VA: For sale by the National Technical Information Service, 19911. SERIES: NASA technical memorandum; 105151. ISN/OTHER No.: N 91-27167 NASA. 0830-D (MF), GOV DOC No.: NAS 1.15:105151 Neural Networks Architectures: An Introduction. AUTHOR: Judith E. Dayhoff. PUBLISHER: New York: Van Nostrand Reinhold, 1990. ISN/OTHER No.: 0442207441 Neural Network Design and the Complexity of Learning. AUTHOR: J. Stephen Judd. PUBLISHER: Cambridge, MA: MIT Press, 1990. SERIES: Neural network modeling and connectionism. ISN/OTHER No.: 0262100452 Neural Network Models in Artificial Intelligence. AUTHOR: Matthew Zeidenberg. PUBLISHER: New York: Ellis Horwood, 1990. SERIES: Ellis Horwood series in artificial intelligence. ISN/OTHER No.: 0136121853, 0745806007 Neural Network Parallel Computing. AUTHOR: Yoshiyasu Takefuji. PUBLISHER: Boston: Kluwer Academic publishers, 1992. The Kluwer international series in engineering and computer science; SECS 0164. ISN/OTHER No.: 079239190X (acid-free paper) Neural Networks: An Introduction. AUTHOR: B. Muller and J. Reinhardt. EDITION: Corr. 2nd print. PUBLISHER: Berlin, New York: Springer-Verlag, 1991. SERIES: Physics of neural networks. ISN/OTHER No.: 3540523804 (Berlin: alk. paper), 0387523804 (New York: alk. paper)
140
Y.-H. Pao
Neural Networks and Natural Intelligence. EDITOR: Stephen Grossberg. PUBLISHER: Cambridge: MIT Press, 1988. ISN/OTHER No.: 026207107X Neural Networks and Speech Processing. AUTHORS: David P. Morgan and Christopher L. Scofield; foreword by Leon N. Cooper. PUBLISHER: Boston: Kluwer Academic publishers, 1991. SERIES: The Kluwer international series in engineering and computer science. VLSI, computer architecture, and digital signal processing. ISN/OTHER No.: 0792391446 (alk. paper) Neural Networks: Concepts, Applications, and Implementations. EDITORS: Paolo Antognetti and Veljko Milutinovic. PUBLISHER: Englewood Cliffs, NJ: Prentice Hall, 1991. SERIES: Prentice Hall advanced reference series. Engineering. ISN/OTHER No.: 0136125166 (Vol. l ) , 0136127630 (Vol. 2) Neural Networks f o r Computing, Snowbird, U T , 1986. EDITOR: John S . Denker. PUBLISHER: New York: American Institute of Physics, 1986. AIP conference proceedings; no. 151. ISN/OTHER No.: 088318351X Neural Networks for Control. EDITORS: W. Thomas Miller, 111, Richard S. Sutton and Paul J. Werbos. PUBLISHER: Cambridge, MA: MIT Press, 1990. SERIES: Neural network modeling and connectionism. ISN/OTHER No.: 0262132613 Neural Networks for Perception. EDITOR: Harry Wechsler. PUBLISHER: Boston: Academic Press, 1992. ISN/OTHER No.: 0127412514 (Vol. 1: acid-free paper), 0127412522 (Vol. 2: acid-free paper) Neural Networks: Theoretical Foundations and Analysis. EDITOR: Clifford Lau. PUBLISHER: New York: IEEE Press, 1992. ISN/OTHER No.: 0879422807 Neural Networks: Theory and Applications. EDITORS: Richard J. Mammone and Yehoshua Y. Zeevi. PUBLISHER: Boston: Academic Press, 1991. ISN/OTHER No.: 0124670504 (alk. paper) Neurale Netuaerk. In English: Neural Networks: Computers with Intuition. AUTHORS: Soren Brunak and Benny Lautrup. PUBLISHER: Singapore: World Scientific Pub. Co., 1988. ISN/OTHER No.: 9971509385, 9971509393 (pbk.) NeuralSource: The Bibliographic Guide to Artificial Neural Network. AUTHORS: Philip D. Wasserman and Roberta M. Oetzel. PUBLISHER: New York: Van Nostrand Reinhold, 1990. ISN/OTHER No.: 0442237766 Neurocomputing. AUTHOR: Robert Hecht-Nielsen. PUBLISHER: Reading, MA: Addison-Wesley, 1990. ISN/OTHER No.: 0201093553 Neurocomputing: Foundations of Research. EDITORS: James A. Anderson and Edward Rosenfeld. PUBLISHER: Cambridge, MA: MIT Press, 1988. ISN/OTHER No.: 0262010976 New Developments in Neural Computing: Proceedings of a meeting on neural computing sponsored by the Institute of Physics and the London Mathematical Society held in London, 19-21 April 1989. EDITORS: J. G. Taylor and C. L. T. Mannion. PUBLISHER: Bristol [England], New York: A. Hilger, 1989. ISN/OTHER No.: 0852741936 Orthogonal Patterns in Binary Neural Networks [microfom]. AUTHOR: Yoram Baram. PUBLISHER: Moffett Field, CA: National Aeronautics and Space Administration, Ames Research Center; (Springfield, VA: For sale by the National Technical Information Service, 1988). SERIES: NASA technical memorandum; 100060. ISN/OTHER No.: A-88068., 0830-D (MF), GOV DOC No.: NAS 1.15: 10060
1.4 Neural Net Computing for Pattern Recognition 141 Pattern Recognition b y Self-organizing Neural Networks. EDITORS: Gail A. Carpenter and Stephen Grossberg. PUBLISHER: Cambridge, MA: MIT Press, 1991. ISN/OTHER No.: 0262031760 The Perception of Multiple Objects: A Connectionist Approach. AUTHOR: Michael C. Mozer. PUBLISHER: Cambridge, MA: MIT Press, 1991. SERIES: Neural network modeling and connectionism. ISN/OTHER No.: 0262132702 (hc) Physical Models of Neural Networks. AUTHOR: Tamas Geszti. PUBLISHER: Singapore: World Scientific, 1990. ISN/OTHER No.: 9810200129 A Real Time Neural Net Estimator of Fatigue Life [microform]. AUTHOR: T. Troudet and W. Merrill. PUBLISHER: Washington, DC: National Aeronautics and Space Administration; [Springfield, VA: For sale by the National Technical Information Service, 19901. SERIES: NASA technical memorandum; 103117. ISN/OTHER No.: N 90-21564 NASA., 0830-D (MF), GOV DOC No.: NAS 1.15:103117 Recursive Neural Networks for Associative Memory. AUTHORS: Yves Kamp and Martin Hasler. PUBLISHER: Chichester, New York: John Wiley and Sons, 1990. SERIES: Wiley-Interscience series in systems and optimization. ISN/OTHER No.: 0471928666 Simulation Tests of the Optimization Method of Hopfield and Tank Using Neural Networks [microform]. AUTHOR: Russell A. Paielli. PUBLISHER: Moffett Field, CA: National Aeronautics and Space Administration, Ames Research Center; [Springfield, VA: For sale by the National Technical Information Service, 19881. SERIES: NASA technical memorandum; 101047. ISN/OTHER No.: A-88275 Structure Level Adaptation f o r Artificial Neural Networks. AUTHOR: Tsu-Chang Lee; foreword by Joseph W. Goodman. PUBLISHER: Boston: Kluwer Academic publishers, 1991. SERIES: The Kluwer international series in engineering and computer science; SECS 133. Knowledge representation, learning, and expert systems. ISN/OTHER No.: 0792391519 VLSI Design of Neural Networks. EDITOR: Ulrich Ramacher. PUBLISHER: Boston: Kluwer Academic Publishers, 1991. ISN/OTHER No.: 0792391276
Appendix B A list of names of journals wholly or partially devoted to publishing neural net computing articles: Advances in Connectionist and Neural Computation Theory. Frequency: Irregular. PUBLISHER: Ablex Publishing Corp., 355 Chestnut St., Norwood, NJ 07648. Tel: (201) 767-8450. EDITOR: John Barnden. Biological Cybernetics. Frequency: monthly. PUBLISHER: Springer-Verlag, Heidelberger Platz 3, D-1000 Berlin 33, Germany (also in New York). Tel: 030-8207-1. EDITOR: W. Reichardt. IEEE Transactions on Neural Networks. Frequency: Bi-monthly. PUBLISHER: IEEE, Inc., 345 E. 47th St., New York, NY 10017-2394. Tel: (212) 705-7366. Subscriptions to 445 Hoes Lane, Box 1331, Piscataway, NJ 08855-1331. Tel: (908) 562-3948. EDITOR: Herbert Rauch. IEEE Transactions on Pattern Analysis and Machine Intelligence. Frequency: monthly. PUBLISHER: IEEE, Inc., 345 E. 47th St., New York, NY 10017-2394. Tel: (212) 705-7366. Subscriptions to 445 Hoes Lane, Box 1331, Piscataway, NJ 08855-1331. Tel: (908) 562-3948. EDITOR: Anil K. Jain.
142
Y.-H. Pao
International Journal of Neural Networks. Frequency: quarterly. PUBLISHER: Learned Information, Inc., 143 Old Marlton Pike, Medford, NJ 08055. Tel: (609) 654-6266. EDITOR: Kamal Karna and Ian Croall. Journal of Parallel and Distributed Computing. Frequency: monthly. PUBLISHER: Academic Press, Inc., JOURNAL Division, 1250 Sixth Ave., San Diego, CA 92101. Tel: (619) 230-1840. EDITOR: Kai Hwang and Howard Siegel. Neural Computation. Frequency: quarterly. PUBLISHER: MIT Press, 55 Hayward St ., Cambridge, MA 02142. T e l (617) 253-2889. EDITOR: Terence Sejnowski, Salk Institute, Box 85800, San Diego, CA 92138. Neural Network Review. Frequency: quarterly. PUBLISHER: Lawrence Erlbaum Associates, Inc., 365, Broadway, Hillsdale, NJ 07642. Tel: (201) 666-4110. EDITOR: Craig Will. Neural Networks. Frequency: Bi-monthly. PUBLISHER: Pergamon Press, Inc., JOURNALS Division, Maxwell House, Fairview Park, NY 10523. Tel: (914) 592-0770. Neurocomputing. Frequency: Bi-monthly. PUBLISHER: North Holland (Subsidiary of Elsevier Science Publishers B. V.), P.O. Box 211, 1000 AE Amsterdam, Netherlands. EDITOR: V. David Sanchez. Pattern Recognition. Frequency: monthly. PUBLISHER: Pergamon Press, Inc. , JOURNALS Division, Maxwell House, Fairview Park, NY 10523. Tel: (914) 592-0770. EDITOR: Robert Ledley. Progress i n Neural Networks. Frequency: annual. PUBLISHER: Ablex Publishing Corp., 355 Chestnut St., Norwood, NJ 07648. Tel: (201) 767-8450. EDITOR: Omid Omidvar.
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 143-181 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
LCHAPTER
1.5
I
ON MULTIRESOLUTION WAVELET ANALYSIS USING GAUSSIAN MARKOV RANDOM FIELD MODELS
C. H. CHEN and G. G. LEE Department of Electrical and Computer Engineering, University of Massachusetts Dartmouth, N . Dartmouth, M A 0274 7, USA This chapter presents a novel algorithm for image segmentation and classification via the use of the multiresolution wavelet analysis and the expectation maximization (EM) algorithm. The development of a multiresolution wavelet feature extraction scheme is based on the Gaussian Markov random field (GMRF) assumption in image modeling. Real-world images are hierarchically decomposed into different resolutions. In general, larger image components are characterized by coarser resolutions whereas higher resolutions show finer and more detailed structures. These hierarchical variations in the anatomical features displayed by multiresolution decomposition is further quantified through the application of the Gaussian Markov random field. Due to its uniqueness in locality, adaptive features based on the nonstationary assumption of GMRF are defined for each pixel of the image. Distinct image regions are then segmented via the Fuzzy c-Means (FCM) algorithm using these localized features. Subsequently the segmentation results are further enhanced via the introduction of a maximum a posteriori (MAP) segmentation estimation scheme based on the Bayesian learning paradigm. Gibbs priors or Gibbs random fields have also been incorporated into the learning scheme with very effective outcomes. In this chapter, the expectation maximization (EM) algorithm for MAP estimation will be formulated. The EM algorithm provides an iterative and computationally simple algorithm based on the incomplete data concept. The algorithm presented is applied to digital mammograms and wafer inspection images.
Keywords: Image segmentation, multiresolution wavelet analysis, Gaussian Markov random field, adaptive local features, EM algorithm, textured images, digital mammograms, wafer inspection.
1. Introduction In recent years, considerable interest has risen concerning new transform techniques and image models for the analysis of real-world images [l].Presently, much of the work of scientists, engineers and even medical doctors relies significantly upon successful extraction of information embedded in different image modalities originated from biomedical, manufacturing, and remotely sensed applications. Many of these intelligent behaviors have been automated by computers. Indeed, automated analysis and interpretation of information conveyed by images have become very important tools in several aspects of science, engineering, and medicine. Moreover, due to limitations of even the most recent and advanced image capturing 143
144
C. H. Chen 6Y G. G. Lee
systems, the recorded images are inevitably degraded in many ways. Thus image analysis algorithms based on new image transform techniques and image modeling paradigms for segmentation and feature extraction are now strongly sought for. In this chapter we discuss the issues of segmentation and classification of natural images using the spectral-spatial properties of the multiresolution wavelet transform in conjunction with the contextual information provided by the Markov random field models. Image segmentation via automated machine recognition algorithms has aroused considerable interest in recent machine vision literature [l]. In the segmentation problem, each disjoint region of the image is assumed t o be a different class. The task of an image segmentation algorithm is to find an optimal classification which best characterizes the regions of the image. In the deterministic approach, a significant emphasis has been placed on the study of textures based on wavelet analysis [2]. Multi-channel and/or tree-structured wave-packets have also been developed for classification purposes. Furthermore many researchers have discovered significant advantages in the use of the multiresolution concept [3,4]. The concepts of multiresolution wavelet analysis (MWA) will be discussed in Section 2. As a result of cross-fertilization of innovative ideas from image processing, spatial statistics and statistical physics, a significant amount of research activities on image modeling and segmentation have also been concentrated on the 2-D Markov Random Field (MRF). Although much of the potential of MRF had been envisioned by the early works of Levy, McCormick, and Abend et al. [5-71, exploitation of the powers of the MRF was not possible until significant recent advances had been made in the appropriate mathematical and computational tools. Kashyap and Chellappa [8] successfully applied the non-causal autoregressive (NCAR) model or the Gaussian Markov random field (GMRF) to the characterization of real world textural images. Woods [9] also reported on the issues of two dimensional discrete Markovian fields. As a result of the MRF and Gibbs distribution (GD) equivalence [10,11],Cross and Jain, Geman and Geman, and Derin and Elliott have also demonstrated substantial successes in image segmentations and restoration [12-141. The fundamental concepts and issues concerning the 2-D MRF’s will be described in Section 3. In Section 4, we introduce a novel multiresolution wavelet analysis and MRF based algorithm for image segmentation. In many of today’s real world applications such as the low contrast mammograms or wafer images, it is necessary that the recognition algorithm be capable of categorizing the complex 2-D signals with considerably high accuracy. Thus many of the real world image processing paradigms necessitate the combination of both the deterministic and stochastic approaches of image studies. The task of automated image recognition can in general be achieved in two steps. The first step involves a careful selection of features that best characterize the class membership of the patterns. The second step requires that a good classifier be designed to differentiate the given or measured patterns based on the information provided by the selected features with minimum error. The performances of classifiers in image recognition vary with the nature of the
I . 5 O n Multiresolution Wavelet Analysis Using. . . 145 images to which they are applied. Thus characterization or modeling of the class of images that are t o be processed or analyzed provides not only profound understanding of image structures but also renders a means of feature extraction. The mammographic and wafer images considered in this chapter are most effectively represented by both the NCAR and GD models. The nonstationary GMRF model was used in the extraction of adaptive local features. Moreover, in computer vision problems, grayscale values of the images are generally insufficient for the tasks of differentiation and interpretation. Many types of natural patterns or primitives consist of hierarchical structures that reveal different information under different resolutions and are not generally available in a single resolution. In general, coarse resolution images reveal macrostructures whereas microstructures can be observed in finer resolutions. These hierarchical pattern variations can be systematically quantified by the GMRF applied a t different resolutions of the image. Chellappa and Krishnamachari [15] introduced Markov filters for wavelet analysis which retains the Markovianity of the lower resolution images after decomposition. In our previous works [16-181 we have shown in both the 1-D and 2-D cases that it is necessary to analyze the signals and select stochastic features from different resolutions for classification. In this chapter, a novel statistical feature extraction scheme using MRF under a multiresolution wavelet decomposition framework is introduced. Many of the recent efforts have also been made on the development of unsupervised segmentation techniques. Several authors [ 191 have reported on the successful applications of K-means clustering for unsupervised image segmentation. The outcomes of the K-means algorithm are based primarily on hard decision which are in general less efficient than the soft decisions from an information point of view. Thus soft decision making classifier or the fuzzy-c-means (FCM) algorithm introduced by Bezdek [20,21] will be used for classification or initial image label estimations. The multiresolution wavelet analysis and MRF feature extraction scheme together with the FCM classifier thus provide a novel paradigm for unsupervised segmentation. This unsupervised segmentation paradigm is outlined in Section 4. In the study of real-world images, prior experiences or a priori data provide significant information for segmentation. It is therefore crucial that the a priori expertise be incorporated into the learning processes of the recognition algorithms. A maximum a posteriori (MAP) segmentation estimation scheme based on the Bayesian learning paradigm will be introduced in recognition of the potential contribution of machine learning to the development of robust computer vision algorithms. The goal of this learning task is aimed a t the development of perceptive capabilities of the computers in decision making based on information acquired from past experiences. Model-based and knowledge-based segmentation have been studied by many authors [14,22] with much successes. The essence of these MAP estimations relies primarily on first problem formulation under a Bayesian framework followed subsequently by an estimation or optimization of the a posteriori distribution. Having assumed some prior understanding or information on the image by the specification of an a priori distribution, the Bayesian framework allows
146
C. H. Chen & G. G. Lee
the computer to learn and accumulate experiences from the observed or measured image data for decision making. Gibbs priors have been incorporated into the learning scheme of many researchers for the purpose of segmentation with very effective results [13,14,23]. The MAP estimation problem usually results in the minimization of the energy function contained in the a posteriori distribution. The main difficulty of the problem is primarily due to the existence of nonconvexity in the energy function so obtained. Thus it is not impossible that the estimations be trapped in local minimas of the energy function. As a remedy to this difficulty, different approaches for global optimization have also been introduced by several authors. Inspired by the applications in statistical physics [24], Geman and Geman I131 have demonstrated significant success via the stochastic relaxation technique of simulated annealing. To avoid the pitfall of dropping into the local minima, the stochastic simulated annealing consists of random fluctuations that free the estimations from these stable points. Blake and Zisserman [25] introduced the graduated nonconvexity (GNC) algorithm in which an approximating convex function to the original energy or objective function is assumed. The construction of these convex approximations however is effected by the specification of the controlling parameters of the system. Geiger and Yuille [26] reported on the use of mean field theory
Multiresolution Wavelet Decomposition
Gaussian Markov Random Field
.............................................................................................................................
(a)
I
I
I
...........................................................................
Standalone,Non-supervised Algorithm (b) Fig. 1. (a) MWA & GMRF Scheme for Microcalcification Detection; and (b) Image Segmentation Scheme for Subtle Mass Localization.
1.5 O n Multiresolution Wavelet Analysis Using. . . 147 and tree approximations have also been investigated [27]. Most of the concurrent research are aimed at the reduction of computational complexity and sensitivity to selection of model parameters during MAP estimations. In this chapter the expectation maximization (EM) algorithm for MAP estimation will be formulated. The EM algorithm is an iterative and computationally simple algorithm based on the incomplete data concept. In this formulation, the observed image pixels are assumed to be the incomplete data whereas the class label or status of each pixel is assumed to be unknown. The initial labels estimated from the previously mentioned unsupervised FCM classification will then serve as an initial estimate to the EM algorithm. The EM algorithm is formulated in Section 5 for MAP estimation. The block diagram of the overall novel algorithm is shown in Fig. 1. 2. Wavelet Analysis The main concept of multiresolution wavelet analysis (MWA) can be understood by either the top-down algorithm from the inner product viewpoint or it can alternatively be considered as the fast wavelet herringbone algorithm providing a bottom-up interpretation of the transform. In the top-down algorithm, the coarser resolution wavelet coefficients are computed before the finer resolution wavelet coefficients. On the contrary, the herringbone bottom-up algorithm starts with the computation of finer resolution wavelet coefficients. In the following two subsections, we provide discussions of both the top-down and bottom-up interpretations of the wavelet transform respectively. 2.1. Top-Down Algorithm In the top-down algorithm of wavelet analysis, a function can be represented as a superposition of wavelet basis functions [28,29]. From a single finite duration mother wavelet $(x) a family of orthonormal bases functions can be obtained through dilations and translations:
where m and n are integers. By variations of the integer n, the translational process transforms the function into a separate vector space with distinct spatial orientation. Similarly, by changing m, the dilation process amounts t o a transformation of the original function into another vector space possessing a different resolution. Representing the vector spaces by Vm’s where m is the resolution, we have,
A function f (x)can then be decomposed by the equation:
m.n
148
G. H . Chen & G. G. Lee
where c,,,’s
are the wavelet coefficients calculated by the inner product via
L oc)
cm,n =
(f(.),
$m,n(x)) =
f(z)$m,n(z)dz.
(2.4)
Each of the wavelet coefficient c ~ therefore , ~ represents the resemblance of the function f(z)to the wavelet bases $)m,n(z) at a specific resolution m and translation n. In the construction of the mother wavelet, it is necessary that an appropriate scaling function 4(z) be first chosen. According to the work of Daubechies [28] this scaling function should satisfy the two-scale difference equation, (2.5) k
Thus if estimation of the mother wavelet is straightforward by the equation
The impulse responses hl(Ic) and hz(Ic) constitute a pair of conjugate Quadracture Mirror Filters (QMF) with frequency responses H1 and H2 respectively. The QMF filters can be constructed with perfect reconstruction or synthesis of the original signal if certain conditions of h l ( k ) and h~(Ic) are satisfied. Thus in the top-down algorithm of the wavelet transform, one computes first the coarse resolution coefficients via the inner product of Eq. (2.4). 2.2. The Herringbone Bottom- U p Interpretation In the herringbone interpretation, the discrete wavelet transform can be considered an efficient application of the two-band subband coding in an iterative fashion. The first step in the wavelet decomposition starts with a simple two-band subband coding scheme in which the signal is initially convoluted or filtered by the half-band lowpass filter with impulse response hl ( k ) followed by subsequent sub-sampling. Similarly the same input function is also convoluted with the half-band highpass filter h2(k) followed by subsequent sub-sampling. The respective outputs 91 ( I c ) and 92(Ic) of the lowpass and highpass filters can then be represented as,
n
n
where f ( n )represents the discrete sequences of the continuous signal f(z).Thus if
H,2+H2=1,
(2.10)
1.5 O n Multiresolution Wavelet Analysis Using. . . 149
f(n)can be reconstructed perfectly using the equation,
c Dc)
f(n)=
[g1(k)h1(2k - n ) + gz(k)h2(2k - n)l
.
(2.11)
k=-cc
In the second subband coding iteration of the discrete wavelet transform, the lowpass signal gl(k) is the input to another two-band subband coding scheme. This process is continued until the lowpass signal consists of only a single point. The original signal f ( n )can of course be reconstructed by putting the lowpass and highpass signals back into the subband coding scheme iteratively using Eq. (4.11). It is evident from the iterative subband coding scheme of the discrete wavelet transform that the finer resolution schemes are computed prior t o that of coarser resolution coefficients. As in the herringbone structure, each set of wavelet coefficients is computed via the convolution of f ( n ) repeated with hl(k) and then once with h z ( k ) . Thus in a way analogous to the top-down scheme described from the inner product point of view, the orthonormal basis functions of the present herringbone scheme are h2(k) and other functions resulting from convolution with hz(k). Observing the equivalent structures of Eqs. (2.5) and (2.8), and Eqs. (2.7) and (2.9), the highpass impulse response can be interpreted as the mother wavelet where the set of wavelet basis functions is derived from the convolution with the highpass filter. Thus in the top-bottom algorithm, the scaling functions in a sense are an intermediate step for the generation of wavelet basis as in the herringbone scheme. By the same token, in two dimensional image analyses, by choosing the analyzing wavelet +(x, y) to be localized in space, spatial information such as shapes and orientations of the images can be emphasized. The multiresolution wavelet representation scheme shown in Fig. 2 allows an image to be decomposed into a hierarchy of localized subimages at different spatial frequencies. This representation divides the 2-D frequency spectrum of an image Y into a lowpass subband image Y$ and a set of bandpass subimages where i = 1,.. . , L and j = 0 , 1 , 2 , 3 . The integer i represents the number of resolution levels used while j represents the number of orientations. Thus the q ’ s for j = 1 , 2 , 3 represent the detail subimages obtained as a result of the 2-D wavelet decomposition outlined in Fig. 2. H1 and H2 represent respectively the lowpass and highpass conjugate quadrature mirror filters (QMF) . The notation y3” = W,,j[Y] (2.12)
q
represents the wavelet decomposition or the subimage of Y a t the ith level and spatial orientation j. Thus by tacitly choosing the resolution level i and spatial orientation j, the multiresolution wavelet scheme provides a natural hierarchy for the embodiment of an iterative paradigm for accomplishing spectral-spatial feature analysis. One may then visualize the analyzed image on the computer from a coarse to fine matching strategy. The transform scheme thus enabIes us t o first visualize the coarse features of the images embedded in the lower frequency components or
150
C. H. Chen & G. G. Lee
Y'
Keep one column out of two:
)2.11)
Keep one row out of two: Convolution with filter H :
(4
Fig. 2. (a) Two dimensional wavelet decomposition of the image Y iby a two-channel QMF. Y;+l represents the resulting low resolution subimage while Y:+', Y;+l, Y;*' denoted the different spectral-spatial detail subimages; and (b) Disposition of the lower resolution and detail snbimages shown in this paper.
resolutions of the transformed images followed by examination of the finer details contained in the higher frequency levels or resolutions. The spectral-spatial and multiresolution properties of wavelet transform have been reported to be intrinsic to the human visual system [30]. Specialized cortical neurons are known to respond specifically to stimuli within certain spatial orientations and frequencies. As will be shown in Section 6, these properties of the MWA provide significantly efficient tools for the extraction of localized spectral-spatial features such as the microcalcifications in digital mammograms [31].
1.5 On Multiresolution Wavelet Analysis Using. . . 151
3. Markov Random Field for Image Modeling Pertinent contextual information contained in natural images is not generally provided by the two-channel wavelet transform. Contextual information extracted via a Gaussian Markov random field (GMRF) and incorporated into the localized spectral-spatial details of MWA can provide remarkable improvements in image segmentation. This subsection presents a brief overview of the Gaussian Markov random field and Gibbs Distribution. Gibbs priors are used in subsequent sections for the expectation maximization. The readers are referred to [13,14,32,33]for more details on the topic.
3.1. Gaussian Markov Random Field Let an image Y be modeled by a finite lattice Gaussian Markov random field (GMRF) where Y = {yij : 0 5 i 5 M , 0 5 i 5 M } and L = {(i, j ) : 0 5 i 5 M , 0 5 i 5 M } . The positions or sites of the pixel in the M by M square lattice, L are denoted by ( i , j ) . For notational conveniences, the pixel sites are also represented as s, r, t , etc. A neighborhood system of the given lattice L is any collection of subsets of L described as q = {qij : ( i , j ) E L , qij C L } such that (i) ( i , j ) is not an element of qij and (ii) if (k,Z) is a n element of qij, then ( i , j ) should be a n element of ( i , j ) which is an element of L.
qkl
given any
The systematic and sequential ordering of the neighborhood system commonly used in image modeling is qo, where 0 = {1,2,3, . . .} represent the order of the neighborhood system. The relative positions of the pixel in the ordered neighborhood system are shown in Fig. 3(a). Due to the finite lattice approximation, the sum of pixels, say (i, j ) ( k ,I ) , are evaluated in modulo which is equivalent to the assumption of L being a toroidal lattice. In modeling the image Y as a GMRF with respect to a certain neighborhood system 17 and reshaping Y into a single vector y = [yl,y2, . . . ,yM2ITin the lexicographic order," the image Y will then be assumed to be a set of jointly Gaussian random variables that also possess the Markov property. Thus the joint probability density function of the random variables constituted by the pixel values in y has the form,
+
where C is the covariance matrix of y. In addition, the Markovianity property requires that, P(YijlYpq7
(P,Q) # ( i , j ) )= p ( Y i j l y ( i , j ) + ( k , ~ ) (, k ,1) E % j ) .
"T represents matrix transpose.
(3.2)
152
C. H. Chen €d G. G. Lee
Fig. 3. (a) Ordered neighborhood system vo for MRF. The numbers indicate the order of the model relative t o x; and (b) The clique types in second order neighborhood system, q 2 .
In the above Markovian expression, the two-dimensional pixel site notation was used whereas the joint Gaussian distribution expression assumed a lexicographic ordering of the pixel sites. Thus under the GMRF assumption of image modeling any pixel ( i , j ) can be predicted by the linear combination of the pixels contained in the neighborhood. That is Yij
=
eklY(i,j)+(k,l)
+ &Wij
7
( i , j )E L
(3.3)
(k,l)Evij
where Bkl are the parameters of linear combination and w i j is a white Gaussian noise with zero mean and unit variance [32]. Due to the definition and choice of the neighborhood system, it can be seen that the above linear combination scheme results in a noncausal and autoregressive (NCAR) prediction. It is clear from the above discussion that the key in image modeling via GMRF lies in the specification of the neighborhood system together with the parameters of linear combination characterizing the image. In our example in the present chapter, we have chosen the second order neighborhood system, q2. Innovative applications of the NCAR MRF were documented in [8,34]. Natural textures were modeled using the GMRF and the issue of the proper selection of
I . 5 O n Multiresolution Wavelet Analysis Using. . . 153 the neighborhood system were also discussed. In the present context, GMRF was chosen due to its close resemblance to the real world textural structure. This is of crucial importance for texture analysis in mammograms and wafer images as significant statistical information can be obtained via MRF modeling. Nonstationarity will require the adaptation of features to the local variations of pixel gray levels. The issue of the choice of the neighborhood system will be discussed further in a subsequent section on multiresolution feature extraction. 3.2. Gibbs Distribution
Gibbs distribution provides a powerful tool for the formulation of the Bayesian learning framework. The origin of GD lies primarily in physics and statistical mechanics. In this subsection, we will briefly discuss the class of GD used in this chapter. The readers are referred however to [10,13] for more detailed and thorough explanations of GD. In the specification of a GD, it is necessary that the cliques be defined in association with the neighborhood system as depicted in the last subsection. Denoted by c, a clique of the graph ( L ,77) is defined to be a subset of the lattice L such that all the pairs of individual sites in c are also neighbors of each other. The clique types for the second order neighborhood system q2 are shown in Fig. 3(b). The set of cliques associated with a specific order of neighborhood system is denoted as C. Let the random field or image Y = be defined over the graph ( L ,77). Then Y is a MRF with respect to 77 if and only if its joint distribution is of the form
{x,}
where (3.5) CEC
is the energy function and Vc(y) is the potential associated with the clique c. In addit ion, Y
represents the partition function which is a normalizing constant obtained by the summation over all the gray level G = (91, g2, . . . , gN}. The clique potentials Vc(g) are constituted only from the pixel values within the clique c. As with its GMRF counterpart, the specification of a neighborhood system is sufficient for the definition of a Gibbs distribution for the image at hand. That is by a proper choice of t,he clique potentials, a wide variety of images can be characterized efficiently. Once again, the second order neighborhood system is assumed in the present chapter. However only the single pixel and double pixel cliques are considered. This results in the following energy function,
1-54 C. H. Chen & G. G. Lee
U(Y) = POYij + Pl[Yi-l,j + Yi+l,jl
+ PZ[Yi,j-l + YZ,j+I] f P3[%-l,j+l f Yi+l,j-l]
+ P4[!/-l,j-l
f Yi+l,j+l].
(3-7)
The parameters in the energy function control the extent of pixel clustering in specific directions described by the corresponding cliques. From the above energy function, image intensity is controlled by the quantity ,Do, whereas PI, pz, P 3 , and ,B4 control the degree of clustering of the image pixels in the vertical, horizontal, crossdiagonal and diagonal directions respectively. This idea, which originated from the modeling of ferromagnetism in statistical physics [34], was applied to image modeling of textural images in [12,34]. As a result of the Hammersly-Clifford theorem which was proved independently by several researchers [10,11], there is a one-to-one correspondence between MRF and Gibbs random field (GRF). Image modeling via both the MRF and GRF assumptions provides a powerful tool in image analysis from both the mathematical and computational perspectives. Due to the nonhomogeneous property of real world images studied in this chapter, it is essential that a nonstationary GMRF assumption be made for the extraction of locally adaptive features. On the other hand, since GD is basically exponential in nature, it can be used with more convenience in the formulation of MAP segmentation estimations and in the EM algorithm. Thus utilization of both assumptions have revealed significant results in the present study. 4. Multiresolution Feature Extraction Scheme and Fuzzy Clustering for Image Segmentation In this section we will introduce the multiresolution wavelet analysis (MWA) and Gaussian Markov random field (GMRF) based feature extraction scheme. By using this feature extraction scheme, adaptive local GMRF features are extracted from different resolutions of the wavelet decomposed image. Each pixel in the original image is characterized by a vector of discriminant features. We assume that the constituent regions in natural images are characterized by distinctive textures. Together with FCM clustering, a novel unsupervised segmentation algorithm is presented for image labeling. 4.1. Wavelet Decomposition and GMRF for Feature Extraction
In the segmentation of real-world images, the multiresolution hierarchical framework of the wavelet transform can be used with significant advantage. Although the Fourier transform provides one of the earliest ways of signal or image decomposition, it performs best primarily for periodic signals and images. As a result of the infinite duration of the complex sinusoids serving as the basis function of the inner product in the transform integral, Fourier decomposed signals do not display the locality
1.5 O n Multiresolution Wavelet Analysis Using.. . 155
Fig. 4. (a) Simple Image; (b) Corresponding Image; (c) 1st pass in Wavelet; and (d) Corresponding Subimages.
property in the spatial domain. Due to the choice of the mother wavelet and the corresponding QMF's described in Section 2, the conjugate filters H1 and H2 serve as a two-channel QMF representation of the image. As a result of their finite duration, both frequency and spatial localities can be observed in the wavelet decomposed image. Since the detailed image contains the higher frequency components of the original image in different orientations, abrupt changes in grayscales such as edge information can be observed. This can be seen from the simple example shown in Fig. 4. Figure 4(a) shows a 64 x 64 image with a simple background having a grayscale of approximately 100. The inner square consists of pixels having grayscales in the vicinity of 200. Figure 4(c) is the result of the first pass in the wavelet decomposition scheme shown in Fig. 2. It can be seen that Y;" emphasizes on the edge of the square in the horizontal direction whereas Y;" shows the edge in the vertical direction. Thus the spatial and frequency domain localization property of the wavelet transform makes it a very efficient tool for image segmentation. However, Fig. 4(a) is a simple image with only two uniform grayscale regions. Figure 5(a) shows a more complex picture with a square texture in the middle of a square background with different textures. Although still placing emphasis on the horizontal and vertical directions, Y;" and Y;'' of Fig. 5(c) were not able t o show
156
C.H. Chen B G. G. Lee
Fig. 5 . (a) Natural Textile; (b) Corresponding Image; (c) 1st Pass in Wavelet; and (d) Corresponding Subimages.
clear cut edges of the square as they did in Fig. 4(c). Thus the two-channel QMF does not in general provide sufficient statistical information contained in real world images necessary for segmentation purposes. This necessitates the introduction of the Gaussian Markov Random Field in conjunction with wavelet decomposition for image analysis. The locally interactive pixels and statistically dependent properties of the GMRF model provide a remedy to the nonuniformity of natural textures in real-world images. As was described in Section 3 , each pixel in the GMRF can be predicted by the linear combination of pixels in the carefully chosen neighborhood system. The parameters in the linear combination can be found from the least square estimate over the entire image [8,34]. In natural images, each pixel and region can display significant variations and complexities that are neither periodic nor homogeneous. The inherent assumption of uniform discriminatory properties at local regions or pixels of real world images such as mammograms is in general inadequate. Thus local features which adapt to each of the individual pixels of the image are selected. In other words, the least square estimation is performed on only a small window centered a t the corresponding pixel rather than on the entire image. In this chapter, the second order neighborhood system or the first eight neighbors of each pixel are
1.5 O n Multiresolution Wavelet Analysis Using. . . 157
used for the estimation of the linear combination parameters. That is if we let,
the least square estimates of the parameters of the eight neighbors of each pixel of y i j are
where the variance estimate at each pixel is
with w being the size of the window. The mean estimates at each pixel are then taken to be,
Therefore each pixel of the image at a specific resolution can be characterized by a set of three features. The first two features arise from the mean and variance estimates and the last is the rotational invariant feature estimated from the sum of parameters of the linear combination estimated from the pixels in the chosen window. Each of the parameters estimated in O i j represents the influence of the corresponding pixel in the neighborhood system on the pixel under study from one of the eight directions. Thus the effects of all eight pixels in the neighborhood from the eight different directions are combined by summing the eight parameters. As a result of the estimation from the windows defined around each pixel, the features will then adaptively reflect the local variations or properties of the textured image. Therefore different regions of the image will be characterized by discriminant local features essential for segmentation or region labeling purposes. Pixels in the same region will be characterized by similar features whereas different regions are modeled by different discriminant features. It will be shown that greater variations of the features will be observed in different image regions whereas less variations of the features can be observed from within the region. These changes or variations of the features within the image are measured by the local variances calculated from the pixels of the feature maps, that is
In the former equation, w represents again the size of the window used in estimating the feature variance, fj:’ represents the ( i , j ) pixel of the kth feature extracted
158
C. H. Chen t3 G. G. Lee
from the nonstationary GMRF described above. The mean, estimation is,
$’”’
of the variance
At each resolution of the image, the variance in effect reveals the relative importance of the corresponding features. Thus every extracted feature at each image pixel is multiplied by its corresponding variance estimates which functions as the weight of a specific feature. Intuitively, discontinuities of gray levels can exist at the edges of the constituent regions in the image. Therefore, if we begin by examining a coarse resolution of the studied image larger structures will be observed with more abrupt changes and larger steps in grayscales. In the course of gradually increasing the resolution, finer details and smaller patterns can then be seen. The changes in the gray levels at the edges of the higher resolution levels in the wavelet pyramid are indeed less. Thus the coarser the resolution, the more abrupt changes there will be in the gray levels of the image. These changes of gray levels at the edge between different regions from different resolutions of the image are quantified by the application of the nonstationary GMRF at each resolution of the wavelet pyramid. Given the original image Y , wavelet decomposition is first applied to find the lower resolution images. Subsequently, features are extracted at each resolution of the image via the nonstationary GMRF model. The features estimated from the lower resolutions are then restored to the original resolution size via simple upsampling. Thus each pixel of the image is characterized by a single discriminant vector consisting of GMRF features estimated from every resolution of the wavelet decomposed image. That is each pixel of the image is characterized by the feature vector,
(4.7)
where f $ ) R is the kth feature of the image pixel ( i , j ) at resolution R. In this study, only two resolutions were used. That is features are extracted from the three images Y:, Yo”,and Yo”. It can be seen from the pixel feat,ure vector that features from different resolutions are embedded. In this manner, information from lower resolution images is propagated t o the higher resolution. The importance of each of the features is weighted by the corresponding variance. This scheme provides a means of careful scrutiny of the individual image pixels which are of great significance for natural images. The inclusion and/or exclusion of a certain pixel in a specific region is then decided by the FCM algorithm based on the discriminatory information conveyed by the GMRF features extracted from each resolution of the wavelet pyramid. These adaptive features will be shown to have highly effective outcomes especially for the nonhomogeneous images such as mammograms and wafer inspection images. The FCM algorithm will be described in the following subsection.
1.5 O n Multiresolution Wavelet Analysis Using. . .
159
4.2. Fuzzy-C-Means ( F C M ) Clustering
Having found the characteristic indicators for different regions in the image represented by pixel gray levels in Section 3, the subsequent task is the partitioning of the set of pixels into the corresponding regional sets. Thus given the set Y = {yi : 0 5 I 5 m2 - 1 = n } ,we intend to find the optimal partition of Y exhibiting categorically homogeneous subsets.b This can be achieved by minimizing a cost [20,21] defined from the similarity measures of feature vectors selected for each of the pixels in the set Y . Before proceeding to the definition of cost function, it is necessary that we review some basic definitions of fuzzy reasoning. Let P ( Y ) denote the algebra of the power set of Y . In other words, P ( Y ) is the set of subsets of Y . Let S be an element of P ( Y ) ,then the function us is defined as the characteristic function which maps the set of elements Y into the set {0,1}. The function us defined as u s = { l , a ~ ”
0, otherwise
}
is the characteristic function of hard subset S c Y . Only one characteristic function us corresponds to every S which is an element of P ( Y ) . The fuzzy subset of Y is thus the function u which maps the set Y into the inclusive set [0, 11. Now suppose that the set Y is made up of mutually exclusive and exhaustive hard subsets of S subsets denoted by,
Y = S ] u s2 u...sc.
(4.9)
Following this notation, the characteristic functions of Sq’s where q = { 1 , 2 , . . . , c } can be represented as uq, (4.10) Having defined the basic notations, a hard c partition of Y can be defined as the set, (4.11) MC = {U E Rcn : uq/eE {0,1}} such that C
for any !
(4.12)
q=l
and
n
(4.13) bThe index k represents one of the coordinate pairs in the M by M square lattice. By using this notation, we have again assumed a lexicographical ordering of the image pixels as in the GMRF. The number n here is the number of pixels in the image and is different from the interger n used in the wavelet decomposition scheme.
160
C. H. Chen €9 G. G. Lee
In words, the hard c-partition of a set Y is the set of c by n matrix with each element of the matrix being zero or one such that the previous two conditions are true. The fuzzy partition of Y is then defined to be,
M f , = {U E 8'" such that
: uqe E [O,1]}
(4.14)
c
(4.15) q=l
and
n
0
0), (0 5 z < n - l , y = p - 1 , z > 0), ( z = O , O 5 y 5 p - 1 , z > O), ( z = n - 1 , O I y 5 p - 1 , z > O)}
577-1
-----r.
n*p*m hollow (up) brick
or ln-l 2P-1 C grnp1 7, where C is the set of cells defined above within the brackets {}. An n x p x m hollow(up) brick is shown in Fig. 10. It is interesting t o compare the figures in Examples 4.1-4.6 with the object figures in [4,8,16,17,19],which are typical 3-D illustrations used for analyzing 3-D objects .
Z
Fig. 10. An upward hollow brick realized from a real 3-D object image in different gray levels.
Example 4.4. The pyramid is an interesting object, which has been widely used by many computer vision and pattern recognition researchers as a challenging test data for image description, understanding, representation, and recognition [8,9,12]. Here we show how a pyramid can be stmcturally represented by a 3-D UAG. Without loss of generality, a 5 * 5 * 3 pyramid is shown in Fig. 11 and its parsing sequence (representation) is shown as follows:
1.6 3 - 0 Object Pattern Representation
S
S
S S S S S S A S S S S S ((1 2 5 I 3 , l 5 y 5 3 , =~0))
sssss sssss sssss -5,
=
193
ci
(2,2,1) = c 2
& 5 * 5 * 3 pyramid or the parsing sequence is
top v i e w s :
side v i e w s :
m
l
Fig. 11. A 5
I 1 I I
* 5 * 3 solid pyramid.
Example 4.5. Figure 1 2 shows a stair, which can also be considered as a n approximation of Fig. 12(b). Its string representation is 1 n 2 m 5 k(z, y
2 k, 2)sk(z, y 2 2k, 2)s'" (z, y 2 3k, z)5k (z, y 2 4k, 2)s'".
Example 4.6. Compare the two stairs in Fig. 13, which cannot be properly distinguished by the method in IS], but can be clearly distinguished by 3-D UAG from their respective representations as follows: (a): 1 ~ 2 (z, ~ y 52 k, ~ 2)s'" (z, y
2 2k, 2 ) ~ ~ ' "
(b): I n 2m 52k ( 5 , y 2 k , z)52k (z, y 2 2k, z)s2"
194 P.
s. P.
Wang
P
.
. (a) a stair array
--.
(b) approximation array
Fig. 12. A stair and its approximation arrays.
stair (a)
stair (b)
Fig. 13. Two similar but different objects (stairs) that cannot be distinguished by Marill’s method in [S].
5. 26-Neighborhood UAG and from Pixels to Object Features
So far, the 3-D UAG uses the six-neighborhood, which can handle changes of n* 90” only. In this section, this restriction is lifted by expanding the six-neighborhood to a 26-neighborhood. In this case, a 3-D UAG using 26-neighborhood is defined in Example 5.1, using two-point normal form [3]. Example 5.1.
G, = (V,, V,, P, S , #), where V, = { S ) , V, = {*) as defined in Section 2, and rewriting rules in P are in the following form: P : (u)( S , SS, w) where in (1, 2, . .. , 6, a , b . .. , n, p , q . . . , u}, and each of these 26 symbols is a neighbor as defined as a vector in Fig. 14. (Note that we exclude the letter “o” (oh) to avoid confusion with the original location (O,O,O)). or ( S , *, -) for a terminal rule.
1.6 3-0 Object Pattern Representation
5
195
m
( -1,-1.1
-X
= 9
...
I'
Cell
-
-Y
( -1, -1.
4
-1) =
-2
6 I
(l,-l,-L)=u
-2
-Y
-2
Fig. 14. 3-D space with its center cell and 2-D-neighborhood, each denoted by an alphanumeral ranging from 1 to 6, and a to u (excluding '(oh").
Notice that there are 27 rules including the terminal rule. Again, since every array sentence must terminate by the terminal rule, in the parsing sequence, the last digit indicating the terminal rule can be omitted, without loss of generality.
Example 5.2. A standing up coat rack is shown in Fig. 15. Its stfring representation is 2m A( 1" 2"3"
5%+4+')B(epfp)C(ipjp)
,
+
where A = (0, m, 0 ) , B = (0, m, n) and C = (0, m, n 4). A more complicated example combining both wire-like and solid volume objects is illustrated in the following example.
Example 5.3. Two types of overhead projectors are shown in Fig. 16, with their back, side and bird's-eye views. Their string pattern representations are: (a): I m 2 n 5 P A ' + q B m ~ s C ( 1 u 2 v 5 ) (yxL, i, 2 ) 5 , i = 1, . . . , w - 1 (b): 1"2" SPAT+'B i ~ s l S C ( 1 u 2 v 5 ) (ys ,2 i, 2)5, i = 1 , . . . , w - 1 where A , B and C are computed and shown in Fig. 16.
196 P. S. P. Wang C
Fig. 15. A standing up coat rack and its digitized 3-D array object.
Again this example demonstrates two structurally similar but distinguishable objects, which are reflected from their respective string pattern representations. In fact, its representation also shows a segmentation of the object into four major portions (feature extraction) shown by four different shadings in Fig. 16, where the key difference between the two objects is indicated by the darkest black region (neck of the overhead projector). 6. Approximating Distorted, Noisy and Curved Objects
Distorted and noisy objects, can be approximated by straight line segments according to probabilistic distribution and thresholding, very much similar to those methods for 2-D line drawings [5,22]. Objects whose line drawing have arbitrary angles 8, rather than a multiple of 45O, can be approximated by x and y = tan 8 line segments along a particular plane. Example 6.1. In Fig. 17, (i) is a noisy line segment along the x-axis in the x-y plane, whose representation is lmaldllndlallp. If m ,n,p >> 1, then this representation can be approximated as l m + n f p . Figure 17 (ii) with an arbitrary angle 8 can be approximated by a sequence of line segments in the x-y plane by l"2" tan '. Curved objects can be approximated by line segments along the quantized planes tangent to the quantized 45O's,as shown in Example 6.2. Example 6.2. Figure 18 shows a type of glass and the different angles of its views. Its string pattern presentation is
lt2t3t4tatbtct dt5" B(e" f vivjvmvn"pvqv)C5w ,
1.6 3-0 Object Pattern Representation 197
y4 ...............
Type1
(
A
!!G
.i5O ................
Fig. 16. Two types of overhead projectors I and 11, and their different views.
where C is a circle described by x2 k d n , u q o ) , where x 5 T .
+
+ y2 = r2 at
z
=
u + Tv, Jz i.e. C
= (&x,
Example 6.3. More examples of curved objects are illustrated in Fig. 19. Their respective pattern representations are as follows:
lt2t3t4tat btct dt5uB(15 2 5 3%4; a f bf c~ d; qqr)D5f,
)qe $rf$ri 4 .3.4 r m $ r n + r
P
$r
P. S. P. Wang
198
Y
I
I:
(a)
Fig. 17. Noisy line segments, with arbitrary angle 0 and their digitized arrays.
Fig. 18. A glass and its 3-D arrays.
1.6 3-D Object Pattern Representation 199
21
Fig. 19. More illustrations of curved objects and their 3-D arrays.
~zF, u),
where C and D are two circles indicated in the figure, i.e. C = (hz, where z 5 ;, and D = ( & z , & d m ,u f), where z 5 T , C
+
hd-,
= (&z,
q u ) ,where x 5 r ; (b) l t 2 t 3 t 4 t a t b t c t d t 5 " B ( l x 5 y 2 ~ 5 y 3 ~ 5 y 4 ~ 5 y ~ ~ 5 y ~ x 5, y c ~ 5 y ~ ~ 5 y ) "
y.
where y = x t a n 0 and w = Notice that the three different types of glasses shown in Figs. 18 and 19 have structural similarities and differences, which are reflected from their respective string pattern representations via 3-D UAG shown in Examples 6.2 and 6.3.
7. Discussions and Future Research We have introduced a formal model known as "3-D universal array grammar" (3-D UAG) for three-dimensional object representation. It is parallel, and simple
200
P. S. P. Wang
to manipulate by computers, including orientations (along the IC-, y- and z-axis), shift, enlargement, elongations and reductions. This model is basically extended from 2-D universal array grammar (2-D UAG) [23]. But the difference here is not just the dimensionality. The types of production rules are different and it is parallel. The 2-D UAG in [23] uses regular (type 3) rules while here the 3-D UAG G, uses “context-free’’ (type 2) rules. Please note that here by “context-free” we mean t o borrow the terminology from Chomsky [3,25]. Because of its dimensions and the use of blanks (#) in the context, it is still more or less sensitive to the # symbols. Nevertheless it is interesting to see that 3-D UAG G, is more powerful (in terms of generative capability) than 2-D UAG, in that the following “multi-branch wire-like” patterns shown in Fig. 20 (i) (symbolizing a digitized Chinese character meaning “center” or “ central”) can be generated by G, but not by any 2-D UAG even in two-dimensional space [3].
X X
X
X
xxxxxxxxxx x x X x x X xxxxxxxxxx X X
X
X
x
x
X
X
X
X
X
x
x
X (i 1
(ii)
Fig. 20. (i) Multi-branch wire-like pattern symbolizing a digitized Chinese character “center”, and (ii) diagonal pattern symbolizing a Chinese character “human” or “man”.
Further, neither G, nor 2-D UAG can generate any “diagonal” patterns shown in Fig. 20 (ii) (symbolizing a Chinese digitized character meaning “human” or “man”). This is because the limitation of “6-neighborhood”. But with “26-neighborhood” introduced in Section 5 of this article, it can. Therefore there is a certain hierarchy in the patterns depending not only on their production rules but also on neighborhood definitions. It would be interesting to investigate such a three-dimensional pattern hierarchy, whose 2-D array counterpart has been investigated in [21]. It is also interesting to explore multi-dimensional arrays in other spaces such as hexagonal space using 60 and 120 degrees (rather than 90 degrees) as illustrated in 2-D space 111. The idea introduced in this paper cannot only generate many interesting 3-D objects, but can also be used for 3-D object learning, understanding, and description. For example, according to the sequence of rules (universal array grammar), Fig. 8 can be described and understood as a 3-D object with 12 sides, forming six perpendicular rectangle surface areas, i.e. a brick. When all sides are of equal
1.6 3-D Object Pattern Representation 201
length, it is a cube, i.e. a cube is a brick (with all six surfaces perpendicular t o each other) with all sides of equal length. Indeed, when one learns, understands, describes, memorizes, and recognizes a cube, these are the key characteristics all reflected by our representation : 1n-l
2n - 1 5 n - l .
From the theory in The Society of Mind [9], the idea of which was reiterated in [26], this can be considered as a small agent that can recognize all sizes of cubes. Another small agent is able to recognize bricks. These two small agents are very much alike in nature, and probably reside in one’s brain (memory) very close to each other. Translating to pattern recognition terms, their string pattern representations occupy nearby or neighboring addresses in the dictionary. There are also many other small agents, each recognizing an infinite subclass of objects sharing some common properties characterized by its representation, and so on. Altogether, we have a society that can recognize (describe, understand, memorize, and interpret) any object that has been taught through training via a 3-D UAG. For future research, more can be done, including: (1) 3-D objects clustering, need alignment, dictionary construction, matching, (2) from pixels to 3-D object feature extraction, segmentation, scene analysis, understanding, description, representation, and recognition, and (3) thinning (skeletonization) 3-D digitized arrays. For example, there are some more interesting applications to the real world, e.g. satellite launching environment such as the one shown in Fig. 21 that can be described by 3-D UAG.
Fig. 21. Some illustrations of 3-D objects from a shuttle launching station.
It is the author’s hope that this ground work can also pave the road for further studies of the 3-D formal model for object pattern recognition and to stimulate research in 3-D object clustering analysis involving noisy and distorted patterns.
202
P. S. P. Wang
Acknowledgement
Part of this work was done when the author was visiting the LIPN Labs of University of Paris VII and XIII. The author is grateful to Profs. M. Nivat and A. Saoudi for providing an excellent environment for research and for the financial support. References [l]K. Aizawa and A. Nakamura, Grammars on the hexagonal array, in P. S. P. Wang (ed.), Array Grammars, Patterns and Recognizers (World Scientific, 1989) 191-200. (21 L. Baird and P. S. P. Wang, 3-D object recognition using gradient descent and the universal 3-D array grammar, SPIE Vol. 1607 Intelligent Robots and Computer Vision, 1992, 711-719. [3] C. Cook and P. S. P. Wang, A chomsky hierarchy of isotonic array grammars and languages, Comput. Graph. Image Process. 8 (1978) 144-152. [4] S. Edelman, H. Bulthoff and D. Weinshall, Stimulus Familiarity Determines Recognition Strategy for Novel 3-D, MIT A1 Lab. Memo 1138, Jul. 1989. [5] K. S. Fu, Syntactic Pattern Recognition and Applications (Prentice-Hall, Englewood Cliffs, NJ, 1982). [6] W. I. Grosky and P. S. P. Wang, The relation between uniformly structured tessellation automata and parallel array grammars, in Proc. I E E E ISUSAL 75, Tokyo, Japan (1975) 97-102. [7] K. Inoue, I. Sakuramoto, M. Sakamoto and I. Itsanami, 2-D automata operating in parallel, in Proc. Int. Colloquium on Parallel Image Processing, Paris, 1991, 239-262. [8] T. Marill, Emulating the human interpretation of line-drawings as 3-D objects, Int. J . Comput. Vision 6, 2 (1991) 147-161. A preliminary version of this paper also appeared as a technical report: Recognizing Three-Dimensional Objects Without the Use of Models, MIT A1 Lab. Memo 1157, Sept. 1989. [9] M. L. Minsky, The Society of Mind (Heinemann, London, 1986). [lo] M. L. Minsky, Steps toward artificial intelligence, in Proc. I R E 49, 1961, 8-30. [ll] M. Nivat, A. Saoudi and V. R. Dare, Parallel generation of finite images, in P. S. P. Wang (ed.), Array Grammars Patterns and Recognizers (World Scientific, 1989) 1-16. [ 121 A. Rosenfeld, Picture Languages: Formal Models for Picture Recognition (Academic Press, New York, 1979). [13] A. Rosenfeld, Preface, in P. S. P. Wang (ed.), Array Grammars, Patterns and Recognizers (World Scientific, 1989). [ 141 A. Rosenfeld, Coordinate grammars revisited: generalized isometric grammars, in P. S. P. Wang (ed.), Array Grammars, Patterns and Recognizers (World Scientific, 1989) 157-166. [15] A. Saoudi, M. Nivat and P. S. P. Wang (eds.), Parallel Image Processing (World Scientific, 1992). [16] R. N. Shepard and J. Metzler, Mental rotation of 3-D objects, Science 171 (1971) 701-703. [17] R. N. Shepard and J. Metzler, Mental rotation: Effects of dimensionality of objects and type of task, J. Exp. Psychol.: Human Perception and Performance 14 (1988) 3-11. [18] R. Siromoney, Array language and Lindenmayer systems - A survey, in G. Rozenberg and A. Salomaa (eds.), The Book of L (Springer Verlag, 1986). [19] S. Ullman, An Approach to Object Recognition: Aligning Pictorial Descriptions, MIT A1 Lab. Memo 931, Dec. 1986.
1.6 3-D Object Pattern Representation 203 [20] P. S. P. Wang, Finite-turn repetitive checking automata and sequential/parallel matrix languages, IEEE Trans. Comput. 30 (1981) 366-370. [all P. S. P. Wang, Hierarchical structures and complexities of isometric patterns, IEEE Trans. Pattern Anal. Mach. Intell. 5 , 1 (1983) 92-99. [22] P. S. P. Wang, An application of array grammars to clustering analysis for syntactic patterns, Pattern Recogn. 17,4 (1984) 441-451. [23] P. S. P. Wang, On-line Chinese character recognition by array grammars, in Proc. 6th IGC Int. Conference on Electronic Image '88 (1988) 209-214. 124) P. S. P. Wang (ed.), Array Grammars, Patterns and Recognizers (World Scientific, 1989). [25] Y. Yamamoto, K. Morita and K. Sugata, Context-sensitivity of 2-D regular array grammars, in P. S. P. Wang (ed.), Array Grammars, Patterns and Recognizers (World Scientific, 1989) 17-41. [26] P. Winston with S. Shellard (eds.), Artificial Intelligence at M I T - Expanding Frontiers (MIT Press, 1990).
PART 2 BASIC METHODS IN COMPUTER VISION
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 207-248 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
1 CHAPTER 2.1 1 TEXTURE ANALYSIS
MIHRAN TUCERYAN Department of Computer and Information Science, Indiana University - Purdue University at Indianapolis, 723 W. Michigan St. Indianapolis, I N 46202-5132 and ANIL K. JAIN Computer Science Department, Michigan State University East Lansing, M I 48824-1027, USA Internet:
[email protected] This chapter reviews and discusses various aspects of texture analysis. The concentration is on the various methods of extracting textural features from images. The geometric, random field, fractal, and signal processing models of texture are presented. The major classes of texture processing problems such as segmentation, classification, and shape from texture are discussed. The possible application are= of texture such as automated inspection, document processing, and remote sensing are summarized. A bibliography is provided at the end for further reading. Keywords: Texture, segmentation, classification, shape, signal processing, fractals, random fields, Gabor filters, wavelet transform, gray level dependency matrix.
1. Introduction In many machine vision and image processing algorithms, simplifying assumptions are made about the uniformity of intensities in local image regions. However, images of real objects often do not exhibit regions of uniform intensities. For example, the image of a wooden surface is not uniform but contains variations of intensities which form certain repeated patterns called visual texture. The patterns can be the result of physical surface properties such as roughness or oriented strands which often have a tactile quality, or they could be the result of reflectance differences such as the color on a surface. We recognize texture when we see it but it is very difficult to define. This difficulty is demonstrated by the number of different texture definitions attempted by vision researchers. Coggins [l]has compiled a catalogue of texture definitions in the computer vision literature and we give some examples here. <Wemay regard texture as what constitutes a macroscopic region. Its structure is simply attributed to the repetitive patterns in which elements or primitives are arranged according to a placement rule.” [2]
208 0
0
0
0
0
M . Tuceryan & A . K. Jain
“A region in an image has a constant texture if a set of local statistics or other local properties of the picture function are constant, slowly varying, or approximately periodic.” [3] “The image texture we consider is nonfigurative and cellular.. . An image texture is described by the number and types of its (tonal) primitives and the spatial organization or layout of its (tonal) primitives.. . A fundamental characteristic of texture: it cannot be analyzed without a frame of reference of tonal primitive being stated or implied. For any smooth gray-tone surface, there exists a scale such that when the surface is examined, it has no texture. Then as resolution increases, it takes on a fine texture and then a coarse texture.” [4] “Texture is defined for our purposes as an attribute of a field having no components that appear enumerable. The phase relations between the components are thus not apparent. Nor should the field contain an obvious gradient. The intent of this definition is to direct attention of the observer to the global properties of the display -i.e. its overall “coarseness,” “bumpiness,” or “fineness.” Physically, nonenumerable (aperiodic) patterns are generated by stochastic as opposed to deterministic processes. Perceptually, however, the set of all patterns without obvious enumerable components will include many deterministic (and even periodic) textures.” [5] “Texture is an apparently paradoxical notion. On the one hand, it is commonly used in the early processing of visual information, especially for practical classification purposes. On the other hand, no one has succeeded in producing a commonly accepted definition of texture. The resolution of this paradox, we feel, will depend on a richer, more developed model for early visual information processing, a central aspect of which will be representational systems at many different levels of abstraction. These levels will most probably include actual intensities at the bottom and will progress through edge and orientation descriptors to surface, and perhaps volumetric descriptors. Given these multilevel structures, it seems clear that they should be included in the definition of, and in the computation of, texture descriptors.” [B] “The notion of texture appears to depend upon three ingredients: (i) some local ‘order’ is repeated over a region which is large in comparison to the order’s size, (ii) the order consists in the nonrandom arrangement of elementary parts, and (iii) the parts are roughly uniform entities having approximately the same dimensions everywhere within the textured region.” [7]
This collection of definitions demonstrates that the “definition” of texture is formulated by different people depending upon the particular application and that there is no generally agreed upon definition. Some are perceptually motivated, and others are driven completely by the application in which the definition will be used. Image texture, defined as a function of the spatial variation in pixel intensities (gray values), is useful in a variety of applications and has been a subject of intense study by many researchers. One immediate application of image texture is the recognition of image regions using texture properties. For example, in Fig. l(a), we
2.1 Texture Analysis 209
Fig. 1. (a) An image consisting of five different textured regions: cotton canvas (D77), straw matting (D55), raffia (D84), herringbone weave (D17), and pressed calf leather (D24) [8]. (b) The goal of texture classification is to label each textured region with the proper category label: the identities of the five texture regions present in (a). ( c ) The goal of texture segmentation is to separate the regions in the image which have different textures and identify the boundaries between them. The texture categories themselves need not be recognized. In this example, the five texture categories in (a) are identified as separate textures by the use of generic category labels (represented by the different fill patterns).
can identify the five different textures and their identities as cotton canvas, straw matting, raffia, herringbone weave, and pressed calf leather. Texture is the most important visual cue in identifying these types of homogeneous regions. This is called texture classification. The goal of texture classification then is to produce a classification map of the input image where each uniform textured region is identified with the texture class it belongs to as shown in Fig. l ( b ) . We could also find the texture boundaries even if we could not classify these textured surfaces. This is then the second type of problem that texture analysis research attempts to solve texture segmentation. The goal of texture segmentation is to obtain the boundary map shown in Fig. l(c). Texture synthesis is often used for image compression applications. It is also important in computer graphics where the goal is to render object surfaces which are as realistic looking as possible. Figure 2 shows a set of synthetically generated texture images using Markov random field and fractal models (91. The shape from texture problem is one instance of a general class of vision problems known as “shape from X”. This was first formally pointed out in the perception literature by Gibson [lo]. The goal is to extract three-dimensional shape information from various cues such as shading, stereo, and texture. The texture features (texture elements) are distorted due t o the imaging process and the perspective projection which provide information about surface orientation and shape. An example of shape from texture is given in Fig. 3. 2. Motivation Texture analysis is an important and useful area of study in machine vision. Most natural surfaces exhibit texture and a successful vision system must be able t o
210
M . Tuceryan €d A . K. J a i n
Fig. 2. A set of example textures generated synthetically using only a small number of parameters. (a) Textures generated by discrete Markov random field models. (b) Four textures (in each of the four quadrants) generated by Gaussian Markov random field models. (c) Texture generated by fractal model.
Fig. 3. We can extract the orientation of the surface from the variations of texture (defined by the bricks) in this image.
deal with the textured world surrounding it. This section will review the importance of texture perception from two viewpoints- from the viewpoint of human vision or psychophysics and from the viewpoint of practical machine vision applications.
2.1 Texture Analysis 211 2.1. Ps ychophysics The detection of a tiger among the foliage is a perceptual task that carries life and death consequences for someone trying to survive in the forest. The success of the tiger in camouflaging itself is a failure of the visual system observing it. The failure is in not being able to separate figure from ground. Figureground separation is an issue which is of intense interest to psychophysicists. The figureground separation can be based on various cues such as brightness, form, color, texture, etc. In the example of the tiger in the forest, texture plays a major role. The camouflage is successful because the visual system of the observer is unable to discriminate (or segment) the two textures of the foliage and the tiger skin. What are the visual processes that allow one to separate figure from ground using the texture cue? This question is the basic motivation among psychologists for studying texture perception. Another reason why it is important to study the psychophysics of texture perception is that the performance of various texture algorithms is evaluated against the performance of the human visual system doing the same task. For example, consider the texture pair in Fig. 4(a), first described by Julesz [ll].The image consists of two regions each of which is made up of different texture tokens. Close scrutiny of the texture image will indicate this fact to the human observer. The immediate perception of the image, however, does not result in the perception of two different textured regions; instead only one uniformly textured region is perceived. Julesz says that such a texture pair is not “effortlessly discriminable” or “preattentively discriminable.” Such synthetic textures help us form hypotheses about what image properties are important in human texture perception. In addition, this example raises the question of how to evaluate the performance of computer algorithms that analyze textured images. For example, suppose we have an algorithm that can discriminate the texture pair in Fig. 4(a). Is this algorithm “correct?” The answer, of course, depends on the goal of the algorithm. If it is a very special purpose algorithm that should detect such scrutably different regions, then it is performing correctly. On the other hand, if it is to be a computational model of how the human visual system processes texture, then it is performing incorrectly. Julesz has studied texture perception extensively in the context of texture discrimination [11,12,13]. The question he posed was “When is a texture pair discriminable, given that they had the same brightness, contrast, and color?” Julesz concentrated on the spatial statistics of the image gray levels that are inherent in the definition of texture by keeping other illumination-related properties the same. To discuss Julesz’s pioneering work, we need t o define the concepts of first- and second-order spatial statistics. (i) First-order statistics measure the likelihood of observing a gray value at a randomly-chosen location in the image. First-order statistics can be computed from the histogram of pixel intensities in the image. These depend only on individual pixel values and not on the interaction or co-occurrence of
212
M . Tuceryan & A . K. Jain
neighboring pixel values. The average intensity in a n image is an example of the first-order statistic. (ii) Second-order statistics are defined as the likelihood of observing a pair of gray values occurring at the endpoints of a dipole (or needle) of random length placed in the image at a random location and orientation. These are properties of pairs of pixel values.
(4
(b)
Fig. 4. Texture pairs with identical second-order statistics. The bottom halves of the images consist of texture tokens that are different from the ones in the top half. (a) Humans cannot perceive the two regions without careful scrutiny. (b) The two different regions are immediately discriminable by humans.
Julesz conjectured that two textures are not preattentively discriminable if their second-order statistics are identical. This is demonstrated by the example in Fig. 4(a). This image consists of a pair of textured regions whose second-order statistics are identical. The two textured regions are not preattentively discriminable. His later counter-examples t o this conjecture were the result of a careful construction of texture pairs that have identical second-order statistics (see Fig. 4(b)) [14,15,16]. Julesz proposed the “theory of textons” to explain the preattentive discrimination of texture pairs. Textons are visual event9 (such as collinearity, terminations, closure, etc.) whose presence is detected and used in texture discrimination. Terminations are endpoints of line segments or corners. Using his theory of textons, Julesz explained the examples in Fig. 4 as follows. Recall that both texture images in Fig. 4 have two regions that have identical second-order statistics. In Fig. 4(a), the number of terminations in both the upper and lower regions is the same (i.e. the texton information in the two regions is not different), therefore the visual system is unable to preattentively discriminate the two textures. In Fig. 4(b), on the other hand, the number of terminations in the upper half is three, whereas the number of terminations in the lower half is four. The difference in this texton makes the two textured regions discriminable. Caelli has also proposed the existence of perceptual analyzers by the visual system for detecting textons [17]. Beck et al. [18] have conducted experiments and argue that the perception of texture segmentation in certain types of patterns is primarily a function of spatial frequency analysis and not the result of higher level symbolic grouping processes.
2.1 Texture Analysis 213 Studies in psychophysiology have suggested that a multi-channel, frequency and orientation analysis of the visual image formed on the retina is performed by the brain. Campbell and Robson [19] performed psychophysical experiments using various grating patterns. They suggested that the visual system decomposes the image into filtered images of various frequencies and orientations. De Valois et al. [20] have studied the brain of the macaque monkey which is assumed t o be close t o the human brain in its visual processing. They recorded the response of the simple cells in the visual cortex of the monkey to sinusoidal gratings of various frequencies and orientations and concluded that these cells are tuned to narrow ranges of frequency and orientation. These studies have motivated vision researchers t o apply multi-channel filtering approaches to texture analysis. 2.2. Applications
Texture analysis methods have been utilized in a variety of application domains. In some of the mature domains (such as remote sensing) texture already has played a major role, while in other disciplines (such as surface inspection) new applications of texture are being found. We will briefly review the role of texture in automated inspection, medical image processing, document processing, and remote sensing. Images from two application domains are shown in Fig. 5. The role that texture plays in these examples varies depending upon the application. For example, in the SAR images of Figs. 5(b) and (c) texture is defined to be the local scene heterogeneity and this property is used for classification of land use categories such as water, agricultural areas, etc. In the ultrasound image of the heart in Fig. 5(a), texture is defined as the amount of randomness which has a lower value in the vicinity of the border between the heart cavity and the inner wall than in the blood filled cavity. This fact can be used t o perform segmentation and boundary detection using texture analysis methods. 2.2.1. Inspection There has been a limited number of applications of texture processing t o automated inspection problems. These applications include defect detection in images of textiles and automated inspection of carpet wear and automobile paints. In the detection of defects in texture images, most applications have been in the domain of textile inspection. Dewaele et al. [21] used signal processing methods to detect point defects and line defects in texture images. They have sparse convolution masks in which the bank of filters are adaptively selected depending upon the image to be analyzed. Texture features are computed from the filtered images. A Mahalanobis distance classifier is used to classify the defective areas. Chetverikov [22] defined a simple window differencing operator to the texture features obtained from simple filtering operations. This allows one to detect the boundaries of defects in the texture. Chen and Jain [23] used a structural approach to defect detection in textured images. They extract a skeletal structure from images, and by detecting
214
M. Tuceryan & A . K. Jain
Fig. 5. Examples of images from various application domains in which texture analysis is important. (a) The ultrasound image of a heart. (b), (c) are example aerial images using SAR sensors.
anomalies in certain statistical features in these skeletons, defects in the texture are identified. Conners et al. [24] utilized texture analysis methods to detect defects in lumber wood automatically. The defect detection is performed by dividing the image into subwindows and classifying each subwindow into one of the defect categories such as knot, decay, mineral streak, etc. The features they use to perform this classification are based on tonal features such as mean, variance, skewness, and kurtosis of gray levels along with texture features computed from gray level cooccurrence matrices in analyzing pictures of wood. The combination of using tonal features along with textural features improves the correct classification rates over using either type of feature alone. In the area of quality control of textured images, Siew et al. [25] proposed a method for the assessment of carpet wear. They used simple texture features that are computed from second-order gray level dependency statistics and from first-order gray level difference statistics. They showed that the numerical texture features obtained from these techniques can characterize the carpet wear successfully. Jain et al. [26] used the texture features computed from a bank of Gabor filters to automatically classify the quality of painted metallic surfaces. A pair of automotive paint finish images is shown in Fig. 6 where the image in (a) has a uniform coating of paint, but the image in (b) has a “mottled” or “blotchy” appearance.
2.1 Texture Analysis 215
Fig. 6. Example images used in paint inspection. (a) A non-defective paint which has a smooth texture. (b) A defective paint which has a mottled look.
2.2.2. Medical Image Analysis Image analysis techniques have played an important role in several medical applications. In general, the applications involve the automatic extraction of features from the image which are then used for a variety of classification tasks, such as distinguishing normal tissue from abnormal tissue. Depending upon the particular classification task, the extracted features capture morphological properties, color properties, or certain textural properties of the image. The textural properties computed are closely related to the application domain to be used. For example, Sutton and Hall [27] discuss the classification of pulmonary disease using texture features. Some diseases, such as interstitial fibrosis, affect the lungs in such a manner that the resulting changes in the X-ray images are texture changes as opposed to clearly delineated lesions. In such applications, texture analysis methods are ideally suited for these images. Sutton and Hall propose the use of three types of texture features t o distinguish normal lungs from diseased lungs. These features are computed based on a n isotropic contrast measure, a directional contrast measure, and a Fourier domain energy sampling. In their classification experiments, the best classification results were obtained using the directional contrast measure. Harms et al. [28] used image texture in combination with color features to diagnose leukemic malignancy in samples of stained blood cells. They extracted texture micro-edges and “textons” between these micro-edges. The textons were regions with almost uniform color. They extracted a number of texture features from the textons including the total number of pixels in the textons which have a specific color, the mean texton radius and texton size for each color and various texton shape features. In combination with color, the texture features significantly improved the correct classification rate of blood cell types compared to using only color features. Landeweerd and Gelsema [29] extracted various first-order statistics (such as mean gray level in a region) as well as second-order statistics (such as gray level
218
M. Tuceryan €9 A . K. Jain
bar code block. A similar method is used for locating text blocks in newspapers. A segmentation of the document image is obtained using three classes of textures: one class for the text regions, a second class for the uniform regions that form the background or images where intensities vary slowly, and a third class for the transition areas between the two types of regions (see Fig. 9). The text regions are characterized by their high frequency content.
Fig. 9. Text/graphics separation using texture information. (a) An image of a newspaper captured by a flatbed scanner. (b) The three-class segmentation obtained by the Gabor filter based texture segmentation algorithm. (c) The regions identified as text.
2.2.4. Remote Sensing Texture analysis has been extensively used t o classify remotely sensed images. Land use classification where homogeneous regions with different types of terrains (such as wheat, bodies of water, urban regions, etc.) need t o be identified is an important application. Haralick et al. [41] used gray level co-occurrence features t o analyze remotely sensed images. They computed gray level co-occurrence matrices for a distance of one with four directions (O", 45", go", and 135"). For a seven-class classification problem, they obtained approximately 80% classification accuracy using texture features. Rignot and Kwok [42] have analyzed SAR images using texture features computed from gray level co-occurrence matrices. However, they supplement these features with knowledge about the properties of SAR images. For example, image restoration algorithms were used to eliminate the specular noise present in SAR images in order to improve classification results. The use of various texture features was studied for analyzing SAR images by Schistad and Jain [43]. SAR images shown in Figs. 5(b) and (c) were used to identify land use categories of water, agricultural areas, urban areas, and other areas. Fractal dimension, autoregressive Markov random field model, and gray level co-occurrence texture features were used in the classification. The classification errors ranged from 25% for the fractal based models to as low as 6% for the MRF features. Du [44] used texture features derived
2.1 Texture Analysis 219 from Gabor filters to segment SAR ima.ges. He successfully segmented the SAR images into categories of water, new forming ice, older ice, and multi-year ice. Lee and Philpot [45] also used spectral texture features to segment SAR images.
3. A Taxonomy of Texture Models Identifying the perceived qualities of texture in an image is an important first step towards building mathematical models for texture. The intensity variations in an image which characterize texture are generally due to some underlying physical variation in the scene (such as pebbles on a beach or waves in water). Modelling this physical variation is very difficult, so texture is usually characterized by the two-dimensional variations in the intensities present in the image. This explains the fact that no precise, general definition of texture exists in the computer vision literature. In spite of this, there are a number of intuitive properties of texture which are generally assumed t o be true. 0 Texture is a property of areas; the texture of a point is undefined. So, texture is a contextual property and its definition must involve gray values in a spatial neighborhood. The size of this neighborhood depends upon the texture type, or the size of the primitives defining the texture. 0 Texture involves the spatial distribution of gray levels. Thus, two-dimensional histograms or co-occurrence matrices are reasonable texture analysis tools. 0 Texture in an image can be perceived a t different scales or levels of resolution [lo]. For example, consider the texture represented in a brick wall. At a coarse resolution, the texture is perceived as formed by the individual bricks in the wall; the interior details in the brick are lost. At a higher resolution, when only a few bricks are in the field of view, the perceived texture shows the details in the brick. 0 A region is perceived to have texture when the number of primitive objects in the region is large. If only a few primitive objects are present, then a group of countable objects is perceived instead of a textured image. In other words, a texture is perceived when significant individual “forms” are not present. Image texture has a number of perceived qualities which play an important role in describing texture. Laws [47] identified the following properties as playing an important role in describing texture: uniformity, density, coarseness, roughness, regularity, linearity, directionality, direction, frequency, and phase. Some of these perceived qualities are not independent. For example, frequency is not independent of density and the property of direction only applies t o directional textures. The fact that the perception of texture has so many different dimensions is an important reason why there is no single method of texture representation which is adequate for a variety of textures. 3.1. Statistical Methods One of the defining qualities of texture is the spatial distribution of gray values. The use of statistical features is therefore one of the early methods proposed in the
220
M. Tuceryan lY A . K. Jain
machine vision literature. In the following, we will use { I ( $ ,y), 0 5 x 5 N - 1 , 0 5 y 5 N - 1) to denote an N x N image with G gray levels. A large number of texture features have been proposed. But, these features are not independent as pointed out by Tomita and Tsuji [46]. The relationship between the various statistical texture measures and the input image is summarized in Fig. 10 [46]. Picard [48] has also related the gray level co-occurrence matrices t o the Markov random field models. 3.1.1. Go-occurrence Matrices
Spatial gray level co-occurrence estimates image properties related to secondorder statistics. Haralick [lo] suggested the use of gray level co-occurrence matrices (GLCM) which have become one of the most well-known and widely used texture features. The G x G gray level co-occurrence matrix pd for a displacement vector d = (dz, dy) is defined as follows. The entry ( i ,j ) of Pd is the number of occurrences of the pair of gray levels i and j which are a distance d apart. Formally, it is given as p d ( i , j ) = l { ( ( r , s ) , ( t , v ):)I ( r , S )= i , I ( t , v )= j } l (3.1)
+
+
where ( T , s), (t,v) E N x N , (t,v ) = ( r dx, s dy), and 1 . 1 is the cardinality of a set. As an example, consider the following 4 x 4 image containing three different gray values: 1 1 0 0 1 1 0 0 0 0 2 2 0 0 2 2 The 3 x 3 gray level co-occurrence matrix for this image for a displacement vector of d = ( 1 , O ) is given as follows:
Pd=
[:: :I 2
2 0
.
Here the entry (0,O) of Pd is 4 because there are four pixel pairs of that are offset by (1, 0) amount. Examples of p d for other displacement vectors i s given below.
Notice that the co-occurrence matrix so defined is not symmetric. But a symmetric co-occurrence matrix can be computed by the formula P = Pd P-d. T h e cooccurrence matrix reveals certain properties about the spatial distribution of the
+
2.1 Texture Analysis 221 gray levels in the texture image. For example, if most of the entries in the cooccurrence matrix are concentrated along the diagonals, then the texture is coarse with respect to the displacement vector d. Haralick has proposed a number of useful texture features that can be computed from the co-occurrence matrix. Table 1 lists some of these features. Here p3: and py are the means and g3:and gy are the standard deviations of pd(2) and pd(y), respectively, where pdc3:) = pd(z,j) and pd(y) = pd(i, I/).
cj
Table 1. Some texture features extracted from gray level co-occurrence matrices. Texture Feature
Entropy
Formula
-
Homogeneity
The co-occurrence matrix features suffer from a number of difficulties. There is no well established method of selecting the displacement vector d and computing co-occurrence matrices for different values of d is not feasible. For a given d, a large number of features can be computed from the co-occurrence matrix. This means that some sort of feature selection method must be used to select the most relevant features. The co-occurrence matrix-based texture features have also been primarily used in texture classification tasks and not in segmentation tasks. 3.1.2. Autocorrelation Features
An important property of many textures is the repetitive nature of the placement of texture elements in the image. The autocorrelation function of an image can be used to assess the amount of regularity as well as the fineness/coarseness of the texture present in the image. Formally, the autocorrelation function of an image I ( z , y) is defined as follows:
222
M. Tuceryan tY A . K. J a i n
The image boundaries must be handled with special care but we omit the details here. This function is related to the size of the texture primitive (i.e. the fineness of the texture). If the texture is coarse, then the autocorrelation function will drop off slowly; otherwise, it will drop off very rapidly. For regular textures, the autocorrelation function will exhibit peaks and valleys. The autocorrelation function is also related t o the power spectrum of the Fourier transform (see Fig. 10). Consider the image function in the spatial domain I(x, y) and its Fourier transform F ( u ,v). The quantity IF(u,. ) I 2 is defined as the power spectrum where I . I is the modulus of a complex number. The example in Fig. 11 illustrates the effect of the directionality of a texture on the distribution of energy in the power spectrum. Early approaches using such spectral features would divide the frequency domain into rings (for frequency content) and wedges (for orientation content) as shown in Fig. 12. The frequency domain is thus divided into regions and the total energy in each of these regions is computed as texture features.
Original Image
Difference Statistics
w
7
Fourier Tmsform
Au toregression Model
Fig. 10. The interrelation between the various second-order statistics and the input image [46]. @ Reprinted by permission of Kluwer Academic Publishers.
Fig. 11. Texture features from the power spectrum. (a) A texture image, and (b) its power spectrum. The directional nature of this texture is reflected in the directional distribution of energy in the power spectrum.
2.1 Texture Analysis 223
r = JiFT7
0 = atan(v/u)
(4
r
=
J2G-3
0 = atan(v/u) (b)
Fig. 12. Texture features computed from the power spectrum of the image. (a) The energy computed in each shaded band is a texture feature indicating coarseness/fineness, and (b) the energy computed in each wedge is a texture feature indicating directionality.
3.2. Geometrical Methods
The class of texture analysis methods that falls under the heading of geometrical methods is characterized by their definition of texture as being composed of “texture elements” or primitives. The method of analysis usually depends upon the geometric properties of these texture elements. Once the texture elements are identified in the image, there are two major approaches to analyzing the texture. One computes statistical properties from the extracted texture elements and utilizes these as texture features. The other tries to extract the placement rule that describes the texture. The latter approach may involve geometric or syntactic methods of analyzing texture. 3.2.1. Voronoi Tessellation Features Tuceryan and Jain [49] proposed the extraction of texture tokens by using the properties of the Voronoi tessellation of the given image. Voronoi tessellation has been proposed because of its desirable properties in defining local spatial neighborhoods and because the local spatial distributions of tokens are reflected in the shapes of the Voronoi polygons. First, texture tokens are extracted and then the tessellation is constructed. Tokens can be as simple as points of high gradient in the image or complex structures such as line segments or closed boundaries.
224
M . Tuceryan & A . K. Jain
In computer vision, the Voronoi tessellation was first proposed by Ahuja as a model for defining “neighborhoods” [50]. Suppose that we are given a set S of three or more tokens (for simplicity, we will assume that a token is a point) in the Euclidean plane. Assume that these points are not all collinear, and that no four points are cocircular. Consider an arbitrary pair of points P and Q. The bisector of the line joining P and Q is the locus of points equidistant from both P and Q and divides the plane into two halves. The half plane H?(H,‘) is the locus of points closer to P ( Q ) than to Q ( P ) . For any given point P , a set of such HQ half planes is obtained for various choices of Q. The intersection defines a polygonal region consisting of points closer to P than any other point. Such a region is called the Voronoi polygon [51] associated with the point. The set of complete polygons is called the Voronoi diagram of S [52]. The Voronoi diagram together with the incomplete polygons in the convex hull define a Voronoi tessellation of the entire plane. Two points are said to be Voronoi neighbors if the Voronoi polygons enclosing them share a common edge. The dual representation of the Voronoi tessellation is the Delaunay graph which is obtained by connecting all the pairs of points which are Voronoi neighbors as defined above. An optimal algorithm to compute the Voronoi tessellation for a point pattern is described by Preparata and Shamos [ 5 3 ] . A simple 2-D dot pattern and its Voronoi tessellation are shown in Fig. 13.
nQEs,QZp
I ......
...... ......
.. .. .. .. .. ..... .. .. ... ...... .. .. ...... .. .. .. .. .. . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Fig. 13. Voronoi tessellation: (a) An example dot pattern, and (b) its Voronoi tessellation.
The neighborhood of a token P is defined by the Voronoi polygon containing P . Many of the perceptually significant characteristics of a token’s environment are manifest in the geometric properties of the Voronoi neighborhoods (see Fig. 13). The geometric properties of the Voronoi polygons are used as texture features. In order to apply geometrical methods to gray level images, we need to first extract tokens from images. We use the following simple algorithm to extract tokens from input gray level textural images. 1. Apply a Laplacian-of-Gaussian (LOG or V2G) filter t o the image. For computational efficiency, the V2G filter can be approximated with a difference
2.1 Texture Analysis 225
Fig. 14. Texture segmentation using the Voronoi tessellation. (a) An example texture pair from Brodatz’s album [92], (b) the peaks detected in the filtered image, and (c) the segmentation using the texture features obtained from Voronoi polygons [49]. The arrows indicate the border direction. The interior is on the right when looking in the direction of the arrow.
of Gaussians (DOG) filter. The size of the DOG filter is determined by the sizes of the two Gaussian filters. Tuceryan and Jain used ~1 = 1 for the first Gaussian and ~2 = 1 . 6 0 ~for the second. According to Marr, this is the ratio at which a DOG filter best approximates the corresponding V2G filter [54]. Select those pixels that lie on a local intensity maximum in the filtered image. A pixel in the filtered image is said to be on a local maximum if its magnitude is larger than six or more of its eight nearest neighbors. This results in a binary image. For example, applying steps 1 and 2 to the image in Fig. 14(a) yields the binary image in Fig. 14(b). Perform a connected component analysis on the binary image using eight nearest neighbors. Each connected component defines a texture primitive (token). The Voronoi tessellation of the resulting tokens is constructed. Features of each Voronoi cell are extracted and tokens with similar features are grouped to construct uniform texture regions. Moments of area of the Voronoi polygons serve as a useful set of features that reflect both the spatial distribution and shapes of the tokens in the textured image. The ( p q)th order moments of area of a closed region R with respect to a token with coordinates (zo, yo) are defined as [55]:
+
226
M. Tuceryan & A . K. Jain
Table 2. Voronoi polygon features used by the texture segmentation algorithm [49]. Here, f 2 gives the magnitude of the vector from the token to the polygon centroid, f3 gives its direction, f 4 gives the overall elongation of the polygon (f4 = 0 for a circle), and f5 gives the orientation of its major axis. ( 5 ,Q) are the coordinates of the Voronoi polygon’s centroid. Texture Feature
Computation
fl
moo
f2
VG-Tjp
f3
atan (g/%)
f5
atan
(
2mll m20 - m02
)
where p + q = 0 , 1 , 2 , . . .. A description of the five features used is given in Table 2 where (3,y) are the coordinates of the Voronoi polygon’s centroid. The texture features based on Voronoi polygons have been used for segmentation of textured images. The segmentation algorithm is edge based, using a statistical comparison of the neighboring collections of tokens. A large dissimilarity among the texture features is evidence for a texture edge. This algorithm has successfully segmented gray level texture images as well as a number of synthetic textures with identical second-order statistics. Figure 14(a) shows an example texture pair and Fig. 14(c) shows the resulting segmentation. 3.2.2. Structural Methods
The structural models of texture assume that textures are composed of texture primitives. The texture is produced by the placement of these primitives according to certain placement rules. This class of algorithms, in general, is limited in power unless one is dealing with very regular textures. Structural texture analysis consists of two major steps: (a) extraction of the texture elements, and (b) inference of the placement rule. There are a number of ways to extract texture elements in images. It is useful to define what is meant by texture elements in this context. Usually texture elements consist of regions in the image with uniform gray levels. Voorhees and Poggio [56] argued that blobs are important in texture perception. They have proposed a method based on filtering the image with Laplacian of Gaussian (LOG) masks at different scales and combining this information to extract the blobs in the image. Blostein and Ahuja [57]perform similar processing in order to extract texture tokens in images by examining the response of the LOG filter at multiple scales. They integrate their multi-scale blob detection with surface shape computation in order to improve the results of both processes. Tomita and Tsuji [46] also suggest a method
2.1 Texture Analysis 227 of computing texture tokens by doing a medial axis transform on the connected components of a segmented image. They then compute a number of properties such as intensity and shapes of these detected tokens. Zucker [58] has proposed a method in which he regards the observable textures (real textures) as distorted versions of ideal textures. The placement rule is defined for the ideal texture by a graph that is isomorphic to a regular or semiregular tessellation. These graphs are then transformed to generate the observable texture. Which of the regular tessellations is used as the placement rule is inferred from the observable texture. This is done by computing a two-dimensional histogram of the relative positions of the detected texture tokens. Another approach to modeling texture by structural means is described by Fu [59]. In this approach the texture image is regarded as texture primitives arranged according to a placement rule. The primitive can be as simple as a single pixel that can take a gray value, but it is usually a collection of pixels. The placement rule is defined by a tree grammar. A texture is then viewed as a string in the language defined by the grammar whose terminal symbols are the texture primitives. An advantage of this method is that it can be used for texture generation as well as texture analysis. The patterns generated by the tree grammars could also be regarded as ideal textures in Zucker’s model. 3.3. Model Based Methods
Model based texture analysis methods are based on the construction of a n image model that can be used not only t o describe texture, but also t o synthesize it. The model parameters capture the essential perceived qualities of texture. 3.3.1. Random Field Models
Markov random fields (MRFs) have been popular for modeling images. They are able to capture the local (spatial) contextual information in a n image. These models assume that the intensity at each pixel in the image depends on the intensities of only the neighboring pixels. MRF models have been applied to various image processing applications such as texture synthesis [60], texture classification [61,62], image segmentation [63,64], image restoration [65],and image compression. The image is usually represented by an N x N lattice denoted by L = {(i,j)Il 5 i 5 M , 1 5 j 5 N } . I ( i , j ) is a random variable which represents the gray level at pixel ( i , j ) on lattice L. The indexing of the lattice is simplified for mathematical convenience to I, with t = (i - l ) N j . Let A be the range set common to all random variables It and let Q = ( ( 2 1 , z2, . . . , z ~ ~ ) \E zA,M} t denote the set of all labellings of L. Note that A is specified according to the application. For instance, for an image with 256 different gray levels A may be the set {0,1,. . . ,255). The random vector I = (11, Iz, . . . , I M N )denotes a coloring of the lattice. A discrete Markov random field is a random field whose probability mass function has the properties of positivity, Markovianity, and homogeneity.
+
228
M. Tuceryan & A . K. Jain
The neighbor set of a site t can be defined in different ways. The first-order neighbors of t are its four-connected neighbors and the second-order neighbors are its eight-connected neighbors. Within these neighborhoods, sets of neighbors which form cliques (single site, pairs, triples, and quadruples) are usually used in the definition of the conditional probabilities. A discrete Gibbs random field (GRF) assigns a probability mass function to the entire lattice: 1 P ( X = x ) = -e--) vx E R (3.4) z where V(x) is an energy function and Z is a normalizing constant called the partition function. The energy function is usually specified in terms of cliques formed over neighboring pixels. For the second-order neighbors the possible cliques are given in Fig. 15. The energy function is then expressed in terms of potential functions VC(.) over the cliques Q:
U(x) = C V C ( X ) .
(3.5)
c€Q
0
m
E % 8
Fig. 15. The clique types for the second-order neighborhood.
We have the property that with respect t o a neighborhood system, there exists a unique Gibbs random field for every Markov random field and there exists a unique Markov random field for every Gibbs random field [66]. The consequence of this theorem is that one can model the texture either globally by specifying the total energy of the lattice or model it locally by specifying the local interactions of the neighboring pixels in terms of the conditional probabilities. There are a number of ways in which textures are modeled using Gibbs random fields. Among these are the Derin-Elliot model [67] and the auto-binomial model [66, 601 which are defined by considering only the single pixel and pairwise pixel cliques in the second-order neighbors of a site. In both models the conditional probabilities are given by expressions of the following form:
2.1 Texture Analysis 229 where Zt = x g E A e - w ( g , R t ) Tis e the normalization constant. The energy of the Gibbs random field is given by:
V(x)
=
c
1M N w(x,Rt)T8 2 t=l
-
(3.7)
where W ( x t , Rt) = [wI(xt) w 2 ( x t ) w3(xt) w4(xt)lT and 8 = [OI82 6% O4IT. The two models define the components of the w vector, w T ( x t ) ,differently as follows:
+
Derin-Elliott model: w,(xt) = I ( x t , zt-T) I ( z t , zt+r) 1 5 r 5 4. Auto-binomial model: w,(xt) = xt(ztVT xt+T) 1 5 r 5 4.
+
Here r is the index that defines the set of neighboring pixels of a site t , and I ( a , b) is an indicator function as follows:
I ( a ,b ) =
-1
ifa=b
1
otherwise.
The vector 8 is the set of parameters that defines and models the textural properties of the image. In texture synthesis problems, the values are set to control the type of texture to be generated. In the classification and segmentation problems, the parameters need to be estimated in order to process the texture images. Textures were synthesized using this method by Cross and Jain [60]. Model parameters were also estimated for a set of natural textures. The estimated parameters were used to generate synthetic textures and the results were compared to the original images. The models captured microtextures well, but they failed with regular and inhomogeneous textures. 3.3.2. Fractals Many natural surfaces have a statistical quality of roughness and self-similarity at different scales. Fractals are very useful and have become popular in modelling these properties in image processing. Mandelbrot [68] proposed fractal geometry and is the first one t o notice its existence in the natural world. We first define a deterministic fractal in order to introduce some of the fundamental concepts. Self-similarity across scales in fractal geometry is a crucial concept. A deterministic fractal is defined using this concept of self-similarity as follows. Given a bounded set A in a Euclidean n-space, the set A is said to be selfsimilar when A is the union of N distinct (non-overlapping) copies of itself, each of which has been scaled down by a ratio of r . The fractal dimension D is related to the number N and the ratio r as follows: (3.9) The fractal dimension gives a measure of the roughness of a surface. Intuitively, the larger the fractal dimension, the rougher the texture is. Pentland [69] has argued
230
M. Tuceryan & A . K. J a i n
and given evidence that images of most natural surfaces can be modelled as spatially isotropic fractals. Most natural surfaces and in particular textured surfaces are not deterministic as described above but have a statistical variation. This makes the computation of fractal dimension more difficult. There are a number of methods proposed for estimating the fractal dimension D. One method is the estimation of the box dimension as follows [70]. Given a bounded set A in Euclidean n-space, consider boxes of size L, on a side which cover the set A . A scaled down version of the set A by ratio r, will result in N = l / r D similar sets. This new set can be covered by boxes of size L = TL,,,. The number of such boxes is then related to the fractal dimension by (3.10)
The fractal dimension is then estimated from Eq. (3.10) by the following procedure. For a given L, divide the n-space into a grid of boxes of size L and count the number of boxes covering A. Repeat this procedure for different values of L. Then estimate the value of the fractal dimension D from the slope of the line ln(N(L)) = -D ln(L)
+ D ln(Lmax).
(3.11)
This can be accomplished by computing the least squares linear fit t o the data, namely, a plot of In(L) vs. - In(N(L)). An improved method of estimating the fractal dimension was proposed by Voss [71]. Assume we are estimating the fractal dimension of an image surface A . Let P ( m ,L) be the probability that there are m points within a box of side length L centered at an arbitrary point on the surface A . Let A4 be the total number of points in the image. When one overlays the image with boxes of side length L, then the ( M / m ) P ( m , L ) is the expected number of boxes with m points inside. The expected total number of boxes needed to cover the whole image is (3.12)
The expected value of N(L) is proportional to L-D and thus can be used t o estimate the fractal dimension D. Other methods have also been proposed for estimating the fractal dimension. For example, Super and Bovik [72] have proposed using Gabor filters and signal processing methods to estimate the fractal dimension in textured images. The fractal dimension is not sufficient to capture all textural properties. It has been shown [70] that there may be perceptually very different textures that have very similar fractal dimensions. Therefore, another measure, called Zacunarity [68,71,70], has been suggested in order to capture the textural property that will
2.1 Texture Analysis 231 let one distinguish between such textures. Lacunarity is defined as (3.13)
where M is the mass of the fractal set E ( M ) and is the expected value of the mass. This measures the discrepancy between the actual mass and the expected value of the mass. Lacunarity is small when texture is fine and it is large when the texture is coarse. The mass of the fractal set is related to the length L by the power law:
M(L)= K L ~ .
(3.14)
Voss [71] suggested computing lacunarity from the probability distribution P(m,L ) as follows. Let M ( L ) = C : = l m P ( m , L ) and M 2 ( L ) = CL=1m2P(m,L). Then lacunarity A is defined as: (3.15)
This measure of the image is then used as a texture feature in order to perform texture segmentation or classification. Ohanian and Dubes [73] have studied the performance of various texture features. They studied the texture features with the performance criteria: “which features optimized the classification rate?” They compared four fractal features, 16 co-occurrence features, four Markov random field features, and Gabor features. They used Whitney’s forward selection method for feature selection. The evaluation was done on four classes of images: Gauss Markov random field images, fractal images, leather images, and painted surfaces. The co-occurrence features generally outperformed other features (88% correct classification) followed by fractal features (84% classification). Using both fractal and co-occurrence features improved the classification rate to 91%. Their study did not compare the texture features in segmentation tasks. It also used the energy from the raw Gabor filtered images instead of using the empirical nonlinear transformation needed to obtain the texture features as suggested in [40] (see also Section 3.4.3).
3.4. Signal Processing Methods Psychophysical research has given evidence that the human brain does a frequency analysis of the image [19,74]. Texture is especially suited for this type of analysis because of its properties. This section will review the various techniques of texture analysis that rely on signal processing techniques. Most techniques try to compute certain features from filtered images which are then used in either classification or segmentation tasks. 3.4.1. Spatial Domain Filters
Spatial domain filters are the most direct way t o capture image texture properties. Earlier attempts at defining such methods concentrated on measuring the edge
232
M. Tuceryan & A . K. Jain
density per unit area. Fine textures tend to have a higher density of edges per unit area than coarser textures. The measurement of edgeness is usually computed by simple edge masks such as the Robert's operator or the Laplacian operator [10,47]. The two orthogonal masks for the Robert's operator and one digital realization of the Laplacian are given below. Robert's Operators
Laplacian Operator -1
-1
L = [-I -1
-1
8
It] -1
The edgeness measure can be computed over an image area by computing a magnitude from the responses of Robert's masks or from the response of the Laplacian mask. Malik and Perona [75] proposed spatial filtering to model the preattentive texture perception in the human visual system. Their proposed model consists of three stages: (i) convolution of the image with a bank of even-symmetric filters followed by half-wave rectification, (ii) inhibition of spurious responses in a localized area, and (iii) detection of the boundaries between the different textures. The even-symmetric filters they used consist of differences of offset Gaussian (DOOG) functions. The half-wave rectification and inhibition (implemented as leaders-takeall strategy) are methods of introducing a nonlinearity into the computation of texture features. A nonlinearity is needed in order t o discriminate texture pairs with identical mean brightness and identical second-order statistics. The texture boundary detection is done by a straightforward edge detection method applied to the feature images obtained from stage (ii). This method works on a variety of texture examples and is able to discriminate natural as well as synthetic textures with carefully controlled properties. Unser and Eden [76] have also looked at texture features that are obtained from spatial filters and a nonlinear operator. Reed and Wechsler [77] review a number of spatial/spatial frequency domain filter techniques for segmenting textured images. Another set of spatial filters are based on spatial moments [47]. The ( p + q)th moments over an image region R are given by the formula
If the region R is a local rectangular area and the moments are computed around each pixel in the image, then this is equivalent to filtering the image by a set of spatial masks. The resulting filtered images that correspond t o the moments are then used as texture features. The masks are obtained by defining a window of size W x W and a local coordinate system centered within the window. Let (2, j ) be the image coordinates at which the moments are computed. For pixel coordinates (m,n )
2.1 Texture Analysis 233 which fall within the W x W window centered at ( i , j ) ,the normalized coordinates ( x m ,yn) are given by:
(3.17) Then the moments within a window centered at pixel (i, j ) are computed by the sum in Eq. (3.16) that uses the normalized coordinates. (3.18) n=-W/2
m=-W/2
The coefficients for each pixel within the window to evaluate the sum is what defines the mask coefficients. If R is a 3 x 3 region, then the resulting masks are given below:
Moo=
1 1 1 1 1 1 1 1 1
-1
-1
1 0
-1
-1
-1
0
1
1 0 1
The moment-based features have been used successfully in texture segmentation [78]. An example texture pair and the segmentation are shown in Fig. 16.
Fig. 16. The segmentation results using moment based texture features. (a) A texture pair consisting of reptile skin and herringbone pattern from the Brodatz album [92]. (b) The resulting segmentation.
3.4.2. Fourier domain filtering
The frequency analysis of the textured image is best done in the Fourier domain. As the psychophysical results indicated, the human visual system analyzes the textured images by decomposing the image into its frequency and orientation components [19]. The multiple channels tuned to different frequencies are also referred
234
M. Tuceryun i 3 A . K. Juan
to as multi-resolution processing in the literature. The concept of multi-resolution processing is further refined and developed in the wavelet model described below. Along the lines of these psychophysical results, texture analysis systems have been developed that perform filtering in the Fourier domain to obtain feature images. The idea is similar to the features computed from the rings and wedges as described in Section 3.1.2, except that the phase information is kept. Coggins and Jain [79] used a set of frequency and orientation selective filters in multichannel filtering approach. Each filter is either frequency selective or orientation selective. There are four orientation filters centered at 0", 45", go", and 135". The number of frequency selective filters depends on the image size. For an image of size 128 x 128 six filters with center frequencies at 1, 2, 4, 8, 16, 32, and 64 cycles/image were used. They were able to successfully segment and classify a variety of natural images as well as synthetic texture pairs described by Julesz with identical second-order statistics (see Fig. 4). 3.4.3. Gabor and wavelet models The Fourier transform is an analysis of the global frequency content in the signal. Many applications require the analysis to be localized in the spatial domain. This is usually handled by introducing spatial dependency into the Fourier analysis. The classical way of doing this is through what is called the window Fourier transform. The window Fourier transform (or short-time Fourier transform) of a one-dimensional signal f ( s ) is defined as:
1, 00
~,(u, =
f ( z ) w ( z - t)e-j2nuzds
.
(3.19)
When the window function w ( s ) is Gaussian, the transform becomes a Gabor transform. The limits on the resolution in the time and frequency domain of the window Fourier transform are determined by the tame-bandwidth product or the Heisenberg uncertainty inequality given by:
AtAu 2
1 47r
-
.
(3.20)
Once a window is chosen for the window Fourier t,ransform, the time-frequency resolution is fixed over the entire time-frequency plane. To overcome the resolution limitation of the window Fourier transform, one lets the At and Au vary in the time-frequency domain. Intuitively, the time resolution must increase as the central frequency of the analyzing filter is increased. That is, the relative bandwidth is kept constant in a logarithmic scale. This is accomplished by using a window whose width changes as the frequency changes. Recall that when a function f ( t ) is scaled in time by a , which is expressed as f ( a t ) , the function is contracted if a > 1 and it is expanded when a < 1. Using this fact, the wavelet transform can be written as: (3.21)
2.1 Texture Analysis 235 Here, the impulse response of the filter bank is defined to be scaled versions of the same prototype function h(t). Now, setting in Eq. (3.21)
h ( t ) = w(t)e--j2"Ut
(3.22)
we obtain the wavelet model for texture analysis. Usually the scaling factor will be based on the frequency of the filter. Daugman [80] proposed the use of Gabor filters in the modeling of the receptive fields of simple cells in the visual cortex of some mammals. The proposal to use the Gabor filters in texture analysis was made by Turner [81]and Clark and Bovik [82]. Later Farrokhnia and Jain used it successfully in segmentation and classification of textured images [40,83]. Gabor filters have some desirable optimality properties. Daugman [84] showed that for two-dimensional Gabor functions, the uncertainty relations AxAu 2 n/4 and AyAu 2 n/4 attain the minimum value. Here Ax and Ay are effective widths in the spatial domain and Au and Av are effective bandwidths in the frequency domain. A two-dimensional Gabor function consists of a sinusoidal plane wave of a certain frequency and orientation modulated by a Gaussian envelope. It is given by: (3.23) where uo and 4 are the frequency and phase of the sinusoidal wave. The values LT, and gY are the sizes of the Gaussian envelope in the x and y directions, respectively. The Gabor function at an arbitrary orientation 8 0 can be obtained from Eq. (3.23) by a rigid rotation of the x-y plane by 00. The Gabor filter is a frequency and orientation selective filter. This can be seen from the Fourier domain analysis of the function. When the phase 13 is 0, the Fourier transform of the resulting even-symmetric Gabor function f ( z , y) is given by
(3.24) ~ . function is realwhere (T, = 1/(27ra,),a, = 1/(27rgY), and A = 2 7 r ( ~ , ( ~This valued and has two lobes in the spatial frequency domain, one centered around uo and another centered around -UO. For a Gabor filter of a particular orientation, the lobes in the frequency domain are also appropriately rotated. Jain and Farrokhnia [40]used a version of the Gabor transform in which window sizes for computing the Gabor filters are selected according to the central frequencies of the filters. The texture features were obtained as follows: (a) Use a bank of Gabor filters at multiple scales and orientations to obtain filtered images. Let the filtered image for the i t h filter be ri(x, y). (b) Pass each filtered image through a sigmoidal nonlinearity. This nonlinearity $(t) has the form of tanh(at). The choice of the value of (Y is determined empirically.
236
M . Tuceryan & A . K. J a i n
(c) The texture feature for each pixel is computed as the absolute average deviation of the transformed values of the filtered images from the mean within a window W of size M x M . The filtered images have zero mean, therefore, the ith texture feature image ei(lc, y) is given by the equation: (3.25) The window size M is also determined automatically based on the central frequency of the filter. An example texture image and some intermediate results are shown in Fig. 17. Texture features using Gabor filters were used in texture segmentation and texture classification tasks successfully. An example of the resulting segmentation is shown in Fig. 18. Further details of the segmentation algorithm are explained in Section 4.1. 4. Texture Analysis Problems
The various methods for modelling textures and extracting texture features can be applied in four broad categories of problems: texture segmentation, texture classification, texture synthesis, and shape from texture. We now review these four areas.
4.1. Tezture Segmentation Texture segmentation is a difficult problem because one usually does not know a priori what types of textures exist in an image, how many different textures there are, and what regions in the image have which textures. In fact, one does not need t o know which specific textures exist in the image in order to do texture segmentation. All that is needed is a way t o tell that two textures (usually in adjacent regions of the images) are different. The two general approaches to performing texture segmentation are analogous to methods for image segmentation: region-based approaches or boundary-based approaches. In a region-based approach, one tries to identify regions of the image which have a uniform texture. Pixels or small local regions are merged based on the similarity of some texture property. The regions having different textures are then considered to be segmented regions. This method has the advantage that the boundaries of regions are always closed and therefore, the regions with different textures are always well separated. It has the disadvantage, however, that in many region-based segmentation methods, one has to specify the number of distinct textures present in the image in advance. In addition, thresholds on similarity values are needed. The boundary-based approaches are based upon the detection of differences in texture in adjacent regions. Thus boundaries are detected where there are differences in texture. In this method, one does not need to know the number of textured regions in the image in advance. However, the boundaries may have
238
M . Tuceryan fd A . K. Jain
Fig. 18. The rc:sults of integrating region-based and boundary-based processing usirig the miultiscale Gabor filtering method. (a) Original image consisting of five natural textures. (b) Seven category region-based segmentation results. (c) Edge-based processing and texture edges detected. (d) New segmentation after combining region-based and edge-based results.
Boundary-based segmentation of textured images have been used by Tuceryan and Jain [49], Voorhees and Poggio [ 5 6 ] ,and Eom and Kashyap [85]. In all cases, the edges (or texture boundaries) are detected by taking two adjacent windows and deciding whether the textures in the two windows belong t o the same texture or to different textures. If it is decided that the two textures are different, the point is marked as a boundary pixel. Du Buf and Kardan [86] studied and compared the performance of various texture segmentation techniques and their ability to localize the boundaries. Tuceryan and Jain [49] use the texture features computed from the Voronoi polygons in order to compare the textures in the two windows. The comparison is done using a Kolmogorov-Smirnoff test. A probabilistic relaxation labeling, which enforces border smoothness, is used to remove isolated edge pixels and fill boundary gaps. Voorhees and Poggio extract blobs and elongated structures from images (they suggest that these correspond to Julesz’s textons). The texture properties are based on blob characteristics such as their sizes, orientations, etc. They then decide whether the two sides of a pixel have the same texture using a statistical test called maximum frequency difference (MFD). The pixels where this statistic is sufficiently large are considered to be boundaries between different textures.
2.1 Texture Analysis 239 Jain and Farrokhnia [40] give an example of integrating a region-based and a boundary-based method to obtain a cleaner and more robust texture segmentation method. They use the texture features computed from the bank of Gabor filters to perform a region-based segmentation. This is accomplished by the following steps: (a) Gabor features are calculated from the input image, yielding several feature images. (b) A cluster analysis is performed in the Gabor feature space on a subset of randomly selected pixels in the input image (this is done in order t o increase computational efficiency. About 6% of the total number of pixels in the image are selected). The number k of clusters is specified for doing the cluster analysis. This is set to a value larger than the true number of clusters and thus the image is oversegmented. (c) Step (b) assigns a cluster label to the pixels (pattern) involved in cluster analysis. These labelled patterns are used as the training set and all the pixels in the image are classified into one of the k clusters. A minimum distance classifier is used. This results in a complete segmentation of the image into uniform textured regions. (d) A connected component analysis is performed to identify each segmented region. (e) A boundary-based segmentation is performed by applying the Canny edge detectlor on each feature image. The magnitude of the Canny edge detector for each feature image is summed up for each pixel to obtain a total edge response. The edges are then detected based on this total magnitude. (f) The edges so detected are then combined with the region-based segmentation results to obtain the final texture segmentation. The integration of the boundary-based and region-based segmentation results improve the resulting segmentation in most cases. For a n example of this improvement see Fig. 18. 4.2. Texture Classification
Texture classification involves deciding what texture category an observed image belongs to. In order to accomplish this, one needs t o have a priori knowledge of the classes to be recognized. Once this knowledge is available and the texture features are extracted, one then uses classical pattern classification techniques in order to do the classification. Examples where texture classification was applied as the appropriate texture processing method include the classification of regions in satellite images into categories of land use [41]. Texture classification was also used in automated paint inspection by Farrokhnia [83]. In the latter application, the categories were ratings of the quality of paints obtained from human experts. These quality rating categories were then used as the training samples for supervised classification of paint images using texture features obtained from multi-channel Gabor filters.
240
M. Tzlceryan & A . K. Jain
4.3. Texture Synthesis Texture synthesis is a problem which is more popular in computer graphics. It is closely tied to some of the methods discussed above, so we give only a brief summary here. Many of the modelling methods are directly applicable to texture synthesis. Markov random field models discussed in Section 3.3.1 can be directly used to generate textures by specifying the parameter vector 8 and sampling from the probability distribution function [62,60]. The synthetic textures in Fig. 2(b) are generated using a Gaussian Markov random field model and the algorithm in [87]. Fractals have become popular recently in computer graphics for generating realistic looking textured images [88]. A number of different methods have been proposed for synthesizing textures using fractal models. These methods include midpoint displacement method and Fourier filtering method. The midpoint displacement method has become very popular because it is a simple and fast algorithm yet it can be used to generate very realistic looking textures. Here we only give the general outline of the algorithm. A much more detailed discussion of the algorithm can be found in [88]. The algorithm starts with a square grid representing the image with the four corners set to 0. It then displaces heights at the midpoints of the four sides and the center point of the square region by random amounts and repeats the process recursively. The iteration n 1 uses the grid consisting of the midpoints of the squares in the grid for iteration n. The height a t the midpoint is first interpolated between the endpoints and a random value is added t o this value. The amount added is chosen from a normal distribution with zero mean and variance 02 at iteration n. In order t o keep the self-similar nature of the surface, the variance is changed as a function of the iteration number. The variance at iteration n is given by 2 on = r 2 n H where r = 1/h . (4.1)
+
This results in a fractal surface with fractal dimension (3 - H ) . The heights of the fractal surface can be mapped onto intensity values to generate the textured images. The example image in Fig. 2(c) was generated using this method. Other methods include mosaic models [89,90].This class of models can in turn be divided into subclasses of cell structure models and coverage models. In cell structure models the textures are generated by tessellating the plane into cells (bounded polygons) and assigning each cell gray levels according to a set of probabilities. The type of tessellation determines what type of textures are generated. The possible tessellations include triangular pattern, checkerboard patterns, Poisson line model, Delaunay model, and occupancy model. In coverage models, the texture is obtained by a random arrangement of a set of geometric figures in the plane. The coverage models are also referred t o as bombing models. 4.4. Shape from Texture
There are many cues in images that allow the viewer to make inferences about the three-dimensional shapes of objects and surfaces present in the image. Examples
2.1 Texture Analysis 241 of such cues include the variations of shading on the object surfaces or the relative configurations of boundaries and the types of junctions that allow one t o infer three-dimensional shape from the line drawings of objects. The relation between the variations in texture properties and surface shape was first pointed out by Gibson [lo]. Stevens observed that certain properties of texture are perceptually significant in the extraction of surface geometry [91]. There are three effects that surface geometry has on the appearance of texture in images: foreshortening and scaling of texture elements, and a change in their density. The foreshortening effect is due to the orientation of the surface on which the texture element lies. The scaling and density changes are due to the distance of the texture elements from the viewer. Stevens argued that texture density is not a useful measure for computing distance or orientation information because the density varies both with scaling and foreshortening. He concluded that the more perceptually stable property that allows one to extract surface geometry information is the direction in the image which is not foreshortened, called the characteristic dimension. Stevens suggested that one can compute relative depth information using the reciprocal of the scaling in the characteristic dimension. Using the relative depths, surface orientation can be estimated. Bajcsy and Lieberman [92] used the gradient in texture element sizes to derive surface shape. They assumed a uniform texture element size on the threedimensional surface in the scene. The relative distances are computed based on a gradient function in the image which was estimated from the texture element sizes. The estimation of the relative depth was done without using knowledge about the camera parameters and the original texture element sizes. Witkin [93] used the distribution of edge orientations in the image to estimate the surface orientation. The surface orientation is represented by the slant (a) and tilt ( T ) angles. The Slant is the angle between a normal to the surface and a normal to the image plane. The Tilt is the angle between the surface normal’s projection onto the image plane and a fixed coordinate axis in the image plane. He assumed an isotropic texture (uniform distribution of edge orientations) on the original surface. As a result of the projection process, the textures are foreshortened in the direction of steepest inclination (slant angle). Note that this idea is related to Stevens’ argument because the direction of steepest inclination is perpendicular to the characteristic dimension. Witkin formulated the surface shape recovery by relating the slant and tilt angles t o the distribution of observed edge directions in the image. Let /3 be the original edge orientation (the angle between the tangent and a fixed coordinate axis on the plane S containing the tangent). Let a* be the angle between the x-axis in the image plane and the projected tangent. The a* is related to the slant and tilt angles by the following expression:
-)
tan /3 o* = atan( cos a
+7
242
M. Tuceryan & A . K. Jain
Here a* is an observable quantity in the image and ( a , ~are ) the quantities to be computed. Witkin derived the expression for the conditional probabilities for the slant and tilt angles given the measured edge directions in the image and then . A* = used a maximum likelihood estimation method to compute the ( a , ~ )Let { a ; ,. . . , a:} be a set of observed edge directions in the image. Then the conditional probabilities are given as: (4.3)
sin a where P(a,T ) = -. The maximum likelihood estimate of P ( a ,T ~ A *gives ) the 76 desired surface orientation. Blostein and Ahuja [57] used the scaling effect to extract surface information. They integrated the process of texture element extraction with the surface geometry computation. Texture element extraction is performed at multiple scales and the subset that yields a good surface fit is selected. The surfaces are assumed planar for simplicity. Texture elements are defined to be circular regions of uniform intensity which are extracted by filtering the image with V2G and &(V2G) operators and comparing the filter responses to those of an ideal disk (here a is the size of the Gaussian G). At, the extremum points of the image filtered by V2G, the diameter ( D ) and contrast (C) of the best fitting disks are computed. The convolution is done at multiple scales. Only those disks whose computed diameters are close to the size of the Gaussian are retained. As a result, blob-like texture elements of different sizes are detected. The geometry of the projection is shown in Fig. 19. Let a and T be the slant and tilt of the surface. The image of a texture element has the foreshortened dimension Fi and the characteristic dimension U,. The area A, of the image texel is proportional to the product FiU, for compact shapes. The expression for the area A, of the image of a texture element is given by:
Ai
= A c ( l - tanBtana)3
(4.4)
where Ac is the area that would be measured for the texel at the center of the image. The angle B is given by the expression B = atan((xcosr
+ ysinT)(r/f)) .
(4.5)
Here, r is the physical width of the image, r / f is a measure of the field of view of the camera, and (2,y) denotes pixel coordinates in the image. Ai can be measured in the image. To find the surface orientation, an accumulator array consisting of the parameters ( A c ,g,T ) is constructed. For each combination of parameter values, a possible planar fit is computed. The plane with the highest fit rating is selected as the surface orientation, and texture elements that support this fit are selected as
2.1 Texture Analysis 243
"m" 4
imane of texture element f length F,
The slant of the plane CT
I
\
Texture element of length Fp on the surface.
Fig. 19. The projective distortion of a texture element in the image.
Fig. 20. Examples of shape from texture computation using Blostein and Ahuja's algorithm [57]. (a) An image of a field of rocks and the computed slant and tilt of the plane. (6) An image of a sunflower field and the extracted slant and tilt values.
the true texture elements. Some example images and the computed slant and tilt values are shown in Fig. 20. 5 . Summary
This chapter has reviewed the basic concepts and various methods and techniques for processing textured images. Texture is a prevalent property of most
244
M. Tuceryan & A . K. Jain
physical surfaces in the natural world. It also arises in many applications such as satellite imagery and printed documents. Many common low level vision algorithms such as edge detection break down when applied to images that contain textured surfaces. It is therefore crucial that we have robust and efficient methods for processing textured images. Texture processing has been successfully applied to practical application domains such as automated inspection and satellite imagery. It is also going t o play an important role in the future as we can see from the promising application of texture to a variety of different application domains. Acknowledgment
The support of the National Science Foundation through grants IRI-8705256 and CDA-8806599 is gratefully acknowledged. We thank the Norwegian Computing center for providing the SAR images shown in Fig. 5. We also thank our colleagues Dr. Richard C. Dubes and Dr. Patrick J. Flynn for the invaluable comments and feedback they provided during the preparation of this document. References [l] J. M. Coggins, A Framework for Texture Analysis Based on Spatial Filtering, Ph.D. Thesis, Computer Science Department, Michigan State University, East Lansing, MI, 1982. [2] H. Tamura, S. Mori and Y. Yamawaki, Textural features corresponding to visual perception, IEEE Trans. Syst. Man Cybern., (1978) 46Ck473. [3] J. Sklansky, Image segmentation and feature extraction, IEEE Trans. Syst. Man Cybern. (1978) 237-247. [4] R. M. Haralick, Statistical and structural approaches to texture, Proc. IEEE 67 (1979) 786-804. [5] W. Richards and A. Polit, Texture matching, Kybernetic 16 (1974) 155-162. [6] S. W. Zucker and K. Kant, Multiple-level representations for texture discrimination, in Proc. IEEE Conf. on Pattern Recognition and Image Processing, Dallas, TX, 1981, 609-614. [7] J. K. Hawkins, Textural properties for pattern recognition, in B. Lipkin and A. Rosenfeld (eds.), Picture Processing and Psychopictorics (Academic Press, New York, 1969). [8] P. Brodatz, Textures: A Photographic Album for Artists and Designers (Dover Publications, New York 1966). [9] C. C. Chen, Markov Random Fields in Image Analysis, Ph.D. Thesis, Computer Science Department, Michigan State University, East Lansing, MI, 1988. [lo] J. J. Gibson, The Perception of the Visual World (Houghton Mifflin, Boston, MA, 1950) . [ll] B. Julesz, E. N. Gilbert, L. A. Shepp and H. L. Frisch, Inability of humans to discriminate between visual textures that agree in second-order statistics -revisited, Perception 2 (1973) 391-405. 1121 B. Julesz, Visual pattern discrimination, IRE Trans. Inf. Theory 8 (1962) 84-92. [13] B. Julesz, Experiments in the visual perception of texture, Sci. Am. 232 (1975) 34-43. [14] B. Julesz, Nonlinear and cooperative processes in texture perception, in T. P. Werner and E. Reichardt (eds.), Theoretical Approaches in Neurobiology (MIT Press, Cambridge, MA, 1981) 93-108.
2.1 Texture Analysis 245 [15] B. Julesz, Textons, the elements of texture perception, and their interactions, Nature 290 (1981) 91-97. [16] B. Julesz, A theory of preattentive texture discrimination based on first-order statistics of textons, Biol. Cybern. 41 (1981) 131-138. [17] T. Caelli, Visual Perception (Pergamon Press, 1981). [18] J. Beck, A. Sutter and R. Ivry, Spatial frequency channels and perceptual grouping in texture segregation, Comput. Vision Graph. Image Process. 37 (1987) 299-325. [19] F. W. Campbell and J. G. Robson, Application of Fourier analysis to the visibility of gratings, J. Physiol. 197 (1968) 551-566. [20] R. L. Devalois, D. G. Albrecht and L. G. Thorell, Spatial-frequency selectivity of cells in macaque visual cortex, Vision Res. 22 (1982) 545-559. [21] P. Dewaele, P. Van Goo1 and A. Oosterlinck, Texture inspection with self-adaptive convolution filters, in Proc. 9th Int. Conf. on Pattern Recognition, Rome, Italy, Nov. 1988, 56-60. [22] D. Chetverikov, Detecting defects in texture, in Proc. 9th Int. Conf. on Pattern Recognition, Rome, Italy, Nov. 1988, 61-63. [23] J. Chen and A. K. Jain, A structural approach to identify defects in textured images, in Proc. IEEE Int. Conf. on Systems, Man, and Cybernetics, Beijing, 1988, 29-32. [24] R. W. Conners, C. W. McMillin, K. Lin and R. E. Vasquez-Espinosa, Identifying and locating surface defects in wood: Part of an automated lumber processing system, I E E E Trans. Pattern Anal. Mach. Intell. 5 (1983) 573-583. [25] L. H. Siew, R. M. Hodgson and E. J. Wood, Texture measures for carpet wear assessment, IEEE Trans. Pattern Anal. Mach. Intell. 10 (1988) 92-105. [26] A. K. Jain, F. Farrokhnia and D. H. Alman, Texture analysis of automotive finishes, in Proc. of S M E Machine Vision Applications Conf., Detroit, MI, Nov. 1990, 1-16. [27] R. Sutton and E. L. Hall, Texture measures for automatic classification of pulmonary disease, IEEE Trans. Comput. 21 (1972) 667-676. (281 H. Harms, U. Gunzer and H. M. Aus, Combined local color and texture analysis of stained cells, Comput. Vision Graph. Image Process. 33 (1986) 364-376. [29] G. H. Landeweerd and E. S. Gelsema, The use of nuclear texture parameters in the automatic analysis of leukocytes, Pattern Recogn. 10 (1978) 57-61. [30] ] M. F. Insana, R. F. Wagner, B. S. Garra, D. G. Brown and T. H. Shawker, Analysis of ultrasound image texture via generalized Rician statistics, Opt. Engin. 25 (1986) 743-748. [31] C. C. Chen, J. S. Daponte and M. D. Fox, Fractal feature analysis and classification in medical imaging, I E E E Trans. Medical Imaging 8 (1989) 133-142. [32] A. Lundervold, Ultrasonic tissue characterization-A pattern recognition approach, Technical Report, Norwegian Computing Center, Oslo, Norway, 1992. [33] D. Wang and S. N. Srihari, Classification of newspaper image blocks using texture analysis, Comput. Vision Graph. Image Process. 47 (1989) 327-352. (341 F. M. Wahl, K. Y. Wong and R. G. Casey, Block segmentation and text extraction in mixed text/image documents, Comput. Graph. Image Process. 20 (1982) 375-390. [35] J. A. Fletcher and R. Kasturi, A robust algorithm for text string separation from mixed text/graphics images, IEEE Trans. Pattern Anal. Mach. Intell. 10 (1988) 910918. [36] T. T a t , P. J. Flynn and A. K. Jain, Segmentation of document images, I E E E Trans. Pattern Anal. Mach. Intell. 11 (1989) 1322-1329. [37] A. K. Jain and S. K. Bhattacharjee, Text segmentation using Gabor filters for automatic document processing, Mach. Vision and Appl. 5 (1992) 169-184.
246
M. Tuceryan & A . K. Jain
[38] A. K. Jain and S. K. Bhattacharjee, Address block location on envelopes using Gabor filters, in PTOC.11th Int. Conf. o n Pattern Recognition, The Hague, Netherlands, Aug. 1992, Vol. B, 264-267. 1391 A. K. Jain, S. K. Bhattacharjee and Y. Chen, On texture in document images, in PTOC.IEEE Conf. o n Computer Vision and Pattern Recognition, Champaign, IL, Jun. 1992, 677-680. 1401 A. K. Jain and F. Farrokhnia, Unsupervised texture segmentation using Gabor filters, Pattern Recogn. 24 (1991) 1167-1186. [41] R. M. Haralick, K. Shanmugam and I. Dinstein, Textural features for image classification, I E E E Trans. Syst. M a n Cybern. 3 (1973) 61G621. [42] E. Rignot and R. Kwok, Extraction of textural features in SAR images: Statistical model and sensitivity, in Proc. Int. Geoscience and Remote Sensing Symp., Washington, DC, 1990, 1979-1982. [43] A. H. Schistad and A. K. Jain, Texture analysis in the presence of speckle noise, in Proc. IEEE Geoscience and Remote Sensing Symp. Houston, TX, May 1992, 147-152. [44] L. J. Du, Texture segmentation of SAR images using localized spatial filtering, in Proc. Int. Geoscience and Remote Sensing Symp., Washington, DC, 1990, 1983-1986. [45] J . H. Lee and W. D. Philpot, A spectral-textural classifier for digital imagery, in PTOC. Int. Geoscience and Remote Sensing Symp., Washington, DC, 1990, 2005-2008. [46] F. Tomita and S. Tsuji, Computer Analysis of Visual Textures (Kluwer Academic Publishers, Boston, 1990). (471 K. I. Laws, Textured Image Segmentation, Ph.D. thesis, University of Southern California, 1980. [48] R. Picard, I. M. Elfadel and A. P. Pentland, Markov/Gibbs texture modeling: Aura matrices and temperature effects, in Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Maui, Hawaii, 1991, 371-377. [49] M. Tuceryan and A. K. Jain, Texture segmentation using Voronoi polygons, IEEE Trans. Pattern Anal. Mach. Intell. 12 (1990) 211-216. [50] N. Ahuja, Dot pattern processing using Voronoi neighborhoods, I E E E Trans. Pattern Anal. Mach. Intell. 4 (1982) 336-343. [51] G. Voronoi, Nouvelles applications des p a r a m h e s continus 5 la thkorie des formes quadratiques. Deuxihme m6moire: Recherches sur les parallkllo2dres primitifs, J . Reine Angezu. Math. 134 (1908) 198-287. [52] M. I. Shamos and D. Hoey, Closest-point problems, in 16th Annual Symposium on Foundations of Computer Science, 1975, 131-162. 1531 F. P. Preparata and M. I. Shamos, Computational Geometry (Springer-Verlag, New York, 1985). [54] D. Marr, Vision (Freeman, San Francisco, 1982). [55] M. K. Hu, Visual pattern recognition by moment invariants, I R E Trans. Inf. Theory 8 (1962) 179-187. [561 H. Voorhees and T. Poggio, Detecting textons and texture boundaries in natural images, in Proc. First Int. c o n f. on Computer Vision, London, 1987 25G-258. [57] D. Blostein and N. Ahuja, Shape from texture: Integrating texture-element extraction and surface estimation, IE E E Trans. Pattern Anal. Mach. Intell. 11 (1989) 1233-1251. [58] S. W. Zucker, Toward a model of texture, Comput. Graph. Image Process. 5 (1976) 190-202. [59] K. S. Fu, Syntactic Pattern Recognition and Applications (Prentice-Hall, New Jersey, 1982).
2.1 Texture Analysis 247 [SO] G. C. Cross and A. K. Jain, Markov random field texture models, IEEE Trans. Pattern Anal. Mach. Intell. 5 (1983) 25-39. [61] R. Chellappa and S. Chatterjee, Classification of Textures using Gaussian Markov random fields, IEEE Trans. Acoust. Speech Signal Process. 33 (1985) 959-963. [62] A. Khotanzad and R. Kashyap, Feature Selection for texture recognition based on image synthesis, IEEE Trans. Syst. Man Cybern. 17 (1987) 1087-1095. [63] F. S. Cohen and D. B. Cooper, Simple parallel hierarchical and relaxation algorithms for segmenting noncausal Markovian random fields, IEEE Trans. Pattern Anal. Mach. Intell. 9 (1987) 195-219. [64] C. W. Therrien, An estimation-theoretic approach to terrain image segmentation, Comput. Vision Graph. Image Process. 22 (1983) 313-326. [65] S. Geman and D. Geman, Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images, IEEE Trans. Pattern Anal. Mach. Intell. 6 (1984) 721-741. [66] J. Besag, Spatial interaction and the statistical analysis of lattice systems, J. Roy. Stat. SOC.B36 (1974) 344-348. [67] H. Derin and H. Elliott, Modeling and segmentation of noisy and textured images using Gihbs random fields, IEEE Trans. Pattern Anal. Mach. Intell. 9 (1987) 39-55. [68] B. B. Mandelbrot, The Fractal Geometry of Nature (Freeman, San Francisco, 1983). [69] A. Pentland, Fractal-based description of natural scenes, IEEE Trans. Pattern Anal. Mach. Intell. 9 (1984) 661-674. [70] J. M. Keller, S. Chen and R. M. Crownover, Texture description and segmentation through fractal geometry, Comput. Vision Graph. Image Process. 45 (1989) 15G166. [71] R. Voss, Random fractals: Characterization and measurement, in R. Pynn and A. Skjeltorp (eds.), Scaling Phenomena in Disordered Systems (Plenum, New York, 1986). [72] B. J. Super and A. C. Bovik, Localized measurement of image fractal dimension using Gabor filters, J. Visual Commun. Image Represent. 2 (1991) 114-128. [73] P. P. Ohanian and R. C. Dubes, Performance evaluation for four classes of textural features, Pattern Recogn, Vol. 25, no. 8 (1992) pp. 819-833. [74] M. A. Georgeson, Spatial Fourier analysis and human vision, Chapter 2, in N. S. Sutherland (ed.), Tutorial Essays in Psychology, A Guide to Recent Advances, vol. 2 (Lawrence Erlbaum Associates, Hillsdale, NJ, 1979). [75] J. Malik and P. Perona, Preattentive texture discrimination with early vision mechanisms, J . Opt. SOC.Am. Series A 7 (1990) 923-932. [76] M. Unser and M. Eden, Nonlinear operators for improving texture segmentation based on features extracted by spatial filtering, IEEE Trans. Syst. M a n Cybern. 20 (1990) 804-815. [77] T. R. Reed and H. Wechsler, Segmentation of textured images and Gestalt organization using spatiallspatial-frequency representations, IEEE Trans. Pattern Anal. Mach. Intell. 12 (1990) 1-12 [78] M. Tuceryan, Moment based texture segmentation, in Proc. 11th Int. Conf. o n Pattern Recognition, The Hague, Netherlands, Aug. 1992, Vol. 111, 45-48. [79] J. M. Coggins and A. K. Jain, A spatial filtering approach to texture analysis, Pattern Recogn. Lett. 3 (1985) 195-203. [80] J. G. Daugman, Two-dimensional spectral analysis of cortical receptive field profiles, Vision Res. 20 (1980) 847-856. [81] M. R. Turner, Texture discrimination by Gabor functions, Biol. Cybern. 55 (1986) 71-82. 1821 M. Clark and A. C. Bovik, Texture segmentation using Gabor modulation/ demodulation, Pattern Recogn. Lett. 6 (1987) 261-267.
248
M. Tuceryan & A . K. Jain
[83] F. Farrokhnia, Multi-channel Filtering Techniques for Texture Segmentation and Surface Quality Inspection, Ph.D. thesis, Computer Science Department, Michigan State University, 1990. [84] J. G. Daugman, Uncertainty relation for resolution in space, spatial-frequency, and orientation optimized by two-dimensional visual cortical filters, J. Opt. SOC.A m . 2 (1985) 1160-1169. [85] Kie-Bum Eom and R. L. Kashyap, Texture and intensity edge detection with random field models, in Proc. Workshop on Computer Vision, Miami Beach, FL, 1987, 29-34. [86] J. M. Du Buf, H. M. Kardan and M. Spann, Texture feature performance for image segmentation, Pattern Recogn. 23 (1990) 291-309. [87] R. Chellappa, S. Chatterjee and R. Bagdazian, Texture synthesis and compression using Gaussian-Markov random field models, I E E E Trans. Syst. Man Cybern. 15 (1985) 298-303. [88] H. 0. Peitgen and D. Saupe, The Science of Fractal Images (Springer-Verlag, New York, 1988). [89] N. Ahuja and A. Rosenfeld, Mosaic models for textures, I E E E Trans. Pattern Anal. Mach. Intell. 3 (1981) 1-11. [go] N. Ahuja, Texture, in Encyclopedia of Artificial Intelligence (Wiley, 1987) 1101-1115. [91] K. A. Stevens, Surface perception from local analysis of texture and contour, MIT Technical Report, Artificial Intelligence Laboratory, no. AI-TR 512, 1980. [92] R. Bajcsy and L. Lieberman, Texture gradient as a depth cue, Comput. Graph. Image Process. 5 (1976) 52-67. [93] A. P. Witkin, Recovering surface shape and orientation from texture, Artif. Intell. 17 (1981) 17-45.
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 249-282 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
1 CHAPTER 2.2 1 MODEL-BASED TEXTURE SEGMENTATION AND CLASSIFICATION
R. CHELLAPPA Department of Electrical Engineering, Center for Automation Research Institute for Advanced Computer Studies, University of Maryland College Park, M D 20742, U S A R. L. KASHYAP School of Electrical Engineering, Purdue IJniversity W. Lafayette, IN 47907, U S A and
B. S . MANJUNATH Department of Electrical and Computer Engineering University of California, Santa Barbara, C A 93106, USA Over the last ten years, several model based methods have been proposed for segmentation and classification of textured images. Models based on random field representations and psychophysical/neurophysiological studies have been dominant. In this chapter, we present examples drawn from both approaches. Related issues on implementation of the various optimal/suboptimal algorithms are also addressed.
Keywords: Texture segmentation, texture classification, artificial neural networks, Markov random fields, fractional differencing model, preattentive segmentation.
1. Introduction Automatic segmentation and classification of textured images has several applications in landsat terrain classification [l],bio-medical applications [2] and aerial image understanding [3]. Previous approaches to segmentation have been based on correlation [4],Fourier transform features [5], Laws features and their extensions [6,7],fractal models [8], and features from co-occurrence matrix [9]. Recently, more emphasis has been given to methods using random field models such as the 2-D non-symmetric half plane models [lo] and non-causal Gauss Markov random field models and their variations [ll-181. Both supervised and unsupervised methods have been developed. Although significant progress has been made using these methods, several problems remain as the methods are sensitive to illumination and resolution changes and transformations such as rotation. Also, these methods do not explain the role of preattentive segmentation as applied to textures. Preattentive
249
250
R. Chellappa, R. L. Kashyap & B. S. Manjunath
segmen tation refers to the ability of humans to perceive textures without any sustained attention. Central to solving this problem are the issues of what features need to be computed and what kind of processing of these features is required for texture discrimination. Some of the early work in this field can be attributed to Julesz [19] for his theory of textons as basic textural elements. The spatial filtering approach has been used by many researchers for detecting texture boundaries not clearly explained by the texton theory [20]. Recently an elegant computational model for preattentive texture discrimination has been proposed by Malik and Perona [21]. Grossberg and Mingolla’s Boundary Contour System (BCS) [22] is one of the first attempts to model the early processing stages in the visual cortex. Texture classification refers to the problem of identifying the particular class label of the input texture and can operate on the output of the segmentation algorithm. Thus, standard pattern classification techniques may be applied by assuming that there is only one texture in the image, the image being constructed from a single segmented region. Features for texture classification have been derived from a variety of approaches such as co-occurrence matrices 11,231, textural features [5,24,25], runlength statistics [5], difference statistics [5],decorrelation methods [26], Fourier power spectrum [5], structural features [9,27], region based random fields [28-301, parametric Gaussian random field models [31-381, fractals and fractional models [36,39,40], etc. A major advantage of the features based on parametric Gaussian non-causal random field models is that they are information preserving in the sense that the features in conjunction with the discrete random field model can be used to synthesize an image which closely resembles the original. The chief disadvantage of the above model and all other related classification methods is that they are not rotation invariant, i.e. if we train the classifier with a set of texture data and test the classifier with a rotated version of the same image, the correct classification rate goes down. We give an approach for achieving rotational invariance in Section 4. We illustrate the different approaches to texture segmentation and classification mentioned above using several deterministic and stochastic algorithms. The first method we describe in Section 2 stems from the idea of using Markov random field (MRF) models for texture in an image. We assign two random variables for the observed pixel, one characterizing the underlying intensity and the other for labeling the texture corresponding to the pixel location. We use the Gauss Markov Random Field (GMRF) model for the conditional density of the intensity field given the label field. Prior information about the texture label field is introduced using a discrete Markov distribution. The segmentation can then be formulated as a n optimization problem involving minimization of a Gibbs energy function. Exhaustive search for the optimum solution is not possible because of the large dimensionality of the search space. For example, even for the very simple case of segmenting a 128 x 128 image into two classes, there are 2214possible label configurations. Derin and Elliott [13] have investigated the use of dynamic programming for obtaining the Maximum a posteriori (MAP) estimate while Cohen and Cooper [11] give a deterministic relaxation algorithm for the same problem. The optimal MAP
&. 2 Model-Based Texture Segmentation and Classification 251
solution can be obtained by using stochastic relaxation algorithms like simulated annealing [41]. However, the computational burden involved because of the theoretical requirements on the initial temperature and the impractical cooling schedules overweigh their advantages in many cases. Recently there has been considerable interest in using neural networks for solving computationally hard problems. Fast approximate solutions can be obtained by using a deterministic relaxation algorithm like the iterated conditional mode rule [42]. The energy function corresponding to this optimality criterion can be mapped into a Hopfield type network in a straightforward manner and it can be shown that the network converges to a n equilibrium state, which in general will be a local optimum. The solutions obtained using this method are sensitive to the initial configuration and in many cases starting with a maximum likelihood estimate is preferred. The second optimality criterion we discuss minimizes the expected percentage of classification error per pixel. This is equivalent t o finding the pixel labels that maximizes the marginal posterior probability given the intensity data [43]. Since calculating the marginal posterior probability is very difficult, Marroquin [44] suggested the Maximum Posterior Marginal (MPM) algorithm (see Section 4) that asymptotically computes the posterior marginal. Here we use this method t o find the texture label that maximizes the marginal posterior probability for each pixel. In Section 3 we discuss a simple biologically motivated approach to detect texture boundaries within a more general context of boundary detection [45]. The input image is first processed through a bank of orientation selective bandpass filters at various spatial frequencies. The convolution of the image with these filters yields a representation which is localized in space as well as in frequency. A special class of this decomposition is the wavelet transformation where the filter profiles are all self-similar. Wavelets are families of basis functions obtained through dilations and translations of a basic wavelet and such a decomposition provides a compact data structure for representing information. Following the wavelet decomposition we introduce local feature interactions. Three distinct types of interactions are considered: competition between spatial neighbors in each orientation channel, competition between orientations at each spatial location, and interscale interactions. Interscale interactions are used in localizing line ends and play an important role in boundary detection. The second stage of interactions groups similar features in the neighborhood. This cooperative processing helps in the boundary completion process. The receptive fields of the cells in this stage have the same orientation selectivity as their inputs and have a larger receptive field, and the filter profiles are modeled by oriented Gaussians. In Section 4, we discuss direct patt,ern classification strategies for classifying textures, assuming that there is only one texture in the image or in the segment of the image. The strategy is to fit varieties of parametric random field models, extract features from them, and use these features for classification using both standard algorithms and new procedures. In Section 5, a multi-level classification
252
R. Chellappa, R. L. Kashyap tY B. S. M a n j u n a t h
method based on fractional differencing models with a fractal scaling parameter is presented. This algorithm can handle arbitrary 3-D rotated textures. Since the fractal scale is known to be a rotational and scaling invariant parameter, the accuracy of classification from the procedure will not be affected by 3-D rotation of the test texture. In the first level of classification, the textures are classified by the first-order Fkactional Differencing model with a fractal scale parameter, and in the second level, classification is completed with the additional frequency parameters of the second-order Fractional Differencing periodic model. 2. Texture Segmentation via Optimization and Artificial Neural Networks The inherent parallelism of neural networks provides an interesting architecture for implementing many computer vision algorithms [46]. Some examples are image restoration [47], stereopsis [48] and computing optical flow [49-511. Networks for solving combinatorially hard problems like the Traveling Salesman problem have received much attention in the neural network literature [52]. In all these cases the networksare designed to minimize an energy function defined by the network architecture. The parameters of the network are obtained in terms of the cost function which is to be minimized and it can be shown that [52] for networks having symmetric interconnections, the equilibrium states correspond t o the local minima of the energy function. For practical purposes, networks with few interconnections are preferred because of the large number of processing units required in any image processing application. In this context MRF models for images play a useful role. They are typically characterized by local dependencies and symmetric interconnections which can be expressed in terms of energy functions using Gibbs-Markov equivalence. The artificial neural net (ANN) approach suggested here stems from the idea of using MRF models for textures in an image. 2.1. Markov Random Fields and the Image Model In modeling images consisting of more than one texture we have to consider two random processes, one for the texture intensity distribution and the second for the label distribution. Various models have been proposed in the literature for textured images. In this section we discuss one such model based on Markov random fields. In most image processing applications the input image is a rectangular array of pixels taking values in the range 0-255. Let R denote such a set of grid points on an M x M lattice, i.e. R = { ( i , j ) , 1 5 i , j L: M } . Let { Y s , s E R } be a random process defined on this grid.
Definition. The process {Ys}is said to be strictly Markov if P(YJal1 Yr,r # s) = P(Y,IY,,r is a neighbor of s)
(2.1)
The neighborhood set of site s can be arbitrarily defined. However in many image processing applications it is natural to consider neighbors which are also spatial
2.2 Model-Based Texture Segmentation and Classification 253 neighbors of the site. The Markov process can further be classified as causal or non-causal depending on the relationship of these neighbors with respect t o the site. The use of MRF in image processing applications has a long history (see for e.g. [53]) and MRF have been used inapplications such as image restoration, segmentation, etc. Cross and Jain [54] provide a detailed discussion on the application of MRF in modeling textured images. In the following we use {L,, s E a } t o denote the label s E } for the zero mean intensity process. process and {Ys,
Intensity Process. We model the intensity process { Y s } by a Gaussian Markov random field (GMRF). Depending on the neighborhood set one can construct a hierarchy of GMRF models as shown in Fig. 1. The numbers indicate the order of the GMRF model relative t o the center location x. Note that this defines a symmetric neighborhood set. We have used the fourth order model for the intensity process.
Fig. 1. Structure of the GMRF model. The numbers indicate the order of the model relative t o x [541.
Let N, denote the symmetric fourth order neighborhood of a site s. Let N* be the set of one-sided shift vectors corresponding to the fourth order neighborhood, i.e. N * is the set of shift vectors corresponding to a fourth order neighborhood system,
and
N, where s
+
7-
= (7- : 7- = S + T ,
7-
E N*}
is defined as s = ( i , j ) , 7- = (x,y), s +
7-
= (i
+ x , j + y).
254
R. Chellappa, R. L. Kashyap & B. S. Manjunath
Assuming that all the neighbors of s also have the same label as that of s, the conditional density of the intensity at the pixel s is:
Equation (2.3) is a Gibbs distribution function, V ( . )is often referred to as a Gibbs measure and Z(Z1yr,r E N,) is called the partition function. In (2.4), cq and 0' are the GMRF model parameters of the I-th texture class. A stationary GMRF model implies that the parameters satisfy O;,, = @-;, = @:.-T = 0;. There are several ways of estimating the GMRF parameters and a comparison of different schemes can be found in [53]. We have used the least squares method in our experiments. We view the image intensity array as composed of a set of overlapping Ic x Ic windows W,, centered at each pixel s E 0. In each of these windows we assume that the texture label L , is homogeneous (all the pixels in the window belong to the same texture) and model the intensity distribution in the window by a fourth order stationary GMRF. Let Yz denote the 2-D vector representing the zero mean intensity array in the window W,. Using the Gibbs formulation and assuming a free boundary model, the joint probability density in the window W, can be written as:
where Z l ( l ) is the partition function and
Label Process. The texture labels are assumed t o obey a first or second order discrete Markov model with a single parameter p, which measures the amount of clustering between adjacent pixels. If N , denotes the appropriate neighborhood for the label field, then we can write the distribution function for the texture label at site s conditioned on the labels of the neighboring sites as: P(L,(L, , where
22
T
E
Ns)=
is a normalizing constant and
e-u2(Ls 2 2
I L-1
2.2 Model-Based Texture Segmentation and Classification 255 In (2.6), p determines the degree of clustering, and S ( i Using the Bayes rule, we can write
-
j ) is the Kronecker delta.
Since YB is known, the denominator in (2.7) is just a constant. The numerator is a product of two exponential functions and can be expressed as (2.8) where Zp is the partition function and Up(.) is the posterior energy corresponding to (2.7). From (2.5) and (2.6) we write
Up(Ls I Y : , L,, r E
is) = w(L,) + U l ( Y , *1 L,) + U2(L, 1 L,,
rE
fi,) .
(2.9)
Note that the second term in (2.9) relates the observed pixel intensities to the texture labels and the last term specifies the label distribution. The bias term w(L,) = log Z1(L,) is dependent on the texture class and it can be explicitly evaluated for the GMRF model considered here using the toroidal assumption (the computations become very cumbersome if toroidal assumptions are not made). An alternate approach is to estimate the bias from the histogram of the data as suggested by Geman and Graffigne [15]. Finally, the posterior distribution of the texture labels for the entire image given the intensity array is (2.10) Maximizing (2.10) gives the optimal Bayesian estimate. Though it is possible in principle to compute the right-hand side of (2.10) and find the global optimum, the computational burden involved is so enormous that it is practically impossible to do so. However we note that the stochastic relaxation algorithms discussed in Section 2.3 require only the computation of (2.8) to obtain the optimal solution. The deterministic relaxation algorithm given in the next section also uses these values, but in this case the solution is only an approximation to the MAP estimate. 2.2. A Neural Network for Texture Classification We describe the network architecture used for segmentation and the implementation of deterministic relaxation algorithms. The energy function which the network minimizes is obtained from the image model discussed in the previous section. For convenience of notationlet U l ( i , j , I ) = U 1 ( Y i , L s = I ) w(Z) where s = ( i , j ) denotes a pixel site and Ul( . ) and w(I) are as defined in (2.9). The network consists of K layers, each layer arranged as an M x M array, where K is the number of texture classes in the image and M is the dimension of the image. The elements (neurons) in the network are assumed to be binary and are indexed by ( i ,j , I ) where
+
256
R. Chellappa, R. L. Kashyap & B. S. Manjunath
( i , j ) = s refers to their position in the image and 2 refers to the layer. The ( 2 , j , 2)-th neuron is said to be ON if its output Kjl is 1, indicating that the corresponding site s = ( i , j ) in the image has the texture label 1. Let Tijl;i~j~ be p the connection strength between the neurons (i, j , 2) and (i’, j’, 2’) and Iiji be the input bias current. Then a general form for the energy of the network is [52] M
E=
M
K
M
-~~~~~
c M
K
M
M
K
c T 2 3 i , 2 ~ 3 ~ i-~JExx123iK31. K3i~~3~i~ (2.11) 2
2=13=1 1=1 2’=13’=1l’=l
2=1 3=11=1
We note that a solution for the MAP estimate can be obtained by minimizing (2.10). Here we approximate the posterior energy by
U ( L ) Y *= ) c(U(Y,’lLs)
+ W L S+ U2(LS)}
(2.12)
S
and the corresponding Gibbs energy to be minimized can be written as
E
1
=
M
M
;ccc c
K
K
M
M
2=1
3=1
5 ~ y ~ u l ( z , ~ , 2 ) -K 3 1 2=13=1 I=1
1=1
K’3‘1K31
(2.13)
(2?,3’)Efi,3
where f i i j is the neighborhood of site ( i , j ) (same as the f i s in Section 2). In (2.13), it is implicitly assumed that each pixel site has a unique label, i.e. only one neuron is active in each column of the network. This constraint can be implemented in different ways. For the deterministic relaxation algorithm described below, a simple method is t o use a winner-takes-all circuit for each column so that the neuron receiving the maximum input is turned on and the others are turned off. Alternately a penalty term can be introduced in (2.13) to represent the constraint as in [52]. From (2.11) and (2.13) we can identify the parameters for the network, (2.14)
and the bias current
I.. = - UI(i,j,Z). 2.7 1
(2.15)
2.2.1. Deterministic relaxation The above equations (2.14) and (2.15) relate the parameters of the network to that of the image model. The connection matrix for the above network is symmetric and there is no self feedback, i.e. Tijl;ijl= 0, Vi,j,1. Let uijl be the potential of neuron (i,j,Z). (Note that 1 is the layer number corresponding to texture class I), then M
M
K
(2.16)
2.2 Model-Based Texture Segmentation and Classajication 257 In order t o minimize (2.13), we use the following updating rule: (2.17)
This updating scheme ensures that a t each stage the energy decreases. Since the energy is bounded, the convergence of the above system is assured but the stable state will in general be a local optimum. This network model is a version of the Iterated Conditional Mode algorithm (ICM) of Besag [42]. This algorithm maximizes the conditional probability P ( L , = Z(Y:,L,j1 s' E fi,) during each iteration. It is a local deterministic relaxation algorithm that is very easy to implement. We observe that in general any algorithm based on MRF models can be easily mapped onto neural networks with local interconnections. The main advantage of this deterministic relaxation algorithm is its simplicity. Often the solutions are reasonably good and the algorithm usually converges within 20-30 iterations. In the next section we study two stochastic schemes which asymptotically converge t o the global optimum of the respective criterion functions. 2.3. Stochastic Algorithms for Texture Segmentation
We look at two optimal solutions corresponding to different decision rules for determining the labels. The first one uses simulated annealing to obtain the optimum MAP estimate of the label configuration. The second algorithm minimizes the expected misclassification per pixel. The parallel network implementation of these algorithms is discussed in Section 2.3.3. 2.3.1. Searching f o r MAP solution
The MAP rule [15] searches for the configuration L that maximizes the posterior probability distribution. This is equivalent to maximizing P ( Y * I L) P(L) as P ( Y * )is independent of the labels and Y *is known. The right-hand side of (2.10) is a Gibbs distribution. To maximize (2.10) we use simulated annealing [41], a combinatorial optimization method which is based on sampling from varying Gibbs distribution functions e - + J p ( L , I Y:,L?,7-€N3) ZTk
In order to maximize
e - w L I Y')
z
1
Tk being the time varying parameter, is referred to as the temperature. We used the following cooling schedule Tk =
10
1
+ log,
lc .
(2.18)
258
R. Chellappa, R. L. Kashyap €9 B. 5’. Manjunath
where lc is the iteration number. When the temperature is high, the bond between adjacent pixels is loose, and the distribution tends to behave like a uniform distribution over the possible texture labels. As Tk decreases, the distribution concentrates on the lower values of the energy function which correspond to points with higher probability. The process is bound to converge to a uniform distribution over the label configuration that corresponds to the MAP solution. Since the number of texture labels is finite, convergence of this algorithm follows from [41]. In our experiment, we realized that starting the iterations with To = 2 did not guarantee convergence to the MAP solution. Since starting at a much higher temperature will slow the convergence of the algorithm significantly, we use an alternative approach, viz., cycling the temperature [43]. We follow the annealing schedule till Tk reaches a lower bound then we reheat the system and start a new cooling process. By using only a few cycles, we obtained results better than those with a single cooling cycle. Parallel implementation of simulated annealing on the network is discussed in Section 2.3.3. The results we present in Section 2.4 were obtained with two cycles. 2.3.2. Maximizing the posterior marginal distribution
The choice of the objective function for optimal segmentation can significantly affect its result. The choice should be made depending on the purpose of the classification. In many implementations the most reasonable objective function is the one that minimizes the expected percentage misclassification per pixel. The solution to the above objective function is also the one that maximizes the marginal posterior distribution of L,, given the observation Y * ,for each pixel s.
P{L,
= I,
I Y *= y*}
cc
c
P ( Y * = y* I L = 1) P ( L = 1)
lJL,=l,
The summation above extends over all possible label configurations keeping the label at site s constant. This concept was thoroughly investigated in [44]. Marroquin [55] discusses this formulation in the context of image restoration, and illustrates the performance on images with few gray levels. The possibility of using this objective function for texture segmentation is also mentioned. In [42] the same objective function is mentioned in the context of image estimation. To find the optimal solution we use the stochastic algorithm suggested in [44]. The algorithm samples out of the posterior distribution of the texture labels given the intensity. Unlike the stochastic relaxation algorithm, samples are taken with a fixed temperature T = 1. The Markov chain associated with the sampling algorithm converges with probability one to the posterior distribution. We define new random variables g$t for each pixel (s E a):
where Lt is the class of the s pixel, at time t , in the state vector of the Markov chain associated with the Gibbs sampler. The ergodic property of the Markov chain
2.2 Model-Based Texture Segmentation and Classification 259 [56] is used to calculate the expectations for these random variables using time averaging:
where N is the number of iterations performed. To obtain the optimal class for each pixel, we simply chose the class that occurred more often than the others. The MPM algorithm was implemented using the Gibbs sampler [41]. A much wider set of sampling algorithms such as Metropolis can be used for this purpose. The algorithms can be implemented sequentially or in parallel, with a deterministic or stochastic decision rule for the order of visiting the pixels. In order to avoid dependence on the initial state of the Markov chain, we can ignore the first few iterations. In the experiments conducted we obtained good results after five hundred iterations. The algorithm does not suffer from the drawbacks of simulated annealing. For instance we do not have to start the iterations with a high temperature to avoid local minima and the performance is not severely affected by enlarging the state space. 2.3.3. Network implementation of the sampling algorithms All the stochastic algorithms described in the Gibbs formulation are based on sampling from a probability distribution. The probability distribution is constant in the MPM algorithm [44] and is time varying in the case of annealing. The need for parallel implementation is due to the heavy computational load associated with their use. We now describe how these stochastic algorithms can be implemented on the network discussed in Section 2.2. The only modification required for the simulated annealing rule is that the neurons in the network fire according to a time dependent probabilistic rule. Using the same notation as in section 3, the probability that neuron (i, j,I ) will fire during iteration k is
where uijl is as defined in (2.16) and T k follows the cooling schedule (2.18). The MPM algorithm uses the above selection rule with T k = 1. In addition, each neuron in the network has a counter which is incremented every time the neuron fires. When the iterations are terminated the neuron in each column of the network having the maximum count is selected to represent the label for the corresponding pixel site in the image. 2.4. Experimental Results
The segmentation results using the above algorithms are given on two examples. The parameters ~1 and 01 corresponding to the fourth order GMRF for each texture
260
R. Chellappa, R. L. Kashyap & B. S. M a n j u n a t h
class were pre-computed from 64 x 64 images of the textures. The local mean (in an 11 x 11 window) was first subtracted to obtain the zero mean texture and the least square estimates [53] of the parameters were then computed from the interior of the image. The parameter values for the different textures used in our experiments is given in Table 1. Table 1. GMRF texture parameters.
$1
e2
e3 e4 e5 06
e7 08
e9 $10
2
calf
grass
pigskin
sand
wool
wood
0.5689 0.2135 -0.1287 -0.0574 -0.1403 -0.0063 -0.0052 -0.0153 0.0467 0.0190 217.08
0.5667 0.3780 -0.2047 -0.1920 -0.1368 -0.0387 0.0158 0.0075 0.0505 0.0496 474.72
0.3795 0.4528 -0.1117 -0.1548 -0.0566 -0.0494 -0.0037 0.0098 0.0086 0.0233
0.5341 0.4135 -0.1831 -0.2050 -0.1229 -0.0432 0.0120 0.0111 0.0362 0.0442
0.4341 0.2182 -0.0980 -0.0006 -0.0836 0.0592 -0.0302 -0.0407 0.0406 -0.0001
79.33
91.44
126.22
0.5508 0.2498 -0.1164 -0.1405 -0.0517 0.0139 -0.0085 -0.0058 -0.0008 0.0091 14.44
The first step in the segmentation process involves computing the Gibbs energies U1(Y*,IL,) in (2.5). This is done for each texture class and the results are stored. For computational convenience these U 1 ( . ) values are normalized by dividing by k2, where k is the size of the window. To ignore the boundary effects, we set U1 = 0 at the boundaries. We have experimented with different window sizes and larger windows result in more homogeneous texture patches but the boundaries between the textures are distorted. The results reported here are based on windows of size 11 x 11 pixels. We obtained w(Zs) by trial and error. The choice of ,B plays an important role in the segmentation process and its value depends on the magnitude of the energy function UI(.). Various values of ,B ranging from 0.2-3.0 were used in the experiments. In the deterministic algorithm it is preferable to start with a small ,B and increase it gradually. Large values of beta usually degrade the performance. We also observed that slowly increasing p during the iterations improves the results for the stochastic algorithms. It should be noted that using a larger value of p for the deterministic algorithm (compared to those used in the stochastic algorithms) does not improve the performance. The nature of the segmentation results depends on the order of the label model. It is preferable to choose the first order model for the stochastic algorithms if we know a przori that the boundaries are either horizontal or vertical. However for the deterministic rule and the learning scheme the second order model results in more homogeneous classification. The MPM algorithm requires the statistics obtained from the invariant measure of the Markov chain corresponding to the sampling algorithm. Hence it is preferable
2.2 Model-Based Texture Segmentation and Classification 261 to ignore the first few hundred trials before starting to gather the statistics. The performance of the deterministic relaxation rule of Section 2.2 also depends on the initial state and we have looked into two different initial conditions. The first one starts with a label configuration L such that L, = I , if Ul(Y,*IZ,) = minl,{Ul(Y,* I l b ) } . This corresponds to maximizing the probability P ( Y * I L) [12]. The second choice fur the initial configuration is a randomly generated label set. Results for both the cases are provided and we observe that the random choice often leads t o better results.
Example 1. This is a 256 x 256 image (Fig. 2(a)) having six textures: calf, grass, wool, wood, pigskin and sand. This is a difficult problem in the sense that three of the textures (wool, pigskin and sand) have almost identical characteristics and are not easily distinguishable even by the human eye. The ICM result obtained with the maximum likelihood estimate (MLE) as the initial condition is in Fig. 2(b). The MAP solution using simulated annealing is shown in Fig. 2(c). As mentioned before, cycling of temperature improves the performance of simulated annealing. The segmentation result was obtained by starting with a n initial temperature To = 2.0 and cooling according to the schedule (2.18) for 300 iterations. Then the system was reset to To = 1.5 and the process was repeated for 300 more iterations. In the case of the MPM rule the first 500 iterations were ignored and Fig. 2(d) shows the result obtained using the last two hundred iterations. As in the previous example the best results were obtained by the simulated annealing and MPM algorithms. For the MPM case there were no misclassifications within homogeneous regions but the boundaries were not accurate and in fact, as indicated in Table 2, simulated annealing has the lowest percentage error in classification. Table 2. Percentage misclassification for Example 1 (six class problem). Algorithm
Percentage Error
Maximum Likelihood Estimate Neural network (MLE as initial state) Neural network (Random initial state) Simulated annealing (MAP) MPM algorithm
22.17 16.25 14.74 6.72 7.05
3. Preattentive Segmentation In this section we discuss a simple biologically motivated approach to detect texture boundaries within a more general contex of boundary detection [45]. Previous approaches to this problem are discussed in [21,22]. In [21], Malik and Perona propose a three stage model involving convolution with even symmetric filters followed by half wave rectification, local inhibition, and texture boundary detection using odd symmetric filters. The BCS processes the intensity data and performs
262
R. Chellappa, R. L. Kashyap & B. S. M a n j u n a t h
Fig. 2. Texture segmentation results for a six class problem. (a) original image. Segmentation results using ICM, MAP and MPM are given in (b)-(d), respectively.
preattentive segmentation of the scene. The first stage of the BCS consists of oriented contrast filters at various scales and orientations and extracts the contrast information from the scene. The outputs of the filters are then fed to a two-stage competitive network whose main goal is to generate end-cuts. Subsequent long range cooperative interactions and a positive feedback to the competitive stage help in boundary completion. The boundary detection takes place independently in different spatial channels. The input image is first processed through a bank of orientation selective bandpass filters at various spatial frequencies. The convolution of the image with these filters yields a representation which is localized in space as well as in frequency. We then introduce three distinct types of local feature interactions for consideration: competition between spatial neighbors in each orientation channel, competition between orientations at each spatial location, and interscale interactions. Interscale interactions are used in localizing line ends and play an important role in boundary
2.2 Model-Based Texture Segmentation and Classification 263 detection. The second stage of interactions groups similar features in the neighborhood. This cooperative processing helps in the boundary completion process. The final step involves identifying image boundaries.
3.1. Gabor Functions and Wavelets Gabor functions are Gaussians modulated by complex sinusoids. Consider a wavelet transform where the basic wavelet is a Gabor function of the form gx(z, Y,0) = e-(X
2x’2+g’2)+i7Tx’
x‘ = xcosO + 1~sinO y’= -asin6+ycos6
where X is the spatial aspect ratio and 0 is the preferred orientation. To simplify the notation, we drop the subscript X and unless otherwise stated assume that X = 1. For practical applications, discretization of the parameters is necessary. The discretized parameters must cover the entire frequency spectrum of interest. Let the orientation range [0,7r]be discretized into N intervals and the scale parameter a be sampled exponentially as 02. This results in the wavelet family
where 81, = k.rr/N. The Gabor wavelet transform is then defined by
3.2. Local Spatial Interactions Following feature extraction using Gabor wavelets, we now consider local competitive and cooperative processing of these features. Competitive interactions help in noise suppression, and in reducing the effects of illumination. These interactions are modeled by non-linear lateral inhibition between features. Two types of such interactions are considered. The first type includes competition between spatial neighbors within each orientation and scale. The second type involves competition between different orientations at each spatial position. For simplicity the transfer function g(z) of all feature detectors is assumed to be the same. The following notation is used in explaining the int,eractions: The output of a cell at position s = (x,y) in the ith spatial frequency channel with a preferred orientation 0 is denoted by yZ(s, O), with I t ( s ,0) being the excitatory input t o that cell from the previous processing stage. For example, I,(s,O) could be the energy in the filter output corresponding to feature ( s ,-9) in the ith frequency channel. For convenience we will drop the subscript i indicating the frequency channel whenever there is no ambiguity. Let N, be the local spatial neighborhood of s. The
264
R . Chellappa, R. L. Kashyap & B. S. Manjunath
competitive dynamics is represented by:
where ( a , b, c ) are positive constants. In our experiments we have used a sigmoid non-linearity of the form h ( z ) = 1/(1+ exp(-,&)). The dynamics of (3.4) can be visualized as follows : At each location within a single frequency channel, the corresponding cell receives an excitatory input from a similarly oriented feature detector (of the same spatial frequency). Further it also receives inhibitory signals from the neighboring cells within the same channel. We assume that all these interactions are symmetric (bs,+t = bsI,+ and c g , v = ce,,e). The competitive dynamics of the above system can be shown to be stable. The Lyapunov function for the system [52] can be written as
Under the assumptions that the interactive synapses are symmetric and that g ( . ) is monotone non-decreasing, the time derivative of E is negative and the system represented by (3.4) always converges. 3.3. Local Scale Interactions We now suggest a simple mechanism to model the end-inhibition property of hypercomplex cells. Hypercomplex cells in the visual cortex differ from simple and complex cells in that they respond to small lines and line endings [57]. For this the hypercomplex cell receptive field must have inhibitory end zones along the preferred orientation. Such a profile can be generated either by modifying the profile of the simple cell itself or through interscale interactions, discussed below. The fact that both simple and complex cells often exhibit this end-stopping behavior further suggests that both these mechanisms are utilized in the visual cortex. If Qij(z,y, 8) denotes the output of an end-inhibited cell at position (2, y) receiving inputs from two frequency channels i and j (ai< a j ) with preferred orientation 0, then Q i j ( z , ~ I 0= )
h(llW(z,~,Q) -YW~(~,Y,~)II)
(3.7)
where y = ( ~ - ~ ( 2 - - 3 ) is the normalizing factor. The logic behind this is simple. At line ends, cells with shorter receptive fields will have a stronger response than those with larger fields, and consequently will be able t o excite the hypercomplex cells. At other points along the line, both small and large receptive field cells are equally
2.2 Model-Based Texture Segmentation and Classification 265 excited and in the process the response of the hypercomplex cells is inhibited. It appears that such scale interactions to generate end inhibition do exist in the visual cortex. Bolz and Gilbert [58] observe that connections between layers 6 and 4 in the cat striate cortex play a role in generating end inhibition. The cells in layer 4 are of hypercomplex type exhibiting end inhibition. Layer 6 cells have large receptive fields and require long bars (or lines) to activate them. In addition, cells in both layers show orientation selectivity. Inactivating layer 6 cells resulted in the loss of end-inhibition property of layer 4 cells, while preserving other properties such as orientation selectivity. Thus, in the absence of layer 6 activity, cells in layer 4 could be excited by short bars and their response did not decrease as the bar lengths increased, suggesting that layer 6 cells have an inhibitory effect on the cells of layer 4. 3.4. Grouping and Boundary Detection
The final stage involves grouping similar orientations. The grouping process receives inputs both from the competitive stage (3.4) and from the end detectors (hypercomplex cells) described in Section 3.3. Note that the orientation of the activating end-detector is orthogonal to the actual orientation of the grouping process. This incorporates the observation made in [59,60] that hypercomplex cells are responsible for detecting illusory contours. Abrupt line endings signal a n occluding boundary almost orthogonal to the edge orientation, and this is represented by these end-inhibited cells providing input t o the grouping process nearly orthogonal in their orientation preference. If Zi(s,8) represents the output of this process, then
ZZ(S,8) = h
(J
di(S -
s’, B)(yZ(sl,8)
+ Q i j ( S ’ , 8’)ds’ ) .
(3.8)
d i ( s , 8) represents the receptive field of Zi(s,19)and in our experiments we have used
+
d(s = (2,y), e) = e x p ( - ( 2 ~ ~ ) - ~ [coso ~ ~ ( zy sine)'
+ (-z
sin0
+ y C O S Q ) ~ ] ) (3.9)
where 8 is the preferred orientation, 8‘ is the corresponding orthogonal direction, and X is the aspect ratio of the Gaussian. The Z cells thus integrate the information from similar oriented cells within each frequency channel and from hypercomplex cells of appropriate orientation, and thus help in grouping the features and in boundary completion. Since the various frequency channels are sampled, the effective standard deviation of the Gaussian is a / & , where ai is the scale parameter for channel a. To summarize, this approach consists of three distinct steps: (a) feature detection using Gabor wavelets, (b) local interactions between features and (c) scale interactions to generate end inhibition. The output Z(.) from different frequency channels is now used to detect edges and texture boundaries.
3.5. Experiments The performance of our approach is illustrated on several images. The following parameter values were used in our experiments described here: p = 4.0 in the
266
R. Chellappa, R. L. K a s h y a p & B. S. M a n j u n a t h
transfer function g(.). The strengths of the inhibitory synapses in (3.4) are b,,s, = 1/ 11 n, 11 and c = 1/N, where 11 N , 11 is the cardinality of the neighborhood set and n is the number of discrete orientations used. Unless otherwise stated, N = 4 and N s consists of the four nearest neighbors of s. The aspect ratio of the Gaussian in both the Gabor wavelets (3.l),and in the receptive field of 2 cells (3.8) is set to 0.5. If more than one channel is mentioned then the result shown is a superposition of the boundaries detected in the individual channels. Regarding implementing the dynamics of competition, we used a simple gradient descent on the corresponding energy function (3.6) instead of solving the set of differential equations. The equilibrium points in general for these two methods will be different, but gradient descent on E in (3.6) will be much faster (typically it takes less than 50 iterations to converge on a 256 x 256 image).
Example 2 (Intensity edges). Figure 3 shows two examplesof edge detection using the energy measures. Figures 3(a) and (c) show the original 256 x 256 images. The edges shown in Figure 3(b) are detected in channels a' = [l/fi,1/21 and in (d) they correspond to the channel a' = 1/&. In both cases (T is set to 1. Example 3. Figure 4 shows the boundaries detected in an aerial image consisting of four textures, grass, water, wood and raffia. The wood texture is present a t two regions at different orientations. The parameter values used are a' = {1/2,1/2&, 1/4} and g = 5.0. Example 4. Figure 5 shows the results on a synthetic texture which is often used in psychophysical experiments. The boundary between L and T s is not easily perceived whereas that between straight and oriented Ts clearly stands out. This boundary can be easily detected in almost all frequency channels and the parameters values used are the same as in the previous example. Example 5. This example illustrates the importance of end inhibition in texture boundary detection. Figure 6 shows another commonly used texture consisting of randomly oriented Ls and +s. Unlike the previous example, orientation information can be used for segmentation. The line segments forming Ls and +s have the same length (seven pixels). The two regions differ in the distribution of corners, line-ends and interactions. As we discussed in Sec. 3.2.1, scale interactions play an important role in detecting these features. None of the scales by themselves contain enough information to segment the two regions, but using these interscale interactions the boundary between the Ls and +s can be detected (Fig. 6(b)). The boundary shown is for the case of using the interactions between scales corresponding t o {1/2,1/4} with u = 16. Example 6 (Illusory contours). The usefulness of scale interactions in detecting line endings and their subsequent grouping to detect illusory contours is illustrated in Figure 7. For the line (Fig. 7(d)) and sine wave (Fig. 7(e)) contours the results shown are for a' = {1/2,1/4},0 = 8. For the circle (Fig. 7(f)) a' = { l / f i , 1/2} and (T = 2.
2.2 Model-Based Texture Segmentation and Classification 267
Fig. 3. (a) and (c) show two 256 x 256 images and the corresponding edges detected are shown in (b) and (d). In (b) the edges are from two channels ai= {1/&, 1/2} and in (d) a2 = 1/*. For both examples u = 1.
4. Rotational Invariant Texture Classification
We discuss direct pattern classification strategies for classifying textures, assuming that there is only one texture in the image. The strategy is t o fit varieties of parametric random field models, extract features from them, and use these features for classification using both standard algorithms and new procedures.
4.1. Rotation Invariant Non-Causal A R Model The model used here is a modified version of a second-order non-causal autoregressive (NCAR) model, where the nearest eight neighbors are interpolated on a circle according to the following formula [61]:
2.2 Model-Based Texture Segmentation and Classification 269
+.
Fig. 6. Texture consisting of randomly oriented L and The boundary shown in (b) is detected using the output of the scale interactions with CT = 16. The scales used in this example are a i= {1/2,1/4}, and figures (c) and (d) show the result of convolution and (e) shows the output after the interactions.
The parameters (Y and p can be estimated by least-squares technique and the estimates 6, can be used as discriminating features. can be interpreted as a measure of the roughness of the texture. Classification experiments performed using only these two features showed that there is room for improvement. Textures like wood have a strong degree of directionality, not captured by & or A feature which measures the degree of directionality can be obtained by fitting to the image two different simultaneous autoregressive
a
6
b.
R. Chellappa, R. L. Kashyap tY B. S. Manjunath
270
._
..........
~
._
..
................_
_
....
........................
::_I
I
........... ._ .
-
i
( Fig. 7. Some examples of illusory contours formed by line terminations (a), (b), and ( c ) , and the corresponding detected contours (d), (e) and (f).
(SAR) models [31,62,61],having the following form: Y(S) =
C 8rY(s
r ) + fiw(s)
TEN
where 8, = O-, and w(riz) is (0,l) an identical, and independently distributed (IID) sequence. N is a neighbor set excluding (0,O). Let us first choose the neighbor set N to consist of four nearest neighbors, namely N , = [(0, l),(0, -l), (1,0), (-1, O)]. Let 8io,l)and 8rl,o)be the ML estimates of 8 ( 0 , ~ and ) 8(l,o). Next, let us fit another SAR model with neighbor set Nb having the four nearest diagonal members, namely
2.2 Model-Based Texture Segmentation and Classification 271 be the ML estimates Nb = [(I,I), (1, -l), (-1, I ) , (-1, -1)]. Let 6Tl,l) and of the corresponding parameters. Consider the feature [ defined as
i measures the extent of variation in the orthogonal directions. For a texture having strong directionality like wood or straw, i will be very large. From a directionless texture like sand, it will be very small. Thus, our feature set is (8,a,i). A supervised recognition approach is used for the classification of textures. The inputs to the system are the digitized images from one of the m texture classes. The images are separated into test and training sets. The class of textures in the training set is known a priorz. In the feature selection state, 8, ,6 and [ are extracted from the processed images. The class parameterization phase computes the sample mean and standard deviation of each category training feature. The classifier is a distance classifier which measures a weighted distance between the features of the test image denoted by X ( t ) = [8(t),/?(t),((t)] and the mean feature of each of the m classes. The texture is then classified to class C: for which such a distance is minimum, i.e.
i*
= mind(X(t),i),i =
1 , ... , m ,
9
where
and f(i) and [a;](i)correspond to the sample mean and variance of class (i) feature, obtained from the training set, respectively. 4.2. Experiments
Twelve different textures, namely, calf leather (D24), wool (D19), beach sand (D29), pigskin (D92), plastic bubbles (D112), herringbone weave (D17), raffia (D84), wood grain (D68), grass (D9), straw (D15), brick wall (D95) and bark of tree (D12) were chosen from the photo album by Brodatz [63]. This selection includes both macrotextures (e.g. brick wall) and microtextures (e.g. sand). Seven rotated 512 x 512 - 8 bit (0-225) digitized images with relative angles of rotation of Odeg, 30deg, 6Odeg, 90deg, 120deg, 150deg, and 200deg are taken from each class of texture. Each 512 x 512 image was first reduced t o a 128 x 128 one by averaging every 4 x 4 window into a single pixel. Each 128 x 128 image is then segmented into four 64 x 64 images. Thus the database has 28 64 x 64 images from each texture. One 64 x 64 digitized window of Odeg orientation of each texture is shown in Fig. 8. Figure 9 shows a 64 x 64 sample of raffia texture for all of the seven orientations.
272
R. Chellappa, R. L. Kashyap & B. S. Manjunath
Fig. 8. A 64 x 64, Odeg digitized sample of each texture of the database. From left to right, first row: calf leather, wool, sand; second row: pigskin, plastic bubbles, herringbone weave; third row: raffia, wood, grass; fourth row: straw, brick wall, bark of tree.
Fig. 9. A 64 x 64 digitized sample of each of the seven orientationsof raffia texture. From top and from left to right: Odeg, 30deg, GOdeg, 9Odeg, l20deg, 150deg, and 200deg.
To remove the variability in the image caused by illumination or quantization schemes, all the 64 x 64 images were first subjected to a gray scale normalization procedure and then normalized so that each image has zero empirical mean and unit empirical variance. To illustrate the discriminating power of each individual feature, a range plot is presented for each of them in Fig. 10. The classes are first ordered for each feature according to the mean value of the respective features. Then the actual range of values that each feature takes for each category is plotted. Underneath
2.2 Model-Based Texture Segmentation and Classification 273 Raff la Pigskin
Sand
L Y c
Tree Brick
-
Bubbles
m-a c
VaOl
w
Wood Grass Leather Strau
-
-
Herring
Y
Lcat her
Snd
-
Strw
I
Raffia
cc w
Wood
c_Lc_
. . I
Piqskin
,130
--M-U
a13
.sin
.3+)
.*m
.uo .un
moo
.7ia1
82
(b) Fig. 10. (a) Range plots of h. (b) Range plots of *U.
p.
x: mean value.
0:
range extrema. *: mean
274
R. Chellappa, R. L. Kashyap & B. S. Manjunath
each range plot the distance corresponding t o two empirical standard deviations of the feature for that texture is also given. These range plots indicate how packed each respective feature is. The amount of vertical overlap between category range plots is an indication of the classification power of the causing feature. The less such an overlap, the better the feature. On the average, the & values of each texture class overlap with & values of four other classes. Raffia texture is an exception with very distinct &. Herringbone and leather textures have very distinct features while the of the rest of the categories overlap with an average of three other classes. Note that the mean value of ,6 for highly circularly nonsymmetric features like herringbone is much higher than the mean value for plastic bubbles which has more of a circular symmetry property. Several experiments were carried out [64] and only one is described here. Recall that we have 28 images with 7 orientations for each texture. The classifier for each texture is trained on samples from the images of three orientations (i.e. 12 images) and the classifier is tested using all other images (i.e. 16 images of that texture and 28 of all other textures). Thus the classifier has not “seen” the orientations it encounters in the test phase. The results are presented in Table 3. A total of ten experiments were carried out and the average classification accuracy obtained is 89 percent.
a
a
Table 3 . Classification results for 12 classes in the database using (&,fJ,() feature vector. In each experiment the available 28 samples from each class are divided into 12 training and 16 test samples. Angle of rotation (degree) Training samples 0, 30, 60 30, 60, 90 60, 90, 120 90, 120, 150 120, 150, 200 0, 60, 120 30, 90, 150 0, 90, 200 0, 150, 200 30. 150. 200
Testing samples 90, 120, 150, 200 0, 120, 150, 200 0, 30, 150, 200 0, 30, 60, 200 0, 30, 60, 90 30, 90, 150, 200 0, 60, 120, 200 30, 60, 120, 150 30, 60, 90, 120 0. 60. 90. 120
Classification accuracy rate 87% 88% 88% 89% 86% 91% 90% 90% 91% 90%
AVERAGE
89%
5 . Classification Using Fractional Models
A multi-level classification method which can handle arbitrary 3-D rotated samples of textures is developed based on fractional differencing models with a fractal scaling parameter. In the first level of classification, the textures are classified by the first-order F’ractional Differencing model with a fractal scale parameter, and in the second level, classification is completed with the additional frequency
2.2 Model-Based Texture Segmentation and Classification 275 parameters of the second-order Fractional Differencing periodic model. This multilevel classification scheme has several advantages over the conventional approaches [31-391.
5.1. Fractional Difference Models (FDM) The Fractional Difference model in one dimension is the discrete version of the continuous fractional Brownian motion process (FBM) introduced by Mandelbrot and Van Ness [65]. The FBM differs from the GMRF models introduced earlier in two respects, namely (i) its correlation function decays with lag much slower than that in the parametric GMRF models, and (ii) it has considerable power at low frequencies and can account for large periodicities unlike the GMRF models which have little power a t low frequencies. The FDM possesses both these properties possessed by FBM. In many images, widely separated image pixels seem to display relatively high degrees of correlation. We will first generalize the first-order FDM given in [65] for two dimensions as follows [66]: y(ml,m2)=
[(l- z1-1 )(1- ~ ; ~ ) ] - ‘ / ~ [ ( m 1 , m 2 )ml ,
, m2 = 0 , 1, . . . , N
-
1 (5.1)
where z1,z2 are the unit lead variables in the two-dimensions, c is the fractional parameter, 0 < c < 1, and S(riz,rjz.) is a two-dimensional independent, identically distributed sequence of random variables with zero mean and finite variance p . The above model is stationary even though it has a zero unit circle. By taking the factor (1 - ~ ; l ) - ‘ / ~ ( l - Z ; ~ ) - ‘ / ~ on the left-hand side and expanding it in an infinite power series, one can interpret the above model as an infinite order (non-Markov)autoregressive model in two dimensions. The above model has only two “tunable” parameters, c and p . Sometimes they are not enough to provide the level of classification. Then we go to the second-order FDM given below [66]: y(ml,m2) = [(l- 22L1 cosw1
+ zL2)(1 - 2 2 i 1 C O S W ~+ ~ , ~ ) ) - “ / ~ ] ] r ( m l(5.2) ,m2).
(cosw1) and ( C O S Ware ~ ) the two additional parameters. In addition, we can choose different scaling parameters c1, c2 in the two directions instead of one parameter c. The corresponding DFTs of these functions are
Y ( k ,k2)
=
“1 - e
-ja *N1)(I -
e - j 2 ” % ) ] - ~ / 2 ~ ( k lI ,C ~ ) ,
and
+
Y ( / ~ , I c=~ [(I ) - 2 c o s w 1 e - j 2 ~ 3 e - j 4 a 3 11 where zi is the delay operator associated with mi, [(ml,m2) is an IID Gaussian sequence, and W(lc1,IC2) is the corresponding DFT. A key property of both these models is that the structure of their DFT given above and the associated parameters are unaltered even if the images are rotated,
276
R. Chellappa, R. L. Kashyap tY B. S. Manjunath
tilted and slanted. The details are in [67]. For any given image, the parameters c and p in the first-order model and the parameters c, w1,w2, p in the second-order model can be estimated [67,68]. 5.2. Multi-level 3 - 0 Rotational Invariant Classification Scheme For this classification scheme, the images are separated into test and training sets. The class of textures and the number of classes in the training set is assumed to be known a priori. In the first level, the different 3-D rotated texture images are classified into M different classes depending on their estimated values of the fractal scale. Actual classification is achieved by applying a distance classifier d(c, i), which measures a weighted distance between the extracted feature of the test image denoted by 2 and the mean feature of each of M classes. Then the texture is classified to class Ai for which such a distance is minimum. That is.
i*
2 .
= min[C - Ci] z = i
1 , . . ., M
where ei corresponds to the sample mean of feature c in class Ai. The class Ai can consist of several different texture classes since several different textures share the same fractal scale (the roughness of the surface). This means that the fractal scale only is not enough to distinguish the different textures. Thus, we need an additional classification scheme to distinguish textures contained in the same class Ai. In the second level, the textures which were already classified to the same class in the first-level are split t o the different subclasses, based on the values of pattern features w1, wz in the second-order fractional differencing periodic function (5.2).
and f ( i ) and [o!](~) correspond to the sample mean and variance of subclass (k) features, respectively. Here, it should be noticed that since we have at most several subclasses from a first-level class, we need to compare only a small number of subclasses to complete the classification, instead of checking the feature distance of whole other texture classes. 5 . 3 . Experiments
For these experiments, nine different classes of texture were taken from Brodatz’s standard texture album for the training set. These are, namely, grass [D9],tree bark [D12],straw [D15], herringbone weave [D17], woolen cloth [D19], calf leather [D24], beach sand [D29], water [D37], and raffia [D84].
2.2 Model-Based Texture Segmentation and Classification 277 Table 4. The sample mean and variance of parameters c, w 1 , w 2 : 16 64 x 64 sample image data are taken for each different texture classes, and the parameter values are extracted from the first and second-order fractional differencing models. ~
Textures
C
-
2
grass tree bark straw herringbone weave woolen cloth calf leather beach sand water raffia
w1
U2
-
w2
X
U2
X
0.636 0.601 1.209 1.175 0.793 0.935 0.571 0.972 0.988
1.209 1.530 0.923 1.003
0.057 0.073 0.053 0.072
0.744 0.691 0.387 1.263
0.078 0.199 0.068 0.114
0.809 1.064 1.195 1.074 1.547
0.024 0.044 0.038 0.055 0.062
0.852 1.175 0.665 0.083 1.042
0.095 0.114 0.107 0.064 0.153
62 __ 0.082 0.324 0.070 0.167
0.098
0.122 0.129 0.132 0.165
For the actual training, 16 64 x 64-sized sample image data were taken for each different texture pattern, and the sample mean and variance of parameters, c, w1, and w2 were obtained for each texture class, based on the first- and second-order fractional differencing models (Table 4). As we can see from Table 5, fractal scale c itself is not enough to classify the different textures, because some of the textures have similar values of c, even though they are different texture patterns. Based on these sample mean and variance values of the parameters c, nine textures are grouped into five classes as indicated in Table 6, which also indicates the corresponding sample mean and variance of each class. Notice that the herringbone weave texture belongs to classes 2 and 3, because of its high value of variance. The second level gives the recognized texture from each class.
2-D rotated texture case. In this experiment, the test input images were taken from the 2-D raffia textures rotated by various angle 0s. Then, each 64 x 64 texture was classified by the proposed multi-level classification scheme. For the first level, the fractal scale parameter c was extracted based on the first-order Fractional Differencing model (5.1), and the parameters w1 and w2 were extracted from the second-order Fractional Differencing periodic model (5.2). Actual classification of the test images was done in each level by comparing weighted distances between the extracted features and the data base. The classification results are presented in Table 7, which shows the parameter values extracted from each rotated texture pattern and demonstrates the perfect result of classification based on these values. Rotated and projected texture case. In this experiment, six 64 x 64 test input images were taken from the straw textures rotated and projected orthographically from the various tilted and slanted texture surfaceslike in previous experiments, for the first level, the fractal scale parameter c was extracted based on the firstorder Fractional Differencing model (5.1), and the parameters, w1 and w2, were
278
R. Chellappa, R. L. Kashyap €9 B. S. M a n j u n a t h
Table 5. Database of the first level of classification. 4 and variance of class i, respectively.
are the sample mean and the
Class
Textures
Ci
Ul
1
woolen cloth
0.809
0.024
2 3 4
straw, herringbone weave herringbone weave, calf leather, water grass, beach sand tree bark, raffia
0.963 1.047 1.202 1.539
0.063 0.055 0.045 0.067
5
Table 6. Classification results from the 2-D rotated texture images. (Result indicates the result class after applying two-level classification method.) Andes
2
Gl
G,
Result
20 40 60 80 100 120 140 160 180
1.523 1.517
1.132 1.144 1.138 1.142 1.139 1.138 1.135 1.140 1.133
1.098 1.102 1.119 1.120 1.118 1.120 1.097 1.113
raffia raffia raffia raffia raffia raffia raffia raffia raffia
1.535 1.537 1.532 1.529 1.527 1.533 1.525
1.099
Table 7. Classification results from the rotated and orthographically projected straw texture images. (Result indicates the result class after applying two-level classification method.) Angles
0 = Odeg, T 0 = 45deg, T 0 = 90deg, T 0 = Odeg, T 0 = 45deg, T 0 = 9Odeg, T
= Odeg, u = Odeg, u = Odeg, u = 45deg, u = 45deg, u = 45deg, u
= 15deg = 30deg
= 45deg = 15deg = 30deg = 45deg
C
bl
G2
Result
0.914 0.932 0.928 0.918 0.922 0.927
0.365 0.371 0.373 0.368 0.375 0.377
1.189 1.224 1.218 1.156 1.191 1.202
straw straw straw straw straw straw
extracted from the second-order F'ractional Differencing periodic model (5.2). The classification results from this experiment are presented in Table 7. Table 7 shows the parameter values extracted from each rotated and projected texture pattern and demonstrates the perfect result of classification based on these values. 6. Summary
In this chapter we presented a number of techniques for texture segmentation and classification. Although significant progress has been made over the last thirty years, completely automated, unsupervised texture segmentation and classification
2.2 Model-Based Texture Segmentation and Classification 279 algorithms that are invariant to transformations such as rotation scaling, illumination, etc. remain elusive.
References [l] R. M. Haralick, Statistical and structural approaches to textures, Proc. I E E E 67 (1979) 786-804. [2] G. H. Landerweerd and E. S. Gelsema, The use of nuclear texture parameters in the automatic analysis of leukocytes, Pattern Recogn. 10 (1978) 57-61. [3] M. Nagao and T. Matsuyama, A Structural Analysis of Complex Aerial Photographs (Plenum Press, New York, 1980). [4] P. C. Chen and T. Pavlidis, Segmentation by texture using correlation, I E E E Trans. Pattern Anal. Mach. Intell. 5 (1983) 64-69. [5] J. Weszka, C. R. Dyer and A. Rosenfeld, A comparative study of texture measures for terrain classification, I E E E Trans. Syst. Man Cybern. 6 (1976) 269-285. [6] K. Laws, Textured image segmentation, Ph.D. Thesis, University of Southern California, 1978. [7] A. Ikonomopoulos and M. Unser, A directional filtering approach to texture discrimination, in Proc. 7th Int. Conf. on Pattern Recognition, Montreal, Canada, Jul. 1984, 87-89. [8] A. P. Pentland, Fractal-based descriptions of natural scenes, IEEE Trans. Pattern Anal. Mach. Intell. 6 (1984) 661-674. [9] S. W. Zucker and D. Terzopoulos, Finding structure in co-occurrence matrices for texture analysis, in Azriel Rosenfeld (ed.), Image Modeling (Academic Press, New York, 1981) 423-445. [lo] C. W. Therrien, An estimation-theoretic approach to terrain image segmentation, Comput. Vision Graph. Image Process. 22 (1983) 313-326. [ll] F. S. Cohen and D. B. Cooper, Simple parallel hierarchical and relaxation algorithms for segmenting noncausal Markovian fields, IEEE Trans. Pattern Anal. Mach. Intell. 9 (1987) 195-219. [12] S. Chatterjee and R. Chellappa, Maximum likelihood texture segmentation using Gaussian Markov random field models, in Proc. Computer Vision and Pattern Recognition Conf., San Francisco, CA, Jun. 1985. [13] H. Derin and H. Elliott, Modeling and segmentation of noisy and textured images using Gibbs random fields, IEEE. Trans. Pattern Anal. Mach. Intell. 9 (1987) 39-55. [14] P. B. Chou and C. M. Brown, Multi-model segmentation using Markov random fields, in Proc. Int. Joint Conf. on Artificial Intelligence, Seattle, WA, 1987, 663-670. [15] S. Geman and C. Graffigne, Markov random fields image models and their application to computer vision, in A. M. Gleason (ed.), Proc. Int. Congress of Mathematicians 1986, Providence, RI, 1987. [16] Z. Fan and F. S. Cohen, Textured image segmentation as a multiple hypothesis test, IEEE Trans. Circuits and Syst. 35 (1988) 691-702. [17] B. S. Manjunath, T. Simchony and R. Chellappa, Stochastic and deterministic networks for texture segmentation, I E E E Trans. Acoust. Speech Signal Process. 38 (1990) 1039-1049. [18] B. S. Manjunath and R. Chellappa, A note on unsupervised texture segmentation, I E E E Trans. Pattern Anal. Mach. Intell. 13 (1991) 472-483. [19] B. Julesz, Visual pattern discrimination, I R E Trans. Inf. Theory 8 (1962) 84-92. [20] J. R. Bergen and E. H. Adelson, Early vision and texture perception, Nature 333 (1988) 363-364.
280
R. Chellappa, R. L. Kashyap & B. S. Manjunath
[21) J . Malik and P. Perona, Preattentive texture discrimination with early vision mechaAm. A 7 (1990) 923-932. nisms, J. Opt. SOC. [22] S. Grossberg and E. Mingolla, Neural dynamics of surface perception: Boundary webs, illuminants, and shape-from-shading, Comput. Vision Graph. Image Process. 37 (1987) 116-165. 1231 L. S. Davis, M. Clearman and J. K. Aggarwal, An empirical evaluation of generalized co-occurrence matrices, IEEE Trans. Pattern Anal. Mach. Intell. 3 (1981) 214-221. [24] D. Chetverikov, Experiments in the rotation-invariant texture discrimination using anisotropy features, in Proc. 6th Int. Conf. on Pattern Recognition, Munich, Germany, Oct. 1982, 1071-1073. [25] L. S. Davis, Polograms: A new tool for image texture analysis, Pattern Recogn. 13 (1981) 219-223. [26] 0.D. Faugeras and W. K. Pratt, Decorrelation methods of texture feature extraction, IEEE Trans. Pattern Anal. Mach. Intell. 2 (1980) 323-332. [27] F. Vilnrotter, Structural analysis of natural textures, Ph.D. Thesis, University of Southern California, 1981. [28] B. J. Schacther, A. Rosenfeld and L. S. Davis, Random mosaic models for textures, IEEE Trans. Syst. Man Cybern. 9 (1978) 694-702. [29] N. Ahuja and A. Rosenfeld, Mosaic models for textures, I E E E Trans. Pattern Anal. Mach. Intell. 3 (1981) 1-11. [30] J. W. Modestino, R. W. Fries and A. L. Vickers, Texture discrimination based upon an assumed stochastic texture model, I E E E Trans. Pattern Anal. Mach. Intell. 3 (1981) 557-580. [31] R.L. Kashyap, R. Chellappa and A. Khotanzad, Texture classification using features derived from random field models, Pattern Recogn. Lett. 1 (1982) 43-50. [32] P. M. Lapsa, New models and techniques for synthesis, estimation and segmentation of random fields, Ph.D. Thesis, Purdue University, 1982. [33] R. Chellappa and S. Chatterjee, Classification of textures using Gaussian-Markov random fields, IEEE Trans. A c o u t . Speech Signal Process. 33 (1985) 959-963. [34] A. Khotanzad and R. L. Kashyap, Feature selection for texture recognition based on image synthesis, I E E E Trans. Syst. Man Cybern. 17 (1987) 1087-1095. [35] P. DeSouza, Texture recognition via autoregression, Pattern Recogn. 15 (1982) 471475. (361 R. L. Kashyap and K.-B. Eom, Texture boundary detection based on the long correlation model, IEEE Trans. Pattern Anal. Much. Intell. 11 (1989) 58-67. [37] J. Zhang and J. W. Modestino, Markov random fields with applications to texture classification and discrimination, Conf. on Information Sciences and Systems, Princeton, NJ, 1986. [38] S. Chatterjee, Classification of natural texture using Gaussian Markov random field models, in R. Chellappa and A. K. Jain (eds.), Markov Random Fields: Theory and Application (Academic Press, 1992). [39] C. Chen, J. S. Daponte and M. D. Fox, Fractal feature analysis and classification in medical imaging, I E E E Trans. Medical Imaging 8 (1989) 133-142. [40] R. L. Kashyap and Y. Choe, Multilevel 3-D rotation invariant classification, in Proc. 11th Int. Conf. on Pattern Recognition, The Hague, Sept. 1992. [41] S. Geman and D. Geman, Stochastic relaxation, Gibbs distributions, and Bayesian restoration of images, IEEE Trans. Pattern Anal. Mach. Intell. 6 (1984) 721-741. [42] J. Besag, On the statistical analysis of dirty pictures, J. Roy. Statist. SOC.B48 (1986) 259-302. 1431 U . Grenander, Lectures in Pattern Theory (Springer-Verlag, New York, 1981).
2.2 Model-Based Texture Segmentation and Classification 281 [44] J. L. Marroquin, Probabilistic solution of inverse problems, Ph.D. Thesis, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1985. (451 B. S. Manjunath and R. Chellappa, A unified approach to boundary perception: Edges, textures and illusory contours, IEEE Trans. Neural Networks 4 (1992). [46] T. Poggio, V. Torre and C. Koch, Computational vision and regularization theory, Nature 317 (1985) 314-319. [47] Y. T. Zhou, R. Chellappa, A. Vaid and B. K. Jenkins, Image restoration using a neural network, IEEE Trans. Acoust. Speech Signal Process. 36 (1988) 1141-1151. [48] Y. T. Zhou and R. Chellappa, Stereo matching using a neural network in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing, New York, NY, Apr. 1988, 940-943. [49] C. Koch, J. Luo, C. Mead and J. Hutchinson, Computation motion using resistive networks, in D. Z. Anderson (ed.), Proc. Neural Information Processing Systems, Denver, CO, 1987. [50] Y. T. Zhou and R. Chellappa, Computation of optical flow using a neural networkjn Proc. IEEE Int. Conf. on Neural Networks, San Diego, CA, Jul. 1988, 71-78. [51] H. Bulthoff, J. Little and T. Poggio, A parallel algorithm for real-time computation of optical flow, Nature 337 (1989) 549-553. 1521 J. J. Hopfield and D. W. Tank, Neural computation of decisions in optimization problems, Biol. Cybern. 52 (1985) 114-152. [53] R. Chellappa, Two-dimensional discrete Gaussian Markov random field models for image processing, in L. N. Kana1 and A. Rosenfeld (eds.), Progress in Pattern Recognition 2 (Elsevier Science Publishers, North Holland, 1985) 79-112. [54] G. R. Cross and A. K. Jain, Markov random field texture models, IEEE Trans. Pattern Anal. Mach. Intell. 5 (1983) 25-39. [55] J. Marroquin, S. Mitter and T. Poggio, Probabilistic solution of ill-posed problems in computer vision, in Proc. Image Understanding Workshop, Miami Beach, FL, Dec. 1985, 293-309. (561 B. Gidas, Non-stationary Markov chains and convergence of the annealing algorithm, J. Stat. Phys. 39 (1985) 73-131. [57] D. H. Hubel and T. N. Wiezel, Functional architecture of macaque monkey visual cortex, in Proc. Royal SOC.of London, Ser. B 198 (1977) 1-59. 1581 J. Bolz and C. D. Gilbert, Generation of end-inhibition in the visual cortex via interlaminar connections, Nature 320 (1986) 362-365. 1591 E. Peterhans and R. von der Heydt, Mechanisms of contour perception in monkey visual cortex. 11. Contour bridging gaps, J. Neuroscience 9 (1989) 1749-1763. 1601 R. von der Heydt and E. Peterhans, Mechanisms of contour perception in monkey visual cortex. I. Lines of pattern discontinuity, J. Neuroscience 9 (1989) 1731-1748. [61] R. L. Kashyap and R. Chellappa, Estimation and choice of neighbors in spatial interaction models, IEEE Trans. Inf. Theory 29 (1983) 6G72. [62] R. L. Kashyap and K.-B. Eom, Robust image models and their applications, in P. Hawkes (ed.), Advances in Electronics and Electron Physics, Vol. 70 (Academic Press, 1988) 79-158. [63] P. Brodatz, Texture: A Photographic Album f o r Artists and Designers (Dover, New York, 1956). [64] R. L. Kashyap and A. Khotanzad, A model-based method for rotation invariant texture classification, IEEE Trans. Pattern Anal. Mach. Intell. 18 (1986) 472-481. [65] B. B. Mandelbrot and J. W. Van Ness, Fractional Brownian motions, fractional noises and applications, SIAM Rev. 10 (1968) 422-437.
282
R. Chellappa, R. L. Kashyap & B. S. Manjunath
[66] R. L. Kashyap and P. M. Lapsa, Synthesis and estimation of random fields using long correlation models, IEEE Trans. Pattern Anal. Mach. Intell. 6 (1991) 800-808. [67] Y . Choe and R. L. Kashyap, 3-D shape from a shaded and textural surface image, ZEEE Tmns. Pattern Anal. Mach. Intell. 13 (1991) 907-918. [68] R. L. Kashyap and K.-B. Eom, Estimation in long-memory time series model, J. Time Series Analysis 9 (1988) 35-41.
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 283-312 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
I CHAPTER 2.3 I COLOR IN COMPUTER VISION: RECENT PROGRESS
GLENNHEALEY Electrical and Computer Engineering, University of California, Iruine, CA 92697, USA E-mail:
[email protected] QUANG-TUAN LUONG Artificial Intelligence Center, SRI International, 333 Rauenswood Awe., Menlo Park, CA 94025, USA E-mail:
[email protected] The use of color in computer vision has received growing attention. This chapter introduces the basic prinrjples underlying the physics and perception of color and reviews the state-of-the-art in color vision algorithms. Parts of this chapter have been condensed from [58] while new material has been included which provides a critical review of recent work. In particular, research in the areas of color constancy and color segmentation is reviewed in detail. The first section reviews physical models for color image formation as well as models for human color perception. Reflection models characterize the relationship between a surface, the illumination environment, and the resulting color image. Physically motivated linear models are used to approximate functions of wavelength using a small number of parameters. Reflection models and linear models are introduced in Section 1 and play an important role in several of the color constancy and color segmentation algorithms presented in Sections 2 and 3. For completeness, we also present a concise summary of the trichromatic theory which models human color perception. A discussion is given of color matching experiments and the CIE color representation system. These models are important for a wide range of applications including the consistent representation of color on different devices. Section 1 concludes with a description of the most widely used color spaces and their properties. The second section considers progress on computational approaches to color constancy. Human vision exhibits color constancy as the ability to perceive stable surface colors for a fixed object under a wide range of illumination conditions and scene configurations. A similar ability is required if computer vision systems are to recognize objects in uncontrolled environments. We begin by reviewing the properties and limitations of the early retinex approach to color constancy. We describe in detail the families of linear model algorithms and highlight algorithms which followed. Section 2 concludes with a subsection on recent indexing methods which integrate color constancy with the higher level recognition process. Section 3 addresses the use of color for image segmentation and stresses the role of image models. We start by presenting classical statistical approaches to segmentation which have been generalized to include color. The more recent emphasis on the use of physical models for segmentation has led to new classes of algorithms which enable the
283
284
G. Healey €4 Q.-T. Luong accurate segmentation of effects such as shadows, highlights, shading, and interreflection. Such effects are often a source of error for algorithms based on classical statistical models. Finally, we describe a color texture model which has been used successfully as the basis of an algorithm for segmenting images of natural outdoor scenes. Keywords: Color, computer vision, modeling, reflectance, color constancy, multispectral, segmentation, recognition, physics-based vision, intrinsic properties, features.
1. Modeling This section introduces physical and perceptual models for color. Much of the work based on the physics of color image formation is covered in more detail in an edited collection of papers published in 1992 [38]. More details on perceptual color models are given in the reference books by Wyszecki and Stiles [89] and Judd and Wyszecki [47]. The first book focuses on a few topics a t great depth and contains a large amount of reference data. The second book is more recent and covers more topics with a practical approach. An interesting book on the physics and chemistry of color formation is [76]. 1.1. Physical Color Models 1.1.1. Sensing the color signal The reflected spectral radiance of a surface point r in the direction of a n observer can be expressed as
where ( i ,e, 9 ) are the photometric angles of incidence, observation, and phase (Fig. l), E(A,r) is the spectral power distribution of the incident illumination a t point r, and p ( i , e, 9 , A, r) is the spectral reflectance of the surface at point r in the direction of the observer. The reflected radiance I ( A ,r) over the visible wavelengths (400 nm-700 nm) is an example of a color signal. In general, the term color signal refers to any spectral power distribution of light.
Fig. 1. Photometric angles.
2.3 Color in Computer Vision: Recent Progress 285 The reflected spectral radiance function I ( X , r) is not represented explicitly by a color imaging system. Instead, the system stores scalars
where (s,y) is the image location corresponding t o scene point r and f j ( X ) is the spectral sensitivity of the j t h sensor. The image sj(z, y) formed using a single f j ( X ) is called a color band. A color camera system uses a set of three color filters with different spectral sensitivities to capture three color bands which represent the red, green, and blue components of the scene. Various technical solutions include: Black-and-white camera with colored optical filters such as the Kodak Wratten 25, 58, and 47B.a Single CCD color camera. A striped or mosaic color filter pattern is affixed to a single CCD so that different CCD cells have different effective spectral sensitivities. This is an inexpensive approach, but sacrifices spatial resolution in each color band. Three CCD color camera. A high precision prism makes three copies of the incoming image which are independently filtered and sensed by three separate CCDs. There is no loss of spatial resolution. Some of the challenges of obtaining accurate color images are presented in [65]. The properties of CCD cameras are reviewed in [37]. 1.1.2. Reflection models Color reflection models play the important role of describing the relationship between the properties of an object and its image. These models are exploited by several of the color constancy and segmentation algorithms presented later in this chapter. Reflection can be modeled as two distinct physical processes (Fig. 2). Surface (interface) reflection describes light which is reflected directly from a surface. Body scattering refers to light which penetrates some depth into a material and emerges following scattering by inhomogeneities. General physical models predict that spectral reflectance is a complicated function of wavelength and geometry [30]. Consequently, an important goal of computer vision is the selection of models which are both accurate and tractable. The dichromatic reflection model [78] characterizes the reflected radiance from any point on an inhomogeneous dielectric material as the sum of a surface reflection aThe transmission curves are in the first edition of the book by Wyszecki and Stiles [89]. They can also be obtained from Kodak.
286
G. Healey 6 Q.-T. Luong
Incident light
Surface component
Body component
Fig. 2. Surface and body components of reflection.
term and a body reflection term according to
where 0 denotes the photometric angles. The relationship in (1.4) assumes that both the surface and body reflection components factor into the product of a term which depends only on geometry and a term which depends only on wavelength. The spectral composition terms c, (A) and c, (A) depend on the spectral properties of the illumination and the spectral properties of the reflectance functions
where p,(A) and pB (A) describe the dependence of the surface and body reflectance on wavelength and E(X) is the spectral power distribution of the illumination. Healey [30] used the Torrance-Sparrow model [85] for surface reflection and the Reichman extension [74] of the Kubelka-Munk model [47] for body scattering to show that the dichromatic model is accurate for a wide range of inhomogeneous dielectric materials. He further showed that the reflected radiance for metals can be modeled using
I ( @ ,A) = m ( @ ) c ( A )
(1.7)
over a wide range of geometries. A common extension to the dichromatic reflection model for inhomogeneous dielectric materials is the neutral interface reflection (NIR) model. The NIR model assumes that p s ( A ) has a constant value independent of wavelength. This implies that the surface reflection (highlight) from inhomogeneous dielectrics has the same
2.3 Color in Computer Vision: Recent Progress 287 spectral composition as the illumination. The NIR model has been verified experimentally by Tominaga and Wandell [83] and Lee et al. [57]. 1.1.3. Linear models Linear models represent functions of wavelength from 400 nm to 700 nm as an additive combination of a small number of basis functions. The functions that are typically represented are the spectral reflectance and the illuminant spectral distribution. These models allow representation of the spectral functions with a small number of parameters. The use of linear models by color constancy algorithms will be discussed in detail in Section 2. Spectral reflectance data is available for analysis from many sources. Spectral reflectance functions of some materials are given in the book by Wyszecki and Stiles [89]. These materials include different building materials (brick, shingles, sheet metals, rocks, and enamel paints) and some natural objects. Other data is taken from the work of Krinov [52] who measured the spectral reflectance for samples of 370 natural materials, such as trees, shrubs, grasses, mosses, crops, soils, roads, water surfaces, and snow. Spectral reflectance data also exists for the Munsell color chips. The first study of finite dimensional linear models for reflectance functions was performed by Cohen [9]. The first analysis for the illuminant is due to Judd et al. [46] (the tables are also in [89])Both studies used a characteristic vector statistical analysis. The resulting first three basis functions for representing the illuminant I l ( X ) , Iz(X), and I 3 ( X ) and the first three basis functions for representing the reflectance R1(X),&(A), and &(A) are presented in Fig. 3. More recent experimental
Fig. 3. Basis functions for the representation of illuminants (Judd) and reflectances (Cohen).
288
G. Healey €9 Q.-T. Luong R3
R2
Ri black/white
red/geen
E
blue/yellow
> A
Fig. 4. Basis functions compatible with the NIR model for (A) body and (B) surface reflectance.
work is due to Maloney [59] and Parkkinen and co-workers [45,70,71]. The conclusion is that generally a t least three basis functions are necessary t o represent accurately spectral reflectance functions. A set of three basis functions compatible with the NIR model is used by D’Zmura and Lennie [16] and is also examined by Wandell [86]. These functions are the first three basis functions of a Fourier analysis (Fig. 4): Rl(X) is constant, &(A) is a red/green function, &(A) is a blue/yellow function. 1.2. Perceptual Color Models 1.2.1. Colorimetry: the trichromatic theory The human vision system uses three classes of color photoreceptors called cones and their spectral sensitivity curves can be found in many references including [89]. The three classes of cones respond respectively to the short (blue), medium (green), and long (red) wavelengths giving a decomposition of the visible spectrum. Similar to Eq. (1.2) for an electronic imaging system, the human vision system can be modeled as storing three scalars s1, s2, and s3 at each retina location to represent an incoming color signal I ( X ) . Two color signals I ( X ) and I ’ ( X ) will appear identical if they give rise to the same values of S ~ , S Z and , s3. Color signals which appear identical but for which I ( X ) # I ’ ( X ) are called metamers. Physically identical color signals ( I ( X ) = I ’ ( X ) ) are called isomers. The trichromatic theory summarizes experimental studies of human color vision. The theory predicts that over a wide range of conditions, most colors can be matched by a single additive mixture of three fixed primary colors. The primary colors can be broadly chosen provided that one cannot be obtained as an additive mixture of the other two. The theory further predicts that proportionality and additivity of color matches holds over a wide range of conditions.
2.3 Color in Computer Vision: Recent Progress 289 Consider a set of primary colors with spectral distributions PI (A), & ( A ) , and P3(A).A monochromatic stimulus is a spectral distribution with energy at a single wavelength. According to the trichromatic theory, a unit energy monochromatic stimulus &(A) can be matched by a n additive combination RPl(A) GP2(A) BP3(A)for some values R , G, and B . A value of R , G, or B may be negative if some amount of a primary must be added t o &(A) to obtain a match with an additive combination of the other two primaries. As we allow the wavelength A of the unit energy monochromatic stimulus &(A) to vary, the values of R , G, and B required for a match vary. The resulting functions R(A),G(A),and B(A) are called the color matching functions for these primaries. It follows from the additivity of color matches that a general stimulus I ( A ) will be matched by an additive combination R,Pl(X) G,P2(A) B,P3(A) where
+
+
R, =
s,
+
R(X)I(A)dX, G , = LG(A)I(A)dA, B ,
=
s,
B(A)I(X)dA.
+
(1.8)
The scalars R,, G I , and B, are called the tristimulus values of I ( A ) with respect to the primaries Pl(A),& ( A ) , and P3(A).For any set of tristimulus values R, GIB , the corresponding chromaticity coordinates T , g, b are defined by
R G B b= g= R+G+B R+G+B R+G+B Since T g b = 1, the chromaticity coordinates are typically specified using only two of the coordinates, say r and g in this case. Chromaticity coordinates specify the color quality of a stimulus independent of its absolute intensity. Thus, stimuli I ( A ) and K I ( X ) with significantly different brightnesses will have the same chromaticity coordinates. A common problem is to represent a stimulus I ( A ) using different sets of primaries. This problem arises, for example, when attempting t o preserve the appearance of a color on two different color monitors. Let PI (A), P2 (A), and P3 (A) be a first set of primaries and let Pl (A), Pi (A), and Pi (A) be a second set of primaries. Since each primary in one set can be matched by an additive combination of primaries in the other set, we can write T =
+ +
(1.10) where A denotes a perceptual match and not necessarily a physical match. It follows from (1.10) that for any color the tristimulus values R , G, and B corresponding to the primaries Pl(A),P 2 ( X ) , and P3(A) are related to the tristimulus values R’, G’, and B’ corresponding to the primaries P:(A),Pi(X),and P$(A)by all
a21
a31
(1.11)
290
G. Healey & Q.-T. Luong
Thus, although the set of possible color signals lies in a n infinite dimensional space, from the standpoint of human color perception three-dimensional representations and transformations may be used. 1.2.2. The CIE colorimetric system In 1931, the CIE (Commission Internationale d’Eclairage) defined a standard set of color matching functions X ( X ) ,Y(X),and Z(X) corresponding to a set of hypothetical primaries which lie outside the space of physically realizable colors. The CIE color matching functions are plotted in Fig. 5. These standard color matching functions have the advantage that they are nonnegative so that the resulting tristimulus values X , Y , and Z will always be nonnegative. The function Y(A)has the additional property that it is the same as the relative luminous efficiency function. This means that the tristimulus value Y is a measure of perceived brightness and two stimuli will match in brightness, but not necessarily color, if they have the same Y tristimulus value. The X Y Z system also has the property that a white stimulus with equal energy at all wavelengths will have equal tristimulus values X = Y = 2. The CIE chromaticity ( x ,y) diagram is shown in Fig. 6. The horseshoe shaped curve in Fig. 6 bounds the set of physically achievable chromaticities known as the color locus. The outer boundary of the color locus corresponds t o the chromaticities of monochromatic stimuli (pure colors) except for the purple line near the bottom which connects the blue and red ends of the spectrum. The coordinates of the standard illuminants are plotted in Fig. 6 . The left side of Fig. 7 shows how the dominant wavelength and excitation purity, psychophysical quantities corresponding respectively to hue and saturation, can be obtained from the chromaticity diagram. For stimuli I ( X ) and I’(X) with respective chromaticities (x,y) and (x’,y’), the chromaticity corresponding to an additive combination a I ( X ) bI’(X) will lie on the line
+
Wavelength (nm)
Fig. 5 . CIE color matching functions.
2.3 Color in Computer Vision: Recent Progress 291
X
Fig. 6 . The CIE chromaticity (z, y) diagram.
-
z. Right:
Fig. 7. Left: The dominant wavelength of color A has chromaticity B, the purity is EB Colors that can be obtained by a mixture of I , J , and K .
in the chromaticity diagram connecting ( 2 ,y) and (x’,y’). The right side of Fig. 7 shows more generally that the set of chromaticities resulting from the mixture of three colors lies inside a triangle in the chromaticity diagram with vertices corresponding to the chromaticities of the three colors. Thus, the chromaticity gamut which can be obtained using an RGB display system can be determined from the chromaticity coordinates of the primaries.
292
G. Healey €4 Q.-T. Luong
1.3. Color Spaces Many color coordinate systems have been proposed. Most of the following formulas are in Pratt [73].
RGB : In the RGB space, a color is specified by its tristimulus values with respect to given red, green, and blue primary colors. The RGB space is frequently used with color display monitors, but the associated set of primaries often varies from device to device. A procedure for calibrating color monitors is described in [ l l ] . The CIE defined a standard set of RGB primaries using the monochromatic stimuli R : 700 nm, G : 546.1 nm, and B : 435.8 nm. In terms of these primaries, the tristimulus values RGB can be converted to the CIE tristimulus values X Y Z using 0.73467 0.27376 0.16658 Y = 0.26533 0.71741 0.00886 (1.12) 0.00000 0.00883 0.82456
I:[
I .I:[
[
CMY: The C M Y system is used for color hard-copy devices such as printers and copiers. In this system, the cyan, magenta, and yellow primaries are the complements of red, green, and blue, respectively. These are known as subtractive primaries because depositing an ink or paint on a surface subtracts some of the wavelengths from incident light. For example, depositing cyan ink on white paper causes the paper to absorb red wavelengths (the complement of cyan) and reflect blue and green wavelengths. If RGB values are normalized to the range 0-1, then RGB can be converted to C M Y using
E] [i] [3 =
(1.13)
-
Additional information on color printing can be found in [43]. Work on calibrated color reproduction with application t o the printing of digital images is given in [801. IHS : The intensity, hue, and saturation color space is an intuitive representation which is related to how humans describe colors. Intensity ( I ) is associated with brightness and is defined by I = R+G+B. (1.14)
If we fix intensity I , then we obtain a constant intensity plane P in the three dimensional RGB space. The point in P for which R = G = B is called the white point or gray point. All points on a line in P emanating from the gray point in one direction have the same hue and correspond to adding some amount of white to a pure color. Red and pink, for example, have the same hue. The
2.3 Color in Computer Vision: Recent Progress 293 hue ( H ) of a color (R,GIB ) is thus defined as an angle in the constant intensity plane containing ( R ,GIB ) . Hue is undefined for any gray point ( R = G = B ) . Otherwise, if G 2 B , then hue is given by
h = arccos
2J(R
( ( R- G ) + ( R - B ) ) - G)2+ ( R - B ) ( G - B )
(1.15)
and, in general, if G 2 B and not ( R = G = B ) undefined
if G 5 B and not ( R = G = B ) if ( R = G = B ) .
(1.16)
The geometric derivation for H is given in many places including [27]. For a given hue, saturation describes the purity or amount of white added t o a color. Pure colors such as red have full saturation, while colors such as pink (red plus white) are less saturated. Gray points have zero saturation. Saturation ( S ) is defined bv 3 [min(R,GIB ) ] . S=l(1.17)
(R+G+B)
The quantities I , HIS have an approximate correlation with luminance, dominant wavelength, and excitation purity. The transformation from RGB to I H S is nonlinear and has drawbacks including singularities, instability, and nonuniformity as detailed by Kender [48]. YIQ : The Y I Q color space defines the coordinates which are encoded in the NTSC color television signal. The Y (luminance) component is a measure of brightness and carries the information used for black-and-white television. The I (in-phase) and Q (quadrature) components carry the chromatic information. The Y I Q system takes advantage of the fact that the human vision system is more sensitive to luminance changes than to chromatic changes. Consequently, in the NTSC signal more bandwidth is devoted to the Y component than to the I and Q components. The RGB to Y I Q transformation is linear and invertible and is defined by
[a] [ =
0.587 0.114 0.299 0.596 -0.275 -0.3211 0.212 -0.523 0.311
[i]
.
(1.18)
Perceptually Uniform Spaces: One property shared by all of the color spaces presented in this section is that a fixed Euclidean distance does not correspond to a fixed perceptual distance. To address this problem, the CIE has proposed two separate uniform color spaces in which the Euclidean metric approximates perceived color differences. Each of these representations is based on computing nonlinear functions of the tristimulus
294
G. Healey
6Y Q.-T. Luong
values. The CIE L*u*u* space is used to represent color lights as might be used in color monitors. The CIE L*a*b* space is used t o represent colorant mixtures as might be used in the formation of dyes. Uniform color spaces are discussed in detail in [89]. Perceptually uniform spaces are not particularly useful for computer vision because the magnitude of measurable differences achievable with a color camera is not directly related to perceptual differences. Properties of several color coordinate systems are summarized in Fig. 8.
color system
transformation
RGB
1 I I
r
normalization
uniformity
no
no
rgb
nonlinear, non-one-to-one
Yes
no
XYZ
linear
no
no
zyz
1 nonlinear, non-one-to-one 1
Yes
CMY
linear
no
no
IHS
nonlinear
yes (2 coordinates)
no
YIQ
I
linear
L*u*v*
I
nonlinear
L*u*b*
I
nonlinear
I I
I
no
no no
I
I I
I
no
no
yes
Yes
I I
I
I
Fig. 8. Properties of classical color spaces.
2. Color Constancy Color constancy refers to the ability of humans to perceive surface colors which are relatively stable under large variations of illumination and scene composition. Such an ability demonstrates that color vision requires significant processing beyond the measurement of the physical quantities described in Section 1. Without elaborating, psychophysical experiments reveal strong spatial effects. From a computational standpoint, color constancy is an underdetermined problem and can be posed as the computation of spectral reflectance or another stable surface descriptor from sensor measurements. Color constancy is an important problem because a measured color signal does not by itself indicate anything very reliable about the world. On the other hand, spectral reflectance is an intrinsic property of a surface which can be used for recognition. The color constancy problem has been studied extensively [38]. However, the work that has been done in an algorithmic framework has the characteristics that: 0 0
much of the work is based on restrictive hypotheses many of the algorithms have been demonstrated only on simplified scenes.
2.3 Color in Computer Vision: Recent Progress 295 For a review of color constancy algorithms with particular attention t o retinex algorithms, the reader is referred to Forsyth [19]. Also of interest is a chapter of Hurlbert [44].
2.1. The Retinex Algorithm Land has published numerous papers on the retinex algorithm which have been both influential and criticized. Nevertheless, it is interesting to consider his experiments which illustrate the issues of color constancy and the sophistication of color vision in complex scenes [53-551. Land’s retinex algorithm was based on three principles: 0
0
0
The color perceived at a point does not depend only on the color signal at this point. The color perceived at a point depends only on the combination of three descriptors. The descriptors can be computed independently in the three color bands.
The retinex algorithm is based on the coefficient rule or von Kries model of color constancy. Under this model of color constancy, surface descriptors are obtained by scaling the measurements in each color band independently. The assumption underlying this model is that if s1(z7y),s2(x,y), and s ~ ( z , y )are the sensor measurements (Eq. (1.2)) for a surface under illuminant l and si(x,y), sa(x,y), and sb (2, y) are the corresponding sensor measurements under illurninant 2, then the sensor measurements are related by a diagonal matrix:
[
si(x1 Y)
]I.,.I:.s$(x,Y)
=
[
mll
0
m,2
m!3]
[
s1
(x7
Y) (2.1)
s 32((xx, Y) , y ) ].
It follows from (1.1)and (1.2) that this diagonal model is an approximation. Equation (2.1) holds exactly, however, for the case of narrowband sensors fj(X). The computational problem addressed by retinex is to recover an approximation to surface reflectance by discarding the effects of the illuminant. From (1.1) we have that I ( X , r) = A X ,r)E(X,r) . (2.2)
A basic assumption of retinex is that there exists an asymmetry between the reflectance p and the illumination E which enables the solution of (2.2): p consists of uniform patches with abrupt changes whereas E varies smoothly over the scene. This is called the Mondrian World. The two conditions required for the retinex algorithm to work are: 0
Hypothesis 1 (Mondrian world): the scene is a flat Mondrian world and the illumination varies slowly and smoothly.
296 0
G. Healey
tY Q.-T.
Luong
Hypothesis 2 (Gray world): The mean value of the scene reflectances in each color band is the same.
The first hypothesis enables the separation, at each spatial location, of the reflectance and illumination components of the sensor measurements. The second hypothesis guarantees that a spectral normalization will give a triplet of color constant surface descriptors. Although many related specifications of the retinex algorithm exist, Brainard and Wandell [4] have shown that for a representative version in the limiting case the measurements in each band are simply scaled by the mean of the sensor measurements in that band. One variation involves weighting the measurements so that spatially distant measurements have less influence in determining the scale factor at a point. Several studies, e.g. [60], have demonstrated that the descriptors computed by the retinex algorithm are relatively stable in the presence of illumination changes. Brainard [4] has shown, however, that the retinex descriptors for a fixed surface can be quite unstable with respect to changes in the color of surfaces surrounding the fixed surface. This instability is often related to the violation of Hypothesis 2 above. The dependence of the descriptors on scene composition renders the retinex algorithm an inadequate model of human color vision and severely limits its usefulness for computer vision.
2.2. Algorithms Based on Linear Models Many approaches to color constancy are based on the use of a finite dimensional linear model for spectral reflectance as described in Section 1.1.3. Using such a model, the spectral reflectance p(X, r) at each location r is approximated by
where the Rj(X) are a set of n fixed basis functions. Several empirical studies [9,59,70] have shown that at least three basis functions are required to approximate accurately naturally occurring spectral reflectance functions. Similarly, the illuminant spectra can be expressed as a linear combination of basis functions using
The derivation of color constancy algorithms from linear models allows explicit characterization of the set of scenes for which a n algorithm will exhibit color constancy. This ability to specify the domain of applicability of an algorithm is a primary advantage of this approach. 2.2.1. Known average reflectance algorithms
The idea of the algorithms of Buchsbaum [7] and Gershon et al. [26]is to estimate the illuminant color using the assumption (similar to the Gray world hypothesis)
2.3 Color in Computer Vision: Recent Progress 297 that some average property of the scene is known. The Buchsbaum algorithm assumes that the mean value of the reflectance over the scene is a gray value. Gershon has slightly improved this algorithm by assuming that the average reflectance is the mean taken over Krinov’s [52] data of reflectances of natural surfaces. These algorithms assume that the spectral reflectance is approximated by a three parameter (n = 3) linear model as in (2.3) and that the illuminant is approximated by a three parameter (m = 3) linear model as in (2.4). The average spectral reflectance is assumed to be a known function p(X) and the illumination E(X) is assumed to be spatially constant over the scene. The first step in the algorithm is to compute the average sensor response vector (Si,S2,S3) over the image. In Gershon’s case, the average is computed using segmented regions so that each region in the image is counted once independent of size. Substituting the linear illumination model into Eqs. (1.1) and (1.2) we have 31 =
s,p(X)[fiIi(X) + €212(X) + f3k(X)]fi(X)dX
(2.5)
giving a system of three linear equations which can be solved for the unknowns €1,€ 2 , €3 which specify the illurninant E(X). Using this estimate of the illurninant leads to three linear equations at each image location
which can be solved for the unknowns al(r ), aa(r), and a3(r) which specify the spectral reflectance. The primary limitation of these algorithms is that for many applications the average reflectance for a scene is not known.
2.2.2. Dimensionality bused algorithms Dimensionality based algorithms do not require any assumptions about the average properties of surfaces in a scene. Instead, the primary assumptions concern the general structure of the sets of reflectance functions and illuminants. If an n parameter linear model for spectral reflectance is used, then the sensor measurements are related t o the components of the spectral reflectance by a linear
298
G. Healey & Q.-T. Luong
transformation M (2.11)
where the elements of M are given by (2.12) Note that for n = 3, (2.11) is equivalent to (2.8)-(2.10). It follows from (2.11) that the sensor measurement vector for any surface is a linear combination of the n column vectors of M . Thus, the set of sensor vectors falls in a n n-dimensional subspace of the space of possible sensor measurements. This subspace depends only on the illurninant, whereas the positions of the responses in this subspace depend only on the reflectances. If n is less than the number of color bands, then this subspace is of lower dimension than the sensor measurement space and can be determined using standard techniques. Figure 9 illustrates the idea for three-classes of sensors with n = 2. The reflectance functions are specified by two parameters which the matrix M maps t o a plane in the sensor space. This plane can be recovered if there are enough distinct surfaces in the scene. Knowledge of this plane and the assumption of a three-dimensional linear model for the illuminant allows recovery of the illumination parameters €1, € 2 , and €3 giving the matrix M . The reflectance parameters al(r) and a2(r) can then be computed at each image location using the pseudoinverse of M . Although the Maloney-Wandell algorithm represents a n important theoretical advance, the restriction for trichromatic ( N = 3) color constancy that reflectance functions be modeled by two parameters prevents the procedure from being useful for most applications. In fact, experiments [18] have demonstrated that in many situations it is better to do nothing than to apply the Maloney-Wandell algorithm. Forsyth [20] has extended the Maloney-Wandell algorithm by developing a procedure MWEXT which recovers more parameters in the linear illumination model
Y
Fig. 9. The idea behind the Maloney-Wandell method.
2.3 Color in Computer Vision: Recent Progress
299
than the original Maloney-Wandell formulation. As with the original MaloneyWandell algorithm, however, MWEXT requires a similar limiting restriction on the dimension of reflectance functions. 2.2.3. Gamut mapping algorithms
Following the work of Maloney-Wandell, several researchers have developed trichromatic color constancy algorithms by introducing additional constraints which allow spectral reflectance models with more than two degrees of freedom. Forsyth [20] developed a n algorithm CRULE which is a form of coefficient rule in that each band is scaled separately to achieve color constancy. This algorithm works for arbitrary reflectance functions provided that they are viewed by narrowband sensors. For conventional color sensors, CRULE requires corresponding constraints on the spectral properties of the illumination. The goal of CRULE is to transform the sensor values measured under an unknown illuminant to the values which would be measured under a known canonical illuminant. The algorithm begins by representing the set of sensor values which can be observed for all reflectances under the canonical illuminant. Given a set of sensor measurements for a scene, there exists a set of triplets of coefficients which transform these sensor measurements into possible corresponding measurements under the canonical illuminant. The coefficients chosen for normalization are selected from the possible set using a maximum volume heuristic. Forsyth shows using real Mondrian images that CRULE performs significantly better than retinex when the composition of the Mondrian is changed. Finlayson [17] presents an insightful review of Forsyth’s work and shows how CRULE can be extended to incorporate prior constraints on illumination color. By working in a two-dimensional normalized color space, Finlayson further suggests how the performance of CRULE can be improved in the presence of curved surfaces and highlights. 2.2.4. Local algorithms
Each of the previous approaches t o color constancy has depended on the use of information obtained over several surfaces with different reflectance functions. Brainard et al. [5] and Ho et al. [42] have examined conditions under which color constancy is possible locally by considering only a single point in an image. The underlying idea is that if reflectance is represented by a linear model as in (2.3) and illumination is represented by a linear model as in (2.4)’ then the color signal received at the sensor is described by P ( X , r)E(X,r) =
cc
a j ( r k i(+j
(Wi (4.
(2.13)
l image
plane
Fig. 5. The intrinsic or interior parameters.
which depend only on position; the latter are therefore called extrinsic or exterior parameters. The first terminology comes from the computer vision community (see for instance [ll]or [12], but the second is more than 50 years old and comes from photogrammetrists. Figure 5 depicts the standard reference frame for a pure perspective imaging system. The image plane has its own reference frame (27 2. The image plane is at a distance f from the principal point 0. f is called the principal axis distance, but also sometimes improperly called focal length. 0 is the origin of the camera reference frame (2, g, 4, which is oriented as in the figure. If the 3-D word reference frame coincides with the camera frame and if the camera coordinate axis (?, $ is parallel to the axis of the image frame, the perspective projection matrix P can be written as:
k , and kv are the scaling factor; they rely on the value o f f and on the image scaling factor on each image axis. uo and wo are the coordinates of the projection of the optical center onto the image plane. It has t o be noted that if the image frame is not orthogonal as it is in Fig. 5, the total number of these camera parameters are then five: U O ,V O , k,, k , and the angle a. So if we add the six degrees of freedom for the three-dimensional positioning of this camera frame in a world frame, we reach the total number of 11 degrees of freedom for the camera, which is to be related to the 11 degrees of freedom of a projective transformation from P 3 on P2.
2.4 Projective Geometry and Computer Vision 323
The general projection matrix P’ can be written as the product of a Euclidean transformation with the projection matrix P, the Euclidean transformation mapping the world coordinate frame in the camera coordinate frame:
(i 2 ”) . ( 0
P’=
uo 210
1
0
000
T1 t ) .
(3.2)
Given an estimation for P’, the computation of the estimation for P and for the rotation R and translation T satisfying (3.2) is a less easy and stable problem. References [ll]or [12] provide solutions and other new methods have been developed using for instance the properties of vanishing points [13]. It has t o be pointed out that, even in the case of a moving camera, such a decomposition is unnecessary. For instance if the camera is calibrated in two positions with the resulting projection matrices P’ and P”, the matrix associated to the motion to be estimated is just a rotation and translation matrix M containing a rotational orthogonal matrix R and a translation T such that
p” = p ’ x
(
000
’“)1
.
4. Application to Stereovision As it was presented in the previous section, a point in a image corresponds to a line in the space, and this line in completely determined if the system is calibrated. Let us now consider the case of two cameras observing the same scene. If we are able to find the projection in the two images of a 3-D point M , then its position in space is simply defined by the intersection of the two lines associated with each image. Usually such an intersection is computed using least square methods, as errors are always introduced in all the steps of the process: calibration, image acquisition, image point determination. Stereovision thus implies several steps: 0 camera calibration, 0 determination of points correspondence from one image t o the other, 0 3-D reconstruction. The reader is referred to books concerned with the subject like the one describing the pioneer work of Marr and Grimson [14] or the book written by Ayache where he describes the use of three cameras [15].
4.1. The Epipolar Geometry We only address here the problem of the geometry of such a stereoscopic system. Figure 6 displays the configuration. The 3-D point A4 observed by the two cameras defines a plane which intersects the image planes respectively by line 1 and 1’. Notice that when M moves, all these lines 1 pass through the intersection of the
324
R. Mohr
0
0 Fig. 6 . Epipolar lines for two images.
image plane P and the intersection of the line 00‘. This point e is called the epipole of image 1 with respect to image 2. Similarly, e’ is the epipole of image 2 with respect to image 1. 1 and I’ are the corresponding epipolar lines for these two images. Let m and m’ be the projection of M on each image. It has to be noticed that the line Om is projected on image 2 as the line l’, so each possible point m’ corresponding to the projection m of M has to be on the epipolar line 1’ associated t o 1 . Such a geometrical constraint reduces nicely the search for corresponding matches in the two images. Consider now the two pencils of epipolar lines when M moves through the space. Let us consider four points M , providing four distinct corresponding epipolar lines I , and 1:. As these pencils are obtained by the intersection of the pencil of planes passing through OO’, their cross ratios are the same; in fact this cross ratio is by definition the cross ratio of the pencil of planes. Therefore the pencil of the corresponding epipolar lines are in a projective correspondence. In conclusion, if three of the corresponding epipolar lines are known, the epipolar line correspondence for a fourth line is deduced in a straightforward manner from the cross ratio of the four lines in the pencil. The problem of epipolar geometry has seven degrees of freedom: 2 x 2 for the coordinates of the epipoles e and el, and three for the three epipolar lines in the second image corresponding to three arbitrarily epipolar lines going through e in the first image. So, the correspondence of between seven points in the two images is enough for defining the epipolar geometry. How it can be done effectively was recently established by Maybank and Faugeras [16].
2.4 Projective Geometry and Computer Vision 325 4.2. A Linear Computation of the Epipolar Geometry [3]
In the case of eight point matches between the two images, the computation of the epipolar geometry becomes much simpler. This elegant construction is inspired by [17]. Let m = ( 5 , y, t ) be a point in the first image and let e = (u,v, w) be the epipole point with respect t o image 2. The three homogeneous coordinates ( a , b, c) of the epipolar line 1 going through e and m are m x e where x denotes the cross product: obviously l.mt = l.et = 0. The mapping m = (5, y, z ) + ~ mx e = ( a , b, c ) ~is linear and can be represented by a matrix C of rank 2:
(H)
yw - zv
=
(-)
=
(-YW u! ); (i)
(4.1)
The mapping of each epipolar line 1 from image 1 to its corresponding epipolar line 1' in image 2 is a collineation defined in the dual space of lines in P2.Let A be one such collineation: 1lt = Alt. A is defined by the correspondence of three distinct epipolar lines. The first two correspondences provide four constraints as the degree of freedom of a line is 2. As the third line in correspondence belongs to the pencils defined by the two first ones, the third correspondence only adds one more constraint. So A only has five constraints for eight degrees of freedom. Let E = AC. Using (4.1) we get
As A has rank 3 and C has rank 2, E has rank 2. As the kernel of C is obviously Xe e , the epipole is the kernel of E. Let m' be the corresponding point of m. Using (4.1) the epipolar constraint mlllt - 0 can be rewritten m'Em = 0. So each matching between the two images provides a constraint on E, and as E is defined up t o a scaling factor, eight independent constraints will allow us to linearly compute E and therefore get the epipolar geometry. Notice that E is defined by seven degrees of freedom: C has two (the epipole position) and A has five. But allowing a redundant set of constraints provides a unique solution which can be linearly computed. N
4.3. Bringing Epipolar Lines i n Parallel: Image Rectification
A visually interesting case occurs when two image planes are parallel: the epipoles are at infinity and the epipolar lines are therefore parallel. Many people working in computer vision use image planes which are the same for the two images, which is an even stronger constraint than parallelism. This makes computation a bit simpler, but adds an unnecessary technical constraint to the vision system. As these constraints are hardly satisfied with the needed precision, this has to be avoided for real applications where the goal is precision in reconstruction.
326
R. Mohr
If parallel epipolar lines are wanted for easy human visual matching, one would prefer the following method: 0 calibration of the stereovision system, 0 computation of the epipoles, 0 computation of the image transformation which provides parallel epipolar lines, 0 reconstruction of all the image features after such a transformation. The transformation computation is easy (see [15] for another explanation and illustration). Let e = ( a ,b, l)t be the epipole of the first image. We have to find a projective mapping which sends e to infinity along the image 2 axis, i.e. a 3 x 3 homogeneous matrix A such that
(t)
=A(!)
The images of four points completely define A. As only one is presently set, we can add some more constraints, for instance leave three of the four corners of the image invariants. Such a choice allows the features to move not too far out of the image border lines. The same process can then be applied on the second image. However people prefer to have the epipolar not only parallel t o the 2 axis, but also that the corresponding epipolar line have the same coordinates. Therefore we chose three epipolar lines in the first rectified image, each having the equation y = yi. On the corresponding epipolar lines on image 2 three points can be chosen and their image specified with coordinates (xi,yi), where the xi are arbitrarily chosen. This allows enough freedom to have the rectified second image with reasonable coordinates. Figure 7 displays such rectified images. It is interesting to notice that such rectifications were already done optically with old photogrammetrist material a hundred years ago: as a perspective projection is just the general case of a projective transformation, this was done by choosing interactively a new projection of the previous image. 4.4. The Dansfer Problem
Having located a 3-D point in two images, the transfer problem is to determine how it can be located in a third one. In order to solve it, we need some knowledge of the three images, and here we assumed that we have the matches of several features within the three images. Computing the epipolar geometry of the imaging systems provides a direct solution. Let pi be the location of the considered point in image i. p l and p2 are already located. So p3 has t o be on the epipolar line corresponding to p l in image 3 , and on the epipolar line corresponding to pa. Thus it lies at their intersection. This fact is widely used in trinocular stereovision 1151. The fact that epipolar geometry can be computed with at least seven point matches was already mentioned in the previous subsection. However a simple case
2.4 Projective GeometnJ and Computer Vision 327
Cmaal
Fig. 7. Rectification of two stereo images. The corresponding epipolar lines are now horizontal lines with equal coordinates (courtesy of N. Ayache).
can be considered here when only six such matches are known, in the three images, and four of these points correspond to coplanar 3-D points. In what remains in this section we suppose that we shall never encounter degenerate cases (for instance two lines coinciding instead of the general case of two different lines). Let A , B , C, D be the four coplanar 3-D points and F, G the two remaining reference points. Oil i = 1, 2, 3 are the principal points of each of the three imaging systems we consider. a’, a”, a”’ are the projections of A in the images 1, 2, 3. The intersection F’ of the view line 0 1 F with the plane ABCD is defined by its projective coordinates measured in image 1, taking the projections a’, b’, c’, d’ as reference frame (see Fig. 8). Now consider image 2. Using a’‘, b”, c”, d” and the projective coordinate of F‘, we can locate its projection 4 in the second image. As we also have the image f” of F , we have therefore in image 2 the projection of two points from the line 0 1 F , i.e. we have the epipolar line associated with F (see Fig. 8). If we proceed similarly with G, the intersection of these two epipolar lines provides the epipole e12 of image 2 with respect t o image 1. Of course the process is symmetrical and allows us to find the epipole e21 of image 1with respect to image 2. Three epipolar lines are needed to complete the epipolar correspondence: using the
328
R. Mohr
Fig. 8. Reconstruction of the epipolar geometry.
reference point matches we have plenty of them: e21a1 with e12a2, e21bl with e12b2, and so on. Now consider the third image. From the epipolar geometry the position of each point matched in the two first images is straightforward. Using the previous construction, the epipolar geometry between images 1 and 3 and between images 2 and 3 is constructed, and the epipolar lines corresponding to the location of the point in images 1 and 2 intersects in only one possible position. 5. Application to 3-D Positioning 5.1. Relative Positioning Let us consider first the simple case of four points Pi lying on a line and viewed on an image where we can compute the cross ratio. We know that the fourth point is uniquely defined from the position of the first three and from the computed cross ratio. So, taking the first point as the origin, the position of the fourth point can be expressed, using as parameters the position of the second one and the third one, and using the cross ratio. The resulting expression is
2.4 Projective Geometry and Computer Vision 329
This simple example shows how relative positioning is possible. Similar construction can be done in the plane using the projective coordinates defined by the cross ratios (2.7) and (2.8). In the more simple case when the four points are the vertices of a parallelogram, the relation simplifies as we can easily choose two sides as reference axis (see Fig. 9).
Fig. 9. Relative positioning using a parallelogram.
In such a case the position of P in the frame (A>, A?7) is easily deduced:
x=-
-k1
kl y=-
+ k2 1
kl + k2 kl = [AB,AC; AD, A P ] , k2 = [BA,BD; BC, BPI.
(5.2)
As no 3-D position can be deduced from a single image [18],extra assumptions have t o be added: the alignment in the space of four points for (5.1) or coplanarity for (2.7) and (2.8). 5.2.
Where is the Camera?
We consider here the problem of finding the location of the principal point (sometimes called optical center) of the viewing system. We first consider the case of seeing six points in the scene, with four of them coplanar. First we are derive the view line associated with an image point relative to a reference point in the scene. Let m be the projection of a point M on a n image where the projection of the planar configuration ABCD is projected as abcd (see Fig. 10). As we mentioned (Section 2.3), we can compute the projective coordinates of f with respect t o the basis a, b, c, d. From the definition, these coordinates are the same for F', where the view line OF intersects with the plane ABCD. So the view line goes through F and F' and is defined. Proceeding in a similar way, we can compute the view line coming through E , and therefore the principal point 0 is the intersection of these two lines.
330
R. Mohr
center of projection
Fig. 10. The back projection of the image point m.
Having the 3-D position for 0, we deduce easily the viewline associated with each point m in the image. Such a computation can also be done using non-coplanar points, but the demonstration is a bit tedious and the reader is referred to [19] for the details. The hint of the technique can however be provided using a planar configuration: we suppose here that the image is restricted to a line and that we are taking such a picture in a planar world. We observe five reference points A , B , C, D, E with their images a, b, c, d, e. Measuring the cross ratio [a,b; c, d ] , we deduce from Chasles’ theorem that the principal point lies on a conic passing through A , B , C and D and completely defined by [a, b; c , d] (see Fig. 11). We can do it again with A , B , C, E and the two conics intersect in four points, three of them already known: A , B , C. So the remaining intersection is the desired position and is computed algebraically from formula (2.11).
Fig. 11. The camera location lies at the two conic intersections.
2.4 Projective Geometry and Computer Vision 331 5.3. Choosing Points a s References in the Scene
The techniques presented may lead to 3-D position estimation of points in the scene. However such an estimation needs at least two images when no constraints on the scene are given. We describe here a simple experiment of computation of the location with two views.
_.
I! I
!i
j(
ij
..-'i
---.a
Fig. 12. Contour image of a scene.
Let us consider the scene described in Fig. 12. It displays contours of a n image taken approximately a t a distance of 1 m with a regular Pulnix camera. Contours were fitted with straight lines and corner point coordinates were computed as intersections of these lines. The same process was applied on a second image. Taking as reference points six points from the background rectangles, the view line associated with each contour was computed using the technique presented in the previous section. Matches between the images was performed by hand as matching was not the primary concern of the present study. Intersection of the view lines associated with two corresponding image points was then computed using least squares and this provided us with the 3-D coordinates of the corresponding point in the scene. Table 1describes the results for the cube vertices. The exact location has no real meaning, it corresponds to the reference frame of the chosen point whose locations were measured with a standard ruler. Much more interesting are the edge lengths computed from these coordinates. As the exact size of the cube is 50 mm, the computed results are accurate within 4% of the value, and this without camera modeling and subpixel edge extractor.
332
R. Mohr Table 1. Experimental results for 3-D reconstruction of the cube. Points 0 1 2 3 4 5 6
z 78.9 79.1 81.3 82.0 33.2 34.4 30.3
Y 140 141 189 188 195 194 145
z
48.5 -2 47.5 -1.5 48.5 -1.5 49.0
Edges 0-1 0-2 0-6 1-3 2-3 2-4 3-5 4-5 4-6
Length 50.5 49.1 48.9 47.1 49.5 48.9 48.0 49.8 50.1
6. Recognition Using Projective Invariant In order to classify patterns, standard pattern recognition techniques use numerical measures which are invariant under the experimental conditions like for instance the movement of the observing camera (see [20] or Chapter 1.2 “Statistical Pattern Recognition” in this book). Therefore such invariants can be applied directly in classification methods. More recently researchers developed indexing for selecting a subset of possible candidate models using hashing techniques based on geometric invariants [21,221. The interesting point with the geometrical approach is that partial information on the image is sufficient to recover points, straight lines, conics, etc. and therefore to compute the invariants. This is not the case for standard global invariant measures like moments. We will first explore some results of the invariant theory and derive from there some invariants in the second subsection.
6.1. Results on Invariant Theory Only a simple introduction of this theory can be provided in this chapter. The reader is referred to the standard textbook on the subject like the second part of [5] or to the more vision application-oriented ones [4]. Let G be a group which acts on a set E l and o the composition operator of G. For instance 4 can be the Euclidean transformations in the plane and E the set of circles in this plane. o is in this case the composition of such transformations. 6 acts on E means that
Finding an invariant for E means computing a measure m that is constant for all z E E : ’v’x, z’ E E m ( % )= rn(z’). Of course if m(.) is such a function, so is f ( m (.)). We are only interested in such independent invariant functions.
2.4 Projective Geometry and Computer Vision 333 Let us consider the example of E being the set of circles with radius r. E is generated from a single circle by applying all the Euclidean transformations. There is an obvious invariant here: the radius r. The area a is also an invariant, but it is not independent with r: a = 2 m 2 . On the other hand if we consider the set of all points in the plane under the Euclidean transformation, there are no invariants. Before stating the basic result, we need few more notations. Let Dc be the degree of freedom of (i.e. more formally its dimension). Let D, be the degree of freedom of the subgroup which leaves an element z E E invariant (such a subgroup is called the isotropic subgroup of z), and let D E be the degree of freedom of E , then the number I of independent invariants is:
I = DE
- (Dc - min 0,) xEE
.
In the previous example, the degree of freedom for the planar Euclidean transformation is 3, a circle is defined by three parameters, and the subgroup which leaves a circle invariant is the one-dimensional subgroup of rotations centered at the circle center. So we get one independent invariant. Notice that we are only dealing with groups here. So we are not addressing the problem of the projection of the 3-D space into a 2-D image. But this result is applicable to planar shapes projected onto an image: the group is then the group of 2-D collineations. In fact there is no invariant in the case of 2-D projection of 3-D data [ 181 without additional conditions such as coplanarity. 6.2. Computing the Invariant Using the Cross Ratio
Formula (6.1) provides us only with the number of possible independent invariants. Here we explore how they can be computed easily in the projective case using the cross ratio. Recall that the degree of freedom for the collineation in P z is 8, as a collineation in the projective plane is defined by a 3 x 3 homogeneous matrix. 6.2.1. Invariant for two conics
Each conic has five degrees of freedom; this provides us with ten parameters for two conics. As there is no collineation which leaves two conics invariant in the general case, two invariants have to be discovered. The two conics intersect in four points, and as stated in Section 2, four points on a conic define a cross ratio. Each of these two cross ratios (one for each conic) are obviously independent. From (6.1) it is then possible to conclude that all other invariants can be obtained as a combination of these two measures. Such invariants are very useful as conics can be found by conic approximation to different shapes [23]. 6.2.2. Two points and two lines
Let A and B be two points and a and /3 be two lines intersecting in 0. This provides us with an eight degrees of freedom configuration (see Fig. 13). There is
334
R. Mohr
Fig. 13. The two points and two lines configuration.
an obvious invariant cross ratio: the line y defined by A and B intersects (Y and p in C and D, and this defines four points on a line; four points on a line provide us with a cross ratio [A, B , C, D]. Formula (6.1) indicates therefore that they should be a collineation subgroup leaving this configuration invariant and with degree of freedom a t least equal to 1. In fact if A and B are at infinity, it is easy to see that the only collineations which leave this configuration invariant are the uniform scaling transformations (similitudes) with origin at 0. This obviously is a one-dimensional subgroup and so there is only one independent invariant: [A, B; C, D] is a solution. 6.2.3. Implementation Experiments were conducted on this kind of invariants at the University of Oxford [24]. Figure 14 shows the relevant features used. In fact these authors computed these invariants using the general algebraic tools from the invariant theory. These algebraic invariants are nicely related to the one presented here [25]. Using such invariants may however lead to small combinatorial problems. For instance five points in a plane have two invariants, the two cross ratios provided by (2.7) and (2.8). However, having five points we have to choose which four points are going to be used as basis, and in which order they have to be considered. This leads us to 5! = 120 possibilities. The solution to this combinatorial problem is the use of the symmetrical polynomials. For instance we know that if k is the cross ratio of four points, then all possible cross ratios obtained by taking these four points in different orders are: 1 k , - , 1-k,-
k
1
I-k’
k-1 -or-
k
k k-1.
2.4 Projective Geometry and Computer Vision 335
Fig. 14. Recognition using conic fitting and conic invariants (courtesy of D. Forsyth et al.).
So for getting an invariant which does not depend on the order of these points, we have to look for a symmetrical polynomial with six variables, each for a value of (6.2). One of the simplest symmetrical polynomials which is not constant is 6
= i=l
2k6 - 6 k 5
+ 9k4
-
8k3 -t- 9k2 - 6k 1- 2
k2(k - 1 ) 2
Finding such symmetrical polynomials avoids the combinatorics of sorting the feature in a feature set. There still remains the combinatorial problem of collecting the right set of features, for instance the right set of five points. No general answer exists for this problem.
7. Discussion This chapter provided a short introduction to the geometry of the image formation system in the case of a pure perspective projection. For real images, optics and electronics provide differences with this ideal model and this can reach some pixels when using standard CCD cameras. For this reason methods are proposed in calibration for correcting the image, bringing the geometry back to the original perspective projection [26,1]. Projective geometry offers the right tool for dealing with such perspective projections. Its basic invariant is the cross ratio. Computing cross ratios can be done easily with image points, lines or conics. Such sets of features provide invariants which can be used in two ways: by indexing models for finding the possible models which can be associated with the invariant measured in a scene, or by finding the relative 3-D location of an object with reference t o another object. The latter case is called relative positioning and has proved t o be more flexible and robust than standard 3-D positioning in a camera reference frame.
336
R. Mohr
New kinds of geometric invariants are presently under investigation. In [27] the reader may find studies of differential invariants, i.e. invariants obtained on a curve using derivatives of different orders. Cross ratios on areas can also be computed [28]. However few experiences were reported on the stability of the values computed in the different cases. Such practical evaluations still remain to be done.
Acknowledgements The Esprit program “Basic Research” and the French national project “Orasis” provided financial and intellectual support to many parts of the work reported here. E. Arbogast, P. Gros, L. Morin, and L. Quan are kindly acknowledged for their insightful discussions and contributions. Figure 7 is displayed with courtesy of N. Ayache, and Fig. 14 with courtesy of A. Zisserman, J. Mundy, D. Forsyth, and Ch. Rothwell. I would like to thank them all for their cooperation.
References [l] R. Y. Tsai, A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shell T V cameras and lenses, IEEE Trans. Robotics and Automation 3, 4 (1987) 323-344. [2] K. W. Wong, Mathematical formulation and digital analysis in closerange photogrammetry, Photogrammetric Eng. Remote Sensing 41 (1975) 1355-1373. [3] 0. D. Faugeras, 3-0 Computer Vision (MIT Press, Cambridge, MA, 1992). [4] J. Mundy and A. Zisserman (eds.), Applications of Invariance in Computer Vision (MIT Press, Cambridge, MA, 1992). [5] J. G. Semple and G. T. Kneebone, Algebraic Projective Geometry (Oxford Science Publication, 1952). [6] S. J. Maybank, The projective geometry of ambiguous surfaces, Technical Report 1623, Long Range Laboratory, GEC, Wembley, Middlesex, UK, Jul. 1990. [7] N. Efimov, Advanced Geometry (MIR, Moscow, 1978). [8] R. Horaud, B. Conio, 0. Leboulleux and B. Lacolle, An analytic solution for the perspective 4-point problem, Comput. Vision Graph. Image Process. 47 (1989) 33-44. (91 J. S. C. Yuan, A general photogrammetric solution for determining object position and orientation, IEEE Trans. Robotics and Automation 5, 2 (1989) 129-142. [lo] Y . Liu, T. S. Huang and 0. D. Faugeras, Determination of camera location from 2-D to 3-D line and point, IEEE Trans. Pattern Anal. Mach. Intell. 12, 1 (1990) 28-37. [ll] R. K. Lenz and R. Y. Tsai, Techniques for calibration of the scale factor and image center for high accuracy 3-D machine vision metrology, in Proc. IEEE Int. Conf. on Robotics and Automation, Raleigh, USA, 1987, 68-75. [12] 0. D. Faugeras and G. Toscani, Camera calibration for 3-D computer vision, in Proc. lnt. Workshop on Machine Vision and Machine Intelligence, Tokyo, Japan, 1987. [13] B. Caprile and V. Torre, Using vanishing points for camera calibration, Int. J. Cornput. Vision 4 (1990) 127-140. [14] W. E. L. Grimson, From Images to Surfaces. A Computational Study of the Human Early Visual System (MIT Press, Cambridge, MA, 1981). [15] N. Ayache, Stereovision and Sensor Fusion (MIT Press, Cambridge, MA, 1990). 1161 S. Maybank, 0. Faugeras and Q. T. Luong, Camera self-calibration: Theory and experiments, in Proc. Second European Conf. on Computer Vision, Santa Margherita, May 1992.
2.4 Projective Geometry and Computer Vision 337 [17] 0. Faugeras, What can be seen in three dimensions with an uncalibrated stereo rig? in Proc. Second European Conf. on Computer Vision, Santa Margherita, May 1992. I181 J. B.'Burns, R. Weiss and E. M. Riseman, View variation of point-set and linesegment features, in Proc. DARPA-ESPRIT Workshop on Applications of Invariants in Computer Vision, Reykjavik, Iceland, Mar. 1991, 55-108. [19] R. Mohr, L. Morin and E. Grosso, Relative positioning with poorly calibrated cameras, in Proc. DARPA-ESPRIT Workshop on Applications of Invariants in Computer Vision, Reykjavik, Iceland, Mar. 1991, 7-45. [20] J. T. Tou and R. C. Gonzalez, Pattern Recognition Principles (Addison-Wesley, 1974). [21] P. C. Wayner, Efficiently using invariant theory for model-based matching, in Proc. Conf. on Computer Vision and Pattern Recognition, Maui, Hawaii, Jun. 1991, 473-478. [22] H. L. Wolfson, Model-based object recognition by geometric hashing, in 0. Faugeras, (ed.), Proc. 1st European Conf. on Computer Vision, Antibes, France (SpringerVerlag, 1990) 526-536. [23] D. Forsyth, J. L. Mundy, A. Zisserman and C. M. Brown, Projectively invariant representation using implicit algebraic curves, in 0.Faugeras (ed.), Proc. 1st European Conf. on Computer Vision, Antibes, France (Springer-Verlag, Apr. 1990) 427-436. [24] D. Forsyth, J. L. Mundy, A. Zisserman and C. Rothwell, Invariant descriptors for 3-D object recognition and pose, in Proc. DARPA-ESPRIT Workshop on Applications of Invariants in Computer Vision, Reykjavik, Iceland, Mar. 1991, 171-208. [25] L. Quan, P. Gros and R. Mohr, Invariants of a pair of conics revisited, in P. Mowforth (ed.), Proc. British Machine Vision Conf., Glasgow, Scotland (Springer Verlag, 1991) 71-77. [26] C. C. Slama (ed.), Manual of Photogrammetry, fourth ed. (American Society of Photogrammetry and Remote Sensing, Falls Church, VA, 1980). [27] A. Zisserman and J. Mundy (eds.), Proc. DARPA-ESPRIT Workshop on Applications of Invariants in Computer Vision, Reykjavik, Iceland, Mar. 1991. [28] E. B. Barrett, P. M. Payton, N. N. Haag and M. H. Brill, General methods for determining projective invariants in imagery, Comput. Vision Graph. Image Process.: Image Understanding 53, 1 (1991) 46-65.
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 339-385 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
I CHAPTER 2.5 I 3-D MOTION ANALYSIS FROM IMAGE SEQUENCES USING POINT CORRESPONDENCES
JOHN. J. WENG Department of Computer Science, Michigan State University East Lansing, Michigan 48824, USA and THOMAS S. HUANG Beckman Institute, University of Illinois at Urbana-Champaign Urbana, Illinois 61801, USA
The objective is to analyze the motion between a rigid scene and the camera. The temporal correspondences between consecutive images are established by a procedure of image matching. Such temporal correspondences are then used for the estimation of the three-dimensional (3-D) interframe motions, as well as the 3-D structure of the scene. Long term motion that covers many image frames is modeled using object dynamics, and the model parameters are determined from the interframe motions. Thus, smooth 3-D motion can be predicted using the model parameters. Keywords: Image matching, optical flow, 3-D motion estimation, structure from motion, motion modeling, motion prediction.
1. Introduction
The projection of a dynamic 3-D scene onto an image plane contains rich dynamic and geometric information about the scene. The projections a t different time instants can be recorded by a sequence of images. To extract the information from the image sequence, several basic subtasks are identified: image matching, interframe motion estimation, motion modeling and prediction.
1.1. Image Matching The first subtask is to establish the correspondences between images. Its objective is to identify image elements in different images that correspond to the same element of the sensed scene. The matching elements, or tokens, can vary significantly from one approach to another. Existing techniques for image matching roughly fall into two categories: continuous and discrete.
339
340
J. J. Weng & T. S. Huang
(1) Continuous approaches. Although the objective of the approaches in this category is to determine image velocity field instead of performing explicit matching between images, the computed velocity field amounts to image matching. Each velocity vector approximates the correspondence between two points in different images. Ideally, one needs the projection of 3-D velocity on the image plane. However, since such a projection is not directly available from visual sensors, an optical flow field (the field representing the apparent motion of the brightness pattern) is used to approximate the actual image plane velocity field. The techniques in this category typically need the assumption that the interframe motion is small and the intensity function is smooth and well-behaved [l-51. (2) Discrete approaches. The techniques in this category treat the images as samples of the scene taken at discrete times, and select discrete features as tokens that are to be matched. Points with high intensity variation are often used as the matching tokens [6,7]. Other features used for matching include closed contours of zero crossings of Laplacian-of-Gaussian images to compute velocity field [8], edges for stereo matching [9-121, lines for stereo matching [13], correlation of intensity patterns [14,15], local phase information [16-181, or some aspects of higher level scene structure [19-211. Discrete approaches allow either small motion or relatively large motion. The image matching algorithm presented here belongs t o the discrete approach, but it takes the advantages of implicit matching which is common in continuous approaches. It associates multiple attributes with the images t o obtain a n overdetermined system of matching constraints. This accommodates, to varying degrees, image noise and slight variations in image intensity that result from changes in viewing position, lighting, shading, reflection, etc. More importantly, the displacement vectors are determined in this overdetermined system without resorting to smoothness constraint. A multi-resolution multi-grid computational structure is employed to deal with relatively large image disparities caused by large interframe motions. The approach is capable of dealing with uniform non-textured object surfaces that are often present in real world images. We also address the problem of discontinuities in the field of displacement, and occlusion. The algorithm computes the displacement field and occlusion maps along a dense pixel grid, based on two perspective images of a scene. This algorithm has been tested on images of real world scenes which contain significant occlusions and depth discontinuities [22]. 1.2. 3 - 0 Motion Estimation 3-D motion estimation has been investigated also under two types of approach: discrete and continuous. In the discrete approaches, the motion is treated as a displacement from one time instant to the next. Therefore, the time separation between the two time instants can be either long or short. The parameters of interframe motion are called two-view motion parameters. The result of image matching, used as input for 3-D motion estimation, is given as the displacement vectors between the
2.5 3-D Motion Analysis from Image Sequences 341 corresponding image points. In the continuous approaches, the interframe motion is approximated by motion velocity and, therefore, in order for such an approximation to be reasonable, the interframe motion must be very small. The 3-D motion is formulated as velocity. The result of image matching, which is needed as input for motion estimation, is given as optical flow. Since the discrete approach more accurately models what actually happens than the continuous approach, we present a discrete approach in this chapter. The possibility of recovering the 3-D motion and structure of a scene from its monocular views has been known t o photogrammetrists for quite long. This subject attracted investigations in the computer vision area around the early 80’s. A few iterative algorithms were proposed [23-271. One drawback of these iterative algorithms is that the solution is not guaranteed because the iterative search may be trapped at local extrema. Two linear algorithms were developed independently by Longuet-Higgins [28], and Tsai and Huang [29]. The linear algorithms guarantee a unique solution if certain nondegeneracy conditions are met. Yen and Huang [30] reported a vectorgeometric approach to this problem. Because these algorithms were designed primarily for noise-free cases, a high sensitivity to noise has been reported [29,31]. In the framework of continuous approach, closed-form solutions for motion velocity from optical flow have been presented by Zhuang et al. [32,33] and Waxman et al. [34]. Since then, improvements have been made in reducing the sensitivity to noise while still keeping the algorithm linear. The post-consideration of the constraint in E through a constrained matrix fitting was independently reported by Faugeras, Lustman and Toscani [35], and Weng, Huang and Ahuja [36]. The latter algorithm is almost the same as the one presented in this chapter (Section 3). It eliminates the need t o compute three false solutions and also uses other measures to improve the stability of the solution. While the above linear algorithms require the solution of only linear systems and therefore no iteration is needed, further improvement of the solution requires a globally optimal nonlinear solution and thus a nonlinear algorithm. The optimal solution is presented in Section 3.4.
1.3. Motion Modeling and Prediction The trajectory of a moving object can be used to understand the motion pattern and predict future motion. Section 4 presents a framework for motion modeling, understanding and prediction. Based on dynamics, a locally constant angular momentum (LCAM) model is introduced [42]. The model is local in the sense that it is applied to a limited number of image frames at a time. Specifically, the model constrains the motion, over a local frame subsequence, to be a superposition of precession and translation. Thus, the instantaneous rotation axis of the object is allowed to change with time. The trajectory of the rotation center is approximated
342
J. J. Weng & T. S. Huang
by a vector polynomial. The parameters of the model evolve in time so that they can adapt to long term changes in motion characteristics. Based on the assumption that the motion is smooth, object position and motion in the near future can be predicted, and short missing subsequences can be recovered. 2. Image Matching Two images, I and 1’,are two functions: i: U -+ B and i‘: U -+ B , where U is a subset of 2-D space, and B is a subset of l-D space for monochrome images (3-D space for color images). U defines the image plane. Functions i and i’ map each point in the image plane to an intensity value. The occlusion map, 0 for image I consists of those image points in image I whose corresponding points are not visible in image 1’. Similarly, we define the occlusion map, 0’ for image 1’. An image matching from I to I’ is a mapping K : U + U such that for any u E U - 0 (the symbol “-” denotes set subtraction), u and K(U) are the projections of the same point of the scene onto images I and 1’, respectively. Notice that the mapping from an occluded point u E 0 is arbitrary. Similarly we define K’ as the matching from I’ to I. The displacement field is defined by d = K - e , where e is a n identity mapping e(u) = u for all u in 2-D space. Therefore, the matched image point for u is u’ = K(U) = u d(u). We will use the term “displacement field” to refer to the result of image matching.
+
2.1. Image Attributes Image attributes are defined for image matching: the matching points should have similar attributes. Some simple image attributes are defined in the following. Image intensity is a simple image attribute. Under certain conditions, e.g. from matte surfaces illuminated by extended light sources, the image intensity value of a scene is in fact quite stable under motion. However, if matching is based on intensity only, a point can be matched to any point with the same or similar intensity. Although intensity may vary, certain relationships among intensity values of nearby points may be relatively stable. These relationships provide some structural information of the scene. A candidate of such image structural information is a sharp transition of intensity, or edge. To get a continuous measure of edgeness, we define edgeness as the magnitude of the gradient of intensity, namely, e = IlVill. Similarly, the edgeness and intensity are generally not sufficient to reliably determine the correct match. The features that relate t o the shape of the local intensity surface are useful in distinguishing otherwise similar looking points. For example, different points on the border of a region may have the same intensity and edgeness values, but the local border shape may vary from point to point on the border. A point at a geometrical corner may be clearly distinguished from others. The cornerness or the curvature of a region border thus can be used as a matching
2.5 3-0 Motion Analysis from Image Sequences 343 criterion. We define the cornerness in the following way that does not use computationally expensive polynomial fitting but achieves very good performance on real world images. As we mentioned earlier, we define positive and negative cornerness separately. Roughly speaking, the edgeness at a point u measures the change in the direction of the gradient at two nearby points, weighted by the gradient at the point. These two points, u ra and u rb are located on a circle centered at u. The radius of the circle is determined by the level of image resolution. We choose ra and rb such that the directional derivative along the circle reaches the minimum and the maximum values, respectively. Let a = V i ( u ra), b = V i ( u +rb), and angle (a, b) be the angle from a to b measured in radians counter-clockwise, ranging from -T to T . The closer the angle is to ~ / 2 the , higher the positive cornerness measure should be. In addition, the measure should be weighted by the magnitude of gradient at the point u since the direction of the gradient in a uniform region is very unreliable. Mathematically, the positive cornerness and negative cornerness are defined, respectively, by
+
+
+
e ( u ) ( l - 11 - 2 / angle(a, ~ b)l)
0 5 angle(a, b) 5 otherwise
T
(2.1)
and n(u) =
e ( u ) ( l - 11
+2
/ angle(a, ~ b)l)
-T 5 angle(a, b) 5 0 otherwise
where column vectors a and b are intensity gradients at u tively:
where ra and rb are such that llrall = llrbll
=T
(2.2)
+ ra, and u + rb, respec-
and
and
where the superscript “I” denotes the corresponding perpendicular vector: if r = ( T ~ T, , ) ~ , then r l = (-T,, T , ) ~ . If ra and rb that correspond t o the minimum in (2.3) and the maximum in (2.4), respectively, are not unique, we choose those that minimize p(u) in (2.1) and (2.2), in addition to satisfying (2.3) and (2.4). The cornerness is defined only if the involved derivatives exist. The value of T is a
344 J. J. Weng d T. S. Huang
parameter of cornerness, and is directly related to image resolution. In the discrete version, T is equal to the pixel size. The framework described below does not depend very much on the type of image attributes used. Different image attributes can be used according to the actual applications. The attributes defined in this section are planar rigid motion invariant (PRMI) in the sense that if the image is rigidly moved in the image plane, the attributes defined at the two corresponding points (before and after the 2-D motion) have the same value. 2.2. Smoothness Smoothness constraints impose some similarity of the displacement vectors over a neighborhood. In addition t o considering the smoothness of the overall displacement vectors, we separately consider the smoothness of the orientation of these vectors. The reason for emphasizing orientation smoothness is that (1) the orientation of the displacement vectors projected from a coarse level is generally more reliable than their magnitude, and (2) at a fine level, the local attribute gradient perpendicular to the displacement vector can easily lead the displacement vector in a wrong direction if the orientational smoothness is not emphasized. Clearly, the smoothness constraint should be enforced only over points whose displacements are related, e.g. over adjacent points from the same surface. To selectively apply the smoothness constraint to two points, we use the similarity of intensities and the similarity of available displacement vector estimates at the two points. We represent the displacement vector field in the vicinity of a point uo by a vector which is intended to approximate the displacement field within the region that uo belongs to. In the implementation, ~ ( u o )is computed as
a(,,), -
JII’
d(uo) = w(i(u) - ~ ( u o ) ,d(u) - d(uo))d(u)du o< 1 Iu-uo I I 2 1 , f(x) x 255, to limit strong edges and relatively enhance the moderate edges. The values of two transition points zo and 2 1 are determined automatically through an analysis of the histogram of gradient magnitudes such that the fractions of the pixels in the edgeness images that have values below f(z0) and above f(z1) are maintained a t predetermined levels.
350
J. J. Weng & T. S. Huang
Fig. 3. Two normalization functions for t h e edgeness.
The edgeness e(u) used in the definition of cornerness, (2.1) and (2.2), should also use the modified definition (2.10) as well. Note that such modified edgeness and cornerness are still PRMI attributes. The preprocessing and normalization steps enable the algorithm to perform consistently for a wide variety of images using a set of standard parameters, which are selected based on a moderate number of image examples. In the implementation, the parameters can be determined through trials. A set of parameters, e.g. those in (2.6) or (2.7) and (2.9), are determined for each level of resolution. At coarse levels, the edgeness, cornerness and smoothness have relatively large weights. Their weights are reduced gradually down to finer levels, since smoothness constraint should be reduced at finer levels where details of the displacement are obtained, and cornerness and edgeness measurements a t finer levels are more susceptible to noise than significantly blurred measurements. The original images are first preprocessed by the methods discussed above. Then as shown in Fig. 1, four pairs of attribute images are generated (intensity, edgeness, positive cornerness and negative cornerness). The attribute images are extended in four directions to provide context for the points that are near the image border. The extension is made by replicating the border row or column. We use recursive blurring to speed up computation. Only integer summations and a few integer divisions are needed to perform such a simple blurring. The blurring for level 1 1 is done using the corresponding attribute image at level 1: For each pixel at level I + 1, the value is equal t o the sum of the values of four pixels at level 1 divided by k ( k = 4 for the intensity, k = 3 for the edgeness and k = 2 for the cornerness). The locations of these four pixels are such that each is centered at a quadrant of a square of a x a. a is equal to 2' at level 1. Therefore, the blurred intensity image at level I is equal to the average over all pixels in a square of size a x a. To enhance sparse edges and corners, k is smaller than 4 for the edgeness
+
2.5 3 - 0 Motion Analysis f r o m Image Sequences 351
and the cornerness. So, the results can be larger than 255. If this occurs, the resulting value is limited t o 255. This multilevel recursive normalization is useful for the algorithm t o adapt to different scenes. 2.6. Occlusion
To correctly match two images, those scene regions which are occluded in one or the other image must be identified. Occlusion occurs when a part of the scene visible in one image is occluded in the other by the scene itself, or a part of the scene near the image boundary moves out of the field of view in the other image. If the occluded regions are not detected, they may be incorrectly matched to nearby regions, interfering with the correct matching of these regions. To identify the occluded regions, we define two occlusion maps, occlusion map 1 showing parts of image 1 not visible in image 2, and similarly occlusion map 2 for image 2 (see Fig. 4, where black areas denote the occluded regions). We first determine the displacement field from image 2 to image 1, without occlusion information. The objective of this matching process is t o compute occlusion map 1. This matching may “jam” the occluded parts of image 2 (e.g. the right-most section in Fig. 4) into parts of image 1 (e.g. the right-most section in Fig. 4). This generally will not affect the computation of the occlusion map 1, since the occluded regions of image 1 may only occur on the opposite side across the “jammed” region (in Fig. 4,e.g. the occluded region of image 1 is to the right of a “jammed” region). Those regions in image 1 that have not been matched (in Fig. 4,no arrows are pointing to them) are occluded in image 2 and are therefore marked in occlusion map 1 (black in Fig. 4). These unmatched patches may also be located at the center of the images, if they are occluded by other parts of the scene. Once the occlusion map 1 is obtained, we then compute the displacement field from image 1 to image 2 except for the occluded regions of image 1. The results of this step determine occlusion map 2 (see Fig. 4).
Occlusion map 1
Occlusion map2
Fig. 4. A 1-D illustration of determining occlusion maps (see text). Images are represented by lines as one-dimensional images. The displacement fields shown just illustrate the correspondences between two 1-D images, and are not the actual displacement fields.
352 J. J. Weng €4 T. S. Huang
From the definition of K and K ' , it is clear that K. and K.' are one-to-one correspondences from U - 0 t o U - 0', and from U - 0' to U - 0, respectively. Therefore, the occlusion map 0 can be determined by 0 = U - K'(U - O'), and similarly 0' = U - n(U - 0). However, this procedure is recursive. Once one occlusion map is determined, the other can also be determined. The procedure outlined in Fig. 4 used preliminary K.' that is computed t o determine 0 without information about 0'. Since regions in 0 and 0' are generally far apart, this preliminary K.' may be good enough to determine 0. 2.7. Outline of the Algorithm
The following summarizes the steps of the procedure that computes the displacement field from one image t o the other (see Fig. 1): (1) Filter the two images using a 3 x 3 low pass filter to remove noise and normalize the pair of images as described in Section 2.5. (2) Compute the image attributes: intensity, edgeness, positive cornerness and negative cornerness as described in Sections 2.1 and 2.5. ( 3 ) Set the level to the highest e.g. 1 = 6, and set the displacement field on the grid of level 6 to zero. (4) Blur the attribute images to level 1 as described in Section 2.5. The scale of the blurring filter at level I is 2'. ( 5 ) Compute the displacement field along the grid. Perform a number of iterations as discussed in Section 2.3. (6) If I = 0, the procedure returns with the resulting displacement field. Otherwise go to 7. (7) Project the displacement field on the grid of level 1 to the grid of level 1 - 1 (replicating the vector at each grid point t o the four corresponding grid points of level 1 - 1);decrement 1 by one and go to (4). Suppose we need to determine the displacement field from image 1 t o image 2. In order to obtain occlusion map 1, first compute the displacement field from image 2 to image 1 using the above procedure, without occlusion information (assuming image 2 has no occluded region in image 1). The displacement field computed is used to determine the occlusion map 1 for image 1. In the implementation, the occlusion maps are filtered by 3 x 3 median filters to remove single-pixel-wide occlusion and noise. Then, the displacement field from image 1 to image 2 is computed using the occlusion map 1 by calling the above procedure starting from step (4).In step ( 6 ) , if a point in image 1 is marked in the occlusion map 1, it is not visible in image 2, and so, the displacement vector from this point cannot be determined. We just copy the vector d to this occluded point. The final computed displacement field assigns a displacement field t o every pixel in image 1. In summary, the matching algorithm computes a displacement vector for every pixel in the source image. This vector points t o the matching point in the target
2.5 3-0 Motion Analysis f r o m Image Sequences
353
image. Since subpixel precision is used, the two components of the displacement vector use real value prepresentation. Mathematically, this is equivalent to computing point correspondences: For each point in the source image, the algorithm determines the corresponding point in the target image so that these two points are projections of the same point in the scene.
3. Motion Estimation This section first presents a linear algorithm that exploits redundancy in the available data to improve accuracy of the solution. Then, the optimization is discussed. We first define a mapping [.Ix from a 3-D vector to a 3 x 3 matrix:
0 [(Xl, 5 2 ,
x3y1x =
-53
53 [ L X 2
52
0
-x1]
51
0
(3.1)
Using this mapping, we can express cross operation of two vectors by the matrix multiplication of a 3 x 3 matrix and a column matrix:
x x Y = [XIx Y .
(3.2)
3.1. Problem Statement
Let the coordinate system be fixed on the camera with the origin coinciding with the projection center of the camera, and the z-axis coinciding with the optical axis and pointing toward the scene (Fig. 5 ) . Since we are only interested in the ratio of image coordinates to the focal length and one can always measure the image coordinates in the unit of focal length, we assume, without loss of generality, that the focal length is unity. We call such a camera model normalized camera model. Thus, in the normalized camera model, the image plane is located a t z = 1. Visible objects are always located in front of the camera, i.e. z > 0. Notice that 0 < z < 1 may occur since the camera model is normalized. Consider a point P on the object which is visible at two time instants. The following notation is used for the spatial vectors and the image vectors. x = (5, y, z ) ~ spatial vector of P at time t l ; x' = ( X I , y', z ' ) ~ spatial vector of P a t time t 2 ;
x = ( u , w, l ) t = X' = (u', w', ly
=
image vector of P at time t l ;
(5,2,l) Y'
t
image vector of P at time
t2;
where ( u , w) and (u', w') are the image coordinates of the point. Therefore, the spatial vector and image vector are related by
x =zx,
x' = z'x' .
354
J. J. Weng & T. S. Huang
Figure 5 shows the geometry and the camera model of the setup. From the figure we can see that the image of a point determines nothing but the projection line, the line that passes through the point and the projection center. The direction of this projection line is all that we need, and the position of the image plane is immaterial. That is why we can normalize the focal length to unity. It is obvious that the model in Fig. 5 is not meant to describe the optical path in a conventional camera. But rather, it is a simple geometrical model that is mathematically equivalent to an ideal pin-hole camera. A conventional camera can be calibrated so that every point in the actual image plane can be transformed to a point in the image plane of this normalized model. X
Fig. 5 . The geometry and the camera model of the setup.
Let R and T be the rotation matrix and the translational vector, respectively. The spatial points at the two time instants are related by x’=Rx+T or for image vectors: z’X’=zRX+T.
If IlTll
# 0, from
(3.3)
(3.3) we get
(3.4)
where
Given n corresponding image vector pairs at two time instants, Xi and Xi, i = 1, 2, . . . , n, the algorithm solves for the rotation matrix R. If the translation vector T does not vanish, the algorithm solves for the translational direction represented by a unit vector T and the relative depths and for object points xi and llTll IlTll
2.5 3-D Motion Analysis from Image Sequences 355
x:, respectively. The magnitude of the translational vector llTll, and the absolute depths of the object points, zi and zi, cannot be determined by monocular vision. This can be seen from (3.4), which still holds when 11T11, zi and z: are multiplied by any positive constant. In other words, multiplying the depths and llTll by the same scale factor does not change the images.
3.2. Algorithm We shall first state the algorithm, and then justify each of the steps.
Step (2). Solvingfor E. Let Xi = (ui, vi, l ) t ,Xi = (u:,v;,l ) t ,i be the corresponding image vectors of n (n 2 8) points, and u1u: u2u;
A = [
u1v: u2v;
u1
v1u:
u2
u2u;
v1v: v2v;
211
v2
u; u;
. .
~
unu;
unvh
un vnu;
V ~ V ; V,
=
1, 2,
v; 1 v; 1 . .],
U;
.
.
. . . , n,
(3.5)
V; 1
and
h = (hi,
h3, h4, h5, h6, h7, h8, h s ) t .
(3.6)
We solve for unit vector h in min h
11 A h 11 , subject to: 11 h II=
1.
(3.7)
The solution of h is a unit eigenvector of AtA associated with the smallest eigenvalue. (Alternatively, the above problem can be transformed to a linear least squares problem by setting a nonzero nonvanishing component of h to one and moving the corresponding column to the right hand side.) The matrix E is determined by
Step ( i i ) . Determining a unit vector T, with T = f T , . Solve for unit vector T, in min 11 EtT, 1 1 , subject to: 11 T, II= 1 . (3.9) TS
The solution of T, is a unit eigenvector of EEt associated with the smallest eigenvalue. If (3.10) (T, x X i ) . (EXi) < 0 , i
356
J. J. Weng & T. S. Huang
then T, t -T,. The summation in (3.10) is over several values of i's to suppress noise (usually three or four values of i will suffice).
Step (iii). Determining rotation matrix R. Without noise, it follows that E
=
[TSIXR
(3.11)
or
Rt[-T,].
= Et
.
(3.12)
In the presence of noise, we find rotation matrix R in
)I Rt[-Ts]x - Et 11 ,
subject to: R is a rotation matrix.
(3.13)
Alternatively, we can find R directly: Let
W = [Wi Wz WJ] = [El x T, Ez x E3 Ez x T,
+
Without noise, R such that min R
=
+ E3 x El
E3 x T, + E l x Ez] . (3.14)
W. In the presence of noise, we find rotation matrix R
11 R - W 11,
subject to: R is a rotation matrix.
(3.15)
We can use either (3.13) or (3.15) t o compute R. They both have the form min R
11 RC - D 11 , subject
to: R is a rotation matrix
(3.16)
where C = [C, Cz C,], D = [D1 Dz D3]. The solution of (3.16) is as follows: Define a 4 x 4 matrix B by (3.17)
where (3.18)
2.5 3-D Motion Analysis from Image Sequences
S t e p (iv). Checking T # 0. If T # 0, d e t e r m i n e t h e sign of T . Let small threshold ( a = 0 without noise). If
II x: x RXi II < I1 x!,II I1 xi I1 for all 1 5 i 5 follows. If
TI,
then report T
= 0.
(Y
357
be a
(Y
Otherwise determine the sign for T as
C(T,x X:) . ( X i x RXi) > 0 ,
(3.20)
i
then T = T,. Otherwise T = -T,. Similar to (3.10), summation (3.20) is over several values of i.
S t e p (w). IfT depth
#
0, e s t i m a t e relative depths. For i , 1
5 i 5 n, find relative (3.21)
to minimize
11 [x: - RXiIZi - T 11
(3.22)
using a standard least squares method for linear equations. A simple method to correct structure based on rigidity constraint is as follows. The corrected relative 3-D position (scaled by 11 T 11-l) of point i at time t 2 equals to XI = (R(&Xi) T ZiX:)/2. Its relative 3-D position (scaled by 11 T 11-1) at time tl equals to j i i = R - ~ ( X ;- TI.
+ +
3.3. Justification of the Algorithm We now justify each step of the algorithm.
For S t e p ( i ) . Let T, be a unit vector that is aligned with T, i.e.
T, x T = O .
(3.23)
Pre-crossing both sides of (3.4) by T, we get, using (3.1) and (3.2), 2’
-T, IlTll
x
XI
=
z -[T,]xRX. IlTll
(3.24)
Pre-multiplying both sides of (3.24) by XIt (inner product between vectors), we get:
X’t[T,] RX = 0
(3.25)
since X’t(T, x XI) = 0 and z > 0. Geometrically, (3.25) means that three vectors X’, T, and RX are coplanar, which can be seen from (3.3). Define E to be
E = [T,],R = [T, x
R1
T, x R2 T, x Rs] = [El E2 E3]
(3.26)
358
J. J. Weng & T. S. Huang
where R = [R1R2 R3]. From the definition of T,, the sign of E is arbitrary since the sign of T, is arbitrary (as long as the sign of T, and that of E match such that (3.26) holds). Using (3.26), the definition of E, we rewrite (3.25) as
XIt EX
=0
.
(3.27)
Our objective is to find E from the image vectors X and X’. Each point correspondence gives one equation (3.27) which is linear and homogeneous in the elements of E . n point correspondences give n such equations. Let
Given n point correspondences, we rewrite (3.27) as linear equations in the elements of E and get AE=O, (3.28) where the coefficient matrix A is given in (3.5). In the presence of noise, we use (3.7). The solution of h in (3.7) is then equal to E up to a scale factor provided rank ( A ) = 8. The rank of the n x 9 matrix A cannot be larger than 8 since E is a non-zero solution of (3.28). Longuet-Higgins [37] gives a necessary and sufficient condition for the rank of A t o fall below. Assuming the relative motion is due to motion of the camera, the condition is that the feature points do not lie on any quadratic surface that passes through the projection center of the camera at the two time instants. To satisfy this condition, at least eight points are required. More points are needed to combat noise. Since the sign of E is arbitrary, we need only to find the Euclidean norm of E to fully determine E (equivalently E) from h. Let Ts = (s1, s2, ~ 3 ) Noticing ~ . T, is a unit vector and using (3.26), we get
llE1I2 = trace{EEt} = trace{ [Tslx R([Tsl x RIt } = trace{[TsIx
([Tsl.)t}
= II[Tslx112 2 = 2(s1
So, E
=
2 + sa2 + s3) = 2.
a h . This gives (3.8).
For Step (ii). We determine T,. From (3.26), T, is orthogonal to all three columns of E . We get EtT, = 0 . With noise, we use (3.9). It is easy to prove that the rank of E is always equal to 2. In fact, let Q 2 and Q3 be such that Q = [T, Q 2 Q3] is an orthonormal 3 x 3 matrix. S = R t Q is then also orthonormal. Post-multiplying the two sides of the first equation of (3.26) by S , we get
ES = [TsIxRS = [ T s I x Q
=
[O Ts x
Q2
Ts x Q3].
2.5 3 - 0 Motion Analysis from Image Sequences 359
We see the second and the third columns of ES are orthonormal, according to the definition of Q. Thus, rank { E } = rank { E S } = 2. Since rank { E } = 2, the unit vector T, is uniquely determined up to a sign by (3.9). To determine the sign of T, such that (3.26) holds, we rewrite (3.24) using
E
=
[T,] R: -I
I
L T , x X' = L E X . (3.29) IlTll IlTll Since z > 0 and z' > 0 for all the visible points, from (3.29) we know the two vectors T, x Xi and EXi have the same directions. If the sign of T, is wrong, they have the opposite directions. Thus, if (3.10) holds, the sign of T, should be changed. For Step (iii). In steps (i) and (ii) we found E and T, that satisfy (3.11). R can be determined directly by (3.14). We now prove W in (3.14) is equal to R without noise:
R = [Ri R2 R3] = [El x T, + E2 x E3 E2 x T,
+ E3 x El
E3 x T, + E l x Ez] .
Using the identity equation (a x b) x c = ( a . c)b - (b . c)a and (3.26), we get
This proves that the first column of R is correct. Similarly we can prove that the remaining columns of R are correct. In the presence of noise, however, the estimated E has errors, and so does the matrix determined by (3.14). In particular, W in (3.14) does not give a rotation matrix in general. For the same reason, generally, one cannot find a unit vector T, and a rotation matrix R so that [T,], R = E if E has errors. This can be understood by considering degrees of freedom in a correct E (3 for rotation and 2 for a unit T,), which is smaller than the degrees of freedom, 8, in a unit h in (3.7). In other words, in solving for h in (3.7), we neglect the constraint in h. This is necessary to be able to derive a linear algorithm. The alternative steps, (3.13) and (3.15), re-consider such a constraint through matrix fitting. To solve the problem of (3.16), we represent the rotation matrix R in terms of a unit quaternion q. R and q are related by Eq. (3.19). We have [38]
llRC
-
Oil2 = q t B q
(3.30)
360
J. J. Weng & T. S. Huang
where B is defined in (3.17) and (3.18). The problem of (3.16) is then reduced to the problem of minimization of a quadratic. The solution of the unit vector q in (3.30) is then a unit eigenvector of B associated with the smallest eigenvalue. Note that R is uniquely determined in (3.12), since the rank of [-T,]. is two and the positions of any two non-collinear vectors completely determine a rotation: If RX1 = Y1, RX2 = Y2, and X1 x X2 = 0, then we have the third equation:
R(X1 x X2) = Y1 x Y2, and [XI X2
X1 x X,] has a full rank.
For Step (iv). Pre-crossing both sides of (3.3) by X’, we get 0 = zX’ x RX+ X’ x T .
(3.31)
If T = 0, for any point X’ we have (note z > 0) X ’ XR X = 0 .
(3.32)
If T # 0, X’ x T # 0 holds for all the points X’ (except at most one). Therefore, (3.32) cannot hold for all points by virtue of (3.31). In the algorithm, we normalize the image vectors in (3.32) and give a tolerance threshold a in the presence of noise.
From (3.31), if T = T, then T, x X’ and X‘ x RX have the same directions. Otherwise they have opposite directions since T = -Ts.We use the sign of the inner product of the two vectors in (3.20) t o determine the sign of
T.
For Step (v). The equations for the least-squares solution (3.21) follow directly from (3.4). The idea for correcting structure based on rigidity is as follows. Moving the recovered 3-D points at time t l using the estimated rotation and translation, their new positions should coincide with the recovered position at time t 2 , if the data is noise free. However, in the presence of noise, the positions do not coincide. Here we adopt a simplistic way of removing this discrepancy: the midpoint between these two positions of a point at time t2 is chosen as the corrected solution for the position of the point at time t 2 . Moving the midpoint back gives the corrected 3-D position of the point at time t l .
T
In summary, we have proved that if rank ( A ) = 8, the solution of R and is unique, and we have derived the close-form solution. Given eight or more point correspondences, the algorithm first solves for the essential parameter matrix E. Then the motion parameters are obtained from E. Finally the spatial structure is derived from the motion parameters. All the steps of the algorithm make use of the redundancy in the data to combat noise. As the results of determining the signs in (3.10) and (3.20), the computations of three false solutions [28,29] are avoided. These steps for determining signs are stable in the presence of noise, since the decisions are made based on the signs of the inner product of the two vectors which
2.5 3-0Motion Analysis from Image Sequences 361 are in the same or opposite direction without noise. Summations over several points in (3.10) and (3.20) suppress the effects of the cases where two noise-corrupted small vectors are used, whose inner products are close t o zero and the signs are unreliable. If T # 0 and the spatial configuration is nondegenerate, the rank of A is 8. In this case, we can determine the unit vector h in (3.7) up to a sign, and determine R and % uniquely. If T = 0, any unit vector T, satisfies (3.24) and so matrix E , and correspondingly the unit vector h, have two degrees of freedom (notice T, and h are restricted to be unit vectors). Therefore, A in (3.5) has a rank less than or equal to 6 . If T = 0, relative depths of the points cannot be determined. However, the rotation parameters can be determined even if T = 0.
3.4. Optimal Motion Estimation The optimization is motivated by the following observations on the linear algorithms (including the one presented in Section 3.2): (a) With certain types of motion, even pixel level perturbations (such as digitization noise of conventional CCD video cameras) may override the information characterized by the epipolar constraint, which is a key constraint used for determining motion and structure by linear algorithms. The epipolar constraint restricts only one of the two components in image point displacement. The other component is related to the depth of the point and the motion. If this component is also used for motion estimation, the accuracy of the estimated motion parameters can be considerably improved. (b) Existing linear algorithms give closed-form solution to motion parameters. However, the constraints in the intermediate parameter matrix (essential matrix E ) are not fully used. The use of these constraints can improve the accuracy of the solution in the presence of noise. The above considerations are unified under a general framework of optimal estimation: Given the noise-contaminated point correspondences, we need the best estimator for motion and structure parameters. In reality, the image coordinates of a n object point as well as the corresponding displacement vector in the image plane are the results from a feature detector and the corresponding matcher, whose accuracy is influenced by a variety of factors including lighting, structure of the scene, image resolution and the performance of the feature matching algorithms. Thus, the observed 2-D image plane vectors u, of image 1 and u: of image 2 are noise-contaminated versions of the true ones. Let (uz,u:) be the observed value of a pair of random vectors (U,, Ui). (With n point correspondences over two time instants, we add subscripts a t o denote the ith point. A subscript-free letter denotes a general example of the vectors.) What we obtain is a sequence of the observed image vector pairs
362
J. J. Weng €4 T.S. Huang
of a sequence of random vector pairs
u 4 i (u;,(Ui)t, u;, (UL)t, . . . , u;,
(UL)t)>".
We need to estimate the motion parameter vector m and the 3-D positions of the feature points (scene structure)
(xi,
'X
I
t t .
xi, ( 4 1 t 7 ... 7 xt,,(Xn)
We assume that the errors are uncorrelated between the different components of a point and between different points. Let h,(m, x) be the noise-free projection of the ith point in the first image, given motion m and structure x, and h',(m,x) be the corresponding projection in the second image. Then, according to the principle of minimum variance estimator, the optimal estimate of m and x is the one that minimizes n
C(llU2 2=
+ 114-
- hz(m, x)l12
(3.33)
x)l12)
1
which is just the sum of discrepancies between the observed projection and the inferred projections. The value of (3.33) measures the differences in the observed images and the inferred images. Equation (3.33) involves both motion parameters and the 3-D position of every feature point. The maximum is over all the possible motion parameters and scene structures. The parameter space for iteration is huge and the computation is very expensive. However we do not have to iterate on the structure of the scene. In fact, given motion parameters m, the structure x that minimizes the value of (3.33) can be estimated analytically. That is, we can compute m;tn{llu2 - h,(m, x)ll2
+ llu: - h:(m,
(3.34)
x)l12} i? g2(m)
from a given m. In fact,
+ llui
= min( m C m i Xn { l ( u i - hi(m, x)(I2 i= 1
-
h~(m,x)l12}} = m$xgi(m). i= 1
So computationally, structure x will not be included in the parameter space of iteration. Given an m, x is computed directly as we will discuss in the following paragraph. This drastically reduces the amount of computation. Otherwise it is computationally extremely expensive to iterate on this huge (m, x) space (iterations on n points needs (3n 5)-dimensional parameter space!). Since the optimal structure x can be determined from motion parameters, we can exclude x from the notation for parameters t o be estimated. That is, symbolically, the parameters to be determined are just m.
+
2.5 3-D Motion Analysis from Image Sequences 363
To derive the closed-form expression for x that gives g2(m)in (3.34), we use the following methods. From motion parameter vector m and the observed projections of point i, the two observed projection lines are determined. These two observed projection lines do not intersect in general. If the true 3-D point is on the observed projection line of the first image, the discrepancy ((u,- hz(m,x)1I2 is equal to zero, but llul - h’,(m,x)l12generally is not. If the true 3-D point is on the other observed projection line, llu: - h’,(m,x)(I2is equal to zero while I(u,- h,(m, x)1I2 is not. Given the motion parameters, we need to find a 3-D point for each feature point such that the corresponding term IJuz- hz(m,x)1I2 llu: - h’,(m,x)1I2 is minimized. Obviously, under a normal configuration the point lies in the shortest line segment L that connects the two observed projection lines, because otherwise the perpendicular projection of a 3-D point onto L is better than the 3-D point. An exact solution of the optimal point requires solving a fourth order polynomial equation. It can be shown that using a reasonable approximation, we can get a closed-form solution. The optimal point is generally not far from the midpoint of the line segment L, unless the distance to the object and the viewing angle differ a lot between two images. For computational efficiency, we may just use the midpoint of the line L as an approximated optimal point, which is the solution in (3.22). Computationally, a two-step approach is proposed here. First, a linear algorithm is applied which gives a closed-form solution. Then in the second step, this solution is used as an initial guess for an iterative algorithm which improves the initial guess to minimize the objective function (3.33). This two-step approach has the following advantages:
+
(1) A solution is generally guaranteed. The linear algorithm always gives a solution provided that degeneracy does not occur. Unless the noise level is very high, this solution is close to the true one. As long as the initial guess is within the convergent region to a globally optimal point, iteration leads to the optimal solution. (2) The approach yields reliable solutions. The linear algorithms use only the epipolar constraint and so, the solution is sensitive to noise and the reliability of solutions varies with motion types. The optimization in the second step employs more global constraints and achieves significant improvements over the first step. (3) The computation is faster than straight iterative methods that start with a “zero” initial guess. Generally, a linear algorithm is fast, and a nonlinear algorithm is slow. When a linear algorithm is followed by a nonlinear algorithm, the amount of computation is not simply equal to the sum of those needed by each algorithm individually. Since the linear algorithm provides a good initial guess, the time taken by the nonlinear algorithm to reach a solution is greatly reduced.
364
J. J. Weng 63 T. S. Huang
4. Motion Modeling and Prediction In general, the moving objects exhibit a smooth motion, i.e. the motion parameters between consecutive image pairs are correlated. From this assumption and given a sequence of images of a moving rigid object, we determine what kind of local motion the object is undergoing. A Locally Constant Angular Momentum model, or LCAM model for short, is introduced. The model assumes short term conservation of angular momentum and a polynomial curve as the trajectory of the rotation center. This constraint is the precise statement of what we mean by smoothness of motion. However, we allow the angular momentum, and hence, the motion characteristics of the object to change or evolve over the long term. Thus, we do not constrain the object motion by some global model of allowed dynamics. We will give a closed-form solution to motion parameters and structure from a sequence of images. As a result of the analysis presented in this section, some of the questions that we can answer are: whether there is precession or tumbling; what the precession is if it exists; how the rotation center of the object (which may be an invisible point!) moves in space; what the future motion would probably be; where a particular object point would be located in image frames or in 3-D at the next several time instants; where the object would be if it is missing from an image subsequence, and what the motion before the given sequence could be. This approach of motion modeling and prediction is based on the two-view motion analysis of image sequences consisting of either monocular images, or stereo image pairs. Generally, two-view motion does not represent actual continuous motion undergone by the object between the two time instants. The physical location of the rotation axis is not determined by such a two-view position transformation. Using a a single camera, the 3-D translation and the range of the object can be determined up t o a scale factor. If stereo cameras are used, we can determine the absolute translation velocities and the ranges of object points. The approach presented in this section is independent of the type of algorithms used to determine two-view motion parameters. To be specific, feature points are used for the discussion here. We assume that there is a single rigid object in motion, the correspondences of feature points between images are given, and the motion does not exhibit any discontinuities such as those caused by collisions. 4.1. Motion of a Rigid Body in 3 - 0
We first present the laws of physics that govern the motion of a rigid body. All external forces acting on a body can be reduced to a total force F acting on a suitable point Q, and a total applied torque N about Q. For a body moving freely in space, the center of mass is to be taken as the point Q. If the body is constrained to rotate about a fixed point, then that point is t o be taken as the point Q. That point may move with the supports. Letting m be the mass of the body, the motion
2.5 3-D Motion Analysis from Image Sequences 365
of the center of mass is given by
Let L be the angular momentum of the body. The torque N and the angular momentum L satisfy [39,40]:
dL
N=-. dt The rotation is about the point Q, which will be referred t o as the rotation center. In the remainder of this subsection, we concentrate on the rotation part of the motion. The motion of the rotation center Q will be discussed in the next subsection. In matrix notation, the angular momentum L can be represented by L=Gw or writing in components:
[f.] [ =
Qxx
QYX
Qzx
W X
QXY
QYY
Qzy
WY
sxz
Qyz
Qzz
wz
where
gzx=gxZ=gzy = g y z
=
.I
x ~ d m ,g y x = g x y = - ] ~ v d m ,
-/zYdm.
The above integrals are over the mass of the body. If the coordinate axes are the principal axes of the body [39,40], the inertia tensor G takes the diagonal form:
[gr ] 0
G=
gyy
0
.
(4.3)
Qzz
Referring to a coordinate system fixed on such a rotating body, (4.2) becomes nx
=QXXGX
+ w y w z ( g z z - gyy)
ny = g y y b y
+
n z = QzzWz
+ WxWy(Qyy
W Z W X ( 9 X X -
Qzz)
- Qxx)
1
, ,
where (n,, ny, n,) = N. These are known as Euler’s equations for the motion of a rigid body. These equations are nonlinear and have generally no closed-form solutions. Numerical methods are generally needed to solve them.
366
J. J. Weng & T. S. Huang
Clearly the motion of a rigid body under external forces is complicated. In fact even under no external forces, the motion remains complex. Perspective projection adds further complexity to the motion as observed in the image. However, in a short time interval, realistic simplifications can be introduced. One simplification occurs if we ignore the impact of the external torque over short time intervals. If there is no external torque over a short time, there is no change in the angular momentum of the object. Thus, if we have a dense temporal sequence of images, we can perform motion analysis over a small number of successive frames under the assumption of locally constant angular momentum. Another simplification occurs if the body possesses an axis of symmetry. The symmetry here means that at least two of i,,, iyy, i,, in (4.3) are equal. Cylinders and disks are such examples. Most satellites are also symmetrical or almost symmetrical in this sense. Under the above two simplifications, Euler's equations are integrable [39,40]. The motion is such that the body rotates about its axis of symmetry m, and at the same time the axis rotates about a spatially fixed axis 1. The motion can be represented by a rotating cone that rolls along the surface of a fixed cone without slipping as shown in Fig. 6, where the body is fixed on the rolling cone, the axis of symmetry coincides with that of rolling cone, and the center of mass or the fixed point Q of the body coincides with the apices of the cones. Then, the motion of the rolling cone is the same as the motion of the body. Figure 6 gives three possible configurations of the rolling cone and the fixed cone.
fixed
Fig. 6. The precessional motion of a symmetrical rigid body.
Let w1 be the angular velocity at which the rolling cone rotates about 1, and w, be the angular velocity a t which the rolling cone rotates about its own axis of symmetry m. Then the instantaneous angular velocity w is the vector sum of w~ and w, as shown in Fig. 6. The magnitudes of w, and w1 are constant. Thus,
2.5 3-D Motion Analysis from Image Sequences 367
the magnitude of the instantaneous angular velocity is also constant. This kind of motion about a point is called precession in the following sections and it represents the restriction imposed by our model on the allowed object rotation. A special case occurs when m is parallel to 1. Then w is also parallel to 1. Therefore, the instantaneous rotation axis does not change its orientation in motion. This type of motion is called motion without precession. 4.2. Motion of Rotation Center
The location of rotation center Q ( t ) changes with time. Assume the trajectory of the rotation center is smooth, or specifically, it can be expanded into a Taylor series:
If the time intervals between image frames are short, we can estimate the trajectory by the first k terms. We get a polynomial of time t . The coefficients of the polynomial are three-dimensional vectors. Letting 1 djQ(0) -bj+1, j ! dtj j = O , 1 , 2, . . . , k - 1 , w e h a v e
Qi = bl
+ b2(ti - to) + b3(ti
-
to)2
+ . . . + bk(ti
-
to)"'
.
(4.5)
For simplicity, we assume the time intervals between image frames are constant to. From (4.5) we get
c, i.e. ti = ci
+
Letting aj = cj-lbj, j
=
1, 2,
. . . , k , we get
Equation (4.7) is the model for the motion of the rotation center. The basic assumption we made is that the trajectory can be approximated by a polynomial. If the motion is smooth and the time interval covered by the model is relatively short, Eq. (4.7) is a good approximation of the trajectory. In the sense of dynamics, (4.7) implies that the total force acting on the center of rotation has zero high order temporal derivatives. A polynomial trajectory of center of rotation in (4.7) together with the precession model presented in the previous subsection, gives the complete LCAM model [42]. The model is characterized by locally constant angular momentum, i.e. the angular momentum of the moving object can be treated as constant over short time intervals.
368
J. J. Weng €9 T.5’. Huang
A point should be mentioned here. Though we derive the model from the assumption of constant angular momentum and object symmetry, the condition leading t o such motion is not unique. In other words, the motion model we derived applies to any moving object whose rotation can be locally modeled by such motion: the rotation about a fixed-on-body axis that rotates about a spatially fixed axis, and whose translation can be locally modeled by a vector polynomial. It is important to motivate the kinematics from dynamic conditions. But in reality, many different dynamic conditions may result in the same type of motion. Our goal here is t o understand 3-D motion of an object over an extended time period using the two-view motion analysis of images taken at consecutive time instants. Thus we would first estimate the motion parameters of the moving object from the images taken at two time instants, using the method presented in the previous section. Such motion parameters give the displacement between two time instants and do not describe the actual motion, since the object can move arbitrarily between the two time instants. The displacement can be represented by a rotation about an axis located at the origin of a world coordinate system, and a translation [41]. We have called this displacement two-view motion. Let the column vector po be the 3-D coordinates of any object point at time t o ; let p1 be that of the same point at time t l , R1 be the rotation matrix from time t o t o t i , and T1 be the corresponding translation vector. Then, p o and p1 are related by P I = Rip0 Ti (4.8)
+
where R1 represents a rotation about an axis through the origin. Given a set of point correspondences, R1 and TI can be determined by two-view motion analysis. In the case of monocular vision, the translation vector can only be determined up t o a positive scale factor, i.e. only the direction of T, T = T/llTll, can be determined from the perspective projection. In Eq. (4.8),letting po be at the origin, it is clear that TI is just the translation of the point at origin. For any point Qo, we can translate the rotation axis so that it goes through Qo and rotate p o about the axis at the new location. Mathematically, from (4.8) it follows that pi = Ri(po - Qo)
+ (R1Qo + T i ) .
(4.9)
Compared with (4.8), (4.9) tells us that the same motion can be represented by rotating PO about QOby R1, and then translating by RlQo Ti. Because QO is arbitrarily chosen, there are infinitely many ways to select the location of the rotation axis. This is an ambiguity problem in motion understanding from image sequences. If we let the rotation axis always be located at the origin, the trajectory described by Ri and Ti, a = 1, 2, 3 . . . would be like what is showed in Fig. 7, which is very unnatural. In Fig. 7 the real trajectory of the center of the body is the dashed line. However, neither the rotation nor the translation components show this trajectory. As
+
2.5 3-D Motion Analysis from Image Sequences
369
Y 4
Fig. 7. Trajectory described by
Ri and Ti if the rotation axis is always located at the origin.
we discussed in Section 4.1, the center of mass of a body in free motion satisfies Newton's equation of motion of a particle (4.1). Rotation is about the center of mass (or fixed point if it exists). Thus, motion should be expressed in two parts, the motion of the rotation center (the center of mass or the fixed point), and the rotation about the rotation center. Let Qi be the position vector of the rotation center at time ti, Ri be the rotation matrix from t i - 1 to ti, and T i - b e the translation vector from t i - 1 to t,. From (4.8) it follows that Qi = RiQo Ti,
+
or, -RlQo
+ Qi = TI.
Similarly we get equations for the motion from
ti-1
to t i , i = 1, 2,
. . . , f:
(4.10)
Equations (4.10) give the relationship among the locations of the rotation center, the two-view rotation matrices and the two-view translation vectors. Substituting (4.7) into (4.10), we get
+ a 2 + a3 + . . . + a k = TI , ~ 2 ) a l + (21 ~ 2 ) a Z+ (41 - ~ 2 1 ~ 1+3 . . . + (2"'1-
( I - R1)al
(I -
-
R2)ak= T ~ ,
......
(4.11)
+
+
(I- R f ) a l + ( f -~ (f - 1 ) ~ f ) a z (f21 - (f - 1 l 2 ~ f ) a 3. . . + ( f " ' I - (f - l)k-lRf)ak = Tf . Vector equations (4.11) are referred to as the coefficient equations. Both sides of the equations are three-dimensional vectors. There are f equations in k unknown
370
J. J . Weng €9 T. 5'. Huang
three-dimensional vectors. Let A = (a:, a;, . . . , a",,", T = (Ti, Ti, . . . , T;)t, and D be the coefficient matrix of the unknowns in (4.11). Let the element of D at the ith row and j t h column be the 3 x 3 matrix Dij, i.e. D = [Dij]rxk. We have
D23. . ij-11- (i - l ) j - l R i . We can rewrite the coefficient equations (4.11) as
DA=T.
(4.12)
D and T are determined by two-view motion analysis. The problem here is to determine A, the coefficients of the polynomial in (4.7). 4.3. Solutions of the Coefficient Equation Let f = k in (4.11). Then the matrix D is a square matrix. We wish to know whether the linear equations (4.12) have a solution. If a solution exists, is it unique? If it is not unique, what is the general solution? The solution of the coefficient equations depends on the types of motion, or the rotation matrices Ri and the translation vectors Ti. Let us first consider a simpler case, where k = 2. This means that the trajectory of the rotation center is locally approximated by a motion of constant velocity. Three frames are used in this case. The coefficient equations become
(I - R l ) a l
(I - &)a1
+ (21
-
+
,
(4.13)
= T2.
(4.14)
a 2 = TI
R2)az
Solving for a2 in (4.13) and substituting it into (4.14), we get
(I - 2R1+ R2Ri)ai If I - 2R1 + R2R1 is nonsingular, a1
Then
a2
= (I - 2R1
a1
=
(21 - R2)T1 - T 2 .
(4.15)
can be uniquely determined from (4.15):
+ R2R1)-'((21-
&)TI - T2).
is determined from (4.13): a2
= TI - (I - R l ) a l .
It can be shown [42] that (I- 2R1+ R2R1) is nonsingular if and only if the following two conditions are both satisfied: (1) the axes of rotations, represented by R1 and R2, respectively, are not parallel; (2) neither rotation angle is zero. Condition (2) is usually satisfied if the motion is not pure translation. If condition (1) is not satisfied, the solution of Eqs. (4.13) and (4.14) is not unique and has some structure. To show this, assume the rotation axes of R1 and R2 are parallel. Let
2.5 3-D Motion Analysis f r o m Image Sequences
371
w be any vector parallel to these axes. Because any point on the rotation axis remains unchanged after rotation, we have Rlw = w, R ~ w = w. For any solution a1 and a 2 , a1 cw and a2 is another solution, where c is an arbitrary real constant. Therefore, there exist infinitely many solutions. The following theorem presents the results for the general case.
+
Theorem 1 [42]. In coefficient equations, let f = k . Define Si to be a 3 x 3 matrix :
Define number
ui,j:
Then k
SLT1,
$a1 = 1=1
If S i is not singular, the first equation given by Theorem 1 uniquely determines Then a k , a k - 1 , . . . , a2 can be determined, sequentially, by the second, third, ..., and last equations in Theorem 1. Thus, if S i is not singular, the solution is unique.
al.
Theorem 2 [42]. In the case of rotation without precession, let w be any column vector parallel to the rotation axes, then
s,ow=o,
(4.16)
372
J. J. Weng tY T. S. Huang
and for any vector a (S,Oa).w=O.
(4.17)
Using Theorem 1 gives k
SEal = -
C SLTi.
(4.18)
1=1
In the case of rotation without precession, Eq. (4.16) implies S; is singular. From (4.17), the left-hand side of (4.18) is orthogonal t o w. However if the real trajectory of the rotation center is not exactly a j t h degree polynomial with j 5 k - 1 in (4.7), the right-hand side of (4.18) can be any vector, which may not be orthogonal t o w. This means that no solution exists for Eq. (4.18). If the real trajectory is a j t h degree polynomial with j 5 k - 1, then Eq. (4.18) has a solution by our derivation of (4.18). Since Eq. (4.7) is usually only an approximation of the real trajectory, a least-squares solution of (4.18) can serve our purpose. Let 61 be a least-squares solution of (4.18) which is solved by using independent columns of SE. If the rank of SE is 2, which is generally true for motion without precession, the general solution is then a1 = a 1 cw, where c is any real number. All general solutions { a 1 + cw} form a straight line in 3-D space. From Eq. (4.7), this line gives the location and direction of the two-view rotation axis of the motion between time instants t o and tl. From Theorem 2 it follows that
+
Then
S;-lal
= ~ E - ~ a lSE-2al , = s k0- 2 a 1 ,
.. ., Syal = ~
y a 1 .
Based on the equations given by Theorem 1, the unknowns a k , a k - 1 , . . . , a 2 are determined without knowing the undetermined number c. If the motion is pure translation without rotation, all the rotation matrices Ri, i = 1, 2, . . . , k, are unit matrix 1. SE is the zero matrix. The first three columns of D are zero. a1 cannot be determined by coefficient equations. From Theorem 1, a 2 , a3, . . . , a k , can still be determined by coefficient equations. Because no rotation exists, any point can be considered as a rotation center. Equation (4.7) can be used to approximate the trajectory of any object points. Thus the solutions of the coefficient equations can be summarized as follows. (1) In the case of rotation with precession, the solution of the coefficient equations is generally unique. The trajectory of the rotation center is described by (4.7). (2) In the case of rotation without precession, the general solution of a1 gives the two-view rotation axis of the first two-view motion. All other coefficients a 2 , a3, . . . , a k are generally determined uniquely by Theorem 1. Thus, the twoview rotation axes of all two-view motions are determined by (4.7). Because no precession exists, any point on the rotation axis can be considered as the rotation
2.5 3-0 Motion Analysis f r o m Image Sequences
373
center. This is the meaning of the general solution al. Once a particular point on the rotation axis is chosen as the rotation center, its trajectory is described by Eq. (4.7). There are infinitely many possible “parallel” trajectories of the rotation center depending on which point on the axis is chosen as the rotation center. (3) In the case of pure translation without rotation, a2, a3, . . . , ak can still be determined by coefficient equations. However a1 cannot be determined by coefficient equations. al can be chosen to be the position of any object point at time to. Then Eq. (4.7) describes the trajectory of this point. In the presence of noise, both a large number of point correspondences and a large number of image frames provide overdetermination. The algorithm presented in Section 3 can be used for the closed-form least-squares solution of two-view motion parameters. To use overdetermination based on a large number of frames, we let f > k in the coefficient equations (4.11). In fact, the coefficient matrix 5’; is essentially a high order deference [42]. 5’; tends t o be ill-conditioned when k gets large. This means f > k is more important when 5 is large. I f f > k, Eq. (4.12) can be solved by a least-squares method. We find a solution A to minimize
In the case of motion with precession, all the columns of D are generally independent. The least-squares solution is
A
=
(DtD)-lDtT.
In the case of motion without precession, the column vectors of D are linearly dependent. This can be shown by letting a1 in Eq. (4.11) be a non-zero vector parallel to the two-view rotation axes. Then the first three columns of D linearly combined by a1 is a zero vector. To get the least-squares solution of the coefficient equations (4.11), the largest set of independent columns of D should be found or tolerance-based column pivoting should be made. Theorem 1 solves a2, a3, . . . , ak. This means the last 3k - 3 columns of D are always independent. In the presence of noise the columns of D are very unlikely to be exactly linearly dependent even in the case of motion without precession. 4.4.
Continuous and Discrete Motions
The LCAM model we discussed is based on continuous precessional motion. We must find the relationship between continuous precession and two-view motion, before we can estimate the precessional parameters of our model based on discrete two-view motions. As we discussed in Section 4.1, a precession can be considered as the motion of a rolling cone which rolls without slipping upon a fixed cone. The angular frequency
374
J. J. Weng & T. 5’. Huang
Q
time 1 ,
Q time t 2
Fig. 8. The relation between rotation angles 0 and
4.
at which the symmetrical axis of the rolling cone rotates about the fixed cone is constant. Assumed at time t l , that an edge point A’ on the rolling cone touches a n edge point A on the fixed cone as shown in Fig. 8. After a certain amount of rolling, the touching points become B’ on the rolling cone and B on the fixed cone at time t 2 . Let 8 be the central angle of points A’ and B’, and q5 be that of A and B. Let r and r’ be the radii of circles 0 and 0’,respectively. The arc length between A and B is equal to that between A‘ and B’. Thus, q5r = Or’ or q5 sin a = 8 sin 0,where a and p are generating angles of the fixed cone and the rolling cone, respectively. We get 8 - sina (4.19) q5 sinp The precession consists of two rotational components. One is the rotation of the rolling cone about its own symmetrical axis. The other is the rotation of the rolling cone about the fixed cone. From Fig. 8 it can be readily seen that the relative position of the rolling cone and the fixed cone is uniquely determined if the touching points of the two cones are determined. Or alternatively, starting from the previous position, the new position of the rolling cone is determined if the two angles q5 and 8 are determined. Thus, no matter how we order these two rotational components, the final positions are identical as long as the angle 4 and 8 are kept unchanged. We can first rotate the rolling cone about its axis m and then rotate the rolling cone about the axis of the fixed cone, 1, or vice versa. We hope to find the equivalent two-view rotation axis of this continuous motion between two frames at time tl and time t 2 , respectively in Fig. 8. If we can find two fixed points which stay in the same positions before and after the motion, then the two-view rotation axis must go through these points. One trivial fixed point is the apex Q of the cones. Another fixed point can be found as follows: In Fig. 9 let the midpoint of arc AB touch the rolling cone (at time (tl - k t 2 ) / 2 ) . Extend line OB
2.5 3-0 Motion Analysis from lmage Sequences 375
Fig. 9. Finding fixed points for two-view rotation.
so that it intersects the plane containing Q, 0’ and B’ at a point PI. Extend line OA so that it intersects the plane containing Q, 0’ and A’ at a point P2. Draw a circle centered at 0 and passing through PI and Pz. Then the midpoint P of arc PI P2 is a fixed point. This can be seen by noting that the rolling cone can also reach its position at the next time instant t 2 in a n alternative manner as follows. First, rotate the rolling cone (slipping along the fixed cone) about 1 by angle 4/2, thus rotating P to its new position a t P I , and axis m reaches the position shown in Fig. 9. Then rotate the rolling cone (slipping on the fixed cone) about its own axis m by angle 8. Point P now reaches position Pz. Finally, rotate the rolling cone (slipping along the fixed cone) about 1 again by angle 4/2, taking the rolling cone to the position at time instant t 2 . This takes the point P back to its starting position. Therefore, the two-view rotation axis q5 found by two-view motion analysis from two image frames, goes through Q and P. Notice that the angular frequency at which the symmetrical axis of the rolling cone rotates about the fixed cone is constant. From the way of finding P, it is clear that the two-view rotation axis also rotates about 1 by a constant angle between consecutive frames. Thus, we have the following theorem:
Theorem 3. If a rigid body undergoes a precessional motion of the LCAM model, the two-view rotation axis between constant time intervals changes by rotating 0 about the precessional vector by a constant angle. Without loss of generality, we assume the time intervals between consecutive image frames are of unit length. We define the precessional vector to be a unit vector 1 parallel to the symmetrical axis of the fixed cone, the precessional angular frequency 4 to be the angular frequency at which the symmetrical axis of the rolling
376
J. J. Weng & T. S. Huang
cone rotates about the precessional axis, the ith body vector. mi to be a unit vector parallel to the symmetrical axis of the rolling cone at time ti, and the body rotation angular frequency 8 to be the angular frequency at which the rolling cone rotates about its symmetrical axis (see Fig. 10).
Fig. 10. Parameters of continuous precession and discrete two-view motion.
From image sequences we find estimates of two-view motion parameters. They are the ith two-view rotation axis vector ni, a unit vector parallel t o the two-view rotation axis between time instants t i - 1 and ti; the corresponding ith two-view rotation angle $i and the ith two-view translation vector Ti. Figure 10 shows the precession parameters of continuous motion and discrete two-view motion. Let R(n, 8) = [rij]denote the rotation matrix representing a rotation with axis unit vector n = (n,, ny,n,) and rotation angle 8, then R(n, 8) is given by
[
+
I
(n: - 1)(1- c o s ~ ) 1 n,ny(l- case) - n, sin8 n,n,(l- cos8) + ny sine n y n , ( ~ - c 0 s 8 ) + n , s i n ~ (np - 1 ) ( 1 - c 0 s ~ ) + 1 nyn,(~-c0s~)-n,sin~. n,n,(l-cos8)-nysin8
nZn,(l-cos8)+n,sin8
(np - i ) ( i - c o s e ) + i
(4.20)
Theorem 4. The continuous precession parameters and discrete two-view motion parameters are related by
Proof. From time t i - 1 , to time ti, the body moves from its previous position to a new position. From Fig. 10 the new position of the rolling cone (or the body) can be reached in the following way: First, the rolling cone rotates about its body vector mi-1 by angle 8. Then, the rolling cone rotates about the precessional
2.5 3-D Motion Analysis f r o m Image Sequences
377
vector 1 by angle q5. The two-view motion combines these two motions into one, which is the rotation about the two-view rotation axis vector ni by angle $i. We get Eq. (4.21). Similarly if we change the order of these two rotational components 0 we get Eq. (4.22). From Theorem 3, the two-view rotation axis rotates about the precessional vector. Therefore, the precessional vector 1 is perpendicular to ni - ni-1 and ni-1 - ni-2. The sign of 1 is arbitrary. Thus, 1 can be determined by (4.23)
We will assume that precessional angular frequency, body rotation angular frequency and two-view rotation angle are not larger than half a turn between every two consecutive frames. This assumption is necessary in practice, since the rotation must be small enough for matching to be possible. The precessional angular frequency 4 is equal to the the angle between ni x 1 and ni-1 x 1: (4.24)
The sign of q5 is the same as the sign of (ni-1
x ni) . 1 .
(4.25)
After 1 and q5 are found by Eqs. (4.23), (4.24) and (4.25), R(1, q5) can be calculated by (4.20). R(mi-1, 0) and R(mi, 0) can be determined by (4.21) and (4.22): R(mi-1,0) = R - l ( 1 , 4 ) R ( n i , + i ) ,
(4.26)
4).
(4.27)
R(mi, 0)
= R(ni,+i)R-'(l,
We can determine mi-1, mi and 0 by (4.26) and (4.27), because n and 0 can be determined from R(n, 0) [41]:
0 = fcos-1 1=
[T32
-
(
TIl
+
T23, T13
\ ) [ T ~-z TZ3, T13
TZZ2+ T33 -
1
- T31, TZl
T12It
>, -
T31, TZl
-
- Tiz]))'
Thus, we get the following theorem.
Theorem 5. The precessional vector, precessional angular frequency, body axes and body rotation angular frequency which define the precession part of the LCAM model can all be determined from three consecutive two-view motions, or 0 four consecutive image frames. In addition to these basic parameters which uniquely determine the motion of the model, some other parameters can also be determined from these basic parameters.
378
J . J. Weng €4 T. 5'. Huang
For example, the generating angles a: and 0 of the fixed cone and the rolling cone, respectively, in Fig. 8 can also be determined from 1, 4, mi, 0 and Eq. (4.19). 4.5. Estimation and Prediction
The LCAM model is applied t o subsequences of the images successively. The parameters of the model are estimated for every (overlapping) subsequence. The estimated model parameters can then be used to describe the current local motion. The following questions can be answered. Is there precession? If so, what are the precession parameters? What are the current or previous body vectors? What is the body rotation angular frequency? What is the probable motion for the next several time intervals? What are the probable locations of the feature points at the next several time instants? If the moving object is occluded in some of the previous image frames, what are the motion and the locations of these feature points during that time period? The number of frames covered by an LCAM model can be made adaptive to the current motion. The number can be changed continuously to cover as many frames as possible so long as the constant angular momentum assumption is approximately true during the time period to be covered. The value of the number of frames chosen can be based on the accuracy with which the model describes the current set of consecutive frames. The residuals of least-squares solutions and the variances of the model parameter samples indicate the accuracy. The noise level also affects the residuals and the variances of parameter samples. However, the noise level is relatively constant or can be measured. The resolution of the cameras and the viewing angle covering the object generally determine the noise level. The noise can be smoothed by determining the best time intervals and the number of frames covered by the model, according to the current motion. Because the LCAM model is relatively general, the time interval an LCAM model can cover is expected to be relatively long in most cases. The following part deals with the estimation of model parameters using overdetermination. Although one can derive formulation for minimum variance estimator here, the computation of the estimates requires iterations. We will discuss this optimal solution in Section 4.7. Here we give a closed-form solution that uses overdetermination. The solution can be directly computed without resorting to iterations. After finding two-view rotation axis vectors nl, n2, . . . , nf, precessional vector 1 should be orthogonal to n2 - nl, n3 - n2, . . . , nf - nf-1. However, because of noise, this may not be true. Thus, we find 1 such that the sum of squares of the projections of 1 onto n2 - nl, n3 - n2, . . . , nf - nf-1 is the smallest. Let
2.5 3 - 0 Motion Analysis from Image Sequences
379
We are to find unit vector 1 in minIIAlI1, subject to: llll1 = 1 . 1
(4.28)
The solution 1 of (4.28) is the unit eigenvector corresponding to the smallest eigenvalue of AtA. Let the precessional angular frequency determined from (4.24) and (4.25) be 4i. The precessional angular frequency of the model 4 can be estimated by the mean
Let the body rotation angular frequency determined by (4.27) be 8i. Body rotation angular frequency of the model can be estimated by the mean
Body vectors are estimated by averaging two consecutive two-view motion using (4.26) and (4.27), respectively. Two-view angular frequency can also be estimated by mean
According to the motion model, the ( f + l ) s t two-view rotation axis vector nf+l is R(1, ( f - i l)4)ni for any 1 5 i 5 f. In the presence of noise, we use the mean over all previous two-view motions to predict the next two-view rotation axis vector:
+
f
nf+l = f - l
C R(Z,( j i + l)+)ni. -
i= 1
From (4.9) the next position of point xf in the f t h frame is predicted by
where Qf and Qf+l are determined by (4.7). The prediction can be made for more than p 2 2 frames by using the following equations successively.
If the object was occluded in parts of image sequences,the positions and orientations of the object as well as the locations of the feature points on the object
380
J . J. Weng & T. 5’. Huang
can be recovered by a n interpolation similar to the prediction procedure discussed above. For the motion of the rotation center, occlusion just means that some rows in the coefficient equations are missing. The solution can still be found if we have enough rows. For the precession part of the emotion, the interpolation can be made in a way similar t o prediction or extrapolation. When making interpolation we use both the “history” and the “future” of the missing part. For prediction, only the “history” is available. Furthermore, we can also extrapolate backwards to find “history”, i.e. to recall what has not been seen before. The essential assumption is that the motion is smooth. 4.6. Monocular Vision
For the monocular case, 3-D positions can be predicted only up to a global scale factor from two images. When more interframe motions are involved, how many scale factors are unable t o be determined? The answer is one, provided that at least one point is visible among every three consecutive image frames. In other words, from point correspondences over a sequence of monocular image sequences, the depth of any visible points and translation between any two image frames can be determined up t o a scale factor c. Therefore, although the 3-D position of the points can only be determined up to a scale factor, the image coordinates of the points can be predicted without knowing the scale factor, since the scale factor c is canceled out in image coordinates. In the following, we derive these results. Suppose a point P is located at xi = ( ~ i , y i , z i a) ~t time t i , i = 0 , 1, 2, . . .. The image vectors are defined by xi = zXi. From image frame fi to image frame fi+l the motion, called ith motion, is a rotation represented by Ri followed by a translation represented by Ti.
or
i = 1, 2, . . , n. As discussed in Section 3, we can determine relative depths (4.29)
and
for i = 1, 2,
. . . , n, assuming the point P is visible for t o to t,. We have (4.30)
2.5 3-D Motion Analysis from Image Sequences
Multiplying both sides for i = 1 to i
=k
381
yields (4.31)
Letting llTlll
s be the unknown scale factor, (4.31) gives
(4.32)
That is, the norm of the kth translation, k factor s and a number
=
1, 2, . . . , n, is a product of the scale (4.33)
which can be determined from the relative depths. Therefore, although many interframe motions are involved in a long monocular sequence, only one unknown scale factor exists, instead of many. If the norm of translation in any interframe motion is known, s is determined from (4.31) and then motion and structure in the entire sequence is determined completely. Similarly, if the absolute depth of a point at a time instance is known, the norm of the translation vector is determined from (4.29), and then so does motion and structure in the entire sequence. Now what happens if there exists no point that is visible through the entire sequence? We can see from (4.30) that (4.31) and (4.32) still hold true if each factor &/Z; corresponds to a different point. For Zi and Z: to be determined for a point P, the point P must be visible from ti-1 to ti+l in three consecutive image frames. Therefore, as long as through any three consecutive frames there is a t least one point visible, the norm of any translation vector can be completely determined from s as in (4.32). Now we have proved the following theorem:
Theorem 6. Suppose the rotation matrix and direction of the translation between every consecutive monocular image pair can be determined. The magnitude of any interframe translation and the depths of visible points can be determined up to the same scale factor s, provided that through every three consecutive images at least one point is visible. 0 Since the depth of every point can be determined up to the same scale factor s, the 3-D position of the points can be predicted up to s as we discussed in the previous section. Since the image coordinates of a point cancel out this scale factor, the image coordinates of the point can be predicted without knowing the scale factor. For example, we can let s = 1 and compute the predicted 3-D position in the next frame based on the previous motion trajectory. Then by projecting the predicted 3-D point onto the image plane, we get the predicted image position of the point. This predicted position is the same for any positive s. In the presence of noise, the error in a single point may significantly influence the accuracy of the product in (4.33). Therefore, the product should be determined
382
J. J. Weng & T. S. Huang
based on many points instead of one. For example, the value of Zi/i?i can be averaged over many points before it is used to evaluate (4.33). 4.7. Optimization
The algorithm discussed above gives closed-form solutions for the model parameters. A simple least-squares solution is obtained when redundant data are available. Those solutions are generally good preliminary estimates. However, since the statistics of noise distribution is not employed, the solution is not optimal. If iterations are allowed, one can obtain higher accuracy through optimization. The optimization for more than two image frames is very similar to the case of two frames (either monocular or stereo). We just need a natural extension. The two-step approach can be used. First, the algorithm discussed above provides initial guesses. Then the initial guesses are improved through an iterative optimization. Similar to the notation in Section 3.4. Let m denote the motion model parameter vector, x = ( X I , x 2 , . . . , x,) denote the structure of the scene at time to, u i 3 k denote the observed ith image point in the j t h camera (e.g. left: l s t , right: 2nd) at time t k ( k = 0, 1, . . . , f), and hijk(m, x) denote the computed projection of the ith point in the j t h camera at time t k . Suppose that the additive noise in the coordinates of the image point is uncorrelated, U , j k = hijk(m, X)
-k b i j k
.
According to the minimum variance estimation, the optimal estimate of m and x is the one that minimizes image plane errors: n
2
f
i = l j = 1 k=O
To keep the dimension of the iterative search space low, only the independent model parameters should be included into m. They include precessional velocity vector defined as 41, 0th body velocity vector defined as Omo, and coefficient vectors in the coefficient equation {ai}. Those parameters uniquely determine the two-view motion parameters through (4.21) and (4.22). Since the optimal 3-D position of a point xi at time to depends only on model parameters and the observations of the point in image sequences, the optimal x i can be determined by minimizing 2
f
(4.34) The solution for x i that minimizes (4.34) can be estimated in a closed-form as in Section 3.4. Then, using the space decomposition technique gives
2.5 3-D Motion Analysis from Image Sequences
383
This decomposition significantly reduces the dimension of search space. If the twoview motion parameters used t o compute the model parameters are optimized, the initial guess of m is usually very good. This will significantly reduce the number of iterations in computing the optimal m. It is clear that either using monocular vision or any multi-ocular vision, we just need t o change the upper limit of j accordingly. Computationally, the minimization of the above function can be performed using a recursive-batch approach [43] which is a revised version of the Kalman filtering approach [44]. The recursive-batch approach is useful for improving the performance and efficiency when the image sequence is very long, or virtually infinite.
References [l] B. K. P. Horn and B. G. Schunck, Determining optical flow, Artif. Intell. 17 (1981) 185-203. [2] A. M. Waxman, An image flow paradigm, in Proc. Workshop on Computer Vision: Representation and Control, Annapolis, MD (IEEE Computer Society Press, Washington D.C., 1984) 49-57. [3] H.-H. Nagel and W. Enkelmann, An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences, ZEEE Trans. Pattern Anal. Mach. Intell. 8 (1986) 565-593. [4] J. K. Kearney, W. B. Thompson and D. L. Boley, Optical flow estimation: An error analysis of gradient-based methods with local optimization, IEEE Trans. Pattern Anal. Mach. Intell. 9 (1987) 229-244. [5] D. J. Heeger, Optical flow using spatiotemporal filters, Znt. J. Comput. Vision 2 (1987) 279-302. [6] S . T. Barnard and W. B. Thompson, Disparity analysis of images, IEEE Trans. Pattern Anal. Mach. Intell. 2 (1980) 333-340. [7] L. Dreschler and H.-H. Nagel, Volumetric model and 3-D trajectory of a moving car derived from monocular TV frame sequences of a street scene, Comput. Graph. Image Process. 20 (1982) 199-228. [8] E. C. Hildreth, The Measurement of Visual Motion (MIT Press, Cambridge, MA, 1983). [9] D. Marr and T. Poggio, A theory of human stereo vision, Proc. Royal Society of London B204 (1979) 301-328. [lo] J. E. W. Mayhew and J. P. Frisby, Psychophysical and computational studies towards a theory of human stereopsis, Artif. Intell. 17 (1981) 349-385. [I11 W. E. L. Grimson, From Images to Surfaces: A Computational Study of the Human Early Visual Systems (MIT Press, Cambridge, MA, 1981). [12] Y. Ohta and T. Kanade, Stereo by intra- and inter-scanline search using dynamic programming, IEEE Trans. Pattern Anal. Mach. Intell. 7 (1985) 139-154. [13] N. Ayache and B. Faverjon, Efficient registration of stereo images by matching graph descriptions of edge segments, Int. J. Comput. Vision 1 (1987) 107-131. [14] H. P. Moravec, Towards automatic visual obstacle avoidance, in Proc. 5th Znt. Joint Conf. on Artificial Intelligence (William Kaufmann, Los Angeles, LA, 1977). [15] F. Glazer, G. Reynolds and P. Anandan, Scene matching by hierarchical correlation, in Proc. ZEEE Conf. on Computer Vision Pattern Recognition (IEEE Computer Society Press, Washington D.C., 1983) 432-441.
384
J. J. Weng & T. 5'. Huang
[16] T. D. Sanger, Stereo disparity computation using Gabor filters, Biol. Cybern. 59 (1988) 405-418. [17] A. D. Jepson and M. R. M. Jenkin, The fast computation of disparity from phase differences, in Proc. IEEE Conf. on Computer Vision Pattern Recognition, San Diego, CA (IEEE Computer Society Press, Washington D.C., 1989) 398-403. [18] J. Weng, A theory of image matching, in Proc. Third Int. Conf. on Computer Vision, Osaka, Japan (IEEE Computer Society Press, Washington D.C., 1990) 200-209. [19] J. J. Hwang and E. L. Hall, Matching of featured objects using relational tables from stereo images, Comput. Graph. Image Process. 20 (1982) 22-42. [20] W. K. Gu, J. Y . Yang and T. S. Huang, Matching perspective views of a polyhedron using circuits, IEEE Trans. Pattern Anal. Mach. Intell. 9 (1987) 390-400. [21] H. S. Lim and T. 0. Binford, Stereo correspondence: A hierarchical approach, in Proc. Image Understanding Workshop (Science Applications Corp., Mclean, VA, 1987) 234-241. [22] J. Weng, N. Ahuja and T. S. Huang, Two-view matching, in Proc. 2nd Znt. Conf. on Computer Vision (IEEE Computer Society Press, Washington D.C., 1988) 64-73. Also, Matching two perspective views, I E E E Trans. Pattern Anal. Mach. Intell. 14 (1992) 806-825. [23] R. Jain and H.-H. Nagel, On the analysis of accumulative difference pictures from image sequences of real world scenes, I E E E Trans. Pattern Anal. Mach. Intell. 1 (1979) 206-214. [24] J. W. Roach and J. K. Aggarwal, Determining the movement of objects from a sequence of images, IEEE Trans. Pattern Anal. Mach. Intell. 2 (1980) 554-562. [25] A. R. Bruss and B. K . Horn, Passive navigation, Comput. Vision Graph. Image Process. 21 (1983) 3-20. [26] G. Adiv, Determining three-dimensional motion and structure from optical flow generated by several moving objects, I E E E Trans. Pattern Anal. Mach. Intell. 7 (1985) 348-401. [27] A. Mitiche and J. K. Agganval, A computational analysis of time-varying images, in T. Y . Young and K. S. Fu' (eds.), Handbook of Pattern Recognition and Image Processing (Academic Press, New York, 1986). [28] H. C. Longuet-Higgins, A computer program for reconstructing a scene from two projections, Nature 293 (1981) 133-135. [29] R. Y . Tsai and T. S. Huang, Uniqueness and estimation of 3-D motion parameters of rigid bodies with curved surfaces, IEEE Trans. Pattern Anal. Mach. Intell. 6 (1984) 13-27. [30] B. L. Yen and T. S. Huang, Determining 3-D motion and structure of a rigid body using the spherical projection, Comput. Vision Graph. Image Process. 21 (1983) 21-32. [31] J. Q. Fang and T. S. Huang, Some experiments on estimating the 3-D motion parameters of a rigid body from two consecutive image frames, IEEE Trans. Pattern Anal. Mach. Intell. 6 (1984) 547-554. [32] X. Zhuang and R. M. Haralick, Rigid body motion and the optic flow image, in Proc. I E E E 1st Conf. on Artificial Intelligence Applications, Denver, CO (IEEE Computer Society Press, Washington D.C., 1984) 366-375. [33] X. Zhuang, T. S. Huang, N. Ahuja and R. M. Haralick, A simplified linear optic flow-motion algorithm, Comput. Vision Graph. Image Process. 42 (1988) 334-344. [34] A. M. Waxman, B. Kamgar-Parsi and M. Subbarao, Closed-form solutions to image flow equations for 3-D structure and motion, Int. J. Comput. Vision 1 (1987) 239-258.
2.5 3-0 Motion Analysis f r o m Image Sequences 385 [35] 0. D. Faugeras, F. Lustman and G. Toscani, Motion and structure from point and line matches, in Proc. Int. Conf. o n Comput. Vision, London, UK (IEEE Computer Society Press, Washington D.C., 1987) 25-34. [36] J. Weng, T. S. Huang and N. Ahuja, Error analysis of motion parameter determination from image sequences, in Proc. 1st Int. Conf. on Computer Vision, London, UK (IEEE Computer Society Press, Washington D.C., 1987) 703-707. [37] H. C. Longuet-Higgins, The reconstruction of a scene from two projections-configurations that defeat the 8-point algorithm, in Proc. I E E E 1st Conf. o n Artificial Intelligence Applications, Denver, CO (IEEE Computer Society Press, Washington D.C., 1984) 395-397. [38] J. Weng, T. S. Huang and N. Ahuja, Motion and structure from two perspective views: Algorithms, error analysis and error estimation, I E E E Trans. Pattern Anal. Mach. Intell. 11 (1989) 451-476. [39] G. R. Fowles, Analytical Mechanics, 3rd ed. (Holt, Rinehart and Winston, New York, 1977). [40] W. D. Macmillan, Dynamics of Rigid Bodies (McGraw-Hill, New Jersey, 1936). (411 0. Bottema and B. Roth, Theoretical Kinematics (North-Holland, New York, 1979). [42] J. Weng, T. S. Huang and N. Ahuja, 3-D motion estimation, understanding and prediction from noisy image sequences, I E E E Trans. Pattern Anal. Mach. Intell. 9 (1987) 37&389. [43] J. Weng, P. Cohen and N. Rebibo, Motion and structure estimation from stereo image sequences, IEEE Trans. Robotics and Automation 8, 3 (1992) 362-382. [44] G. S. Young and R. Chellappa, 3-D motion estimation using a sequence of noisy stereo images: Models, estimation, and uniqueness results, I E E E Trans. Pattern Anal. Mach. Intell. 12 (1990) 735-759.
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 387-424 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
I CHAPTER 2.6 1 SIGNAL-TO-SYMBOL MAPPING FOR LASER RANGEFINDERS
KENONG WU* and MARTIN D. LEVINE Centre for Intelligent Machines €4 Department of Electrical Engineering, McGill University, Montre'al, P. Q. Canada H3A 2A7 A new approach for computing qualitative part-based descriptions of 3-D objects is presented. The object descriptions are obtained in two steps: Object segmentation into parts and part model identification. Beginning with single- or multi-view range data of a 3-D object, we simulate the charge density distribution over an object's surface which has been tessellated by a triangular mesh. We detect the deep surface concavities by tracing local charge density minima and then decompose the object into parts at these points. The individual parts are then modeled by parametric geons. The latter are seven qualitative shapes, each of which is formulated by a restricted globally deformed superellipsoid. Model recovery is performed by fitting all parametric geons to a part and selecting the best model for the part, based on the minimum fitting residual. A newly defined objective function and a fast global optimisation technique are employed to obtain robust model fitting results. Experiments demonstrate that this approach can successfully recover qualitative shape models from input data, especially when part shapes are not fully consistent with model shapes. The resultant object descriptions are well suited for symbolic reasoning and fast object recognition. Keywords: Computer vision, 3-D shape representation, object segmentation, object description, range data, shape characterisation, electrical charge density distribution, volumetric primitives, parametric geons, superellipsoids.
1. Introduction
A major problem in machine vision is the development of an object recognition system which is not based on accurately known models, but rather on coarse, qualitative ones representing classes of objects. Sensor data usually only provide point-by-point measurements such as the distance from the sensor to objects in the viewed scene. Thus, for example, these numerous and unstructured data are not appropriate for representing the environment for a mobile robot executing quick and complicated tasks. On the one hand, such a robot must make use of symbolic models which are concise and organised descriptions about the structure of the world. On the other, the robot must transform sensor data to symbolic descriptions which 'Current address: Integrated Surgical Systems, 829 West Stadium Lane, Sacramento, CA 95831, USA.
388
K. Wu i 3 M. D. Levine
are consistent with the models in a stored database and support efficient model matching. Thus this signal-to-symbol mapping is at the heart of any functioning robot carrying out complex tasks. In this chapter, we present a new approach t o 3-D shape representation of objects based on parts. The input t o our system is a single range image or multiple range images of an object. Our task is twofold. The first is to segment the object into individual parts. The second is to select a particular part model which describes the best shape approximation of each segmented object part from a few predefined model candidates. The segmentation approach works as follows. An object to be segmented is viewed as a charged perfect conductor. It is a well known physical fact that electrical charge on the surface of a conductor tends t o accumulate at sharp convexities and vanish at sharp concavities. Thus object part boundaries, which are usually denoted by a sharp surface concavity [l],can be detected by locating surface points exhibiting local charge density minima. Beginning with range data of a 3-D object, we tessellate the object surface with a closed triangular mesh and simulate the charge density distribution over the surface. We then detect the deep surface concavities by tracing local charge density minima and decompose the object into parts at these boundary points. The segmentation method efficiently deals with certain thorny problems in traditional approaches, such as unrealistic assumptions about surface smoothness and instability in the computation of surface features. Part model recovery employs parametric geons as the part models. These are seven qualitative shapes associated with pose and attribute parameters governing model size, tapering rate and bending curvature. Parametric geons are formulated in terms of restricted deformed superellipsoids [2]. The equations of the models provide explicit global constraints on the qualitative shape of the part models. This permits the algorithm to directly compare model shapes with a given shape and restricts the part model to a predefined shape family, no matter how the input data vary. We obtain part models by fitting all parametric geons to a part and selecting a particular model based on the minimum fitting residual. Thus our approach implements explicit shape verification of the resultant part models, obtaining them more robustly and accurately than in previous work. We begin this chapter with a review of previous related research on both part segmentation and model recovery, and make a comparison with our work in Section 2. Then we present the part segmentation method in Section 3. We describe the parametric geon models and the model recovery approach in Sections 4 and 5, respectively. In Section 6 we report experimental results with both single- and multi-view range data. The characteristics of our approach are further discussed in Section 7 and conclusions are drawn in the last section. 2. Related Work The significance of object descriptions at the part level is well understood [1,3-61. Many objects consist of parts or components which have perceptual salience
2.6 Signal-to-Symbol Mapping for Laser Rangefinders 389 and reflect the natural structure in the world [7]. Building part-based object descriptions for various tasks has been a major strategy in computer vision for many years [3,7-151. To obtain part-based descriptions, one needs to address the following two points: (1) Which are the parts? and (2) What is the model for each of the parts? The former is the issue of object segmentation into parts (part localisation), while the latter deals with part model recovery (part identification). 2.1. Object Segmentation into Parts The problem of object segmentation into parts can be stated as follows: Given a set of data points that represent a multi-part object, classify these data into meaningful subsets, each of which belongs to a single physical part of the object. By a physical part, we mean a portion of the object surface which can be distinguished perceptually, geometrically or functionally from the rest of the object. Definitions of parts are discussed in [1,4,6,16]. Part segmentation algorithms can be categorised as being shape- or boundary-based. Shape-based approaches decompose objects into parts by measuring the shape similarity between image data and a n arrangement of predefined part models. For example, spheres [17-191, quadrics [20],and superellipsoids [9,21-231 have been used for part segmentation. These approaches first hypothesise an object configuration composed of part models, and then evaluate a measure of the similarity between the hypothesis and the true object shape. If the measure is worse than a preselected threshold, another hypothesis is generated and evaluated until the similarity measure is less than the threshold. The last hypothesis is then adopted as the desired object segmentation. Another type of shape-based segmentation uses an aspect hierarchy of part shapes [13,24,25]. These methods commonly employ a finite but relatively large set of distinctive part models. Each model exhibits a restricted number of configurations of surface patches in all possible views. Thus, even for all models, the number of surface configurations in all possible views is quite limited. The procedure is to first identify surface patches using region growing [26] or edge detection [27], and then group surface patches into a potential part according to the permissible surface configurations. For perfect object shapes, shape-based approaches can be quite efficient. However, problems arise when the part shapes are not very consistent with the available model shapes. This produces non-unique or incorrect part segmentations. Boundary-based methods segment objects into parts by seeking part boundaries instead of complete shapes. This type of approach can segment an object without incorporating part shape information. For example, Koenderink and Van Doorn [4] have proposed parabolic lines as part boundaries. At a parabolic line, one of the principal curvatures [28] of the surface changes from being convex to concave. Rom and Medioni [29] have performed part decomposition based on this theory using range data as input. The drawback of this scheme is that parabolic lines cannot indicate part boundaries on cylindrical surfaces [l].Also, since this method is based
390
K. Wu €3 M. D. Levine
on the classification of regions of positive and negative Gaussian curvature, it is not clear how t o apply it t o objects containing planar surfaces. Hoffman and Richards [l]have proposed segmenting objects into parts a t deep surface concavities. Ferrie and Levine have analysed surface principal curvatures to locate surface concavity and segment range data containing objects [30]. This strategy has also been applied t o the segmentation of edge-junction graphs of object range images [31]. In addition to range data, cross-sections of 3-D objects have also been taken as input [32-341. In this case, part segmentation is performed by examining the connectivity and shape of cross-sections. Such an approach requires a voxel-based coordinate system t o relate the data across the cross-sections. Our approach to part segmentation is consistent with the boundary-based approaches. As indicated above, all previous methods [11,29,30,35] are based on surface curvature, a geometrical property. In contrast, we employ a physical property, the simulated charge density distribution over a n object surface, t o find part boundaries. This approach has some distinguishing characteristics and advantages. Since the curvature computation involves the first and second partial derivatives of the surface, an assumption on smoothness of the object surface is mandatory [36]. Curvature computation is also very sensitive to local noise [37] and a smoothing operation on the range data is usually required. Alternatively, a larger area or scale may be employed to reduce noise effects. However, selecting a suitable scale is in general a difficult problem. Also, alarger scale will increase the computational time. In contrast, our approach solves an integral equation rather than performing surface curvature computations and thus does not require smoothness of the object surface. Since the charge density computation uses global data, the influence of noise is reduced. Furthermore, although the charge density must be computed on a closed triangular mesh, the variables are restricted only to the surface of objects. Thus, one does not require a voxel-based coordinate system or need t o compute the object interior. Because of this, our approach involves many fewer unknown variables than voxel-based approaches [32,33,38].
2.2. Part Identification The problem of part identification can be stated as follows: Given a set of data points on a particular part and all candidate part models, find the model which is the best description of that part. On the basis of psychological experimentation, Biederman’s theory of Recognition-by-Components (RBC) [39] proposed a modest set of volumetric primitives, called geons, as qualitative descriptions of object parts. The theory postulated that if an arrangement of a few geons could be recovered from the line drawings of an object, it could be quickly recognised with great tolerance for variations in viewpoint and shape. Geons have therefore been proposed as a basis for recognition of 3-D objects observed in single 2-D views [13-15,401. The idea of using a finite set of qualitative shape primitives as part models is also adopted here.
2.6 Signal-to-Symbol Mapping f o r Laser Rangefinders
391
Nearly all research has focused on the recovery of geon models from complete edge maps or ideal line drawings which have depicted objects whose parts are instances of geon models [41]. However, in general, “clean” or complete line drawings of objects cannot be obtained due to the colour and texture of object surfaces or complex illumination configurations. Because of this, and also for practical reasons, some research has focussed on data obtained from laser rangefinders [24,25,31]. In all of the latter, part descriptions were determined in a bottom-up fashion, inferring global properties by aggregating local features. This type of approach will fall short when local features do not fully satisfy the exact definitions of the geons. Clearly, any computer vision system, which successfully recovers qualitative descriptions, would have t o approximate various object shapes by ideal shape models. It would seem that the most popular volumetric model for part shape approximation is the superellipsoid, a parameterised family of closed surfaces [42]. The implicit equation of superellipsoids is given as follows [42]:
Here €1 is the “squareness” parameter in the north-south direction; €2 is the “squareness” parameter in the east-west direction. a l l a2, a3 are scale parameters along the z, y, z axes, respectively. The advantage of the superellipsoid model is that by using only two more parameters than ellipsoids, it can describe a large variety of volumetric shapes. In addition, its mathematical definition provides a useful global constraint which restricts its shape during model recovery. The latter adapts and molds the model to the object shape, thereby, reducing the influence of missing data, image noise and minor variations in shape. In this way, a n approximate shape description can be obtained efficiently [2,9,11,43,44]. The fundamental distinction between superellipsoids and geons is that the latter is a prescribed set of individual primitive component shapes, which are qualitatively discriminable, thereby supporting fast object recognition as indicated by the RBC theory [39]. The only previously reported attempt to derive geons from superellipsoids is due to Raja and Jain [45] in a continuation of previous work on superellipsoid model recovery [2,9,12]. They explored the recovery of 12 geons from single-view range images by classifying the actual parameters of globally-deformed superellipsoids [2]. Although they obtained 89% accuracy for objects with smooth surfaces, they found that the estimated parameters were very sensitive to viewpoint, noise, and objects with coarse surfaces. They also noticed that their major classification errors were due t o the misclassification of straight and curved geon cross-sections. This is mainly caused by the Euclidean distance measure they used for classifying part shapes. The problem can be easily illustrated by Fig. 1, which shows a series of superellipses representing the cross-sections of superellipsoids. The shape parameter of the superellipse changes uniformly from 0.1 t o 1. Accordingly, the shape changes gradually (row by row) from a square to a circle. The number under each figure
392
K. Wu tY
M. D. Levine
0.10
0.13
0.16
0.19
0.22
Fig. 1. Classification of cross sections of superellipsoids. A series of shapes of a superellipse is given row by row. The shape parameter of the superellipse changes from 0.1 to 1, and consequently, its shape changes from a square to a circle. The number under each figure indicates the value of the shape parameter. The task is to classify these shapes into two groups, square-like shapes and circle-like shapes. If the classification were based upon the shape parameter, the shapes in the first three rows would be classified into square-like shapes. However, human perception seems to classify more of these shapes into square-like shapes.
indicates the value of the associated shape parameter. If these shapes were t o be classified into two groupsa based on the Euclidean distance of the shape parameter, the top three rows would be classified into one group and the rest into the other. However, we clearly observe that the shapes in at least the first four rows are more similar to the square than the circle. Thus, there is a significant difference in shape discrimination between a n Euclidean distance Euclidean distance-based method and human perception. Another reason is the ambiguity between superellipsoid shapes and their associated parameters, as noted by Solina and Bajcsy [2]. According to RBC theory (391, “the memory of a slight (sic) irregular form would be coded as the closest regularized neighbor of that form”. Our work is the aAccordingly, the corresponding superellipsoids are classified as one of two geons (a cuboid and a cylinder).
2.6 Signal-to-Symbol Mapping for Laser Rangefinders
393
first attempt to accomplish this. The importance of this approach is the ability t o achieve an explicit shape verification of the resdtant part models. The equations of the parametric geons provide explicit global constraints on the qualitative shape of part models. This constraint permits the algorithm to directly compare the model shapes with a given part shape. The resultant shape must be one of the predefined parametric geon shapes, no matter how the input data vary. Therefore, our approach can compute the qualitative shape models of parts reliably from data representing parts whose shapes are not fully consistent with their models. In our research, we define a new objective function which measures (i) the Euclidean distance Euclidean distance from data points t o model surfaces and (ii) the squared difference between the normal vectors of a model and the object. Model fitting is performed by minimising this function using a stochastic global optimisation approach (Very Fast Simulated Re-annealing), which statistically guarantees finding the global minimum. A similar approach has been presented by Yokoya et al. [44]. The first term of their objective function was the squared Euclidean distance, and their optimisation technique was the classical simulated annealing [46]. We will show that our objective function and optimisation approach significantly improve the efficiency of the model fitting procedure. 3. Part Segmentation
3.1. Motivation According to Hoffman and Richards [l],the concept of a part is based upon a particular regularity in nature - transversality [47]. The theory states that when two arbitrarily shaped surfaces are made t o interpenetrate, they always meet at a contour of concave discontinuity of their tangent planes. This is illustrated in Fig. 2 by ellipsoids. Transversality has been widely applied to part segmentation [11,30,31,34,35]. Interestingly, there exists an analogy between the singularity in surface tangents and the singularity in electrostatics. When a charged conductor with an arbitrary
contour of concave discontinuity of tangent olanes
Fig. 2. Transversality. Two ellipsoids joined together create a contour of concave discontinuity at their intersection.
394
K. Wu kY M . D. Levine
Fig. 3. Charge distribution over an object. The crosses represent charge on the surface of the object. The charge density is very high and very low at sharp convexities and concavities, respectively. Thus, the object part boundary can be located at local charge density minima.
Fig. 4. Charge densities near edges. (a) An edge formed by two planes with an angle charge density at P(7,B).
p. (b) The
shape is in electrostatic equilibriumlb all charge resides unevenly on the outer surface of the conductor [48], as shown in Fig. 3. The charge density is very high at sharp convex edges and corners. Conversely, almost no charge accumulates at sharp concavities. Electrical charge density at sharp edges and corners has been carefully studied by Jackson [49]. An object edge or corner is defined as a C1 discontinuity of an object surface. By ignoring secondary global effects, Jackson has derived an approximate relationship governing the charge density p at an edge formed by two conducting planes, as shown in Fig. 4. Here p is the angle between two planes defining an edge and 7 is the distance from the edge t o a point P , where the charge density is measured. It has been shown [49] that the larger p and the smaller 7 , the greater the charge density at P. For a constant 7 , the charge density increases monotonically as p increases. A theoretical singular behaviour of charge densities bThe conductor is in electrostatic equilibrium when there is no net motion of charge within the conductor.
2.6 Signal-to-Symbol Mapping for Laser Rangefinders
at edges (for
= 0)
395
has also been suggested as follows [49]:
This means that the charge density is infinite, constant and zero when the angle defined by the two planes is convex, flat and concave, respectively. The singular behaviour of charge densities at corners, which is similar t o that at edges, has also been investigated [49]. We have observed that at slightly smoothed edges and corners, the positions of the local extrema of charge density remain unchanged. Consequently, by assuming that a multi-part object is a charged conductor, we can detect deep surface concavities, which we have noted delineate part boundaries [l],by isolating significant charge density minima.
3.2. Charge Density Computation Our physical model is the charge density distribution on a charged conductor in 3-D free space, where no other charge or conductors exist. To begin with, we list three physical facts which can be derived from physical laws.
Fact 3.1. I n electrical equilibrium, any charge o n a n isolated conductor m u s t reside entirely o n its outer surface [50]. This means that there is no charge inside the conductor. Thus the structure within an object does not affect its charge density distribution. This fact indicates that, under these circumstances, the charge density distribution is a surface property.
Fact 3.2. T h e surface of a n y charged conductor in electrical equilibrium i s a n equipotential surface [50]. Fact 3.3. Conservation of Charge: Charge cannot be created or destroyed, since the algebraic s u m of positive and negative charges in a closed or isolated system does not change under any circumstances [51]. These facts provide us with the conditions needed to establish mathematical equations with charge densities as their variables. Consider the electrical potential at the vector position r E R3, produced by a point charge q , located at the vector position r' E R3, as shown in Fig. 5. The corresponding electrical field at r can be calculated by an application of Gauss's law. Thus,
Here € 0 is a constant, known as the permitivity of free space. The electrical potential 4(r) at r can be derived by an integration of (3.2) along the dashed line from ro E R3, the vector position of the reference point, to r
396
K. Wu & M. D. Levine observation point
Fig. 5. Configuration for a point charge. r is the vector position where the electrical potential is observed. r’ is the vector position of the source point charge. ro is the vector position of the potential reference point.
Fig. 6. Configuration of charge distribution over the surface. 0 is the origin of the coordinate system.
(see Fig. 5):
It is customary to choose the reference potential to be zero at Iro I = w. Accordingly, 4(ro) = 0 and Eq. (3.3) becomes: (3.4) Secondly, consider that the charge is continuously distributed over the object surface S (see Fig. 6 ) . Thus the electrical potential at r is contributed by all of the charge on S and satisfies the principle of superposition. It can be expressed as follows:
Here q = p(r’)dS’, p(r’) is the charge density at r‘, and S‘ is the area over S. Thirdly, according to Fact 3.2 that all points on a charged conductor in electrical equilibrium are at the same electrical potential, if we restrict r in Eq. (3.5) to the
2.6 Signal-to-Symbol Mapping for Laser Rangefinders 397 conductor surface, 4(r) is constant. Thus, (3.5) may be rewritten as follows:
Here V = 4 7 r ~ 4 ( r is ) a constant. Since S in Eq. (3.6) is a n arbitrary surface, it is impossible t o solve the equation analytically. However, we can obtain an approximate solution to Eq. (3.6) by using finite element methods [52], as described in the next sect ion.
3.3. Finite Element Solution To compute the charge density distribution based on Eq. (3.6), we approximate the 3-D object by a polyhedron, each face of which is a planar triangle which possesses a constant charge density. Then the problem of integration over the complete surface (see Eq. (3.6)) can be converted into a summation of integrations over each triangle. Since the latter can be solved analytically, the charge density on each triangle can be easily computed. The finite element solution is obtained as follows. We tessellate the object surface using a triangular mesh having N planar triangles, Tk, Ic = 1,.. . , N . Each triangle is assumed to have a constant charge density, pk, as shown in Fig. 7. A set of basis functions f k , k = 1,. . . ,N is defined on this triangular mesh as follows: (3.7) Thus the basis function, f k , is nonzero only when r’ is on the triangle T k , as shown in Fig. 7. Therefore, the charge density p(r’) can be approximated by a piecewise constant charge density function as follows: N
dr’)
xpkfk(r’). k=l
Fig. 7. Polyhedral approximation of an ellipsoid. When r’ is on T k , fk = 1 and
(3.8)
fi
= O(i
# k).
398
K. Wu & M. D. Levine
Substituting (3.8) into Eq. (3.6), we have (3.9) Since the charge density is assumed to be constant on each Tk, we may take ri as the observation point on each Ti and rewrite Eq. (3.9) as: (3.10) According to Fact 3.3, the sum of the charges on each triangle equals the total charge on the object surface. Let Q be the total charge on the object surface and s k be the area of T k . Then we have N
(3.11) Assuming Q is known, and given (3.10) and (3.11), we obtain a set of linear equations with N l unknowns, p 1 , . . . , p~ and V. Since the integral in (3.10) can be evaluated analytically [53],the charge density distribution p k and the constant V can be obtained by solving the set of linear equations. Because the potential on a particular triangle is actually contributed by the charge on all of the triangles, the matrix A is dense. In the actual computation, the observation point ri on each triangular patch is selected at its centroid. The set of linear equations is solved by a conjugate gradient-squared method [54]. To compute the charge density distribution, we need to tessellate a closed triangular mesh on the object surface. A triangular mesh also specifies a data indexing system for object surfaces, which are represented by a set of discrete 3-D points. Thus, triangulation establishes a specific spatial relationship between these points and facilitates the extraction of part boundaries. The objects to be segmented are represented in either single or multiview range data. The former indicate the 3-D coordinates of data points only on the visible object surfaces. To obtain a closed triangular mesh for this case, we tessellate the visible surface and artificially construct a mesh for the invisible surface. Depending on how this shape completion is done, the resulting mesh will affect the absolute values of charge density on the visible surfaces. However, we observe that the locations of the charge density extrema which indicate object part boundaries do not change. We may combine these two meshes to construct a closed triangular mesh. A detailed description of this mesh construction is given in [55]. Multiview data are obtained by merging range data acquired from several viewpoints. They are a sequence of 3-D data points representing the complete surface of the object. In our experiments, surface tessellation for multiview data was performed by Decarlo and Metaxas at the University of Pennsylvania using a mesh blending approach [56].
+
2.6 Signal-to-Symbol Mapping for Laser Rangefinders 399
Fig. 8. Direct Connection Graph (DCG). (a) A triangular mesh. (b) DCG of the triangular mesh in (a). (c) Subgraphs of (b) after boundary node deletion. Here triangular patches 1, 2, 3 and 8 are assumed to be located on the part boundary.
3.4. Object Decomposition
The object is decomposed into parts after the simulated charge density distribution is obtained. The method is based on a Direct Connection Graph (DCG) defined on the triangular mesh, as shown in Fig. 8. Here nodes represent triangles in the mesh and branches represent the connections between direct neighbors. Two triangles which share two vertices are considered t o be direct neighbors. For example in Fig. 8(a), triangles 1 and 2 are direct neighbors while 2 and 3 are not. Thus the DCG provides a convenient coordinate system on the object surface and indicates the spatial relationship between a triangle and its neighbors. It permits the tracing of the part boundaries on the triangular mesh without employing a voxelbased coordinate system. This significantly reduces the required memory space for describing the object and increases the computational speed. For a triangular mesh of multiview range data, we decompose the complete mesh. For single-view range data, only the visible surface of the object is segmented. On the basis of the transversality principle described in Section 3.1, we have assumed that a part boundary is explicitly defined by deep surface concavities. For a complete object the part boundary is a closed contour. This ensures that the decomposition algorithm will be able to segment a part from the rest of the object. The assumption also provides a stopping criterion for the boundary tracing procedure. Since the part boundary is located at local charge density minima, it can be traced along the LLvalle”’of the charge density distribution. We note that for singleview data, the visible surface is not closed, and therefore the part boundary may not be a closed contour. In this case, the tracing process stops when it reaches a mesh boundary, that is, a triangle which has only two direct neighbors. The algorithm examines the charge density on all triangles to find an initial triangle for tracing each boundary which must satisfy the following conditions: 1. It must be a concave extremum; that is, its charge density must be a local minimum.
400 K . W u & M . D. Levine
2. It must be located at a deep concavity. Thus the charge density on the triangle must be lower than a preselected threshold.c 3. It, as well as its neighbors, must not have been visited before. This ensures that the same boundary will not be traced again.
Beginning at the initial triangle, the algorithm proceeds to the neighbor with the lowest charge density. During the tracing procedure, all triangles detected on the boundary are marked. These will not be checked again and eventually will be deleted from the DCG. For the mesh constructed from mdtiview data, the process continues until it returns to the initial triangle. Since we have assumed that a part boundary is closed, this means that all triangles on this part boundary are visited. For a mesh constructed from single-view data, as illustrated in Fig. 8, the process continues until it reaches a triangle (face 3) on the boundary of the mesh. If the initial triangle (for example, face 2) possesses three direct neighbors, the procedure will move in the other direction until reaching a triangle (face 8) on the boundary of the mesh. Thus all triangles on the part boundary are visited. Next the algorithm finds a new initial triangle and traces another boundary. It repeats the same tracing procedure, and finally stops when the charge density at an initial triangle is higher than the preselected threshold. After all triangles on part boundaries have been found, the nodes of the DCG representing these triangles are deleted. In this way, the original DCG is divided into a set of disconnected subgraphs, as shown in Fig. 8(c). Physically the object has been broken into parts. Each object part can be obtained by applying a component labeling algorithm to a subgraph of the DCG. The result is several lists of triangles, each containing those belonging to the same object part. 4. Parametric Geons
4.1. The Model
Parametric geons are a finite set of distinct volumetric shapes, which are used to describe the shapes of object parts. We believe that such model shapes should reflect the essential geometry of objects in the real world. Seven volumetric shapes were chosen, primarily motivated by the art of sculpture, perhaps the most traditional framework for 3-D object representation. One of the most obvious features of sculptured objects is that they consist of a configuration of solids with different shapes and sizes which are joined together but which we can perceive as distinct units. The individual volume is the fundamental unit in our perception of sculptural form, as indeed it is in our perception of fully 3-D solid forms in CThisthreshold determines when an object should not be decomposed any further. If the charge density at an initial triangle is greater than this threshold, we assume that all the boundary points have been found. The selection of the threshold depends on a priori knowledge of the surface concavity and there is no universal rule for determining it. Currently we choose 1.5 times the lowest charge density on the object surface as the threshold.
2.6 Signal-to-Symbol Mapping for Laser Rangefinders
ellipsoid:e,= 1 E,
=1
TAPERING
BENDING (6)
(2)
tapered cylinder
401
cylinder:E,=o.iE,=
1
curved cylinder
0.1 E ,=o.I
curved cuboid
I
Ez=O.1
tapered cuboid
cuboid:+
Fig. 9. The seven parametric geons. e l and
€2
are the superellipsoid shape parameters.
general [57]. From a sculptor's point of view, all sculptures are composed of variations of five basic forms: the cube, the sphere, the cone, the pyramid and the cylinder [58,59]. Another important belief in the world of sculpture is that each form originates either as a straight line or a curve [59]. Straightness and curvature are significant for characterising the main axes of elongated objects and were employed in defining the original geon properties [39]. By generalising the five primitive shapes used in sculpture and adding two curved primitives, we arrive a t the following seven shapes for parametric geons (see Fig. 9): the ellipsoid, the cylinder: the cuboid, the tapered cylinder, the tapered cuboid, the curved cylinder and the curved cuboid. These seven shapes are derived from the superellipsoid Eq. (2.1) by (i) specifying the shape parameters, €1 and €2 and (ii) applying tapering and bending deformations [2,42]. Each parametric geon shape can be expressed by a compact implicit function,
g(x,ai)=O i = 1 ,
...,7 .
dActually this could be a cylindrical shape with an elliptical cross-section.
(4.1)
402
K. Wu & M. D. Levine
Fig. 10. Some other examples of parametric geons. The number beside each shape indicates its geon type: 1 - ellipsoid, 2 - cylinder, 3 - cuboid, 4 - tapered cylinder, 5 - tapered cuboid, 6 - curved cylinder, 7 - curved cuboid.
Here x E R3 and ai is the nine- to eleven-dimensional vector. The elements of the vector are three scale parameters, six spatial transformation parameters, two tapering parameters if it is a tapered primitive, and one bending parameter if it is a curved primitive. Since these seven shape types are defined quantitatively, their variations can represent a variety of different shapes. Some of this diversity is shown in Fig. 10. The detailed derivations of the implicit and normal equations for all of the seven parametric geons can be found in [55]. 4.2.. Comparison with the Original Geons
The major distinction between parametric geons and the conventional geons of Biederman [39] is that the latter are defined in terms of certain attributes of volumetric shapes, which do not impose global shape constraints. By contrast, parametric geons are defined in terms of different analytical equations, which do provide such constraints. In addition, the original geons are described in strictly qualitative terms. However, parametric geon descriptions simultaneously include both qualitative and quantitative characterisations of object parts. The geometrical differences between these two sets of primitives are given in Table 1. Certain qualitative properties of the parametric geons have been simplified in comparison with the original geons of Biederman. For example, an asymmetrical cross-section is not used in defining any of the parametric geons because of the symmetrical nature of superellipsoids. The assumption that all parametric geons are symmetrical with respect to their major axes is consistent with the well-known human perceptual tendency toward phenomenological simplicity and regularity [60]. Symmetrical primitives have also been employed in alternatives to the original geons discussed by other researchers [45,61].
2.6 Signal-to-Symbol Mapping f o r Laser Rangefinders
403
Table 1. Difference in qualitative properties between parametric geons and Biederman’s original geons. Attributes
Parametric Geons
~~
Geons ~
cross-sectional shape
symmetrical
symmetrical, asymmetrical
cross-sectional size
constant, expanding
constant, expanding, expanding and contracting
combination of properties
either tapering or bending
both tapering and bending
5. Part Model Recovery
5.1. The Objective Function The strategy for recovering parametric geons bears some resemblance to that for other parametric primitives. That is, a fitting scheme is used to minimise an objective function which measures some property difference between a n object and a model [2,9,44]. However, there is an additional requirement for parametric geon recovery. The process must also produce discriminative information such that the resultant metric data can be converted t o a qualitative description. The objective functions studied previously by several researchers were neither intended nor used for this purpose. An approach has been reported which did use fitting residuals t o guide object segmentation into parts, but for another purpose [20]. To identify individual qualitative shapes based on fitting residuals, were quire that the values of the objective function correctly reflect the difference in size and shape between the object data and the parametric models. Our objective function consists of two terms expressed as follows:
E
= tl
+ Xytz .
The first term, t l , measures the distance between object data points and the model surface; the second term, t 2 , measures the difference between the object and model normals. X and y are parameters controlling the contribution of t 2 made to the objective function. 5.1.1. T h e distance measure The first term of the objective function is given by
Here N is the number of data points, {di E E 3 ,i = 1 , . . . , N } is the set of data points described in terms of the model coordinate system, and a is the vector of model parameters.
404 K . Wu tY M . D. Leuine
I'
f
\
] Imagedata
Fig. 11. Defining the objective function. n, and nd are the model and object surface normals, respectively. 0 is the origin of the model. A is the distance between a particular data point and the centre of the model. zsis a point on the model surface. l?i is the angle between a model and object surface normals.
For the three regular primitives (ellipsoid, cylinder and cuboid), e(di,a) is defined as the Euclidean distance from a data point to the model surface along a line passing through the origin 0 of the model and the data point [62,63] (see Fig. 11):
where
P=
{
2 20
for the ellipsoid for the cylinder and cuboid.
A is the distance from di to 0 and g(di,a) is an implicit function for a parametric geon. Since tapering or bending significantly complicates the implicit equations of the deformed primitives, in these cases we cannot obtain a closed-form solution for e(di,a),as was done in Eq. (5.3). Thus an iterative method would be indicated. However, the objective function evaluation is the largest computational component of the model recovery procedure. Hence, for the sake of simplicity, we compute an approximate distance measure for the tapered and curved models. No iteration is required. First, we apply an inverse tapering or bending transformation to both the data and the model in order to obtain the transformed data di, as shown in Fig. 12; this gives either a regular cuboid or regular cylinder. Second, we use (5.3) to compute the distance from the transformed data point d{ to the transformed model surface along a line passing through d: and the model origin 0. We interpret e(d'i,a) as the approximation of the distance along a line from di to the model surface. Although this approximation creates a small error in the distance measure, it greatly speeds up computation.
2.6 Signal-to-Symbol Mapping for Laser Rangefinders 405
distance
Fig. 12. The cylinders on the right in (a) and (b) are obtained by applying inverse tapering and bending transformations t o the left tapered and curved cylinder, respectively. e(di, a) is the Euclidean distance along a line Odi in the inverse transformed case.
5.1.2. The normal measure We define the second term ( t 2 ) of the objective function by measuring a squared difference between the surface normal vectors n d of objects and the surface normal vectors n, of models at each corresponding position (see Fig. 11): (5.4) Here N is the number of data points and
+ +
In (5.1), y = (a, ay a , ) / 3 , which makes the second term adapt to the size of the parametric geons, and a,, ay and a, are model size parameters. This factor also forces the selection of a model with a smaller size if object data can be fitted
406 K . Wu €4 M. D. Levine
equally well by a model with different parameter sets. This case can happen when the data on the bottom surface of an object cannot be obtained. However, the size of the model is prevented from being arbitrarily small since the value of the objective function increases if the model size is smaller than the object size. This is similar t o the volume factor used in superellipsoid recovery [2]. A, a weighting constant, controls the contribution of the second term t o the objective function. It should be selected based on assumptions of shape differences between objects and their models. There is no general rule for selecting it. We chose X = 5 according to a heuristic based on the shape difference between each pair of parametric geons [64]. 5.2. Minimising the Objective Function The procedure for fitting parametric geons is a search for a particular set of parameters, which minimises the objective function in (5.1). This function has a few deep and many shallow local minima. The deep local minima are caused by a n inappropriate orientation of the model. The shallow minima are caused by noise and minor changes in object shape. In order to obtain the best fit of a model t o an object, we need to find model parameters corresponding to the global minimum of the objective function. To accomplish this, we employ a stochastic optimisation technique, Very Fast Simulated Re-annealing (VFSR) [65]. Motivated by an analogy to the statistical mechanics of annealing in solids, the simulated annealing technique uses a “temperature cooling” operation for non-physical optimisation problems, thereby transforming a poor solution into a highly optimised, desirable solution [46]. The salient feature of this approach is that it statistically finds a global optimal solution. VFSR uses an annealing schedule which decreases exponentially, making it much faster than traditional (Boltzmann) annealing [46], where the annealing schedule decreases logarithmically. In addition, a re-annealing property permits adaptation to changing sensitivities in the multidimensional parameter space. 5.3. Biasing the Objective Function with Different N o r m s
We have suggested a n L1 norm in (5.2) and LZ norm in (5.5) t o measure differences in distance and orientation, respectively. It is known that the sensitivity of an Lz norm gradually increases [66]. In other words, this norm is insensitive to small values of the objective function and becomes sensitive to outliers. On the other hand, the sensitivity of a n L1 norm is the same for all residual values. It is also known that the absolute size of a model is independent of the measurement of the differences between normals. These properties can be used to construct an efficient parameter search during model fitting. Effectively, the procedure automatically endeavours to compute the correct result in what amounts t o two successive “stages”. In the first “stage”, when the fitting procedure begins, the models and objects are not well aligned, so most of the data can be viewed as outliers. Thus
2.6 Signal-to-Symbol Mapping for Laser Rangefinders
407
140
c
0 .3
tj
120
c
2
.
10,000 : 1 > 14% 514 to 850 nm > 1000 f/s
> 100 : 1 > 65% 0.5 to > 1 nm > 100 f/s
1282
light clockwise or counterclockwise, depending upon which of the two homogeneous magnetic states the pixel is switched to. When an incident linearly polarized beam is transmitted through the device, the output polarization is rotated clockwise or counterclockwise in a manner dictated by the pixels’ switched states. This polarization modulation is translated t o a n amplitude or phase modulation by placing an analyzer polarizer in the path of the transmitted beam. When the polarizer’s transmission axis is oriented perpendicular to one of the two output polarization angles, the light passing through pixels switched to that state is extinguished, while that passing through pixels switched to the opposite homogeneous state is partially transmitted. This modulation is referred t o as “binary amplitude only” modulation, and is shown below in Fig. 4.
Fig. 4. Principle of Operation of the MOSLM.
If alternatively, the analyzer polarizer’s transmission axis is oriented perpendicular t o the bisector of the two output polarization angles, the transmitted amplitude is the same for all pixels, but the phase of light from pixels switched in one homogeneous state is 180 degrees out of phase with that of the light from pixels switched in the opposite homogeneous magnetization state. This modulation is referred to as “binary phase only” modulation. For non binary operations, liquid crystal devices with lower speed need be considered.
3.4
Multi-Sensory Opto-Electronic Feature Extraction
...
545
3.1.2. Multifocus Fourier transform device
To accomplish the desired replication of the input pattern’s two-dimensional Fourier transform, the inherent structure of the MOSLM spatial light modulator may be utilized. However, t o achieve intensity uniformity throughout the replication plane, an alternative holographic solution would consist of custom manufacturing a single hologram that would be placed after the input MOSLM, and which would result in the required uniform replication of the Fourier transformed input pattern. Figure 5 shows an example of a Fourier replication due t o pixelation of an input function. The computer simulation program used t o demonstrate this feature assumes a fixed focal length, and thus only shows a fixed portion of the replicated field, but in fact the replication period may be adjusted at will by varying the Fourier lens.
Fig. 5. Example of Fourier pattern replication due to input SLM pixel array.
The MOSLM, as well as other similar devices, contains opaque rows and columns between the rows and columns of active pixels. This becomes a two-dimensional grating when the MOSLM is used in a Fourier system, such as here, and the result is a replication at the Fourier plane of whatever image is programmed onto the MOSLM. The spacing of the replications depends upon the focal length of the Fourier transforming lens, the spacing of the pixels in the MOSLM, and the incident wavelength. Thus by choosing the focal length of the Fourier lens, we achieve a twodimensional periodic replication of the desired Fourier transform pattern. The intensity of the replicated Fourier transforms falls off with increasing order. This falloff was experimentally measured using a 1282 MOSLM array and an incident 633 nm laser beam, and values are given in Table 3.2, along with the required recording time for each t o cancel the intensity variation. If one assumes that the efficiency of the recorded hologram is proportional t o the recording time, then by inverting the intensity value of each order, and using that number as a relative recording time, the net transmission will be a constant. Thus the 1.M.H hologram recorded using these first five orders of the MOSLM would require a recording time up t o 100 times longer for the outer orders than for the strongest zero order replicated pattern.
546
H.-K. Liu et al. Table 3.2. Experimental 128’ array replicated output intensities, and calculated corrective recording times. Order
Measured Intensity
Recording Time
0
47.5 pw 27.3 35.5 3.00 7.95 1.40 2.00 0.45 0.90 0.85 2.30
1.00 T(0) 1.74
+1 -1 +2 -2 +3 -3 +4 -4 +5 -5
1.34 15.8 6.00 33.9 23.8 105.0 52.8 55.9 20.7
Binary optic gratings are another applicable technology which potentially could be used to complement the effect of the SLM grating itself, making the replication intensity more uniform.
3.1.3. Interconnect matrix hologram (I.M.H.) This element, placed at the Fourier plane of the MOFENAR architecture, must enact the required phase and amplitude modulation upon the incident optical beam to satisfy the learning rules which are to be implemented within the MOFENAR system. This element must also display high resolution recording capability in order to enact the desired transmission function at each of the locations of the replicated optical Fourier transform incident from the input MOSLM. The device will be encoded in the following manner. Refer to Fig. 6. A desired reference data pattern, a ( z , y ) b ( x , y ) , is first encoded on the spatial light modulator, which in turn encodes the pattern onto the
+
Aperture,
Plxelated S.L.H.,
‘*I ......
& 1.tl.H. n
F.T.0
4 F.T. -1
+ F.T. Fig. 6. Interconnect Matrix Hologram Recording Process.
-2
3.4
Multi-Sensory Opto-Electronic Feature Extraction
...
547
laser beam. This pattern is then optically Fourier transformed and replicated, as discussed above. An aperture is placed just in front of the I.M.H., equal in size t o a single replicated Fourier pattern, and centered over one of the replicated order locations. A holographic recording is then made of the joint Fourier transform intensity pattern. Note that only the section of the I.M.H. illuminated by the aperture is recorded. After recording one Fourier hologram in the I.M.H., the input laser is blocked, the input MOSLM is reprogrammed with a different desired pattern (either a(z,y), b(z,y) or both are changed), and the aperture is translated to an adjacent location on the I.M.H. This process is repeated throughout the plane of the I.M.H., until a complete array of reference holograms is achieved. The recording time is varied to correct for the incident intensity variation. By using a recording time that is inversely proportional to the intensity of the particular replicated order, the efficiency of the I.M.H. may be altered t o cancel this variation and produce equivalent outputs for any of the recorded orders. Table 3.2 presents calculated recording times for the case of the 128 x 128 MOSLM array. Noting the dramatic fall off in intensity for the device in going from the zero order to the plus and minus fifth orders, this seems a reasonable limit of replications to attempt in a prototype MOFENAR architecture. This results in 121 different reference data patterns being stored in the I.M.H. When these data patterns have been recorded, the aperture is removed. Because the non-uniformity of the diffraction pattern may cause a problem in the reading of the hologram and in the amplification, reflection, and thresholding operations, it may be necessary that an HOE be utilized to achieve a more uniform replication. The drawback here is the reduction of light throughput, as limited by the HOE’S diffraction efficiency. 3.1.4. Phase conjugate mirrors
Another, perhaps most challenging element of the proposed breadboard prototype are the phase conjugate mirrors (PCMs). These devices produce a phase conjugated, opposing beam t o that which is incident upon the surface of the mirror. Such a beam exactly retraces the path of the incident beam, and therefore may be described as a “time reversed” equivalent to the incident beam. It is this “time reversal” of the incident wave that makes possible the iterative optical processing of the input pattern. The architecture that has been studied over the last several years to accomplish phase conjugation is known as “four wave mixing”, and is shown in Fig. 7. Interference of the incident signal beam and a plane wave pump beam 1 forms an intensity pattern in the crystal. This intensity distribution creates a charge migration in the photorefractive crystal that results in a corresponding refractive index change. Pump beam 2, also a plane wave, enters the crystal from the opposite direction as pump beam 1, and is diffracted by the index modulation. The result
548 H.-K. Liu et al.
Signal Beam
\
Fig. 7. Phase conjugate mirror architecture.
of this diffraction is the creation of a phase conjugated beam counter propagating along the path of the incident signal beam. If the intensity of pump beam 2 is much greater than that of the signal beam, the diffracted intensity of the phase conjugated beam may exceed that of the incident signal beam, and gain is achieved. This gain is simply a result of coupling between the incident and plane wave beams. Such amplification has been previously shown in BaTiOs. Here we present some specific points regarding the performance and characteristics of four photorefractive materials used in four wave mixing experiments: LiNb03 is a well studied material, and is perhaps more readily available than others. It has a small electro-optic coefficient, fast response time, and small diffraction efficiency. The response time increases for smaller grating periods, while sensitivity (gain) increases with period. Sensitivity is low due to small carrier drift and diffusion lengths. BaTi03 has a large electro-optic coefficient, and relatively slow response times (100 ms for approximate 100 mW/cm2 intensity). Its response time decreases for smaller grating periods. Its gain increases for smaller grating periods, and for less intense signal beams. Amplified phase conjugation has been demonstrated in BaTiO3, and this material is capable of very good performance under correct conditions. BSO has a small electro-optic coefficient, poor sensitivity, and fast response time. It is strongly optically active, and poorly understood (e.g. input-output polarization relationship, transmitted-diffracted beam interaction, polarization states). SBN is another material that is not well documented. It has been suggested that it can be made to perfor; [21] comparably to BaTi03. It has a slow response time (in seconds with 1 W/cm2 intensity). In general, for all of the above materials the steady state gain saturates for large pump/input beam intensity ratios (Ipump/Iinput >> 1). As this ratio gets smaller, e.g. less than 1000, the gain decreases. Two requirements of the PCM for the MOFENAR application include the ability to achieve gain between the incident and reflected phase conjugate beam, as well as the ability to perform nonlinear optical thresholding upon the incident intensity distribution. The first requirement is necessary due to attenuating effects present
3.4 Multi-Sensory Opto-Electronic Feature Extraction . . . 549 elsewhere within the cavity, namely losses due to absorption and scattering. The second requirement exists to achieve the desired self correcting, adaptive capability described above. A third requirement of the PCM is that it respond at least as quickly as the input pattern cycles. This is seen by recalling above that it is the incident beam itself which forms the grating within the PCM that creates the phase conjugate beam. Therefore every time the input changes, and thus the incident beam distribution changes, the desired grating in the PCM must be reformed before the new incident beam will be phase conjugated. Finally, an important aspect of this element of the MOFENAR architecture is the availability of photorefractive crystals. An investigation into availability led to the conclusion that there are in fact few sources available to those interested in simply purchasing crystals to experiment with. Much of the research that has been performed in this area has been done by those who have the capability to grow such devices themselves (or whom are directly associated with those who can grow the devices). There are devices available however, and one centimeter cube single crystals of both BaTiOs and BSO can be obtained within a reasonable amount of time. 3.2. Experimental Demonstrations
There are many examples of the applications of optical pattern recognition techniques [34-521. In this section, we present two optical pattern recognition experiments to demonstrate the feasibility of the MOFENAR. First we discuss the input/out format to the system. The input to the MOFENAR architecture consists of electrical signals from one or more detectors, which are fed to the input MOSLM. The formatting of these signals requires standard, consistent mapping onto the MOSLM for efficient associative recall to be achieved. To accomplish this input data formatting for the SLM, software may be written which processes the individual signals from each of the utilized detectors. Frame grabbers are available for the acquisition, digitization, and processing of input images and signals. Real images may also be processed by the MOFENAR architecture. Examples of real image input and multi-sensor input architecture that are feasible with the neural net are shown in Figs. 8(a) and (b). The output of the MOFEhAR architecture consists of the reconstructed reference pattern which is found by the iterative optical neural network algorithm implemented in the MOFENAR system. This output may be read directly in the case of real image analysis and pattern recognition, or may be analyzed with a dedicated electronic state circuitry custom tailored to the specific application in which the MOFENAR is utilized. The optically generated output of the MOFENAR may also be analyzed optically, using standard optical processing architecture such as the Vander Lugt correlator or a digital optical signal processing network.
550
H.-K. Liu et al.
m-+3? Input I
n-
S.L.H. Pattern
Input 3
(4
Real Image
A r r a y Detector
.
S.L.H. P a t t e r n
(b) Fig. 8. (a) Real image input to the MOFENAR neural net; and (b) One or multi-sensor MOFENAR input architecture.
M
COMPLETE
OBJECT
Pmnu OBJECT (RECOUING)
Fig. 9. Optical feedback SOFENAR by using volume holographic medium and conventional mirror.
First, the experimental demonstration of an architecture of single-sensory optoelectronic feature extraction neural associative retriever (SOFENAR) is accomplished. Then, a MOFENAR with electronic feedback and angularly or spatially multiplexed is demonstrated.
3.4 Multi-Sensory Opto-Electronic Feature Extraction . . . 551 3.2.1. Single-sensory opto-electronic feature extraction neural associative
retriever (SOFENAR) First, we discuss two kinds of SOFENARs with optical feedback and electronic feedback, respectively. The experimental demonstration for the scheme with optical feedback is presented. a. SOFENAR with Optical Feed-back An optical feedback SOFENAR is shown in Fig. 9. In the recording (memorizing) step, the shutter is turned on, and the object beam carrying complete input information passes through a Fourier transform lens (FL), interferes with a reference beam to write a dynamic grating in a holographic medium. The grating includes three components, 10l2 lrI2, 0,. and 0;)where 0 represents the Fourier transform of the object 0, r denotes the reference beam, and symbol “*” means phase conjugation. In the retrieval step, the shutter is turned off, the object beam carrying incomplete input information (0’)to read out the hologram t o generate a part of the related reference beam ( T ’ ) , the reference beam is counter-propagated by using a mirror (M) t o read out the same hologram again, so that the corresponding complete object, F{O‘O*)~T~~O*}, can be obtained from output plane via a beam splitter (BS), where 0‘ means the Fourier transform of the incomplete input object 0’ and F represents Fourier transform operation. In this scheme, the holographic medium can be either planner or volume materials. Figure 10 shows experimental results of 2-D image associative sensing by using the optical feedback SOFENAR, two images are shown respectively in (a) and (b). The left part and right part represent a n incomplete input object and a completely reconstructed complete object, respectively. The reversed reconstructed-objects in the vertical direction comes from the reverse imaging of the optical setup. In addition t o the images shown in Fig. 9, the image of a finger print is used to demonstrate the feasibility of the system. Results are shown in Figs. 11-15.
+
(4
(b)
Fig. 10. Experimental results. (a) Image object 1, and (b) Image object 2. The left part and the right part are the incomplete input objects and reconstructed complete objects, respectively.
552
H.-K. Liu et al.
Fig. 11. The original image of a finger print stored in a PC and reproduced from a laser printer.
Fig. 12. The retrieved image of the finger print with an original input that is limited by an aperture of 1/5 of the diameter centered at the image.
Fig. 13. The retrieved image of the finger print with an original input that is limited by an aperture of 1/2 of the diameter centered at the image.
3.4 Multi-Sensory Opto-Electronic Feature Extraction .. , 553
Fig. 14. The retrieved image of the finger print with an original input with its right half blocked.
Fig. 15. The retrieved image of the finger print with an original input with its left half blocked.
Fig. 16. The a laser printer.
.educed from
554
H.-K. Liu et al.
Fig. 17. The retrieved image of the resolution chart with an original input that is limited by an aperture of 1/5 of the diameter centered a t the image.
Fig. 18. The retrieved image of the resolution chart with an original input that is limited by an aperture of 1/2 of the diameter centered at the image.
An Air Force resolution chart is used to show the resolution of the system in the retrieved image. Results are shown in Figs. 16-18. From these results, we may conclude that when the diameter of the aperture of the image is 1/5, only incomplete retrieval can be obtained. When the diameter is increased to 1/2, reasonable retrieval is obtainable. Also, the image retrieval process is asymmetrical in the horizontal direction due to the thick volume recording property of the photorefractive crystal.
3.4 Multi-Sensory Opto-Electronic Feature Extraction . . .
555
COMPLETE
FL2
OBJECT OUTPUT
HOLOGRAPHIC MEDIUM
:.___.___._.____ :...........F b____- - {
DETECTOR:
THRESHOLD
1 ---:
COMPLETE
PARTIAL OBJECT (RECOLLING)
Fig. 19. Electronic feedback SOFENAR. The holographic medium is either a thick volume or a thin planner material.
b. Electronic feedback SOFENAR The electronic feedback SOFENAR is shown in Fig. 19. In the memorizing step, both shutter 1 and shutter 2 are turned on, and the interference grating is recorded with a holographic medium passing through FL1. In the sensing step, shutter 1 is turned off and shutter 2 is turned on at the beginning, the incomplete input object generates a partial reference beam incident into a detector. The electronic signal output from the detector passes though a thresholding device and a controller to reverse the states of these two shutters as long as the intensity of the reconstructed reference beam is larger than a certain value set in advance. The original reference beam passes through shutter 1 t o read out the complete object from output plane via FL2. The SOFENARs described above can be expanded to become a MOFENAR by using angularly and/or spatially multiplexed recording with either optical and/or electronic feedback arrays. We first present the architecture of electronic feedback MOFENAR with angular multiple5ing and spatial multiplexing and then show a n experimental feasibility demonstration. 3.2.2. Multi-sensory opto-electronic feature extraction neural associative retriever ( M O F E N A R )
a. Electronic feedback angularly-multiplexed MOFENAR The architecture of the optical feedback SOFENAR (see Fig. 9) may be modified to form a MOFENAR by using angularly multiplexed reference beam in a volume
556
H.-K. Liu et al.
Fig. 20. Angularly multiplexed MOFENAR with an electronic feedback, a photorefractive crystal, and a Dammann grating.
holographic medium, as shown in Fig. 20. An N x M (1x 3 in the Fig. 20) Dammann grating (DG) is used to generate a N x M angularly multiplexed replicates of an incident plane-wave reference. Lenses L2 and L3 are used to image the rear surface of the DG to a photorefractive crystal (PRC). The passage of each angularly multiplexed reference beam in the array can be individually controlled by using a corresponding shutter array (SA), which is located on the focal plane of L2. In the recording step, a series of inputs from a spatial light modulator (SLM) are Fourier transformed with lens L1 and recorded on the PRC with a different reference beam by controlling the SA. In the retrieval step, all elements of the SA are turned off, and an incomplete object input from the SLM is Fourier transformed and used to read out a part of the related reference beam which is then focused by lens L5 and detected with a certain element of the detector array (DA). The electronic signal output from the detector element is then used to control the SA and SLM (after removing the incomplete input object from the SLM and turning on the corresponding element of the SA) via the threshold device and the controller. The specific reference beam passing through the SA, generates a complete object out of the incomplete input object. This output is obtained at the image plane of lens L4. b. Electronic feedback spatially-multiplexed MOFENAR bl. Architecture An electronic feedback spatiaily-multiplexed MOFENAR can be constructed by using a spatially-multiplexed reference beams in a planner holographic medium, as shown in Fig. 21. In this architecture, the configuration of the reference beam is the same as that of the previous figure. A Dammann grating (DG2) is used to generate an N x M spatially-multiplexed replications of the Fourier transform of the input object. A shutter array (SA2) is used to individually control the propagation state (passing or blocking) of each replicated beam, and only one replicate at one time is allowed to address the holographic medium. If one input is recorded at one location
3.4 Multi-Sensory Opto-Electronic Feature Extraction . . . 557 REFERENCE BEAM
OUTPUT
.....................-...................... Fig. 21. Spatially-multiplexed MOFENAR with an electronic feedback, a planner holographic medium, and two Dammann gratings.
of the hologram, then N x M different inputs may be recorded (e.g. N = M = 3 in Fig. 21). At the beginning of the retrieval, SA2 is turned on and SA1 is turned off. The partial reference beam generated by the related incomplete input is then detected, thresholded, and used t o turn off SA2 and turn on the element of SA1 to read out the corresponding complete image. b2. Experimental Results The experimental set-up of a three-channel electronic-feedback MOFENAR are shown in Figs. 21 and 22. Attention is called to Fig. 22. The laser beam comes from the 514.5 nm line of a Coherent Innova 306 Ar+3 laser which passed through a half-wave plate (HPl), and is split by a polarizing beam splitter (PBS) into two; an ordinary beam and a n extraordinary beam. The transmitted extraordinary beam is expanded, spatially-filtered, and collimated using lens L1, a pin-hole spatial filter, and lens L2. This is the object beam. A double Mach-Zehnder interferometer is used to provide three input paths for inputs of A, B, and C. Three shutters SH1-SH3 can be used to select the specific input for recording and recalling. The input object beam is then Fourier transformed by lens L3 and applied t o a photorefractive crystal (PRC). The reflected ordinary beam from the PBS is converted to an extraordinary beam using a half-wave plate (HP2) which then illuminates a 1 x 3 optical fan-out element (OFE) and generates three reference beams with equal intensities. The transmission of the three reference beams are controlled by three shutters (SH4SH6). During learning, the six shutters are turned on a pair at a time (SH1 and SH4, SH2 and SH5, SH3 and SH6) t o record three individual complete input objects via angular multiplexing in the PRC. At the beginning of the retrieval process, shutters SH4-SH6 are turned off, and one of the shutters SH1-SH3 (e.g. SH1) is turned on to apply an incomplete object (e.g. A) to reconstruct the corresponding reference
558 H.-K. Liu et al. HP1-HPZ: HALF-WAVE PLATES
MI-M7: MIRHOAS
BSI-BS4 BEAM SPLITTERS Ll-L5: LENSES SH1-SH6: SHUTTERS PBS: POLARIZING BEAM SPLITTER PRC: PHOTOREFRACTIVE CRYSTAL PHI PIN-HOLE SPETIAL FILTER OFE: OPTICAL FAN-OUT ELEMENT (1 X 3) D1-D2; DETECTORS A-C: INPUTS
2
D1 Fig. 22. The experimental set-up of a MOFENAR with electronic feedback.
Fig. 23. The three reference beams reconstructed by three complete inputs corresponding respectively to the image of space shuttle DISCOVERY (left spot); ATLANTIS (middle spot); and COLUMBIA (right spot).
beam and unavoidably also due to cross-talk some other reference beams. These reference beams are sensed by a ;hoto-detector array D1 with a proper threshold for the suppressing of the cross-talk terms. The correctly associated reference beam which is selected after thresholding is then feedback to turn on the corresponding shutter (e.g. SH4) to reconstruct the complete image of the object (e.g. A) through lens L5 and detector array D2. In the experiment, the objects chosen are three different space shuttle images (i.e. DISCOVERY, ATLANTIS, and COLUMBIA). Figure 23 shows the three reconstructed reference beams when the three incomplete objects are applied.
3.4 Multi-Sensory Opto-Electronic Feature Extraction . .. 559
(b)
(c)
Fig. 24. Associative retrieval experiment; input with DISCOVERY. (a) Incomplete input, (b) reconstructed reference, and (c) retrieved.
(b)
(c)
Fig. 25. Associative retrieval experiment; input with ATLANTIS. (a) Incomplete input, (b) reconstructed reference, and (c) retrieved.
560 H.-K. Liu et al.
(b)
(c)
Fig. 26. Associative retrieval experiment; input with COLUMBIA. (a) Incomplete input, (b) reconstructed reference, and ( c ) retrieved.
Figures 24-26 show three sets of associatively retrieved images. In each of the figures, (a) shows the incomplete input object; (b), the reconstructed reference beam; and (c), the retrieved complete object. 4. Conclusions
We have presented the results of a thorough theoretical and experimental investigation on the characteristics and capabilities of a novel optical neural network pattern recognition architecture. We have presented detailed analyses and experimental results and have constructed experimental systems utilizing state-of-the-art optical elements and devices which are either currently available or can be obtained in a reasonable time and manner. The major elements of this investigation include the theoretical design and explanation of the proposed system architecture and operation, a computer simulation investigation into the implementation of desired learning rules and performance of the system in various applications, and an investigation of the hardware elements required for the implementation of the breadboard system. The individual elements required for implementation of such a system are currently available. A spatial light modulator which is currently capable of demonstrating frame rates of over 1000 per second has been documented, and standard dichromated gelatin holographic technology is shown to allow the resolution necessary
3.4 Multi-Sensory Opto-Electronic Feature Extraction . . . 561 for a prototype implementation such as is proposed. The phase conjugate mirrors show great promise, and experimental application of various crystals as phase conjugate mirrors is investigated and referenced. However, the body of experimental data on this new field of optics is as yet not sufficient for a definite conclusion t o be drawn on the exact performance of such materials in the proposed MOFENAR system. Some important questions regarding these materials are raised in the following section. A mathematical model of the MOFENAR principle of operation was derived. This formalism was subsequently translated t o computer code form, and implemented t o provide simulation results of the performance of such a system. These results were documented and presented, which support the conclusion that the MOFENAR system does indeed offer the capability of adaptive, fault tolerant operation. Using both angularly multiplexed and spatially multiplexed MOFENARs described above, a large number (an order of thousands) of objects may be memorized and sensed, in parallel. The designed MOFENAR with electronic feedback is of large information capacity, good discrimination capability, and high image quality. As shown above, the MOFENAR architecture offers significant potential capability to perform parallel multi-sensory recognition of input patterns or vector sets originating from multiple sensors, which measure either identical or different types of signals. This capability has direct potential applications in several fields, including parallel database search, image and signal understanding and synthesis, and robotics manipulation and locomotion, in addition to real-time multi-channel real pattern recognition. In the following sections, we examine some of the possible applications that may utilize the capability of the MOFENAR neural net, and indicate how they might be investigated using the proposed prototype system.
4.1. Real Images The most obvious application of the MOFENAR architecture is a neural pattern recognition tool, for use in NASA and/or industrial environments. The input patterns may be directly fed to a spatial light modulator from a standard or nonstandard video camera. This input may be one of a known set of potential inputs. The set of known potential inputs may be stored in matrix form in the Interconnect Matrix Hologram. Thus, the input pattern, which may contain noise or variations, will create a closest “match’’ with the most similar stored reference image, and the thresholding, iterative optical processing of the MOFENAR will distinguish and recreate this reference pattern at the output plane of the neural network. This output may either be presented for direct visual confirmation, or may be used as an input into a n optical post processor, such as a standard optical correlator.
562
H.-K. Liu
et al.
One example of such a utilization of the MOFENAR is robotics vision. The proposed neural network offers the capability of a relatively compact optical processing system to handle imperfect visual input and draw conclusions based upon its own reference library of patterns. This may allow real-time vision analysis to be performed, which could be translated to independent robotics locomotion and environmental interaction. 4.2. Related Vector Sets
The second major area in which the MOFENAR neural network might be well utilized is in the processing of large and complex data sets, either from a single detector or source, or from more than one detector simultaneously. Known data patterns which are to be searched for may first be encoded in the Interconnect Matrix Hologram of the architecture, as described in Section 4.1. The input SLM may then be fed continuous information from detectors aimed at the field(s) of interest, and the MOFENAR architecture will in parallel compare each input frame with all of the stored reference data patterns in the I.M.H. Any input data frame which contains a pattern sufficiently similar t o one of those stored in the IMH will result in the ideal reconstruction of only that pattern at the output plane of the neural network. Not only does this allow for direct large scale comparison and recognition of data to be accomplished in parallel, but it also presents the capability of inferring information from the input, and establishing new recognition rules to be sought and found. This is due t o the fact that each input pattern will result in some output that indicates how the input data pattern compares and is similar or dissimilar to, the data patterns stored in the I.M.H. Thus one may attempt t o “teach” the neural network to recognize any and all levels of data patterns present in the incoming signals, both on the level of the discrete data itself and on higher levels of data field. Documentation of the “MOFENAR” output in a training mode may allow the repeated ability to infer information from the input of previously unusable or unseen data patterns. One specific application is a common problem which currently requires massive electronic computing capability. This is weather condition analysis. Individual sensors measuring humidity, temperature, wind velocity, and pressure for example, at one or several locations, and representing a single time or a set of consecutive temporally separated measurements, may be translated t o spatial binary representation, relayed to the input spatial light modulator of the MOFENAR, and processed. The Interconnect Matrix Hologram in this case could consist of experimentally recorded data from the same sensors, which is known to precede specific weather patterns or conditions. The output from the MOFENAR in this case may be evaluated on different levels as described above, and statistical or empirical relationships between the output weather condition vector set and the actual measured weather condition may be drawn.
3.4 Multi-Sensory Opto-Electronic Feature Extraction . . . 563 In summary, a few potential NASA and commercial applications are summarized below:
NASA Applications (a) (b) (c) (d) (e) (f) (g)
Planetary exploration in-situ data analysis and information screening Space surveillance specific object identification and navigation guidance Space image understanding and classification Space station automated rendezvous and docking Space habitation and utilization environmental evaluation and assessment Navigation collision avoidance in Moon and Mars Satellite repair, maintenance and sensing
Commercial applications (a) (b) (c) (d) (e)
Criminal finger prints random access memory and retrieval for police Security check commercial building entrances Automobile plate identification Large capacity free space interconnection for future computers Border patrol and illegal drug traffic prevention
Acknowledgement The authors would like to acknowledge the support of the research work by the Jet Propulsion Laboratory, California Institute of Technology, under contract with the National Aeronautical and Space Administration. The work of Standard International Corporation, and Professor Pochi Yeh of the University of California at Santa Barbara was funded under a NASA-SBIR contract No. NAS7-1307.
References [l] H. C. Longuet-Higgins, Nature 217 (1968) 104. [2] D. Gabor, IBM J. Res. Dewel. 13 (1969) 156. [3] D. Psaltis and N. Farhat, in ICO-13 Conf. Digest (International Commission for Optics, Amsterdam, 1984), paper A1-9. [4] A. D. Fisher and C. L. Giles, in Proc. of IEEE, Compcon Spring (Institute of Electrical and Electronics Engineers, New York, 1985) 342. [5] G. J. Dunning, E. Marom, Y. Owechko and B. H. Soffer, J. OSA. A2, 13 (1985) 48. [6] H. J. Caulfield, Opt. Commun. 55 (1985) 80. [7] H. Mada, Appl. Opt. 24 (1985) 2063. [8] S. Y. Kung and H. K. Liu, SPIE PTOC.613 (1986) 214. [9] H. K. Liu, S. Y. Kung and J. Davis, Opt. Eng. 25 (1986) 853. [lo] B. H. Soffer, G. J. Dunning, Y. Owechko and E. Marom, Opt. Lett. 11 (1986) 118. [ll]Y. Owechko and B. H. Soffer, Opt. Lett. 16 (1991) 675. [12] A. Yariv, S. K. Kwong and K. Kyuma, Appl. Phys. Lett. 48 (1986) 1114. [13] 0. Changsuk and P. Hankyu, SPIE Proc. 963 (1988) 554. [14] H. Yoshinaga, K. Kitayama and H. Oguri, Opt. Lett. 16 (1991) 669. [15] J. W. Goodman, Introduction to Fourier Optics (McGraw-Hill, New York, 1968).
564
H.-K. Liu et al.
[16] D. Z. Anderson, D. M. Lininger and M. J. O’Callahan, Competitive learning, unlearning and forgetting in optical resonators, IEEE Proc. Conf. on Neural Information Processing Systems - Natural and Synthetic (1987). [17] Y. Z. Liang, D. Zhao and H. K. Liu, Multifocus dichromated gelatin hololens, Appl. Opt. 22 (1983) 2351. [18] J. Davis and J. Waas, Current status of the magneto-optic spatial light modulator, SPIE 0-ELase 89, San Diego. [19] J. Waas and M. Waring, Spatial Light Modulators; A User’s Guide, in The Photonics Design and Applications Handbook 1989, Teddi C. Laurin (ed.) (Pittsfield, MA, 1989). [20] Y. Fainman, E. Klancnik and S. Lee, Optimal coherent image amplification by twowave coupling in photorefractive BaTiO3, Opt. Eng. 25, 2 (1986). [21] G. Valley and M. Klein, Optimal properties of photorefractive materials for optical data processing, Opt. Eng. 22, 6 (1983). 1221 G. Rakuljic, R. Ratnakar et al., Self-starting passive phase conjugate mirror with Ce-doped strontium Barium Niobate, Appl. Phys. Lett. 50, 1 (1987). [23] G. Gheen and L. Cheng, Optical correlators with fast updating speed using photorefractive semiconductor materials, Appl. Opt. 27, 3 (1988). [24] P. Yeh et al., Photorefractive nonlinear optics and optical computing, Opt. Eng. 28, 4 (1989). (251 F. Laeri, T. Tschudi and J. Albers, Coherent CW image amplifier and oscillator using two-wave interaction in a BaTiO3 crystal, Opt. Commun. 47, 6 (1983). [26] J . Fienberg et al., Photorefractive effects and light-induced charge migration in Barium Titanate, J. Appl. Phys. 51, 3 (1980). [27] D. Ledoux and J. Huignard, Two-wave mixing and energy transfer in BaTiO3 application to laser beamsteering, Opt. Commun. 49, 4 (1984). [28] V. Vinetskii et al., Dynamic self-diffraction of coherent light beams, Sou. Phys. Usp. 22, 9 (1979). [29] A. Marrakchi and J. Huignard, Diffraction efficiency and energy transfer in two-wave mixing experiments with Bil2SiOzo crystals, Appl. Phys. 24 (1981). [30] H. Rajbenbach, J. Huignard and B. Loiseaux, Spatial frequency dependence of the energy transfer in two-wave mixing experiments with BSO crystals, Opt. Commun. 48, 4 (1983). [31] B. Fischer et al., Amplified reflection, transmission, and self-oscillation in real-time holography, Opt. Lett. 6, 11 (1981). [32] J. P. Huignard and A. Marrakchi, Coherent signal beam amplification in two-wave mixing experiments with photorefractive BilzSiOzo crystals, Opt. Commun. 38, 4 (1981). [33] A. Yariv, Phase conjugate optics and real-time holography, IEEE J. Quantum Electron. QE-14 (1978) 650-660. 134) A. Vander Lugt, The effects of small displacements of spatial filters, Appl. Opt. 6, 7 (1967) 1221-1225. [35] D. Casasent and A. Farman, Sources of correlation degradation, Appl. Opt. 16, 6 (1977) 1652-1661. [36] A. Shimizu and M. Hase, Entry method of fingerprint image using prism, Trans. Inst. Electronic Commun. Engineers Japan, Part D, J67D,5 (1984) 627. [37] L. Cai, S. Zhou, P. Yeh, Y.Jin, N. Marzwell and H. K. Liu, Translation sensitivity adjustable compact optical correlator and its application for fingerprint recognition, Opt. Eng. 35 (1996) 415. [38] A. B. Vander Lugt, Signal detection by complex spatial filtering, IEEE Trans. Inf. Theory IT-10 (1964) 139-145.
3.4 Multi-Sensory Opto-Electronic Feature Extraction . . . 565 [39] V. V. Horvath, J. M. Holeman and C. Q. Lemmond, Holographic technique recognizes fingerprints, Laser Focus 6 (1967) 18-23. [40] F. T. Gambe, L. M. Frye and D. R. Grieser, Real-time fingerprint verification system, Appl. Opt. 31 (1992) 652-655. [41] C. S. Weaver and J. W. Goodman, A technique for optically convolving two functions, Appl. Opt. 5 (1966) 1248. [42] K. H. Fielding, J. L. Horner and C. K. Makekau, Optical fingerprint identification by binary joint transform correlation, Opt. Eng. 30, 12 (1991) 1958-1961. (431 J. Ohta, J. Sharpe and K. Johnson, An optoelectronic smart detector array for the classification of fingerprints, Opt. Commun. 111 (1994) 451-458. [44] Z.-K. Chen, Y. Sun, Y.-X. Zhang and G.-G. Mu, Hybrid optical/digital access control using fingerprint identification, Opt. Eng. 34,3 (1995) 834-839. [45] A. Vander Lugt, Practical considerations for the use of spatial carrier-frequency filters, Appl. Opt. 5,11 (1966) 176C1765. [46] L. Sadovnik, A. Rizkin, 0. Rashkovskiy and A. A. Sawchuk, All-optical invariant target recognition based on intensity-to-phase coding, Opt. Eng. 35 (1996) 423. [47] X. J. Lu, C. Y. Wrigley and D. A. Gregory, Basic parameters for miniature optical correlators employing spatial light modulators, Opt. Eng. 35 (1996) 429. (481 R. Burzynski, M. K. Casstevens, Y. Zhang and S. Ghosal, Novel optical components: second-order nonlinear optical and polymeric photorefractive materials for optical information storage and processing applications, Opt. Eng. 35 (1996) 443. [49] X. Yang and Z. H. Gu, Three-dimensional optical data storage and retrieval system based on phase-code and space multiplexing, Opt. Eng. 35 (1996) 452. [50] M. Montes-Usategui, J. Compos, J. Sallent and I. Juvells, Complex sidelobe removal by a multichannel procedure, Opt. Eng. 35 (1996) 514. [51] A. Gonzalez-Marcos and J. A. Martin-Pereda, Digital chaotic output from an optically processing element, Opt. Eng. 35 (1996) 525. [52] P. Blonda, V. la Forgia, G. Pasquariello and G. Satalino, Feature extraction and pattern classification of remote sensing data by a modular neural system, Opt. Eng. 35 (1996) 536.
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 567-578 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
[CHAPTER 3.5
I
CLASSIFICATION OF HUMAN CHROMOSOMES - A STUDY OF CORRELATED BEHAVIOR IN MAJORITY VOTE
LOUISA LAM Hong Kong Institute of Education, Northcote Campus, 21 Sassoon Road, Hong Kong E-mail: 1lamOnc.ied.edu.hk and CHING Y.SUEN Centre for Pattern Recognition and Machine Intelligence, Concordia University, Montreal, Quebec H3G 1 M8, Canada Methods for combining multiple classifiers have been developed for improved performance in pattern recognition. This paper examines nine correlated classifiers from the perspective of majority voting. It demonstrates that relationships between the classifiers can be observed from the voting results, that the error reduction ability of a combination varies inversely with the correlation among the classifiers t o be combined, and that the correlation coefficient is an effective measure for selecting a subset of classifiers for combination to achieve the best results. Keywords: Combination of Classifiers; Majority Vote; Classification of Chromosomes.
1. Introduction In pattern recognition, there has been a recent trend towards combining the decisions of several classifiers in order to arrive at improved recognition results. Its culmination arises from a number of reasons, among which are the demands for highly reliable performance imposed by real-life applications, and the difficulty for a single algorithm to achieve such performance. One area in which combinations of classifiers have been widely applied is that of word and character recognition, i s which many different combination methods have been developed and studied. For example, when each of the classifiers outputs a single class, the decisions can be combined by majority vote [1,2]. If the individual classifiers output ranked lists of decisions, these rankings can be used to derive combined decisions by the highest rank, Borda count, logistic regression and other methods [3,4]. Further developments in obtaining a combined decision include statistical approaches [5,6], formulatidns based on Bayesian and DempsterShafer theories of evidence [7,5], the use of neural networks [8] and fuzzy theory [9]. Some very recent theoretical studies and evaluations of classifier combinations 567
568 L. Lam & C. Y. Suen
are presented in [10,11], while comparisons of results from different combination methods can be found in [12-141. In general, the methods applicable t o combining multiple classifier decisions depend on the types of information produced by the individual classifiers. For abstract-level classifiers that output only a unique label or class for each input pattern, suitable combination methods may be a simple or weighted majority vote, the Bayesian formulation, or the Behavior-Knowledge Space (BKS) method 161. Among these methods, the simple majority vote is the only one that does not require prior knowledge of the behavior of the classifiers. For the other methods, this knowledge should be based on the results obtained from a large set of representative data, and the appropriate size of the database increases sharply with the number of pattern classes considered [15]. For example, to combine k classifiers in an n-class problem, a Bayesian formulation would require the estimation of the n2 entries of the confusion matrix for each classifier, while the BKS method would need the determination of O(nk+') such entries. Consequently, to model these behaviors on a problem such as the classification of human chromosomes (a 24-class problem), or to use neural networks to combine the classification results, would demand huge databases. This being the case, the simple majority vote is a reasonable method t o use for combining the decisions of chromosome classifiers. In this work, we study the results of combining nine such classifiers. The perspective will be different from that of previous publications [16,17], in which the majority vote problem has been theoretically analyzed based generally on the assumption of independence of opinions. While independence cannot generally be assumed in practice, it is a closer approximation when the algorithms are developed separately, using different features and classification methods. This happened in some previous applications of majority vote to OCR [1,14]. In the present context, however, all the classifiers in the study make use of the same extracted features, and they differ only in the classification phase. This creates a dependency in their decisions which can be observed in the combined results. We will first examine the effects of this dependence on the combined results, after which some aspects of correlated voting will be studied. Based on a measure of correlation, we will establish a criterion for selecting a subset of classifiers that would be effective in combination.
2. Classification of Human Chromosomes Many different techniques have been used in the automatic classification of human chromosomes into 24 classes (1-22, X, Y). Researchers have reported their results on some standard databases, among which are the Copenhagen, Edinburgh, and Philadelphia datasets. In addition, feature sets have been extracted from these databases and made available by Dr. Jim Piper at Vysis, Inc. (formerly Imagenetics) in Napierville, Illinois. Different classification methods have been used by
3.5 Classification of Human Chromosomes . . . 569 various researchers on these feature sets. These databases and features are summarized below.
2.1. Databases The first database was collected at Rigshospitalet, Copenhagen in 1976-1978 and consists of 180 peripheral blood cells. This database contains the ‘(cleanest” images. The second database was obtained at the Medical Research Council of Edinburgh in 1984, and it contains 125 male peripheral blood cells. The third database was obtained at the Jefferson Medical College, Philadelphia in 1987 and contains 130 chorionic villus cells. This last database is widely recognized as being difficult to analyze. Due to the differences in quality of the datasets, it was found that classifiers should be trained and tested within each database [HI. For this reason, each database is divided into sets A and B, so that each classifier can be trained on one set and tested on the other, after which the roles of the two sets are reversed. The numbers of chromosomes contained in the databases are shown in Table 1. In this paper, test results from both sets are used together for experimentation. Table 1. Summary of databases.
A B Total
Copenhagen
Edinburgh
Philadelphia
3416 4690 8106
2617 2931 5548
2947 3000 5947
For each chromosome, a 30-dimensional feature vector is obtained [18]. The features include normalized area, size, density, convex hull perimeter and length, weighted density distributions, global shape features, as well as centromeric indices. 2.2. Classifiers and Results Classification results are provided by the following nine methods: C1: Constrained classification by transportation method applied to negative log-likelihoods, using 16 out of the 30 features. C2: Same as C1, but all 30 featu,res are used. C3: Constrained classification by transportation method applied to Mahalanobis distance, using 16 out of the 30 features. C4: Same as C3, but all 30 features are used. C5: Constrained classification by transportation method applied to logarithm of Mahalanobis distance after weighting off-diagonal covariance terms by 0.8, using 16 out of the 30 features. C6: Same as C5, but all 30 features are used. C7: A maximum likelihood classifier applied independently to each chromosome.
570 L. Lam & C. Y.Suen Table 2. Error rates of individual chromosome classifiers. Classifier Copenhagen
Database Edinburgh
Philadelphia
3.36 4.89 2.65 3.97 2.05 2.04 4.80 2.62 3.92
14.85 14.62 13.82 14.11 11.88 11.19 16.93 12.44 16.76
19.00 23.68 17.79 21.89 14.53 14.71 23.09 16.53 22.84
c1 c2 c3 c4 c5 C6 c7 C8 c9
C8: Rearrangement classifier. C9: A probabilistic neural network. Of these methods, C1 - C6 are described in [19-21], discussions of C7 and C8 can be found in [22] and [23] respectively, and C9 is presented in [24]. Each of the above classifiers outputs a class label for every chromosome without rejections, and their error rates for each database (sets A and B combined) are shown in Table 2. These results clearly reflect the differences in quality between the databases, and they also show that the Copenhagen database contains images that are probably cleaner than can reasonably be obtained under normal circumstances. 3. Majority Vote Results As stated in Section 1, the simple majority vote is an effective method for combining the decisions of the classifiers in this problem. In particular, it is of interest to note that relationships between the classifiers can be deduced from the majority vote results. The pattern of majority votes has been studied in [17],where the following results have been established: 1. Combinations of even numbers of classifiers tend to produce both lower recognition and error rates than that of odd numbers. Adding one classifier to an even number would increase both the recognition and error rates, while adding this to an odd number would decrease both rates. These conclusions hold regardless of the performances of the individcal classifiers and whether they are independent or not. 2. Assuming that classifiers make independent decisions, adding two classifiers to an even number tends to increase the recognition rate, while the effect on the error rate would depend on the individual performances and cannot be determined a priori. On the other hand, adding two classifiers to an odd number tends to reduce the error rate, while the effect on the recognition rate depends on the individual results.
3.5 Classification of Human Chromosomes . . . 571 When multiple classifiers are used in pattern recognition, it means that the different classifiers will consider the same pattern images, and hence independence of decisions cannot be assumed. In the present context, the classifiers make use of the same feature sets, and are even more strongly dependent than usual. However, the combination results do show the tendencies established in (1) and (2) above. This is illustrated in Fig. 1, which contains the classification results from combining 2 to 6 (out of 7) classifiers on the Copenhagen database. Only combinations among 7 classifiers are shown in this figure, because 2 pairs of classifiers have been found to be highly correlated (as described below), and therefore one classifier has been selected from each such pair. Besides, combining more classifiers would produce a denser distribution of points, which makes the pattern less clearer. From Fig. 1, it can be seen that the addition of one classifier creates a zigzag effect which is stated in (1) above. The behavior of (2) is discernible in that the addition of two classifiers to an even number does cause a general increase in the recognition rate, while such an addition to an odd number produces a movement to the left caused by a lower error rate. This behavior is less prominent, in part due to the dependence among the decisions. To clarify the behavior of majority voting results in Fig. 1, the average rates and ranges of values are shown in Table 3. These values show that adding 2 classifiers to an even number increases both the correct and error rates, while adding 2 classifiers to an odd number mainly decreases the error rate.
m
m 95 -
m
m
m
m
m
m
m
m
m m
94-
m
2 classifiers
X
3 classifiers
+
4 classifiers
*
5 classifiers
0
6 classifiers
m m 93 0.6
m
4
I
I
0.8
1
1.2
I
I
1.4 1.6 ERROR
I
I
1.8
2
2.2
Fig. 1. Results from combinations of classifiers on Copenhagen database.
I 2.4
572 L. Lam €$ C. Y. Suen Table 3. Range of values for combinations on Copenhagen database. Number of classifiers in combination
Average recog. rate
Range of recog. rates
Average error rate
Range of error rates
2 3 4 5 6
95.36 97.52 96.89 97.67 97.26
[93.45,97.251 [96.68,98.041 [96.05,97.431 [97.41,97.941 [97.10,97.461
1.06 1.86 1.19 1.65 1.22
[0.69,1.581 [1.57,2.221 [0.96,1.441 [1.42,1.801 [1.12,1.371
It is noted that combinations of classifiers in this problem are not as effective in reducing errors as in a previous application to OCR [14]. Since this is postulated to be a consequence of correlated behavior, the relation between correlation and error reduction is examined in the next section.
3.1. E m r Reduction and Correlation Majority voting results can be effective indicators of correlations between pairs of classifiers. For example, Table 4 contains the error rates obtained from the majority (also unanimous) decisions of all pairs of classifiers on the Edinburgh database. (In the rest of this paper, results are usually shown for one database when the same patterns of behavior are observed in all three databases). In Table 4, the high error rate resulting from the combination of C1 and C3 (or C2 and C4), when compared to the other entries, shows that these classifiers are highly correlated. Since these four classifiers are actually based on two classifiers using 16 and 30 features respectively, it implies that the transportation method, whether applied to negative log-likelihoods or Mahalanobis distance, tends to make the same misclassifications. The results also indicate that the probabilistic neural network makes different erroneous classifications from all the other methods, and is therefore very effective for reducing the error rates in combination. This is found to be true for combinations of any number of classifiers; in other words, among all combinations composed of the same number of classifiers, C9 is almost always present in the combination that produces the lowest error rate. In order to quantify the reduction in error rates, we define the error reduction ratio E, in the following way. Suppose Eave is the average error rate of two classifiers, and E, is the error rate of their combination. Then E, = (Eaue - Ec)/Eave becomes a “normalized” measure of the effectiveness of majority vote in error reduction, and E, tends to vary inversely with the value of E,. This can be seen by comparing the values of E, shown in Table 4 with those of E, in Table 5. From Table 5, the negative effect of combining C1 and C3 (or C2 and C4) is similarly evident, as opposed to the positive effect of combining C9 with any other classifier. Furthermore, it is found that for the three databases, the average E, (averaged over all different pairs of classifiers) takes on values 0.6548, 0.4917,
3.5 Classification of Human Chromosomes . . .
573
Table 4. Error rates from pairs of classifiers on Edinburgh database.
C1 c2 c3 c4 c5 C6 c7 C8
c2
c3
C4
C5
C6
C7
C8
C9
7.71
12.02 7.43
7.66 12.11 7.35
8.22 6.11 8.98 6.27
6.72 7.30 7.16 7.59 7.44
6.47 7.14 6.52 6.92 6.00 6.90
6.74 7.48 7.12 7.70 7.34 9.52 7.39
5.21 4.87 5.25 4.85 5.68 5.32 5.23 5.43
Table 5. Values of error reduction ratio E,.
C1 c2 c3 c4 c5 C6 c7 C8
C2
C3
C4
C5
C6
C7
C8
C9
0.48
0.16 0.48
0.47 0.16 0.47
0.38 0.54 0.30 0.52
0.48 0.43 0.43 0.40 0.35
0.59 0.55 0.58 0.55 0.58 0.51
0.51 0.45 0.46 0.42 0.40 0.19 0.50
0.67 0.69 0.66 0.69 0.60 0.62 0.69 0.63
and 0.6233 respectively, whereas the average for the OCR algorithms cited in [14] is 0.9261. This supports the conclusion that majority vote is not as effective in reducing misclassifications in the current problem. Another way of testing the validity of E, as a measure of correlation is by comparison with the correlation coefficient p between 2 classifiers. Suppose these classifiers are Ci and Cj with probabilities p i and p j of being correct, r i j is the probability that they are both correct, and u! is the variance of Ci (so that u i = Then
d w ) .
is the coefficient of correlation between Ci and Cj. It is expected that classifiers having high pij would have low ET,and vice versa, which is the case for the databases and classifiers studied here. For exakple, a plot of p versus E, for the Edinburgh database in Fig. 2 shows a strong linear relationship between the two quantities (with coefficient -0.9944). For the Copenhagen and Philadelphia databases, the coefficients are -0.9635 and -0.9863 respectively. For more than 2 classifiers, the p in the above discussion can be replaced by the average correlation coefficient
574
L. Lam & C. Y. Suen V.7
0.85
-
0.8
-
22 0.75 -
1 1l
-
0.65
1
7
2 8
m
0.7
m
m m m
m m x
m
0.6
0.5
-
mn m 3 i
I
Fig. 2. A plot of correlation coefficient versus error reduction ratio.
In the next section, we study the relationship between the combined performance of classifiers and the value of pave, from which we establish a criterion for selecting a subset of classifiers whose combined decision would be most accurate. 4. Behavior of Correlated Voting One of the most cited theorems about voting is Condorcet’s jury theorem (CJT) which shows that, under suitable conditions, a majority of a group is more likely to choose the “better” of two alternatives than any one member of the group. Among the conditions are independence of voting and equal competence among the group members, which prove to be severe limitations in real situations. An interesting generalization of this theorem is presented in [25]. Translated into the present context, this recent publication shows the following: Suppose a group of n (possibly correlated) classifiers has average correct recognition rate pave > 0.5 and combined correct rate P,. Then Pn
> pave if rave
n
pave -
___ * ( n - 1)
(Pave - 0.25)
* (1 - P a v e ) ,
Pave
(3)
Since raveis also a measure of independent voting and requiring raveto be small is asking the same of the average correlation coefficient pave, the above theorem in fact stipulates that if the classifiers are not highly correlated, then a majority vote of the classifiers would produce a higher correct rate than the average.
3.5 Classajiication of Human Chromosomes . . . 575 The above condition, while sufficient, is far from necessary. It has been found that, for the three databases and all combinations of classifiers, the hypothesis is never satisfied even though the 'conclusion is true in 2/3 of all cases. Almost all of the exceptions (when P,, 5 p a v e ) occur in combinations of 2, 4,and 6 (i.e. even numbers of) classifiers, when the combined recognition rate is expected to be lower [17]. Obviously, the hypothesis is far more restrictive than needed in the sense that even when the classifiers are highly correlated, P, can still be greater than pave.
5. Experimental Results Given that P, > pave in most cases, the selection of an effective subset of classifiers for combination cannot depend on the average correct rate. Consequently, we wish to consider the validity of making a selection based on the average correlation coefficient pave between the classifiers, and to verify this criterion by experimentat ion. In order to do this, we examine the performances of all combination pairs containing the same number n of classifiers, where n = 2, 3, . . .,8. Suppose S1 and S2 are two such combinations. They are compared provided that: (i)
1 P,',,, - P&, I
0.02,
where Pive is the average correct rate of classifiers in S,, and pkve is the average correlation coefficient between these classifiers. Condition (i) is imposed so that the comparison can be based on factors other than different levels of performance, and condition (ii) establishes a threshold for significant differences. Under these conditions, and supposing that pive > p i u e , we wish to determine whether S2 produces better performance than S1, where better may mean one of the following cases: (a) P2 > PI and E2 < El, where Pa and E, are respectively the correct and error rates of the combination S,. This is a clear case of superior performance by 5'2. (b) Condition (a) does not hold but R2 > R1, where R, is the reliability or accuracy of S,, and it is defined by R, = P,/(P, E,); in other words, rejections (created by lack of a majority decision) are ignored in the determination of reliability.
+
These will be referred to as Cases (a) and (b) respectively. Case (c) occurs when
R1 > R z , or a lower pave coincides. with a lower reliability, which is the opposite of our hypothesis. The statistics of all comparisons made on the 3 databases are shown in Table 6 . From the results of Table 6 , the following observations can be made: 1. The number of comparisons that can be made depends on the correct rates, which is a consequence of making comparisons only when the average correct rates are commensurate, since these occur more often when the individual correct rates are closer to each other. This is reflected in the databases, where the qualities of
576
L . Lam €5 C. Y. Suen
the data also create different ranges for the correct rates in the increasing order of Copenhagen, Edinburgh, and Philadelphia databases. 2. When only two classifiers are combined, pave is their correlation coefficient, and the correct and error rates both increase or decrease with it in general. Consequently, Case (a) should not happen, and this has been found to be true. 3. The low incidences of Case (c) clearly support our hypothesis. This shows the effectiveness of the average correlation coefficient as an indicator of reliable performance in the majority vote when the votes may be correlated, and it provides a valid basis for selecting a subset of classifiers for optimal combination. Table 6. Results of all comparisons on 3 databases. (a) Copenhagen database
2
Case (a) Case (b) Case (c) Total # pairs
0 43 6 49
Number of classifiers in 3 4 5 465 623 189 181 48 47 40 63 40 276 686 734
combination 6 7 312 58 43 1 8 3 363 62
8 3 0 0 3
(b) Edinburgh database
2
Case (a) Case (b) Case (c) Total #pairs
0 17 0 17
Number of classifiers in combination 3 4 5 6 7 178 29 145 308 405 9 88 10 10 0 6 1 4 4 1 0 160 410 419 189 29
8 0 0
0 0
(c) Philadelphia database
Case (a) Case (b) Case (c) Total # pairs
Number of classifiers in combination 2 3 4 5 6 7 0 65 146 227 100 15 9 8 6 3 1 2 8 0 1 1 3 6 0 0 10 74 212 245 108 15
8 1 0 0 1
6. Concluding Summary
In this paper, we have studied the majority vote patterns of 9 chromosome classifiers and compared the results to established theoretical ones. It is found that the theoretical behavior not based on independence are reflected in these voting results, whereas those based on the independence assumption are less clear. The
3.5 Classification of Human Chromosomes . . . 577 dependence between these classifiers is partly a result of using the same feature sets, but some classifiers are also more highly correlated than others. From the voting results, we can identify pairs of classifiers that are highly correlated (with resulting lower error reduction capability in combination), as well as the classifier that is most effective in contributing to error reduction. Further studies show the strong linear relationship between the average correlation coefficient of classifiers and the error reduction capability of their combination. The extensive experimental results indicate that this coefficient is an effective measure of reliability for combinations of any number of classifiers.
Acknowledgements The authors are deeply grateful to Drs. J. Piper, I. Mitterreiter and W. P. Sweeney for taking considerable trouble to provide their classification results for this study. This research was supported by the Natural Sciences and Engineering Research Council of Canada, the National Networks of Centres of Excellence program of Canada, and the FCAR program of the Ministry of Education of the province of Quebec. The first author is also supported by a research grant from the Hong Kong Institute of Education.
References [l] C. Y. Suen, C. Nadal, R. Legault, T. A. Mai and L. Lam, Computer recognition of unconstrained handwritten numerals, Proc. I E E E 80 (1992) 1162-1180. [2] L. Xu, A Krzyzak and C. Y. Suen, Methods of combining multiple classifiers and their application to handwritten numeral recognition, IEEE Trans. Syst. Man Cybern. 22 (1992) 418-435. [3] T. K. Ho, J. J. Hull and S. N. Srihari, Decision combination in multiple classifier systems, I E E E Trans. Pattern Anal. Mach. Intell. 16 (1994) 66-75. [4] S. Yamaguchi, K. Hagata, T. Tsutsumida, F. Kawamata and T. Wakahara, The third IPTP character recognition competition and study on multi-expert system for handwritten Kanji recognition, Fifth Int. Workshop on Frontiers in Handwriting Recogn. Colchester, UK, Sept. 1996, 479-482, World Scientific, 1997. [5] J. F'ranke and E. Mandler, A comparison of two approaches for combining the votes of cooperating classifiers, Proc. 11th Znt. Conf. Pattern Recogn., The Hague, Netherlands, Sept. 1992, Vol. 2, 611-641. [6] Y. S. Huang and C. Y. Suen, Combination of multiple experts for the recognition of unconstrained handwritten numerals, I E E E Trans. Pattern Anal. Mach. Intell. 17 (1995) 9(t94. [7] E. Mandler and J. Schuermann, Combining the classification results of independent classifiers based on the Dempster/Shafer theory of evidence, in Pattern Recogn. and Artzficial Intell. E. S . Geselma and L. N. Kanal (eds.) (North Holland, Amsterdam, 1988) 381-393. [8] D. S. Lee and S. N. Srihari, Handprinted digit recognition: A comparison of algorithms, Pre-Proc. 3rd Int. Workshop on Frontiers in Handwriting Recognition, Buffalo, USA, May 1993, 153-162. [9] F. Yamaoka, Y. Lu, A. Shaout and M. Shridhar, Fuzzy integration of classification results in a handwritten digit recognition system, Proc. 4th Znt. Workshop on Frontiers in Handwritting Recognition, Taipei, Taiwan, Dec. 1994, 255-264.
578
L. Lam B C. Y. Suen
[lo] J. Kittler, Improving recognition rates by classifier combination, Fifth Int. Workshop on Frontiers in Handwriting Recognition, Colchester, UK, Sept. 1996, 81-101. [ll] G. Pirlo, G. Dimauro, S. Impedovo and S. Rizzo, Multiple experts: a new methodology for the evaluation of the combination processes, Fifth Int. Workshop on Frontiers in Handwriting Recognition, Colchester, UK, Sept. 1996, 131-136. [12] N. Gorski, Practical combinations of multiple classifiers, Fifth Int. Workshop on Frontiers in Handwriting Recognition, Colchester, UK, Sept. 1996, 115-118. [13] T. Tsutsumida, F. Kimura, S. Yamaguchi, K. Nagata and A. Iwata, Study on multiexpert system for handprinted numeral recognition, Fifth Int. Workshop on Frontiers i n Handwriting Recognition, Colchester, UK, Sept. 1996, 11S124. [14] L. Lam and C. Y. Suen, Optimal combinations of pattern classifiers, Pattern Recogn. Lett. 16 (1994) 945-954. [15] L. Lam, Y.-S. Huang and C. Y. Suen, Combination of multiple classifier decisions for optical character recognition, Handbook on Character Recognition and Digital Image Analysis, H. Bunke and P. S. P. Wang (eds.) World Scientific (to appear). [16] L. Lam and C. Y. Suen, A theoretical analysis of the application of majority voting to pattern recognition, Proc. 12th Int. Conf. on Pattern Recognition, Jerusalem, Israel, Oct. 1994, 418-420. (171 L. Lam and C. Y . Suen, Increasing experts for majority vote in OCR: theoretical considerations and strategies, Proc. 4th Int. Workshop on Frontiers in Handwriting Recognition, Taipei, Taiwan, Dec. 1994, 245-254. [18] J. Piper and E. Granum, On fully automatic feature measurement for banded chromosome classification, Cytometry 10 (1989) 242-255. [19] P. Kleinschmidt, I. Mitterreiter and J. Piper, Improved chromosome classification using monotonic functions of Mahalanobis distance and the transportation method, Mathematical Methods of Operations Research 40 (1994) 305-323. [20] P. Kleinschmidt, I. Mitterreiter and C. Rank, A hybrid method for automatic chromosome karyotyping, Pattern Recog. Lett. 15 (1994) 87-96. [21] M. Tso, P. Kleinschmidt, I. Mitterreiter and J. Graham, An efficient transportation algorithm for automatic chromosome karyotyping, Pattern Recogn. Lett. 12 (1991) 117-126. [22] J. Piper, The effects of zero feature correlation assumption on maximum likelihood based classification of chromosomes, Signal Processing 12 (1987) 49-57. [23] J. Piper, Classification of chromosomes constrained by expected class size, Pattern Recogn. Lett. 4 (1986) 391-395. [24] W. P. Sweeney, Jr., M. T. Musayi and J. N. Guidi, Classification of chromosomes using a probabilistic neural network, Cytometry 16 (1994) 17-24. [25] K. K. Ladha, The Condorcet jury theorem, free speech, and correlated votes, Amer. J. Polit. Sci. 36 (1992) 617-634.
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 579-612 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
I CHAPTER 3.6 I DOCUMENT ANALYSIS AND RECOGNITION BY COMPUTERS
YUAN Y. TANG*,', M. CHERIETt, JIMING LIU', J. N. SAID* and CHING Y. SUENt *Department of Computing Studies, Hong Kong Baptist University 224 Waterloo Road, Kowloon Tong, Hong Kong t Centre for Pattern Recognition and Machine Intelligence Conconlia University, 1455 de Maisonneuve Blvd. West Montreal, Quebec H3G 1M8, Canada Surveys of the basic concepts and underlying techniques are presented in this chapter. A basic model for document processing is described. In this model, document processing can be divided into two phases: document analysis and document understanding. A document has two structures: geometric (layout) structure and logical structure. Extraction of the geometric structure from a document refers to document analysis; mapping the geometric structure into logical structure deals with document understanding. Both types of document structures and the two areas of document processing are discussed in this chapter. Two categories of methods have been used in document analysis, namely, (1) hierarchical methods including topdown and bottom-up approaches, (2) nohierarchical methods including modified fractal signatures. Tree transform, formatting knowledge and description language approaches have been used in document understanding. All the above approaches are presented in this chapter. A particular case - form document processing is discussed. Form description and form registration approaches are presented. A form processing system is also introduced. Finally, many techniques, such as skew detection, Hough transform, Gabor filters, projection, crossing counts, form definition language, etc. which have been used in these approaches are discussed in this chapter. Keywords: Document processing, document analysis and understanding, geometric and logical structures, hierarchical and no-hierarchical methods, tree transform, formatting knowledge, description languages, texture analysis.
1. Introduction
Documents contain knowledge. Precisely, they are the medium for transferring knowledge. In fact, much knowledgelis acquired from documents such as technical reports, government files, newspapers, books, journals, magazines, letters, bank cheques, t o name a few. The acquisition of knowledge from such documents by an information system can involve an extensive amount of hand-crafting. Such hand-crafting is time-consuming and can severely limit the application of information systems. Actually, it is a bottleneck of information systems. Thus, automatic knowledge acquisition from documents has become an important subject. Since
580
Y. Y. Tang et al.
the 1 9 6 0 ’ ~ much ~ research on document processing has been done based on Optical Character Recognition (OCR) [71,5]. Some OCR machines which are used in specific domains have appeared in the commercial market. Surveys of the underlying techniques have been made by several researchers [89,70,18,66,29,40,69,83,33].The study of automatic text segmentation and discrimination started about two decades ago [71,49]. With rapid development of modern computers and the increasing need to acquire large volumes of data, automatic text segmentation and discrimination have been widely studied since the early 1980’s [1,98,105]. To date, a lot of methods have been proposed, and many document processing systems have been described [36,75,17,24,72,100,10].About 500 papers have been presented in the International Conferences on Document Analysis and Recognition ICDAR’91, ICDAR’93 and ICDAR’95 [37-391 and nine articles in the special issue of the journal Machine Vision and Applications [50] were concerned with document analysis and understanding, where a lot of papers deal with new achievements of the research on these areas, such as [11,13,42,52,54,59,85,106,48,97].Several books which deal with these topics have been published [9,86,79]. What is document processing? Different definitions have caused a bit of confusion. In this chapter, the definition is chosen from a basic document processing model proposed by [90,95,93]. And its principal ideas will be seen throughout the entire chapter. According to this model, this chapter is organized into the following sections: (i) Basic Model for Document Processing (ii) Document Structure (a) Strength of Structure
(b) Geometric Structure (c) Logical Structure (iii) Document Analysis (a) Hierarchical Methods Top-down Approach Bottom-up Approach (b) No-hierarchical Methods (iv) Document Understanding (a) Tree Transform Approach (b) Formatting Knowledge Approach (c) Description Language Approach
(v) Form Document Processing (a) (b) (c) (d)
Characteristics of Form Documents Form Description Language Approach Form Registration Approach A Form Document Processing System
3.6 Document Analysis and Recognition by Computers 581 (vi) Major Techniques Hough Transform, Skew Detection, Projection Profile Cuts, Run-Length Smoothing Algorithm (RLSA), Neighborhood Line Density (NLD), Connected Components Algorithm, Crossing Counting, Form Description Language (FDL), Texture Analysis, Local Approach, Other Segmentation Techniques. 2. A Basic Model for Document Processing A basic model for processing the concrete document was first proposed in our early work, which was presented at the First International Conference on Document Analysis and Recognition [95],and also appeared in the Handbook of Pattern Recognition and Computer Vzsion [94]. A graphic illustration can be found in Fig. 1, where the relationships among the geometric structure, logical structure, document analysis and document understanding are depicted.
Fig. 1. Basic Model for Document Processing.
The following principal concepts .were proposed in this model: A concrete document is considered to have two structures: the geometric (or layout) structure and the logical structure. Document processing is divided into two phases: document analysis and document
understanding. Extraction of the geometric structure from a document is defined as document analysis; mapping the geometric structure into a logical structure is defined as
Y. Y. Tang e t al.
582
0
document understanding. Once the logical structure has been captured, its meaning can be decoded by A1 or other techniques. But in some cases, the boundary between the two phases just described is not clear. For example, the logical structure of bank cheques may also be found during an analysis by knowledge rules. The basic model of document processing can be formally described below [94]:
Definition 1. A document R is specified by a quintuple
such that 3 = {Ol,O2,. . . ,Oi,. . . , O m } ,
where
and
where 0 0 0
0
0 0
3 is a finite set of document objects which are sets of blocks 0 2 ( i = 1 , 2 , . . . , m). {Of}* denotes repeated sub-division. a is a finite set of linking factors. cpl and cpr stand for leading linking and repetition linking respectively. S is a finite set of logical linking functions which indicate logical linking of the document objects. CY is a finite set of heading objects. p is a finite set of ending objects.
Definition 2. Document processing is a process t o construct the quintuple represented by Eqs. (1)-(3). Document analysis refers t o extracting elements 3, O i and Of in Eq. (2), i.e., extraction of the geometric structure of 0. Document understanding deals with finding a, 6 , a, and p in Eq. (3), considering the logical structure of R.
3.6 Document Analysis and Recognition by Computers 583
Fig. 2. A simple example of document processing described by the basic model.
Example - A simple example is illustrated in Fig. 2, we have
cs = { 0 1 , 0 2 , 0 3 , 0 4 , 0 5 } 04=
{04}*= {Of, @}
0 5 =
{0j5}* = {0;,0;,0;}
a = {01,02}
p
=
{04, 05)
From the above definition, it is obvious that there is a nondeterministic mapping from the geometric structure into the logical structure. However, as the geometric structure is extracted, a deterministic mapping can be achieved. It can be formally described below:
Theorem 1. Let R be a document defined by a quintuple (3,@, 6, a , p) having nondeterministic mapping from geometric structure into logical structure, then there exists a quintuple (V, a’,d’, a’, p’) ’which contains a deterministic mapping from the geometric structure of R into a logical structure. 3. Document Structures The key concept in document processing is that of structure. Document structure is the division and repeated subdivision of the content of a document into
584
Y. Y. Tang et al.
increasingly smaller parts which are called objects. An object which can not be subdivided into smaller objects is called a basic object. All other objects are called composite objects. Structure can be realized as a geometric (layout) structure in terms of its geometric characteristics, or a logical structure due to its semantic properties.
3.1. Strength of Structure To measure a document structure, a Strength of Structure S, has been introduced [104].
Definition 3. Suppose a document is divided into n objects associated with n variables. Hi stands for the partial entropy of the i-th variable, and H for the entropy of the whole document. The strength of structure is n
S, =
C H~
-
H.
(4)
i=l
For instance, if the entire document consists of four composite objects associated with the variables ~ 1 - 2 4 , the strength will be 4
n
n
3.2. Geometric Structure Geometric structure represents the objects of a document based on the presentation, and connection among these objects. According to the International Standard IS0 8613-1:1989(E) [44], the geometric or layout structure can be defined below:
Definition 4. Geometric or layout structure is the result of dividing and subdividing the content of a document into increasingly smaller parts, on the basis of the present at ion. Geometric (Layout) Object is an element of the specific geometric structure of a document. The following types of geometric objects are defined: Block is a basic geometric object corresponding to a rectangular area on the presentation medium containing a portion of the document content; n a m e is a composite geometric-object corresponding to a rectangular area on the presentation medium containing either one or more blocks or other frames; Page is a basic or composite geometric object corresponding to a rectangular area, if it is a composite object, containing either one or more frames or one or more blocks; Page Set is a set of one or more pages;
3.6 Document Analysis and Recognition by Computers 585 0
Document Geometric (Layout) Root is the object at the highest level in the hierarchy of the specific geometric structure. The root node in the above example represents a page.
The geometric structure can be formally described by the following definition according to the basic model given by Eqs. (1) and (2).
Definition 5. The geometric structure is described by the element S in the document space R = (3, 9,6 , a, 0) shown in Eqs. (1) and (2) and Pu which is a set of operations performed on S such that 3 = {SB,SC}
where S B represents a set of Basic objects, and SC stands for a set of Composite objects. Sc = {O1,02,.
. . ,Om}
This is a general definition of the geometric structure. Different types of specific documents have their specific forms. For example, for a specific document shown in Fig. 3(a) which is a page extracted from a newspaper, it is divided into several blocks as illustrated in Fig. 3(b). According to the above general model, its specific document geometric structure can be presented graphically in Fig. 3(c). In this page, a document is divided into several composite objects - text areas and graphic areas, which are broken into headline blocks, text line blocks, graphic blocks, etc. 3.2.1. Geometric Complexity The geometric complexity of a document can be measured by a complexity function p which is defined below:
Definition 6. Let ISTI and l S ~be l the number of elements in sets ST and SG respectively. Complexity function p can be presented as p=
lSTl + l S G l .
(7)
In terms of the complexity, documents can be classified into four categories: 0
0 0 0
Documents without graphics (e.g. editorials): SG = 4; Document forms (e.g. bank cheques and other business forms): SG = {OF}; Documents with graphics (e.g. general newspaper articles): SG # 4; Documents with graphics as the main elements (e.g. advertisements, front page of magazine): ISTI 5 ISG~.
586
Y. Y. Tang et al.
Fig. 3. Geometric structure and logical structure of a page in a newspaper.
3.3. Logical Structure
Document understanding emphasizes the finding of logical relations between the objects of a document. To facilitate this process, a logical structure and its model have been developed in our early work [95] which can be summarized as shown below.
3.6 Document Analysis and Recognition by Computers 587 Logical structure represents the objects of a document based on the humanperceptible meaning, and connection among these objects. According to the International Standard I S 0 8613-1:1989(E), the logical structure can be defined as follows [44]:
Definition 7. Logical structure is the result of dividing and subdividing the content of a document into increasingly smaller parts, on the basis of the human-perceptible meaning of the content, for example, into chapters, sections, subsections, and paragraphs. Logical Object is an element of the specific logical structure of a document. For logical object, no classification other than Basic logical object, Composite logical object and Document logical root is defined. Logical object categories such as Chapter, Section and Paragraph are application-dependent and can be defined using the Object class mechanism [44]. The document understanding process finds the logical relations between the objects of a document. According to the basic model represented by Eqs. (1)and (2), a formal description of the logical structure is presented as follows:
Definition 8. The logical structure is described by the elements CP, 6, a , and P in the document space R = (S,CP, 6, a , P ) in Eqs. (1) and (2), such that
For a specific document shown in Fig. 3, its logical structure can be represented graphically by Fig. 3(d).
4. Document Analysis Document analysis is defined as the extraction of the geometric structure of a document. In this way, a document image is broken down into several blocks, which represent coherent components of a document, such as text lines, headlines, graphics, etc. with or without the knowledge regarding the specific format [100,95]. This structure can be represented as a geometric tree shown in Fig. 3(c). To build a such tree, there are many methods which can be classified into two categories: Hierarchical Methods: When we break a page of document into blocks, we consider the geometric relationship among the blocks. In this way, we have three approaches, i .e . Top-down approach Bottom-up approach - Adaptive split-and-merge approach
-
588
Y. Y. Tang et al.
No-hierarchical Methods: When we are break a page of document into blocks, we do not consider the geometric relationship among the blocks.
4.1. Hierarchical Methods
In the hierarchical methods, we have two ways: (1) from parents t o children, or (2) from children to parents. Corresponding to these two ways, there are two approaches: top-down and bottom-up approaches. These two approaches have been used in document analysis. Each has its advantages and disadvantages. The top-down approach is fast and very effective for processing documents that have a specific format. On the other hand, the bottomup approach is time consuming. But it is possible t o develop algorithms which are applicable to a variety of documents. A better result may be achieved by combining the two approaches [73]. 4.1.1. Top-Down Approach The top-down (knowledge based) approach proceeds with an expectation of the nature of the document. It divides the document into major regions which are further divided into sub-regions, etc. [28,32,42,55-57,60,75,77]. The geometric structure of a document can be represented by a tree. Suppose this tree contains K levels. Figure 4 indicates the i-th and (i 1)-th levels. Suppose the upper layer has nodes N j , N i , . . . , N A ; and the lower layer has nodes N ; + l , N;+l, . . . , N;+l. The relations between these two layers are expressed by edges between the nodes. They can also be represented in the form of
+
1 1 ... 1 0 0 ... 0 0 0 ... 0 0 0 ... 0
... 0 0
1 1
... 1 0 0 ... 0
1
...
... 0 0 0 ... 0 1 1 ...
1
Values 1's in Eq. (9) correspond t o the edges in Fig. 4 meaning that
3.6 Document Analysis and Recognition by Computers
Fig. 4. The ith and (i
589
+ 1)th levels of a structure tree.
Equation (9) gives two ways, “+” from left t o right corresponding to “from top to bottom” in the tree structure (Fig. 4), and “t-” from right t o left corresponding to “from bottom to top” in the same structure. In the top-down approach, the former way is used, and a document is divided into several regions each of which can be further divided into smaller sub-regions. Let S be the set of objects which can be split into w disjoint subsets 01,02,. . . , OP, . . . , O”, opts,
p = 1 , 2 ,.”,
21
A C-function [lo41 has been defined as C(0P) 2 0,
p = 1 , 2 , ..., w
such that
C(0Pu 0 4 ) 2 C(OP)+ C ( O Q ) . From (4), the strength of structure S, will be S,(OP, 0 9 )
= C(OP u 0 9 ) - C(OP)- C ( 0 Q 2) 0.
(10)
The criterion of topdown splitting is that we should divide 0‘ = OP U OQ into two subsets OP and 0 4 such that the strength of structure S, becomes minimum. This policy will maximize the intra-subset cohesion and minimize the inter-subset cohesion.
590
Y. Y. Tang et al.
For multiple splitting, the strength of structure Ss(OP, 0 9 , . rived by repeating Eq. (10):
. . , OY) can be de-
To achieve a good splitting, S,(OP, 0 9 , . . . , OY) should be minimized. Many methods have been employed in the top-down approach, e.g. smearing [49,105,51],projection profile cut [74,46,4,56,60],Fourier transform detection [35], template [21], and form definition language (FDL) [32,28]. 4.1.2. Bottom-Up Approach The bottom-up (data-driven) approach progressively refines the data by layered grouping operations. The bottom-up approach is time consuming. But it is possible to develop algorithms which can be applied to a variety of documents [105,22,41,36,46,17,27,4,78]. The bottom-up approach corresponds to the direction of "+" in Eq. (9). In this way, basic geometric components are extracted and connected into different groups in terms of their characteristics, then the groups are combined into larger groups, etc. An analysis of this approach based on the entropy theory is given in terms of the dynamic coalescence model [104]. In this model, we start with N ( 0 ) objects of equal "mass", suppose a region is formed by m original objects, such that this region has a mass m. N ( t ) stands for the number of regions at time t. X ( * ) , R(Q)and represent the position, size and mass of the a-th region respectively. We have
N ( 0 ) > N ( t ) > N(2t) > . . . N ( n t )
R(a)= 4 n ] ( M ( a ) ) & , where Ro indicates a constant called coalescence parameter. The dynamic equation can be represented in the form of
where
3.6 Document Analysis and Recognition by Computers 591 If we want t o include the second order effect in the equation in order to enhance the chain effect, then Eq. (12) can be replaced by the following formula
where E is a constant to be adjusted. Two blocks a and /3 coalesce into a new block y when they satisfy the following condition: IX(P) - X(")I = R(P) - R(")
There are two practical bottom-up methods: (1) neighborhood line density (NLD) indicating the complexity of characters and graphics [57,45,46];and (2) connected components analysis indicating the component properties of the document blocks [64,88,8,27,78]. 4.1.3. Adaptive Split-and-Merge Approach Liu, Tang, and Suen have developed an adaptive split-and-merge approach [62,63], which draws on the conventional split-and-merge image-processing advantage of spontaneous separation of unhomogeneous regions and merging of homogeneous ones, and furthermore empowers such a process with an adaptive thresholding operation that computes the segmentation borders. The novelty of their approach consists in the document representation of a tree-like data structure that results from the block identification can readily be utilized in reasoning about the geometric relationships in those blocks. As an integral step of this approach, the relative spatial relationships are inferred at the same time as the block convergence takes place. The proposed approach has been implemented and tested with real-life documents. 4.2. N o -hiemrchical Methods
Traditionally, two approaches have been used in document analysis, namely, topdown and bottom-up approaches [93]. Both approaches have their weaknesses: They are not effective for processing documents with high geometrical complexity. Specifically, the topdown approach can process only the simple documents which have specific format or contain some a priori information. It fails t o process the documents which have complicated geometric structures. To extract the geometric (layout) structure of a document, the top-down approach needs iterative operations t o break the document into several blocks while the
592
Y. Y. Tang et al.
bottom-up approach needs t o merge small components into large ones iteratively. Consequently, both approaches are time consuming. Tang e t al. [91] presented a new approach based on modified fractal signatures for document analysis. It does not need iterative breaking or merging, and can divide a document into blocks in only one step. This approach can be widely used to process various types of documents including even some with high geometrical complexity. An algorithm has been developed in [91], and is briefly presented as follows:
Algorithm 1. (fractal signature) Input: a page of document image; Output: the geometric structure of the document; Step-1 For x = It0 Xmax do For y = 1 t o YmaXdo F is mapped onto a gray-level function gk(z, y); Step-2 For x = 1 to X,, do For y = 1 to Yma, do Substep-1 Initially, taking 6 = 0, the upper layer u,"(z,y) and lower layer bE(z,y) of the blanket are chosen as the same as the gray-level function gk (z, y), namely:
Substep-2 Taking 6 = 61, (a) u g , (z,y) is computed according t o the formula:
(b) bg, (z,y) is computed according t o the formula:
(c) The volume Volg, of the blanket is computed by the formula: VOl6,
= c ( u 6 1(z, 51) - b6l (z, Y)); X>Y
Substep-3 Taking 6 = 6 2 , (z, y) is computed according to (a)
3.6 Document Analysis and Recognition by Computers 593 (b) b g 2 ( z ,y) is computed according to
(c) The volume Vols, of the blanket is computed by
Step-3 The sub fractal signature A: is computed by the formula:
At =
Volg, - V O l 6 , 2
Step-4 Combining sub fractal signatures A:, k = 1 , 2 , .. . , n into the whole fractal signature: n
A6 = IJ A t . k=l
5. Document Understanding
As document analysis extracts geometric structures from a document image by using the knowledge about the general document and/or the specific document format, document understanding maps the geometric structures into logical structures considering the logical relationship between the objects in specific documents. There are several kinds of mapping methods in document understanding: [loo] proposed a tree transformation method for understanding multi-article documents. [98] discussed the extraction of Japanese newspaper articles using a domain specific knowledge. [41] constructed a special purpose machine for understanding Japanese documents. [32] proposed a flexible format understanding method, using a form definition language. Our research [92,107,94] has led to the development of a form description language for understanding financial documents. These mapping methods are based on specific rules applied to different documents with different formats. A series of document formatting rules are explicitly or implicitly used in all these understanding techniques. In this section, document understanding based on tree transformation, document formatting knowledge and document description language will be discussed. 5.1. Document Understanding Based o n Free Transformation
This method defines document understanding as the transformation of a geometric structure tree into a logical structure tree [loo]. A document has an obvious hierarchical geometric structure, represented by a tree, and the logical structure of a document is also represented by a tree. Three
594
Y. Y. Tang et al.
kinds of blocks are defined: H (head), B (body) and S (either body or head). During the transformation, a label is attached to each node. Labels include title, abstract, sub-title, paragraph, header, foot note, page number, and caption. The transformation, which moves the nodes in the tree, is based on four transformation rules. These rules are created according to a layout designed according to the manner in which humans read. Rules 1 and 2 are based on the observation that a title should have a single set of paragraphs as a child in the logical structure. The paragraph body in another node is moved to the node under the body title by these rules. Rule 3 is mainly for the extraction of characters or sections headed by a sub-title. By rule 4, a unique class is attached to each node. This method was implemented on a SUN-3 workstation. Pilot experiments were carried out using 106 documents taken from magazines, journals, newspapers, books, manuals, letters, scientific papers, and so on. The results show that only 12 out of 106 tested documents were not interpreted correctly. 5.2. Document Understanding Based on Formatting Knowledge Since a logical structure can correspond to a variety of geometric structures, the generation of logical structure from the geometric structure is difficult. One of the promising solutions to this problem is the use of formatting knowledge. The formatting rules may differ from each other because of the type of document and language to be used in it. However, for a specific kind of document, once the formatting knowledge is acquired, its logical structure can be deduced. An example can be found in [98] where a method of extracting articles from Japanese newspapers has been proposed. In this method, six formatting rules of Japanese newspaper layout are summarized. An algorithm for extracting articles from Japanese newspaper has been designed based on the formatting knowledge. Another example can be found in [20] where a business letter processing approach has been developed. Because business letters are normally established in a single-column representation, letter understanding is mainly the identification of the logical objects, like sender, receiver, date, etc. In this approach, the logical objects of the letter are identified according to a Statistical Database (SOB). As the author reported, the SDB consists of about 71 rule packages derived from the statistical evaluation of a few hundred business letters. Other knowledge, like the shape, size and pixel density etc. of the image block can also be used for document understanding. References [108,23] use statistical features of connected components to identify the address blocks on envelopes.
5.3. Document Understanding Based on Description Language One of the most effective ways to describe the structures of a document is the use of a description language. [32] detects the logical structure of a document and makes use of the knowledge rules represented by a f o r m definition language ( F D L ) . The basic concept of the form definition language is that both the geometric and
3.6 Document Analysis and Recognition by Computers 595 logical structures of a document can be described in terms of a set of rectangular regions. For example, a part of a program in form definition language coded for the United Nations’ (UN) documents is listed below: (defform UN-DOC# (width 210) (height 297) (if (box (? ? ? ?) (mode IN Y LESS) (area (0 210 60 100)) (include (160 210 1 5))) (form UN-DOC-A (0 210 0 297)) (form UN-DOC-B (0 210 0 297)))) (defform UN-DOC-A . . . ) (defform UN-DOC-B . . .) It means that the UN documents have a width of 210 mm and a height of 297 mm. The if predicate is one of the control structures. If the box predicate succeeds, the document named UN-DOC# is compared with UN-DOC-A and UNDOC-B, and analyzed as UN-DOC-A. Otherwise, it is analyzed as UN-DOC-B. The box states that a rule line should exist inside the region (0 210 60 100) and satisfy the conditions that the width of the ruled line is between 160 mm and 210 mm and the height is between 1 mm and 5 mm. (defform UN-DOC-A . . . ) and (defform UN-DOC-B . . . ) will give the definition of the UN documents with and without a ruled line with these properties stated above. According to the definition, a form dividing engine will analyze the document and produce the images of some logical objects, such as the organization which issued the document, document number, and section, etc. More details about this method can be found in [32]. 6. Form Document Processing
Form document is a type of special-purpose documents commonly used in our daily life. For example, millions of financial transactions take place every day. Associated with them are form documents such as bank cheques, payment slips and bills. For this specific type of documents, according to their specific characteristics, it is possible to use specific methods to acquire knowledge from it. 6.1. Chamcteristics of F o m Documents
Specific characteristics of form documents have been identified and analyzed in our early work [96,92,107,94]which are listed below: In general, a form document may consist of straight lines which are oriented mostly in horizontal and vertical directions.
596 0
0
Y. Y. Tang et al.
The information that should be acquired from a form is usually the filled data. The filling positions can be determined by the above lines as references. Texts in form documents often contain a small set of known machine-printed, hand-printed and handwritten characters, such as legal and numeric amounts. They can be recognized with current character recognition techniques.
6 . 2 . Form Document Processing based on Form Description Language
According to the above analysis, a form document processing method based on form description has been proposed in [96,92,107,94]. A block diagram of this method is illustrated in Fig. 5. The goal of this method is to extract information called items from the form documents.
Fig. 5. Diagram of form processing based on the FDPL.
To acquire the items from the form documents, the item description (IDP) has been developed [107,94]. Suppose there exists a finite set of relations r = {rl,rz, . . . , r h } between the finite set of items a = (01, a2, . . . , a,} and the finite set of graphs C = {El, C2, . .. , C,}, and it can be represented by 0-ri matrix. We call it an Item Description Matrix: M I D ,such that
3.6 Document Analysis and Recognition by Computers 597 satisfying the following condition:
where R, L , A and B represent Right, Left, Above and Below respectively. For example, the finite set of items and the finite set of graphs are given by a = { a l , a2, a 3 , a4} and c = { L l , L2, L3, L4, L5, L 6 ) respectively. Let r = {R,L , A , B } . M I D is represented by the following matrix:
Equation (15) means that
(a) (b)
is located above line L1 and also on the left of line L4; is located below line L1 and above line L2 and also on the right of line L4 and left of line L5; (c) a3 is located below line L3 and also on the right of line L5 and left of line L6; (d) a4 is located below line L3 and also on the right of line L6. a1 a2
6.3. Form Document Processing based on Form Registration
A form document processing system based on the pre-registered empty forms has been developed in [75]. The process includes two steps: (1) empty form registration, and (2) data filled form recognition. During the registration step, a form sample without any data is first scanned and registered with the computer. Through line enhancement, contour extraction and square detection, both the label and data fields are extracted. The relationships among these fields are then determined. Man-machine conversation is required during this registration process. The result of registration is stored as the format data of the form sample. During the recognition step, only the data fields are extracted according to the locations indicated by the format data. 6.4. Form Document Processing System
An intelligent form processing system (IFPS) has been described in [lo]. It provides capabilities for automatically indexing form documents for storage/retrieval to/from a document library, and for capturing information from scanned form images using OCR software. The IFPS also provides capabilities for efficiently storing form images. The overall organization of IFPS is shown in Fig. 6, which contains
598
Y. Y. Tang et al. Difference image
Data extraction lications
Fig. 6. An intelligent document form processing system.
two parallel paths, one for image applications such as retrieval, display and printing of a form document, the other for data processing applications that deal with information contained on a form. IFPS consists of six major processing components: 0 0 0 0
0 0
Defining form model; Storing the form model in a form library; Matching input form against the model stored in the form library; Registering the selected model to the input form; Converting the extracted image data to symbol code for input t o data base; Removing the fixed part of a form, and retaining only the data filled in for storage.
7. Major Techniques To implement the above approaches, a lot of practical techniques have been developed. In this section, the major techniques will be presented. 0
0 0 0 0 0 0 0 0 0 0
Hough Transform, Skew Detection, Projection Profile Cuts, Run-Length Smoothing Algorithm (RLSA), Neighborhood Line Density (NLD) , Connected Components Algorithm, Crossing Counting, Form Description Language (FDL), Texture Analysis, Local Approach, Other Segmentation Techniques.
3.6 Document Analysis and Recognition by Computers 599
7.1. Hough Transform The Hough transform maps points of Cartesian space (s,y) into sinusoidal curves in a ( p , 8 ) space via the transformation: p = zcos(8)
+ ysin(8).
Each time a sinusoidal curve intersects another at particular values of p and 8 , the likelihood that a line corresponding to these p8 coordinate values is present in the original image also increases. An accumulator array (consisting of R rows and T columns) is used t o count the number of intersections at various p and 8 values. Those cells in the accumulator array with the highest number of counts will correspond to lines in the original image. Because text lines are actually thick lines of sparse density, Hough transforms can be used t o detect them and their orientation. Three major applications of the Hough transform in document analysis are listed below: 0 Skew Detection: An important application of Hough transform is skew detection. A typical method can be found in [34]. It detects the document skew by applying the Hough transform to a “burst image”. At first, the resolution of the document image is reduced from 300 dpi (dots per inch) t o 75 dpi. Next, a vertical and a horizontal burst image will be produced based on the reduced document image. The Hough transform is then applied t o either the vertical or the horizontal burst image according to the orientation of the document. Compared to the original image, the number of black pixels in the burst image is significantly reduced compared t o the original image. It speeds up the skew detection procedure. In order to eliminate the negative effects of the large run-length contributed by the figures and black margins, only small run-lengths between 1 and 25 pixels are mapped to the ( p , 8 ) space. The skew angle can then be calculated according to the accumulator array. In [34], all skews have been detected correctly for the thirteen test images of five different types of documents. 0 Text block identification: The accumulator array produced by the transform has different properties corresponding t o the different contents of the document images. The high peaks in the array correspond t o graphics in the document, while the cells with regular value and uniform width in the array correspond to texts in the document [82,87]. Thus, the different document contents can be identified according to these properties. 0 Grouping the characters in a line for text/graph separation: The Hough transform can also be used t o iletect the text lines by means of grouping the characters together and separating them from the graphics [27].
7 . 2 . Techniques for Skew Detection Many techniques have been applied to skew detection, and many algorithms have been developed [79].For example, Akiyama and Hagita [4] developed an automated
600
Y. Y. Tang et al.
entry system for skewed documents. But it failed in a document which is mixed of text blocks, photographs, figures, charts, and tables. Hinds, Fisher and D’amato [34] developed a document skew detection method using run-length encoding and the Hough transform. Nakano, Shima, Fujisawa, Higashino and Fujiwara [76] proposed an algorithm for the skew normalization of a document image based on the Hough transform. These methods can handle documents in which the non-text regions are limited in size. Ishitani [43]proposed a method t o detect skew for document images containing a mixture of text areas, photographs, figures, charts, and tables. To handle multi-skew problems, [log] developed a method using Least Squares, and the basic idea is presented below: Given a set of N data points, i.e., the reference points of the text line, a linear function is assumed to exist between the dependent variable f (x)and the independent variable x,
They can therefore be solved t o give
Consequently, from a1 and a2, the slope of the text block as well as the skewed angle 0; can be calculated. Furthermore, the skewed angle can be rotated t o the correct position.
7.3. Projection Profile Cuts Projection refers to the mapping of a two-dimensional region of an image into a waveform whose values are the sums of the values of the image points along some specified directions. A projection profile is obtained by determining the number of black pixels that fall onto a projection axis. Projection profiles represent a global feature of a document. They play a very important role in document element extraction, character segmentation and skew normalization. Let f ( x , y ) be a document image, and R stand for a n area of the document image. Assume that f(x,y) = @lies outside the image. b[. . .] denotes a delta function. t = xsin4 - ycos4 gives the Euclidean distance of a line from the origin [80]. If the projection angle from the x-axis is 4, the projection can be defined as follows: f(x,y)b[xsin@- ycos4 - tldzdy . p(4, t) = (19)
L
For a digitized image, the symbol
sRshould be replaced by CR.
3.6 Document Analysis and Recognition by Computers 601 All objects in a document are contained in rectangular blocks. Blanks are placed between these rectangles. Thus, the document projection profile is a waveform whose deep valleys correspond to the blank areas of the documents. A deep valley with a width greater than an established threshold, can be cut as the position corresponding to the edge of an object or a block. Because a document generally consists of several blocks, the process of projection should be done recursively until all of the blocks have been located. More details about various applications of this technique in the document analysis can be found in Refs. [74,46,99,103,4,101]. 7.4. Run-Length Smoothing Algorithm (RLSA)
The basic RLSA is applied to a binary sequence in which white pixels are represented by 0’s and black pixels by 1’s. It transforms a binary sequence x into an output sequence y according to the following rules: (a) 0’s in x are changed to 1’s in y, if the number of adjacent 0’s is less than or equal to a predefined limit C. (b) 1’s in x are unchanged in y.
For example, with C = 4 the sequence x is mapped into y as follows:
x : 00010000010100001000000011000 y : 11110000011111111000000011111
When applied to pattern arrays, the RLSA has the effect of linking together neighboring black areas that are separated by less than C pixels. With an appropriate choice of C, the linked areas will be regions of a common data type. The degree of linkage depends on the following factors: (a) the threshold value C, (b) the distribution of white and black pixels in the document, and (c) the scanning resolution. On the other hand, the RLSA may also be applied to the background. It has the effect of eliminating black pixels that are less than C in length [49]. The choice of the smoothing threshold C is very important. Very small horizontal C values simply close individual characters. Slightly larger values of C merge together individual characters in a word, but are not large enough to bridge the space between two words. Too large values of C often cause sentences to join to non-text regions, or to connect to adjacent columns. In general, the threshold C is set according to the character height, gap between words and interline spacing [49,26]. The RLSA was first proposed by Johnston [49] to separate text blocks from graphics. It has also been used to detect long vertical and horizontal white lines [1,102]. This algorithm was extended to obtain a bit-map of white and black areas representing blocks which contain various types of data [105]. Run-length smoothed document images can also be used as basic features for document analysis [24,26,28].
602 Y. Y. Tang et al.
7.5. Neighborhood Line Density (NLD) For every pixel on the document, its NLD is the sum complexity of its four directions.
ci =
c
(l/Lij)
iEN
N = (L,R,U,D), where, L, R, U and D stand for the four directions, i.e., left, right, up and down respectively. Ci indicates the complexity of a pixel for the direction i. Lij represents the distance from the given pixel to its surrounding stroke j in the direction i. Based on the following features, the NLD can be used to separate characters from graphics including the situation that some characters are touching the graphics: (1) NLD is higher for character fields than graphic fields, and (2) there are high peaks of NLD in the character fields and their height is affected by the character’s size and pitch [58]. The NLD consists of three processing steps. First, the NLD for all the black pixels of the input document is calculated using the method stated above. Second, an NLD emphasis processing is carried out in order to enlarge the NLD difference between the graphic fields and character fields. The third step is thresholding, the pixels which have an NLD value greater than a threshold f? are classified as character fields, otherwise they are classified as graphic fields.
7.6. Connected Components Analysis (CCA)
A connected component is a set of connected black or white pixels such that an 8-connected path exists between any two pixels. Different contents of the document tend to have connected components with different properties. Generally, graphics consist of large connected components. Texts consist of connected components with regular and relatively smaller size. By analyzing these connected components, graphics and texts in the document can be identified, grouped together to different blocks and separated from each other. The size and location of the connected component can be represented by a fourtuple [98]. The analysis of a document can be regarded as the process of merging these four-tuples. Taking the newspaper as an example, its content is classified into several regions like index, abstract, article body, picture and figure, etc. During image analysis, the four-tuples are merged and classified into these regions using the features found in the regions. In [98], 13 features about the six regions of Japanese newspaper are summarized. According to these features, a table is created summarizing the properties of the four-tuples in each region. All the four-tuples can be classified and merged following the rules described in this table. Since the
3.6 Document Analysis and Recognition by Computers 603
four-tuples contain information about the location of the components, all the regions can be classified and located at the end of the four-tuple merging process. Two typical applications of the CCA in document processing can be illustrated below.
Envelope Processing: An important application is automatic envelope processing [108,6,7,19,23,61].By placing the connected components into several groups and further analyzing the components in them, CCA has been used to locate address blocks on envelopes [108]. Mixed Text/Graphics Document Processing: [27] describes the development and implementation of a Robust algorithm where the CCA is successfully used to separate a text string from a mixed text/graphics document image. This algorithm consists of five steps: (a) Connected component generation, (b) Area/ratio filter, (c) Collinear component grouping, (d) Logical grouping of strings into words and phrases, and (e) Text string separation. 1'.7. Crossing Counts
A crossing count is the number of times the pixel value turns from 0 (white pixel) to 1 (black pixel) along horizontal or vertical raster scan lines. It can be expressed as a vector whose components are defined as follows. (1) Horizontal crossing counts:
(2) Vertical crossing counts:
Crossing counts can be used to measure document complexity. In [4],crossing counts have been used as one of the basic features to separate and identify the document blocks.
7.8. Form Definition Language (FDL) [32]proposed a top-down kno&edge representation called Form Definition Language (FDL), to describe the generic layout structure of document. The structure can be represented in terms of rectangular regions, each of which can be recursively defined in terms of smaller regions. An example is given in Fig. 7. These generic descriptions are then matched to the preprocessed input document images. This method is powerful, but is complicated to implement. [28] developed a simplified version of FDL so that it may be implemented more easily.
604
Y. Y. Tang et al.
I
10-
40- .....................
F 2 w -
(defform F (form F1 (10, 90, 10,40)) (form ) (form F3 )) (defform F1 (form F11 ) (form F1.2
............ ............ ............ ............
))
Fig. 7. Representation of structure using the FDL.
7.9. Texture Analysis - Gabor Filters
A text segmentation algorithm using Gabor filters for document processing has been proposed by [48]. The main steps of this algorithm are described below:
Step 1. Filter the input image through a bank of n even-symmetric Gabor filters [65], to obtain n filtered images. Step 2. Compute the feature image consisting of the “local energy” estimates over windows of appropriate size around every pixel in each of the filtered images. Step 3. Cluster the feature vectors corresponding to each pixel using a squarederror clustering algorithm to obtain a segmentation of the original input image into K clusters or segments. This algorithm has been used for locating candidate regions of the destination address block (DAB) on images of letters and envelopes in [47]. It treats the document image as a multi-textured region in which the text on the envelope defines a specific texture and other non-text contents including blank regions produce different textures. Thus, the problem of locating the text in the envelope image is posed as a texture segmentation problem. A great variety of methods in texture analysis have been developed for image processing and pattern recognition. Although many of these methods have not been used in document processing directly, a lot of texture classification and segmentation techniques can be useful. Some of those published since 1990 have been selected and are listed below: A new set of textural measures derived from the texture spectrum has been presented in [31]. The proposed features contain more complete texture characteristics of the image. Based on the texture spectrum, a texture edge detection method has been developed in [30]. The basic concept of the method is to use the texture spectrum
3.6 Document Analysis and Recognition by Computers 605 as the texture measure of the image and combine it with the conventional edge detectors. In [81], two new methods have been described which use geometric proximity t o reference points in region growing. The first one is based on Voronoi tessellation and mathematical morphology while the second one is based on the “radiation model” for region growing and image segmentation. In [67], a multi resolution simultaneous autoregressive (MR-SAR) model has been presented for texture classification and segmentation.
7.10. Local Approach In [15,16], a formal model has been proposed to seek specific regions inside a document and perform appropriate local, instead of global, thresholding techniques that lead t o extracting valuable information from the discovered regions with the challenge of preserving its topological properties. The proposed local approach can be summarized into the following key steps: (1) image enhancement, (2) image segmentation, (3) guide lines elimination, (4) morphological processing, (5) restoring lost information, (6) topological processing, (7) edge detection, (8) gaps identification and filling, and (9) information extraction. This approach allows for the visualization and understanding of the finest granularity of data written or printed on documents such as bank cheques.
7.11. Other Segmentation Techniques Segmentation techniques can be roughly categorized as [3,84,2,68,12,25,53]: project ion-based, pitch-based, recognition-based, region-based. The first two techniques are suitable tor typewritten texts where characters are equally spaced and there is a significant gap between adjacent characters. In the “recognition-based” methods, segmentation is performed by recognizing a character in a sequential scan. For handwritten or hand-printed texts where variations in handwriting are unpredictable, the performance of these methods is dubious. The “region-based” method is the only alternative for the segmentation of totally unconstrained handwritten characters. This category of techniques consists of finding and analyzing the input image components as well as how these components are related in order to detect suitable regions for segmentation. Also, segmentation techniques can be categorized as the following types by the sorts of pixels to be worked on: Methods working on foreground pixels (black pixels) [3,84,68,12,25,53], Methods working on background pixels (white pixels) [14].
606
Y. Y. Tang et al.
8. Conclusions
Every day, millions of documents including technical reports, government files, newspapers, books, magazines, letters, bank cheques, etc. have to be processed. A great deal of time, effort and money will be saved if it can be executed automatically. However, in spite of major advances in computer technology, the degree of automation in acquiring data from such documents is very limited and a great deal of manual labour is still needed in this area. Thus, any method which can speed up this process will make a significant contribution. This chapter deals with the essential concepts of document analysis and understanding. It begins with a key concept, document structure. The importance of this concept can be seen in the whole chapter: constructing a geometric structure model and a logical structure model; considering document analysis as a technique of extracting the geometric structure; regarding document understanding as a mapping from the geometric structure into logical structure, etc. This chapter attempts to theoretically analyze document structure and top-down, bottom-up approaches which are commonly used in document analysis in terms of entropy function. Some open questions and problems still exist, especially in document understanding. Any practical document can be viewed differently depending on its geometric structure space and logical structure space. Because there is no one-to-one mapping between these two spaces, it is difficult to find a correct mapping to transform a geometric structure into a logical one. For example, rules based on knowledge may vary in different documents, how to find the correct rules is a profound subject for future research. References [l] L. Abele, F. Wahl and W. Scheri, Procedures for an automatic segmentation of text graphic and halftone regions in document, Proc. 2nd Scandinavian Conf. on Image Analysis (1981) 177-182. [2] P. Ahmed and C. Y . Suen, Computer recognition of totally unconstrained handwritten Zipcodes, Int. J. Pattern Recognition and Artificial Intelligence 1, 1 (1987) 1-15. [3] P. Ahmed and C. Y . Suen, Segmentation of unconstrained handwritten postal zipcodes, Proc. 6th Int. Conf. on Pattern Recognition (1982) 545-547. [4] T. Akiyama and N. Hagita, Automated entry system for printed documents, Pattern Recognition 23, 11 (1990) 1141-1154. [5] R. N. Ascher, G. M. Koppelman, M. J. Miller, G. Nagy and G. L. Shelton Jr, An interactive system for reading unformatted printed text, IEEE Trans. on Computers, C-20, 12 (1971) 1527-1543. [6] N. Bartneck, Knowledge based address block finding using hybrid knowledge representation schemes, Proc. 3rd USPS Advanced Technology Conf. (1988) 249-263. [7] A. Bergman, E. Bracha, P. G. Mulgaonkar and T. Shaham, Advanced research in address block location, Proc. 3rd USPS Advanced Technology Conf. (1988) 218-232. [8] J. P. Bixler, Tracking text in mixed-mode document, Proc. ACM Conf. Document Processing Systems (1988) 177-185.
3.6 Document Analysis and Recognition by Computers 607 [9] H. Bunke, P. S. P. Wang and H. S. Baird (Eds.) Document Image Analysis, Singapore: World Scientific Publishing Co. Pte, Ltd., 1994.
[lo] R. G. Casey, D. R. Ferguson, K. M. Mohiuddin and E. Walach, An intelligent forms processing system, Machine Vision and Applications 5, 3 (1992) 143-155. [ll] R. G. Casey and G. Nagy, Document analysis - a broader view, Proc. First Int. Conf. on Document Analysis and Recognition, Saint-Malo, France, Sept. 3COct. 2 (1991) 839-850. [12] M. Cesar and R. Shinghal, An algorithm for segmentation handwritten postal codes, Man Machine Studies 33 (1990) 63-80. [13] Y. Chenevoy and A. Belaid, Hypothesis management for structured document recog-
nition, Proc. First Int. Conf. on Document Analysis and Recognition, Saint-Malo, France, Sept. 3CkOct. 2 (1991) 121-129. [14] M. Cheriet, Y. S. Huang and C. Y. Suen, Background region-based algorithm for the segmentation of connected digits, Technical Report, Centre for Pattern Recognition and Machine Intelligence, Concordia University (1991). [15] M. Cheriet, J. N. Said and C. Y. Suen, A formal model for document processing of bank cheques, Proc. 3-nd Int. Conf. on Document Analysis and Recognition, Montreal, Canada, Oct. 14-16, (1995) 21C213. [16] M. Cheriet, J. N. Said and C. Y. Suen, A recursive approach for image segmentation, IEEE Rans. Image Processing, (submitted 1995). [17] G. Ciardiello, M. T. Degrandi, M. P. Poccotelli, G. Scafuro and M. R. Spada, An experimental system for office document handling and text recognition Proc. 9th Int. Conf. on Pattern Recognition (1988) 739-743. [18] R. H. Davis and J. Lyall, Recognition of handwritten characters - a review, Image and Vision Computing 4, 4 (1986) 208-218. [19] V. Demjanenko, Y. C. Shin, R. Sridhar, P. Palumbo and S. Srihari, Real-time connected component analysis for address block location, Proc. 4th USPS Advanced Technology Conf. (1990) 1059-1071. [20] A. Dengel, Document image analysis - expectation-driven text recognition, Proc. Syntactic and Structural Pattern Recognition (SSPRSO) (1990) 78-87. [21] A. Dengel and G. Barth, Document description and analysis by cuts, Proc. RIAO, MIT, 1988. (221 W. Doster, Different states of a document’s content on its way from the gutenbergian world to the electronic world, Proc. 7th Int. Conf. on Pattern Recognition (1984) 872-874. [23] A. C. Downton and C. G. Leedham, Preprocessing and presorting of envelope images for automatic sorting using OCR, Pattern Recognition 23, No. 3/4 (1990) 347-362. [24] F. Esposito, D. Malerba, G. Semeraro, E. Annese and G. Scafuro, An experimen-
tal page layout recognition system for office document automatic classification: an integrated approach for inductive generalization, Proc. 10th Int. Conf. on Pattern Recognition (1990) 557-562. [25] R. Fenrich, Segmentation of automatically located handwritten words, Proc. 3rd International Workshop on Frontiers in Handurnding Recognition, Chateau de Bonas, France (1991) 33-44. [26] J. L. Fisher, S. C. Hinds and D. P. D’Amato, A rule-based system for document image segmentation, Proc. 10th Int. Conf. on Pattern Recognition (1990) 567-572. [27] L. A. Fletcher and R. Kasturi, A robust algorithm for text string separation from mixed textlgraphics images, IEEE Rans. on Pattern Analysis and Machine Intelligence 10, 6 (1988) 910-918.
608
Y. Y. Tang et al.
[28] H. Fujisawa and Y. Nakano, A top-down approach for the analysis of document images, Proc. SSPRSO (1990) 113-122. [29] V. K. Govindan and A. P. Shivaprasad, Character recognition - a review, Pattern Recognition 23, 7 (1990) 671-683. [30] D. C. He and L. Wang, Detecting texture edges from image, Pattern Recognition 25, 6 (1992) 595600. [31] D. C. He and L. Wang, Texture features based on texture spectrum, Pattern Recognition 24, 5 (1991) 391-399. [32] J. Higashino, H. Fujisawa, Y. Nakano and M. Ejiri, A knowledge-based segmentation method for document understanding, Pmc. 8th Int. Conf. on Pattern Recognition (1986) 745-748. [33] T. H. Hilderbrandt and W. Liu, Optical recognition of handwritten Chinese characters: advances since 1980, Pattern Recognition 26, 2 (1993) 205-225. [34] S. C. Hinds, J. L. Fisher and D. P. D'Amato, A document skew detection method using run-length encoding and the Hough transform, Proc. 10th Int. Conf. on Pattern Recognition (1990) 464-468. [35] M. Hose and Y. Hoshino, Segmentation method of document images by twodimensional Fourier transformation, System and Computers in Japan 16, 3 (1985) 38-47. [36] N. Hagita I. Masuda, T. Akiyama, T. Takahashi and S. Naito, Approach to smart document reader system, Proc. CVPR'85 (1985) 550-557. [37] ICDAR'91, Proc. First Int. Conf. on Document Analysis and Recognition, SaintMalo, France, Sept. 30-Oct. 2, 1991. [38] ICDAR'93, Proc. Second Int. Conf. on Document Analysis and Recognition, Tsukuba Science City, Japan, Oct. 20-22, 1993. [39] ICDAR'95, Proc. Third Int. Conf. on Document Analysis and Recognition, Montreal, Canada, August 14-16, 1995. [40] S. Impedovo, L. Ottaviano and S. Occhinegro, Optical character recognition - A survey, Znt. J. Pattern Recognition and Artificial Intelligence 5, 1 (1991) 1-24. [41] K. Inagaki, T. Kato, T. hiroshima and T. Sakai, MACSYM: A hierarchical parallel image processing system for event-driven pattern understanding of documents, Pattern Recognition 17, 1 (1984) 85-108. [42] R. Ingold and D. Armangil, A top-down document analysis method for logical structure recognition, Proc. First Int. Conf. on Document Analysis and Recognition, Saint-Malo, France, Sept. 30-Oct. 2 (1991) 41-49. [43] Y. Ishitani, Document Skew Detection Based on Local Region Complexity, Proc. Second Int. Conf. on Document Analysis and Recognition, Tsukuba Science City, Japan, Oct. 20-22 (1993) 49-52. [44] ISO, 8613: Information Processing- Text and Ofice Systems-Ofice, Document Architecture (ODA) and Interchange Format, International Organization for Standardization, 1989. [45] 0. Iwaki, H. Kida and H. Arakawa, A character/graphic segmentation method using neighbourhood line density, Dans. of the Institute of Electronics and Communication Engineers of Japan, Part IV, J&D, 4 (1985) 821-828. [46] 0.Iwaki, H. Kida and H. Arakawa, A Segmentation method based on office document hierarchical structure, Proc. IEEE Int. Conf. Syst. Man. Cybern. Alexandria, VA, Oct. (1987) 759-763. [47] A. K. Jain and S. K. Bhattacharjee, Address block location on envelopes using Gabor filters: supervised method, Proc. 11th Int. Conf. on Pattern Recognition (1992) 264-266.
3.6 Document Analysis and Recognition by Computers 609 [48] A. K. Jain and S. K. Bhattacharjee, Text segmentation using Gabor filters for autcmatic document processing, Machine Vision and Applications 5, 3 (1992) 169-184. [49] E. G. Johnston, Short note: printed text discrimination, Computer Graphics and Image Processing 3, 1 (1974) 83-89. [50] Journal, Machine Vision and Applications, (Special Issue: Document Image Analysis Techniques) 5,3 (1992). (511 J. Kanai, M. S. Krishnamoorthy and T. Spencer, Algorithms for manipulating nested block represented images, Advance Printing of Paper Summaries, SPSE’s 26th Fall Symposium, Arlington, Virginia, Oct (1986) 190-193. [52] S. M. Kerpedjiev, Automatic extraction of information structures from documents, Proc. First Int. Conf. on Document Analysis and Recognition, Saint-Malo, France, Sept. 30-Oct. 2 (1991) 32-40. [53] F. Kimura and M. Shridhar, Recognition of connected numerals, 1st Znt. Conf. on Document Analysis and Recognition, Saint-Malo, France, Sept. 30-Oct. 2 (1991) 731-739. [54] J. Kreich, A. Luhn and G. Maderlechner, An experimental environment for model based document analysis, Proc. First Int. Conf. on Document Analysis and Recognition, Saint-Malo, France, Sept. 30-Oct. 2 (1991) 50-58. [55] J. Kreich, A. Luhn and G. Maderlechner, Knowledge based interpretation of scanned business letters, IAPR Workshop on C V (1988) 417-420. [56] M. Krishnamoorthy, G. Nagy, S. Seth and M. Viswanathan, Syntactic segmentation and labeling of digitized pages from technical journal, IEEE Trans. on Pattern Analysis and Machine Intelligence 15, 7 (1993) 737-747. [57] K. Kubota, 0. Iwaki and H. Arakawa, Document understanding system, Proc. 7th Int. Conf. on Pattern Recognition (1984) 612-614. [58] K. Kubota, 0. Iwaki and H. Arakawa, Image segmentation techniques for document processing, Proc. 1983 Int. Conf. on Text Processing with a Large Character Set (1983) 73-78. [59] S. W. Lam and S. N. Srihari, Multi-domain document layout understanding, Proc. First Int. Conf. on Document Analysis and Recognition, Saint-Malo, France, Sept. 30-Oct. 2 (1991) 112-120. [60] K. K. Lau and C. H. Leung, Layout analysis and segmentation of Chinese newspaper articles, Computer Processing of Chinese and Oriental Languages 8 , 8 (1994) 97-114. [61] S. W. Lee and K. C. Kim, Address block location on handwritten Korean envelope by the merging and splitting method, Pattern Recognition 27, 12 (1994) 1641-1651. [62] J. Liu, Y . Y . Tang and C. Y . Suen, Adaptive rectangle-shaped document segmentation and geometric relation labeling, Proceedings of the 13th International Conference on Pattern Recognition, Vienna, Austria, Aug. 25-30 (1996) 763-767. [63] J. Liu, Y . Y . Tang and C. Y . Suen, Chinese document layout analysis based on adaptive split-and-merge and qualitative spatial reasoning, Pattern Recognition (in press). [64] H. Makino, Representation and segmentation of document images, Proc. IEEE Comput. SOC.Conf. Pattern Recognition and Image Processing (1983) 291-296. [65] J. Malik and P. Perona, Preattentive texture discrimination with early vision mechanisms, Journal Opt. SOC.Amer. A . 7, 5 (1990) 923-932. [66] J. Mantas, An overview of character recognition methodologies, Pattern Recognition 19,6 (1986) 425-430. [67] J. C. Mao and A. K. Jain, Texture classification and segmentation using multiresolution simultaneous autoregressive models, Pattern Recognition 25, 2 (1992) 173-188.
610
Y. Y. Tang et al.
[68] B. T. Mitchell and A. M. Gillies, A Model-based computer vision system for recognizing handwritten ZIP codes, Machine Vision and Applications 2 (1989) 231-243. [69] S. Mori, C. Y. Suen and K. Yamamoto, Historical review of OCR research and development, Proceeding of the IEEE 80, 7 (1992) 1029-1058. [70] S. Mori, K. Yamamoto and M. Yasuda, Research on machine recognition of handprinted characters, IEEE h n s . on Pattern Analysis and Machine Intelligence 6 , 4 (1984) 38G405. [71] G. Nagy, A preliminary investigation of techniques for the automated reading of unformatted text, Comm, ACM 11, 7 (1968) 480-487. [72] G . Nagy, Towards a structured-document-image utility, Proc. SSPRSO (1990) 293-309. [73] G. Nagy, J. Kanai and M. Krishnamoorthy, Two complementary techniques for digitized document analysis, Proc. ACM Conf. on Document Processing Systems (1988) 169-176. [74] G. Nagy, S. C. Seth and S. D. Stoddard, Document analysis with an expert system, In E. S. Gelsema and L. N. Kanal eds., Pattern Recognition Practice II, pp. 149-159, Elsevier Science Publishers B. V. (North-Holland), 1986. [75] Y. Nakano, H. Fujisawa, 0. Kunisaki, K. Okada and T. Hananoi, A document understanding system incorporating with character recognition, Proc. 8th Int. Conf. on Pattern Recognition (1986) 801-803. [76] Y. Nakano, Y. Shima, H. Fujisawa, J. Higashino and M. Fujiwara, An algorithm for the skew normalization of document image, Proc. 10th Int. Conf. on Pattern Recognition 2 (1990) 8-13. (771 D. Niyogi and S. N. Srihari, A rule-based system for document understanding, Proc. AAAI’86 (1986) 789-793. [78] L. O’Gorman, The document spectrum for structural page layout analysis, IEEE f i n s . on Pattern Analysis and Machine Intelligence 15, 11 (1993) 1162-1173. [79] L. O’Gorman and R. Kasturi (eds.) Document Image Analysis, New York: IEEE Computer Society Press, 1995. [80] T. Pavlidis, Algorithm for Graphics and Image Processing, Maryland: Computer Science Press, 1982. [81] I. Pitas and C. Kotropoulos, A texture-based approach to the segmentation of sesmic image, Pattern Recognition 25, 9 (1992) 929-945. [82] A. Rastogi and S. N. Srihari, Recognizing textual blocks in document images using the Hough transform, T R 86-01,Dept. of Computer Science, SUNY Buffalo, NY, 1986. [83] M. Sabourin, Optical character recognition by a neural network, Neural Networks 5 , 5 (1992) 843-852. [84] M. Shridhar and A. Badreldin, Recognition of isolated and simply connected handwritten numerals, Pattern Recognition 19, 1 (1986) 1-12. [85] J. C. Simon and K. Zerhouni, Robust description of a line image, Proc. First Int. Conf. on Document Analysis and Recognition, Saint-Malo, France, Sept. 30-Oct. 2 (1991) 3-14. [86] A. L. Spitz and A. Dengel (e’ds.) Document Analysis Systems, Singapore: World Scientific Publishing Co. Pte, Ltd., 1995. [87] S. N. Srihari and V. Govindaraju, Analysis of textual images using the Hough transform, Machine Vision and Application 2 (1989) 141-153. [88] S. N. Srihari, C. H. Wang, P. W. Palumbo and J. J. Hull, Recognizing address blocks on mail pieces: specialized tools and problem-solving architecture, AI Mag. 8 , 4 (1987) 25-40.
3.6 Document Analysis and Recognition by Computers 611 [89] C. Y. Suen, M. Berthod and S. Mori, Automatic recognition of handprinted characters - The state of the art, Proc. IEEE 68, 4 (1980) 469-487. [go] C. Y. Suen, Y. Y. Tang and C. D. Yan, Document layout and logical model: A general Analysis for document processing, Technical Report, Centre for Pat tern Recognition and Machine Intelligence (CENPARMI), Concordia University, 1989. [91] Y. Y. Tang, Hong Ma, Dihua Xi, Yi Cheng and C. Y. Suen, A new Approach to document analysis based on modified fractal signature, Proc. 3-nd Int. Conf. on Document Analysis and Recognition, Montreal, Canada, Oct. 14-16 (1995) 567-570. (921 Y. Y. Tang, C. Y. Suen and C. D. Yan, Chinese form pre-processing for automatic data entry, Proc. Int. Conf. on Computer Processing of Chinese and Oriental Languages, August 13-16 (1991) Taipei, Taiwan, 313-318. [93] Y. Y. Tang, C. Y. Suen and C. D. Yan, Document processing for automatic knowledge acquisition, ZEEE Trans. on Knowledge and Data Engineering 6, 1 (1994) 3-2 1. [94] Y. Y. Tang, C. D. Yan, M. Cheriet and C. Y. Suen, Automatic analysis and understanding of documents, Handbook of Pattern Recognition and Computer Vision, pp. 625-654, edited by Patrick S. P. Wang, C. H. Chen and L. F. Pau, Singapore: World Scientific Publishing Co. Pte, Ltd., 1993. [95] Y. Y. Tang, C. D. Yan, M. Cheriet and C. Y. Suen, Document analysis and understanding: a brief survey, Proc. First Int. Conf. on Document Analysis and Recognition, Saint-Malo, fiance, Sept. 30-Oct. 2 (1991) 17-31. [96] Y. Y. Tang, C. D. Yan and C. Y. Suen, Form description language and its mapping onto form structure, Technical Report, Centre for Pattern Recognition and Machine Intelligence (CENPARMI), Concordia University, 1990. [97] S. L. Taylor, R. fiitzson and J. A. Pastor, Extraction of data from preprinted forms, Machine Vision and Applications 5, 3 (1992) 211-222. [98] J. Toyoda, Y. Noguchi and Y. Nishimura, Study of extracting Japanese newspaper article, Proc. 6th Int. Conf. on Pattern Recognition (1982) 1113-1115. [99] Y. Tsuji, Document image analysis for generating syntactic structure description, Proc. 9th Int. Conf. on Pattern Recognition (1988) 744-747. [loo] S. Tsujimoto and H. Asada, Understanding multi-articled documents, Proc. 10th Int. Conf. on Pattern Recognition (1990) 551-556. [loll M. Viswanathan, Analysis of scanned documents - a syntactic approach, Proc. SSPRSO (1990) 450-459. [lo21 F. Wahl, L. Abele and W. Scheri, Merkmale fuer die segmentation von dokumenten zur automatischen textverarbeitung, Proc. 4th DAGM-Symposium, 1981. (1031 D. Wang and S. N. Srihari, Classification of newspaper image blocks using texture analysis, CVGIP 47 (1989) 327-352. [lo41 S. Watanabe, Pattern Recognition: Human and Mechanical, Wiley-Interscience Publication, 1985. [lo51 K. Y. Wong, R. G . Casey and F. M. Wahl, Document analysis system, IBM J. Research Develop, 26, 6 (1982) t47-656. [lo61 A. Yamashita, T. Amano, H. Takahashi and K. Toyokawa, A model based laout understanding method for document recognition system (DRS), Proc. First Int. Conf. on Document Analysis and Recognition, Saint-Malo, France, Sept. 30-Oct. 2 (1991) 130-138. [lo71 C. D. Yan, Y. Y. Tang and C. Y. Suen, Form understanding system based on form description language, Proc. First Int. Conf. on Document Analysis and Recognition, Saint-Malo, France, Sept. 30-Oct. 2 (1991) 283-293.
612
Y. Y. Tang et al.
[lo81 P. S. Yeh, S. Antoy, A. Litcher and A. Rosenfeld, Address location on envelopes, Pattern Recognition 20, 2 (1987) 213-227. [log] C. L. Yu, Y. Y. Tang and C. Y. Suen, Document skew detection based on the fractal and least squares method, Proc. 3-nd Int. Conf. on Document Analysis and Recognition, Montreal, Canada,Oct. 14-16 (1995) 1149-1152.
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 613-624 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
I CHAPTER 3.7 I PATTERN RECOGNITION AND VISUALIZATION OF SPARSELY SAMPLED BIOMEDICAL SIGNALS
CHING-CHUNG LI, T. P. WANG Department of Electrical Engineering, University of Pittsburgh Pittsburgh, PA 15261, USA and A. H. VAGNUCCI, M.D. Department of Medicine, University of Pittsburgh Pittsburgh, PA 15261, USA A variety of biomedical signals such as hormonal concentrations in peripheral blood can only be sampled and measured infrequently over a limited period of time; hence, they are considered as sparsely sampled non-stationary short time series. Discrete pseudo Wigner distribution is a transform which can be applied t o such signals to provide timedependent spectral information at an improved frequency resolution in comparison t o the short-time Fourier transform. When appropriately clipped and scaled, it can be visualized as an image showing the characteristic pattern of the signal. Spectral features can be extracted from the Wigner distribution for use in automatic pattern recognition. The basic technique is described in this article along with an example of its application to cortisol time series. Keywords: Biomedical signal; cortisol; pattern recognition; pattern visualization; short time series; Wigner distribution.
1. Introduction
Various biological signals are often measured in a clinical setting for providing information to aid medical diagnosis. Some signals, such as EEG and ECG, etc., can be continuously measured with relative ease; it is well known that their spectra and other analyses have been successfully applied in characterizing the state of health [1,2]. Other types of biological signals, such as chemical signals in blood samples, can be measured only infrequently for a very limited period of time; such sparsely sampled data constitute short time series which are of non-stationary nature [3]. The short-time Fourier transform gives some crude spectral information at a coarse resolution. However, the pseudo Wigner distribution can be applied to provide a better estimate of the time-dependent spectral information. This time-frequency domain analysis and its use in biomedical pattern recognition will be discussed in the following sections. 613
614 C.-C. La, T. P. Wang €4 A. H. Vagnucca
The Wigner distribution was introduced by E. P. Wigner [4] in 1932 in the context of quantum mechanics, and then applied to signal theory by J. Ville [5] in 1948. During the past ten years, the methods and applications of the Wigner distribution to non-stationary signals have been developed rapidly [6-131. An exposition of the important mathematical background of the Wigner distribution can be found in a series of three papers by Claasen and Mecklenbrauker 161,and a recent review is contained in a paper by Hlawatsch and Boudreaux-Bartels [14]. We will summarize some of the most useful properties of the Wigner distribution and the techniques of applying the discrete pseudo Wigner distribution to sparsely sampled biomedical signals. The plasma cortisol time series is taken as an example to illustrate its application to recognition and visualization of normal and abnormal patterns. 2. Wigner Distribution
2.1. Continuous Wigner Distribution Let f(t) be a continuous function of time variable t ; f ( t ) may be either real or complex, and f*(t) is the complex conjugate of f ( t ) . The Wigner distribution of f ( t ) is defined by 00
W f ( t , w ) = / -00 f ( t + ; ) f * ( t - i ) e - j w T d T where T is the correlation variable, +$ and -$ denote the time advance and time delay respectively. f(t + $)f*(t - $) forms a kernel function of the time variable t and the correlation variable T . The Fourier transform of this kernel function with respect to T gives the Wigner distribution W f(t,w ) which is a real-valued continuous function of both time t and frequency w. If F ( w ) is the Fourier transform of f(t) and F * ( w ) is its complex conjugate, the Wigner distribution W F ( W t ), can be defined as
w ~ ( wt ), =
2.rr
F (w -m
+
g)
F* ( w -
g)
eitcd
50 cm) special care has to be taken to get good image quality over the whole field of view. The shape of the object, to guide the selection of features to be identified. This can be a difficult task because food products are often non-rigid and easily damaged.
690
H. Arnarson & M. Asrnundsson
2.1.2. Sensors
There are several important characteristics one has to take into consideration when selecting cameras for vision systems. In this section the most important ones, when selecting between Charge Transfer Devices (CTD) [13] cameras and tube cameras, will be described. (i) Shape of the sensor. When using CTD cameras it is possible to select between different shapes of sensors. The three most common types are: array, line and disk shaped sensors. Tube cameras are limited to array shaped sensors. The shape of the sensor is important, especially if the objects are moving. When the object is stationary or slow moving an array shaped sensor can be used. For rapidly moving objects improved performance is obtained by using linear or disk shaped sensors [14] coupled to a motion synchroniser. (ii) Sensor resolution. In array sensors the resolution is normally expressed as the number of lines per pictures height. In CTD cameras the limit is set by the pixels available in the image area. In tube cameras resolution is influenced by the type and size of the photoconductive layer, the image size on the layer, beam and signal levels, and the spot size of the scanning beam. (iii) Spectral sensitivity. Spectral sensitivity is the sensor’s relative response at different wavelengths of light. Usually CTD cameras cover wavelengths in the range 0.3-1.2 pm, while tube cameras cover 0.2-14 pm (not by one tube). (iv) Sensitivity. Sensitivity is the efficiency of the light to the charge conversion of the sensor. There are several ways of measuring this sensitivity. The two most frequently used are luminous sensitivity and radiant sensitivity. 0
0
Luminous sensitivity is measured at a specific colour of light, usually 2856 K. The output is expressed in mA/(lumen mm2) or V/(mW mm’). Radiant sensitivity is measured over a range of wavelengths, usually from 400 to 1000 nm. The output is expressed in mA/(Wmm2).
In CTD cameras, sensitivity is influenced by variables such as quantum efficiency, the length of integration time and the dominant source of noise in the device. Tube camera sensitivity is dependent on the type and size of the photoconductive layer. It also varies with the target voltage level in certain types of tube cameras. (v) Dynamic range. This represents the overall usable range of the sensor. It is usually measured as the ratio between the output signal at saturation and the RMS value of the noise of the sensor (sometimes peak to peak noise). In CTD cameras this RMS noise does not take into account dark signal nonuniformities [13]. In CTD cameras the saturation voltage is proportional to the pixel area. Factors influencing the dynamic range of tube cameras include photoconductive characteristics of the faceplate, as well as scanning rate and electronic gun characteristics.
4.1 Computer Vision in Food Handling and Sorting 691 Table 1. Performance of CTD and tube cameras
Units Resolution Dynamic range Max. sensitivity Geometric distortion Nonuniformity Lag Spectral sensitivity Mean time between failure Frame rate Damage by overlighting Price Supply voltage
CTD camera Typical value Max. value
lines peak sign /ms noise lux
500 4000 : 1
% % %
750 100 : 1
1000 : 1
10-6
*
20
0 2
0.1
1
2
12
10
20
0 300-1200
0.1
10 200-14000
hours
unlimited
* *
15-20
nm
10000
*
25
400
frames/s
*
1000 10000 : 1
Tube camera Typical value Max. value 2000
*
*
*
25
2000
No
Yes
*
USD V
2500 15
* *
1500 500
*
*
Among other important characteristics of cameras are: signal to noise ratio, geometric distortion, lag, nonuniformities, readout speed, camera synchronisation, mean time between failure, operating temperature, damage by overlighting, operating power, operating voltage, size, weight, price. Table 1 lists typical performances of CTD and tube (Vidicon) cameras currently available on the market. 2.1.3. Lighting and viewing techniques Selection of illumination equipment and viewing geometry is a n important step in the development of the acquisition part of a vision system [13,14]. Based on the application (inspection, handling, sorting) to be implemented and the characteristics of the object, both the physical and optical characteristics, the optimum lighting and viewing technique is defined. In one- and two-dimensional size and shape measurements, diffused or direct backlighting are most likely to give good image quality. Although special care has to be taken with some food products, e.g. fish where the fins can be partly transparent. In three-dimensional size and shape measurements, structured light [15] is often used, but the use of two or more sensors can also give robust and accurate three-dimensional measurements. When doing surface inspection the most appropriate set-up is diffused front lighting, where the contrast is often enhanced using coloured light and colour filters (Section 2.1) in front of the camera. Using front lighting techniques it can be difficult to cope with variations in the color of the object.
692
H. Arnarson & M . Asrnundsson
Inspection inside the food object, includes search for parasites and bones. In these applications it is generally very difficult to develop the lighting and viewing part of the vision system, because of the optical characteristics of the food [IS]. In the fish industry different lighting techniques have been tested. This includes Xrays €or bone detection [17], laser scanning for bones and parasites [IS], ultrasound for bones and parasites [19],and fluorescence of fish bones [20]. Today it is possible to detect bones inside meat and fish flesh using soft X-rays, while the problem of parasites in fish still remains unsolved [18]. 2.1.4. Optics The optical front end of a vision system must be designed with equal care to that applied to electronics, otherwise there is a risk that an apparently precise measurement will hide a significant error caused by the limitation in optics. Special care has also to be taken, because applications in the food industry involve sensing of large images outside the optical axis of the lens system. In this section important characteristics of lenses for food handling and sorting will be discussed (see Fig. 1).
FOV
V
U
Fig. 1. Image forming basics.
(i) Magnification m. The optical magnification is defined as the image distance IJ over object distance u or Field-Of-View (FOV) over the sensor size d.
m = FOV/d
= v/u
.
(2.1)
The FOV should be large enough to see the object, and because of object movement which is very often the case in the food industry an FOV 30% larger than the largest object is recommended. (ii) Focal length f . The optimum focal length for each application is related to u and IJ through the well known lens equation: l/f = 1 / u
+l/v.
(2.2)
4.1 Computer Vision an Food Handling and Sorting
693
Equation (2.2) assumes the light is perpendicular to the optical plane of the lens. In practice, for many applications in handling and sorting in the food industry, u >> v so a good approximation of Eq. (2.2) is :
l/f
FZ
l/v.
(2.3)
(iii) Lens quality. There are two main factors that influence lens quality. Special care has to be taken when working on applications involving large objects. 0
Resolution T . Because of the diffraction effect of the light going through the lens, there is a theoretical limitation on the resolution of the lens. For lenses working at high demagnification, this theoretical value is given by [13]: T
=
1.22 * X * f / A
(2.4)
where
X : wavelength of light f : focal length of lens
A : diameter of the aperture of the lens. 0
Aberration. There are two kind of aberrations: monochromatic and chromatic [21]. Monochromatic aberrations are divided into five subclasses and can be calculated theoretically using a lens formula assuming oblique line directions [22]. The chromatic aberrations are caused by the changing diffraction index of the lens material, with wavelength of light. All the aberrations get worse as the lens aperture is increased, and all of them except one subclass of monochromatic aberrations get worse with increased field angle. Monochromatic aberrations need special attention when doing accurate measurements outside the optical axis of the lens system. Typically, no information on resolution or aberrations are provided from the lens producer, instead a measure of the Modulation Transfer Function (MTF) is given. The MTF is the ability of a lens to image a particular sine wave pattern. The MTF is determined by measuring, through a lens, the contrast of such a sine wave pattern image, while changing the aperture and the off axis position of the object.
(iv) Depth of View (DOV) is defined as the distance along the optical axis on which the object can be located and still be properly imaged :
DOV = C U / ( A - C)
+ C U / ( A+ C)
where
A : lens aperture diameter u : object distance c : blur circle diameter at object.
(2.5)
694
H. Arnarson & M. Asrnundsson The blur circle is the amount of blur which can be tolerated, often set at one pixel. From Eq. (2.5), it can be seen that there is a maximum A for a given DOV.
2.1.5. Filtering Filtering is used to improve image quality, reduce noise, and enhance features of interest. Three types of filtering are described in this section: neutral density filtering, polarisation filtering, and colour filtering. (i) Neutral density filtering. Neutral density filters [23] are used to attenuate the intensity of a beam of light over a broad spectral region, without altering its spectral distribution. A neutral density filter can thereby for example be used to decrease the light intensity incident on a photodetector. Because of optical resolutions it is important to allow a large enough aperture of the lens (Section 2.1). Using a neutral density filter allows a larger aperture of the lens. A neutral density filter is characterised by its optical density D:
where
I0 : incident power IT : transmitted power T : transmittance. (ii) Polarisation filtering. Light travels as a transverse electromagnetic wave, the electric and magnetic fields being perpendicular to each other as well as to the direction of propagation. A light beam is said to be linearly polarised if its electric field vectors are oriented in the same direction. A substance can affect the polarisation of light, reflected or transmitted, giving a significant feature for that same substance. The polarisation state of the resulting light beam can be detected with the aid of polarising filters, and by comparing it to the polarisation of the incident light information regarding the substance can be obtained. A polarisation filter can also be used to reduce glinting in an image. Dichroic (iii) film polarisers, fabricated from sheets made of long grain organic molecules are probably the most convenient type of polarisers for image processing purposes. They are inexpensive and have a convenient shape. (iv) Colour filtering. Colour filtering [23] may be of the most obvious importance in image processing for the food industry. With colour filtering it is possible to extract information from well defined bands of the spectrum or to increase the amount of information in the image by examining more than one different band (wavelength regions). Colour images, as we know them (for example, TV
4.1 Computer Vision in Food Handling and Sorting 695 images), are often based on the combination of three images in separate bands, which all together cover the visible spectrum. These bands are referred to as Red, Green and Blue (RGB) [23]. Such images are most often acquired using three well defined colour filters. A three-band colour image can sometimes have excess information and, in that way, will slow down the processing. It is therefore essential to analyze the optical characteristics of the subject with the purpose of choosing a spectral band of interest. Accordingly the right colour filtering can be applied. Often a much narrower band than R, G or B is more effective, where the wave bands are selected using spectroradiographic study of the product to be investigated. Different types of colour filters are available. Coloured glass filters operate through ionic absorption or via absorptive and scattering crystallites formed within the glass. They are available as Long Wave Pass filters (LWP) with a relatively sharp cut and a variety of bandpass filters which are not so sharp. Gelatin filters have similar characteristics to glass filters. They operate through absorption as well. They are commonly used in photography and are inexpensive. Gelatin is a “plastic-like” material and therefore gelatin filters are vulnerable to scratches. Interference filters operate through interference to select a range of wavelengths. Wavelengths not falling within this range are reflected. They are fabricated as thin coatings of various dielectric materials on a glass plate. Interference filters are available as LWP, SWP (short wave pass) and BP (bandpass) filters with very sharp cut characteristics. By tilting an interference filter its characteristics are changed, the cut wavelength(s) being displaced. This effect can also occur when observing an object which is not lying on the system’s optical axis, through such a filter. This should be noted or taken into account when used with cameras.
2.2. Environment
The environment in the food industry is generally harsh. The main characteristics are: 0
0
0
0
The humidity is often high (95%-100%). This is caused by continuous washing of the machinery for sanitary reasons. It is frequently recommended to keep the temperature in the processing plants between 5-10°C. There are strict limitations on what types of waste are allowed from machines in the food industry. There are regulations on what types of materials are allowed in the food processing plants. For example, it is often forbidden to use conventional glass for direct contact with the food.
696
H . Arnarson €4 M. Asrnundsson
2.3. Image Processing for Food Products All algorithms used in sorting and handling food products, use a priori knowledge, although at different levels. This a priori knowledge is used to build up a model of the process and provide strategies for the algorithms to be designed. Some of this knowledge is imposed on the process by selecting colour and texture of the background and by the viewing angle of the light source and the camera. Another part is controlled by the feeding system which determines the direction of motion of the object and whether or not the objects are overlapping. Also the object to be sorted gives information on what kind of algorithm should be used, for example, for fish, the fish has a head and a tail and some fins that can be used for classification. It is desirable to use as much a priori information on the object and the process as possible. In that way the algorithms can be simplified, the hardware requirements reduced, and the possibility of satisfying the needs of the industry at a cost and speed it accepts are increased. Algorithms used in the food industry are made of the same basic elements as in most other industries, i.e. 0 0 0
preprocessing feature extraction classification.
However the emphasis on these basic elements can be quite different compared to other types of applications. 2.3.1. Pre-processing
Although in real-time industrial applications special care is taken in designing the optimum image acquisition part, there is often a need to improve the image quality before extracting the feature of interest from the image. In applications where the results are presented to the user in an image format, image enhancement techniques such as histogram equalisation, and look-up table operations are used to improve the contrast of the feature of interest in the image. On the other hand, in automatic control systems the computer controls some actions. Based on results from the image processing this type of enhancement technique is of no use and can in fact degrade the quality of the image because of quantization effects. The nature of noise in applications in the food industry is different from what most textbooks discuss, where the focus is on random noise or spot noise. In applications in the food industry the noise usually has some physical explanation, e.g. dirt on the background or shadows because the camera has a different viewing angle than the light source. [9]describes methods based on mathematical morphology for filtering out noise, where the noise has some predefined form and some maximum size. In Section 3.2 on colour inspection, results are shown on how noise can be filtered out using this method when detecting surface defects on fish fillets.
4.1
Computer Vision in Food Handling and Sorting
697
The primary goal in pre-processing images is to reduce the amount of data in the image, for example by binarizing [24] the image. Global thresholding is used when possible. Otsu [25] describes a method for calculating the optimum threshold between classes. His method is theoretically well based but requires too much computation to be used on-line in a real-time application especially when using more than two classes. It can however be very useful in a training phase of an automatic system. When the Field Of View (FOV) is large it is difficult to get even lighting in the whole FOV. In these cases it is necessary to use local thresholding, especially when the performance of the application is dependent on accurate t hresholding. 2.3.2. Feature extraction
When selecting features to be measured it is important to select features that can be measured with good accuracy and good repeatability. Generally there are two kinds of errors that affect the feature extraction: Measurement error, because of limited accuracy of the sensing equipment, or the sensing process. This includes limited resolution of the sensor and optics, blur caused by the movement of the object, and quantization error in the A/D conversion process. The measurement error is controlled by selecting the appropriate sensing equipment. Presentation error, because of variations in the way the object is presented to the vision system. This error needs special attention when dealing with food products of non-uniform shape, varying size, flexibility and imperfect operation of feeding systems dealing with these kind of products. It is important, based on knowledge of the products to be processed and also very importantly on the nature of the feeding system, to select features to be extracted. Generally features such as location, dimension, and orientation, are common to most handling and sorting problems in the food industry. These features are used to localise the parts to be classified. Further feature extraction includes identification of corners, lines, holes, and curves. It is very useful to use information on object location to reduce the amount of data to be processed, on dimensions of the object to get size invariance and on orientation of the object to reduce dependency on orientation. In this way it is possible to focus the attention of the vision system to Areas of Interest (AOI) and in that way speed up processing. The algorithms used to extract the features selected have to be able to work on objects of random orientation and in real time. A number of algorithms are used for feature extraction. A good overview of these are in 1261. One example of algorithms used is based on identifying the contour of the object. Edge detectors are used to enhance the edge pixels which are then connected in a chain code and further connected into shape primitives which are used to describe the object. These algorithms are very time consuming and of
698 H. Arnarson €9 M . Asrnundsson
limited use in real-time applications in the food industry. Another type of feature extraction algorithm is a space domain technique like skeleton algorithms, where the object is characterised by its skeleton. These algorithms provide useful feature descriptions of the object using a limited amount of data. Of special interest is the distanced labelled skeleton [27], which can give a complete description of binary images. Another example of feature extraction algorithms is scalar transform techniques, such as moment invariants [28] and Fourier transform techniques. Mathematical morphology [29] is a technique well suited for real-time applications in the food industry. This is due to the parallel nature of its basic operations. It is a set-theoretical approach to image analysis and its purpose is the quantitative description of geometrical structures. Mathematical morphology extracts information about the geometrical structures of image objects by transforming the object through its interaction with another object (structuring element) which is of simpler shape and size than the original image object. Information about size, spatial distribution, shape, connectivity, smoothness, and orientation can be obtained by transforming the image object using different structuring elements. In [30] the Hit or Miss transform [29] from mathematical morphology is used to extract information on the presence and position of shape primitives (line, corner, convex, concave) in a fast and reliable way. 2.3.3. Classification Based on the features extracted from the image, the object is classified into one of the possible classes. There exist numerous methods for classification based on features extracted from an image. Statistical pattern classification [31] is a very sound theoretical method for classification of patterns. It is well suited in applications where a limited number of features is used. However when using many features it can be difficult and time consuming to design the classifier. Graph matching [32] is a method where the presence and position of features in relation to other features is used to classify the object. The features used could for example be corners and lines used to recognise fish species. Neural networks [30] is a suitable method, when a large number of features are available, but it is difficult to identify which are the most important features for classification. Of special interest is the possibility of training the classifier, in such a way that the classification rules are determined automatically. 2.4. Hardware
The performance of conventional sequential computer architectures is inadequate for the majority of machine vision applications in the food industry. The problem arises from the sheer amount of data presented in the image. Simple real time neighbourhood operations require 20 million operations per second (MOPS). The typical computational power of a sequential computer is less (e.g. 1-5 MOPS),
4.1 Computer Vision in Food Handling and Sorting 699 and therefore they are too slow. Confronted with such computational problems numerous researchers and manufacturers have sought to develop new computer architectures to provide the necessary computational power required by real time machine vision applications [33,34]. One approach is to develop more powerful processors to handle the workload, typified by the newest Digital Signal Processor (DSP) chips. Another approach is to create architectures which allow many processors to work on the image data in parallel at the same time. It is important to note that algorithms have quite different possibilities of being implemented in parallel, and right from the beginning it is important to focus on the software side on algorithms that are parallel in nature. 2.4.1. Image processing hardware
There is no single architecture that is the optimum one for all vision algorithms or industrial applications [35]. Therefore it is important that there is a flexibility in the selection of arithmetic units that can be installed in an industrial vision system. Figure 2 shows an example of a modular hardware structure, available from several companies (ITI, Vicom, Datacube, Eltec) today at a price of less than 25K USD.
Host bus
Vi&obm
0Pipehebus
Fig. 2. Example of a modular hardware structure in an industrial vision system.
The basic blocks of this kind of industrial vision system are: Host computer which controls the system. This is typically an Intel 80x86 or a Motorola (MC 680x0) based computer. Analogue/Digital interface, to provide an interface to cameras. This is typically an 8-bit flash A/D converter with a 10 MHz sampling rate. Frame buffer, to store images. Often it is possible to install up to four frame buffers, where each frame buffer stores a 512 x 512 x 8 image. It is important that there is more than a single port access to the frame buffer.
700
H . Arnarson &
M. Asrnundsson
(4) Special purpose arithmetic units. It is very important that it is possible to select between different types of architectures dependent on the algorithms to be performed at each time. An example of the arithmetic units available are: (i) Pipeline processors, well suited for simple arithmetic, logical, conditional and bit-plane operations, that can be performed in real time. (ii) Rank value filters which perform real time median filtering and grey scale morphological operations. (iii) Binary correlators which perform real binary operations including convolution, correlation, erosion and dilation. (iv) Signal processors, general purpose vision algorithms. (v) FUSC processors, for general purpose vision algorithms. Which of the modules described above are used in each application is highly dependent on both the application and the way the object is presented to the vision system.
2.4.2. Complexity of the vision system The levels of complexity for a vision system for food products is highly determined by the feeding part of the system. These levels are mainly determined by: (i) Distance between the objects. Is there a minimum distance between the objects? Can they be side be side, or overlapping? (ii) Orientation of the object. Is the object oriented, or not?
If objects are fed to the system on a conveyor and there is no minimum distance between the objects, then there is a need for at least two processes, one which constantly searches for the object while the other which measures the features of interest of the object. The orientation on the other hand directly influences the complexity of the algorithm. Table 2 lists the different levels of complexity in a vision based sorting system. In the food industry the practical levels are mainly levels 2-4. Level 1 is excluded because of the orientation demand which is difficult to obtain for an elastic object like food. Level 5 is excluded because it normally results in high sorting errors, and Table 2. Levels of complexity in the vision part of a sorting system. Level 1 2 3 4
5
Feeding System Objects oriented, and there is a minimum distance between objects. Objects not oriented, and there is a minimum distance between objects, Objects oriented, no minimum distance between, not overlapping. Objects random, not overlapping. Objects random, and overlapping.
4.1 Computer Vision an Food Handling and Sorting 701 although the food products can be measured accurately, it is difficult to direct it mechanically to different places. The compromise, which has to be made when selecting a working level for a vision based sorting system, lies between the requirements (cost) of the feeding system and the complexity and the speed of the algorithm. The level of complexity selected when specifying a vision application, does not only determine the cost of the system, but also the possible accuracy of the features to be measured.
3. Applications 3.1. Size Sorting of Fish 3.1 .l. Motivation
Sorting dead or dying fish is required on board fishing boats for packaging and storage purposes. Typically, the catch must be sorted by species, length or weight before going into boxes, the contents of which are compatible with the auctioning process. At the processing level, the machines (filleting and head cutters) still cannot handle in one batch diverse fish types or sizes without reduction in yield. Any set-up due to such variations is both time consuming and costly. By sorting the fish by size the fishing industry is able to get both increased yield and added production control [36]. 3.1.2. Image acquisition
There are different size parameters that can be measured on a fish using computer vision, these include volume, area, length, thickness and width [37,38]. Much depends on the feeding system as to how difficult it is to measure these features. Here we will assume that the feeding system is working on level 4 (Section 2.4), that is the fish is lying randomly but not overlapping when fed to the vision system. Further we assume that the fish is round (e.g. cod, haddock). When measuring the volume and the area, a high (& 15%) presentation error (Section 2.3) is observed due to the irregular position of fins and belly flaps [9], whereas the length can be defined and measured with a low presentation error. Therefore the length is selected as a size feature when sorting round fish by size, where the length of the fish is defined as the length of a line starting at the middle of the tail to the top of the head, following the bone structure of the fish. A fish is a highly reflecting object when under direct illumination. Fish colour varies for many species. Usually the fish is dark on the back, and white or greyish on the belly. Because of this it is very difficult to get a good contrast in an image using front lighting techniques. Diffused backlighting is used, but special care has to be taken because of transparency of fins, especially the tail fin. This application has to be able to work on board rolling ships. This fact excludes the use of line-scan and circular-scan sensors, because of their dependency on object
702
H. Arnarson €d M. Asrnundsson
motion. A frame camera which is able to sense the whole fish is therefore selected. A CTD camera is selected here mainly because of its robustness.
3.1.3. Image processing
A block diagram of the algorithm is shown in Fig. 3. In start-up the system goes through a training phase where the threshold for the background and for the fish are automatically determined using the method in [25]. In the training phase the optical magnification of the system is determined, by measuring n objects of a known size. This training phase can also be entered into interactively by the user. The algorithm starts by searching for a fish, in binary images that are snapped continuously. As soon as a whole fish is detected inside the field of view, its position is registered and a rectangular area of interest encircling the fish is determined. Based on a priori knowledge on the shape of the fish, that the fish is an elongated object and that the fish is thicker close to the head than close to the tail, the position
Initialize system
I
Search for fish Search-I
Estimate an aoi encircling the fish
Reduce blur
Register length in database
Fig. 3. Block diagram of an algorithm for length sorting of fish.
4.1 Computer Vision in Food Handling and Sorting 703 of the head, tail, fins and belly flaps are determined. Then the position of the length estimation line can be determined accurately. The length is then measured using piecewise linear approximation and knowledge on the optical magnification of the system. Based on sorting criteria programmable by the user the fish is then classified as belonging to different groups based on the length measured. The measured length is also registered in a database. 3.1.4. Results
The length estimation system is now a commercial system [5]. See Fig. 4.
Fig. 4. Prototype for length estimation of whole fish.
Fish are fed to the vision system on a conveyor running at 1.2 m/s. The system is able to length estimate whole fish with an accuracy of f 0.3 cm (one standard deviation), independent of fish orientation (Fig. 5). The processing time for each fish is 0.2-0.3 seconds, depending on fish size. 3.2. Colour Inspection 3.2.1. Motivation
Fish flesh (for example, that of cod fish) is graded in quality groups according to: colour, coloured spots (blood spots and liver spots), gaping, and shape. The most important factor in quality control is the colour of the fish flesh. Briefly, one can say that the lighter the flesh the better the quality. Today quality control is
704 H. Arnarson & M . Asmvndsson
Fig. 5. Length estimation of cod fish.
done manually in all processing plants and under different circumstances in each place. Therefore the manual quality control is bound to be very dependent upon the individual performing it and the circumstances it is performed under. The benefits of automation in quality control in the fish industry are evident. Coordination of control and standardisation will benefit both sellers and buyers.
3.2.2. Image acquisition
A first proposal for sensing equipment for a “colour grader” would presumably be a colour camera or a colorimeter. A wide variety of colorimeters are available on the market [39], some of them have been successfully applied in the food industry, for example, in inspection of fruit. Most colorimeters feature three sensors as the sensing equipment: these sensors are light sensitive in the visible (VIS) range but filtered with red, green, and blue filters respectively. Furthermore most colorimeters are point measurement devices. This is not a feasible alternative for the purpose of grading fish as none of the R, G, and B ranges fit the narrow range of wavelengths representing the difference between quality groups. Point measurement is not attractive either since the measurement must be more “intelligent”. The colour must be measured locally in certain areas of the fish and other areas or picture elements must be avoided if they do not represent the colour of the healthy fish flesh, for example, blood spots, bones and skin. A colour camera could be an alternative, but it gives excessive and unwanted information in spectral ranges we do not want to measure. Because no theoretical colour standard is available for the different quality groups of the fish, the most important issue is to define the colour of the fish and the colour difference between quality groups. The most accurate way to represent colour is by its reflectance or transmittance of different wavelengths of light.
4.1 Computer Vision in Food Handling and Sorting 705 In an effort to characterise the colour of fish, a considerable amount of fish was chosen, from each of the quality groups used, as a sample for the measurement. The fish was graded by five quality control personnel. Three samples were cut from each fish and the spectrum of each sample was measured with a spectrophotometer. The measurement covered all of the VIS spectrum and stretched into the NIR. All samples were measured in the range 350-1050 nm, with a 5 nm resolution, and some also in the range 1050-1600 nm. The results from the measurement showed that the spectral difference between groups was high in a certain narrow range of wavelengths in the VIS spectrum while there was little or no difference outside this range. From results of the spectrum measurements the optimum lighting and sensing equipment was selected. For lighting we choose diffused front lighting, light sources with spectral characteristics strong in the range where the difference between groups was most evident and weaker outside this range, thereby exaggerating the difference. A sensing equipment that fits our purpose is a black and white CTD frame camera with the appropriate bandpass filter in the specific wavelength range. Another colour feature that is to be taken into account when estimating the quality of fish is reddish bloodspots [40] that can occur in the fish flesh and which decrease the quality of the fish. These bloodspots must be detected locally in the fish flesh in some well defined areas, since the position of the spot plays a role concerning the weight of the defect (a blood spot on the more expensive loin piece of the fish is a more serious defect than a spot on the tail). A camera is therefore also suitable for detecting bloodspots. When using a black and white CTD camera to detect reddish bloodspots on the light fish flesh, the use of a bluish filter is appropriate to enhance the difference between healthy flesh and blood, making the discrimination easier. 3.2.3. Image processing
As the fish arrives on a conveyor under the camera an image of it is acquired. The fish position and orientation in the image is determined and then the image is segmented into predefined areas based on the form and the aspect ratio of the fish. The segmentation is necessary because of the different weights the fish pieces have in the quality evaluation. The colour of the fish flesh is computed individually for each area. A grey scale histogram is computed for the area and after smoothing the histogram with a moving average, two thresholds are computed deciding the interval of grey values belonging to the healthy fish flesh. The colour of the area is characterised by the average grey value of the healthy fish flesh pixels (in this case however the “grey values” of the image represent a very narrow range in the spectrum since the image is filtered). Bloodspots are also tackled locally in predefined areas. The area is thresholded with a local thresholding operator, and a binary image of blood spots on healthy
706
H. Arnarson & M. Asrnundsson
Fig. 6. Noise reduction, and detection and classification of surface defects of fish fillets. (a) Original image of fillet, (b) binary image of fillet, (c) results from classification of spots by size and shape.
fish flesh is produced. Morphological operators are then used to evaluate the size and shape of the bloodspots (Fig. 6). 4. Concluding Remarks
By studying the work effort that people perform in the food industry, it is clear that one of the main obstacles in automation is the need of intelligent human-like operation of the machines. That is the machine has to be able to sense the food product to adjust and optimise the handling of the food. If increased automation in the food industry is to come, it has to rely on intelligent sensing, where computer vision will play a major role. In this chapter we have discussed the use of computer vision in the food industry. To be successful in this type of application special attention has to be put into the development of the image acquisition part of the vision system. This includes study of the object characteristics, lighting and viewing techniques, sensors and optics. The image processing algorithms have t o be able to work in real time on randomly oriented objects of varying size, and shape. This processing demand limits what type of algorithms can be used, and imposes the need for special purpose arithmetic units to perform the processing. There exist several commercial applications today where computer vision is used to guide food handling and sorting. Nevertheless it is a fact that this field has
4.1 Computer Vision in Food Handling and Sorting 707 been growing slower than expected in the past five years. The main reason being that people underestimated the difficulties of applying this technique to objects of varying size and shape as food products are. These characteristics in fact demanded a processing power that was not available at a price acceptable to the food industry. The evolution of the computer industry is also clear, as the price is still going down and the performance of the system is increasing. In recent years there has been significant development in sensors, for example the Charge Injection Cameras [41] offering random scanning of the sensor. Another interesting development is also in the developing of intelligent camerase, where using CMOS technology the sensor and driving circuit are developed in one computer chip. This makes it possible to access the sensor faster than conventional CCD cameras, taking this step further where a processor and 110 chips can be added to the system, resulting in a Vision system capable of fast and effective preprocessing. An example of this is the 3-D range systemf which can process up to 70,000 range valuesls on a low cost hardware. This kind of intelligent system will open up a range of new applications for the Vision system where the price performance ratios were too high before. Because of the continuous evolution of the technology we believe that it is only a question of time before computer vision plays a major role in controlling and handing of food products.
References R. C. Harrell, D. C. Slaugter and P. D. Adsit, A fruit-tracking system for robotic harvesting, Machine Vision and Applications 2 (1989) 69-80. C. Pellerin, CRE cross the pond with a DAB hand for food inspection, Sensor Review 11,4 (1991) 17-19. K. Khodabandeloo, Getting down to the bare bones, The Industrial Robot 16,3(1989) 16G165. A. MacAndrew and C. Harris, Sensors detect no food contamination, Sensor Review 11,4 (1991) 23-26. Mare1 H/F, 1989. Product Information, Reykjavik, Iceland. W. V. D. Sluis, A camera and P C can now replace the quality inspector, Misset- World Poultry 7,10 (1991) 29-31. R. K. Dyche, INEX, 100 per cent on-line visual inspection of consumer products, Sensor Review 11,4 (1991) 14-17. L. F. Pau and R. Olafsson (eds.), Fish Quality Control by Computer Vision (Marcel Dekker, New York 1991). H. Arnarson, Fish Sorting Using Computer Vision, Ph.D. report LD 78, EMI, Technical University of Denmark, 1990. N. J. C. Strachan and C. K. Murray, Image analysis in the fish and food industries, in [5]. Lumitech, 1988. Product Information, Copenhagen, Denmark. Baader, 1990. Product Information, Lubeck, Germany. VISI Vision Ltd. (1996) Product Information, Aviation House, 31 Pinkhill, Edinbourgh, UK. fIVP AB (1996), Product Information, Teknikringen 2C, S-58330, Linkoping, Sverge.
708
H. Arnarson F4 M. Asmundsson
(131 B. G. Batchelor, D. A. Hill and D. C. Hodgson, Automated Visual Inspection (IFS, Bedford, UK, 1985). [14] A. Novini, Fundamentals of machine vision lighting, Proc. SPIE, Vol. 728, 1987,84-92. [15] D. Poussard and D. Laurendeau, 3-D sensing for industrial computer vision, in J. L. C. Sanz (ed.), Advances in Machine Vision (Springer, New York, 1988). [16] J. PBtursson, Optical spectra of fish flesh and quality defects in fish, in L. F. Pau and R. Olafsson (eds.), Fish Quality Control by Computer Vision (Marcel Dekker, 1991) 45-70. [17] Pulsar, 1990. Product Information, Eindhoven, Holland. [18] D. L. Hawley, Final Report: Fish Parasite Research, Federal grant No. NA-85-ABH00057, USA, 1988. [19] H. Hafsteinsson and S. S. H. Rizvi, Journal of Food Protection 50, 1 (1987) 70-84. [20] H. H. HUSS,P. Sigsgaard and S. A. Jensen, Fluoresence of fish bones, Journal of Food Protection 48, 5 (1984) 393-396. (211 K. Harding, Lighting & Optics Tutorial, VISION’87, SME, Detroit, Jun. 1987. [22] W. T. Welford, Abemtion of the Symmetrical Optical System (Academic Press, New York, 1974). [23] Oriel Corporation, 1990. Optics and Filters, Vol. 111, Stratford, CT, 1990. [24] J. S. Weszka, A survey of threshold selection techniques, Comput. Graph. Image Process. 7 (1978) 25+265. [25] N. Otsu, A threshold selection for gray-level histograms, IEEE Trans. Syst. Man Cybern. 9, 1 (1979) 62-66. [26] M. D. Levine, Vision in Man and Machine (McGraw-Hill, New York, 1985). [27] P. Maragos and R. W. Schafer, Morphological skeleton representation and coding of binary images, IEEE Trans. Acoust. Speech Signal Process. 34 (1986) 1228-1244. [28] M. K. Hu, Visual pattern recognition by moment invariants, IRE Trans. Inf. Theory 8 (1962) 179-187. [29] S. Serra, Image Analysis and Mathematical Morphology (Academic Press, New York, 1982). [30] H. Arnarson and L. F. Pau, Shape classification in computer vision by the syntactic, morphological and neural processing technique PDL-HM, in PTOC.ESPRIT-BRA Workshop on Specialized Processors f o r Real Time Image Analysis, Barcelona, Spain, Sept. 1991. [31] K. F’ukunaga, Introduction to Statistical Pattern Recognition (Academic Press, New York, 1972) 260-267. [32] A. K. C. Wong, Knowledge representation for robot vision and path planning using attributed graphs and hypergraphs, in A. K. C. Wong and A. Pugh (eds.), Machine Intelligence Knowledge Engineering Robotic Applications (Springer-Verlag, New York, 1977). [33] J. Kittler and M. J. B. Duff (eds.), Image Processing System Architectures (Research Studies Press Ltd, UK, 1985). [34] L. Uhr, K. Preston, S. Levialdi and M. J. B. Duff (eds.), Evaluation of Multicomputers for Image Processing (Academic Press, Orlando, 1986). [35] J. L. C. Sanz, Which parallel architectures are useful/useless for vision algorithms? Machine Vision and Applications 2 , 3 (1989). [36] J. Heldbo, Information teknologi og Productionsstyring i Konsumfiske industrien (in Danish), Ph.D. Report, EF201, Technical University of Denmark, 1989. [37] H. Arnarson, K. Bengoetxea and L. F. Pau, Vision applications in the fishing and fish product industries, Int. J. Pattern Recogn. Artif. Intell. 2 , 4 (1988) 657-673. [38] H. Arnarson, Fish and fish product sorting, in [5].
4.1
Computer Vision in Food Handling and Sorting
709
[39] Honeywell, USA, Product Information. [40] K. Bengoetxea, Lighting setup in the automatic detection of ventral skin and blood spots in cod fish fillets, Report No. 497, EMI, Technical University of Denmark, 1988. [41] CID Technologies Inc., 1988. Product Information, Liverpool, USA.
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 711-736 Eds. C. H. Chen, L. F. P a u and P. S . P. Wang @ 1998 World Scientific Publishing Company
I CHAPTER 4.21 APPROACHES TO TEXTURE-BASED CLASSIFICATION, SEGMENTATION AND SURFACE INSPECTION
MATTI PIETIKAINEN, T I M 0 OJALA and OLLI SILVEN Machine Vision Group, Infotech and Department of Electrical Engineering, University of Oulu, FIN-90570 Oulu, Finland E-mail: {mkp,skidi,olli} @ee.oulu.fi Over the last few years significant progress has been made in applying methods using distributions of feature values to texture analysis. Very good performance has been obtained in various texture classification and segmentation problems. This chapter overviews recent progress and presents some examples to demonstrate the efficiency of the approach. Problems of analyzing textured surfaces in industrial applications are also discussed. A general overview of the problem space is given, presenting sets of solutions proposed and their prerequisites. Keywords: Texture, classification, segmentation, feature distributions, visual inspection, quality control.
1. Introduction Texture analysis is important in many applications of computer image analysis for classification, detection, or segmentation of images based on local spatial variations of intensity or color. Important applications include industrial and biomedical surface inspection, for example for defects and disease, ground classification and segmentation of satellite or aerial imagery, segmentation of textured regions in document analysis, and content-based access to image databases. Most of the texture analysis applications can be regarded as texture classification or segmentation problems, or as a combination of both. A wide variety of techniques for discriminating textures have been proposed. For recent surveys of texture analysis, see [l-51. The methods can be divided into four categories: statistical, geometrical, model-based and signal processing. Various methods of each category are decribed in Chap. 2.1 [4]. Among the most widely used approaches are statistical methods based on co-occurrence matrices of secondorder gray level statistics or first-order statistics of local property values (difference histograms), signal processing methods based on local linear transforms, multichannel Gabor filtering or wavelets, and model-based methods based on Markov random fields or fractals. In texture classification, the goal is to assign an unknown sample to one of several predefined categories. The choice of proper texture measures is crucial for 711
712
M. Pietikainen, T. Ojala & 0. Silven
the performance of classification. Most of the approaches to texture classification quantify texture measures by single values (means, variances etc.), which are then concatenated into a feature vector. The feature vector is fed to an ordinary statistical pattern recognition procedure or neural network for purposes of classification. Recent research results, however, demonstrate that methods based on comparison of distributions of feature values provide very good classification performance for various types of textures [6,7], and should be considered for many applications. Segmentation is a process of partitioning the image into regions which have more or less homogeneous properties with respect to color, texture, etc. [8].The methods of texture segmentation are usually classified as region-based, boundarybased or as a hybrid of the two. For a recent survey of texture segmentation techniques, see [3]. The segmentation can be supervised or unsupervised. In unsupervised segmentation, no a priori information about the textures present in the image is available. This makes it a very challenging research problem in which only limited success has been achieved. The application of the histogram comparison approach to texture segmentation appears to provide significant improvement in performance [9]. There are many potential areas of application for texture analysis in industry [lo-121, but only a limited number of examples of successful exploitation of texture exist. A major problem is that textures in the real world are often not uniform, due to changes in orientation, scale or other visual appearance. In addition, the degree of computational complexity of many of the proposed texture measures is very high. Before committing effort into selecting, developing and using texture techniques in an application, it is necessary to thoroughly understand its requirements and characteristics. Section 2 of this chapter presents an approach to texture classification using feature distributions. It describes two simple but powerful texture measures and a classification principle based on comparison of feature distributions, and demonstrates the performance of the approach in a case study. Section 3 describes a powerful unsupervised texture segmentation method utilizing texture discrimination techniques discussed in Section 2. The focus of Section 4 is more application specific and not directly linked to the previous sections. The use of texture-based approaches in industrial surface inspection problems is considered. 2. Texture Classification Using Feature Distributions In texture classification an unknown image sample is assigned to one of a priori known texture categories (Fig. 1). The features derived from the unknown sample are compared to the features of each category in the training set, and the sample is assigned to the class with the closest match. The performance of texture classification is largely dependent on the efficiency of texture features used. Chap. 2.1 [4] provides a description of various types of texture features and Chap. 2.2 [5] discusses model-based approaches in more detail.
714 M . Pietikainen, T. Ojala & 0. Silven
can be obtained by using distributions of simple texture measures, like absolute gray level differences, local binary patterns and center-symmetric auto-correlation. The performance is usually further improved with the use of two-dimensional distributions of joint pairs of complementary features [7]. In experiments involving various applications we have obtained very good results with the distribution-based classification approach. Among the studies we conducted are the determination of the composition of mixtures of two materials [24] (Section 2.3) and the average grain size of chrome concentrate [25] (Section 4.2.1), metal strip inspection [26], discrimination of melanoma (skin cancer) cell samples from naevus cell samples [27], and rotation-invariant texture classification [28]. A similar approach has also been successfully applied to accurate color discrimination [29]. 2.1.1. Gray level difference method
The method based on histograms of absolute differences between pairs of gray levels or of average gray levels has been successfully used for texture classification, for example in [13,30,7]. It should be noted that the difference histograms can also be derived from co-occurrence matrices which were described in Chap. 2.1 [4]. For any given displacement d = (dz,dy), where dx and dy are integers, let f’(z,y) = lf(z,y) - f(z + dx,y dy)l. Let P’ be the probability density function off‘. If the image has m gray levels, this has the form of an m-dimensional vector whose ith component is the probability that f‘(z,y) will have the value i. P’ can be easily computed by counting the number of times each value of f’(z, y) occurs. For a small d the difference histograms will peak near zero, while for a larger d they are more spread out. As an example consider the 4 x 4 image used in Chap. 2.1:
+
1100 1100 0022 0022
The difference histogram for this image for a displacement vector of d
=
(1,O)
is: DIFFX = [8, 2, 21. The rotation invariant feature DIFF4 used in Section 2.3 is computed by accumulating, in the same one-dimensional histogram, the absolute gray level differences in all four principal directions at the chosen displacement D. If D = 1, for example, the displacements d = ( O , l ) , (l,l),(1,O) and (1,-1) are considered.
4.2 Approaches to Texture-Based Classification . . . 715 The difference histogram approach is not invariant with respect to gray scale variance, which means that the textures to be analyzed should be gray scale corrected by, e.g. histogram equalization.
2.1.2. Local binary patterns Recently, Ojala et al. [7] introduced a Local Binary Pattern (LBP) texture operator, which is a two-level versions of the texture operator proposed by Wang and He [31]. The LBP histogram computed over a region is then used for classification. LBP provides us with knowledge about the spatial structure of the local image texture. However, LBP does not address the contrast of texture which is important in the discrimination of some textures. For this purpose, we can combine LBP with a simple contrast measure C. By considering joint occurrences of LBP and C we usually achieve better discrimination than just with LBP alone. The descriptions of LBP and C are shown in Fig. 2. For each local 3 x 3 neighborhood, if Pi
1. Tkeshold pixels Pi by the value of the center pixel: P,! =
< PO
8
2. Count the number n of resulting non-zero pixels: n = X
Pl
i=l 8
3. Calculate the local binary pattern LBP = c P i 2 i - 1 i=l
pixels
example
thresholded
weights
LBP = 1+8+32+128 = 169 C = (6+7+9+7)/4-(5+2+1+3)/4
= 4.5
Fig. 2. Computation of Local Binary Pattern (LBP) and contrast measure C.
LBP and LBP/C perform well also for small image regions (e.g. 16 x 16 pixels), which is very important in segmentation applications. A simple way to define a “multiresolution” LBP would be to choose the eight neighbors of the center pixel from the corresponding positions in different neighborhoods (3 x 3, 5 x 5, 7 x 7, etc.). By definition, LBP is invariant against any monotonic gray scale transformation.
716
M. Pietikainen, T. Ojala €9 0. Silven
The method is rotation variant which is undesirable in certain applications. A rotation invariant version of LBP is considered in [32,28].
2.2. Classification Using Feature Distributions
A log-likelihood-ratio pseudo-metric, the G statistic, is used for comparing feature distributions in the following experiments, but it could be replaced with some other related method, like histogram intersection [33] or by the statistical chi-square test. The value of the computed G statistic indicates the probability that the two sample distributions come from the same population: the higher the value, the lower the probability that the two samples are from the same population. The distribution of G asymptotically follows a chi-square distribution, but it has some theoretical advantages and it is computationally simpler. For a goodness-of-fit test the G statistic is: n G = 2 X s i l o g - ,si mi i=l where s and m are the sample and model distributions, n is the number of bins and si, mi are the respective sample and model probabilities at bin i. In the experiments presented in this section, a single model distribution for every class is not used. Every sample is in its turn classified using the other samples as models, hence the leave-one-out approach is applied. The model samples are ordered according to their probability of coming from the same population as the test sample. This probability is measured by a two-way test-of-independence:
where s, m are the two texture samples (test sample and model), n is the number of bins and fi is the frequency at bin i. For a detailed derivation of the formula, see Sokal and Rohlf [34]. After the model samples have been ordered, the test sample is classified using the Ic-nearest neighbor principle, i.e. the test sample is assigned to the class of the majority among its Ic nearest models. The feature distribution for each sample is obtained by scanning the texture image with the local texture operator. The distributions of local statistics are divided into histograms having a fixed number of bins. The histograms of features with continuous-valued output, like contrast C in LBP/C, are quantized by adding together feature distributions for every single model image in a total distribution which has been divided into N bins having an equal number of entries. Hence, the cut values of the bins of the histograms correspond to 100/N percentile of
4.2 Approaches to Texture-Based Classification
...
717
total distribution
I bin 0
bin 1
bin 2
bin 3
cutvalues
(b) Fig. 3. Quantization of the feature space, when 4 bins are requested. Single distributions are added together in a total distribution (a), which is divided into 4 equal portions, i.e. the cut value between bin 0 and bin 1 corresponds t o 25% percentile of the combined data (b).
the combined data (Fig. 3). Deriving the cut values from the total distribution and allocating every bin the same amount of the combined data guarantees that the highest resolution of the quantization is used where the number of entries is largest and vice versa. Output of discrete operators like LBP does not require any quantization; operator outputs are just accumulated into a histogram. The empty bins are set to one. To compare distributions of complementary feature pairs, like LBP/C, metric G is extended in a straightforward manner to scan through the two-dimensional histograms. If quantization of the feature space is required, it is done separately for both features using the same approach as with single features.
2.3. Case: Determining Composition of Mixtures of Materials Kjell proposed that the composition of grain mixtures can be determined by texture classification when different compositions are seen as different textures and the classification into discrete classes is considered as a measurement event [35]. The accuracy of the measurement is heavily dependent on the number of classes and the discriminative power of texture features. Kjell examined the performance of Laws’ texture energy measures [23] and ordinary feature vector based classification using images of eleven different mixtures of rice and barley as test material. He achieved promising results when using all nine of Laws’ 3 x 3 features at the same time. About 61 percent of the samples were classified into correct classes, and misclassified samples were close t o their own classes on the diagonal of the confusion matrix. A sample size of 128 x 128 pixels was used. Recently, the distribution-based classification approach was applied to the same problem [24]. In order to have comparable test material, eleven different mixtures of rice and barley grain were prepared. Four images of each different mixture were taken using a SONY 3 CCD DXC-755 color camera. Images were converted to 512 x 512 gray scale images with square pixels. Four test images of different mixtures are shown in Fig. 4.
718
M. Pietikainen, T. Ojala €9 0. Silven
Fig. 4. Rice 100% (a), Barley 100% (b), Rice 70%, barley 30% (c), Rice 30%, barley 70% (d).
We present results of distribution based classification for the gray scale difference feature DIFF4 (Section 2.1.1), with histograms quantized into 32 and 256 bins, respectively. For nearest-neighbor selection, a value of 3 was used for k(3-NN classification). The sample data was split into training and test sets using the leave-one-out method. The classifier was designed by choosing all but one sample for inclusion in the design set and the single sample in the set was then classified. This procedure was repeated for all samples and the classification error rate was determined as the percentage of misclassified samples out of the total number of samples. The effects of sample size and image preprocessing were also examined. The samples were obtained by dividing the original 512 x 512 images into non-overlapping subimages, resulting in 176, 704, 2816 and 11264 samples in total for sample sizes of 256 x 256, 128 x 128,64 x 64 and 32 x 32 pixels, respectively. Histogram equalization was performed prior to feature extraction t o remove the effects of unequal brightness and contrast. It was applied to the whole set of 512 x 512 test images instead of the separate samples. Table 1 shows the results. The numbers in the tables denote the percentages of misclassified samples. Misclassification rate does not reveal how close the misclassified samples are to their correct classes, but it gives enough information t o decide which features are suitable for this kind of application. More detailed information can be extracted from confusion matrices.
4 . 2 Approaches to Texture-Based Classification . . . 719 Table 1. Distribution based classification. DIFF4 sample size
32 bins D = l
128 x 128
38.21
256 bins
D=2
D = l
D = 2
38.35
43.61
42.61
256 x 256
28.41
30.11
30.68
34.66
EQ 32 x 32
43.47
63.22
7.28
14.58
EQ 64 x 64
22.83
37.82
0.11
0.32
EQ 128 x 128
13.07
21.16
0.00
0.00
EQ 256 x 256
11.93
22.73
0.00
0.00
The results are tabulated for two different quantizations (32 and 256 bins) and displacements (D = 1, 2). Histogram equalization improved significantly the performance. The reason for this is that DIFF4 is not invariant with respect t o gray scale variance. This means that the textures t o be analyzed should be gray scale corrected in order to have gray level differences of equal scale in all textures. For example, using a 128 x 128 pixel sample size and distributions with 256 bins, the error for DIFF4 was reduced from 43.61% to 0.00%. Even with 64 x 64 samples, the classification error was as small as 0.11%. This means that only 3 out of all 2816 samples were misclassified. All these misclassified samples were classified to the neighboring class of their correct class which means that the measurement error of the composition for these samples was 10%. The sample size 32 x 32 appeared to be too small for this kind of measurement purpose. Even if the total error rate was only 7.28% for D = 1, a lot of samples were classified far away from their correct classes. One reason for this might be that with a little sample size like 32 x 32 the composition of a sample is not necessarily correct. Table 2. Robustness tests. DIFF4 for sample size 64 x 64 image size (k)
32 bins
256 bins
D = l
D=2
D = l
D = 2
512 x 512(k = 1)
22.43
42.40
0.11
0.64
512 x 512(k = 3)
22.83
37.82
0.11
0.32
512 x 512(k = 5)
22.02
36.08
0.11
0.32
256 x 256(k = 1)
36.79
55.11
0.43
0.00
256 x 256(k = 3)
32.81
51.28
0.00
0.00
256 x 256(k = 5)
34.52
50.71
0.00
0.14
720
M. Pietikainen, T. Ojala €4 0. Silven
In order to test the robustness of the presented approach, experiments with three different values of k(k = 1, 3, 5) and with two image resolutions (original 512 x 512 images and 256 x 256 images obtained by bilinear interpolation from the original images) were performed. A sample size of 64 x 64 was used. Table 2 presents the results. It can be seen that distributions with 256 bins provide robust performance with respect to variations of Ic, displacement D, and image resolution, achieving error rates of 0.64% or less in all cases. 3. Texture Segmentation Using Feature Distributions
Segmentation of an image into differently textured regions is a difficult problem. Usually one does not know a priori what types of textures exist in an image, how many textures there are, and what regions have which textures [4]. In order to distinguish reliably between two textures, relatively large samples of them must be examined, i.e. relatively large blocks of the image. But a large block is unlikely to be entirely contained in a homogeneously textured region and it becomes difficult to correctly determine the boundaries between regions. The performance of texture segmentation is largely dependent on the performance of texture features used. The features should easily discriminate various types of textures. The window size used for computing textural features should be small enough to be useful for small image regions and to provide small error rates at region boundaries. Recent comparative studies performed by Ohanian and Dubes (151 and Ojala et al. [7] indicate that texture measures based on co-occurrence matrices, difference histograms and local binary patterns (LBP) perform very well for various types of textures. The performances of these features are good for small window sizes as well. Recently, an unsupervised texture segmentation algorithm utilizing the LBP/C texture measure (Sec. 2.1.2) and histogram comparison (Section 2.2) was developed. The method has performed very well in experiments. It is not sensitive t o the selection of parameter values, does not require any prior knowledge about the number of textures or regions in the image, and seems to provide significantly better results than existing unsupervised texture segmentation approaches. The method can be easily generalized, e.g. to utilize other texture features, multiscale information, color features, and combinations of multiple features. This section presents a n overview of the method and shows some experimental results. 3.1. Segmentation Method The segmentation method consists of three phases: hierarchical splitting, agglomerative merging and pixelwise classification. First, hierarchical splitting is used to divide the image into regions of roughly uniform texture. Then, an agglomerative merging procedure merges similar adjacent regions until a stopping criterion is met. At this point, we have obtained rough estimates of the different textured regions present in the image, and we complete the analysis by a pixelwise classification to
4.2 Approaches to Texture-Based Classafication . . .
721
Fig. 5. Texture mosaic #l; the main sequence of the segmentation algorithm.
improve the localization. The method does not require any prior knowledge about the number of textures or regions in the image, as many existing approaches do. Figure 5 illustrates the steps of the segmentation algorithm on a 512 x 512 mosaic containing five different Brodatz [36] textures. 3.1.1. Hierarchical splitting
A necessary prerequisite for the agglomerative merging to be successful is that the individual image regions be uniform in texture. For this purpose, we apply the hierarchical splitting algorithm, which recursively splits the original image into square blocks of varying size. The decision on whether a block is split into four subblocks is based on a uniformity test. Using Eq. (2.2) we measure the six pairwise G distances between the LBP/C histograms of the four subblocks. If we denote the largest of the six G values by G,,, and the smallest by Gmin,the block is found to be non-uniform and is thus split further into four subblocks, if a measure of relative dissimilarity within the region is greater than a threshold:
R=- Gmax > X Gmin
Regarding the proper choice of X , one should rather choose too small a value for X than too large a value. It is better to split too much than too little, for the following agglomerative merging procedure is able to correct errors, in cases where a uniform block of a single texture has been needlessly split, but error recovery is not possible if segments containing several textures are assumed to be uniform. To begin with, we divide the image into rectangular blocks of size S,,,. If we applied the uniformity test to arbitrarily large image segments, we could fail to detect small texture patches and end up treating regions containing several textures as uniform. The next step is to use the uniformity test. If a block does not satisfy the test, it is divided into four subblocks. This procedure is repeated recursively on each subblock until a predetermined minimum block size Sminis reached. It is necessary to set a minimum limit on the block size, for the block has to contain a sufficient number of pixels for the LBP/C histogram to be reliable.
722
M. Pietikainen, T. Ojala €4 0. Silven
Figure 5(b) illustrates the result of the hierarchical splitting algorithm with = 64 and Smin 16. As expected, the splitting goes deepest around the texture boundaries.
X
= 1.2, S,
3.1.2. Agglomerative merging Once the image has been split into blocks of roughly uniform texture, we apply an agglomerative merging procedure, which merges similar adjacent regions until a stopping criterion is satisfied. At a particular stage of the merging, we merge that pair of adjacent segments which has the smallest Merger Importance ( M I ) value. M I is defined as MI+pxG (3.2) where p is the number of pixels in the smaller of the two regions and G is the distance measure defined in Eq. (2.2). In other words, at each step the procedure chooses that merge, of all possible merges, introduces the smallest change in the segmented image. Once the pair of adjacent segments with the smallest M I value has been found, the regions are merged and the two respective LBP/C histograms are summed to be the histogram of the new image region. Before moving to the next merge we compute the G distances between the new region and all regions adjacent to it. Merging is allowed to proceed until the stopping rule (3.3) triggers. Merging is halted if M I R , the ratio of MI,,,, the Merger Importance for the current best merge, and MI,,,, the largest Merger Importance of all the preceding merges, exceeds a present threshold Y . Threshold Y determines the scale of texture differences in the segmentation result and therefore the choice of Y depends on the application. In theory, it is possible that the very first merges have a zero MI value (i.e. there are adjacent regions with identical LBP/C histograms), which would lead to a premature termination of the agglomerative merging phase. To prevent this the stopping rule is not evaluated for the first 10% of all possible merges. Figure 5(c) shows the result of the agglomerative merging phase after 174 merges. The M I R of the 175th merge is 9.5 and the merging procedure stops. For comparison, the highest M I R value up to that point had been 1.2. 3.1.3. Pixelwise classification If the hierarchical splitting and agglomerative merging phases have succeeded, we have obtained quite reliable estimates of the different textured regions present in the image. Treating the LBP/C histograms of the image segments as our texture models, we switch to a texture classification mode. If a n image pixel is on the boundary of at least two distinct textures (i.e. the pixel is 4-connected t o at least one pixel with a different label), we place a discrete disc with radius r on the pixel
4.2 Approaches to Texture-Based Classification . . . 723 and compute the LBP/C histogram over the disc. The reason for using a discrete disc instead of a square window is that the latter weighs the four principal directions unequally. We compute the G distances between the histogram of the disc and the models of the regions which are 4-connected to the pixel in question. We relabel the pixel, if the label of the nearest model is different from the current label of the pixel and there is at least one 4-connected adjacent pixel with the tentative new label. The latter condition improves the smoothness of texture boundaries and decreases the probability of small holes occurring inside the regions. If the pixel is relabeled, i.e. it is moved from an image segment to an adjacent segment, we update the corrsponding texture models accordingly; hence the texture models become more accurate during the process. Only those pixels at which the disc is entirely inside the image are examined; hence the final segmentation result will contain a border r pixels wide. In the next sweep over the image we only check the neighborhoods of those pixels which were relabeled in the previous sweep. The process of pixelwise classification continues until no pixels are relabeled or a maximum number of sweeps is reached. This is set to be two times Smin, based on the reasoning that the boundary estimate of the agglomerative merging phase can be a t most this far away from the “true” texture boundary. Setting an upper limit on the number of iterations ensures that the process will not wander around endlessly, if the disc is not able to capture enough information about the local texture to be stable. According t o our experiments the algorithm generally converges quickly with homogeneous textures, whereas with locally stochastic natural scenes the maximum number of sweeps may be used. Figure 5(d) demonstrates the final segmentation result after the pixelwise classification phase. A disc with a radius of 11 pixels was used and 16 sweeps were needed. The segmentation error is 1.7%.
3.2. Experiment a Results Next we present some quantitative results obtained with the method. The segmentation results for three texture mosaics and a natural scene are presented. The same set of parameter values was used for all texture mosaics to demonstrate the robustness of the approach: b = 8 (number of bins for contrast measure C in the texture transform),, , S , = 64 (largest allowed block size in the hierarchical splitting algorithm), Smin= 16 (smallest allowed block size in the hierarchical splitting algorithm), X = 1.2 (threshold determining when a block is divided into four subblocks in the hierarchical splitting algorithm), Y = 2.0 (threshold determining when the agglomerative merging algorithm is to be halted), and r = 11 (radius of the disc in the pixelwise classification algorithm). For each image, we provide the original image, the rough segmentation result after the agglomerative merging phase and the final segmentation result after the pixelwise classification phase. The segmentation results are superposed on the original image and the final segmentation result contains only the area processed by
724
M. Pietakainen, T. Ojala & 0. Silven
Fig. 7. Texture mosaic #3.
the disc. Also, in each case we collect values M I h & , and MIRhi. MI&top is the value of M I R when the agglomerative merging is stopped and MIRhi is the highest M I R of the preceding merges. The relationship between MI&top, MIRhi and the threshold Y reflects the reliability of the result of the agglomerative merging phase. Mosaic #2 is a 512 x 512 image with a background made by a GMRF process and four distinct regions; the square and the circle are painted surfaces with different surface roughnesses and the ellipse and the triangle are made by a fractal process [15]. As we can see from the values of MI&top (8.1) and M I & (1.2), the rough estimate (Fig. 6(b)) of the texture regions is obtained relatively easily. Figure 6(b) contains 4.6% and Fig. 6(c) only 1.9% misclassified pixels. Mosaic #3 (Fig. 7(a)), which is 384 x 384 pixels in size, is composed of textures taken from outdoor scenes [37]. Our method gives a very good segmentation result of 2.1%. Note that the pixelwise classification clearly improves the result of the agglomerative merging phase (7.8%). The difference between M I h t o p (2.8) and M I & (1.2) is still noticeable, but by far the smallest in the three cases, reflecting the inherent difficulty of this problem. We also applied the texture segmentation method t o natural scenes. The scenes were originally in RGB format [38], but we converted them to gray level intensity
726
M. Pietikainen, T. Ojala & 0. Silven
by straightforwardly computing the desired feature for suitably symmetrical discrete neighborhoods of any size, such as disks, or boxes of odd and even size. The remaining question is how to combine the multiple feature channels obtained with several features and/or scales. We can hardly expect to reliably estimate joint distributions for a large number of features. Also, multidimensional histograms with large numbers of bins are very computationally intensive and consume a lot of memory. An alternative is to use an approximation with marginal distributions and to employ each independent feature separately, as a 1-D histogram, to compute a similarity score such as G for each feature, and then integrate individual scores into an aggregate similarity score, as was done in recent texture and color classification experiments [32,29]. Combining it with a carefully chosen set of nonredundant complementary features, we expect to further improve the performance of the segmentation method. In a similar way, joint pairs of features, like LBP/C, can be combined with other single features or feature pairs. It would also be possible to use single features or joint features one by one, e.g. by first comparing the uniformity of regions with respect to texture and then with respect t o color. 4. Texture Analysis in Surface Inspection Textured materials may have defects that should be detected and identified as in crack inspection of concrete or stone slabs, or the quality characteristics of the surface should be measured as in granulometry. In many applications both objectives must be pursued simultaneously, as is regularly the case with wood, steel and textile inspection. Because these and most natural and manufactured surfaces are textured, one would expect this application characteristic to be reflected by the methodological solutions used in practical automatic visual inspection systems. However, only a few examples of successful explicit exploitation of texture techniques in industrial inspection exist , while most systems, including many wood inspection devices, attempt to cancel out or disregard the presence of texture, trying to transform the problems solvable with other detection and analysis methods, e.g. as done by Dinstein et al. [39]. This is understandable against the high costs of texture inspection, and the fact that often the defects of interest are not textured, but embedded in it like cracks. The inspection of textured surfaces is regularly treated more as a classification and less as a segmentation task, simply because the focus is on measuring the characteristics of regions and comparing them t o prior trained samples. Actual working texture based industrial inspection solutions are available mostly for homogeneous periodic textures, such as on wallpaper and fabric, where the patterns normally exhibit only minimal variation, making defect detection a two category classification problem. Natural textures are more or less random with large non-anomalous deviations, as anyone can testify by taking a look at a wood surface, resulting in the need to add features just to capture the range of normal variation, not to mention of the detection and identification of defects.
4.2 Approaches
to Texture-Based Classification . . .
727
Defect detection may require continuous adaptation or adjustment of features and methods based on the background characteristics, possibly resulting in a complex multi-category classification task already at the first step of inspection. Solutions providing adaptability have been proposed, among others, by Dewaele et al. [40]and Chetverikov [41].Proprietary adaptation schemes are regularly used in commercial inspection systems. In most industrial applications inspection systems must process 1&40 Mpixels/s per camera, thus requiring dedicated hardware for at least part of the system, so the calculation of each new texture feature can be a significant expense that should be avoided. Therefore, the system developers try to select a few powerful straightforwardly implementable features and tune them precisely for the application problem. A prototypical solution depicted in Fig. 9 uses a bank of filters or texture transforms characterising the texture and also defect primitives, each transform producing a feature image that is used in either pixel-by-pixel or window based classification of the original image data.
classifier
transform n Fig. 9. Typical methodological architecture of texture based inspection systems.
The dimensions of the filters used in applications have ranged up to 63 x 63 for pixel classification [42], while most implementors rely on 3 x 3 Laws’ masks [23] or other convolution filters in classification of partially overlapping or non-overlapping windows, e.g. based on means and variances of texture measures. The developments in feature distribution based classification of texture should have a major simplifying impact on future systems, as the techniques have recently matured to the brink of real applicability. The improved efficiency in using the texture measures cuts the number of features needed in an application, enables classifying small regions, and potentially reduces training effort by relieving the dimensionality problem of classification. Nevertheless, many applications will always demand dedicated techniques for the detection of their vital defects. Regardless of the feature analysis methodology, the effort needed for training an inspection system to detect and identify defects from sound background remains a key cost driver for system deployment and use. As texture inspection methods are notoriously fragile with respect to resolution, a minor change between the distance of the camera to the target may result in a retraining need. This need may also arise from normal variations between product batches. Typically, training done in
728
M . Pietikainen, T . Ojala €4 0. Silven
6;
+background transition defect
I I I 1 I
regions with defects selected and used for training with remaining good background L___d I I I
I
(b)
( 4
(c)
Fig. 10. Alternatives for training defect detection methods. (a) Defect; (b) Pixel based training; and (c) Region based training.
laboratory turns out to be useless after an inspection system has been installed on-line. Furthermore, on-line training performed by production personnel tends to concentrate on teaching in “near-misse~~~ and ‘hear-hits” rather than representative defects and background, so non-parametric classifiers should be favored. Figure 10 shows two basic approaches t o training defect detection: pixel-based training assumes that a human operator is able t o correctly pinpoint pixels belonging to defects in the image and pixels that are from sound background. In region-based training the operator roughly labels regions that contain a defect or defects, but may also have a substantial portion of sound background, while the non-labeled regions are assumed sound. We strongly advocate the latter approach, because it is less laborious, and because it is difficult for a human to precisely determine the boundaries of defects. It should be noticed that pixel based training disregards the transition region t o defect, the characteristics of which may have high importance. For instance, the grain around a suspected defect in a lumber board helps in discriminating frequent stray bark particles from minor knots.
defect
-
/ b 1 0 b - t ~-~--c texture deviation
1 line-type
non-texturedblob known shape 1 meandering J
Fig. 11. Categories of defects on textured surfaces.
The detection and recognition of large defects from textured surfaces is relatively straightforward as changes of texture characteristics, but many defects are small local imperfections rather than ‘real’ texture defects such as knots with exactly the color of defectless background in wood. The detection of minor flaws from the background requires application of specific knowledge. In addition, segmentation may be required for measuring the defects and determining their characteristics. Figure 11 presents a categorization of defects on textured surfaces. In the following we briefly review means for detecting of defects from texture and then proceed into application examples on quality measurements and defect detection.
4.2 Approaches to Texture-Based Classification . . . 729 4.1. Detection of Defects
from Texture
The detection and segmentation of “sufficiently” large defects in texture images can be performed reliably with pure texture measures both for periodic and random textures using proposed texture measures [4]. But because texture is a statistical concept, texture measures are good only for regions that have the minimum size that allows the definition of features [41]. The relative sizes of these minimum patches for various features and textures can be roughly concluded from the boundaries in texture segmentation results given in literature: The lower the error, the smaller the defects that can be detected using that family of features. With a small patch size even most of the local texture imperfection can be detected, reducing, if not eliminating the need for application specific detection solutions for purposes such as the locating non-textured blobs from the background. In practice, choosing the patch size for an application depends on the desired balance between false alarm and error escape rates. A smaller patch size increases the number of misdetections from normal variations, while using larger patches may contribute to detection failures. Normally all detections are subjected to further scrutiny, so in the end the patch size is defined by the general purpose computational resources available for detailed analysis. The minimum patch size is smallest for periodic textures such as in textiles that must be inspected for both large and small weaving flaws that are generally multiples of the mesh size. In textile inspection, Ade et al. [43] found that using an imaging resolution of three pixels per mesh width and averaged outputs of 3 x 3 to 5 x 5 pixel filters, derived via Karhunen-Loeve expansion of the spatial covariance matrix, the diameter of the minimum patch is around 15 pixels. The smallest detected defects in [43] appear to be around 10% of the patch area. Neubauer [44], using approximately the same imaging resolution, exploited three 5 x 5 FIR-filters and performed classification using histograms of features calculated from 10 x 10 pixel regions, achieving 1.6% false alarm and 9.3% escape rates. The tests with LBP/C method for quasi-periodic textures (Fig. 5) performed by Ojala [9] with 16 x 16 pixel patch size and distribution classification detected 100% of the cases where more than about 25% of the block area did not belong to the same category. With natural textures (Fig. 7), the average detection threshold of other categories increased to around 35%. It is evident that the inspection accuracy may significantly benefit from dedicated methods for detecting small defects. Crack or scratch detection is undoubtedly the most common purpose for which specific techniques have been included in visual surface inspection systems. 4.1.1. Crack detection The relative difficulty of detecting cracks depends on whether their shape and typical orientation is known a priori, whether they start from the edge of the object, and on whether the texture is periodic or random. A key problem is the
730 M . Pietikainen, T. Ojala & 0. Silven
typically very small transverse dimensions and poor contrast of cracks: the human visual system may easily detect them, but they may actually consist of "chains" of nonadjacent single pixels in the image. In the worst case, the surface is randomly textured and the cracks may meander freely, starting and ending anywhere, leaving few application specific constraints that could be exploited. The detection of cracks having a known shape is often the straightforward application of Hough-transform or RANSAC to edge detected or high-pass filtered versions of the image. For instance, Gerhardt et al. [45] used Hough transform for detecting wrinkles in sandpaper in this manner. With meandering cracks, the problem of discriminating them from other high frequency components in the image is very difficult. If the texture is periodic or quasi-periodic, texture measures characterizing the background may be powerful enough for detecting their presence. An alternative, a rather unusual simple method for defect detection from periodic patterns, based on a model of human preattentive visual detection of pattern anomalies, has been proposed by Brecher [46]. Detection is performed by comparing local and global first order density statistics of contrast or edge orientation. Song et al. [47] have presented a trainable technique based on a pseudo-Wigner model for detecting cracks from periodic and random textures. The motivation behind selecting the technique is the better cojoint spatial and spatial frequency resolution offered by the Wigner distribution when compared to Gabor, differenceof-Gaussians and spectrogram approaches: this is an important factor due to the localness of cracks. The technique is trained with defectless images. During inspection it produces probabilistic distance images that are then postprocessed using rough assumptions on the shape of the cracks in the application. 4.2. Application Cases
Before committing effort into selecting, developing and using texture techniques in an application, it is necessary to thoroughly understand its requirements and characteristics. The developer should consider at least the following questions: 0
0
Is the surface periodically, randomly, or only weakly textured? Strongly periodic textures can be efficiently characterized using linear filtering techniques that are also relatively cheap to implement with off-the-shelf hardware. For random textures LBP/C and gray level difference features with distribution based classification are computationally attractive and rank among the very best. With weakly textured surfaces, plain gray-level and color distribution based classification may work very well [48]. Are any of the properties of the defects known? In particular, are there any defects that cannot be discriminated from the background by their color or intensity? Due to their cost texture methods should usually be the last ones to be thrown
4.2 Approaches
t o Texture-Based Classification
...
731
in. They are generally much better in characterizing surfaces than in detecting anomalies. Thus, whenever feasible, application specific non-texture method solutions may be justified for detection, while texture measures may be powerful in eliminating false alarms and recognizing the defects. The following application cases, particle size determination, carpet wear assessment and leather inspection are demonstrations of analysis of random and quasiperiodic textures, and defect detection from random textures, respectively. 4.2.1. Case 1: determination of particle size distribution On-line measurement of the size distribution of granular materials, e.g. coke, minerals, pellets, etc., is a common problem in the process industry, where knowledge of the mean particle size and shape of the distribution are used for control. The traditional particle size distribution measurement instruments, such as sieves, are suitable for off-line use in the laboratory. The off-the-shelf machine vision systems developed for this purpose are based on blob analysis and require mechanical set-ups for separating the particles from each other. Separation is often necessary, because smaller particles may fall t o the spaces between the bigger ones and are no longer visible, so the particle size distribution of the surface may not be representative. This happens if the relative size range of the particle diameter is around 1.5 or larger. Texture analysis has clear potential in granulometric applications, as has been demonstrated by the previous example on mixture determination in Section 2.3. In principle, a measurement instrument could be trained with pictures of typical distributions, but the preparation of samples with known distributions is a laborious task, making this approach unattractive. Furthermore, the training problem is amplified by the need for frequent recalibrations, because the appearance of the material may change with time. The desired approach is to train the instrument by sieved fractions of the material, or to eliminate the need for training, as is the case with particle separation based measurements. Rautio et al. [25] performed distribution measurement experiments using chrome concentrate that was sieved into 15 fractions, 37 t o 500 pm, for use as training samples, and mixtures of three adjacent fractions were prepared for use as test samples. Various texture features, and distribution based and ordinary statistical classifiers were used in analysis. Figure 12 shows examples of the training material and mixtures, imaged at 7 x 7 pm resolution. The relative diameter range of particles in each mixture was 1.7 which results in only a minor “autosieving” phenomenon. Gray level differences were found to be the best performing features with all classification schemes. Using the G metric and kNN classifier (Ic = 3), the error of the leave-one-out test for training samples was 6%, when the gray level difference histograms with displacement 2 were used. The classification of mixture samples was
4.2 Approaches to Texture-Based Classification . . . 733 manufacturing shoes, belts, furniture and other leather goods, hides were selected on the basis of their characteristics and were cut into pieces of various shapes using moulds in a manner that the pieces have the desired quality, taking into account acceptable minor defects. The defects can be categorized as area faults that are local variations of gray-level or texture, line defects that are often scars or folds of skin, and point faults that are groups of spots, whose gray-levels differ from the background. The dimensions of the smallest defects that should be detected are around 2 mm. A methodology for inspecting leather hides has been investigated by Wamback and his co-workers [49]who found that gray-level distributions for hides are symmetric even for the areas for defects, making plain histogram based detection schemes insufficient. They make a simplifying assumption that the gray values in the image are Gaussian distributed, and check whether the pixels in a 5 x 5 neighborhood are from the distribution determined for the good part of the hide using mean, variance and edginess tests. Because parts of the faults have the same characteristics as the defectless regions, the most deviating parts of the flaws are located first using stricter confidence intervals and requiring a certain number of detections in the 5 x 5 neighborhood to avoid overdetection. The reported difficulties with the methodology were mostly with very small spot and weak line faults. 5. Conclusion This chapter has presented some recent approaches t o texture analysis using a classification principle based on comparison of distributions of feature values. The choice of proper texture measures for classification or segmentation is extremely important. Very good performance has been obtained by using distributions of simple texture features, like gray level difference histograms or local binary patterns, in various texture classification and segmentation problems. The results suggest that the presented approach should be considered for many practical applications of texture analysis. Despite the progress in texture analysis methodology, the application of texture analysis to industrial problems is usually not easy. A major problem is that textures in the real world are often not uniform, due to changes in orientation, scale or other visual appearance. In addition, the degree of computational complexity of many of the proposed texture measures is very high. Before committing effort into selecting, developing and using texture techniques in a n application, it is necessary to thoroughly understand its requirements and characteristics.
References [l] L. Van Gool, P. Dewaele and A. Oosterlinck, Texture analysis anno 1983, Comput. Vision Graph. Image Process. 29 (1985) 336-357. [2] R. M. Haralick and L. G . Shapiro, Computer and Robot Vision, Vol. 1 (AddisonWesley, 1992).
734
M. Pietikainen, T. Ojala
6 0. Silven
[3] T. R. Reed and J. M. H. Du Buf, A review of recent texture segmentation and feature extraction techniques, CVGIP. Image Understanding 57 (1993) 359-372. [4] M. Tuceryan and A. K. Jain, Texture analysis, in Handbook of Pattern Recognition and Computer Vision, 2nd edn., C. H. Chen, L. F. Pau and P. S. P. Wang (eds.) (World Scientific Publishing Co., Singapore, 1997). [5] R. Chellappa, R. L. Kashyap and B. S. Manjunath, Model-based texture segmentation and classification, in Handbook of Pattern Recognition and Computer Vision, 2nd edn., C. H. Chen, L. F. Pau and P. S. P. Wang (eds.) (World Scientific Publishing Co., Singapore, 1997). [6] D. Harwood, T. Ojala, M. Pietikilnen, S. Kelman and L. S. Davis, Texture classification by center-symmetric auto-correlation, using Kullback discrimination of distributions, Pattern Recogn. Lett. 16 (1995) 1-10. [7] T. Ojala, M. Pietikainen and D. Harwood, A comparative study of texture measures with classification based on feature distributions, Pattern Recogn. 29 (1996) 51-59. [8] R. M. Haralick and L. G. Shapiro, Image segmentation techniques, Comput. Vision Graph. Image Process. 29 (1985) 100-132. (91 T. Ojala and M. Pietikilnen, Unsupervised texture segmentation using feature distributions, Report CAR-TR-837, Center for Automation Research, University of Maryland, 1996. [lo] T. S. Newman and A. K. Jain, A survey of automated visual inspection, Comput. Vision Image Understanding 61 (1995) 231-262. [ll] M. Pietikilnen and T. Ojala, Texture analysis in industrial applications, in Image Technology - Advances in Image Processing, Multimedia and Machine Vision, J. L. C. Sanz (ed.) (Springer-Verlag, Berlin, 1996) 337-359. (121 K. Y. Song, M. Petrou and J. Kittler, Texture defect detection: a review, SPIE Vol. 1708 Applications of Artificial Intelligence X : Machine Vision and Robotics, 1992, 99-106. [13] J. Weszka, C. Dyer and A. Rosenfeld, A comparative study of texture measures for terrain classification, IEEE Trans. Syst. Man Cybern. 6 (1976) 269-285. (141 J . M. H. Du Buf, M. Kardan and M. Spann, Texture feature performance for image segmentation, Pattern Recogn. 23 (1990) 291-309. [15] P. P. Ohanian and R. C. Dubes, Performance evaluation for four classes of textural features, Pattern Recogn. 25 (1992) 819-833. [16] W. Siedlecki and J. Sklansky, On automatic feature selection, Int. J . Pattern Recogn. Artif. Intell. 2 (1988) 197-220. [17] P. A. Devijver and J. Kittler, Pattern Recognition: A Statistical Approach (PrenticeHall, London, 1982). [18] K. Fukunaga, Statistical pattern recognition, in Handbook of Pattern Recognition and Computer Vision, 2nd edn., C. H. Chen, L. F. Pau and P. S. P. Wang (eds.) (World Scientific Publishing Co., Singapore, 1997). [19] Y.-H. Pao, Neural net computing for pattern recognition, in Handbook of Pattern Recognition and Computer Vision, 2nd edn., C. H. Chen, L. F. Pau and P. S. P. Wang (eds.) (World Scientific Publishing Co., Singapore, 1997). [20] A. L. Vickers and J. W. Modestino, A maximum likelihood approach to texture classification, IEEE Trans. Pattern Anal. Mach. Intell. 4 (1982) 61-68. [21] M. Unser, Sum and difference histograms for texture classification, IEEE Truns. Pattern Anal. Mach. Intell. 8 (1986) 118-125. [22] D. Harwood, M. Subbarao and L. S. Davis, Texture classification by local rank correlation, Comput. Vision Graph. Image Process. 32 (1985) 404-411.
4.2 Approaches to Texture-Based Classification . . . 735 [23] K. I. Laws, Textured image segmentation, Report 940, Image Processing Institute, Univ. of Southern California, 1980. [24] T. Ojala, M. Pietikiiinen and J. Nisula, Determining composition of grain mixtures by texture classification based on feature distributions, Int. J. Pattern Recogn. Artif. Intell. 10 (1996) 73-82. [25] H. Rautio, 0. Silven and T. Ojala, Grain size measurement using distribution classification, submitted to loth Scandinavian Conf. Image Analysis. [26] M. Pietikiiinen, T. Ojala, J. Nisula and J. Heikkinen, Experiments with two industrial problems using texture classification based on feature distributions, SPZE Vol. 2354 Intelligent Robots and Computer Vision XIII, 1994, 197-204. (271 J. Kontinen, J. Roning and R. M. MacKie, Texture features in classification of melanocytic samples, Computer Engineering Laboratory, University of Oulu, 1996. [28] M. Pietikiiinen, Z. Xu and T. Ojala, Rotation-invariant texture classification using feature distributions, submitted to 10th Scandinavian Conf. Image Analysis. [29] M. Pietikgnen, S. Nieminen, E. Marszalec and T. Ojala, Accurate color discrimination with classification based on feature distributions, in Proc. 13th Int. Conf. Pattern Recognition, Vol. 3, Vienna, Austria, 1996, 833-838. [30] L. Siew, R. Hodgson and E. Wood, Texture measures for carpet wear assessment, IEEE Raw. Pattern Anal. Mach. Intell. 10 (1988) 92-105. [31] L. Wang and D. C. He, Texture classification using texture spectrum, Pattern Recogn. 23 (1990) 905-910. [32] T. Ojala, Multichannel approach to texture description with feature distributions, Report CAR-TR-846, Center for Automation Research, University of Maryland, 1996. [33] M. Swain and D. Ballard, Color indexing, Int. J . Comput. Vision 7 (1991) 11-32. [34] R. R. Sokal and F. J. Rohlf, Introduction to Biostatistics (W. H. Freeman and Co, New York, 1987). [35] B. Kjell, Determining composition of grain mixtures using texture energy operators, SPZE Vol. 1825 Intelligent Robots and Computer Vision X I , 1992, 395-400. [36] P. Brodatz, Textures: A Photographic Album for Artists and Designers (Dover Publications, New York, 1966). [37] A. K. Jain and K. Karu, Learning texture discrimination masks, IEEE Bans. Pattern Anal. Mach. Intell. 18 (1996) 195-205. [38] D. K. Panjwani and G. Healey, Markov random field models for unsupervised segmentation of textured color images, IEEE Bans. Pattern Anal. Mach. Intell. 17 (1995) 93%954. [39] I. Dinstein, A. Fong, L. Ni and K. Wong, Fast discrimination between homogeneous and textured regions, in Proc. 7th Int. Conf. Pattern Recogn., Montreal, Canada, 1984, 361-363. [40] P. Dewaele, L. Van Gool, P. Wambacq and A. Oosterlinck, Texture inspection with self-adaptive convolution filters, in Proc. 9th Int. Conf. Pattern Recognition, Rome, Italy, 1988, 56-60. [41] D. Chetverikov, Texture imperfections, Pattern Recogn. Lett. 6 (1987) 45-50. [42] B. K. Ersball and K. Conradsen, Automated grading of wood slabs: The development of a prototype system, Industrial Metrology 2 (1992) 317-342. [43] F. Ade, N. Lins and M. Unser, Comparison of various filter sets for defect detection in textiles, in Proc. 7th Int. Conf. Pattern Recognition, Montreal, Canada, 1984, 428-431. [44] C. Neubauer, Segmentation of defects in textile fabric, in Proc. 11th Int. Conf. Pattern Recognition, Vol. 1, The Hague, The Netherlands, 1992, 688-691.
736 M. Pietikainen, T. Ojala & 0. Silven [45] L. A. Gerhardt, R. P. Kraft, P. D. Hill and S. Neti, Automated inspection of sandpaper products and processing using image processing, SPZE Vol. 1197 Automated Inspection and High-speed Vision Architectures IZI, 1989, 191-201. [46] V. Brecher, New techniques for patterned wafer inspection based on a model of human preattentive vision, SPIE Vol. 1708 Applications of Artificial Intelligence X , 1992, 452-459. (471 K. Y. Song, M. Petrou and J. Kittler, Texture crack detection, Mach. Vision Appl. 8 (1995) 63-76. [48] 0 .Silv6n and H. Kauppinen, Recent developments in wood inspection, Znt. J. Pattern Recogn. Artif. Zntell. 10 (1996) 83-95. [49] P. Wamback, M. Mahy, G. Noppen and A. Oosterlinck, Visual inspection in the leather industry, in Proc. ZAPR Workshop on Computer Vision, Tokyo, Japan, 1988, 153-156.
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 737-764 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
1 CHAPTER 4.3 I CONTEXT RELATED ISSUES IN IMAGE UNDERSTANDING
L. F. PAU Ericsson, PO Box 1505, Sl.2585 ALVSJO Sweden This chapter gives a formal model for scene understanding, as well as for context information; it helps in adapting image understanding procedures and software to varying contexts, when some formal assumptions are satisfied. We have defined and formalized context separation and context adaptation, which are essential for many applications, including to achieve the robustness of the understanding results in changing sensing environments. This model uses constraint logic programming and specialized models for the various interactions between the objects in the scene and the context. A comparison is made with, and examples are given of, the context models in more classical frameworks such as multilevel understanding structures, object based design in scenes, knowledge based approach, and perceptual context separation. Keywords: Image understanding, computer vision, context models, scene models, constrained logic programming, object oriented design, prolog, object recognition, noise filtering.
1. Introduction 1.1. Contezt Definitions Biophysics as well as perceptual studies, and also image understanding/ computer vision research [7,46]continuously stumble over the issue of context adaptation and separation: “Given a well defined image understanding task pertaining to objects or dynamic events in a scene, how do we render the formalization and implementation of this task independent from the scene and its varying parameters?” Context adaptation means the ability t o redesign an image understanding task and software by removing the context dependent information about one known context, and updating it with similar information for another known context. Context separation means the ability t o design an image understanding task not knowing the context and calibration information (e.g. when mapping 2-D information into 3-D information [27,36]). If, exceptionally, only a finite number of contexts are assumed possible, and if context separation applies, then context recognition is the task of identifying in the specific scene the applicable context selected from that finite list. 737
738
L. F. Pau
If, furthermore, context adaptation applies, then image understanding redesign can be carried out for the recognized context. In general, however, the number of possible contexts is infinite, even if scene, sensor, object and geometric calibration models are applied. The fundamental reasons for this are phenomena, geometric projections, or processes which violate the underlying required context separation: 0
0
0
Object-object interactions with direct effects on the context, e.g. geometric occlusion. Object-context and context-object interactions, each obeying causality relations, e.g. shadows from objects onto the environment or, vice versa, from the environment onto the objects in the scene; this becomes even more complex when non visible phenomena are taken into account, such as induced irradiation. Context unstationarity, due to random or slowly changing events, e.g. weather or failure in the lighting systems. Four basic approaches to context adaptation have been taken so far:
0
0 0
multilevel image understanding structures object oriented design in scenes knowledge based approach perceptual context separation
as sometimes exemplified in defect recognition in machine vision, obstacle avoidance modeling in robot navigation [17], aerial imagery [15,18], and a diversity of other areas.
1.2. Multilevel Image Understanding Structures The ability to represent information extracted from image data in a multilevel knowledge structure facilitates the hierarchical analysis needed for the image understanding (object detection, location, and completion of the understanding task at hand). In the now classical approaches, intermediate-level image processing operators (typically region and shape related) invoke lower-level operators (typically registration, feature extractors and measurement), which are then passed to higherlevel operators to derive complex relations between objects and their task related meanings.
1.3. Object Oriented Design in Scenes More recently, in relation to implementations of the previous multilevel understanding structures, object-oriented design has attempted to group similar low-level features or middle-level elements (such as regions or neighborhoods), while also separating those which are context related, into classes corresponding immediately to a hierarchical representation. In addition, the programming environments selected for the software solutions provide polymorphism, inheritance and encapsulation. This has been easing the updating of the instances and operators (methods) when
4.3 Context Related Issues an Image Understanding 739 changing contexts and tasks or objects. Method and object inheritance [13,16] offer a code-sharing alternative to supplying special-purpose operators for handling specific classes of objects.
1.4. Knowledge Based Approach Work has also taken place to break the hierarchical understanding structure by having a fully-fledged knowledge based system reason about all allowable combinations of low-level, intermediate level, and higher-level concepts or objects, but by introducing different depths into the selection and search according to the nature and ontology of these concepts or objects (e.g. [9]).Fundamentally, the image understanding task has become a goal, and backward inference is carried out to search for evidence along each possible explanation path. There was in this type of work an implicit hope, now largely lost, that the knowledge base itself could be segmented into context related and context independent pieces of knowledge; for example, it was hoped that generic rules would apply to, e.g. illumination, object centered geometrical transformations, clusters of physically related objects, etc.
1.5. Perceptual Context Separation Perception and psychophysics research sometimes suggest that image understanding tasks can almost always be carried out by the human, with the exception of illusions, thanks to the filtering out of perceptual cues about the task irrespective of the context, thus carrying out at once both context separation and adaptation. For example, car driving in day and night time rely on a perceptual image flow and object distance cues which are analyzed equally well in both contexts. However, no one yet knows how to implement and generalize these perceptual cues or groupings (also called collated features), apart from some simple scenes or processes. Some considerations have gone into using neural or Hopfield networks to coalesce and discriminate perceptual groupings such as edges and gaps between lines, line intersections, parallels, U’s, rectangles, etc. The hope here was to be able to add or remove them according to the geometry of the problem and of the context in general. In the neural networks, after training of the understanding task, weights are trained for the links in a global competition between collated features. 1.6. Review In the best case, image understanding work has focussed on the representation and control issues [24,30,35,42], such as those related to semantic network representations and their execution on distributed architectures; some work has focussed on the opposite of context modeling, that is, designing universal applications development environments able to cope with all kinds of specifics. As a result, context modeling has been largely ignored. When the context was not or was insufficiently known, the hypotheses were simply ranked by experimental tolerances or assigned likelihoods.
740
L. F. Pau
Experience proves that none of the first three approaches above, or combinations hereof, can deliver context adaptation and separation, except for very simple problems and scenes. The multilevel image understanding structures fundamentally cannot allow for the lower-level feature parametrization and detection in changing contexts, even if mixed forward-backward reasoning is applied between levels. And the more cluttered the environment, the more numerous the interactions between levels. The object oriented design in scenes is useful in highly structured environments, and especially for those understanding tasks which concern themselves with just a few objects with few interactions with the context (e.g. shadows, reflections, changes in reflection of). But that design is even more hierarchical, and thus more rigid, than the previous approach, and suffers from all the drawbacks hereof as well. The knowledge based approach suffers from the well known knowledge elicitation and accumulation problems; it is probably a never-ending process to acquire heuristic information or models to cover all possible separate context relations and context related processes. Perceptual context separation in humans and animals seems very powerful, especially as it achieves robustness versus perceptual deficiencies and anomalies. It is certainly a research goal of high importance t o be able t o formalize it, yet little is yet understood of the implementation sort for complex tasks. In [51] is analyzed contextual reasoning from text, which has parallels with the reasoning about scenes.
1.7. Plan In this chapter, we will formalize the context adaptation and separation problems, in both theoretical and practical ways, which have been shown to help out significantly, although not yet with resolution of the full range of context related issues. The formal models for the scene and context, and especially for their interactions, are given in Sec. 2. Sections 3, 4 and 5 illustrate those models, stressing especially the context modeling, via one example from car recognition in traffic images, by taking successively the multilevel understanding approach, the object oriented design of the same, and finally by addressing some perceptual context cues. Conclusions are given in Sec. 6 , while two appendices give introductory definitions or explanations about object oriented design and constraint logic programming.
2. Formal Image Understanding and Context Description 2.1. Approach The basic approach proposed here is to: (i) provide a formal description of the scene images via logic (ii) assume massive parallelism in both the spatial description as well as in the processing/understanding
4 . 3 Context Related Issues an Image Understanding 741 (iii) describe each context as a set of logic predicates and constraints propagated through the formal description (i) (iv) model the basic object +) context interactions. The need for the formal description is due to the context separation requirement; the need for the massive parallelism is due t o the implementation requirement; while the need for the context model description and interaction modeling, is due to the context adaptation requirement. It should be highlighted right away that image context simulation from physical processes has made significant progress, as evidenced by the flurry of image synthesis applications, and that they all contribute to the content of context modeling via call in-call out facilities to a battery of physical or other behavioral models. In the following the notation in logic shall be the one from the Prolog language [Ill,although the image understanding task solution implementation may very well be done later on in other languages. The notation a + b is a predicate saying that a is true iff b is true; the notation A t B is a rewriting rule saying that the list B is syntactically rewritten as A. The arity of a predicate is the number of arguments it has. One fundamental remark is also that, although difficult t o achieve, the goal of these scene and context models is t o help in image understanding tasks even in unstructured environments; this leads to the use of rather general context information data structures, i.e. the causal graphs and influencing domains (as defined below). This of course would be impossible unless the physical and causal image formation processes are not taken into account, and therefore we have t o assume known the range of such processes existing in a given scene. The other assumption is about the fact that the image understanding task relates to objects which are significant in terms of their overall presence in the scene, as e.g. measured by the total solid angle of these objects from the point of view of the sensor viewing the scene. 2.2. Scene Model
Assuming in general a four-dimensional space (z, y, z , t ) each sensed pixel gray value/color code “pixel (z, y, z , t)” is true iff its value is true, which in logic corresponds to the fact/statement : pixel(z, y, z , t ) . To each location (z, y, z , t ) is attached a causal graph G of all other locations having an influence on its pixel code value/color. This dependency is explicitly shown by increasing the arity of the “pixel” predicate:
The graph G is built from the causal influences mapped out pairwise on an influencing domain D (2, y, z , t , p ) : influenced (z, y, z , t , x’,y’, z’, t’) .
742
L. F. Pau
and this predicate is true iff (d, y’, z’, t‘) belongs to D ( s , y , z , t , p ) , where p is a causal process type. The influence “influenced” can in general not be related to a single process p , as the paths in the graph G ( z , y, z , t ) leading to the location (2,y, z , t ) may travel through a sequence of pixels each influenced by the previous pixel, but in different ways. 2.3. Causal Processes p
The range of causal processes { p } is not bound, as they may be physical, relational, geometric, qualitative, behavioral [50], or model-based. It is here assumed that the same range of causal processes apply t o the context related information. A “default” set of causal processes t o be considered are: { p } = {lighting, sensor, optics, shadows, orthogonal-projection}
.
In [48] is given an example of such process models in a simple case of context separation, irrespective of the image understanding task relating to objects in the scene. 2.4. Image Understanding Task
The image understanding task [33,35]is then a goal I to be satisfied in the scene in view of a finite number n of logical conditions applying to sets of pixels in the scene: I + cond-1 (pixel (.)), cond-2 (pixel (.)), . . . , cond-n (pixel (.)) or equivalently via the composite condition (applying for example to a composite region [29,33]): I + cond (pixel (.)) .
The n logical conditions cond-1, . . ., cond-n are here treated as constraints in a constraint logic programming framework (see Appendix B). The understanding process itself is then the search process S (set-of (pixel), setof (G), set-of ( D ) , set-of ( p ) ) needed to establish the previous goal I as true or false. The predicate “set-of” is self-explanatory. For reasons of clarity we assume here that S is the sequential ordered list of all nodes traversed in the total image, although of course a fundamental assumption made here is that a massively parallel architecture is used and reflected by a propagation scheme in this architecture. In an earlier work [21], the satisfaction of the conditions cond (pixel (.)) was defined as a truth maintenance problem [31,32],in view of sensor fusion and of the disambiguation of scene contexts in a threcdimensional fusion task.
2.5. Context Model The context is another massively parallel field “context-pixel (2, y, z , t)”with the corresponding causal graphs “Context-G (x,y, z , t)” and influencing domains “Context-D (z,y, z , t , p ) ” .
4 . 3 Context Related Issues in Image Understanding 743 We can then formalize the basic assumptions and definitions: (i) there is context separation iff the following implication holds true:
{I=> cond (pixel (. , . ,. , Context-G)), for all Context-G} which means that all constraints kond-i (pixel (.))” are independent of all “Context-G’ for the range of processes { p } . (ii) context adaptation, assuming context separation, can be carried out by the following rewriting process: S(set-of (pixel), set-of (Context-G), set-of (D), set-of ( p ) ) t S(set-of (pixel’), set-of (Context-G’), set-of (D’), set-of ( p ) ) where the primed symbols pertain to the same problem/goal but in an old context. 2.6. Interaction Models: Object- Object
Defining an object is done easily by defining the LLinfluenced)) predicate for influencing domains D (. , . ,., object-name) covering the spatial and time extent of this object. Context objects are defined equivalently. The influence of an object on a context object, or vice versa, is a predicate and rewriting rule, which in the most general form is in two parts: new-D (. , . , . , new-object-name)
+ intersect (D (. , . , ., object-name), Context-D (. , . , . , context-object-name)).
new-object-name
t (object-name
context-object-name)
where the “intersect” predicate says whether the two sets indeed intersect. The last rewriting rule is a possible, but not compulsory, object relabeling. Example: Intersections of two objects
We find the intersections between objects A and B ; these intersections divide the boundary of each object into contour segments. The contour segments of each object are then assigned to one of three disjoint sets, one containing segments that lie outside the other object, one containing segments that lie inside the other object, and one containing segments shared by the two objects. The relations between various collated features are represented in the context-graph G, which is labeled as a causal graph, in such a way that collated features which support each other perceptually are connected via positively weighted links, while mutually conflicting collations are linked via negatively weighted links. The following cases exist, each modeled by specific predicates, rewriting rules, and attribute changes eventually described by an attributed grammar [22]; the
744 L. F. Pau
definitions below apply to any pair of objects, but we are especially interested by the case where A is a real object and B a context-object: (i) Subsumption. If the outside-segment set of a shape A is empty, and the shared-segment set non-empty, and the edge support for segments in the insidesegment set is non-existent, then we say that object A is subsumed by object B , and can be removed. (ii) Occlusion. If the contour segments of A inside B have strong edge support, and those of B inside A have weak intensity edge support, then A occludes B . This applies even if the rest of the contour segments of A and B belong to the shared set or outside set. (iii) Merger compatibility. If the segments in the inside-segment and shared-segment set for both objects A and B have poor edge support, then A and B represent segmentation of one object into two parts, and can thus be merged into one object. (iv) Disconnected. If A and B have null inside-segment sets and null sharedsegment sets, they are disconnected. If A and B have a non-empty sharedsegment set, and null inside-segment sets, but the shared segments have good edge support, then A and B are still unrelated though adjoining. (v) Incompatible. If A and B have non-empty inside-segment sets and the elements of the inside-segments of both A and B have strong edge support, then at least one of A and B is a wrong structural grouping and must be deleted.
2.7. Interaction Models: Object
+)
Contezt
The influence between an object and the context is a predicate and rewriting rule, which in the most general form is in two parts: pixel (. , . , . , new-G) t (pixel (. , . , . ,G) pixel (. , . , . , Context-G)) new-D (. ,. , . , object-name) + intersect (D (. , . , . , object-name), Context-D (. , . , . , object-name)). which shows that the pixel value or code may be changed because of the change in the influenced domains.
2.8. Interaction Models: Contezt Unstationarity This unstationarity is of course first achieved by the stochastic point processes linked to the location (x,y, z , t ) and thus to the domains G and D. The latter are of course the most interesting due to combining stochastic deviations made of spatial stationarity and temporal stationarity, to modify the influences. In practice, it is indeed very difficult to have or estimate the characteristics of these point processes, and thus to compensate for them in the search processes S. The simpler case is when (x,y, z ) is deterministic but where t is an independent variable driving the scene, task and context.
4.3
Context Related Issues in Image Understanding
745
2.9. Context Causal Graph G Operations The context causal graph Context-G (.) can be manipulated by standard predicates operating on that causal graph and its attributes. It can for example be built (see [25,38] for complete predicate definitions) using: (i) causal graph merger, and adjacencies (ii) coalescence by graph join operations corresponding to overlaps between image scene contexts, with respect t o an angle of view and perspective transformations (iii) perception graph for the context, resulting from joining all context graphs for context-objects (iv) extensions to sensor fusion tasks [21,49].
2.10. Constraint Based Languages as Resolution Strategies Once the image understanding task has received a formal description as above, the big question is of course how to synthesize the search processes S. Here is where the impact of a new research field is felt the most, that is, of constraint based logic programming (see Appendix B for a n introduction). These languages do exist and are in use in the industrial world under trade-names such as: Prolog 111, CHIP, CHARME, PRINCE, etc. [l-61. They include constraint domains (as formalized via the constraints cond(pixe1 (.))) which can be both finite or infinite trees [5,14], linear algebras with infinite precision or rational numbers, boolean algebras, and lists. They also allow for domains such as finite domains (as related to the objects or the influencing domains D),and interval arithmetic (for pixel value rewriting rules such as most gray value “mixing” operations or thresholding). These languages also include the constraint solving algorithms right into their kernels (see Appendix B), while maintaining the declarative nature of the goal I and of all the predicates “influenced”. Most of these languages have pre-compilers or compilers, which is most appreciated in applications development; the search strategies S may be synthesized interactively in the interpreted mode, or compiled [28], with all the underlying constraint propagation carried or by the constraint solving algorithms. One area still unexplored is their implementation on massively parallel storage and processing architectures, although Digital Equipment is collaborating with some research partners on this subject.
2.11. Implementation of the Context Adaptation and Separation This implementation follows from the logic formalism and problem specification described above:
If there is no context separation over { p } , the assertion of the context separation definition will be false. This “fail” can in turn be used t o authorize or deny further rewritings which assume this separation, via the standard “/” predicate, or via the delayed “dif” predicate [ll].If the “fail” happens, then the developer
746
0
L. F. Pau
has the option to change the vocabularies in { p } and change the range and types of processes. If there is context adaptation and this can be carried out, the simple rewriting rule of the context adaptation definition applies. This can be further eased by separating out all predicates and data structures for each Context-G and Context-D in separate “worlds” or “modules” of the asserted predicate knowledge base. This segmentation is precious for context adaptation and modularity. Incidentally, Prolog allows us to write very simply regular grammars and others to implement the rule rewriting.
It should also be noted, and this is very important in practice, that consistency of all “influenced” predicates is maintained, as dynamic updates in the “worlds” or “modules” will check out possible tautologies/contradictions and deny any if happening. 2.12. Time Dependencies in the Context
It is worth underlining once more that the context graph Context-G ( 5 , y, z , t ) and the influencing domain Context-D (x,y, z , t , p , context-object-name) are both time dependent. This is mandatory as the context-objects move, and also as the causal graphs G change over time. In the threat assessment [20], scene monitoring [34], or target tracking problems [20,37], there is an allowed domain attached to the transitions over time between spatial zones occupied by the objects in the scene, corresponding to constraint “scripts”. 2.13. Comparison
This model is much more formal and powerful than the approaches surveyed in Sec. 1, except perceptual context separation. It opens one way to the latter by having ‘(retinas” specified via the influencing domains D, and matching vision processes (see Sec. 5). In Sec. 3, an example will show how easily the multilevel image understanding structures can be represented, with knowledge bases also, and Sec. 4 will show how object oriented design can be incorporated if needed. 3. Multilevel Image Context Representation in a Logic Programming Environment This section describes in a simple case how a simple context information model can be combined with multilevel image understanding, as discussed in Sec. 1, to carry out a simple recognition task. The task I is to recognize car objects in a real-life scene (see Fig. 9). Extensions have been made to three-dimensional scenes in [21], with a full example therein.
4 . 3 Context Related Issues an Image Understanding 747 All image processing predicates mentioned below are available as Prolog [ll] predicates in an environment described in [8,38], and organized into a three-level hierarchy summarized in Table 1. The car objects are defined by clauses too, with orientation as a parameter (see Fig. 1). It should be noted that all predicates affected by the context are found in the upper Context level layer, and only there. About the implementation of the search process S, using the Prolog unification algorithms, see [2,8] for extensive details.
Table 1. Hierarchy of Principal Image Operators; all listed here are predicates which all allow for unification/backtracking.Parameters are explained in Sections 3 and 4.
I
CONTEXT car (Img, Theta, ObjLlrt) I* ObjLlrt form8 a car In Img at < Theta *I LEVEL carSlde (Wl, W2, Theta) I* W1, W2 have 8Imllar orlent< Theta ' I carcorner (Wl, W2) I* W1, W2 are roughly perpendlcular *I carWlndow (Img, ObjNum, W) I* car wlndow W ha8 plxvaliObjNum ' I
APPLIC. LEVEL
trapezold (Img, ObjNum, T) parallelogram (Img, ObjNum, P) rectangle (Img, ObjNum, R) qu8drllateral (Imp, ObjNum, 0 ) orderllder (W, Edger) a8pectRatlo (W, Ratlo) orlentatlon (W, Or)
1'
Edger=Top,Bottom,L,R 8Id.8 of W ' 1 I* Ratlo la Perlm**2 I Area *I I* bared on varlanco ratlos .I
blnarlze (Src, Doat, Threah) lop08 (Src, Dert, Iter) FEATURE traceContour (Src, Deat, Obj, DIR) I* follows edge In DIR dlrectlon*l (LOW) clo8edPolygon (Cornera) LEVEL IocateCorners (Img, ObjNum, Corners, [SCAN, Thresh, COUNT]) I
car( Img, Orientation, [Wl, w2, W31) :carWindow ( I m g , PixVall, Wl), nextTo( I m g , W1, W2). carwindow ( I m g , PixVal2, W2), carside( W1, W2, Orientation), nextTo( I m g , W1, W3), carwindow ( I m g , PixVal3, W3), carcorner( W1, W3).
Fig. 1. Example clause within 'car' predicate, showing one allowable transformation.
3.1. Lower Level Predicates: Image Features
A common characteristic of the predicates at this level is that each is task ( I ) and context (G) independent: edges, vertices, geometric features, etc. Predicates
748
L. F. Pau
such as binary threshold and low-pass filter predicates preprocess the original image pixel gray level values. It is however object registration, scaling, labeling, contour tracking, and corner detection, which are the principal means for extracting information at this level. The attributes derived include, for each labeled object/region, its area, perimeter, centroid, x- and y- variances, chain-coded contour, number and locations of corners. Descriptions of the labeling and contour tracking algorithms are provided in [S]. For example, the general form of the corner location predicate could be [S]: locateCorners (Src, Dest, ObjNum, Corners, [SCAN, Thres, COUNT]) in which Src, Dest are source and destination atoms, ObjNum is the object label number/name, Corners is the corner pixel location pointer pixel (x,y, z , t ) ,and Thres is the gray level threshold for ObjNum. SCAN and COUNT are parameters which can be bound, unbound, or constrained. According to the way these bindings are specified at query time by constraints LLcond(.)l’, then one can easily get declaratively, answers to questions such as (see Fig. 2): 0
0
How many corners COUNT can be found using a scan window length of SCAN = 22 pixels? What scan length SCAN should be used to find exactly COUNT = 4 corners?
and any combinations of similar questions, including on other atoms. 1ocateCorners (Src, Dest, ObjNum, Corners, [SCAN, Thresh, LINECOUNT]) . (a) locatecorners (Src, Dest, ObjNum, Corners, [12, 0.75, LINECOUNT]) . LINECOUNT = 5 (b) 1ocateCorners (Src, Dest, ObjNum, Corners, [SCAN, 0.75, 4 1 ) . SCAN = 12 ; SCAN 16 : SCAN = 20
-
(C) 1ocateCorners (Src, Dest, ObjNum,
Corners,
[SCAN, 0.75, LINECOUNT])
-.
SCAN = 6, LINECOUNT 5 ; SCAN = 8 , LINECOUNT = 5 ; SCAN = 12, LINECOUNT = 4 : SCAN = 24, LINECOUNT = 3
Fig. 2. Three predicate calls to illustrate effects of parameter bindings.
The search process yielding answers to these questions is based on unification and backtracking, and explained in [8,10,38]. Essentially, unification can, not only trigger a search for a suitable parameter value satisfying the goal constraints, but
4.3 Context Related Issues in Image Understanding
749
ot-car-models-found
/
I car-model(lmg,Y,V) I
I
u tlgurs
5
Fig. 3. Class allocation control structure.
can also match the unbound parameter against all possible values which satisfy the predicate’s constraints. 3.2. Intermediate Level Predicates: Application Dependent Predicate
Knowledge Base The predicates defined at this level are all I-application specific and are employed to identify car windows as objects belonging to a particular class of shapes (trapezoid, rectangle, parallelogram), and with shape attributes (e.g. aspect ratio) conforming to specified ranges of values for cars. In addition, allowable transformations are defined, to permit the classification of shapes with incomplete contours. Spatial relationships between pairs of objects in the cars are also specified here; thus we see at this level, predicates for identifying adjacent objects in the cars, objects with the same orientation, etc. The same predicates can exclude objects which do not belong to a car although belonging to the same class of shapes. These predicates altogether constitute what might be called a formal specification of the object model from regions [40], the objects being cars and car elements. They should be stored in a separate predicate base, or world; the use of such worlds is mandatory in sensor fusion tasks [49]. In [8,38] examples are given in detail as to how the search by unification/ backtracking allow detection of the shapes of the object model, if any exist, and to adapt the parameters as explained above for the lower level predicates: 0 0
list of edges for a shape [8]; region attributes for context dependent regions, producing attributes of the influencing domains Context-D.
750
L. F. Pau
class polygon (Image, ObjNum) checks ( get-cornerData (Image, ObjNum, CornerList), closedPolygon (CornerList) ) body img (Image) => ( ! ) . quad (Image, ObjNum) class polygon checks ( get-cornerNum( Image, ObjNum, 4 ) , orderEdges ( Image, ObjNum, Edges), assert-once ( quadrilateral(Image, ObjNum, Edges)) ) body edges( [T, B, L, RI) => clause (quadrilateral(Image, ObjNum, [TI B, L, Rl ) ) -&-
topBottomTheta ( Theta) => clause (quadrilateral(Image, ObjNum, [TIB I-] transorigin( T, B, Tnew, Bnew), rotateToXaxis ( Tnew, Bnew, Theta) .
,
creation new(quad( image01, 141, QuadOl)
QuadOl#topBottomTheta( ThetaQOl
)
Fig. 4. Example class and subclass definitions, showing object creation and message passing. (Body’ predicates (methods) are inherited to subclasses. ‘Checks’ are also inherited, meaning that the call shown to create an instance of ‘quad’ will first evaluate the checks in ‘polygon’ and then in ‘quad’ before completing the instantiation.
3.3. Higher Level Predicates: Goal Satisfaction Constraints
At this level, coexisting in a multilevel representation are two very different types of predicates: a
0
the conditions “cond-i” which are constraints defining the image understanding task I all context related predicates, such as “influenced”, “Context-G” , “Context-D” , etc.
As to the constraints, they essentially specify in the specific case at hand, that an object may be classified as a car of a given car modelftype, if all component car windows are spatially dispersed in a certain way, specified for example by the
4.3 Context Related Issues in Image Understanding 751 constraints of the following car detection goal:
I
+ cond(Img(.)).
I = car (Img, Orientation, [WI, WZ, W3]) cond-1 (Img(.)) = carwindow (Img, PixVall, W l ) , nextTo(Img, W1, W2). cond-2 (Img (.)) = carwindow (Img, PixVal2, W2), carside (Wl, W2, Orientation), nextTo (Img, W1, W3). cond-3 (Img (.)) = carwindow (Img, PixVal3, W3), carcorner (Wl, W2). Unification again generates the search process S, satisfying the goal I , as well as all constraints on the objects or regions with specific characteristics, acting like special-purpose filters. Other goals are then satisfied by other sets of constraints: for example, the initially mentioned sequential car recognition goal will be stated as (again allowing for identical predicates with variable arity):
I + cond(Img(.)). I = list ( Y1, match (Img, Y1 , YZ), List-car-types). cond-1 (Img (.)) = different (Yl, Y2), car-model(Img, Y1, V l ) , car-model (Img, Y2, VZ), element-of (Yl, list-car-models), element-of (Y2, list-car-models). cond-2 (Img (.)) = car-model (Img, Y1, Vl). cond-3 (Img (.)) = car-model (Img, Y2, V2). cond-4 (Img (.)) = equal (Vl, [Thetal, Cornersl, SCAN, Threshl, COUNTl]). cond-5 (Img (.)) = equal (V2, [ThetaZ, Corners2, SCAN, Thresh2, COUNT21). In the above, the search process S will return the list of car models seen in the image “Img” , by ensuring that these car instances are spatially distinct, meaning that there is yet no object +) object interaction. The car-model predicate will first unify the unbound V parameter lists before it can backtrack with one or several solutions. The predefined “different” predicate should guarantee the difference at the term level, and spatial distinction. Figure 5 illustrates the class allocation constraints nesting and hierarchy, and is made graphical to help in object oriented design. We have however to analyze this predicate “different” in more detail below. 3.4. Context Model
The above goal satisfaction constraints are still not fully specified, precisely because the “different” predicate referred to above is obviously context dependent, as it pertains to spatial distinction and non-occlusion, which falls into the class of object +) object interaction models discussed in Sec. 2. More precisely, the context separation in a multi-object recognition task can only be achieved by: first a scene model for isolated single objects alone next a context model for each additional object
752
L. F. Pau
2735 car(lmg,Theta.ObJLlat)
I
1
arMndow(lmg,PlxVall,Wl)
carWlndow(lmg,PlxVaI2,WZ)
carSMe(Wl.W2,Theia)
\
\
trapezold(lmg,PlxVai2,W2) \
I
corder sides,
locataCornen(i~.PixVai2,Comers,[SCAN,THRESH,4])
\ lncaContour(lmg,PlxValZ,OlRECTlON,Ch~lnCode
Fig. 5 . Example. Invoking the ‘car’ predicate sequentially activates its subgoals. When the first call to ‘carwindow’ succeeds, W1 is bound t o an object in ‘img’ with pixel assignment PixVall. The subgoal ‘nextTo’ succeeds in finding a neighbor object, unifying it with W2. Now suppose that the second call to ‘carwindow’ succeeds (visiting the nodes shown in the expansion), but ‘carside’ fails. Backtracking returns to the second ‘carwindow’ call, but since W2 is still bound, the search does not look for a new object. Instead, ‘1ocateCorners’ is invoked via backtracking to analyze the object with new scan parameters, to derive a new corner placement which may satisfy the ‘carside’ constraints. A new object would be searched for if backtracking returned t o ‘nextTo’, freeing W2 to be unified with a new neighbor object.
so that each isolated car is located and recognized individually, and overlapping cars are treated as an object +) object interaction as formalized in Sec. 2. Reusing the goal specification for the car recognition task, and assuming the scene in which the isolated car must be located and recognized, is car-model (Img, Y1, Vl). The context model for this object +) object interaction perfectly fits the
4.3 Context Related Issues an Image Understanding 753 generic definition: new-D (. , . , . , new-object-name)
* intersect (D (. , ., . , object-name), Context-D (. , ., . , context-object-name)).
new-object-name t (object-name
context-object-name)
provided the application level specifies the predicates or definitions: object-name = car-model (Img, Y1, V1) context-object-name = car-model (Img, Y2, V2) different (Yl, Y2) + or (diff (new-object-name, Yl), diff (new-object-name, Y2)). In case of spatial overlap, the rewriting rule must be defined otherwise, e.g. in relation to the attributes of the domains D-1 and 0 - 2 and thus of the domain “newD” (see the formal definitions above leading to the corresponding formal predicates). In this car recognition example, the influencing domains D-1, 0-2, new-D are explicitly described by the simple causal process of spatial, stationary, neighborhood, as explicated by the predicates “nextTo” in relation to the car windows. In this example too, it is especially powerful to use the causal graphs G(.) and Context-G (.) to represent all possible relative attitudes of the “car-windows” and “car-corners” with respect to the sensor. For example, it is obvious that the car detection goal “car (Img, Orientation, [Wl, W2, W31)” made explicit above, corresponds to one such causal graph. In this specific example, as discussed in [8],there is no object t)context interaction, nor context unstationarity: 0
The object t)context interaction would apply both to shades cast by the isolated cars, as well as to reflections between cars and the physical surroundings. However the causal graphs G and Context-G allow for the filtering out of the shade seen in Fig. 9 (once compared to Fig. 8), and no further object t)context interaction model is needed. The context unstationarity would be capturing the randomness in the car speeds, but not speed (as pixel (. , . , . , t ) encapsulates time).
3 . 5 . Constraint Resolution Engine for Object Classification
We can illustrate, in relation with the example treated in this section, how a constraint satisfaction engine would operate for the car classification goal I. We will further particularize this procedure, borrowed from [8],by jointly treating scene objects such as cars and context objects such as trees or road signs, assuming context separation. Consider ‘ ‘ ~ b jand ” “c-obj” as facts belonging to the lists “mi’ and “c-n” respectively, where the prefix c- applies to context information, and “a” is the context. The fact ‘Y as observed in the scene with its context will be resolved by the following
754
L. F. Pau
predicate base which constitutes a constraint resolution engine, for the constraint that the object “obj” is found in that context “a”: (i) T + obj c-obj. (ii) class (r, a.1) + obj c-obj class (obj, m) class (c-obj, c-n) element-of (a, m) element-of (a, c-n). (iii) class(r, nil) + fail. (iv) delete(r, obj c-obj) + class(r, nil). (v) add-predicate (no (r), obj c-obj) +. where
0
0
the first rule (i) belongs to the goal specification in the predicate base; the other rules (ii)-(v) belong to the constraint satisfaction engine by deleting inconsistent facts in the predicate base and adding conditions by (iv) and (v) to get smaller consistent fact lists; the notation “a.l” designates the list made of “a” followed by “l”, “nil” the null fact; the other predicates are self-explanatory.
4. Image Understanding and Object Oriented Context Modeling
This section deals with the issue of showing how object oriented design can be used for context modeling, and the limitations hereof (see Sec. 1). To ease the reading, this is exemplified by extending the example of Sec. 3. Basic definitions of object oriented design terminology are given in Appendix A. 4.1. Object Oriented Context Representation
Even when using a logic based representation, as in Secs. 2 and 3, it is possible to extend the predicate representations by class definition capabilities, including mechanisms for inheritance and methods inheritance. A class definition is composed of two groups of predicates: 0
0
The group labeled “checks” is evaluated when an instance is requested via the predicate “new ((class name))”; if these predicates are satisfied, the instantiation is successful; The group labeled “body” contains the predicates available as methods. Both groups are inherited to subclasses (see Fig. 4).
The context classes then correspond to classes of “context-object”-names, the classes of influencing domains “Context-D”, and the classes of causal processes { p } , as featured via the “Context-D (, , , context-object-name)” term in the formal model of Sec. 2. A more specific example of this, but by no means the only one, is the class of context regions shapes, found in a shape library as used in [9].
4 . 3 Context Related Issues in Image Understanding 755 Method definitions allow a segment of Prolog code to be encapsulated within a class definition and invoked by sending a message to the class instance. Inheritance allows methods defined in a class to be available (inherited) to any of its subclasses. Figure 3 illustrates definitions for a class (e.g. %ar models”) and subclass, as well as the syntax for object creation and message passing; “body” and “checks” of the corresponding class definitions are illustrated in Fig. 4; also [43] gives another example for ship classification. Methods applicable to context classes obviously include the rewriting rules specified above in the context model of Secs. 2 and 3, for context adapt at ion and object-cont ext interact ions. 1- car(0,Thera.ObjLls~).
1 CALL car(0,_361,[_452,_4541) 2 CALL carWlndow(O,_t83._45i) 3 CALL locCorner(O,l,[~lC88,_1490._1C92.imgl],~l486) 4 EXIT 1ocCorner(0,1.~16,0.5,4~~mgl).I1110.210),[180,170],[190,200],[120,250]]) 5 EXIT carWindow(0,1,trapezoi0(0,l,lmgl)) 6 ChLL nextTo(trapezold(O,l,lmgl),-4?3,-494) 7 CALL 1ocCorner(0,1,[~21696,~2169~,~21700,~mg1],~21694) 8 F A I L 1ocCorner(0,1,[l6,0.5,4,1mg1).II110,210],[180,170], 9 RE00 locCorner(0,2,I,21696,~21698,~21700,1mg2],~216?~) 10 EXIT 1ocCorner(0,2,(12,0.5,4,1mg2].([200,160],[2C0,120] 11 W I T nextTo(trapezold(0,1,lmgll,quad(0,2,lmg2),104.043) 12 CkLL carWlndow(qusd(0,2,lmg2),_C54) 13 ChLL carWlndow(0,2,~454) 14 CALL locCorner(0,2,~_29070,~29072,~29074,img2],_29068) 15 W I T locCorner(0,2,[12,0.5,4,img2],[(200,160],(240,120] 16 EXIT carUlndow(0,2,tra~azoid(0,2,lm~2)) 17 EXIT carWindw( quad( 0,i,lmq2 j ,trape;oi;l( 0.2, img2) ) 1 8 CALL carSlde(trapezold(O,l,imgl),trapezold(0,2,img2),~361) 19 FAIL carSlde(trapezold(O,l,lmgl),trcpe~old(0,2,img2),~361) 2 0 REDO carWlndw(0,2,trapezoid(0,2,lmg2)) il REDO ca~indw(qu~d(0,2,lmg2),~ra~e:aid(0,2,impi)) 22 CALL locCorner(0,2,~~29070,~29072,~29074,lmg3],~29068) 23 EXIT 1 o c C o r n e r ( 0 , 2 , [ 1 8 , 0 . 5 , C . i m g ) l . 1 [ 2 0 0 , 1 5 0 ] , [ 2 4 0 , 1 2 0 ] , [ 2 8 0 , 1 3 0 ~ , [ 2 2 0 , 1 9 0 ] ] ) 24 EXIT carWlndov(0,2,trapezold(0,2,lmg3)) 25 W I T carWlndov(quad(0,2,11~g2),trapezold(0,2,lmg3)) 26 CALL carSide(trapetold(0,l,i~ngl),tscpezoid(0,2,lmg3),~361) 27 W I T carSide(trapetold(0,l,imgl),trapezold(0,2,img3),-1.47424) 28 EXIT car(0,-1.47424,[trapezoid(O,l,lm~l),~rapezold(O,2,im~3)]) Theta -3.47424, ObjLlst = [trapezold(O,l,lmgl),trapezoid(0,2,lmg3)]
-
Fig. 6. Example Prolog trace showing toplevel goal ‘car’ (1); subgoal ‘carwindow’ identifies an object as a car window ( 5 ) ; ‘nextTo’ finds a neighboring object (11); neighbor object identified as a car window (17); these two windows fail t o satisfy ‘carside’ (19); backtracking to ‘carwindow’ causes corner detection t o retry same object with new parameters and derives a new corner placement (22); two windows satisfy the ‘carside’ predicate (27).
The visualization of the search process S is displayed in Fig. 5, while Fig. 6 gives the trace of the “I” car recognition goal execution. Figure 7 displays the corner detection results achieved by this search process, for varying values of the bound SCAN parameter (see Sec. 3). Figure 8 displays the regions characterizing the car class instance, in the binary/thresholded image in Fig. 9. Please note that in Fig. 9 there are reflections from the roof and shadows, with the shadows eliminated by the context modeling, whereas the roof is treated as a supplementary LLwindow” but not found to abide by the neighborhood graph Context-G, and thus is eliminated by the constraint satisfaction.
756
L. F. Pau
Fig. 7. Four corner detection results achieved by backtracking, varying the scan window length. Each result shows the same contour with a different corner placement.
Fig. 8. Image of car after preprocessing (upper left) is identified as a car instance, matching the three extracted contours shown as car windows t o satisfy one of the 'car' predicates.
4.2. Extensions to Sensor Fusion
The object oriented design comes in handy to represent context diversity and changes in the case of sensor fusion tasks [25,26,49], where different detection ranges of heterogeneous sensors overlap in the feature domains while having distinct attributes.
4 . 3 Context Related Issues in Image Understanding 757
Fig. 9. Regions extracted by the understanding procedure in the car image. Some connected components have been filtered out thanks t o context knowledge, kept separate from the generic one, and also the thresholds are adapted locally.
Above the context classes, a super-class must be defined for the instances of the same contexts according to different perceptual/sensor ranges. The sensor fusion tasks apply to this root class level. Inheritance applies for the context classes and the methods underneath, but the context causal graphs and influencing domains stay specific to each sensor. 4.3. Comparisons
A summary comparison of full object oriented design to the object oriented elements considered above, is presented in Table 2. But some comments are required. Table 2. Summary comparison of full object oriented design and elements used here. Attributes
Full 00 Design
Objects/Classes Inheritance
Yes Full
Encapsulation
Yes
Last binding
Yes
Current Image Understanding Environment Yes Inheritance for methods No inheritance for instance variables No data in instances Methods exist only in class definitions Yes - inherent in Prolog’s interpretive environment
758
L. F. Pau
An instantiation exists only within the predicate which created the object (with “new”). Thus we do not require here full object oriented design; this however is consistent with a formal object-oriented inheritance model such as [13]. Class inheritance is limited t o methods; there are no instance variables per se. Furthermore, code is not actually encapsulated: objects, e.g. relating t o the context, contain no code of their own, but forward messages to their class definitions. This type of behavior resembles “delegation” [16], but includes the message based inheritance presented in [13]. There is no loss of generality with respect t o Prolog, as the class definitions are translated by Prolog into Prolog; inheritance is also implemented within Prolog [23,41,44,47]. Unification/backtracking/constraintsatisfaction are fully applicable t o class inheritances, especially for the context model, because of the object oriented modalities above. The form of polymorphism presented here for context modeling is most flexible when the instances of influencing domains “Context-D” , processes { p } , and possibly “context-object”-names, each strictly abide to a class tree structure. This holds if basically the image understanding context contains few context elements (i.e. few “Context-D” domains), but with a high variability within each class. This variability is well modeled by the subclasses, instances, and inheritance. Such a n object oriented design is insufficient if the “Context-G’ graphs are rich, with many arcs, labels, and loops all corresponding to contextual ambiguities.
5. Perceptual Context Modeling 5.1. Perceptual Context Modeling This fourth approach is also covered by the model of Sec. 2, by a suitable joint selection of the topologies and of the causal structures in the context graphs “ContextG” and the influencing domains “Context-D” .
Example: Perceptual noise filtering in images Many image understanding tasks consist in removing the perceptual consequences of noise, as this interferes with the interpretation. Many noise removal algorithms rely on bandpass filtering of the gray levels, thus blurring or deleting or fragmenting, e.g. linear elements, such as the car element border lines in the example. Context based modeling consists in representing the “influenced process as a stochastic process, triggered by the local variations in gray values in the influencing domain “Context-D” around each pixel. This context model will be parametrized also by the orientation and curvature of edges or lines found in the influencing domain. Areas containing a lot of structure, with strong contrast between edges/lines and the background, have a higher gray value variance in local areas. This determines, by a context predicate, whether noise reduction by low-pass filtering is to be
4.3 Context Related Issues in Image Understanding 759 applied or not. The result is that noise can be reduced, while the edges in areas with prominent structures are kept sharp. By implementing separation of the context model and of the filter, noise removal filters can be designed with no regard to the structure of the image content. 5.2. Ontologies
One difficulty with perceptual context modeling is that, when this approach is used, it is often not clear what the categories, objects, attributes, entities and conceptual structures, are. This is what an ontology is about, i.e. about the study of the concepts and categories of the world. The understanding and semantic depth of an image in a domain depend on the richness of the ontology of that domain. Most A1 ontologies treat situations or states of the world as objects which can participate in relations; situations get changed by events and generate scripts. Underlying the ontologies are causal links which dictate the behavior: causal links are specialized relational links which indicate the propagation of change. The knowledge and mapping of the causal links driving perceptual changes is still not known (see however [50] and (511). 6. Conclusions
Context modeling is of course difficult, but the time has come where the availability of context simulation models on one hand, and the urgent need for code sharing and reuse on the other hand, just simply impose the use of some still imperfect solutions. Until recently, there was little or no progress, just because of a lack of formal descriptions of the understanding task and of the context. While the jury is still out on the specific forms of inheritance and polymorphism in object oriented design implementations, dynamic inheritance is clearly lacking in image understanding. By dynamic inheritance we mean allowing a method to change the inheritance specifications of the methods within an object. This is required to simplify the representation of context related physical phenomena and constraints, like changes in contrast, shape discontinuities, etc., and to be able to address perceptual cues better than now. Furthermore, image understanding research should not ignore work on contexts in natural language [19,45]. In this field, a cohesive and coherent discourse relies on a three-tiered representation system based on linguistic and knowledge bases; semantic plausability relies finally on an overall discourse model, with the invocation of context specific handlers which: a a
specify which types of interpretations are possible for each specific handler type; combine all information they get into a single text which reflects constraints on the plausability of a proposed interpretation.
This is very similar to the approach proposed here.
760 L. F. Pau
Finally, the extension of context modeling to sensor fusion should not necessarily we viewed as a “complication”; t o the contrary, sensor diversity may often allow for disambiguation, provided the causal and physical processes are entirely known.
References [l] A. Van Hentenryck, Constraint satisfaction in logic programming (CHIP) (MIT Press, Cambridge, MA, 1989). [2] J. Cohen, Constraint logic programming languages, J. ACM 33, 7 (1990) 52-67. [3] J. Jaffar and J.-L. Lassez, Constraint logic programming, in Proc. 14th ACM Symp. on Principles of Programming Languages (POPL-87), Miinchen, 1987, 111-119. [4] W. Leler, Constraint Programming Languages: Their Specification and Generation (Addison Wesley, 1987). [5] A. Colmerauer, An introduction to Prolog 111, J. ACM 33, 7 (1990) 67-90. [6] P.-J. Gailly et al., The Prince project and its applications, in G. Comyn and N. E. fichs (eds.), Logic Programming in Action, Lecture Notes in Artificial Intelligence, Vol. 636 (Springer Verlag, Berlin, 1992) 55-63. [7] D. H. Ballard and C. M. Brown, Computer Vision (Prentice-Hall, Englewood Cliffs, NJ, 1982). [8] B. Bell and L. F. Pau, Contour tracking and corner detection in a logic programming environment, IEEE Trans. Pattern Anal. Mach. Intell. 12, 9 (1990) 913-916. [9] D. Cruse, C. J. Oddy and A. Wright, A segmented image data base for image analysis, Proc. 7th Int. Conf. on Pattern Recognition, Montreal, Canada (IEEE, 1984) 493-496. [lo] E. C. Freuder, Backtrack-free and backtrack-bounded search, in L. Kana1 and V. Kumar (eds.), Search in Artificial Intelligence (Springer-Verlag, New York, NY, 1988) 343-369. Ill] F. Giannesini, H. Kanoui, R. Pasero and M. van Caneghem, Pmlog (Addison Wesley, Reading, MA, 1986). [12] A. Goldberg and D. Robson, Smalltalk-80: The Language and Its Implementation (Addison-Wesley, Reading, MA, 1983). [13] B. Hailpern and Van Nguyen, A model for object-based inheritance, in B. Shriver and P. Wegner (eds.), Research Directions in Object-Oriented Programming (MIT Press, Cambridge, MA, 1987) 147-164. [14] R. M. Haralick and G. L. Elliot, Increasing tree search efficiency for constraint satisfaction problems, Artif. Intell. Journal 14 (1980) 263-313. 115) A. Huertas and R. Nevatia, Detecting buildings in aerial images, Comput. Vision Graph. Image Process. 41 (1988) 131-152. [16] H. Lieberman, Using prototypical objects to implement shared behavior in object oriented systems, Proc. ACM Conf. on Object Oriented Programming, Systems, Languages, and Applications, Portland, OR, 1986, 214-223. [17] S. Matwin and T. Pietrzykowski, Intelligent backtracking in plan-based deduction, IEEE Trans. Pattern Anal. Mach. Intell. 7, 6 (1985) 682-692. [18] D. M. McKeown, Jr., W. A. Harvey, Jr. and J. McDermott, Rule-based interpretation of aerial imagery, IEEE Trans. Pattern Anal. Mach. Intell. 7, 5 (1985) 57c585. [19] B. Neumann, Natural language description of time-varying scenes, in D. Waltz (ed.), Advances in Natural Language Processes, Vol. I (Morgan Kaufmann, 1984). [20] L. F. Pau, Knowledge-based real-time change detection, target image tracking, and threat assessment, in A. K. C. Wong and A. Pugh (eds.), Machine Intelligence and Knowledge Engineering for Robotic Applications, NATO A S I Series, Vol. F-33 (Springer Verlag, Berlin, 1987) 283-297.
4.3
Context Related Issues in Image Understanding
761
[21] L. F. Pau, Knowledge representation for three-dimensional sensor fusion with context truth maintenance, in A. K. Jain (ed.), Real Time Object Measurement and Classification (Springer-Verlag, Berlin, 1988) 391-404. [22] K. C. You and K. S. Fu, A syntactic approach to shape recognition using attribute grammars, IEEE %ns. Syst. Man Cybern. 9, 6 (1979) 334-345. [23] L. Leonardi, P. Mello and A. Natali, Prototypes in Prolog, J. Object Oriented Programming 2,3 (1989) 2Cb28. [24] A. R. Rao and R. Jain, Knowledge representation and control in computer vision systems, IEEE Expert, Spring (1988) 64-79. [25] L. F. Pau, Knowledge representation for sensor fusion, in Proc. IFAC World Congress 1987 (Pergamon Press, Oxford, 1987). (261 S. B. Pollard, J. E. W. Mayhew and J. P. Frisby, PMF: a stereo correspondence algorithm using a disparity gradient limit, J. Perception 14,449-470. [27] R. A. Brooks, Model based 3-D interpretation of 2-D images, in Proc. 7th Int. J. Conf. on Artificial Intelligence, 1981, 619-623. 1281 L. Wes, R. Overbeck, E. Lusk and J. Boyle, Automated Reasoning: Introduction and Applications (Prentice Hall, Englewood Cliffs, NJ, 1984). [29] L. Kitchen and A. Rosenfeld, Scene analysis using region-based constraint filtering, Pattern Recogn. 17,2 (1984) 189-203. [30] Y. Ohta, Knowledge Based Interpretation of Outdoor Natural Scenes (Pitman Advanced Publishing Progr., 1985). [31] J. Doyle, A truth maintenance system, Artif. Intell. J. 12 (1979) 231-272. (321 J. De Kleer, Choices without backtracking, in Proc. AAAI Nat. Conf. on Artificial Intelligence, Aug. 1984. [33] A. Rosenfeld et al., Comments on the Workshop on Goal-Directed Expert Vision Systems, Comput. Vision Graph. Image Process. 34,1 (1986) 98-110. [34] Harbour change of activity analysis, AD 744332, NTIS, Springfield, VA, 1982. [35] Proc. DA RPA Image Understanding Workshop, Science Applications Report SAI-84176-WA, or AD 130251, NTIS, Springfield, VA, June 1983. [36] J. Ebbeni and A. Monfils (eds.), Three-Dimensional Imaging, Proc. SPIE, Vol. 402, Apr. 1983. [37] N. Kazor, Target tracking based scene analysis, CAR-TR-88, CS-TR-1437, Univ. of Maryland, College Park, MD, Aug. 1984. (381 B. Bell and L. F. Pau, Context knowledge and search control-issues in object oriented Prolog-based image understanding, Pattern Recogn. Lett. 13 (1992) 279-290. [39] P. Coad and E. Yourdon, Object Oriented Analysis (Prentice Hall, Englewood Cliffs, NJ, 1991). [40] A. Palaretti and P. Puliti, A Prolog approach to image segmentation, J. Appl. Artif. Intell. 3,4 (1990) 56-68. [41] R. Knaus, Message passing in Prolog, AI Expert, May 21-27 (1990). [42] D. T. Lawton, Image understanding environments, Proc. IEEE 7 6 , 8 (1988) 1036-1050. [43] S.-S. Chen (ed.), Image Understanding in Unstructured Environments (World Scientific, Singapore, 1988). [44] D. Pountain, Adding objects to Prolog, BYTE, Aug. (1990). [45] S. Luperfoy and E. Rich, A computational model for the resolution of contextdependent references, MCC Technical report NL-068-92, MCC, Austin, TX, Mar. 1992. [46] B.J&ne, Digital Image Processing (Springer Verlag, Berlin, 1991).
762 L. F. Pau [47] B. Bell and L. F. Pau, Prolog object oriented embedded manager, Tech. Report, Technical University of Denmark, 21 Jul. 1989. [48] G . Ciepel and T . Rogon, Background modelling in an object oriented, logic programming, image processing environment, Tech. Report, Technical University of Denmark, May 1990. [49] L. F. Pau, Sensor and Data Fusion (Academic Press, NY, 1993). [50] L. F. Pau, Behavioral knowledge in sensor and data fusion systems, J. Robotic Syst. 7, 3 (1990) 295-308. [51] V. Akman and N. Surav, Step toward formalizing context, A1 Magazine 17, 3, 55-72.
Appendix A. Object Oriented Definitions
A good introductory article and a more detailed glossary are found e.g. in [12,39]. Class: A set of elements sharing the same behavior and structure characteristics which are represented in a “class definition”. A class which adopts the behavior and structure of another class, but specializes some characteristics t o form a subcategory, is a “subclass”. Constraints: Predicates or parameters which control goal evaluation and backtracking so that the resulting object classification or scene understanding is consistent with the known physical characteristics of the object or class. Context: A consistent subset of facts derived during the evaluation of a higher level goal. Context knowledge includes predicates t o establish a context, i.e. t o evaluate a top-level image understanding goal, and constraints t o enforce consistency. Instance: An element created from the descriptions in a class definition. An image object is classified when it is determined t o be a n instance of a class. Method: A segment of code appearing within a class definition, which can be invoked by sending a message t o any instance of that class t o evaluate the named method.
Appendix B. Constraint Logic Programming Logic is a powerful way of representing problems. Constraints appearing in logic programming are tests used for checking a solution. With CLP (Constraint Logic Programming), the constraints are part of the description of the problem, i.e. in this chapter. It is the image understanding task. The way they are used allows for a more efficient search for solutions. Further details on various CLP concepts and implementations are found in [la].
B . l . Syntaz
A problem is represented in a CLP by a set of clauses (or rules), like in logic programming languages like Prolog [ll]. However, the syntax of the clauses is
4.3 Context Related Issues in Image Understanding 763 different. The common part is that a clause consists of a term, the head of the clause, and a list of terms (which can be empty) in the body of the clause; in both cases it means “the head term is true if the body terms are true”. The difference with respect to logic programming languages is that CLP clauses can also contain a list of constraints (see next section), and in this case the meaning becomes: “the head term is true if the body terms are true and the constraints are not violated”.
B .2. Constraints Constraints apply t o terms, boolean values, identifiers, lists and trees, numbers (integers, rationals and/or reals, depending on the CLP language used). Constraints are equations or inequations or logicals or fixed lists. The variables in the constraints behave like unknown quantities in mathematical equations:
Examples: X = X / 2 + 1 , implies X = 2 ; X = X + 1 , has no solution; Y = X + 1, means that, if the two variables X , Y are unbound, they belong t o one line: A + B C, says that A is true if B and C are; 0 < T < 3X/4 + 5Y, defines a region for the unbound variable T . Execution efficiency depends very much on the time at which the constraint is treated, and on the algorithm used for testing the satisfiability of systems of constraints, in what is called the constraint solver.
B .3. Resolution The resolution mechanism in CLP languages is based both on a constraint solving mechanism, which is in charge of testing if the constraints can have a t least one solution, and upon a unification algorithm which attempts t o prove each term of the goal by replacing it by an equivalent set of terms and constraints, as they appear in a clause. At each step of the attempt to prove a logical goal, the constraint solver must decide if there is at least a solution for the set of constraints on the variables which appear in the terms considered. The mechanism is as follows: given a set of variables W (appearing in the query), a list of terms to, t l , . . . , tn, and a list of currently satisfiable constraints S, two states are defined:
(I): (W, to, t l , ..., tn, S ) ( 2 ) : (W, sl, . . . , sm,t l , . . . , tn, (S U R U (SO = to) )), with
U=
“union”
An inference step consists of making a transition from the state (1) to the state (2) by applying the program rewriting rule ( r ) :
r: SO
t sl, ..., sm,R
764
L. F. Pau
in which the (si)’s are terms and R is the set of constraints specified by the CLP rule. The new state after inference becomes (2) if the new set of constraints (S U R U (SO = to)) is satisfiable; it has at least one possible solution. Here, (SO = to) represents the set of constraints applied to variables so that SO and t O become identical. If the new set of constraints is not satisfiable, another rule has to be tried in the CLP program to attempt to replace the term. There are two types of non-determinisms that arise in the sequential interpretation of such CLP programs: the first is the selection of the term in the list of terms that will be processed first, and the second is the choice of an applicable rule in the CLP program. The constraint solver must be incremental to minimize the computational effort required to check if the constraints remain satisfiable or not. If the set S of constraints has solutions, adding a new set of constraints R will not require to solve for (SU R), but to transform the solutions of S into solutions of (Su R).
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 765-796 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
I CHAPTER 4.4I POSITION ESTIMATION TECHNIQUES FOR AN AUTONOMOUS MOBILE ROBOT-A REVIEW
RAJ TALLURI and J. K. AGGARWAL Computer and Vision Research Center, Department of Electrical Engineering University of Texas at Austin, Austin, Texas 78712, USA In this paper, we review various methods and techniques for estimating the position and pose of an autonomous mobile robot. The techniques vary depending on the kind of environment in which the robot navigates, the known conditions of the environment, and the type of sensors with which the robot is equipped. The methods studied so far are broadly classified into four categories, landmark-based methods, methods using trajectory integration and dead reckoning, methods using a standard reference pattern, and methods using a priori knowledge of a world model which is matched the sensor data for position estimation. Each of these methods is considered and its relative merits and drawbacks are discussed. Keywords: Autonomous navigation, mobile robots, position estimation, self-location, landmarks, world model.
1. Introduction Autonomous mobile robots are one of the important areas of application of computer vision. The advantages of a vehicle that can navigate without human intervention are many and varied, ranging from providing access to hazardous industrial environments to battlefield surveillance vehicles. A number of issues and problems must be addressed in the design of a n autonomous mobile robot, from the basic scientific issues to state-of-the-art engineering techniques. The tasks required for successful autonomous navigation by a mobile robot can be broadly classified as (1) sensing the environment; (2) building its own representation of the environment; (3) locating itself with respect to the environment; and (4)planning and executing efficient routes in this environment. It is advantageous for a robot t o use different types of sensors and sensing modalities to perceive its environment, since information available from one source can be used to better interpret information from other sources, and can be synergically fused to get a much more meaningful representation. Some of the different sensor modalities considered by previous researchers are visual sensors (both monocular and binocular stereo), infrared sensors, ultrasonic sensors, and laser range finders.
766
R. Talluri & J. K . Aggamal
Building a world model also termed map-making is an important problem in mobile robot navigation. The type of spatial representation system used by a robot should provide a way to consistently incorporate the newly sensed information into the existing world model. It should also provide the necessary information and procedures for estimating the position and pose of the robot in the environment. Information t o do path-planning, obstacle avoidance and other navigational tasks must also be easily extractable from the built world model. Section 3 presents a review of various mapmaking strategies and their associated position estimation methods. Determining the position and the pose of a robot in its environment is one of the basic requirements for autonomous navigation. In this discussion, position refers t o the location of the robot on the ground plane and the pose refers t o the orientation of the robot. We use the term position estimation t o refer to the estimation of both position and pose. The problem of self-location has received considerable attention, and many techniques have been proposed to address it. These techniques vary significantly, depending on the kind of environment in which the robot is to navigate, the known conditions of the environment, and the type of sensors with which the robot is equipped. Most mobile robots are equipped with wheel encoders that can be used to estimate the robot’s position at every instant; however, due to wheel slippage and quantization effects, these estimates of the robot’s position contain errors. These errors build up and can grow without bounds as the robot moves, and the position estimate becomes more and more uncertain. So, most mobile robots use an additional form of sensing, such as vision or range, to aid the position estimation process. In this paper we review various techniques studied for estimating the position and pose of an autonomous mobile robot. Broadly, we classify the position estimation techniques into four categories, landmark-based methods, methods using trajectory integration and dead reckoning, methods using a standard reference pattern, and methods using a priori knowledge of a world model which is matched to the sensor data for position estimation. These four methods are briefly described below. In landmark-based methods, typically the robot has a list of stored landmark positions in its memory. It then senses these landmarks using the onboard sensors and computes the position and pose using the stored and the sensor information. Section 2 reviews the different approaches using landmarks for self-location. In the second type of position estimation technique, the position and pose of a mobile robot are estimated by integrating over its trajectory and by dead reckoning, i.e. the robot maintains an estimate of its current location and pose at all times and, as it moves along, updates the estimate by dead reckoning. Section 3 reviews these methods. A third method of estimating the position and pose of the mobile robot accurately is to place standard patterns in known locations in the environment. Once the robot detects these patterns, the position of the robot can be estimated from the known location of the pattern and its geometry. Different researchers have used different
4.4
Position Estimation Techniques for an Autonomous Robot 767
kinds of patterns or marks, and the geometry of their methods and the associated techniques for position estimation vary accordingly. These methods are discussed in Section 4. Finally, some researchers consider the position estimation problem using a priori information about the environment in which the robot is to navigate, i.e. a preloaded world model is given. The approach is to sense the environment using onboard sensors and to match these sensory observations to the preloaded world model to arrive at an estimate of the position and pose of the robot with a reduced uncertainty. Section 5 presents a review of the different methods studied in solving these issues. Once the robot has the capability to sense its environment, build a representation of it, and estimate its position accurately, then navigational tasks such as pathplanning and obstacle avoidance can be performed.
2. Landmark-Based Methods Using landmarks for position estimation is a popular approach. The robot uses the knowledge of its approximate location to locate the landmarks in the environment. Once these landmarks are identified and the range/attitude of these relative to the robot is measured, in general, the position and pose of the robot can be triangulated from these measurements with a reduced uncertainty. Landmarks used for position estimation can include natural or man-made features in the outdoor environment, such as the tops of buildings, roof edges, hilltops, etc., or can be identifiable beacons placed at known positions to structure the environment. One basic requirement of the landmark-based methods is that the robot be able to identify and locate the landmarks, which is not an easy task. The position estimation methods based on landmarks vary significantly depending upon the sensors used (eg. range or visual sensors); the type of landmarks (i.e. whether they are point sources or lines etc.); and the number of landmarks needed. Case [4] summarizes the landmark-based techniques quite well and presents a new method, called the running fix method, for position estimation. Case classifies the sensor data for navigation purposes as either angular or range type inputs. In general, any combination of two of these is sufficient to yield a fix. In Fig. 1, for instance, the angle between the x-axis and each of the two landmarks is used to construct two lines of position (LOP) which intersect at the robot’s location. In Fig. 2, arcs are struck at the measured range corresponding to the two landmarks. The intersection points of these two arcs yield two possible positions for the robot, thus requiring either correlation with an estimated position or the use of a third landmark to resolve the ambiguity. A range and an angle may also be used, as in Fig. 3. This requires only one landmark, but requires either multiple sensors or a sensor capable of measuring both range and attitude. The angle measurements can be either absolute or relative. Absolute angle measurements require the robot to maintain an
768
R. Talluri €4 J. K. Aggarwal LOP 2 Landmark 2
Fig. 1. Two landmarks and lines of position.
Landmark 2
Robot
Landmark 1 Fig. 2. Ranges from two landmarks.
Fig. 3. LOP and range from one landmark.
4.4
Position Estimation Techniques for an Autonomous Robot
769
internal reference using a gyrocompass or a n inertial sensor. Any error in this reference usually affects the position estimation accuracy. Case also presents a method called the running fix for use in areas with a low density of landmarks. The underlying principle of the technique in that an angle or range to a landmark obtained at a time t - 1 can be used at a time t. To do this the cumulative measurement vector recorded since the reading was obtained is added to the position vector of the landmark, thus creating a virtual landmark. Case presents experimental results using an ARCTEC Gemini mobile robot with an infrared beacon/detector pair. Clare D. McGillem et al. [35] also describe an infra-red location system for autonomous vehicle navigation. They present a n efficient method of resection for position estimation using three infrared beacons t o structure the environment and an optical scanner on the robot capable of measuring the angles between a pair of beacons. They also discuss the sensitivity of the method to errors. They point out that by judiciously placing the beacons in the environment, regions of high error sensitivity can be minimized or avoided. They demonstrate the feasibility of their approach by implementing it on a n inexpensive experimental system. Nasr and Bhanu [42] present a new approach t o landmark recognition based on the perception, reasoning, and expectation (PREACTE) paradigm for autonomous mobile robot navigation. They describe a n expectation driven, knowledge-based landmark recognition system that uses a priori map and perceptual knowledge. Bloom [2] also describes a landmark-based system for mobile robot navigation that uses a grid-based terrain map (MAP), a landmark database, and a landmark visibility map (LVM). Bloom describes the landmarks by one or more of three distinctive attribute sets: a color attribute set, a textural attribute set, and a 2-D geometry attribute set. He then describes the contents of these attribute sets and shows how the vision system uses these attributes to recognize the landmarks. Sugihara [47,48] presents algorithms for the position estimation of a mobile robot equipped with a single visual camera. He considers the problem of a robot given a map of a room in which it navigates. Vertical edges are extracted from the images taken by the robot’s camera, with the optical axis parallel to the floor. The points from where the vertical edges can arise are assumed to be given. Sugihara then considers two classes of problems. In the first class, all vertical edges are identical, and he searches for the point where the image is taken by establishing a correspondence between the vertical edges in the images and those in the map. In the second class of problems, the vertical edges are not distinguishable from each other and the exact directions in which the edges are seen are not given; only the order in which they are found in the image is given. The problems are considered mainly from the point of view of computational complexity. In the case where the vertical lines are distinguishable from one another, Sugihara shows that if we establish a correspondence between three image points and three poles (vertical lines) and measure the angles between the rays joining the image points to the lines, we can uniquely determine the camera’s position.
770 R . Talluri & J . K. Aggarwal
Fig. 4. The unique camera position determined by three rays and the corresponding mark points.
In Fig. 4, p l , p2, and p3 are the three poles and R is the robot’s position. In the case involving four poles, the solution is not necessarily unique. So, in general, when we have k poles and, hence, k rays, Sugihara suggests using the first three rays, rl,rz,rg to determine the position R and then using the other rays to check if the solution R is correct. Now, in the general case when the k lines are not distinguishable from one another, the suggested approach is: First, choose and fix any four rays, say r1 ,7-2, rg, 7-4, and next, for any quadruplet (pi,p j , pk,p ~ ) of marks (vertical lines), solve for the position on the assumption that r1, 7-2, rg, 7-4 correspond to ( p i , p j , p k , p l ) , respectively. Then repeat for the n(n- l ) ( n - 2 ) ( n - 3 ) different quadruplets for a consistent solution. The above naive procedure can solve for the position in O(n4) time. He then gives a less naive algorithm for the position estimation, with n identical marks, which runs in O(n3logn) time with O ( n ) space or in O(n3)time with O(n2)space. Sugihara also considers variations of this problem of n indistinguishable vertical lines, such as: (1) the existence of spurious edges; (2) the existence of opaque walls; (3) linearly arranged marks; and (4)a case in which the robot has a compass. He discusses the possible solutions and simplifications of the original algorithm to these special cases. The case in which the marks are distinguishable from one another but the directions are inaccurate is considered in the second part of the paper. He shows that this case is essentially the same as the problem of forming a region which generates a circular scan list in a given order. Krotkov [25] essentially followed Sugihara’s work of localizing a mobile robot navigating on a flat surface with a single camera using the vertical lines in the image plane as landmarks. He formulates the problem as a search in a tree of interpretations (pairings of landmark directions and landmark points), The algorithm he uses is the naive algorithm, discussed by Sugihara, that runs in O(n4) time. In his work, Krotkov also considers the errors in the ray directions and, using the worst-case analysis, comes up with bounds on the position and pose estimated using this method of localization. He shows that in the case when the angles of the rays
4.4
Position Estimation Techniques for a n Autonomous Robot
771
R, Region of possible robot locations
Fig. 5 . Possible locations given by noisy rays.
are erroneous, the robot position estimated lies not on one point but in a region of possible points, R (see Fig. 5). He also presents simulation results with random ray errors and worst-case ray errors and makes the following observations from his analysis: (1) the number of solution poses computed by the algorithm depends significantly on the number k of angular observations and the observation uncertainty 64; and (2) the distribution of solution errors, given angular observation errors that are either uniformly or normally distributed, is approximately Gaussian, whose variance is a function of 64. Krotkov also presents real data results using a CCD imagery. Most of the landmark-based approaches considered above suffer from the disadvantages of : (1) assuming the availability of landmarks in the scene around the robot; (2) depending on the visibility and the ability to recognize these landmarks from the image to estimate the range/attitude to them from the current location; (3) requiring an approximate starting location to check for the landmarks; and (4) needing a database of landmarks in the area to look for in the image.
2.1. Photogrammetric Methods Photogrammetry generally deals with the mathematical representation of the geometrical relations between physical objects in three-dimensional space based on their images recorded on a two-dimensional medium. Over the years, photogrammetry has been routinely used in aerial photography, cartography, and remote sensing [61]. One of the problems of cartography is to determine the location of a n airborne camera from which a photograph was taken by measuring the positions of a number of known ground objects or landmarks on the photograph. This problem is sometimes known as the camera calibration problem. The orientation and position of the camera in the object space are traditionally called the camera’s exterior orientation partimeters as opposed to its interior orientation parameters, which are independent
772
R. Talluri & J. K. Aggarwal
of the co-ordinate system of the ground objects. The interior orientation parameters include such elements as the camera’s effective focal length, lens distortion, decentering, image plane scaling, and optical axis orientation. These parameters generally do not vary as much, or as quickly, as the exterior orientation parameters and need not be updated at image sampling rates. For nonmetric cameras, standard off-line calibration procedures are available for determining the elements of the interior calibration. The problem of estimating the position and pose of an autonomous mobile robot is, in essence, similar to the camera exterior orientation problem in photogrammetry. However, since the robot is ground-based and has position encoders and other sensors on it, these can be used to constrain the possible orientation and pose. In general, the exterior camera orientation problem involves solving for six degrees of freedom, three rotational and three translational. Traditionally, in single camera photogrammetry, by observing the object’s feature points on the image, it is possible to solve the exterior orientation calibration problem using a traditional method known as space resection [12]. The method is based on the perspective geometry of a simplified camera model, derived from pinhole optics, in which the image of each feature point is projected onto the image plane by a ray connecting the feature point with the pinhole lens. This collinearity condition results mathematically in two nonlinear equations for each feature point. Hence at least three non-collinear points are required to solve for the six degrees of freedom. These collinearity equations are linearized and solved in a n iterative fashion. When the images are noisy, more than three points can be used, with least squares criteria, to take advantage of data smoothing. These methods are now standard in the photogrammetry literature [61]. Iterative solutions are generally more computationally demanding, so that simplifying assumptions are usually necessary for real-time applications. Over the years, a number of alternate methods have been proposed in an effort t o improve the efficiency of the camera calibration procedure. Some of these are reviewed below. Szczepanski [49] surveys nearly 80 solutions, beginning with one by Schrieber of Karlsruhe in 1879. The first robust solution in computer vision literature is by Fischler and Bolles [15]. They studied the exterior calibration problem in connection with the concept of random sample consensus (RANSAC), a methodology proposed for processing large data sets with gross errors or outliers. They argue against the classical techniques of parameter estimation, such as least squares, that optimize (according to a specified objective function) the fit of a functional description (model) to all the presented data. Their argument is that the above techniques are usually averaging techniques that rely on the smoothing assumption, which is not usually valid when the data has outliers or gross errors. The RANSAC paradigm they present can be stated as follows: Given a model that requires a minimum of n data points to instantiate its free parameters and a set of data points P such that the number of points in P is greater than n,randomly select a subset S1 of n data points from P and instantiate the
4.4
Position Estimation Techniques for a n Autonomous Robot
773
model. Use the instantiated model M1 to determine the subset S1*of points in P that are within some error tolerance of M1. The set S1*is called the consensus set of s1. If # (S1*)is greater than some threshold t , which is a function of the estimate of the number of gross errors in P , use 5'1' t o compute (possibly using least squares) a new model M1*. If # (S1*)is less that t , randomly select a new subset S 2 and repeat the above process. If, after some predetermined number of trials, no consensus set with t or more members has been found, either solve the model with the largest consensus set found or terminate in failure. Fischler and Bolles then discuss methods to determine the three unspecified parameters in the RANSAC paradigm: (1) the error tolerance, (2) the number of subsets to try, and (3) the threshold t. They then present a new solution to the Location Determination Problem (LDP) based on the RANSAC paradigm. They reduce the LDP problem to the perspective-n-point problem, i.e. if we can compute the lengths of the rays from three landmarks to the center of perspective projection, then we can directly solve for the location and orientation of the camera. They obtain solutions in a closed form for three and four coplanar feature points; the latter, as well as the case of six points in general position, are demonstrated to be unique. Unfortunately, these analytic solutions cannot be extended to the general case involving more than four points. Nevertheless, the paper does demonstrate graphically the existence of multiple solutions with four or five noncoplanar points. Beyond these qualitative observations, however, no conclusion was offered regarding the existence and uniqueness in the general case. The four point solution has been implemented in a power line inspection system [29]. Ganapathy [19] presents a noniterative, analytic technique for recovering the six exterior orientation parameters as well as four of the interior orientation parameters (two for scaling and two for the location of the origin in the image plane). His method assumes that the perspective transformation matrix relating the world model and image plane points is determined by experimental means. Ganapathy essentially presents an algorithm to decompose the given transformation into the various camera parameters that constitute the components of the matrix. He linearizes the system of equations represented by the transformation matrix by increasing the number of unknowns. He then adds additional constraints, drawn from the properties of the rotation matrix, to solve these systems of equations. The algorithm is independent of the number or distribution of the feature points, since this information has already been distilled into the transformation matrix. Although the matrix may be obtained through experimental means, it is not known whether the effort will be feasible for operation in real time. Kumar and Hanson [28] report that their implementation of the method is extremely susceptible t o noise, and suggest that the susceptibility may be due to the nonlinear least square minimization used, where it is assumed that all the parameters are linearly independent while they actually are not.
774 R. Talluri 63 J. K . Aggarwal
Tsai [58] presents a two stage technique for the calibration of both the exterior and interior parameters of the camera that is probably the most complete camera calibration method proposed so far. The interior parameters include the effective focal length, the radial lens distortion, and the image scanning parameters. The basic idea used is to reduce the dimensionality of the parameter space by finding a constraint or equation which is only a function of the subset of the calibration parameters. Tsai introduces a constraint called the radial alignment constraint, which is a function of only the relative rotation and translation (except for the z component) between the camera and the calibration points. Although the constraint is a nonlinear function of the above mentioned calibration parameters (called Group I parameters), a simple and efficient way exists for computing them. The rest of the calibration parameters (called Group I1 parameters) are computed with normal projective equations. A good initial estimate of the Group I1 parameters can be obtained by ignoring the lens distortion and using simple linear equations in two unknowns. The precise values for these Group I1 parameters can then be computed in one or two iterations, minimizing the perspective equation error. One of the limitations of this technique is that although the method calls for a minimum of five coplanar feature points (seven in the non-coplanar case), a much larger number is required for accuracy (60 points were used in the experiment). Furthermore, restrictions in the relative positions between the objects and the camera exist. For instance, the plane containing the feature points must not be exactly parallel to the image plane of the camera. Although these conditions can be easily arranged in a laboratory environment, they cannot be guaranteed to hold in a real life operating environment for a mobile robot. Finally, the range parameter (Group 11) must still be generated by a nonlinear optimization procedure (specified only as a steepest descent), the choice of which could have a major influence on the efficiency of the algorithm. Horaud et al. [21] consider the perspective-Cpoint problem. They derive an analytic solution for the case of four non-coplanar points, namely a biquadratic polynomial in one unknown. h o t s of such an equation can be found in closed form or by an iterative method. Finding a solution for four non-coplanar points is equivalent to finding a solution to a pencil of three non-coplanar lines: The three lines share one of the four points. The authors show the various line and point configurations that are amenable to solving the P4P problem. Liu, Huang, and Faugeras [31] present a new method for determining the camera location using straight line correspondences. Since the lines can be created from given points, the method can be used for point correspondences also. They show that the rotation matrix and the translation vector can be solved for separately. Both linear and nonlinear algorithms are presented for estimating the rotation. The linear met hod needs eight line correspondences or six point correspondences, while the nonlinear method needs three line or point correspondences. For the translation vector, the method needs three line correspondences or two point correspondences and the algorithm is linear. The authors argue that since the nonlinear methods
4.4
Position Estimation Techniques for an Autonomous Robot
775
need fewer correspondences and have a wide convergence range, they may be preferable in practical problems. The constraint used by Liu, Huang, and Faugeras is that the 3-D lines in the camera coordinate system must lie on the projection plane formed by the corresponding image line and the optical center. Using this fact the constraints of rotation can be separated from those of translation. They suggest two methods to solve for the rotation constraint. In the first, they represent the rotation as an orthonormal matrix and the device as an eigen-value solution. However, they do not enforce the six orthonormality constraints for an orthonormal matrix. The second method represents rotation by Euler angles and is a nonlinear iterative solution obtained by linearizing the problem about the current estimate by the output parameters. The translation constraint is solved by a linear least-squares method. Kumar [27] argues that the decomposition of the solution into the two stages of solving first for rotation and then for translation does not use the set of constraints effectively. His argument is that since the rotation and translation constraints, when used separately, are very weak constraints, even small errors in the rotation stage become amplified into large errors in the translation stage. This, he says, is particularly true in the case of an autonomous mobile robot in an outdoor environment, where the landmark distances from the camera are large. He suggests solving for both the rotation and translation matrices simultaneously to achieve better noise immunity. He uses the same constraints as Liu, Huang, and Faugeras but a different nonlinear technique. The technique he uses is one adapted from Horn [22] to solve the problem of relative orientation. Kumar presents two algorithms, R-then-T and R-and-T. The former solves for the rotation first and then for the translation using the rotation matrix. The latter solves for both rotation and translation simultaneously. He presents experimental results which show that R-and- T performs better in all cases. In addition, he also develops a mathematical analysis of the uncertainty measure, which relates the variance in the output parameters to the noise present in the input parameters. For the analysis, he assumes that that there is no noise in the 3-D model data and that the only input noise occurs in the image data. To handle the problem of outliers or errors in the data and landmark correspondences, Kumar and Hanson [28] present a technique that performs quite well even in the presence of up to 49.9%outliers or gross errors. The work is basically an extension of their previous work [27]. They present an algorithm called Med-R-and-T which minimizes the median of the square of the error over all lines, or the LMS (least median of squares) estimate. The outliers can be arbitrarily large. The algorithm is based on the robust algorithm by Rosseeuw [45]. LMS algorithms have been proven to have a 49.9% breakdown point. Haralick et al. [20] summarize the various cases of the position estimation problem using point data. They consider the pose estimation problem to involve, essentially, the estimation of the object position and orientation relative to a model reference frame or relative to the object position and orientation at a previous time using a camera sensor or a range sensor. They divide the problem into four cases, depending on the type of model and sensor data: (1)2-D model data and 2-D sensor
776
R. Talluri €4 J. K. Aggarwal
data, (2) 3-D model data and 3-D sensor data, (3) 3-D model data and 2-D sensor data, and (4) two sets of 2-D sensor data. All data considered is point data, and the correspondence between the model and sensor data is assumed. The 2-D sensor data is usually the camera perspective projection. The 3-D sensor data refers to range data. The authors refer to Case 3 as absolute orientation and Case 4 as relative orientation. Case 4 occurs in multicamera imagery or time-varying imagery. Haralick et al. present a solution to each of the above four problems and characterize their performance under varying noise conditions. They argue for robust estimation procedures in machine vision, since all machine vision feature extractors, recognizers, and matchers seem to make occasional errors which are indeed blunders. Their thesis is that the least square estimators can be made robust under blunders by converting the estimation procedure to an iterative, reweighted least squares procedure, where the weight for each observation depends on the residual error and its redundancy number. So, they first find the form of the least-square solution, establish their performance as a baseline reference, put the solution technique in an iterative reweighted form, and, finally, evaluate the performance using non-normal noise, such as slash noise. The least-squares solution for both the 2-D2-D and the 3-D-3-D cases are constrained to produce rotation matrices guaranteed to be orthonormal. Yuan [60] presents a general method for determining the 3-D position and orientation of an object relative to a camera based on a 2-D image of known feature points located on the object. The problem is identical to the camera exterior Calibration problem. In contrast to the conventional approaches, however, the method described here does not make use of the collinearity condition, i.e. the condition of the pinhole camera and the perspective projection. Instead, the algebraic structure of the problem is fully exploited to arrive at a solution which is independent of the configuration of the feature points. Although the method is applicable to any number of feature points, Yuan says that no more than five points are needed from a numeric stand point and, typically, three or four points suffice. A necessary condition for the existence of the solution is presented and also a rigorous proof for the uniqueness of the solution in the case of four coplanar points. He shows with simulation results that in the case of four feature points, non-coplanar configurations generally outperform the coplanar feature point configurations, in terms of robustness, in the presence of image noise. In a more recent work, Chen [6] describes a polynomial solution to the pose estimation problem that does not require an a przori estimate of the robot location, using line-to-plane correspondences. He describes the situations when such a problem arises. In the case of a mobile robot, the lines are the 2-D image features and the planes are the projection planes joining these lines to the 3-D world model features. As do Liu et al. [31], Chen also solves for the rotations first and then for the translations. The crux of the approach is that it converts a problem with three unknowns (the three rotation angles) into one that has two unknowns
4.4
Position Estimation Techniques for an Autonomous Robot
777
by transforming the co-ordinate system into a canonical configuration. The two unknowns are then computed by evaluating the roots of an eighth degree polynomial using an iterative method. Chen also presents closed form solutions for orthogonal, co-planar and parallel feature configurations. He also derives the necessary and sufficient conditions under which the line-to-plane pose determination problem can be solved. 3. Trajectory Integration and Dead Reckoning
In this section, we consider techniques for estimating the position and pose of a mobile robot by integrating over its trajectory and dead reckoning, i.e. the robot maintains an estimate of its current location and pose at all times and, as it moves along, updates the estimate by dead reckoning. In order to compute a n accurate trajectory, the robot detects features from the sensory observations in one of the positions and these are used to form the world model. As the robot moves, these features are again detected, and correspondence is established between the new and the old features and the trajectory of the robot is computed. These techniques do not rely on the existence of landmarks and the robot’s ability to identify them. However, to successfully implement such techniques a fundamental problem of environment perception and modeling must be addressed. Indeed, the model of the environment and the location model are the two basic data for position estimation, path planning, and all other navigation tasks involving interaction between the robot and its environment. 3.1. Spatial Representation Using preloaded maps and absolute referencing systems can be impractical because they constrain the robot’s navigation to a limited, static, and structured environment. In this section we survey the various approaches for map making and position estimation using trajectory integration and dead reckoning and evaluate their relative merits. The various approaches are influenced by the environment in which the robot navigates and the type of sensing used. Some approaches try to reason away errors and uncertainties to simplify the map-making process, while others take explicit account of errors and uncertainties using either static error factors or stochastic approaches that use probability distributions to model the errors. Most of the methods deal with an indoor factory or office-type environment made up of walls, corridors, and other man-made obstacles in which the robot navigates. Map-making in an outdoor scenario is a much more complex problem which relies on the existence of landmarks and digital elevation maps of the area. The different methods of mobile robot map-making studied so far are quite varied and differ chiefly in terms of: The environment in which the mobile robot is to navigate. The map-making strategies for an indoor ofice type robot differ significantly from those of a n outdoor terrain autonomous land vehicle.
778 R. Talluri Ed J. K . Aggarwal 0
0
0
The type of world representation (either 2-D or 3-D). Most methods consider a 2-D representation or a flopr map type approach. Since the mobile robot is essentially interested only in obstacle avoidance and path planning, a map of the vacant/occupied areas of the floor should suffice for these tasks. The types of sensors used. To a certain extent, the sensing modality affects the mapping strategies used. Typically, all mobile robots use some kind of range sensor. If a passive range sensor such as binocular stereo is used, the map so constructed will usually be sparse and feature-based. On the other hand, a laser range finder gives dense, high resolution depth estimates, which affect the mapping strategy differently. Sonar-based range finding techniques give less accurate and hence more uncertain depth estimates, so the map-making technique used should have the capability to deal with these uncertain readings. The navigational tasks to be accomplished by the robot. Most mobile robots consider tasks such as position estimation, obstacle avoidance, and path planning.
Keeping in view the above differences, map-making approaches can be broadly classified into the following four types: (1)object feature-based methods; (2) graphbased approaches; (3) certainty grid-based approaches; and (4)qualitative methods These categories are not exacting, since some approaches do not fit into any of the categories and some have properties of more than one approach. However, such a classification may help put things in a better perspective. 3.2. Object Feature-Based Methods
In these methods, object features detected from the sensory observations in one of the robot’s positions are used to form the world model. As the robot moves, these features are again detected, and correspondence is established between the new and the old features. Usually the motion of the robot is known to a certain degree of accuracy as given by its position sensors. These motion estimates are then used to predict the occurrence of the new positions for the features in the world model. The prediction is then used as an aid to limit the search space and to establish a correspondence between the detected features and those already in the current world model. A mechanism to consistently update the world model is also provided when new features are detected. One significant advantage of the object feature-based methods is that after the position sensors are used to establish correspondence, the motion parameters of the robot between the old and the new positions can be solved for explicitly. The solution provides a much more accurate estimate of the robot’s position. The loop then continues, and the world model is continuously updated. The type of sensing used is typically stereo triangulation or other types of visual sensing [38,39,33,43]. Crowley [8] uses a ring of 24 sonar sensors for a similar paradigm. Moravec’s Cart [38] was one of the first attempts at autonomous mobile robot navigation using a stereo pair of cameras. He defines an interest operator to locate the features in a given image. Essentially, the interest operator picks regions that
4.4
Position Estimation Techniques for an Autonomous Robot
779
are a local maxima of a directional variance and uses these to select a relatively uniform scattering of good features over the image. A coarse to fine correlation strategy is used to establish correspondence between the features selected by the interest operator between different frames. The Cart uses a unique variable baseline stereo mechanism called slider stereo. At each pause, the computer slides its camera left to right on a 52 cm track, taking nine pictures a t 6.5 cm intervals. A correspondence is established by using a coarse to fine correlation operator between the central image and the other eight images, so that the features’ distance is triangulated in the nine images. These are then considered as 36 stereo pairings and the estimated (inverse) distance of the feature is recorded in a histogram. The distance to the feature is indicated by the highest peak in the histogram if it crosses a given threshold; otherwise, it is forgotten. Thus, the application of a mildly reliable (correlation) operator is used to make a very reliable distance measurement. Position estimation in the Cart is carried out in exactly the same manner as described before, i.e. the features used t o establish correspondence are then used to estimate the motion parameters and, hence, the location. The world model developed by the Cart is a set of these matched object features. The uncertainty and error modeling of the object features used in the Cart is a simple scalar uncertainty measure. This measure was proportional to the distance of the feature from the robot location; the further the feature, the larger the error associated with it, and, hence the less reliable the measure. Matthies and Shafer [33] show that Moravec’s approach is very similar to using a spherical probability distribution of error centered around the object feature. They argue that a 3-D Gaussian distribution is a much more effective way t o explicitly deal with stereo triangulation errors and errors due to the image’s limited resolution. They detail a method to estimate the 3-D Gaussian error distribution parameters (mean and covariance) from the stereo pair of images. They then present a method to consistently update the robot’s position, explicitly taking into account the Gaussian error distribution of the feature points and motion parameters and their error covariances. A Kalman filter approach is used t o recursively update the robot position from the detected features and the previously maintained world model. They assume the correspondence problem to be solved, and show by simulation data and experimental results that the Gaussian error model results in a more accurate stereo navigation paradigm. The Cart suffers from the requirement of a huge memory to store the object features, the lack of speed (typically it moves in lurches of 1meter in 10 to 15 minutes), and errors in position estimation due to insufficient error modeling. However, as one of the first autonomous mobile robots, it performed very well and made clear the various problems associated with autonomous navigation. The CMU Rover dealt with and corrected many of these problems [39]. Faugeras and Ayache [14,13,1]also address the problem of autonomous navigation. They use trinocular stereo to detect object features. The features they use are
780
R. Talluri & J. K. Agganual
3-D line segments. They propose a paradigm to combine coherently visual information obtained at different places to build a 3-D representation of the world. To prevent the system using line segments as primitives from running out of memory, they want their system t o “forget intelligently” i.e. if a line segment “S” is detected at different positions 1, 2, 3 , . . . , n of the robot as S1, Sz, . . . ,S,, they want to establish a correspondence between all these, to form the line segment S from them, and to forget all others. Thus, the end result is a representation of the environment by a number of uncertain 3-D line segments attatched to co-ordinate frames and related by a n uncertain rigid motion. The measurements are combined in the presence of these uncertainties by using the Extended Kalman Filtering technique. The authors present these ideas, and detail with experimental data the technique for building, registering, and fusing noisy visual maps. A framework presented by Smith and Cheesman [46] for the representation and estimation of position uncertainty is relevant in this context. They describe a general method for estimating the nominal relationships and expected error (covariance) between coordinate frames representing the relative locations of objects. They introduce the concept of Approximate Transformations (ATs) consisting of an estimated mean of one co-ordinate frame relative to another and an error co-variance matrix that expresses the uncertainty of the estimate. They present two basic operations that allow the estimation of the relationship between any two coordinate frames given another relative transformation linking them. The first, Compounding, allows a chain of ATs to be collapsed (recursively) into a single AT. The final compounded AT has a greater uncertainty than its components. The second operation, Merging, combines information from parallel ATs t o produce a single resulting AT with an uncertainty less than either of its components. Crowley [8,9] has a similar approach to Ayache and Faugeras [14]; he also uses a line segment based representation of the free space using Extended Kalman filtering techniques for dealing with error covariances. However, he uses a circular ring of 24 Polaroid ultrasonic sensors, while Faugeras and Ayache use trinocular stereo. 3.3. Graph-Based Approaches Rodney Brooks [3] was one of the first to suggest the idea of using a relational map, which is rubbery and stretchy, rather than place observations in a 2-D coordinate system. The key idea is to represent free space as freeways, elongated rectangular regions, which naturally describe a large class of collision-free straight line motions of objects to be moved. Some places are described as convex regions and called meadows. So a map representation of free space is a graph; nodes of the graph are meadows; and arcs of the graph are freeways. Meadows and freeways are further described with metric and relational position and orientation properties. Brooks also suggests a method for dealing with uncertainties in the position of the robot. He argues that the position uncertainty is a 2-D manifold in a 3-D
4.4
Position Estimation Techniques for an Autonomous Robot
781
space and that dealing with this explicitly makes it mathematically complex. He proposes to use instead a n upper bound on the uncertainty which is cylindrical and mathematically easier t o handle. Brooks also suggests how the uncertainty in the position estimation can be reduced if landmarks can be detected in a meadow. Miller [37] presents a surface representation for real-world indoor robots equipped with a ranging device such as sonar and robot odometry as sensors. He assumes that the world consists of a flat, open plane on which walls and obstacles are placed. Since the robot is limited to motion on the plane of the floor, the projection of walls and obstacles on the plane of the floor captures all the relevant world information. The basic unit of the spatial representation system is the map, composed of linked regions. Regions have a local coordinate frame. Walls and obstacles are themselves represented by line segments, whose end point positions are designated by coordinates in the frame of the region. The borders of the regions are marked with labels that specify the adjoining regions. Regions can be of four types, 0-F, 1-F, 2-F, and 3-F, since a floor dwelling mobile robot has three degrees of freedom, two translational (x and y), and one rotational (orientation 0). A type designation of j-F means that a sensor (here a sonar range sensor with a maximum range of Dmax)can be used t o eliminate j degrees of freedom. Regions are made up of a set of edges. Each edge is represented by a pair of end points, whose Cartesian coordinates are specified in the frame of reference of a particular region. The relative positions of features in two different regions cannot be known with great precision. The more 0-F, 1-F, and 2-F regions on the path between the two regions in question, the less the accuracy with which the two regions can be related. It is, however, possible to arrive at a n approximate idea of the distance t o be traveled between regions by using the lower bounds of the regions. Having set the mapping scheme, Miller then presents methods for position estimation as a heuristic search paradigm. The type of region in which the robot operates determines the amount of position information that can be calculated. If the robot is in a n 0-F region, then the only position information available would be extrapolations from the last known position, based on the robot's ability t o do dead reckoning. If the robot is known t o be in a region that is 1-F or greater, then position information can be found by taking several sensor readings and conducting a heuristic search over the tree of possible matches between the observations and the edges in the map. Chatila and Laumond [5] present a world modeling and position referencing system on their mobile robot HILARE. They take a multisensor approach using a laser range finder for measuring depth and optical shaft encoders on the drive wheel axis for the trajectory integration. The random errors are modeled as Gaussian distributions and their parameters are determined experimentally. They present a three-layer model consisting of geometric, topological, and semantic levels. In the model construction paradigm, the robot at every instance has:
782 R . Talluri 6 J. K. Aggamal
(1) a current environmental model with geometric, topological, and semantic levels related to an absolute reference frame, (2) knowledge about the attitude and position of the robot, and (3) a robot-centered geometric model of the environment perceived at that point. The central problem is to update the models of (1) using (2) and (3), and to correct the information of (2), if possible. 3.4. Certainty Grid-Based Methods
Moravec and Elfes [11,40] use a grid-based representation for mapping the environment a mobile robot will inhabit. The basic idea is to represent the floor as a rectangular grid and to store the information about the occupancy of different portions of the floor on this grid as probability distributions. A sensor range reading provides information concerning empty and occupied volumes in a cone in front of the sensor. The readings are modeled as probability profiles and are projected onto a rasterized 2-D map where somewhere occupied and everywhere empty regions are represented. Range measurements from multiple points of view (taken from multiple sensors on the robot and from the same sensor after the robot moves) are symmetrically integrated into the map. Overlapping empty volumes reinforce each other and serve to condense the range of the occupied volumes. The map definition improves as more readings are added. The final map shows regions probably occupied, probably empty, and unknown areas. The method deals effectively with clutter and can be used for motion planning and extended landmark recognition. The system was tested and implemented on a CMU mobile robot called Neptune. The authors also develop and present a fast algorithm for relating two maps of the same area to determine relative displacement, angle, and goodness of the match. These can then be used to estimate the position and pose of the robot. A measure of the goodness of the match between two maps at a trial displacement and a rotation is found by computing the sum of products of corresponding cells in the two maps. An occupied cell falling on an occupied cell contributes to a positive increment to the sum, as does an empty cell falling on an empty cell. An empty cell falling on an occupied one reduces the sum, and any comparison involving an unknown value causes neither an increase nor a decrease. Moravec and Elfes then offer more efficient versions of this naive algorithm, which take into account only the occupied cells and also use a hierarchy of reduced resolution versions of each map. The authors argue that the advantages of the sonar maps are that they : (1) are much denser than stereo maps, (2) require less computation, (3) can be built more quickly, and (4)can be used for position estimation. Of course, the disadvantages of sonar maps are that the large uncertainty areas associated with the features detected and the difficulties associated with active sensing.
4.4
Position Estimation Techniques for an Autonomous Robot
783
Moravec [40,41] presents a new Bayesian statistical foundation for the mapmaking strategies in the certainty grid framework which seems to hold promise. The fundamental formula used is for the two occupancy cases of a cell o (cell is occupied) and 6 (cell is empty) with prior likelihoods p(o) and p ( 6 ) and new information M ; Bayes’ theorem can be expressed as
The new information, M , occurs in terms of the probability of M in the situation that a cell is or is not occupied, i.e., P ( M / o ) and P ( M / B ) , respectively. This inversion of o and M is the key feature of using the Bayesian framework, and it combines independent sources of information about o and M into a single quantity P ( o / M ) . Moravec then elaborates on this principle and derives formulas for the various cases of multiple sensor readings and presents a Context-free and ContextSensitive method. The former is much faster, but the latter is much more reliable. The former has a linear cost while the latter has a cost proportional to the cube of the volume. These methods are illustrated by simulations. The certainty grid representation also provides an easy framework for fusing information from different sensing modalities, such as sonar, stereo, thermal, proximity, and contact sensors. Matthies and Elfes [34] present several approaches and results in integrating sonar and stereo in a certainty grid.
3.5. Qualitative Methods Levitt et al. [30] and Kuipers et al. [26] argue that the existing robot navigation techniques use absolute range information and, hence, tend to be brittle, to accumulate error and to use little or no perceptual information. They propose qualitative methods which do not depend as much upon metrical information as on perceptual information to build a topological map. Levitt et al. [30] describe a formal theory that depends on visual landmark recognition for the representation of environmental locations. They encode perceptual knowledge in structures called wiewframes. Paths in the real world are represented as a sequence of sets of landmarks, viewframes, and other distinctive visual events. Approximate headings are computed between viewframes that have their lines of sight to common landmarks. Range-free, topological descriptions called orientation regions are rigorously abstracted from viewframes to yield a coordinate-free model of the visual landmark memory that can also be used for navigation and guidance. With this approach, a robot can opportunistically observe and execute visually cued short-cuts. Map and metric information are not required but, if available, are handled in a uniform representation with the qualitative navigation technique. Most of the examples they present are of simulated outdoor scenes using a visual sensor. Kuipers et al. [26] present an approach similar in spirit for an indoor robot with a sonar sensor. They draw a parallel from cognitive science and argue that a powerful description of the environment is a topological description. Their topological
784
R. Talluri €d J. K. Aggamal
description consists of a set of nodes and arcs. The nodes represent distinctive places and the arcs represent travel edges connecting them. A distinctive place is defined as the local maximum of some measure of distinctiveness appropriate to its immediate neighborhood and is found by a hill climbing search. Local travel edges are described in terms of local control strategies required for travel. How to find the distinctive places and how to follow edges is the procedural knowledge which the robot learns dynamically during the exploration stage and which guides the robot in the navigation stage. An accurate topological model is created by linking places and edges, and allows metrical information to be accumulated with reduced vulnerability to metrical errors. The authors describe a simulated robot called NX to illustrate the technique. The position estimation strategies that use trajectory integration and dead reckoning thus rely on the robot’s ability to sense the environment and to build a representation of it, and to use this representation effectively and efficiently. Each of the approaches detailed above has relative merits and works well in different environments. The sensing modalities used significantly affect the map-making strategy. Error and uncertainty analyses play an important role in accurate position estimation and map building. It is important to take explicit account of the uncertainties; modeling the errors by probability distributions and using Kalman Filtering techniques are good ways to deal with these errors explicitly. Qualitative methods propose to overcome the brittleness of the traditional approaches by relying on perceptual techniques. In general, a 2-D floor map of the environment is good enough for most navigation problems such as path planning and position estimation. This approach conserves memory and is easier to build. Certainty grid-based methods are novel and use the Bayesian probability techniques to advantage in combining information from various viewpoints consistently. 4. Techniques Using a Standard Pattern Another method of estimating the position and pose of the mobile robot accurately is to place standard patterns in known locations in the environment. Once the robot detects these patterns, the robot’s position can be estimated from the known location of the pattern and its geometry. The pattern itself is designed to yield a wealth of geometric information when transformed under the perspective projection. Ambiguous interpretations are avoided, and a minimum of a priori knowledge about the camera is desirable. These methods are particularly useful in those applications where a high degree of accuracy in the positioning of the robot is required only after it is near a particular workstation. Simple trajectory integration systems could be used to locate the robot near the work station. Then by identifying the mark (standard pattern) located near the workstation, the robot can be positioned more accurately. Researchers have used different kinds of patterns or marks, and the geometry of the method and the associated techniques for position estimation vary accordingly.
4.4
Position Estimation Techniques for an Autonomous Robot
785
circular arc with constant distance r from the origin
\
A
0
B
circular arc with constant angular width y formed by the segment AB
Fig. 6. Determination of point Q by r and $J.
Fukui [18] uses a square mark rotated by 45 degrees. As the robot moves on the floor, h k u i determines the position of the robot by two co-ordinates, r and p , where r is the distance between the standard point and the robot and p is an angle between the r vector and the normal line t o the mark. Figure 6 shows this situation. Two circles are drawn, one with the center at the standard point and with a radius of r , the other with an arc of a constant visual angle $J made by watching a segment A B on the mark. Generally these two circles intersect a t two points, and it is easy to judge in another way which is the real point. However, if $J is a right angle, the two circles become the same and the position cannot be determined. The height of the camera is adjusted to the square mark ACBD, and it is imaged. If 8 is the visual angle made by viewing CD in the square and $J is that made by viewing AB, then the robot position in polar co-ordinates ( p , r ) can be determined from the following relations: r=w
+
(1 case) sin 19
p = farctan
f i r 2 + w ~ ) ( c o s $-J ()r~2- w2)2 (r2 - w2) sin $J
where A B is equal to CD,which equals 2w in length, and r # w (see Fig. 7). To know the sign of p , Fukui measures the two angles in the image which correspond to LCAD and LCBD, and decides if LCBD 5 LCAD, p 5 0, or else p > 0. The angles $J and 8 are determined as follows: Using all the data of the image and the method of least squares, the equations of the lines AC, AD, C B , and DB are determined on the image. Then the four points A, B , C,D are determined as the points of intersections of these lines. Next, assuming that $J is proportional to the corresponding length A B on the image and
786 R. Talluri d J. K . Agganual Rhombusmark
C
D C
Tv
Fig. 7. Diagram to measure the distance from the origin.
that 8 is proportional to CD, Fukui determines the proportional coefficients that can be used to convert the measured lengths into the desired angles. Fukui also outlines image processing techniques to extract the lines of the pattern from the images. In addition, he presents experimental results to determine the camera position using this method and discusses the effects of errors in the measurement of the angles 8 and $J on the position @, r ) . Courtney, Magee, and Aggarwal [7] use the same mark as Fukui but relax the constraint of having the lens center at the same height as the mark center by partitioning the problem into two planes. Each plane passes through the lens center and either the vertical or the horizontal diagonal. Their results initially yield two equations in three unknowns, which must be further constrained by adding a second mark at a known height above the original mark or by assuming that the height of the camera relative to the mark is known. Since adding the second mark forces the solving of a system of six nonlinear equations, they opt for the later solution, which involves straightforward substitution. Magee and Aggarwal [32] consider the use of a standard mark which would always directly produce at least one of the three position parameters (distance, elevation, or azimuth) and whose geometric properties would be such that its basic shape would be unchanged when its center is viewed along the optical axis. A sphere is such an object, and its projection is always a circle whose radius may be used to determine the distance. On the other hand, an unmarked sphere produces no information regarding the orientation, and so horizontal and vertical great circles are added to the sphere for computing the elevation and azimuth. The resulting selflocator system is mathematically quite simple. The preprocessing stage requires that four values be determined. These are the center and radius of the sphere’s projected circle and the co-ordinates of the points on the projections of the great circle that are closest to the center of the sphere’s outline. The three position
4.4
Position Estimation Techniques for an Autonomous Robot
787
Fig. 8. The robot locator sphere.
Fig. 9. Geometry for finding the distance to the center of the sphere.
estimation parameters used are: (1) the distance of the lens center D , (2) the elevation angle 4 of the lens center above the horizontal great circle, and (3) the azimuth angle 8 of the lens center with respect to the plane of the vertical great circle. The value of D can be computed from the relation
where f is the focal length of the camera, R is the radius of the sphere, and r is the radius of the circular projection of the sphere on the image plane (see Figs. 8 and 9 for details). Similarly, the authors give relations to determine the azimuth and the elevation angles from the projections of the great circles. The preprocessing used to extract these primitives from the images are also discussed and experimental results in estimating the position are shown. From the error analysis presented, the authors show that the errors in the computed distance increase as the camera is moved farther from the sphere and the errors in the computed angles increase as their respective great circles approach the edge of the sphere. This method is robust as long as the primary features are not lost in the sphere’s shadow. Drake et al. [lo] present a method of estimating the position and pose of a mobile robot using a Navigation Line, for use in factory environments. The navigation line is a long line with parallel edges on the floor, that does not intersect other lines.
788
R. Talluri & J. K. Aggarwal
z t \ \uI
T
Fig. 10. The sensor geometry.
Boreslght of the sensor
x =x-
I
x=x+
x=o
Fig. 11. Mobile robot with -0 orientation angle error and
-20
x-shift position error.
Navigation Line
I-
_.
-xO
Fig. 12. Global coordinates of the navigation line.
4.4
Position Estimation Techniques for an Autonomous Robot
789
Once the line is imaged and detected by the robot, the position of the robot with respect to the line can be easily computed. The geometry used by the authors, illustrated in Figs. 10, 11, and 12 below, explains the method. 8 is the pan angle and is the tilt angle of the sensor. Two coordinate systems are shown; the unprimed coordinates (5,y, z ) represent the global coordinate system and the primed coordinates (x’, y’) represent the coordinate system of the image plane. In Fig. 10, the gimbal center coincides with the focal point and is centered on the xy plane, and z = 0 is defined as the ground plane so that the sensor is at zo. The authors also assume that the sensor is centered on the navigation line so that the pan angle 0 = 0. Since the navigation line is coincident with the y-axis, the line has the co-ordinates as shown in Fig. 11. The robot’s position may be described by two parameters: the lateral position along the x-axis between the sensor and the navigation line (the x-shift) denoted by 20, and the angle between the robot’s orientation vector (the direction of travel) and the navigation line denoted by 8. The authors develop relations for these two parameters in terms of the focal length of the camera f, the image plane co-ordinates of the edges of the line x’,and z’, given below. x=xo-
zox’ cos 8
+ (z’ sin d, - f cos d,) (zo sin 8) z‘ cosd, + f sin4
(4.4)
While this equation is only one equation in two unknowns, by using a number of (x’, 2‘) points along the line and using numerical techniques, the values of x and 0 can be solved for quite accurately. The authors also present a specialized operator to detect edges in an image that occur at a specific angle, and use the operator to detect the edges of the navigation line. In addition a Hough transform is used to completely segment the navigation line from the image. The authors also present experimental results to illustrate the robustness of the method. Kabuka and Arenas [23] consider the problem that the robot might end up in a position that will not allow it to view the standard pattern. To alleviate the problem, they suggest using multiple patterns in the navigation environment. It is assumed that the location of each pattern in some standard world coordinate system is known. They associate a unique code with each pattern that will enable the robot to identify and distinguish that pattern from all the others. It consists of two parts: a relative displacement part and an identification code. The relative displacement pattern is used, as in the previous methods, to obtain the relative position of the viewing point with respect to the pattern by analysis of the particular geometric characteristics of its projection onto the image plane. The identification codes serve two purposes: they provide a unique code to discern the viewed pattern from other patterns in the environment, and they provide an aid to scan for the pattern in a minimal amount of time. The displacement pattern used by the authors is a circle, and the identification codes used are similar to bar codes. The authors present a detailed analysis of the application of this method and study the effects of errors in the input parameters on the position estimation.
3 J . K. Agganual 790 R. Talkuri 6
5 . Model-Based Approaches
Some researchers consider the problem of the position estimation of a mobile robot when a priori information is available about the environment in which the robot is to navigate. This could be provided in terms of a CAD model of the building (or a floor map, etc.) in the case of a n indoor mobile robot, or a Digital Elevation Map (DEM) in the case of an outdoor robot. In these cases, the position estimation techniques used take on a different flavor. The basic idea is, of course, t o sense the environment using onboard sensors on the robot and t o match these sensory observations to the preloaded world model t o arrive at an estimate of the position and pose of the robot with a reduced uncertainty. One problem with such an approach is that the sensor readings and the world model may be in different forms. For instance, given a CAD model of the building and a visual camera, the problem is to match the 3-D descriptions in the CAD model to the 2-D visual images. This is the problem addressed by Kak et al. [24]. They present PSEIKI, a system that uses evidential reasoning in a hierarchical framework for image interpretation. They discuss how the PSEIKI system can be used for mobile robot self-location and how their approach is utilized by the navigational system of the autonomous mobile robot PETER. The robot’s position encoders are used to maintain an approximate estimate of its position and heading at each point. However, to account for errors in the quantization effects of the encoders and the slippage of the wheels, a visual sensor in conjunction with a CAD model of the building is used to derive a more accurate estimate of the robot’s position and pose. The basic idea is that the approximate position from the encoders is utilized t o generate, from the CAD model, an estimated visual scene that would be seen. This scene is then matched against the actual scene viewed by the camera. Once the matches are established between the features of the two images (expected and actual), the position of the robot can be estimated with a reduced uncertainty. Tsubouchi and Yuta [59]discuss the position estimation techniques used in their YAMABICO robot, which use a color camera and a map of the building in which the robot navigates. The authors propose a vision system using image and map information with consideration of real time requirements. This system consists of three operations. The first operation is the abstraction of a specified image from a TV camera. The image is processed, and highly abstracted information, called the real perspective information, is generated. The second operation is the generation of the estimated perspective information by coordinate transformation and map information, using information about the robot’s position and direction. The third operation is the establishment of correspondence between the two perspectives. The authors use color images in their real perspective views. They argue for color images, saying that they are invariant under lightness and shadow. From the color images, the authors extract regions of similar color and fit trapezoids t o these regions. From the map information, trapezoids are also extracted and, in the matching process, these trapezoids from the two sources are used as matching primitives.
4.4
Position Estimation Techniques for an Autonomous Robot
791
The authors provide a method of representing the map information efficiently and also discuss techniques for matching trapezoids. Real image data are provided as examples. As pointed out earlier, one of the key issues involved in determining the position of a mobile robot given a world model is to establish a correspondence between the world model (map) and the sensor data (image). Once this correspondence is established, the position of the robot in the environment can be determined easily as a coordinate transformation. Indeed, this problem of imagelmap correspondence is of fundamental importance not only to the mobile robot position estimation problem, but also to many other computer vision problems, such as object recognition, pose estimation, airborne surveillance and reconnaissance, etc. Other work addressing this image/map correspondence problem is described in [30,24,36,16,17,44]. Freeman and Morse [17] consider the problem of searching a Contour Map for a given terrain elevation profile. Such a problem is encountered, for example, when locating the ground track of an aircraft (the projection of the flight path on the ground) given the elevation of the terrain below the aircraft during the flight. The authors describe a solution that takes advantage of the topological properties of the contour map. A graph of the map topology is used to identify all the possible contour lines that would have been intersected by the ground track. So, the topological constraints of the terrain elevation profile and the geometric constraints of the flight path are used in estimating the location of the elevation profile in the given map. Ernst and Flinchbach [16] consider the problem of determining the correspondence between maps and the terrain images in low altitude airborne scenarios. They assume that an initial estimate of the three-dimensional position is available. Their approach consists of partially matching the detected and expected curves in the image plane. Expected curves are generated from a map using the estimate of the sensor position and the simulated curves are matched with the curves in the image plane. Rodriguez and Aggarwal [44] consider the problem of matching aerial image to a Digital Elevation Map (DEM). They use a sequence of aerial images to perform stereo analysis on successive images and recover an elevation map. Then they present a method to match the recovered elevation map to the given DEM and thereby estimate the position and pose of the airborne sensor. Talluri and Aggarwal [50-521 describe a position estimation technique for autonomous mobile robots navigating in outdoor, mountainous environment equipped with a visual camera that can be panned and tilted. A DEM of the area in which the robot navigates is provided to the robot. The robot is also assumed to be equipped with a compass and an altimeter to measure the altitude. Typical applications could be that of an autonomous land vehicle, such as a planetary rover. The approach presented formulates the position estimation problem as a constrained search problem. The authors also follow the idea of computing the expected image and comparing it to the actual image. In particular, the main idea of their work is to hypothesize a robot location, render the model (DEM) data, extract the Horizon Line Contour (HLC) and compare it to the HLC extracted from the camera images. In order
792
R. Talluri €9 J. K . Aggarwal
to reduce the complexity of the search, the authors propose a two stage search strategy. First, all possible camera locations are checked by comparing the predicted (from the DEM) HLC height at the center of the image with the height computed from the camera image. This is done in the four geographic directions (N, S, E, and W). Only the candidate locations with the HLC height within some threshold of the actual height remain. Second, for each remaining candidate location the terrain image is rendered and the complete HLC is extracted and matched with the actual HLC (from the camera image). Examples of the position estimation strategy using real terrain data and simulated images are presented. The algorithm is made robust to errors in the imaging process by accounting for the worst case errors. In a separate work, Talluri and Aggarwal [53-571 also consider the navigational aspects of an autonomous mobile robot navigating in an outdoor, urban environment consisting of polyhedral buildings. The 3-D descriptions of the rooftops of the buildings are assumed to be given as a world model and the robot is assumed to be equipped with a visual camera. The position and pose are estimated by establishing a correspondence between the lines that constitute the rooftops of the buildings (world model features) and their images. A tree search is used to establish a set of consistent correspondences. The tree is pruned using the geometric constraints between the world model features and their images. To effectively capture the geometric relations between the world model features with respect to their visibility from various positions of the robot, the free space of the robot is partitioned into a set of distinct, non-overlapping regions called the Edge Visibility Regions (EVRs). Associated with each EVR is a list of the world model features that are visible in this region called the Visibility List (VL). Also stored for each entry in the VL is a range of orientations of the robot for which this feature is visible. The uses of these EVRs in pruning the tree in searching for a consistent set of correspondences between the world model and the image features is discussed in this paper. An algorithm for forming such an EVR description of the environment from the given world model is presented. The authors also derive worst case bounds on the maximum number of EVRs that will be generated for a given world model and show that this is polynomial in the number of world model features. The uses of this EVR description in the path-planning tasks of the robot are also outlined. Results of the position estimation are provided using a model of a real airport scene. 6. Conclusions
In this paper we have illustrated the various aspects of the problem of estimating the position and pose of a mobile robot and provided a comprehensive review of the various methods and techniques used. These techniques vary significantly depending on the known conditions of the navigation environment and the type of sensors with which the robot is equipped. Landmark-based methods are suitable for robots with the ability to identify the landmarks and measure the rangelattitude
4.4
Position Estimation Techniques for an Autonomous Robot
793
to them. This usually requires the robot to have a database of landmarks occurring in the environment and a n approximate location t o start to search the database. Computing the exterior orientation parameters in the camera calibration problem is dealt with in the photogrammetry literature. This problem is quite similar t o the position estimation problem of a mobile robot using landmarks. Some of the techniques used in photogrammetry can thus be modified and applied in localizing the robot’s position and orientation. Methods using trajectory integration and dead reckoning usually require the robot to address the problem of environment perception and modeling. Various methods of modeling the environment and forming a map of it for navigation tasks are also reviewed in this paper. The position estimation techniques used depend on the map-making strategy and representation used. Techniques which use a standard pattern to structure the environment by placing a standard reference pattern at known locations in the environment are particularly useful in those applications where a high degree of accuracy in positioning the robot is required only after it nears a particular workstation. Simple trajectory integration techniques could be used to locate the robot near the workstation, and then the standard pattern can be used. Model-based methods are best applied when a priori information of the robot’s environment is available in the form of a world model. The problem t o be solved in this instance is t o match the model and the sensor observations, which may be in different forms.
Acknowledgments This research was supported by the Army Research Office under contract DAAL03-91-G-0050.
References [l]N. Ayache, 0. D. Faugeras, Building a consistent 3-D representation of a mobile robot environment by combining multiple stereo views, in Proc. 10th IJCAI, 1987, 808-810. [2] B. C. Bloom, Use of landmarks for mobile robot navigation, in SPIE Proc., Intelligent Robots and Computer Vision, Vol. 579, 1985, 351-355. [3] R. A. Brooks, Visual map making for a mobile robot, in Proc. IEEE Int. Conf. on Robotics and Automation, St. Louis, MO, 1985, 824-829. [4] M. Case, Single landmark navigation by mobile robots, in SPIE Proc., Mobile Robots, Vol. 727, Oct. 1986, 231-238. [5] R. Chatila and J.-P. Laumond, Position referencing and consistent world modeling
for mobile robots, in Proc. IEEE Int. Conf. on Robotics and Automation, St. Louis, MO, 1985, 138-145. [6] H. H. Chen, Pose determination from line-to-plane correspondences: Existence condition and closed-form solutions, IEEE Trans. Pattern Anal. Mach. Intell. 13,6 (1991) 53ck541. [7] J. Courtney, M. Magee and J. K. Aggarwal, Robot guidance using computer vision, Pattern Recogn. 17,6 (1984) 585-592. [8] J. L. Crowley, Dynamic world modeling for an intelligent mobile robot using a rotating
ultra-sonic ranging sensor, in Proc. IEEE Int. Conf. on Robotics and Automation, St. Louis, MO, 1985, 128-135.
794
R. Talluri €9 J. K. Aggarwal
[9] J. L. Crowley, World modeling and position estimation for a mobile robot using ultrasonic ranging, in Proc. IEEE Int. Conf. on Robotics and Automation, Scottsdale, May 1989. [lo] K. C. Drake, E. S. McVey and R. M. Iiiigo, Experimental position and ranging results for a mobile robot, IEEE Trans. Robotics and Automation 3, 1 (1987) 31-42. [ll] A. Elfes, Sonar based real-world mapping and navigation, IEEE Trans. Robotics and Automation 3, 3 (1987) 249-265. [12] I. M. El Hassan, Analytical techniques for use with reconnaissance from photographs, Photogrammetric Eng. Remote Sensing 47, 12 (1981) 1733-1738. [13] 0. D. Faugeras, N. Ayache and B. Faverjon, Building visual maps by combining noisy stereo measurements, in Proc. IEEE Conf. on Robotics and Automation, San Francisco, CA, 1986, 1433-1438. [14] N. Ayache and 0. Faugeras, Maintaining representations of the environment of a mobile robot, IEEE Trans. Robotics and Automation 5, 6 (1989) 804-819. (151 M. A. Fischler and R. C. Bolles, Random sample consensus : A paradigm for model fitting with application to image analysis and automated cartography, Commun. ACM 24, 6 (1981) 726-740. [16] M. D. Ernst and B. E. Flinchbaugh, Image/map correspondence using curve matching, Texas Instruments Technical Report, CSC-SIUL-89-12, 1989. [17] H. Freeman and S. P. Morse, On searching a contour map for a given terrain elevation profile, Journal of the Franklin Institute 284 (1967) 1-25. [18] I. Fukui, TV image processing to determine the position of a robot vehicle, Pattern Recogn. 14, 1-6 (1981) 101-109. [19] S. Ganapathy, Decomposition of transformation matrices for robot vision, in Proc. 1st IEEE Int. Conf. on Robotics, Atlanta, GA, Mar. 1984, 130-138. [20] R. M. Haralick et al., Pose estimation from corresponding point data, IEEE Trans. Syst. Man Cybern. 19, 6 (1989) 1426-1445. [21] R. Horaud, B. Conio and 0. Leboulleux, An analytical solution to the perspective 4-point problem, in Proc. IEEE Conf. on Computer Vision and Pattern Recognition, CVPR '89, San Diego, CA, Jun. 1989, 500-507. [22] B. K. P. Horn, Relative orientation, Proc. Image Understanding Workshop, Vol. 2, 1988, 826-837. [23] M. R. Kabuka and A. E. Arenas, Position verification of a mobile robot using a standard pattern, IEEE Trans. Robotics and Automation 3, 6 (1987) 505-516. [24] A. Kak, K. Andress and C. Lopez-Abadia and M. S. Carroll, Hierarchical evidence accumulation in the PSEIKI system and experiments in model-driven mobile robot navigation, in Uncertainty in Artificial Intelligence, Vol. 5 (Elsevier Science Publishers B.V., North-Holland, 1990) 353-369. [25] E. Krotkov, Mobile robot localization using a single image, in Proc. IEEE Int. Conf. on Robotics and Automation, Scottsdale, May 1989, 978-983. [26] B. J. Kuipers and Y. T. Byun, A robust qualitative method for robot spatial learning, in AAAI-88, The Seventh Nat. Conf. on Artificial Intelligence, St. Paul/Minneapolis, MI, 1988, 774-779. [27] R. Kumar, Determination of the camera location and orientation, in Proc. DARPA Image Understanding Workshop, 1988, 870-881. [28] R. Kumar and A. Hanson, Robust estimation of the camera location and orientation from noisy data having outliers, in Proc. Workshop on Interpretation of 3-d scenes, Austin, TX, Nov. 1989, 52-60.
4.4
Position Estimation Techniques f o r an Autonomous Robot
795
[29] J. Lessard and D. Laurendeau, Estimation of the position of a robot using computer vision for a live-line maintenance task, in Proc. IEEE Int. Conf. on Robotics and Automation, Raleigh, NC, 1987, 1203-1208. [30] T. S. Levitt, D. T. Lawton, D. M. Chelberg and P. C. Nelson, Qualitative navigation, in Proc. DA RPA Image Understanding Workshop, 1987, 447-465. [31] Y. Liu, T. Huang and 0. Faugeras, Determination of the camera location from 2-d to 3-d line and point correspondences, IEEE Trans. Pattern Anal. Mach. Zntell. 12, 1 (1990) 28-37. [32] M. J. Magee and J. K. Aggarwal, Determining the position of a robot using a single calibration object, in Proc. 1st IEEE Int. Conf. on Robotics, Atlanta, GA, Mar. 1984, 140-149. [33] L. Matthies and S. A. Shafer, Error modeling in stereo navigation, IEEE Trans. Robotics and Automation 3 (1987) 239-248. [34] L. Matthies and A. Elfes, Integration of sonar and stereo range data using a grid based representation, in Proc. ZEEE Int. Conf. on Robotics and Automation, Philadelphia, PA, Apr. 1988, 727-733. [35] C. D. McGillem and T. S. Rappaport, Infra-red location system for navigation of autonomous vehicles, in Proc. IEEE Int. Conf. on Robotics and Automation, Philadelphia, PA, Apr. 1988, 1236-1238. [36] G. Medioni and R. Nevatia, Matching images using linear features, IEEE Trans. Pattern Anal. Mach. Zntell. 6, 6 (1984) 675-685. [37] D. Miller, A spatial representation system for mobile robots, in Proc. IEEE Int. Conf. on Robotics and Automation, St. Louis, MO, 1985, 122-127. (381 H. P. Moravec, Robot Rover Visual Navigation (UMI Research Press, Ann Arbor, MI, 1981). (391 H. P. Moravec, The Stanford Cart and the CMU Rover, in Proc. IEEE 71, 7 (1983) 872-884. [40] H. P. Moravec, Sensor fusion in certainty grids for mobile robots, A I Mag. 9, 2 (1988) 61-74. [41] H. P. Moravec and D. W. Cho, A Bayesian method for certain grids, in A A A I Spring Symposium Series on Mobile Robot Navigation, Stanford, CA, Apr. 1989. [42] H. Nasr and B. Bhanu, Landmark recognition system for autonomous mobile robots, in Proc. IEEE Int. Conf. on Robotics and Automation, Philadelphia, PA, Apr. 1988, 1218-1223. [43] A. Robert de Saint Vincent, A 3-D perception system for the mobile robot HILARE, in Proc. IEEE Conf. on Robotics and Automation, San Francisco, CA, 1986, 1105-111 1. [44] J. J. Rodriguez and J. K. Aggarwal, Matching aerial images to 3-D terrain maps, ZEEE Trans. Pattern Anal. Mach. Zntell. 12, 12 (1990) 1138-1149. [45] P. J. Rosseeuw and A. M. Leroy, Robust Regression and Outlier Detection (John Wiley and Sons, NY, 1987). [46] R. C. Smith and P. Cheeseman, On the representation and estimation of spatial uncertainty, Znt. J. Rob. Res. 5, 4 (1987) 56-58. [47] K. Sugihara, Some location problems for robot navigation using a single camera, Comput. Vision Graph. Image Process. 42, 1 (1988) 112-129. [48] K. Sugihara, Location of a robot using sparse visual information, in Robert Bolles and Bernard Roth (eds.), Robotics Research: The Fourth International Symposium (MIT Press, 1987) 319-326. [49] W. Szczepanski, Die Losungsverchlage ftir den raumlichen Ruckwiirtseinschnitt, Deutche Geodatische Komission, Reiche C: Dissertationen-Heft Nr, 1958, 1-44.
796
R. Talluri €3 J. K. Aggarwal
[50] R. Talluri and J. K. Aggarwal, A position estimation for a mobile robot in an unstructured environment, in Proc. IEEE Workshop on Intelligent Robots and Systems, IROS '90, Tsuchiura, Japan, Jul. 1990, 159-166. [51] R. Talluri and J. K. Aggarwal, A positional estimation technique for an autonomous land vehicle in an unstructured environment, in Proc. A I A A / N A S A Int. Symp. on Artificial Intelligence and Robotics Applications in Space, ISAIRAS '90, Kobe, Japan, Nov. 1990, 135-138. [52] R. Talluri and J. K. Aggarwal, Position estimation for an autonomous mobile robot in an outdoor environment, IEEE Trans. Robotics and Automation 8, 5 (1992) 573-584. [53] R. Talluri and J. K. Aggarwal, Edge visibility regions-a new representation of the environment of a mobile robot, in Proc. I A P R Workshop on Machine Vision Applications, MVA '90, Tokyo, Japan, Nov. 1990, 375-380. [54] R. Talluri and J. K. Aggarwal, Positional estimation of a mobile robot using edge visibility regions, in Proc. IEEE Conf. on Computer Vision and Pattern Recognition, CVPR '91, Hawaii, Jun. 1991, 714-715. [55] R. Talluri and J. K. Aggarwal, Positional estimation of a mobile robot using constrained search, in Proc. IEEE Workshop on Intelligent Robots and Systems, IROS '91, Osaka, Japan, Nov. 1991. [56] R. Talluri and J. K. Aggarwal, Transform clustering for model-image feature correspondence, in Proc. IAPR Workshop on Machine Vision Applications, MVA '9.2, Tokyo, Japan, Dec. 1992, 579-582. [57] R. Talluri and J. K. Aggarwal, Autonomous navigation in cluttered outdoor environments using geometric visibility constraints, in Proc. Int. Conf. on Intelligent Autonomous Systems: I A S - 3, Pittsburgh, PA, Feb. 1993. [58] R. Y. Tsai, A versatile camera calibration technique for high accuracy 3-D machine vision metrology using off the shelf TV cameras and lenses, IEEE Trans. Robotics and Automation 3, 4 (1987) 323-344. [59] T. Tsuboushi and S. Yuta, Map assisted vision system of mobile robots for reckoning in a building environment, in Proc. IEEE Int. Conf. on Robotics and Automation, Raleigh, NC, 1987, 1978-1984. (601 J. S.-C. Yuan, A general photogrammetric method for determining object position and orientation, IEEE Trans. Robotics and Automation 5, 2 (1989) 129-142. [61] P. R. Wolf, Elements of Photogrammetry (McGraw Hill, New York, 1974).
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 797-815 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
I CHAPTER 4.5 I COMPUTER VISION IN POSTAL AUTOMATION*
G. GARIBOTTO and C. SCAGLIOLA Elsag Bailey, a Finmeccanica Company R b D Department, Genoua, Italy E-mail: giouanni.garibottoOe1sag.it The objective of this chapter is to provide a critical analysis of Computer Vision within the context of Postal Automation services. The main functional requirements of this application field are briefly referred, as well as the involved Vision functions, which are considered here in a broad sense, including Pattern Recognition, Image Processing and understanding, Signal Processing and Robot Vision. New trends as well as new services emerging in Postal Automation are also discussed, in an attempt t o highlight the expected impact on the development of Computer Vision technology. The aim of the chapter is also to refer t o the most relevant chievements of Computer Vision as well as to discuss why other promising techniques did not succeed, in spite of the advanced results obtained a t prototypical level in laboratory experiments. The ultimate goal is to provide a contribution t o stimulate the basic and applied research efforts in this important field of industrial automation and possibly support the recent initiatives of technology transfer from research to industry. Keywords: Mail processing, character recognition, image processing, material handling, electronic reading systems.
1. Introduction
Mail sorting and postal automation has always represented an important area of application for Image Processing and Pattern Recognition techniques. Since the early developments in the first half of this century, postal mechanisation has grown a lot in the sixties and seventies, pushed primarily by the initiatives of the different national postal administrations. Around the middle of the seventies, the escalating use of faxes and data transfer, and more recently of e-mail, led to predictions that within 20 years relatively few people would communicate by letter. In spite of such predictions mail volume grows steadily, to reach such levels of about 5.6 billion pieces a year in Italy in 1994 [l],10.3 billion in France in 1994 [2], 10.6 billion in Canada in 1992/1993 [3], and 181 billion in the USA in 1995 [4]. However, the scenario of postal services is rapidly changing all around the world, because of the gradual transformation of national post institutions into private companies looking for service quality and efficiency. On the other hand, *The paper has been partially supported by ECVnet, the European Computer Vision network, an ESPRIT Network of Excellence.
798
G. Garibotto €9 C. Scagliola
the telematics revolution enables software developers to offer alternate services in competition with traditional providers. Finally, global market rules force the breakup of national monopolies in communication and mail services, also encouraging information exchange among all providers. In conclusion, the new situation is such that Postal Administrations are hardpressed by strong competition. In order t o meet the new challenges while remaining profitable or at least avoiding losses, at the same time ensuring the continuity of service to small, often rural, post offices, Postal Administrations move essentially along two lines: 0
0
To improve their service through a re-engineering process of mail handling; To introduce new services to adapt their operations to the changing needs of the users.
Many of the foreseen improvements and new services are based on Image Processing and Computer Vision functions, as will be described later in this report. The following section is devoted to the description of the present situation and trends in postal mechanisation and mail processing, while the next one describes new developments that are under study to improve the mechanisation process. Section 4 points out new image-based functions that are needed for a further improvement of the postal service and/or for introducing new services to the customers, and finally some conclusions are drawn.
2. Description of the Industrial Sector and the Current Trend: Postal Mechanisation and Mail Processing Mail handling is a very labour intensive process and labour costs have been increasing during the last three decades. In addition t o the cost factor, the knowledge level required for the sorting process is quite considerable. Mail has to be sorted for a large number of destinations. In the US, the national delivery network reaches nearly 128 million addresses [4].For important destinations like large cities, direct bundles are formed, but for small villages mail is combined into bundles and dispatched to regional sorting centres for further inward sorting. The policy of most Postal Administrations is t o introduce new postcodes containing information which could be used for the entire mail-handling process up to the final delivery point. Sorting is usually performed by machines that read a barcode as an identifier of the mail piece destination. The barcode is impressed on the envelope by an “encoding” function, performed either automatically by a postal OCR machine or by an employee through a videocoding station. Traditionally, encoding mirrors sorting: a first encoding step identifies the destination city for outward sorting, by reading the classical postcode. The second encoding step identifies the final delivery point for inward sorting by reading the new and complete postcode (where it exists and
4.5 Computer Vision in Postal Automation 799 when the user has written it on the address). Otherwise the full address must be read, i.e. street name and number and possibly apartment number, or Post Office Box number, or the name of a large customer, like a Bank or a Company. In order to reduce the cost of this encoding function, the tendency of Postal Administrations is to gradually increase the percentage of mail that is encoded automatically, and to perform this operation only once, i.e. to encode mail to the destination point directly in the first sorting centre. This operation, which requires the on-line consultation of a nation-wide address database, can be already done for a very large proportion of typewritten mail, but a goal of most Postal Administrations is to also automatically encode handwritten mail to the destination point. Just to give an example, we may refer to the figures provided by the Royal PTT in the Netherlands [5], as the service objectives for the near future. 0
0
98 per cent of mail items smaller than 380 x 265 x 32 mm (machinable mail) will be sorted automatically. So-called non-standard sorting machines will handle larger items.
Standard sorting machines
Vide;o;ding Non-Standard sorting machines
'ostcode address directories
Fig. 1. Flow chart of a network service for mail handling.
800 G . Garibotto & C. Scagliola 0
0 0
As many addresses as possible will be read automatically using OCR systems (over 90% of all items are expected to be read in this way). Sorting will take place down t o the level of an individual postman’s delivery route. Parcels will be handled in separate infrastructures which will be newly constructed according to standardised design.
The flow-chart of Fig. 1 refers to a network architecture proposed t o manage all information in a uniform way and sharing the appropriate resources for the mail sorting process.
2.1. Mail Processing: Statement of the Problem and Application Requirements Automatic reading is necessary for all the address fields needed by the carrier to bring the mail to the final destination. The current mail flow is roughly sketched in Fig. 2, from the collection of all mail items t o the first office. All items are separated (culling), oriented and packed together (facing and cancelling), and are sorted according to the respective post codes. At the destination office, two further sorting processes are implemented in order t o obtain the final carrier sequencing of the mail.
Outward sorting Acceptance and payment
Induction, culling, facing and cancelling
Transport Return services
c Delivery
Sequencing
Inward sorting
Transport
Fig. 2. Current mail flow.
4.5 Computer Vision in Postal Automation 801
Letter processing A brief description of the basic components is given as follows: 0
0
0
0
0
A Culler Facer Canceller (CFC) machine is commonly used as a pre-processor. It also provides for image capturing to allow image processing while the physical mail is being transported in the centre. An OCR machine reads addresses written on the face of the mail and prints a fluorescent bar code on the mail item. Should it be unable to recognise the address, it will capture the mail image and send it for on-line video coding. If the mail cannot be resolved by OCR or on-line coding, the image will be sent for off-line video coding. An off-line OCR and the video coding system will be used to process mail images from the CFC. A Bar Code Sorter (BCS) machine can pre-sort mail items whose images have been resolved during either off-line video coding or off-line OCR. It can also pre-sort letters pre-printed with a bar-code by bulk mailers. A Delivery Bar Code Sorter (DBCS) operates on a two-pass sorting process by sorting bar-coded letters to postman delivery routes in the first pass and to a delivery point sequence in the second pass.
In Appendix A, the functional architecture of a typical letter reading process is presented .
3. Main Functions Involving Computer Vision and Related Computer Vision Techniques The main function required in postal automation, involving Computer Vision is definitely address reading and interpretation. In this sense it belongs to the basic perceptual functions of biological vision. Nevertheless, due to the inherent 2-D nature of the problem, its computer implementation is strongly based on basic technologies such as image processing and pattern recognition. At present, the most challenging tasks, performed by such Vision technologies in postal automation, are handwritten address reading, including cursive handwriting recognition, flats handling and reading, grey level and colour image processing, improved man-machine interaction, and robotic material handling. 3.1. Handwritten Address Reading The new frontiers in handwriting recognition make extensive use of the context to achieve unconstrained address reading, using both large vocabulary and grammar constraints as well as heuristics. A first short-term objective consists in reading the last line including the postcode, city name and state and integrates such information in order to improve the reliability of the system and minimise the use of off-line video coding. The French Postal Administration expects a 10% increase in automatically sorted (outward
802
G. Garibotto & C.Scagliola
sorting) handwritten mail [6],by integrating the reading of the city name with that of the postcode. Next generation machines will also include the capability to read the full handwritten address line, with street name and civic number, in order to manage the final postman’s delivery. The goal of USPS, the United States Postal Service, is to encode to the delivery point 50% of the handwritten mail, with 1%error rate [7]. Laboratory tests indicate that this goal can be achieved [8]. Automatic reading of off-line cursive handwriting is presently a forefront technology for automatic reading systems. Considered too difficult to find useful solutions for common use until the early go’s, it is now one of the most important subjects studied by research groups in Pattern Recognition. There are several difficulties in reading cursive handwriting; letters are usually connected, there is a large variety of letter shapes and individual styles and pixel patterns have an intrinsic ambiguity when taken in isolation. The pattern on the right could be equally well interpreted as a “u”, a double “1” , a double “e”, an “n” or part of an “m”. In the classical approach to OCR, characters are first segmented and then recognised. Lexical knowledge is used at the end, in a postprocessing stage, to correct possible recognition errors and find the correct word interpretation. This approach works pretty well with typewritten images, and even with handwritten ones, if characters are hand-printed separately. In the case of cursive handwriting, unfortunately, it is quite difficult to segment without recognising, and of course it is also difficult to recognise without segmenting. Moreover, as seen in the example before, the real identity of character patterns can often be determined only with the aid of the contextual knowledge. An over-segmentation approach is always recommended [9] to produce a series of segmentation, which are measured against the alphabet of characters, without taking a crisp decision. The optimal interpretation is found for the image by looking for the sequence of character hypotheses that best matches one of the sequences of characters allowed by the a priori knowledge, i.e. by the lexicon and/or grammar. Other similar promising approaches have been recently experimented and some of them are already operative in some prototypical installations [lo].
3.2. Flats Sorting Machines Flats reading represents the most challenging objective for postal sorting machines, besides more conventional letter manipulation. There are different categories of flats to be handled. One class includes large A4 size envelopes with more or less additional information printed on it (sender and destination address, advertisement messages, stamps and mail class service information). Another class consists of journals, newspapers and catalogues with or without plastic covers. Moreover in the flat category are often included also small parcels with a maximum thickness of about 40 mm.
4.5 Computer Vision in Postal Automation 803 Only very few flat sorting machines are in operation, often without automatic reading capability, but the traffic for this kind of mail is constantly growing, and the need for automatic encoding is emerging rapidly. However, automatic reading of addresses on flats cannot be achieved through a simple re-engineering of a letter reader. The different characteristics of the mail pieces pose different problems to the image acquisition subsystem and t o the processing algorithms, that are not yet fully satisfactory in unconstrained operating conditions and that require more sophisticated image processing and computer vision capabilities. One of the main problems comes from the management of plastic covers which prevents a sharp and well contrasted image acquisition from the input vision sensor. Improvements in the acquisition process (both in resolution and dynamic range) as well as adaptive grey level image processing tools, represent key factors in the solution of this problem. The second critical point is Address Block Location. In fact, in some cases, like magazines and advertisements, the destination address is usually written on a small label floating under a plastic cover, which means that it may be found in any position and with any orientation. In other cases the destination address has to be located in a complex image, full of text and graphics, as in newspapers, where the statistical properties of the address block are very close t o those of the full size image. Appendix B refers to a processing scheme and the current main problems in flat sorting machines.
Fig. 3. An example of a flat with plastic cover, Address block on a label, advertisement messages and postage code information.
804 G. Garibotto & C. Scagliola
3.3. Man-Machine Interaction and Intelligent Interface There is much improvement in man-machine interaction tools for video coding, which represent a n increasingly important component of the system, especially in the current approach aimed at increasing off-line and remote encoding by system operators. Ergonomic problems, friendly interfaces, quick and easy panning and scrolling of the image on the screen are priority issues. One of the most important problems is still the optimal display of the letter or flat image on the screen, with the appropriate resolution, grey level scale and adaptive contrast adjustment, to remove the uneven perception of the foreground and background information. Many efforts have also been made to use a combination of technologies and the integration of different sensors (i.e. speech recognition t o simplify the input of address information). Eye tracking is also a new area of research [ll]to speed up the localisation of the address block on the screen, when very complex mail images are involved. These technologies may have an important role in the new generation of postal automation systems. 3.4. Parcel Classification
Huge and complex machines are currently used for 3-D parcel sorting. State of the art sorting equipment can handle some 250,000 items per day. Presently parcel processing is highly labour intensive; in fact parcels are introduced manually by human operators and during this input stage a preliminary selection is already performed according to their size and shape (roll and cylinders, regular and irregular packets, etc.). They are also labelled with some ID code label and the operators place the parcel item on the sorter tray with the label side facing up. At the input stage an overhead scanner is installed to automatically read the label and assign the destination information to the sorter tray. During the last 10 years there has been a significant research effort carried out by the most important Postal Administrations, to automate the parcel input stage and reduce the cost associated with this very low-level work. The main objective is to estimate the correct dimensions of the parcel size for billing purposes. Quite interesting results using Computer Vision technology have been achieved in the late eighties, with the realisation of prototype systems which made use of active light laser sensors to recover the 3-D shape of the parcels and allow a presorting of the mail items [12]. 3-D reconstruction has been a hot research topic for a long time in the Computer Vision scientific community with proposed solutions based on both passive vision (stereovision and motion analysis) and geometric analysis of projected light patterns (lasers or white light projectors). The main industrial thrust in the development of this technology came from robotic metrology and co-ordinate measuring machines, in order to substitute the traditional slow contact mechanical sensors with optical non-contact area sensors, and improve speed in surface reconstruction.
4.5 Computer Vision in Postal Automation 805
There are many industrial products available in the market for the reconstruction of 3-D surfaces, besides some new results from advanced research lab-oratories [13]. In any case, the final solution requires a careful system integration, with controlled lighting conditions and a n optimal arrangement of the sensor w.r.t. the positioning fixture of the samples to measure. Actually, the requirements of 3-D parcel reconstruction in terms of precision are not as severe as in robotic metrology. As far as 3-D shape representation is concerned, the most common representation techniques can be classified into volumetric or surface based schemes. A volumetric reconstruction by voxels is quite heavy in terms of data storage but classical arrangement of data as octrees [13] or skeletons may save a lot of memory space. The use of deformable models, like superquadrics [14], allows for a very efficient 3-D shape representation, as a composition of individual elements, and this approach has been successfully investigated for parcel modeling too. The most consolidated technique for 3-D shape representation is definitely the Delaunay triangulation, which allows a n optimal interpolation of sparse data points. The extension of such a tool from two-dimensions to three-dimensions is thoroughly discussed in [16], where formal definitions and examples are referred to as well as the relationshipship which exists between the Delaunay triangulation of a set of points on the boundary of an object and the skeleton of that object. Nowadays there are a few installations of 3-D dimensioning systems for parcel measuring and classification. It is worth while to mention a light curtain technology, as well as an infrared Laser Rangefinder technology [17], which is currently in use for on-line parcel dimensioning in different European countries. 3.4.1. 2 - 0 Bar-code image readers The term “2-D bar-codes” refers to any of the new ID-codes that do not rely on a single row of marks/spaces t o encode data. 2-D codes provide high capacity (up to 2,000 characters per label, compared to 50 characters of conventional 1D Bar codes), and very robust error correction capabilities. There are two main types of 2D codes: stacked linear (PDF) and matrix codes (like the Maxicode referred to in Fig. 4) [18]. Postal administrations are currently using new generation portable 2-D image readers for parcel delivery and high speed sortation. UPS is starting to roll out Maxicode, by marking packages with this 2-D matrix code and reading the code in its large-scale hub sortation centres. The main problem of image processing and recognition is the development of a reliable location and positioning of the 2-D code in the acquired image. For instance in the example of Fig. 4 the detected elliptical shape of the inner circles are used to identify the central position of the code and further geometric reasoning techniques are used to properly locate the polygonal shape of the code, which may be affected by perspective distortion. The following decoding process is quite trivial and is based on standard encoding/decoding procedures.
806
G . Garibotto i 3 C. Scagliola
Fig. 4. An example of Maxicode printed on a parcel.
However, a wider range of image recognition applications is considered using both fixed and hand-held image readers. They include some postal-specific symbols like Postnet as well as digital indicia, a new service that the United States Postal Service (USPS) will offer t o their customers, to purchase postage over the Internet, download the postage t o their PC and then use their own laser printers t o output certified and secure electronic postage. 3.5. Material Handling and Station Loading/Unloading
One of the most intensive tasks t o be performed in a large mail distribution centre is the transportation of mail items between different machines and to/from the input/output stage of the centre. The loading/unloading of the letters to/from mail sorting machines is still mainly performed by human operators and the automation of this process represents an essential target to work towards. A commonly agreed approach consists in the realisation of standardised letter containers which could be handled automatically by robotic machines. From recent studies it has been demonstrated that such a transport service is carried out for about 80% of the latters through the use of trolleys pushed by human operators and just 20% are managed by electric trucks (mainly driven again by human operators). The available technologies to solve such problems of transportation between different working cells (intercell service) are: 0
Electrical trucks, often used to tow a convoy of passive trolleys, with obvious problems of manoeuverability and requirement of human driving.
4.5 Computer Vision an Postal Automation 807
Rail transport system with the well known disadvantages of fixed installations and no flexibility in the management of the mail distribution centre. Rollers chains which are efficiently used for point-to-point service, but again there are strong limitations due to space occupancy and no flexibility. The use of AGVs (Autonomous Guided Vehicles) represents nowadays the most flexible solution. There are conventional systems using inductive guides buried into the floor, as well as new generation navigation systems, mainly based on active sensors (magnetic tags or lasers) for self-orientation. Still there is poor flexibility in the reconfiguration of the navigation map, high precision requirements in the positioning of the loads (pallets), difficulty of switching from automatic to manual driving of the vehicle. New generation mobile robots, based on advanced sensors, with free ranging capabilities and the possibility of an easy reconfiguration of the navigation route, able to detect the presence of other vehicles at crossing points, with automatic and manual driving capability, represent the expected solution. In this domain Computer Vision should play a fundamental role to give flexibility and intelligence to the robotic logistic system. 3.5.1. Computer vision for autonomous navigation
There are not many industrial examples of Computer Vision applied to AGVs even if this is commonly considered one of the most promising solutions to achieve the necessary flexibility and performance. On the other hand there are interesting vision-based control techniques which have been developed in the automotive industry as a support to the driver, to provide information about the distance from the nearest forward cars and to perform autonomous tracking and car following in traffic congestion [19]. Autonomous navigation has been strongly pushed by military research for both normal road drive and off-road navigation. Interesting results can be found in [20] and in [21] where this subject has been investigated for many years and has been tested with prototype vehicles operating at nearly normal speed conditions (about 100 km/h with obstacle detection capability). There have been also attempts to use Computer Vision to drive AGV systems and in mobile robotics for service applications, as in [22] for hospital transportation functions. Some experiments of using AGV systems in mail distribution have been implemented [23]. Quite recently, a fully passive vision approach has been proposed to allow selfpositioning and autonomous navigation of a fork-lift carrier named Robolift, [20]. In this case Computer Vision is used to recognise artificial geometrical landmarks, placed along the navigation pathway and correct the relative odometer estimates. Furthermore, Computer Vision is also used for docking control, to recognise the correct position of the pallet so that Robolift control can suitably correct the displacement of the fork prongs before loading.
808
G. Garibotto & C. Scagliola
Advanced research are in progress to use Computer Vision and multisensor integration (laser, inertial sensors) for autonomous navigation, in order to make use of existing landmarks and features already present in the environment and allow easy and fast reconfiguration of the navigation mission. 4. A Projection of Future Trends
Today large volume mailers increasingly demand faster, more reliable service and customised products. They want day-certain delivery, shipment and piece tracking, and an electronic data interface. Moreover, the most important administrations now compete with express mail service companies like DHL, UPS, FedEX, newspapers, telecommunications companies, and alternative delivery services. A key function to survival and success of the Postal Administrations is the integration of the service in a network, by connecting most of the plants with each other, with transportation suppliers, mailer plants and postal customers. For instance, the information infrastructure has to be redesigned in order to avoid the repetitive capture of the same data and to put all available information ready at each stage in the physical mail handling process. Standardisation of tools and processing interfaces is therefore another fundamental requirement for the next generation of machines. Moreover there is a great effort to improve the efficiency of the service through the realisation of distributed architectures which may provide remote access and obtain a wider access to the processing resources (both geographically and logically). However, besides re-engineering the mail handling process through wise use of Information Technology, other benefits can be achieved by the implementation of new functions based on Image Processing and Computer Vision capabilities. The previous section has already referred to the main direction of research and investment aimed at improving the performance of mail processing systems in the field of handwriting recognition, address block location, parcel processing, etc. In the following we try to focus on new emerging services which represent the new frontiers for competition and system providers in order to enhance the level of service and increase the added value to the final customer. 4.1. New Functions and Services i n Mail Processing Systems
The mail handling process can be divided into three main functional areas, i.e. acceptance and payment, sorting and transportation, and delivery services. A thorough review of the whole postal process, from customer payment for and submission of mail through to delivery to the addressee, is the main goal of the Esprit project TIMBRE (Technology In Mail Business Re-Engineering), conducted by a consortium of postal administrations and technology providers and led by IPC Technology (251. While most of the improvements in the postal process would be based on a heavy use of communication and networking functions, new image-based functions
4.5 Computer Vision in Postal Automation
809
and services can also be used to improve efficiency and provide new services. We indicate here a list of such possible functions, some of which most probably would require colour image processing: 0
0
0 0
0 0 0
verification of the presence and value of stamp(s) and postage, with reference to weight, destination and class of the postal object; Detection of false or recycled stamps; Identification of special stamps and logos for “controlled delivery by time”; Reading of the amount of additional stamps for effecting payments; Reading and verification of postal permits; Reading the name of the addressee for redirection services; Reading the address of the sender for “return to sender” mail.
As a whole, the listed functions would constitute what could be called a “postal image understanding system”. While such a complete system most probably would never be implemented, some of the above services are under consideration in different Postal Administrations. For instance, Canada Post Corporation is considering the automation of “Return to Sender” mail [3] and electronic redirection of mail [26]. 4.1.1. Paper to electronic mail service The mail process has been primarily considered as an end-to-end paper mail service. Over the last few years, the rapid growth of computer and communication technology has led to a corresponding rapid growth of an end-to-end electronic mail, especially in the business sector. On the other hand the integration of different technologies can provide excellent opportunities for new postal services as hybrid mail, as an example of electronic to paper mail service. In this case large mailers can produce and forward messages in electronic form and use the distributed postal network for the printing and delivery of the mail. Image-based technologies make it possible now to cross the paper/electronic barrier in the opposite direction and implement paper-to-electronic services. An example of a new service of this type is the Reply Card Processing, or automation of Business Reply Mail. This service allows the interception of all business reply cards or courtesy reply cards addressed to a specific customer (i.e. a mail order company) and the capture of the image on both sides of the card. At this point, the image can be stored in appropriate electronic mail boxes and later transmitted to the customer’s fulfilment centre instead of the physical card. This paper-to-image transformation alone can reduce the delivery time from 2 to 4 days down to 12 to 18 hours [27] and it is the first step foreseen for a new service in the USA [4]. The next steps are the automatic electronic reading of the content (handwritten information) and its translation into frameworks suitable for computer processing,
810 G. Garibotto & C. Scagliola
its decoding and delivery t o the customer using data transmission networks. In this way the customer’s fulfilment centre would have the information transferred directly into its database, thus avoiding the usual manual data entry operation. This kind of new service is being considered not only by USPS, but also by other Postal Administrations, like UK’s Royal Mail [28], and by technology providers [29]. 5 . Conclusions
The mail automation sector is a quickly evolving area of industrial automation, where the development of Electronics and Parallel processing, Sensors, Robotics, Information Technology and Telecommunications, have opened up new perspectives of development. In the last twenty years the Postal sector has been a very closed domain with special purpose solutions and approaches and a poor connection with similar applications and research disciplines. The mail sorting process has been, and it is, heavily dependent by mechanical constraints. It is always an exciting experience to visit a Mail sorting and Distribution Centre, with huge and complex machines and thousands of letters running back and forth at incredibly high speeds along the rubber transport chains. The other foremost technology has been definitely OCR for address reading. But now there are new problems and new solutions emerging, as we have tried to briefly mention in this chapter. The reading process is much more complex than simply measuring accuracy or speed of the individual character recognition module. The intelligence to distinguish between the different pieces of information in the mail (background, form structures, etc.), the possibility of taking advantage of all available context information to drive the OCR and text reading with feedback control, is the real challenge of the new generation reading systems. The availability of ever increasing processing power, and the possibility of off-line distributed reading systems is pushing the development of software solutions, possibly with hardware accelerators. But severe requirements are also expected of the imaging sensor which has to be able to deal with grey-scale information (and possibly colour) at very high resolution, large formats and with fast pre-processing tools. This is the area where real time hardware solutions are unavoidable. Finally 3-D Computer Vision appears t o have become a n important enabling technology in the Postal Automation sector, being mainly focused on supporting robotic applications in the material handling and logistics services. The most relevant impact of Computer Vision is expected in the following areas: 0 0
3-D object recognition and classification (parcel sorting). Pose recognition for standardised trays handling in loading/unloading stations.
4.5 Computer Vision in Postal Automation 811 0
Vision based autonomous navigation (self-localisation, obstacle detection, docking) to provide the required flexibility and free-navigation capability in a crowded environment, for inter-cell transportation services.
It is time for new individuals t o come into the arena of Mail Automation and Postal Services in order to introduce fresh ideas from close technological fields of Vision and Pattern Recognition, and give a new impulse to the improvement of this communication service of vital importance in the modern society. Appendix A. Functional Architecture of a Postal Address Reader In a very schematic way, address reading may be described as a data compression process, from the raw data coming at 8 pixel/mm and 256 grey levels, (something about 2 Mbytes data) down t o a few bytes, corresponding to the content of the postal destination address. It consists essentially of three modules: (1) Acquisition and image processing (2) Segmentation and recognition of the individual characters in the mail piece (3) Context analysis and address recognition.
Appendix A. 1. Acquisition and Image Processing The objective of this stage is the compression of the grey level image into a binary image. This represents a fundamental step of the process, since any amount of information lost at this stage cannot be recovered any more. Moreover, adaptive processing capability is essential. In fact some of the postal items have a good image contrast, but many of them are much more confused, with poor contrast and limited reading possibilities. A further strong requirement is real-time processing, due to the high speed of the mail stream in front of the input sensor (about 17 letters/second).
Appendix A.2. Individual Character Recognition The input is the bit-map obtained from the single letter image and this module consists of the following steps. Appendix A.2.1. Localisation The objective of this processing step is to identify: 0
0
text lines (both in the case of handwritten or typewritten text). Other geometric or information features t o be detected in the mail item (stamp, codes, etc.)
This stage represents a classical binary pre-processing (using clustering techniques and morphological processing tools).
812
G . Garibotto €9 C.Scagliola
Appendix A.2.2. Segmentation From the localised block of text it is necessary to segment the individual characters for the following recognition. The major problems here are the correct segmentation of broken and touching characters, especially for handwritten text (both numerals and alphabetic characters). Appendix A.2.3. Character recognition The literature of character recognition is extremely wide and rich [30], including the use of feature based statistical approaches, a variety of pattern matching schemes and a combination of neural network techniques. There has been always a great interest in establishing evaluation criteria and benchmarking procedures, to help attain a quantitative and objective comparison of such a wide range of solutions and implementation techniques. The US National Institute of Standards (NIST) has organised specific conferences [31]to promote a thorough comparison of results on the basis of selected data bases representative of machine print and hand print text. The Conference and related comparison exercises focused on a single step in the reading process: machine recognition of individual (or segmented) characters, with no context. To further improve the performance of handwritten character recognition it is quite common to use the following schemes: 0
0
0
to manage multi hypothesis until the end of the process, avoiding an early pruning of the decision tree; to use a combination in parallel or sequential order of different (possibly uncorrelated) character recognition techniques (statistical, neural, etc); a combination of character recognition methods on pair of consecutive characters, rather than on the individual segmented ones.
Appendix A.3. Context Analysis Module This module is relevant not only for the obvious objective of minimising the error rate of the address, but also because there are often writing errors in the original address (it has been estimated to be more than 10% in the UK). This module takes into account the following information: 0
0
a database which describes all the spectrum of the expected addresses (including some possible errors); some coding rules which describes how the address is supposed to be arranged.
Different solutions are usually implemented for typewritten or handwritten context analysis, since different fields with different content are typically involved. Ultimately, the performance achieved by the context analysis code is significantly better than the possible result from a simple code reader as depicted in the following table [32].
4.5 Computer Vision in Postal Automation 813 Reading rate
Rejection rate
Typewritten
code reader context address
2 72%
2 92%
5 1.6% 5 0.5%
Handwritten
code reader context address
2 62% 2 69%
5 1.8% 5 0.9%
Appendix B. Address Block Location in Flat Sorting This section describes the essential features of a module for address block location, in flat sorting machines.
Input data They consist of grey level images of size 2000 x 2000 pixels or more. Visual criteria and the information content of such mail pieces are briefly summarised in the following. The address block usually contains dark ink characters on a brighter background (which may be a white label or the grey colour of the envelope). The format and size of the characters is arbitrary and cannot be established a priori, especially for handwritten addresses. The address lines do not have a fixed known direction, although typewritten text is mostly oriented horizontally or vertically (except when a free label is inserted in a plastic wrapping). The most critical noise source comes from plies or folds on the surface (especially for plastic wrapping). Other useless information which have to be identified and discarded are headlines, patches of text, photos, graphics, etc., with a large variety of colours.
Address block location: A short description of the process Pre-processing and noise removal The objective of this stage is image enhancement t o remove input noise and minimise the effect of interferences (like light reflections or plies and other overlapped noise structures). Multiresolution Region of Interest analysis Data reduction is the primary objective, together with a more efficient data representation to find out candidate regions for the address block. The detection of regular and repetitive patterns, local measures of density and frequency, and blob analysis, are common tools used at this stage of the process. Segmentation of block candidates Geometrical constraints are commonly used to segment and isolate some blocks and rank them on the basis of similarity measures with respect to the available
814
G. Garibotto & C. Scagliola
address prototype models. The number of detected text lines (horizontal or vertical), their alignment (left, centre or right), the size and shape of the block are used as discriminating features. (4) Context analysis Topological constraints as well as heuristic criteria are used t o classify the detected blocks and decide on their arrangement onto the mail piece. It is worthwhile to point out that this stage of the process is usually carried out at lower resolution where text characters cannot be recognised and interpreted. Many research results on this subject, as well on the other topics of mail process automation can be found on the proceedings of the USPS Conference [33]. In the previous scheme we have described a traditional forward processing approach, but the present availability of ever increasing powerful processing power allows the exploitation of feedback information to improve the selection of the processing parameters, and converge t o better reading results. References [I] Poste Italiane, Volume Frafico Nazionale, World Wide Web site: http://www.nettuno.it /fiera/posteitaliane/roma/html/4bc-corr.gif [2] La Poste, Progress Report, 1 l t h Int. Conf. Postal Mechanisation, Melbourne, March 7-11, 1994. [3] Canada Post Corporation, Progress Report, 1 l t h Int. Conj. Postal Mechanisation, Melbourne, March 7-11, 1994. [4] United States Postal Service, Postal Facts for Fiscal Year 1995, World Wide Web site: http://www.usps.gov/history/pfact95f.htm. [5] Postal Technology International '96, UK & Int. Press, ISSN 1362-5209. [6] J. J. Viard, New technologies and their Impact on communication markets, Troika '95, postal service infrastructure of a modern society international symposium, Saint Petersburg, Russia, June 12-16, 1995. [7] S. N. Srihari, V. Govindaraju and A. Shekhawat, Interpretation of handwritten addresses in US mail stream, First European Conference dedicated to Postal Technologies J E T P O S T E 93, Nantes, June 14-16, 1993. [8] F. Kimura and M. Shridhar, Handwritten address interpretation using extended lexicon word matching, in Progress in Handwriting Recognition, A. C. Downton and S. Impedovo (eds.) (World Scientific, Singapore, 1997); Proc. 5th Int. Workshop on Frontiers in Handwriting Recognition, Colchester, Sept. 2-5, 1996. [9] C. Scagliola, Search algorithms for the recognition of cursive phrases without word segmentation, Proc. 6th Int. Workshop on Frontiers in Handwriting Recognition, Taejon, Corea, Aug. 12-14, 1998. [lo] C. Saulnier, Revolution in OCR processing, Postal Technology '98, 112-115 UK & International Press, 1998. [ll] F. Morgan, R. V. O'Toole, D. A. Simon and M. Blackwell, Optotrak Validation Experiments, Technical Report CMU-RI-TR-95-26, The Robotics Institute, Carnegie Mellon Unin., 1995. [12] P. Mulgaonkar, Automated postal system, S R I Int. Report ITAD-733-MK-95,026, pp. 13, February 1995. [13] T. Kanade, A. Gruss and L. R. Carley, A very fast VLSI rangefinder, Proc. 1991 IEEE Int. Conf. Robotics and Automation, Sacramento, CA, Apr. 1991, 1322-1329.
4.5 Computer Vision in Postal Automation 815 [14] H. Samet, Design and Analysis of Spatial Data Structures: Quadtrees, Octrees, and other Hierarchical Methods (Addison-Wesley, 1989). [15] R. Bajcsy and F. Solina, Three-dimensional object representation revisited, in Proc. 1st Int. Conf. Computer Vision, June 1987. [16] 0. Faugeras, Three-Dimensional Computer Vision; A Geometvic Viewpoint (The MIT Press, Cambridge, Massachusetts). [17] J. Spandow, New dimensions in data capture, Postal Technol. Int. '98, UK & International Press, 1998, 136-139. [18] D. Flynn, Advances in barcode scanning, Postal Technology '98, UK & International Press, 1998, 132-134. [19] B. Ulmer, VITA I1 - Active Collision Avoidance in Real Traffic, IEEE Symp. Intelligent Vehicles '94, Oct. 1994, Paris, 1-6. [20] C. Thorpe and M. Hebert, Mobile robotics: perspectives and realities, Proc. ICAR'95, Saint Feliu, Spain, Sept. 1995. [21] E. D. Dickmanns, R. Behringer, D. Dickmanns, T. Hildebrandt, M. Maurer, F. Thomanek and J. Schielen, The seeing passenger car VaMoRs-PI IEEE Symp. Intelligent Vehicles '94, Oct. 1994, Paris, 68-73. [22] J. M. Evans, HelpMate: A service robot success story, Service Robot: A n Int. J. 1, 1 (1995) 19-21 (MCB University Press). [23] S.Tansey and 0. Holland, A system for automated mail portering using multiple mobile robots, 8th Int. Conf. Advanced Robotics, ICAR97, Monterey, Ca, 1997, 27-32. [24] G. Garibotto, ROBOLIFT: Vision guided autonomous fork-lift, Service Robot: A n Int. J . 2, 3 (1996) 31-36 (MCB University Press). [25] European Commission ESPRIT, Information Technologies RTD Programme, Domain 7: Technologies for Business Processes, Summaries of projects of Fourth Framework Programme, Sept. 1996. [26] Canada Post Corporation, Lettermail Mechanization, 11th Int. Conf. Postal Mechanisation, Melbourne, March 7-11, 1994. [27] D. Bartnik, V. Govindaraju and S. N. Srihari, Reply Card Processing, VJWW site:
http://www.cedar.buffalo.edu/RCP/. [28] J. Kavanagh, Investment spawns spin-offs, Financial Times, Jan. 14, 1996. (291 D. Roetzel, Strategic aspects for electronic and hybrid mail services, Troika '95, Postal Service Infrastructure of a Modern Society Int. Symp., Saint Petersburg, Russia, June 12-16, 1995. [30] Proc. Third Int. Conf. Document Analysis and Recognition (IEEE Computer Society Press, Aug. 1995). [31] The first census optical character recognition system conference, NISTIR 4912, Aug. 1992, U S . Dep. of Commerce, NIST. [32] B. Belkacem, Une Application Industrielle de Reconnaissance d'addresses, 4 eme Colloque National sur 1'Ecrit et le Document, CNED'96, Nantes, July 1996. (331 Proc. Advanced Technology Conf. USPS, 1992.
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 817-854 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
I CHAPTER 4.6 1 VISION-BASED AUTOMATIC ROAD VEHICLE GUIDANCE
DIETER KOLLER
EE Dept., 109 Moore Labs, MC 136-93 California Institute of Technology, Pasadena, CA 91125, USA QUANG-TUAN LUONG* Artificial Intelligence Center, SRI International 333 Ravenswood Ave. Menlo Park, CA 94025, USA JOSEPH WEBER Autodesk, 2465 Latham St., Suite 101 Mountain View, CA 94040, USA JITENDRA MALIK Computer Science Division, University of California Berkeley, CA 940.20, USA During the last decade, significant progress has been made towards the goal of using machine vision as an aid to highway driving. This chapter describes a few pieces of representative work which have been done in the area. The two most important tasks to be performed by an automatic vehicle are road following and collision avoidance. Road following requires the recognition of the road and of the position of the vehicle with respect to the road so that appropriate lateral control commands (steering) can be generated. Collision avoidance requires the detection of obstacles and other vehicles, and the measurement of the distances of these objects to the vehicle. We first explain the significance of vision-based automatic road vehicle guidance. We then describe the different road models, and contrast the approaches based on modelbased lane marker detection with adaptive approaches. We describe in detail the important approach of road following by recursive parameter estimation, which is the basis for the most successful systems. We then address the issue of obstacle detection, first detailing monocular approaches. We finally describe an integrated stereo approach which is beneficial not only for obstacle detection, but also for road following. Keywords: visual navigation, intelligent highway vehicle systems, intelligent control, real-time machine vision, recursive estimation, spatio-temporal modeling, stereovision. video image processing. *Corresponding author:
[email protected] 818
D. Koller et al.
1. Introduction 1.1. Vision for Automatic Vehicle Guidance There has been much research in machine vision on the basic problem of building a dynamically updated map of the three-dimensional environment. This includes techniques based on binocular stereopsis-using the slight differences in the images of the scene from two cameras - and techniques based on structure from motion - which is the problem of using the optical flow extracted from a sequence of images acquired from a single moving camera. The literature on these problems is extensive - we refer the reader to Horn [18], and Faugeras [13]. Autonomous vehicle navigation was one of the first applications of machine vision to have been investigated (see for instance [36]). In the United States, many earlier projects were supported by the Department of Defense, and the desired capacity focused on cross-country, all-terrain driving. The CMU NavLab project [46] is the key university-based project on this theme with major activity at sites such as Martin Marietta. The cross-country terrain requirement means that the problem is quite unstructured and hard. Other recent developments in this line of research include applications to planetary rovers [30]. By contrast, this chapter will focus on work specifically aimed at using vision for guiding a vehicle driving on a road. Since automobile transportation is a major component of modern societies, the social and economic implications are far-reaching in terms of comfort, economy, and safety. The ultimate goal is to perform fully autonomous driving within an urban environment. However, achieving this goal within the next decade seems to be out of reach. Because of the complexity of such an environment, many tasks such as identification of the road configuration (including intersections), other vehicles’ behaviors, traffic sign reading, and omnidirectional sensing, have to be performed in addition to those required for autonomous highway cruising under standard conditions. On the other hand, highway driving is sufficiently constrained that simple assistance systems, such as autonomous cruise control, lane keeping, and vehicle following will be operational in the next few years, and fully autonomous highway cruising can be envisioned within this decade. The two most basic tasks to be performed by a n automatic vehicle are road following and collision avoidance. Road following requires the recognition of the road and of the positioning of the vehicle with respect t o the road so that appropriate lateral control commands (steering) can be generated. Collision avoidance requires the detection of obstacles and other vehicles, and the measurement of the distances of these objects to the vehicle. These two capacities are sufficient for highway driving under normal conditions, and will be studied in detail in this chapter.
1.2. Vision Compared to Other Sensors Vision (i.e. the real-time processing of video images) is a rich source of information, however:
4.6 Vision-based Automatic Road Vehicle Guidance 819 The processing is computationally more expensive and complex than with other sensors. The visual input is degraded in some particular atmospheric conditions and at night. Many other sensors have been investigated for use in autonomous vehicles. For the purpose of lateral control, magnets embedded under the road have been proposed. They require an upgrade of the highway infrastructure. However, nearly 70% of single vehicle roadway departure accidents occur in rural or suburban settings on undivided two lane roads. Since it is unlikely that these roads will be upgraded in the foreseeable future, a system for preventing these crashes must rely on the existing road structure. Another approach is to use radar to detect the roadsides, using the fact that at a low incident angle asphalt possesses a lower reflectivity than typically rough roadside surfaces. However, such an approach does not take advantage of the existing lane markings. For the purpose of longitudinal control, Doppler radars, laser range-finders, and sonars have been proposed. Each of these sensors have their own weaknesses. For instance, the magnetic field of embedded magnets can be perturbed by the presence of nearby magnetic bodies, and the sensitivity of the sensors might not be sufficient for performing lane changes. Sonars need a reflective surface and there are problems with detection range and time considerations. The advantage of vision is that it is a passive sensor: it does not send out a signal. Active sensors such as radar can be a potential problem in crowded scenarios. Issues such as environmental safety and interference need to be addressed. While all these different sensors have been shown to provide adequate information to support lateral control and longitudinal control, they might not provide enough information for more complex tasks, where the environment is complex, and a large number of possibilities must be taken into account in order to provide safe automation. Examples of such tasks include lane change maneuvers, or obstacle detection in cluttered environments and/or curved roads. Many approaches to automatic vehicle guidance take a multi-sensor fusion approach in order to gain robustness through redundancy, and to combine the strength of each sensor. Visual sensors are always considered in such approaches, because they have properties which complement non-visual sensors. The combination can provide a reliable system under all weather conditions.
1.3. The State of the Art Coordinated research programs towards autonomous vehicle guidance have been developed around the world. In Europe, impressive work on a visually guided autonomous vehicle has been done in the group of Prof. E. D. Dickmanns of Universitat der Bundeswehr, Munich, Germany [ll].Their work resulted in a demonstration in 1987 of their 5-ton van, the VaMoRs running autonomously on a stretch of the Autobahn at speeds of up to 100 km/h. Vision was used to provide input for
820 D. Koller et al.
both lateral and longitudinal cqntrol on free roads. Subsequently they have demonstrated successful operation on cross-country roads (at lower speeds) where the road boundaries are difficult t o determine. All this was achieved with rather simple hardware-IBM PCs. Subsequently, the PROMETHEUS initiative by the major European car manufacturers has led t o implementations such as VITA [48] and VaMP. The platform for the latter project is a passenger vehicle which demonstrated in 1995 autonomous driving of more than 95% of a 1700 km trip at an average speed of 120 km/h, where hundreds of lane change maneuvers were performed automatically [3]. In the US, the IVHS research initiative triggered the development of several projects. The most visible example is the Navlab-5 which drove in 1995 98% of a 4500 km trip with automatic lateral control [39]. Another example is the LaneLock and LaneTrack projects at General Motors [1,27], which have resulted in real-time implementations. In Japan, research is being conducted at a number of industrial and academic research laboratories. These include the Harunobo project at Yamanashi University ongoing since 1982 which has resulted in an autonomous vehicle tested on roads. The Japanese Ministry of Transport started in 1991 the AVHS and PVS projects, to which major car manufacturers such as Toyota, Nissan, Honda contributed individually. Test vehicles exhibiting various functionalities were demonstrated at the 1996 IEEE Symposium on Intelligent Vehicles. Korea has a similar project, PRV. For an extensive survey, we recommend the proceedings of the IEEE Symposium on Intelligent Vehicles which has been held yearly since 1990 in Tokyo (1990,1991,1993,1996), Detroit (1992,1995), and Paris (1994). 1.4. A Few Representative Approaches
There are many ways to design a system to perform lateral and longitudinal control based on vision. In this chapter, we will not attempt to conduct an extensive survey, but instead detail some representative approaches. A first critical choice is that of the type of representation to choose for lane keeping. In the work by conducted at UBM by Dickmanns et al., a totally explicit representation has been chosen. The idea is that of recursively maintaining a set of road and vehicle state parameters. The road parameters include estimates of horizontal and vertical curvature of the road; the vehicle parameters include estimates of heading angle, slip angle and lateral offset relative to the road. The dynamical model represents knowledge about the motion of a vehicle and serves as a tool both for fusion of conventionally measured data (such as speed and distance from an odometer) and for control determination as well as the prediction of the effects of this control input on the evolution of the trajectory and on corresponding changes in the perspective image over time. The image analysis required is thus made much simpler than in the general machine vision setting. Only at the initialization stage is there a need to search extensively t o determine the lane boundaries. After that the task becomes much simpler by exploiting the temporal continuity conditions captured in the dynamical model. By contrast, in the approach of Pomerleau et al.,
4.6 Vision-based Automatic Road Vehicle Guidance 821 little or no explicit representation is used. As a consequence, temporal continuity is not fully exploited. To obtain real time operation, the images have to be considerably sub-sampled. The knowledge of its position and of the road available to the system is much less precise. However, it is argued that it is easier for such a system to adapt to new road configurations, since the absence of explicit representation would make it more flexible. A second critical choice is the type of camera system to use, monocular or binocular. With a monocular (or bifocal) system it is necessary to use model-based techniques which exploit heuristics such as symmetry of the vehicles. The drawbacks are that more general obstacles (such as pedestrians) cannot be recognized, and the estimation of the distance is not precise. For this reason, in monocular systems the obstacle detection part is often performed using radar. A binocular system requires more visual processing. However, it makes it possible to detect general obstacles and to obtain a precise estimate of their distances. Therefore, for the purpose of obstacle detection, binocular stereopsis is becoming a popular option. Interestingly, the task of highway driving seems to be constrained enough so that the systems with different design options are relatively robust and have exhibited relatively similar performance. For instance Dickmann’s system assumes a relatively fixed road appearance, an assumption easily violated in practice. Pomerleau’s system uses a very simplified geometric model and does not take into account dynamics. Yet both systems exhibit reasonable performance. In Section 2, we concentrate on road modeling and detection, with a focus on the low-level techniques. We first describe the different road models, and contrast the approaches based on model-based lane marker detection with more adaptive approaches. In Section 3, we describe in detail the important approach of road following by recursive parameter estimation, which is the basis for the most successful systems. In Section 4 we address the issue of obstacle detection by monocular systems. We finally describe in Section 5 an integrated stereo approach which is beneficial not only for obstacle detection, but also for road following.
2. Road Modeling and Localization
2.1. Road Models The road is often represented as a triangle in the image plane (for example [8,22,23]) which assumes the road is straight and planar. In this model there are only three parameters: the road width, the orientation and offset of the vehicle. The advantage of this model is its simplicity. Since the features to be detected (lane markers or road borders) are parallel, techniques based on vanishing point detection and Hough transforms work well. An example of detected lane markers found this way is shown in Fig. 1. Within a more sophisticated model, this technique could be used when the system is started (or reinitialized in the case of detected inconsistencies), However, modeling also the road horizontal curvature makes it possible to predict more accurately feature locations on curved sections, at a cost of only a few
822
D. Koller
et al.
additional parameters. The curvature information provides a look-ahead parameter which also helps to generate smoother steering commands. This approach is favored in some of the most successful systems, including UBM’s approach (see Section 3.4), YARF [25], one of the latter lane finders used in the NavLab project, the RALPH system (see Section 2.4), and Berkeley’s StereoDrive project (see Section 5). In Europe, the roads are built using the clothoid model described in more detail in Section 3.4, whereas in North America, a constant curvature model is used. This latter model is easier to implement, since it provides a closed-form solution (a circular arc is projected into a section of an ellipsis), which can even be linearized by a parabolic approximation [25]. In more refined representations, the three-dimensional road shape is also modeled. This is part of UBM’s approach. Other attempts make only the assumption that the road does not bank [9,21], or that the road edges are locally parallel [47]. These approaches didn’t appear to be robust: due to the lack of constraints, errors in feature localization could easily result in changes in road shape. 2.2. Structured Roads: Methods Based on Lane Markers On structured roads like highways, specific techniques can be applied to take advantage of the presence of lane markers, since their position, as well as their appearance, can be modeled. In [40] it is argued that the observation of a single point along a lane marker, together with its optical Aow, could be sufficient for road following. The fact that the observation of the lane flow is a reliable source of information for human drivers had been established in [16]. One typical approach [11,25,28] to find the road parameters is as follows: Predict new parameters for each lane marker. Define horizontal search bands in the image (an example is given Fig. 1) based on the predicted parameters and their uncertainties. This limits the amount of image processing to be done. Localize within the search zone the center of lane markers. This is done by applying an operator tuned to respond to the pattern corresponding to the lane marker. In the case of a bright bar, an example of such an operator is a convolution with a double derivative of a Gaussian (DDG) of a width matching that of the bar. An example of the points found by this method is shown in Fig. 1. Backproject the points just found to the ground plane. It is well known that there is a one-to-one correspondence between any plane in space and the image plane, which takes the form of a projective homography:
f
hiiX IC
= h31X
+ h12Y + hi3
+ h32Y + h33
4.6 Vision-based Automatic Road Vehicle Guidance 823
Fig. 1. The initialization of the algorithm is done by detection of portions of straight lines of common orientation (top left). Within the search zone predicted (top right), a precies localization of line markers points is performed (bottom).
where x,y are image coordinates and X, Y are coordinates in the 3-D plane. This plane is taken to be the road plane, assuming that the road is locally planar. The coefficients hij are defined up to a common scale factor, and depend on the camera parameters, as well as the position and orientation of the camera with respect to the road. They can alternatively be determined by specifying four points on the road and on the image. This construction is fundamental in autonomous driving and has been used by many authors. It makes it possible to build a bird-eye’s view from the ground-level view. In the bird-eye’s view important features such as lane markings, which converged towards the top of the original image, now appear parallel. Fit a new road model to the backprojected points, using a statistical fitting method, and the equation of the model. A linear least-median-of-squares (LMedS) algorithm is efficient and provides robustness against outliers. In the UBM’s approach [ll]a variant of this technique is used, which makes an even stronger use of temporal constraints. Provided that the correct operator is applied, this technique has proven to be very effective. The amount of image
824
D. Koller et al.
processing is relatively limited, and it is possible to take into account relatively sophisticated road models (i.e. with curvature), and use sound statistical techniques. Difficulties might arise if an inappropriate operator is used. To cope with this problem, the LANELOCK [22,23] and YARF [25] systems integrate several feature trackers. Another refinement made possible by the use of stereopsis in [28] is to exclude the areas which correspond to obstacles from the search bands, which is potentially useful in crowded traffic scenes. 2.3. ALVINN: A Learning-Based System
The system previously described relies on a precise modeling of the road appearance. In this section, we describe the opposite approach: systems with weak and flexible representations, designed to cope with the variation in road appearance. Systems which have to deal with unstructured roads have to cope with widely changing road appearances, and must be adaptive. They often have to rely only on the road boundary for road localization, and therefore use region-based segmentation schemes, using color [8], or texture [52]. A favored technique to recover the road parameters is parameter space voting methods: detected features vote for all possible roads they are consistent with. These techniques are relatively robust to segmentation errors, but they are impractical with more than three parameters, therefore they are compatible only with the simplest road representations. A typical example of a system for driving on unstructured roads is SCARF [8], which repeats the following steps: classify image pixels, find the best-fitting road model, and update the color models for classification. Alternative approaches that combine machine vision and machine learning techniques have demonstrated an enhanced ability to cope with variations in road appearance. ALVINN [37,38],which we describe next, is a typical system of this type. It evolved from the Navlab project which was initially concerned with all-terrain driving. Other related approaches are [24,29,41]. ALVINN (Autonomous Land Vehicle In a Neural Network) is a perception system which learns to control the NAVLAB vehicles by watching a person drive. ALVINN’s architecture consists of a single hidden layer back-propagation network. The input layer of the network is a 30 x 32 unit two-dimensional “retina” which receives input from the vehicle’s video camera. Each input unit is fully connected to a layer of five hidden units which are in turn fully connected to a layer of 30 output units. The output layer is a linear representation of the direction the vehicle should travel in order to keep the vehicle on the road. To drive the vehicle, a video image from the onboard camera is injected into the input layer. Activation is passed forward through the network and a steering command is read off the output layer. The most active output unit determines the direction in which to steer. To teach the network to steer, ALVINN is shown video images from the onboard camera as a person drives, and told it should output the steering direction
4.6 Vasion-based Automatic Road Vehicle Guidance 825 in which the person is currently steering. The backpropagation algorithm alters the strengths of connections between the units so that the network produces the appropriate steering response when presented with a video image of the road ahead of the vehicle. After about three minutes of watching a person drive, ALVINN is able to take over and continue driving on its own. Because it is able to learn what image features are important for particular driving situations, ALVINN has been successfully trained to drive in a wider variety of situations than other autonomous navigation systems which require fixed, predefined features. The situations ALVINN networks have been trained to handle include single lane dirt roads, single lane paved bike paths, two lane suburban neighborhood streets, and lined divided highways. In this last domain, ALVINN has successfully driven autonomously at speeds of up to 70 mph, and for distances of over 90 miles on a public highway north of Pittsburgh. While systems of this type have been quite successful at driving on a wide variety of road types under many different conditions, they have several shortcomings. First, the process of adapting to a new road requires a relatively extended “retraining” period, lasting at least several minutes. While this adaptation process is relatively quick by machine learning standards, it is unquestionably too long in a domain like autonomous driving, where the vehicle may be traveling at nearly 30 meters per second. Second, the retraining process invariably requires human intervention in one form or another. These systems employ a supervised learning technique such as back propagation, requiring the driver to physically demonstrate the correct steering behavior for the system to learn. One attempt to solve these problems has been to train specialized networks for each road type. In order to overcome the problem on which network to use, a connectionist superstructure, MANIAC [19], incorporates multiple ALVINN networks with the hope that the superstructure would learn to combine data from each ALVINN network. 2.4. Ralph: A Hybrid System
ALVINN relied entirely on learning, which has been found to be insufficient, therefore its successor, RALPH [39] uses a more hybrid approach which involves a partial modeling of road features. Both systems (ALVINN and RALPH) use simple calculations on low-resolution images which makes it possible to implement them in real time. Temporal continuity however is not exploited. It is argued [39] that even for driving on structured roads, adaptive systems are necessary. There are some variations in road markings depending on the type of road (e.g. suburban street us. interstate highway), and the state or country in which it is located. For example, many California freeways use regularly spaced reflectors embedded in the roadway, not painted markings, to delineate lane boundaries. Further challenges result from the fact that the environmental context can impact road appearance. RALPH is an adaptive approach where two parameters are implicitly determined. These parameters are the road curvature, and the vehicle’s lateral position
826
D. Koller
et al.
Fig. 2. (a) RALPH’s control panel showing the view of the road and the resampled bird-eye’s view. (b) The mapping of the image into a bird-eye’s view. Illustration courtesy of D. Pommerleau.
Fig. 3. RALPH’s method for determining curvature. Illustration courtesy of D. Pommerleau.
Fig. 4. RALPH’s method for determining offset. Illustration courtesy of D. Pommerleau.
relative to the lane center. The latter parameter is used to generate directly a steering command. RALPH does not take into account other parameters such as vehicle heading, or variations between the road plane and the camera position and orientation.
4.6 Vision-based Automatic Road Vehicle Guidance 827 In the RALPH system, the image is first sub-sampled and transformed so as to create a low resolution (30 x 32 pixels) bird-eye’s view image, as illustrated in Fig. 2. To determine the curvature of the road ahead, RALPH utilizes an “hypothesize and test” strategy. RALPH hypothesizes a possible curvature for the road ahead, subtracts this curvature from the parallelized low resolution image, and tests to see how well the hypothesized curvature has “straightened” the image. After differentially shifting the rows of the image according to a particular hypothesis, columns of the resulting transformed image are summed vertically to create a scanline intensity profile. When the visible image features have been straightened correctly, there will be sharp discontinuities between adjacent columns in the image. By summing the maximum absolute differences between intensities of adjacent columns in the scanline intensity profile, this property can be quantified to determine the curvature hypothesis that best straightens the image features. The next step in RALPH’Sprocessing is to determine the vehicle’s lateral position relative to the lane center. This is accomplished using a template matching approach on the scanline intensity profile generated in the curvature estimation step. The scanline intensity profile is a one-dimensional representation of the road’s appearance as seen from the vehicle’s current lateral position. By comparing this current appearance with the appearance of a template created when the vehicle was centered in the lane, the vehicle’s current lateral offset can be estimated. Because RALPH uses procedural methods to determine the two relevant parameters, the techniques assume only that there are visible features running parallel to the road, and that they produce a distinct scanline intensity profile, which is more general than other schemes. A second strength of the approach stems from the simplicity of its scanline intensity profile representation of road appearance. The 32-element template scanline intensity profile vector is all that needs to be modified to allow RALPH to drive on a new road type. There are several strategies on how to adapt this template to changing conditions: 0
0
0
0
A human driver centers the vehicle in its lane, and presses a button to indicate that RALPH should create a new template. RALPH selects one from a library of stored templates recorded previously on a variety of roads. RALPH slowly “evolves” the current template by adding a small percentage of the current scanline intensity profile to the template. RALPH uses the appearance of the road in the foreground to determine the vehicle’s current lateral offset and the curvature of the road ahead, as described above. At the same time, RALPH is constantly creating a new “rapidly adapting template” based on the appearance of the road far ahead of the vehicle (typically 70-100 meters ahead). If the appearance of the road ahead changes dramatically, the new template is used.
There are several possible drawbacks of lacking an explicit representation of the geometry of the road and of the vehicle position: the approach generates only
828
D. Koller
el al.
very coarse steering actions, it is difficult to apply refined control strategies (which require precise measurements of the vehicle status) in order to obtain a smooth ride. It might also be more difficult to plan more complex maneuvers such as lane changes.
3. Road Following by Recursive Parameter Estimation 3.1. Theoretical Background Visual-based road following is a special application of temporal image sequence processing. Using temporal image sequences is also referred to as dynamic scene analysis, which means we are analyzing dynamically changing scenes, opposed to stationary scenes. The only changes which can occur in temporal image sequences are changes due t o moving objects, a moving observer, or a change in lighting conditions. Now Physics tells us all about the dynamic behavior of moving objects or moving observers by using well established (continuous) differential equations (socalled motion equations). These motion equations describe the motion of movable objects under the application of forces. The simplest motion is the force-free motion, in which case a moving object keeps its state constant over time. Such a simple motion, however, plays only a theoretical role, since we always have to cope with at least some kind of frictional forces. Motion equations are usually established by variation methods minimizing the action, resulting in Euler-Lagrange equations (see e.g. [15]). Motion equations are second order differential equations of the spatial coordinates which involve the spatial coordinates, their velocities, and the time. In this context motion estimation or model-based tracking refers to estimating the motion parameters according to a motion model. We first provide a brief background before we describe the most popular parameter estimation method in J Section 3.2. A very crucial issue is establishing a correct motion model, which should approximate the real motion as close as possible. We will address this issue in Section 3.3. We conclude this section by describing one of the most successful approaches for road following using recursive parameter estimation techniques in Section 3.4. 3.1.1. Continuous time linear dynamic systems We start by considering a system whose dynamic behavior is reasonably well approximateda by a set time differential equations, also known as plant equations, or motion equations in the case of moving objects:
i ( t )= A(t)s(t)+ B(t)u(t)
use the term approximate here, since we cannot expect to have an exact description of the dynamic system.
4.6 Vision-based Automatic Road Vehicle Guidance 829 with
s ( t ) = the system state vector a t time t
u(t)= the system input or control vector at time t
A(t) = the system matrix a t time t B ( t ) = the input gain matrix at time t . The system state s is defined to be the smallest vector which summarizes the entire state of the system. The system output z ( t ) , which is actually observable is, in general, a vector of a dimension less than the system state:
with the measurement matrix C ( t ) ,which maps the system state t o the measurement. Other important terms in the context of continuous time linear dynamic systems are: (i) controllability, which states that for any arbitrary combination of initial and destination point in state space there is a n input function for which the destination state can be reached from the initial point in finite time, and (ii) observability, which holds, if the initial state can be fully and uniquely recovered in finite time from its observed output and given input. 3.1.2. Discrete time linear dynamic systems In image sequence analysis, however, we have to deal with discrete time linear dynamic systems, with the time step being the inverse video-rate. In this case we have t o turn the continuous differential equation (3.1) into a time discrete dafference equation:
sk+l = Fksk
+ GkUk
(3.3)
with Sk+l
Uk
= the system state vector at time t k = the system input or control vector a t time t k
Fk = the transition matrix at time G k = the
tk
input gain matrix at time t k .
The transition and input gain matrix are obtained in a straightforward manner when the system matrix and the input gain matrix of the continuous case are integrable: S k + l is the result of integrating Eq. (3.1) along a time interval T = t k + l - t k . The measurement equation is then given by:
830 D. Koller et al.
with the measurement matrix Hk at time t k . Similar expressions for controllability and observability are defined as for the continuous case, where finite time is replaced by a finite number of steps and finite number of observations, respectively. 3.2. Linear Recursive Estimation i n Dynamic
Systems - Kalman Filtering We do not provide a discussion of all necessary properties which lead to the so-called Kalman filter equations given below. We will just summarize the basic concept and refer the interested reader to one of the excellent text books [ 2 , 1 4 , 3 2 , 4 3 ] . We confine the equations to the discrete time case. Consider a discrete time linear dynamic system described by a vector difference equation with additive white, Gaussian noise that models unknown disturbances or inaccuracies in the plant equation:
with the n-dimensional state vector S k , the m-dimensional known input or control vector u k , and W k a sequence of zero-mean white Gaussian process noise with covariance E [ W k W F ]= &kb. The measurement equation is
where z k denotes the 1-dimensional measurement vector and V k is a sequence of zero-mean white Gaussian measurement noise with covariance E [ v ~ v=~ Rk. ] The matrices F k , G k , H k , Q k , and Rk are assumed to be known and possibly time varying. The measurement noise and process noise are assumed to be uncorrelated: E [ V k W T ] = 0, vl, k. Regarding Eq. ( 3 . 5 ) , the question is how do we estimate the state vector s k from measurements z k corrupted by noise and given an initial estimate $0 with initial covariance PO. Optimal estimates that minimize the estimation error, in a welldefined statistical sense, are of particular interest. The optimal estimate of s k will be denoted by $ k . If we have more measurements than parameters to estimate ( I > n) we have an over-determined system and can apply weighted least squares estimation and obtain (we dropped the time index k here):
i=(H~R-'H)-'H~R-~Z,
(3.7)
which reduces to the least squares (pseudo-inverse) solution for uniform measurement noise (R = a21). We obtain the same solution if we apply a probabilistic approach using maximum a-posteriori (MAP) estimation.
b~r
denotes the transpose of vector W k , and E [ x ]computes the expectation value of vector x.
4.6 Vision-based Automatic Road Vehicle Guidance 831 3.2.1. Discrete Kalman filter
A major breakthrough in parameter estimation has been achieved in [20] by formulating a time recursive estimation technique, which is now called the Kalman filter (KF) technique. Filtering refers here to estimating the state vector at the current time, based upon all past measurements. Kalman formulated and solved the Wiener problem for gauss-markov sequences through state space representation and the viewpoint of conditional distributions and expectations. His results reduce to Eq. (3.7), which concludes that for Gaussian random variables weighted least squares, MAP, and Bayes estimation lead to the same result as long as the same assumptions are being used. Given a prior estimate of the system state at time t k , denoted G,, we seek an updated estimate, G l , based on use of the measurement Z k . In order to avoid a growing memory filter, this estimate is sought in a linear, recursive formC
where K; and K k are yet to be determined time-varying weighting matrices. The optimum choice of these matrices can be obtained by considering that the estimator should be unbiased and the associated updated error covariance matrix should have a minimal weighted scalar sum of the diagonal elements (e.g. [14]). The result are the so-called Kalman filter equations for discrete time linear dynamic systems:
(3.11)
Equation (3.9) is referred to as the state update equation and Eq. (3.10) is called the state error covariance update equation. K k denotes the so-called Kalman gain and I stands for the identity matrix. The transition equations being used for computing predictions for the state and the state error covariance become: .-
sk+l
= Fkb;
P;+'
=F
+
GkUk
k P z F F
+W k .
(3.12) (3.13)
Kalman's technique, originally only formulated for linear motion equations and update equations with a linear relation between measurements and the estimated state, has been extended to apply also to nonlinear equations, by linearizing the problem and using only first order terms - the so-called extended Kalman filter CThroughoutthe text we denote with + and - entities immediately before and immediately after a discrete measurement, respectively.
832
D. Koller et al.
(EKF). Using an iterative minimization method is also referred to as iterated extended Kalman filter (IEKF). 3.2.2. Discrete extended Kalman filter
In an extended Kalman filter the measurement equation is linearized by expanding the nonlinear measurement function hk up to first order:
where Hk(gL) is the Jacobian of the measurement function h k . We can still use the nonlinear plant equation for the state vector corresponding to Eq. (3.12), however, we have to linearize the transition function fk to predict the state covariance. Instead of Eq. (3.13) we use:
where Fk(6;)
is the Jacobian of the transition function fk.
3.2.3. Kalman filter summary
A Kalman filter is basically a time recursive formulation of weighted least squares and exhibits the following key features: Recursive: Time recursive means that we do not need to store previous images or measurements, opposed to applying batch weighted least squares, where all images (measurements) need to be available at once. This makes it well suited for real-time applications, where we have only one image at a time. Predictive: Based on current estimates and on the applied motion model we can compute a predictive state for the next image frame. This predicted state is an expectation and enables new measurements to beconfined to a limited (expected) search space in the image. This provides a major performance improvement, also a requirement for real-time performance. Optimal: Kalman filters are optimal in the sense of the minimization method. For the linear case it can be shown that it provides the optimal solution. The nonlinear case provides only sub-optimal solutions.
4.6
Vision-based Automatic Road Vehicle Guidance
System (real world)
System Error
System state s ( t ) Uncertainty P ( t )
Transition function f ( t )
P-(t
Observation z ( t ) Initialization i0,PO
833
-
+ 1)
1
Update (internal world)
Prediction
Fig. 5. Block diagram of a Kalman filter: a real world system is supposed to be completely described by a state vector s ( t ) and its uncertainty P ( t ) . The system dynamic behavior is covered by the transition function f(t) and the system (process) noise. An internal representation of the system - initialized by SO and Po - is being updated using a measurement (observation) z ( t ) resulting in system state estimates s + ( t ) and Pf(t). These values are then used, after a time delay, to compute predictions s - ( t + 1) and P-(t l ) , which in turn support the measurement process for the next time step.
+
Another nice feature of using Kalman filters is that by exploiting dynamical models, all state variables of the state vector can be recovered, even though only a subset of the output variables can be observed a t a time. Such a situation is inevitable in vision, since images basically allow only 2-D measurements, whereas the process t o be recovered is described in 3-D. A block diagram overview of the Kalman filter technique is given in Fig. 5. 3.3. Scene Modeling Applying Kalman filtering to dynamic scene analysis requires an extensive modeling of the entire scene with spatial representations of all objects (movable and non-movable) and motion models (motion equations) for all movable objects. In dynamic scene analysis we have to distinguish between objects and subjects: objects denote those entities which just obey physical laws without any internal mission (e.g. a ball kicked by a soccer player), whereas subjects are entities with a certain mission or goal (e.g. the soccer player who kicked the ball). Subjects are much harder to model, since they can exhibit a very complex behavior. In the con-
834
D. Koller et al.
text of traffic scenes and driving we have mainly t o cope with subjects, e.g. other cars, being driver by a driver with a certain goal, or pedestrians. An easy and straightforward approach, however, is to model subjects like objects with their simple physical motion behavior and model any unpredictable deviations due to higher level decisions of the subject as process noise in the Kalman filter. This approach is often feasible, since the time sampling rate of the input measurements is high compared to the high level changes in the motion of subjects. 3.4. Dickrnanns’
4 - 0 Approach
The first successful and by far the most important and impressive work on a visually guided autonomous vehicle has been done in the group of Prof. E. D. Dickmanns of Universitat der Bundeswehr, Munich, Germany. (See [ll]and the references cited therein.) Their work resulted in a demonstration in 1987 of their 5-ton van, the VaMoRs (Versuchsfahrzeug fur autonome Mobilitat und Rechnersehen) , running autonomously on a stretch of the Autobahn (freeway) at speeds of up to 100 km/h (only limited by the power of the engine). Increased computer performance and smaller and lighter hardware allowed them since 1994 t o modify their system to be implemented in a passenger car, the VaMP (VaMoRs Passenger) car, a Mercedes Benz S-class car. This car is running autonomously at normal cruising speeds of about 130 km/h using bifocal visual sensor input, which provides a long enough look-ahead range. A long distance test ride between Munich (Germany) and Odense (Denmark) in November 1995 has been reported in [3]. Their current implementation employs a set of transputers and PowerPCs [31]. The key feature of Dickmann’s approach is what he coined the 4 - 0 approach, which makes extensive use of spatial-temporal models of objects and processes. His approach works very well since for this type of application he can use special spatial and temporal models, the parameters of which are recursively updated while the vehicle is driving. 3.4.1. Spatial-temporal models
-
4 - 0 approach
The basic idea of this 4 D approach, illustrated in Fig. 6 , is to generate an internal representation of the real world using measurements from the real world. The internal representation consists of models instantiated through model hypotheses verification and model parameter estimation. Measurements from the real world are compared to predictions from the internal representation, which describe geometric and dynamic models. The entire internal model is parameterized through a state vector which is being updated based on the difference between the measurements and predictions using minimal square error estimation. This time recursive estimation process is basically a n extended Kalman filter, applied to the problem of state reconstruction from image sequences. The non-linear mapping from 3-D shape parameters and 3-D object states into 2-D image feature locations by perspective projection is locally approximated by first order relations and covered by the
4.6 Vision-based Automatic Road Vehicle Guidance 835
Fig. 6. The 4-D approach to visual road tracking. The left part symbolizes the real world (3D + time). Snapshots are thinned down to a set of features by image processing techniques. Objects are represented by their state parameters, which describe geometric and dynamical models. Illustration courtesy of R. Behringer.
Jacobian matrix for the measurement function as described in Subsection 3.2.2. It turns out that a crucial parameter in these equations is the focal length determined by the camera lens. This is why bifocal vision is used, one camera with a wide angle lens providing a large field of view and a second camera with a tele lens for the large look-ahead ranges required for high speed driving. In addition to visual input from video cameras they are also using conventionally measured data like velocity from the odometer. The Kalman filter application is therefore embedded in a sensor fusion approach. The next subsection covers a description of the road and vehicle model used for lateral vehicle control. Further development of this work has been in collaboration with von Seelen’s group in Bochum [44,55] and the Daimler Benz VITA project [48,49], in order to combine both lateral and longitudinal control. We will discuss the longitudinal control in the context of obstacle detection in Section 4. Their system is now also capable of performing autonomous lane changes on request ([lo]). 3.4.2. Road and vehicle modeling for lateral control Dickmanns’ work models roads in accordance to the European road layout using the so-called clothoid model. Front-wheel steered road vehicles follow a clothoid path when driven at constant speed and at a constant steering rate, which is a reasonable driver model. This is why civil engineers in Europe build roads in accordance with clothoids. Clothoids are planar curves which are characterized by their linear change in curvature C = 1/r, the inverse of the turning radius:
836 D. Koller et al.
C = Co + dC/dl * 1 = Co + C1* 1 .
(3.17)
COis a constant and 1 is the afc length, i.e. clothoids change their curvature linear with the arc length. C1 = 1/A2 is piecewise constant and A is the so-called clothoid parameter. An essential task for smooth road vehicle guidance is hence to recover the clothoid coefficients COand C1 of the road using vision. The arc length 1 is conventionally recovered from velocity readings of the odometer, since the cycle time is known. Since the ideal skeletal line of the road is usually not visible, the road parameters have to be recovered using only the visible parts of the road boundary. In order to robustly detect the road boundary and to resolve the ambiguities between image features from road boundaries and from shadows on the road cast from trees, buildings and other structures next to the road, the assumption of parallel road boundaries is being applied. Dickmanns also accounts for vertical road curvature by applying a 3-D road model. The vertical mapping geometry is mainly determined by the camera position above the local tangential plane and the camera pitch angle. It is assumed that both, horizontal and vertical road curvatures are so small compared to the visual lookahead range, that they can be recovered independently by decoupled differential equations, which makes the problem much more tractable. In addition to the two decoupled horizontal and vertical road curvature models, he applies a simplified but sufficient dynamic model for the vehicle motion and steering kinematics. This model accounts for the slip angle due to softness of the tire and tire slipping which itself is constrained by a differential equation. The overall dynamical model for 3-D road recognition and relative vehicle egomotion estimation consists of three subsystems: (i) for the lateral vehicle dynamics, (ii) for the horizontal, and (iii) for the vertical road curvature dynamics. The full state vector comprises a total of nine parameters. However, the system can be almost completely decoupled into the above three subsystems of which only the horizontal road curvature affects the lateral vehicle dynamics. Finally a prediction error feedback scheme for recursive state estimation according to the extended Kalman filter technique (Subsection 3.2.2) is being applied for continuous state update. Special care has to be taken during the initialization phase, when good object hypotheses are required, a crucial issue in all Kalman filter applications. Dickmanns applies certain constraints about the size of the road and also assumes low initial road curvature as well as a normal initial vehicle position to help the initialization phase. Image measurements are taken in subwindows given by the predicted image location of the road boundary. 4. Monocular Obstacle Detection The problem of obstacle detection and avoidance emerges in the context of longitudinal vehicle control and in car following. The question here is how can computer vision be used to keep a safe driving distance to a car driving in front.
4.6 Vision-based Automatic Road Vehicle Guidance 837 The latter is of special interest for the platooning concept being studied in the
PATH project [50]. Keeping a safe driving distance requires: (i) detecting cars driving ahead on the road, (ii) tracking a car driving ahead, and (iii) accurately measuring the cars’ changing relative distance for speed control. Contrary to stereo approaches, single camera approaches for obstacle detection require additional information in order to reconstruct the depth coordinate, which is lost in the projection from 3-D objects points to 2-D image points. In this section we describe some of the attempts at recovering range information from monocular images.
4.1. Optical Flow Based Methods Classical approaches for monocular obstacle detection are based on motion stereo or optical flow interpretation [6,12]. The key idea of these approaches is to predict the optical flow field for a moving observer under constraint motion (e.g. planar motion). Obstacles are then detected by a significant difference between the predicted and the actual observed optical flow field. The major drawbacks of these approaches are: (a) computational expense and (b) the lack of reliable and accurate optical flow fields and the associated 3-D data (it is well known that structure-from-stereopsis approaches perform better than structure-from-motion approaches). A combination of stereo and optical flow is suggested in [7] in order to perform a temporal analysis of stereo image sequences of traffic scenes. They do not explicitly address the problem of obstacle detection in the context of vehicle guidance, but the general problem of object identification. They extract and match contours of significant intensity changes in (a) stereo image pairs for 3-D information and (b) subsequent frames to obtain their temporal displacement ([34,35]). Object descriptions are finally obtained by grouping the Kalman filtered 3-D trajectories of these contours using a constant image velocity model. In order to distinguish between obstacles and road boundaries or lane markers, they also exploit some heuristics like horizontally and vertically aligned contour segments as well as 3-D information extracted from the stereo data [33]. 4.2. Methods Based on Qualitative 3-0 Reconstruction
Unlike another group of conventional approaches for obstacle detection based on full 3-D reconstruction, which is known to be error prone, it has also been suggested to use only qualitative 3-D reconstruction [53,54]. Reference [54] describes three algorithms for obstacle detection based on different assumptions: two of them just return yes/no answers in terms of the presence of an obstacle in a view without 3-D reconstruction, only based on the solvability of a linear system which expresses the consistency of a set of points under the same motion. Their third algorithm is quantitative in the sense that it continuously updates ground plane estimates and reconstructs partial 3-D structures by determining the height above the ground plane for each point in the scene.
838 D. Koller et al.
4.3. Model-Based Approaches
Other methods for monocular obstacle detection exploit the use of spatial obstacle models. However, full model-based approaches require detailed models of the obstacles, which is not feasible for car following applications or collision avoidance.
4.3.1. Detecting obstacles using assumptions about mirror symmetry CARTRACK system
-
the
A more recent approach exploits heuristics such as symmetry of the bounding box of the vehicle in front, which is based on the fact that rear or front views of most vehicles exhibit a strong mirror symmetry about the vehicle’s vertical axis. This symmetry provides a striking generic shape feature for object recognition [44,55]. They start by using an intensity-based symmetry finder to detect image regions that are candidates for a leading car. The vertical axis from this step is also an excellent feature for measuring the leading car’s relative lateral displacement in consecutive images because it is invariant with respect to vertical movements of the camera and changes in object size. To exactly measure the image size of a leading car, a novel edge detector is being proposed which enhances pairs of edge points if the local edge orientation at these locations is mutually symmetric with respect to a known symmetry axis. The 2-D symmetry is formed by a systematic coincidence of 1-D symmetries and is hence well suited for parallel processing. The problem is then to detect local 1-D symmetries along horizontal scan lines (only strict vertical symmetry axes are considered).
A Local Symmetry Measure: Reference [55] defines a measure for local symmetry within a 1-D intensity function by means of a contrast function: J E n ( u , z s , w ) 2 d u- J O ( U , X s , W ) 2 d U S(XS,
w) =
JEn(U,2,,W)2dU
+
JO(U,2s,W)2dU
(4.1)
where
denotes the odd part of the intensity function I ( x ) ,and
En(u,z,, w ) := E ( u ,z,, w)
-
w
s”i2
E ( v ,zs,w)dv
-w/2
(4.3)
is the normalized even part of the intensity function I(z) (corrected by the bias in order to compare it with the odd counterpart) with
4.6 Vision-based Automatic Road Vehicle Guidance 839
The parameter x, stands for the location of a potential symmetry axis and w is the width of the symmetry interval about 2,. Since they are interested in the maximum symmetry support interval along a scan line, they introduce the following confidence measure for the hypothesis that there is a significant symmetry axis originating from an interval of width w about the position x,: s A ( x s , w )=
W
-( S ( x s ,W )+ 1) 2Wmax
w 5 wrnax.
(4.5)
The values for SA(Z,, w) are recorded in a symmetry histogram. Two-dimensional symmetry detection requires the combination of many such symmetry histograms. This is easily accomplished by summation of the confidence values for each axis position, provided the symmetry axis is straight.
Symmetry Enhancing Edge Detection: Edge detection is finally performed using a feed-forward network whose connection weights represent a symmetry condition, which results from eight oriented filter outputs at two different image locations. It combines evidence for the different categories of discrete orientation symmetry and can serve as a mechanism for detecting edges that are related to other edges by a certain degree of symmetry. The CARTRACK system: The described approach has been applied to detecting and tracking cars with video input from the viewpoint of a driver. The symmetry finder produces a symmetry histogram, the peaks of which are tentatively taken t o represent the horizontal object position. The symmetry enhancing edge detector is then used to verify and improve the localization as well as to estimate the width of the leading car. However, additional heuristics, which need to be confined on the road like cars and obstacles, are required for robust performance, since the proposed symmetry-based approach also picks up any vertical mirror-symmetric object like road signs or certain buildings. This symmetry-based approach for obstacle detection and longitudinal vehicle control has also been successfully combined with the road detection and lateral control system of Dickmanns (cf. Subsection 3.4). 5. The Use of Stereopsis for Autonomous Driving In real traffic scenes, other vehicles are usually present, and this raises two problems. First, they are potential obstacles, which need t o be detected. Second, lane markers are often obstructed by other vehicles, which might defeat the algorithms which do not allow for occlusion of the lane markers.
840
D. Koller et al.
Stereopsis uses two or more cameras to obtain actual range information of every object visible by both cameras. In this way an easy distinction between road plane and obstacle is available. For instance, stereopsis with linear cameras was considered in the PROMETHEUS project [5]. A system based on stereopsis was first advocated in [26]. While providing more information, historically this approach has been considered to be computationally too expensive. With the emergence of more powerful processors however, they are becoming common and even necessary for fully autonomous vehicles, as seen in the 1996 edition of the Intelligent Vehicles Symposium. This section will outline the use of stereopsis for autonomous driving. It will examine the exploitation of domain constraints to simplify the search problem in finding binocular correspondences. This reduces the computational load of stereopsis correspondence. Temporal integration of the results of the stereo analysis is used to build a reliable depth map of obstacles. In crowded traffic scenes where substantial portions of the lane boundaries may be occluded, this makes the road boundary detection more robust. In addition to supporting longitudinal control (i.e. maintaining a safe, constant distance from the vehicle in front) by detecting and measuring the distances to leading vehicles, stereopsis also measures the relative position of the road surface with respect to the vehicle. Measurements of the road surface are used for dynamic update of (a) the lateral position of the vehicle with respect to the lane markers and (b) the camera parameters in the presence of camera vibration. Lane markers are detected and used for lateral control, i.e. following the road while maintaining a constant lateral distance to the road boundary as discussed in the previous sections. Since the areas of the image belonging to the ground plane are identified, this ensures that the search area for lane markers is not corrupted by occlusions.
I
geometric parameters
I-
Dynamic Disparity
search area for
localization of lanemarkers
-
description of lane markers
longitudinal control
control
Fig. 7. The flow of information in the integrated stereo approach.
4.6 Vision-based Automatic Road Vehicle Guidance 841 The principle of the approach, which was presented in [28,51] is illustrated in Fig. 7. Recent developments include a real time implementation described in [45]. A similar approach is [4]. 5.1. Stereo Imaging of a Planar Surface
In order to simplify the identification of potential obstacles and also limit the required computational load of stereo disparity, a stereopsis based system for autonomous driving makes the assumption that there exists a planar surface within the visual field of the stereo sensors. This surface is the driveable road plane. Obstacles are represented by objects which are either above or below this plane. Under the assumption of a planar surface viewed by a stereo camera system consisting of two cameras with parallel optical axes, the resulting disparity of imaged points from that surface is a linear function of the image coordinates. That is, from the relative orientation of the planar surface and intrinsic camera parameters, the disparity in the image is a predetermined function linear in both image coordinates. In this section the equations relating the relative orientation of the road plane to this linear function are derived. They lead to a simple linear least squares solution based on the measured disparities. Assume two identical cameras with parallel optical axes. The baseline separation between camera centers is b. For such a camera setup, it is well known that a visible point in the world is projected onto the same row in both images. This is equivalent to the epipolar lines relating points between the two cameras being horizontal. The column number of the point projections in the two images will be different however. This disparity between column numbers is a function of the distance Z of the point. This distance is measured along the optical axis. Assuming a pin-hole camera model where world points (X, Y,Z ) are projected to image points (z,y) via z=
X Y f-, y = f-
z
z
the pixel disparity between left and right cameras is
A plane in front of the cameras can be parameterized by a unit vector, n, normal to the plane, and a perpendicular distance from the plane. to the center of the baseline connecting the two camera centers (Fig. 8). The components of the normal vector can be written in the coordinate system of the cameras. Its components are n = ( A ,B , C) = (sin0 cos 4, sin 0 sin 4, cos 0)
(5.3)
where 0 is the inclination angle of the plane from the line of sight direction, Z , and 4 is the roll about this direction. Any point on this plane follows the following
842
D. Koller
et al.
Fig. 8. A plane represented by a unit normal vector and distance, h. The normal vector can be written in terms of the inclination and roll angles (0 and 4) of the camera setup.
constraint on its coordinates:
AX
+ BY + C Z = h .
(5-4)
Using the projection equations (5.1), this constraint can be written in terms of image coordinates: AZx/ f B Z y / f CZ = h . (5.5)
+
+
Solving for 2 : Z
= h(Ax/f
+ B y / f + C)-l.
(5.6)
Using this in the disparity equation (5.2), we find that the disparity is linear in both image coordinates x and y:
b d ( ~ , y= ) -(Ax h
+ B y + fC)= CYX + p y + 7 .
(5.7)
This equation proves that the disparity of points on a plane imaged by a stereo camera system consisting of identical cameras with parallel optic axes will be a linear function of image coordinates, independent of the orientation of that plane with respect to the cameras. Note that when the camera roll is zero (4 = 0 ) , the coefficient a! is zero. The disparity is then a linear function of image row number, y . In fact, the disparity will be zero at the horizon line in the image, and linearly increasing from that line. This linearity in the image plane is referred to as the HeZmhoZtz shear, a configuration where the process of computing the stereo disparity is tremendously simplified. This insight is due to Helmholtz [17] who more than a hundred years ago observed that objectively vertical lines in the left and the right view perceptually appear slightly rotated. This led him to the hypothesis that the human brain performs a shear of the retinal images in order to map the ground plane to zero disparity.
4.6 Vision-based Automatic Road Vehicle Guidance 843 Then, any object above the ground plane will have non-zero disparity. This is very convenient because the human visual system is most sensitive around the operating point of zero disparity. Given a collection of disparity measurements in the image, it is trivial to recover the parameters of the plane being imaged. The linear relationship between disparity and plane parameters makes this a standard least squares problem. Thus, given at least three, non co-linear disparity measurements di(zi,yi), we can recover the height h and orientation e l # of the road plane by using a least squares fit to the data: N
From the fit parameters a,
and y we can recover the plane parameters via:
b
h= Ja2
+ p2 + y 2 / p
Note that in Eq. (5.7) the constants relating image coordinates to disparity are all scaled by the inverse of the height of the plane, h. This can be used to determine if a point is above or below the plane by a certain distance. That is, given a disparity d at location (2, y), Eq. (5.7) gives the height of this point above or below the plane. Based on this, each point can be labeled as being either on the plane, within a small tolerance, or an object which is not on the plane and therefore an obstacle to avoid. 5.2. Effects of Error on the Ground Plane Estimate Stereo disparities are known to contain a fair amount of noise. Uncertainty in matching between left and right images due to changes in lighting, inexact camera calibration and different viewpoints, leads to errors in the disparity measurement. This error is usually modeled as consisting of both a stochastic component and an outlier component. The first coming from the finite pixel size of the image, quantization effects and random noise in the intensity values. The second comes from mis-matches in the disparity computation. Whereas the first component is usually on the order of one pixel or less, the second can be many pixels in magnitude. Since the ground plane parameters are recovered from a standard least squares solution, the effect on the solution due to stochastic noise in the data is well known. If this noise is modeled as Gaussian, then the propagation of noise from data to solution is straightforward. For a noise term with standard deviation 0 , the resulting noise component in the solution will have standard deviation roughly equal to t w / n where K. is a scale factor dependent on the measurement distribution. Since the entire image is used to collect disparity measurements, the number will be on
844
D. Koller et al.
the order of thousands of measurements. This makes the solution quite robust to stochastic noise in disparity measurements. In addition to the magnitude of the noise, the error in the solution depends also on the distribution of the disparity measurements in the image. Because of the large number of measurements, it is this term which could affect the solution the most. If the data lie on a single line in space, then no unique solution to the plane fit exists. When the data is close to co-linear, small errors in measurements can lead to large errors in the solution. This can be minimized by using many disparity measurements from different regions of the image. Whereas the effect of stochastic noise on disparity measurements is small when there is a sufficient number of measurements, the solution can be affected strongly by an error in camera calibration. The solution for the ground plane’s relative orientation with respect to the camera developed in Section 5.1 assumed that the optic axes of both cameras are parallel. If the axes are not parallel it can have a large effect on the accuracy of the estimate. For small angles, the error in estimated 4 as a function of relative pitch angle, p is 4h (5.10) 64-.-i;-P while the error in estimated 8 as a function of vergence angle, v is 2h 68 -. - v . b
(5.11)
Since the height of the stereo camera rig is often larger than the baseline separation, the scale factor h/b is greater than one. In our test vehicle the baseline separation was 40 cm while the height above the road plane was 1.8 m. Larger baseline systems placed closer to the road will show fewer effects from calibration error. Figure 9 shows the estimated horizon for two cameras whose optical axes were approximately two degrees from parallel, both in vergence and relative pitch. The estimated horizon in the top image, calculated by assuming the axes were parallel, is clearly incorrect. The second image shows the same estimate after the error in calibration is taken into account. The deviation from parallel can be accommodated by warping either the ihages, or the resulting disparity map by an affine warp. This is due to the fact that a pure camera rotation can be compensated for via an affine warping based on the rotation matrix.
5.3. Binocular Stereopsis Although proposed algorithms in the literature for computing binocular stereopsis are quite computationally expensive, the complexity can be reduced considerably by using region-of-interest processing and exploitation of domain constraints. Region-of-interest processing recognizes that in regions of uniform image brightness, the stereo correspondence between camera views cannot be determined accurately. To reduce the computational load, the search for disparity is not performed in
4 . 6 Vision-based Automatic Road Vehicle Guidance 845 -
Fig. 9. Estimated horizon location for a pair of cameras with optical axis about two degrees away from parallel. The lower image is the estimate obtained after the images are restored t o parallel axes configuration via an affine warping of one of the images.
those regions. The domain constraints provide the Helmholtz shear described in Section 5.1 which limits the range of possible disparities thus reducing the search requirements. The disparity between images is found by computing the normalized correlation between small horizontal windows in the two images at the locations of the pointsof-interest. The normalized correlation for disparity shift T , a t horizontal image location z is: (5.12) where the correlations
ai,j (z, y)
are approximated by summations
c
+W/2
ai,j(.,Y)
=
gi(z
+ u ) g j ( y + u).
(5.13)
u=-w/2
The subscripts ( i , j ) can be either E or T , representing the left or right image. The summation is calculated over a window of size W .
846
D. Koller
et al.
(4 (b) Fig. 10. (a) view from the left camera. (b) light region indicates objects were detected t o be on the road surface, dark indicates objects are above the road surface. Points-of-interest are those locations in the right image where the value of ur,r is above a threshold. The normalized correlation function is calculated only in those regions. Sub-pixel disparities are obtained by quadratic interpolation of the function about the maximum r. Residual disparities - which appear in the image after the ground plane disparity has been mapped to zero - indicate objects which appear above the ground plane. A simple threshold is used t o distinguish between features lying on the ground plane (e.g. lane markers or other stuff painted on the road) and features due to objects lying above the ground plane (which may become future obstacles). Figure 10 shows the result on a single frame. 5.4. Determining Ground Plane Objects
The least squares solution for the ground plane’s relative orientation assumed that all disparity points belonged to the ground plane. In this way a least squares estimator could be used. In reality there will be many objects in the field of view which do not lie on the ground plane. If the disparity measurements of these objects were used, the ground plane estimate would be incorrect. Therefore only the parts of the image corresponding to points on the road surface should be used for the update. In order to accomplish this the relative orientation of the surface needs t o be known. Figure 11 plots the disparity measurements versus image row number for all points in a cluttered traffic scene. If the imaged points all lay on a single plane in front of the cameras (and there was no camera roll, which is assumed here) then all of the points in this graph would lie on a single line. This comes from Section 5.1 where it was shown that the disparity is a linear function of the image row number. This line would intersect the zero disparity line at the image row number corresponding to the horizon in the image. The least squares solution, using all disparity measurements, obtains a plane orientation which places the horizon line more than 50 pixels above its true location. This results in a majority of obstacles being incorrectly labeled as part of the ground plane.
4.6 Vision-based Automatic Road Vehicle Guidance Disparity vs Row Number
., ,
30
0 0
50
100
150 200 image row number
847
250
300
io
Fig. 11. Disparity versus row number for a crowded traffic scene. All points from the ground plane lie on a single line in this space. All objects below or above the ground plane contribute points which do not lie on this line. Here other vehicles contribute a majority of the disparity points. Estimating the orientation of the ground plane using all points results in a bias toward a plane higher than the true one.
Two different methods are used to delineate the two distributions. The first uses the predicted orientation of the plane obtained from the Kalman filter model of the dynamics of the vehicle. The predicted orientation of the plane, plus the uncertainty in that orientation, defines a search window in disparity space to look for candidate ground plane points. This search region corresponds to a narrow wedge in disparity space. Within this search region, a majority of points belong to the ground plane and the least squares estimate approximates the true value. A systematic bias in the estimate still exists when using only the above method. This is due to the fact that most objects are above the ground plane. To improve the estimate, the assumption that objects are connected to the ground plane is used. That is, objects are not floating above the ground plane. Under this assumption, the image of objects above the road plane will be connected to the ground plane image. The pixels in the image just below a detected obstacle are not used when updating the ground plane estimate. As an example, this method would not include the parts of a vehicle’s tires where they touch the ground plane. If these regions are not removed from consideration, the parts of the tires just above the road surface would be included in the update because of the uncertainty in the predicted ground plane orientation. This new technique combined with the limited search based on the Kalman filter prediction produces better results.
848 D. Koller et al.
A similar stereo vision based system developed by the Suburu car company uses only those image points corresponding to lane markers t o make its estimate of the ground surface [42]. To make this distinction, the system selects parts of the image containing white bars on a dark background and uses the disparity of those regions to estimate the ground plane orientation. This works as long as white lane markers exist and are visible. On highways in the United States the first assumption is not always valid. Lane markers are often hard to distinguish due to a lack of paint, dirty roads or the use of tactile markers instead of painted ones. Also, the markers can be occluded by other vehicles. Moreover, vehicles with coloration which resembles lane markings would confuse the system. To demonstrate the performance of the system, Fig. 12 shows a single frame of a sequence in which the test vehicle is passing a stalled vehicle on the shoulder of the road. Detecting this vehicle early is important for safe driving. The relative speed between vehicles is very high and the lateral distance between vehicles is small. This figure also shows the estimated positions of the stalled vehicle with respect t o the test vehicle (located at the origin in this plot). The ellipses centered at the position
'20 \ Relalivs speed vemus lrsme number
-l01 2 0-
100-
3 0-
no -
4 0-
-50 -
1 60-
-60
40 -
-80
-
-DO
-
20-
(I
-
-70 -
-10
-5
0
5
-100 10
5
10
15
20
25
30
Fig. 12. A single frame from a sequence in which the test vehicle passes a stalled vehicle. The first plot shows the estimated position (with uncertainty ellipses) of the vehicle over time. The test vehicle is at rest at the origin in this reference frame. The second plot shows the estimated relative velocity of the stalled vehicle, with its uncertainty envelope, as a function of time. The frame rate was 30 frames per second so these plots represent less than one second of time. The correct relative velocity was 55 mph.
4.6 Vision-based Automatic Road Vehicle Guidance 849 estimates are the uncertainty in position as given by the Kalman filter. Note the relative horizontal and vertical scales. The second plot shows the estimate of the relative velocity (negative in this case since the vehicle is at rest with respect to ours) and its uncertainty. The estimate improves rapidly with new measurements. This is necessary since the object is detected for only 28 frames which is less than one second at the 30 frames per second video rate.
back
view from front
front
Fig. 13. Model of the suspension system of the vehicle. The springs are critically damped with a natural frequency of 1.8 Hz. The cameras are also free to rotate about their viewing direction (roll). The roll dynamics are modeled by a critically damped torsion model.
. measurements -filter
C1.8-
zl
L
100.5' 0
I
50
100
150
200
250
I 300
frame number Fig. 14. The height and inclination angle of the cameras as tracked by the Kalman filter. The measurements obtained from the least squares solution are also shown.
850
D. Koller
et al.
5.5. Dynamic Ground Plane Estimation The measurements of the current ground plane orientation, (e,q5,h ) , are used to update a Kalman filter model of the dynamics of the vehicle. This simple model assumes that the cameras are mounted on top of two critically damped springs (Fig. 13). The stereo camera rig also contains a roll component. The roll is modeled by a second order critically damped (spring) system. The total state dimension is 6, consisting of the position and velocity of both springs, as well as the rotation and velocity of the roll term. By assuming small angles about the rest position, the measurement model is a linear transformation from spring positions to camera angles. Figure 14 shows the state estimate for the height and inclination angle of the cameras, along with the measurements provided from the least squares solution. The filter effectively smoothes the measurements while following the trend of the motion. Using a Kalman filter to estimate the plane orientation offers a number of advantages. The first is a temporal integration and smoothing of measurements. The filter also provides a prediction which is used to search the disparity measurements for the next plane update. Finally, by using a critically damped system, the system will gracefully return to its rest position given erroneous or missing measurement data. The rest position currently is based on calibration parameters. In reality, this rest position is a function of vehicle load which must be estimated each time it changes. Integration with accelerometers as well as driver controls would improve the model. 5.6. Temporal Integration Computing depth from just a pair of images is known to be sensitive to noise. One can improve the accuracy of the depth estimation by exploiting the temporal integration of information with time using the expected dynamics of the scene via Kalman filters. Objects of interest will be assumed to be either other vehicles on the road or stationary objects connected to the road plane. In addition we can exploit the physical constraints of the environment. We can assume we are interested in connected, rigid objects. This would allow us to use spatial coherence in identifying objects from the depth map. The spatial coherence of objects is used to segment the depth map into objects of interest. First, connected components are found in a 3-D space consisting of the two image dimensions plus the depth dimension. In the two image dimensions, points are connected if they are one of the four nearest neighbors. In the depth dimension they are connected if the difference in depth is less than the expected noise in the depth estimates. Figure 15 gives an example of two objects which are connected in this imageldepth 3-D space. These connected components form the basis of potential objects which are to be tracked with time. If the same object appears in two consecutive frames, a Kalman filter is initialized to track its position and velocity with respect to our vehicle. Figure 16 shows the objects found by this method.
4.6 Vision-based Automatic Road Vehicle Guidance 851 Image y
Image x
Fig. 15. Connected components in image/depth space consist of those pixels which are nearest neighbors in image coordinates as well as having depth differences less than depth uncertainty.
Fig. 16. Objects identified as being in the same lanes of traffic as the test vehicle. On the right side of the image is a “bird’s-eye-view” from above the road surface showing the relative position of the tracked objects with respect to the test vehicle.
References [I] 0. D. Altan, H. K. Patnaik and R. P. Roesser, Computer architecture and implementation of vision-based real-time lane sensing, in Proc. Intelligent Vehicles Symp. (1992) 202-206. [2] Y. Bar-Shalom and X.-R. Li, Estimation and Tracking: Principles, Techniques, and Software (Artech House, Boston, London, 1993). [3] R. Behringer and M. Maurer, Results on visual road recognition for road vehicle guidance, in Proc. Intelligent Vehicles Symp. (1996) 415-420. [4]M. Bertozzi and A. Broggi, Real time lane and obstacle detection on the gold system, in Proc. Intelligent Vehicles Symp. (1996) 213-218.
852
D. Koller et al.
[5] J.-L. Bruyelle and J.-G. Postaire, Disparity analysis for real time obstacle detection by linear stereovision, in Proc. Intelligent Vehicles Symp. (1992) 51-56. [6] S. Carlsson and J.-0. Eklundh, Object detection using model based prediction and motion parallax, in Proc. European Conf. Computer Vision (1990) 297-306. [7] S. Chandrashekar, A. Meygret and M. Thonnat, Temporal analysis of stereo image sequences of traffic scenes, in Proc. Vehicle Navigation and Infor. Sys. Conf. (1991) 203-2 12. [8] J. Crisman, Color Vision for the Detection of Unstructured Roads and Intersection, PhD Thesis, Carnegie-Mellon-University, 1990. (91 D. DeMenthon and L. Davis, Reconstruction of a road by local image matches and 3-D optimization, in Pmc. Int. Conf. Robotics and Automation (1990) 1337-1342. [lo] E. D. Dickmanns and N. Muller, Scene recognition and navigation capabilities for lane changes and turns in vision-based vehicle guidance, Control Engineering Practice 4, 5 (1996) 589-599. (111 E. D. Dickmanns and B. D. Mysliwetz, Recursive 3-D road and relative ego-state recognition, IEEE Trans. Pattern Anal. Mach. Intell. 14 (1992) 199-213. [12] W. Enkelmann, Obstacle detection by evaluation of optical flow fields from image sequences, in Proc. European Conf. Computer Vision (1990) 134-138. [13] 0. D. Faugeras, Three-dimensional Computer Vision: A Geometric Viewpoint (MIT Press, 1993). (141 A. Gelb (ed.), Applied Optimal Estimation (MIT Press, Cambridge, MA, 1974). [15] H. Goldstein, Classical Mechanics (Addison-Wesley Press, Reading, MA, 1980). [16] D. A. Gordon, Perceptual basis of vehicular guidance, Public Roads 34, 3 (1996) 53-68. [17] H. v. Helmholtz, Tkatise on physiological optics (Dover, NY, translated by j.p.c. southall edition, 1925). [18] B. P. K. Horn, Robot vision (MIT Press, 1986). (191 T. M. Jochem, D. A. Pomerleau and C. E. Thorpe, Maniac: A next generation neurally based autonomous road follower, in Image Understanding Workshop (1993) 473-479. [20] R. E. Kalman, A new approach to linear filtering and prediction problems, J . Basic Engin. (ASME) 82D (1960) 35-45. [21] K. Kanatani and K. Watanabe, Reconstruction of 3-D road geometry from images of autonomous land vechicles, IEEE Trans. Robotics and Automation 6, 1 (1990) 127-132. [22] S. K. Kenue, Lanelock: detection of lane boundaries and vehicle tracking using image processing techniques, Part 1, in SPIE Mobile robots I V , 1989. [23] S . K. Kenue, Lanelock: detection of lane boundaries and vehicle tracking using image processing techniques, Part 2, in SPIE Mobile robots I V , 1989. [24] K. I. Kim, S. Y. Oh, J. S. Lee, J. H. Han and C. N. Lee, An autonomous land vehicle: design concept and preliminary road test results, in Proc. Intelligent Vehicles Symp., 1993. [25] K. Kluge and C. Thorpe, Representation and recovery of road geometry in YARF, in Proc. Intelligent Vehicles Symp. Detroit, MI, 1992, 114-119. [26] D. Koller, Q.-T. Luong and J. Malik, Using binocular stereopsis for lane following and lane changing maneuvers, in Proc. Intelligent Vehicles Conf., Paris, France, 1994. [27] B. B. Litkouhi, A. Y. Lee and D. B. Craig, Estimator and controler for lanetrak, a vision-based automatic vehicle steering system, in Proc. IEEE Conf. Decision and Control (1993) 11-468-478.
4.6 Vision-based Automatic Road Vehicle Guidance 853 [28] Q.-T. Luong, J. Weber, D. Koller and J. Malik, An integrated stereo-based approach to automatic vehicle guidance, in Proc. Int. Conf. Computer Vision, Cambridge, MA, (1995) 52-57. [29] W. MacKeown, P. Greenway, B. Thomas and W. Wright, Road recognition with a neural network, in Proc. 1st IFAC Int. Workshop on Intelligent Autonomous Vehicles (1993) 151-156. [30] L. Matthies, Stereo vision for planetary rovers: Stochastic modeling to near real-time implementation, The Int. J. Computer Vision 8 (1992) 71-91. (311 M. Maurer, R. Behringer, S. Fherst, F. Thomanek and E. Dickmanns, A compact vision system for road vehicle guidance, in Proc. Int. Conf. Pattern Recogn. (1996) C7A.1. [32] P. S. Maybeck, Stochastic Models, Estimation and Control (Academic Press, London, 1979). [33] A. Meygret and M. Thonnat, Object detection in road scenes using stereo data, in Pro-Art Workshop on Vision, Sophia Antipolis, Apr. 19-20, 1990. (341 A. Meygret and M. Thonnat, Segmentation of optical flow and 3-D data for the interpretation of mobile objects, in Proc. Int. Conf. Computer Vision, Osaka, Japan, Dec. 4-7, 1990, 238-245. [35] A. Meygret, M. Thonnat and M. Berthod, A pyramidal stereovision algorithm based on contour chain points, in Proc. European Conf. Computer Vision, S . Margherita, Ligure, Italy, May 18-23, 1992, 83-88, G. Sandini (ed.), Lecture Notes in Computer Science 588 (Springer-Verlag, Berlin, Heidelberg, New York, 1992). [36] H. Moravec, Robot Rover Visual Navigation (UMI Research Press, 1981). [37] D. A. Pomerleau, Neural Network Perception for Mobile Robot Guidance (Kluwer Academic Publishing, 1994). [38] D. A. Pomerleau, Progress in neural network-based vision for autonomous robot driving, in Proc. Intelligent Vehicles Symp. (1992) 391-396. [39] D. Pommerleau, Ralph: Rapidly adapting lateral position handler, in Proc. Intelligent Vehicles Symp., 1995. (401 D. Raviv and M. Herman, A new approach to vision and control for road following, in Proc. Int. Conf. Computer Vision and Pattern Recogn. (1991) 217-225. [41] M. Rosenblum and L. S. Davis, The use of a radial basis function network for visual autonomous road following, in Proc. Intelligent Vehicles Symp., 1993. [42] K. Saneyoshi, Drive assist system using stereo image recognition, in Proc. Intelligent Vehicles Symp., Tokyo, 1996. [43] L. E. Scales, Introduction to Non-Linear Optimization (Macmillan, London, 1985). [44] M. Schwartzinger, T. Zielke, D. Noll, M. Brauckmann and W. v. Seelen, Visionbased car-following: Detection, tracking, and identification, in Proc. Intelligent Vehicles Symp. (1992) 24-29. [45] C. J. Taylor, J. Malik and J. Weber, A real-time approach to stereopsis and lanefinding, in Proc. Intelligent Vehicles Symp. (1996) 207-212. [46] C. Thorpe (ed.), Vision and Navigation: The Carnegie-Mellon Navlab (Kluwer Academic Publishers, Norwell, Mass, 1990). [47] M. A. Turk, D. G. Morgenthaler, K. D. Gremban and M. Marra, Vits - a vision system for autonomous land vehicle navigation, IEEE Trans. Pattern Anal. Mach. Intell. 10 (1988) 342-361. [48] B. Ullmer, Vita - an autonomous road vehicle (arv) for collision avoidance in traffic, in Proc. Intelligent Vehicles Symp. (1992) 3 6 4 1 . 149) B. Ulmer, Vita ii - active collision avoidance in real traffic, in Proc. Intelligent Vehicles Symp. (1994) 1-6.
854
D. Koller et al.
[50] P. Varaiya and S. E. Shladover, Sketch of an ivhs architecture, in Proc. V N I S '91 Conference, Paper No. 912772, Dearborn, MI, Oct. 1991, 325-341. [51] J. Weber, D. Koller, Q.-T. Luong and J. Malik, New results in stereo-based automatic vehicle guidance, in Proc. Intelligent Vehicles Conf., Detroit, MI, 1995, 530-535. (521 J. Zhang and H. H. Nagel, Texture based segmentation of road images, in Proc. Intelligent Vehicles Symp. (1994) 26G265. (531 Z. Zhang, R. Weiss and A. R. Hanson, Qualitative obstacle detection, in Proc. Int. Conf. Computer Vision and Pattern Recogn., Seattle, WA, June 19-23, 1994, 554-559. [54] Z. Zhang, R. Weiss and A. R. Hanson, Obstacle detection based on qualitative and quantitative 3d reconstruction, I E E E Trans. Pattern Anal. Mach. Intell. 19, 1 (1997) 15-26. [55] T. Zielke, M. Brauckmann and W. Von Seelen, Intensity and edge-based symmetry detection with an application to car-following, CVGIP 58, 2 (1993) 177-190.
PART 5 ARCHITECTURE AND TECHNOLOGY
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 857-867 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
ICHAPTER 5.1 I VISION ENGINEERING: DESIGNING COMPUTER VISION SYSTEMS
RAMA CHELLAPPA Department of Electrical Engineering, Center for Automation Research, University of Maryland, College Park, Maryland 20742, USA and AZRIEL ROSENFELD Center for Automation Research, University of Maryland College Park, Maryland 20742, USA The goal of computer vision is to derive descriptive information about a scene by computer analysis of images of the scene. Vision algorithms can serve as computational models for biological visual processes, and they also have many practical uses; but this paper treats computer vision as a subject in its own right. Vision problems are often ill-defined, ill-posed, or computationally intractable; nevertheless, successes have been achieved in many specific areas. We argue that by limiting the domain of application, carefully choosing the task, using redundant data (multi-sensor, multi-frame), and applying adequate computing power, useful solutions t o many vision problems can be obtained. Methods of designing such solutions are the subject of the emerging discipline of Vision Engineering. With projected advances in sensor and computing technologies, the domains of applicability and ranges of problems that can be solved will steadily expand. Keywords: Computer vision, vision engineering.
1. Introduction The general goal of computer vision is to c-rive information about a scene by computer analysis of images of that scene. Images can be obtained by various types of sensors; the most common kind are optical images obtained by a TV camera. An image is input t o a digital computer by sampling its brightness at a regularly spaced grid of points, resulting in a digital image array. The elements of the array are called pixels (short for “picture elements”), and their values are called gray levels. Given one or more digital images obtained from a scene, a computer vision system attempts t o (partially) describe the scene as consisting of surfaces or objects; this class of tasks will be discussed further in Section 2. Animals and humans have impressive abilities t o successfully interact with their environments - navigate over and around surfaces, recognize objects, etc. - using 857
858
R. Chellappa & A. Rosenfeld
vision. This performance constitutes a challenge to computer vision; at the same time, it serves as a n existence proof that the goals of computer vision are attainable. Conversely, the algorithms used by computer vision systems to derive information about a scene from images can be.regarded as possible computational models for the processes employed by biological visual systems. However, constructing such models is not the primary goal of computer vision; it is concerned only with the correctness of its scene description algorithms, not with whether they resemble biological visual processes. Computer vision techniques have many practical uses for analyzing images. Areas of application include document processing (e.g. character recognition), industrial inspection, medical image analysis, remote sensing, target recognition, and robot guidance. There have been successful applications in all of these areas, but many tasks are beyond current capabilities (e.g. reading unconstrained handwriting). These potential applications provide major incentives for continued research in computer vision. However, successful performance of specific tasks on the basis of image data is not the primary goal of computer vision; such performance is often possible even without obtaining a correct description of the scene. Viewed as a subject in its own right, the goal of computer vision is to derive correct (partial) descriptions of a scene, given one or more images of that scene. Computer vision can thus be regarded as the inverse of computer graphics, in which the goal is to generate (realistic) images of a scene, given a description of the scene. The computer vision goal is more difficult, since it involves the solution of inverse problems that are highly underconstrained ("ill-posed"). A more serious difficulty is that the problems may not even be well defined, because many classes of real-world scenes are not mathematically definable. Finally, even well-posed, well-defined vision problems may be computationally intractable. These sources of difficulty will be discussed in Section 3. In spite of these difficulties, vision systems have achieved successes in many domains. The chances of success are greatly increased by limiting the domain of application, simplifying the task to be performed, increasing the amount of image data used, and providing adequate computing power. These principles can be stated concisely as: Define your domain; pick your problem; improve your input; and take your time. They will be illustrated in Section 4. Following these principles in attempting t o solve vision problems provides a foundation for a discipline which we may call Vision Engineering, as discussed in Section 5. 2. Vision Tasks
If a scene could be completely arbitrary, not very much could be inferred about it by analyzing images. The gray levels of the pixels in an image measure the amounts of light received by the sensor from various directions. Any such set of brightness measurements could arise in infinitely many different ways as a result of
5.1 Vision Engineering
859
light emitted by a set of light sources, transmitted through a sequence of transparent media, and reflected from a sequence of surfaces. Computer vision becomes feasible only if restrictions are imposed on the class of possible scenes. The central problem of computer vision can thus be reformulated as follows: given a set of constraints on the allowable scenes, and given a set of images obtained from a scene that satisfies these constraints, derive a description of that scene. It should be pointed out that unless the given constraints are very strong, or the given set of images is large, the scene will not be uniquely determined; the images only provide further constraints on the subclass of scenes that could have given rise t o them, so that only partial descriptions of the scene are possible. Computer vision tasks vary widely in difficulty, depending on the nature of the constraints that are imposed on the class of allowable scenes and on the nature of the partial descriptions that are desired. The constraints can vary greatly in specificity. At one extreme, they may be of a general nature - for example, that the visible surfaces in the scene are all of some “simple” type (e.g. quadric surfaces with Lambertian reflectivities). [Constraints on the illumination should also be specified - for example, that it consists of a single distant light source. Note that the surfaces may be “simple” in a stochastic rather than a deterministic sense; for example, they may be fractal surfaces of given types, or they may be smooth surfaces (e.g. quadric) with spatially stationary variations in reflectivity (i.e. uniformly textured surfaces).] At the other extreme, the constraints may be quite specialized - for example, that the scene contains only objects having given geometric (“CAD”) descriptions and given optical surface characteristics. Similarly, the desired scene descriptions can vary greatly in completeness. “Recovery” tasks call for descriptions that are complete as possible, but “recognition” and “navigation” tasks usually require only partial descriptions - for example, identification and location of objects or surfaces of specific types if they are present in the scene. In its earliest years (beginning in the mid-l950s), computer vision research was concerned primarily with recognition tasks, and dealt almost entirely with single images of (essentially) two-dimensional scenes: documents, photomicrographs (which show thin “slices” of the subject, because the depth of field of a microscope image is very limited), or high-altitude views of the earth’s surface (which can be regarded as essentially flat when seen from sufficiently far away). The mid-1960’s saw the beginnings of research on robot vision; since a robot must deal with solid objects at close-by distances, the three-dimensional nature of the scene cannot be ignored. Research on recovery tasks began in the early 1970’s, initially considering only single images of a static scene, but by the mid-1970’s it was beginning to deal with time sequences of images (of a possibly time-varying scene) obtained by a moving sensor. By definition, recovery tasks require correct descriptions of the scene; but recognition and navigation tasks can often be performed successfully without
860
R. Chellappa & A . Rosenfeld
completely describing even the relevant parts of the scene. For example, obstacles can often be detected, or object types identified, without fully determining their geometries. Thirty-five years of research have produced theoretical solutions to many computer vision problems; but many of these solutions are based, explicitly or tacitly, on unrealistic assumptions about the class of allowable scenes, and as a result, they often perform unsatisfactorily when applied to real-world images. As we shall see in the next section, even for static, two-dimensional scenes, many vision problems are ill-posed, ill-defined, or computationally intractable.
3. Sources of Difficulty 3.1. Ill-Posedness
As already mentioned, the gray levels of the pixels in a n image represent the amounts of light received by the sensor from various directions. If the scene does not contain transparent objects (other than air, which wewill assume t o be clear), the light contributing t o a given pixel usually comes from a small surface patch in the scene (on the first surface intersected by a line drawn from the sensor in the given direction). This surface patch is illuminated by light sources, as well as by light reflected from other patches. Some fraction of this illumination is reflected toward the sensor and contributes to the pixel; in general, this fraction depends on the orientation of the surface patch relative to the direction(s) of illumination and the direction of the sensor, as well as on the reflectivity of the patch. In short, the gray level of a pixel is the resultant of the illumination, orientation, and reflectivity of a surface patch. If all these quantities are unknown, it is not possible to recover them from the image. Only under limited conditions of smoothly curved Lambertian surfaces with constant albedo can one recover estimates of illuminant direction, surface albedo and shape from a single image [l]. This example is a very simple illustration of the fact that most vision problems are “ill-posed”, i.e. underconstrained; they do not have unique solutions. Even scenes that satisfy constraints usually have more degrees of freedom than the images to which they give rise; thus even when we are given a set of images of a scene, the scene is usually not uniquely determined. In some special cases, with the availability of singular points, unique solutions may be obtained [2]. In applied mathematics, a common approach to solving ill-posed problems is to convert them into well-posed problems by imposing additional constraints [3]. A standard method of doing this, known as regularization, makes use of smoothness constraints; it finds the solution that minimizes some measure of nonsmoothness (usually defined by a combination of derivatives). Regularization methods were introduced into computer vision in the mid-19807s, and have been applied to many vision problems [4]. Evidently, however, solutions found by regularization often do not represent the actual scene [5]; for example, the actual scene may be piecewise smooth, but may also have discontinuities, and a regularized solution tends to
5.1 Vision Engineering 861 smooth over these discontinuities. To handle this problem, more general approaches have been proposed which allow discontinuities [6], but which minimize the complexity of these discontinuities - e.g. minimize the total length and total absolute curvature of the borders between smooth regions. In effect, these approaches [7] find solutions that have minimum-length descriptions (since the borders can be described by encoding them using chain codes). However, the actual scene is not necessarily the same as the scene (consistent with the images) that has the simplest description. Evidently, not all scenes of a given class are equally likely; but the likelihood of a scene depends on the physical processes that give rise to the class of scenes, not on the simplicity of its description, and certainly not on the simplicity of a description of its image. As an alternative to the regularization approach, direct methods have been suggested for shape recovery from radar [8] and visible images [9]. For illumination sources near the camera, good results have been obtained on simple optical images. Direct methods are rigid, in that they cannot be easily generalized to arbitrary illumination directions or to incorporate additional information. In addition, the lack of smoothing may present problems in the presence of noise.
3.2. Ill-Definedness It is often assumed in formulating vision problems that the class of allowable scenes is “piecewise simple”, e.g. that the visible surfaces are all smooth (e.g. planar or quadric) and Lambertian. This type of assumption seems at first glance to strongly constrain the class of possible scenes (and images), but in fact, the class of images is not constrained at all unless a lower bound is specified on the sizes of the “pieces”. If the pieces can be arbitrarily small, each pixel in an image can represent a different piece (or even parts of several pieces), so that the image can be completely arbitrary. For a two-dimensional scene, it suffices to specify a lower bound on the piece sizes; but for a three-dimensional scene, even this does not guarantee a lower bound on the sizes of the image regions that represent the pieces of surface; occlusions and nearly-grazing viewing angles can still give rise to arbitrarily small or arbitrarily thin regions in the image. Lower bounds on piece sizes are important for another very important reason: they make it easier to distinguish between the ideal scene and various types of “noise”. In the real world, piecewise simple scenes are an idealization; actual surfaces are not perfectly planar or quadric or perfectly Lambertian, but have fluctuating geometries or reflectivities. [Note that these fluctuations are in the scene itself; in addition, the brightness measurements made by the sensor are noisy, and the digitization process also introduces noise.] If the fluctuations are small relative to the piece sizes, it will usually be possible to avoid confusing them with “real” pieces. [Similarly,the noisy brightness measurements - assuming that they affect the pixels independently - yield pixel-size fluctuations, and digitization noise is also of at most pixel size; hence these types of noise too should usually not be
862
R . Chellappa & A . Rosenfeld
confused with the pieces.] Of course, even if we can avoid confusing noise fluctuations with real scene pieces, their presence can still interfere with correct estimation of the geometries and photometries of the pieces. Most analyses of vision problems (e.g. for piecewise simple ideal scenes) do not attempt to formulate realistic models for the “noise” in the scene; they usually assume that the noise in the image (which is the net result ofthe scene noise, the sensor noise, and the digitization noise) is Gaussian and affects each pixel independently. Examination of images of most types of real scenes shows that this is not a realistic assumption; thus the applicability of the resulting analyses to real-world images is questionable. The problem of ill-definedness becomes even more serious if one attempts to deal with scenes containing classes of objects that do not have simple mathematical definitions - for example, dogs, bushes, chairs, alphanumeric characters, etc. Recognition of such objects is not a well-defined computer vision task, even though humans can recognize them very reliably.
3.3. Intractability Even well-defined vision problems are not always easy to solve; in fact, they may be computationally intractable [10,11]. An image can be partitioned in combinatorially many ways into regions that could correspond to simple surfaces in the scene; finding the correct (i.e. the most likely) partition may thus involve combinatorial search. For example, even for scenes consisting of polyhedral objects, the problem of deciding whether a set of straight edges in an image could represent such a scene is NP-complete. Even identifying a subset of image features that represent a single object of a given type is exponential in the complexity of the object, if more than one object can be present in the scene, or if the features can be due to noise. Parallel processing (e.g. [12]) is widely used to speed up computer vision computations; it is also used very extensively and successfully in biological visual systems. Very efficient speedup can be achieved through parallelism in the early stages of the vision process, which involve simple operations on the image(s); but little is known about how to efficiently speed up the later, potentially combinatorial stages. Practical vision systems must operate in “real time” using limited computational resources; as a result, they are usually forced to use suboptimal techniques, so that there is no guarantee of correct performance. In principle, the computations performed by a vision system should be chosen to yield maximal expected gain of information about the scene at minimal expected computational cost. Unfortunately, even for well-defined vision tasks, it is not easy to estimate the expected gain and cost. Vision systems therefore usually perform standard types of computations that are not necessarily optimal for the given scene domain or vision task; this results in both inefficiency and poor performance.
5.1 Vision Engineering 863 4. Recipes for Success
4.1. Define Your Domain Well-defined vision problems should involve classes of scenes in which both the ideal scene and the noise can be mathematically (and probabilistically) characterized. For example, in scenes that contain only known types of man-made objects, the allowable geometric and optical characteristics of the visible surfaces can be known to any needed degree of accuracy. If the objects are LLclean”, and the characteristics of the sensor are known, the noise in the images can also be described very accurately. In such situations, the scene descriptions that are consistent with the images are generally less ambiguous (so that the problem of determining these descriptions is relatively well-posed) because of the relatively specialized nature of the class of allowable scenes. If, in addition, the number of objects that can be present is limited, the complexity of the scene description task and the computational cost of recognizing the objects are greatly reduced. For example, it has been shown [ll]that when all the features in the image can be assumed to arise from a single object, the expected search cost to recognize the object is quadratic in the number of features, and the number of possible interpretations drops rapidly to one as the number of features extracted from the image increases. The number of interpretations and the search cost are much higher when the scene is cluttered, so that the object of interest may be occluded and a significant part of the data may come from other objects in the scene. 4.2. Pick Your Problem
Even for specialized scene domains, deriving complete scene descriptions from images - the general recovery problem - can still be a very difficult task. However, there is no reason to insist on unique solutions to vision problems. The images (further) constrain the class of possible scenes; the task of the vision system is to determine these constraints. This yields a partial description of the scene, and for some purposes this description may be sufficient. In fact, in many situations only a partial description of the scene is needed, and such descriptions can often be derived inexpensively and reliably. A partial description may require only the detection of a specific type of object or surface, if it is present, or it may require only partial (“qualitative”) characterizations of the objects that are present (e.g. are their surfaces planar or curved). Two illustrations of the value of partial descriptions are: (i) An autonomous vehicle can rapidly and accurately follow the markers on a road; it need not analyze the entire road scene, but need only detect and track the marker edges [13,14]. By using additional domain-specific knowledge about the types of vehicles, their possible motions, etc., significant improvements in 3-D object and motion estimation have been reported in [15].
864
R. Chellappa €9 A . Rosenfeld
(ii) An active observer, by shifting its line of sight so that the focus of expansion due to its motion occupies a sequence of positions, can robustly detect independent motion anywhere in the region surrounded by these foci [16]. In this region, independent motion is indicated by the sign of the normal flow being opposite to that of the expansion. 4.3. Improve Your Inputs
Vision tasks that are very difficult to perform when given only a single image of the scene generally become much easier when additional images are available. These images could come from different sensors (e.g. we can use optical sensors that detect energy in different spectral bands; we can use imaging sensors of other types such as microwave or thermal infrared; or we can use range sensors that directly measure the distances to the visible surface points in the scene). Alternatively, we can use more than one sensor of the same type - for example, stereo vision systems use two or more cameras. Even if we use only a single sensor, we can adjust its parameters - for example, its position, orientation, focal length, etc. to obtain multiple images; control of sensor parameters in a vision system is known as active vision [17]. It has been shown that by using the active vision approach, ill-posed vision problems can become well-posed, and their solutions can be greatly simplified. These improvements are all at the sensor level; one can also consider improving the inputs to the higher levels of the vision process by extracting multiple types of features from the image data using different types of operators (e.g. several edge detectors). This strategy leads to a situation where “less is required from more”, i.e. where it is easier to derive the desired results if more input information is available, unlike the traditional situation where “more is required from less”. Animals and humans integrate different types of sensory data, and control their sensory apparatus, to obtain improved or additional information (e.g. tracking, fixation). Obtaining additional constraints on the scene by increasing the amount of image data is evidently a sounder strategy than making assumptions about the scene (smoothness, simplicity, etc.) that have no physical justification. Many successful computer vision systems have made effective use of redundant input data. In the following paragraphs we give three examples: (i) In [18], thermal ( 8 . 5 ~ - 1 2 . 5 ~and ) visual imagery are combined to identify objects or regions such as vehicles, buildings, areas of vegetation and roads. The visual image is used to estimate the surface orientation of the object. Using the surface orientation and other collateral information such as the ambient temperature, wind speed, and the date and time of image acquisition, an estimate of the thermal capacitance of the object is derived. This information, in conjunction with the surface reflectivity of the object (derived from the visual image) and the average object temperature (derived from the thermal image), is used in a rule-based system to identify the types of objects mentioned above.
5.1 Vision Engineering 865
(ii) Photometric stereo [19] is an excellent example of using more inputs t o resolve the inherent ambiguities in recovering shape from shading using a single image irradiance equation. In this scheme, the viewing direction is held constant, but multiple images are obtained by changing the direction of illumination. One then generates as many coupled irradiance equations as there are illumination directions. By solving these equations, robust estimates of the surface orientation can be obtained. Photometric stereo can be very useful in industrial applications where the incident illumination can be controlled. (iii) Stereo matching is the process of fusing two images taken from different viewpoints t o recover depth information in the scene. The process involves identifying corresponding points or regions in two views and using their relative displacements together with camera geometry to estimate their depths. If the baseline (the distance between the two cameras) is large, accurate depth estimates can be obtained, but at considerable added computational cost in the feature matching process. With a short baseline the cost of matching is less, but the depth resolution is low. In [20] a method is described that uses multiple stereo pairs with different baselines generated by lateral displacements of a camera. A practical system with seven cameras has been developed. This is a very good example in which, by using more inputs, the complexity of the algorithms is considerably reduced, while at the same time the results are improved. 4.4. Take Your Time
Since the early days of computer vision, the power of general purpose computational resources has improved by many orders of magnitude. This, combined with special purpose parallel hardware, both analog [21] and digital (VLSI), has greatly expanded the range of tractable vision tasks. The availability of increasingly powerful computing resources allows the vision system designer much greater freedom to adopt an attitude of “take your time” in vision algorithms, as well as freedom to use redundant input data. With no end in sight as regards expected improvements in computing power, the required time t o solve given vision problems will continue to decrease. Conversely, it will become possible to solve problems of increased complexity and problems that have wider domains of applicability.
5. Vision Engineering
Perception engineering has been defined by Jain [22] as the study of techniques common to different sensor-understanding applications, including techniques for sensing and for the interpretation of sensory data, and how t o integrate these techniques into different applications. He pointed out the existence of a serious communication gap between researchers and practitioners in the area of machine perception, and proposed establishing the field of perception engineering t o bridge this gap. However, he did not formulate any principles that could serve as guidelines
866
R. Chellappa & A. Rosenfeld
for the design of successful machine perception systems. We believe that the principles discussed in Section 4 can serve as foundations for an approach to computer vision that we shall refer to as Vision Engineering. The central task of vision engineering is to make vision problems tractable by applying the four principles: carefully characterizing the domain, choosing the tasks to be performed (breaking a given problem up into subtasks, if necessary), and providing adequate input data and adequate computational resources. We feel that these principles and their extensions will find increasing application in the design and construction of vision systems over the coming years. References [l] Q. Zheng and R. Chellappa, Estimation of illuminant direction,albedo and shape from shading, I E E E Trans. Pattern Anal. Mach. Intell. 13 (1991) 680-702. [2] J. Oliensis, Uniqueness in shape from shading, Int. J. Comput. Vision 6 (1991) 75-104. [3] A. N. Tikhonov and V. Y. Arsenin, Solution of Ill-Posed Problems (Winston, New York, 1977). [4] T. Poggio, V. Torre and C. Koch, Computational vision and regularization theory, Nature 317 (1985) 314-319. [5] J. Aloimonos and D. Shulman, Integration of Visual Modules: An Extension of the M a m Paradigm (Academic Press, Boston, MA, 1989). [6] D. Terzopoulos, Regularization of inverse visual problems involving discontinuities, I E E E Trans. Pattern Anal. Mach. Intell. 8 (1986) 413-426. [7] Y. C. Leclerc, Constructing simple stable descriptions for image partitioning, Int. J. Comput. Vision 3 (1989) 73-102. [8] R. L. Wildey, Topography from a single radar image, Science 224 (1984) 153-156. [9] J. Oliensis, Direct method for reconstructing shape from shading, in Proc. D A R P A Image Understanding Workshop, San Diego, CA, Jan. 1992, 563-571. [lo] L. M. Kirousis and C. H. Papadimitriou, The complexity of recognizing polyhedral scenes,J. Comput. Syst. Sci. 37 (1988) 14-38. [ll] W. E. L. Grimson, Object Recognition by Computer (MIT Press, Cambridge, MA, 1990) Chapter 10. [12] V. K. Prasanna Kumar, Parallel Architectures and Algorithms f o r Image Understanding (Academic Press, New York, 1991). [13] E. D. Dickmanns and V. Graefe, Dynamic monocular machine vision, Mach. Vision Appl. 1 (1988) 223-240. [14] E . D. Dickmanns and V. Graefe, Applications of dynamic monocular machine vision, Mach. Vision Appl. 1 (1988) 241-261. [15] J. Schick and E. D. Dickmanns, Simultaneous estimation of 3D shape and motion of objects by computer vision, I E E E Workshop o n Visual Motion, Princeton, NJ, Oct. 1991, 256-261. [16] R. Sharma and J. Aloimonos, Robust detection of independent motion: An active and purposive solution, Center for Automation Research Technical Report CAR-TR-534, University of Maryland, College Park, 1991. [17] J. Aloimonos, I. Weiss and A. Bandophadhay, Active vision, Int. J. Comput. Vision 1 (1987) 333-356. [18] N. Nandhakumar and J. K. Aggarwal, Integrated analysis of thermal and visual images for scene interpretation, I E E E Trans. Pattern Anal. Mach. Intell. 10 (1988) 469-481. [19] R. J. Woodham, Photometric method for determining surface orientation from mul-
5.1 Vision Engineering 867 tiple images, in B. K. P. Horn and M. J. Brooks (eds.), Shape from Shading (MIT Press, Cambridge, MA, 1989). [20] M. Okutomi and T. Kanade, A multiple-baseline stereo, in Proc. IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, Miami, FL, June 1991, 63-69. [21] C. Mead, Analog VLSI and Neural Systems (Addison-Wesley, Reading, MA, 1989). [22] R. Jain, Perception engineering, Mach. Vzsion Appl. 1 (1988) 73-74.
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 869-890 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
I CHAPTER 5.2 I OPTICAL PATTERN RECOGNITION FOR COMPUTER VISION
DAVID CASASENT Department of Electrical and Computer Engineering Center for Excellence in Optical Data Processing Carnegie Mellon University, Pittsburgh, Pennsylvania 15213, USA Optical processors offer many useful operations for computer vision. The maturity of these systems and the repertoire of operations they can perform is increasing rapidly. Hence a brief updated overview of this area merits attention. Many of the new algorithms employed can also be realized in digital and analog VLSI technology and hence computer vision researchers should benefit from this review. We consider optical morphological, feature extraction, correlation and neural network systems for different levels of computer vision with image processing examples and hardware fabrication work in each area included. Keywords: Classifier neural net, correlator, distortion-invariant filters, feature extractor, Hough transform, Hit-or-Miss transform, morphological processor.
1. Introduction
A book could be written on each aspect of optical pattern recognition. Thus, only the highlights of selected optical processing operations can be noted here. The reader is referred to the references provided, several texts [l-31, recent conference volumes [4,5], journal special issues [6] and review articles [7,8] for more details on each topic. To unify and best summarize this field, we consider (in separate sections) optical systems for low, medium, high and very high-level computer vision operations. Although the boundaries between these different levels are not rigid, we distinguish low-level vision by noise and image enhancement operations, mediumlevel vision by feature extractors, high-level vision by correlators and very high-level vision by neural net operations. As we shall show, optical processing has a role in each area. We will mainly emphasize recent work at Carnegie Mellon University in these different areas. Section 2 discusses the major optical processing architectures we consider (feature extractors, correlators and neural nets) and presents one possible unified hierarchical approach to the use of all techniques for scene analysis. Section 3 details and provides examples of optical morphological processors for low-level vision and for detection. Section 4 considers the role for optical processing in medium-level vision with attention to feature extractors for product inspection and for subsequent analysis of regions of interest (ROIs). Section 5 details a variety of advanced 869
870
D. Casasent
distortion-invariant optical correlation filters for several applications in high-level computer vision. We then consider in Section 6 very high-level vision operations with attention to optical neural nets for object identification and brief remarks on their use as production systems. 2. Operations Achievable
Many optical processing architectures exist that are of use in computer vision. The ability to compute the Fourier transform (FT) at Pa of 2-D input data at P1 with a simple lens (Fig. (la)) is probably the most widely used concept in optical processing. To simplify analysis, the IFTI2 is often sampled with a detector with wedge and ring shaped detector elements [9] as in Fig. l(b). This is a very attractive feature space for analysis of an input object since the magnitude FT is shift invariant, the wedge samples are scale invariant and the ring samples are rotation invariant. The use of 32 wedge and 32 ring detector elements also greatly simplifies analysis by dimensionality reduction.
-
I
L
Fig. 1. Optical Fourier transform system (a) and wedge ring detector (b) [23].
Many other operations and feature space descriptions of an input object are possible and can be implemented optically [8]. These include: moments, chord distributions, polar-log FT spaces, and the Hough transform. From this we see that optical processors can implement a wide variety of image processing functions beyond the classic FT. Figure 2 shows an optical system that computes the Hough transform at TV frame rates. The input object at P1 is imaged onto a computer generated hologram (CGH) at P2 which forms the Hough transform at P3 in parallel. The CGH consists of a set of N cylindrical lenses at different angles. The Hough transform input denotes the position, orientation and length of all lines in the input. Extensions to other curved shapes are possible and have been optically demonstrated. All of the aforementioned feature spaces can be optically produced using CGHs [8]. The optical correlator is also one of the most used optical processors. Figure 3 shows the schematic of a space and frequency multiplexed optical correlator. The input is placed at PI, a set of spatially-multiplexed filters is shown at P2.
5.2 Optical Pattern Recognition for Computer Vision 871
Fig. 2. Optical Hough transform system using a computer generated hologram.
L,
Multiple Laser Diodes
1
P,
" Input Data
LI
P9
F i l t e r Bank and HOES
SELECT FILTER AND OPERATION
M u l t i p l e or Single Output Correlation Plapes
I
Fig. 3. Space and frequency-multiplexedoptical correlator architecture [35].
Different laser diodes activated at Po allow different P2 filters to be accessed. At each spatial P2 location several (e.g. four) frequency-multiplexed filters are placed. When one Po laser diode is activated it selects a set of filters and the P3 output is the correlation of the PI input and a set (e.g. four) of P2 filters with the four correlations appearing in parallel in the four quadrants of P3. With access to a large filter bank at Pz, many operations are possible (with a real-time device at P2, adaptive filters are possible). Optical correlators have two major advantages in scene analysis: they can detect multiple objects in parallel (and are essential for parallel analysis of scenes containing multiple objects) with correlation peaks occurring at the locations of each object in the field of view and they are the optimum detector systems when noise is present. The optical correlator is also quite versatile. With CGH filters at P2, the P3 output can be any of the feature spaces noted. With large banks of filters possible, one can use different filters and achieve detection, recognition and identification. When the Pz filter used is a structuring element and when the P3 output is properly thresholded, the P3 output can be any morphological operation [lo].
872
D. Casasent
As we have just noted (to be detailed in subsequent sections), the optical correlator is a most versatile and multifunctional optical image processing architecture. This is of major importance since optical correlators are rapidly reaching a significant level of maturity. As one example, we consider the solid optics correlator fabricated by Teledyne Brown Engineering [Ill. The system uses modular optical elements for laser diode collimation, Fourier transform, imaging and beam splitter, etc. components. These are assembled into a rugged optical correlator of small size as shown in Figs. 4 and 5. In Fig. 4,the output from the laser diode light source on the left is collimated and passes through the input spatial light modulator (SLM) and its FT is formed at the right end where a reflective filter is placed. The light reflected from the filter is Fourier transformed and reflected onto the output detector via the beam splitter (BS) to produce the output correlation. Figure 5 shows the actual optical correlator system with a magneto-optic MO SLM input and a dichromated gelatin (DCG) filter. This is typical of the high state of maturity that this key optical architecture has reached [12]. With the advanced filters we describe, this system will be most suitable for image processing.
f- DETECTOR ARRAY
LASER INPUT APERTURE
I
I I
LASER COLLIMATOR
I
FOCUSING LENS
II
I
I
I
I1
f \
TELEPHOTO FOCUSING FILTER RELAY SOLID BLOCK, LENS FOURIER LENS SECOND TRANSFORM BEAMS PL ITTER PLANE OPTION I I
FOURIER TRANSFORM LENS
Fig. 4. Schematic diagram of the solid optics correlator [ll].
The final basic optical processor architecture we consider is the optical matrixvector multiplier [13] of Fig. 6. The 1-D P1 input vector g can be realized by a linear LED or laser diode array or by a 1-D SLM. The light leaving PI is expanded in 1-D to uniformly illuminate the columns of a 2-D matrix mask or SLM at Pz with transmittance M (a matrix). The light leaving Pz is then integrated horizontally to produce the output vector y = & that is the matrix-vector product. We consider the use of this system as t h e basic building block for an optical neural net for object identification and as an artificial intelligence production system (Section 6). As a multilayer neural net, the P1 outputs are the input neurons, the matrix M is a set of weights and the P3 outputs are the hidden layer neurons. A cascade
5.2 Optical Pattern Recognition for Computer Vision 873
Fig. 5 . Photograph of the solid optics correlator showing its major elements [ll].
Fig. 6. Optical matrix-vector neural net processor.
of two such systems yields the standard multilayer neural net of Fig. 7 (one matrixvector system is an associative processor) and with P3 to PI feedback it is a production system (as we discuss in Section 6). The optical matrix-vector element has also achieved a high degree of maturity as seen in the schematic of Fig. 8 which shows this system component fabricated in integrated optics [14]. The basic optical image processing architectures can be viewed as low, medium, high and very high level computer vision modules. They can be used in many ways for computer vision. The approach we find to be the most useful is shown in Fig. 9. We consider the general scene analysis problem when multiple objects are present in high clutter. We separate the scene analysis problem into a hierarchy of detection, recognition and identification steps. For detection, we employ morphological correlator processors (Section 3). For recognition, we use distortion-invariant correlation filters (Section 5 ). For identification, we use feature extractors (Section 4) applied to the regions of interest (ROIs) obtained from detection and a neural net (Section 6) to analyze the feature space and provide the final object identification.
874
D. Casasent
INPUT
CLUSTERS SEVERAL PER CLASS
CLASS
Fig. 7. Three-layer nonlinear neural net classifier used [23].
tmnsporent insubtor
n -electrode of detectcx army
GoAs l h - Show detector army
, 23 .... .
... .
31 32
Fig. 8. Integrated optical matrix-vector processor schematic [14].
INPUT SCENE
MORPHOLOGICAL PROCESSOR
DISTORT ION INVARIANT CORRELATOR
DETECT ION
RECOGNITION
+
FEATURE EXTRACTOR
NEURAL NET
IDENTIFICATION
Fig. 9. One optical realization of three levels of scene analysis.
3. Low-Level Optical Morphological Processors The two basic morphological operations are dilation (region growing) and erosion (region shrinking). We achieve both on a correlator using a filter that is a structuring element (typically a disc), whose size determines the size of a hole or inlet to be
5.2 Optical Pattern Recognition for Computer Vision 875
filled in or the size of a noise blob or protrusion to be removed. If the correlation output is thresholded low (high) dilation (erosion) results [15]. Thus, including structuring element filters at Pz and an output threshold at P3 allows the system of Fig. 3 to also implement morphological operations. These are local operators. Since filling in holes on a white object (dilation) and removing noise and regions (erosion) distorts the boufidary of the object, these operations are generally used in pairs. A dilation followed by an erosion is a closure and an erosion followed by dilation is an opening. Figure 10 shows examples of these operations. The noisy input with holes (the treads of the tank) on the object is shown in Fig. lO(a). The opening of it is shown in Fig. 10(b) (the erosion removes noisy background smaller than the size of the structuring element used and the dilation restores the boundary). The closure of Fig. 10(b) is shown in Fig. 1O(c) (it fills in holes on the object). Edge enhancement (Fig. 10(d)) is also easily achieved by the difference between a dilation and an erosion and appears to be preferable to conventional edge-enhancement methods [lS]. Other operations such as removal of a nonuniform background [IS] etc. are also possible.
Fig. 10. Optical morphological image enhancement [35].(a) input, (b) opening of (a), (c) closure of (b), (d) edge-enhanced (b).
We find these standard morphological operations to be most useful to improve the image of an object after detection. To achieve detection, we use a modified [17] hit-or-miss (HOM) [18] morphological transform. In the basic HOM algorithm, the input image is thresholded, correlated with a hit structuring element, and thresholded; the complement of the thresholded input is then correlated with a miss
876
D. Casasent
Fig. 11. Optical morphological HOM detection example. (a) Input scene, (b) HI hit structuring element, (c) M I miss structuring element, (d) output data.
structuring element (typically the complement of the hit element with a white border or background present) and thresholded; the intersection of the two correlations is the HOM result. Figure 11 shows an example of our new algorithm. Figure ll( a ) shows a scene with hot (bright) and cold (dark) objects present. We threshold the image above the mean and perform a n HOM correlation with the structuring element (not to scale with Fig. ll(a)) in Fig. l l ( b ) . We then threshold the image below the mean and perform an MOH (miss or hit) correlation with the structuring element in Fig. 11(c). The union of the two correlations detects all objects (Fig. l l ( d ) ) . The hit filter (Fig. l l ( b ) ) has a white region equal t o the smallest object and the central dark part of the miss filter (Fig. l l ( c ) ) is the size of the
5.2 Optical Pattern Recognition for Computer Vision 877
largest object (the size of the white border region in Fig. l l ( c ) depends upon the background expected). The HOM correlation detects hot objects and the MOH correlation detects cold objects. The hit correlation detects all objects larger than the smallest object, the miss correlation detects all objects less than the largest object and their union detects only objects within the desired range of sizes. We find this morphological function to be most attractive for the first (detection) phase of scene analysis in Fig. 9. When necessary, we use conventional image enhancement morphological operations prior t o the last feature extraction step in Fig. 9. 4. Medium-Level Computer Vision (Feature Extraction) Once regions of interest (ROIs) have been extracted (detection) from a scene, one must learn more about the contents of each such ROI. One technique that is very general (since it extends t o a large number of multiple classes) is to calculate features associated with each ROI. These features include those noted in Section 2 and others. They are a reduced dimensionality description of each ROI and hence are easier to analyze (from a computation standpoint). They are generally also a n in-plane distortion-invariant feature space. They almost always have shift invariance (this is essential since the location of the object in the ROI is not known) and this greatly simplifies training. In conventional pattern recognition, these features (as a feature vector) are input to a linear classifier (consisting of one or a number of linear discriminant functions, LDFs). As an example of the power of a feature space processor, we consider the recognition of multiple classes of objects (two aircraft: an F4 and F104) with about 128 x 128 pixel resolution with four degree-of-freedom distortions (roll, pitch, and z and y translations). We considered f60" distortions in both pitch and roll at 2.5" increments (for each roll angle, all f 6 0 " pitch variations are considered). We trained a modified linear Ho-Kashyap classifier [19,20] on distortions every 5" in roll and pitch (625 distorted images per class). For each image, the 32-element wedge FT feature space was calculated and fed to the classifier algorithm. We then tested the classifier on 1152 distorted test images not present in the training set with cn = 0.1 of white Gaussian noise also present and obtained a very respectable 91.2% correct recognition. This demonstrates the ability of feature extractors to provide object discrimination in the face of very severe object distortions. For our present discussion, their major use is their potential to handle many classes of objects. The wedge ring detector sampled FT is the most widely used optical feature space with many product inspection applications and with a well engineered system having been fabricated [21]. Here we describe a product inspection application of the Hough transform in which the specific locations and orientations of portions of a product to be inspected are of concern [22]. Figure 12 shows the product, a package of cigarettes. The specific issues of concern are that: (1) the package wrapper be aligned within 1.8", (2) the closure seal (A) at the top be present, aligned within 3.2", and that the bottom of it extends properly within 0.5 mm,
878 D. Casasent
D
C
Fig. 12. Cigarette package to be inspected [22].
and (3) that the tear strip (B) be present, parallel to the top within tolerances, and be properly positioned within 0.5 mm. To achieve these inspection tasks, we form the Hough transform of each package as it is assembled. We form four slices of the Hough transform at 0 = 38", 142", 0" and 90". The 38" and 142" angular slices denote the presence of and proper location of the two angular lines (C and D) and hence determine if the package is properly aligned. The 90" Hough transform slice has peaks corresponding to horizontal lines in the object (from top to bottom of the image, peaks occur due to the top of the package, the tear strip and the bottom of the closure seal). These indicate the presence of the tear strip and the seal and if they are at the proper location from the top of the package within tolerances. If either is at an angle, the corresponding Hough transform peak on the 90" slice becomes broader and its height decreases. The 0" slide of the Hough transform denotes vertical lines, specifically the two edges of the seal. If the seal is perfectly aligned, both Hough transform peaks will be of the same height and in the proper position horizontally on the package. If the seal is not aligned properly, the Hough transform peaks will be different in height. Figure 13 shows the Hough transform of a cigarette package with six regions along the four Hough transform slices noted with the portions of the product to which they correspond indicated. For each product, we thus investigate the six indicated Hough transform regions for a Hough transform peak and the value of each peak. The laboratory real-time Hough transform system assembled operated at 30 products per second and exhibited over 99% correct inspection. A-om errors in the Hough transform peak positions or heights, the nature of each product defect can be determined.
5.2 Optical Pattern Recognition
for Computer Vision
879
Fig. 13. Real time optical laboratory Hough transform of Fig. 12 [22].
It is important that only one object be present in the field of view and that noise be reduced when feature extractors are employed. The detection ROI location system achieves the one object requirement for scene analysis and LED or laser diode sensors achieve this for product inspection applications. Morphological processing techniques can be employed to reduce noise and improve the image if needed. Figure 9 allows for such operations prior to feature extraction and object identification. 5. High-Level Computer Vision (Correlators) For this level of computer vision we consider advanced distortion-invariant filters used in correlators. Such correlation filters use internal object structure or the boundary shape of the object rather than simple rectangular filters as in the morphological HOM detection filters in Section 3. A wide variety of such filters exist and are generally extensions of the synthetic discriminant function (SDF) filters [24]. These SDF filters used a training set of different distorted images. The vector inner product matrix of the training set was used with a control vector that specified the correlation peak value to calculate the filter function. The filter is a linear combination of the training set of images. 5.1. Filter Synthesis
The synthetic discriminant function filters control only one or several points in the correlation plane and hence have limited storage capacity (number of training images N T ) before large sidelobes occur that cause false alarms. This filter clutter is due to [25] the reduced SNR that occurs for large NT. The minimum average correlation energy (MACE) filter was the next significant development since its intent is to reduce correlation plane sidelobes. It achieves this by minimizing the
880 D. Casasent
correlation plane energy [26]
E=E+gE, where is the vector version of the FT (Fourier transform) of the desired filter function and is a diagonal matrix with elements equal to the sum of IFTI2 of the training images. We minimize (5.1) subject to a constraint on the correlation peak value for all training images
H +=X- &-,
(5.2)
where + denotes the conjugate transpose, the columns of the matrix & are the FTs X iof the training set images and the elements of the control vector u are the correlation peak values specified for each training image (the elements of u are typically chosen to be one). The solution to (5.1) subject to the constraint in (5.2)is found by Lagrange multiplier techniques to be [26] (5.3) The MACE filter solution yields a sharp correlation peak, which localizes the target’s position well. However, such sharp correlation peaks result because the spectrum of the filter has been whitened emphasizing high frequencies. As a result, this filter has poor recognition of non-training set intra-class images and it is sensitive to noise. To overcome these problems, we recently introduced the minimum noise and average correlation energy (MINACE) filter [27]. This uses a better bound on the spectral envelope of the images. And it also inherently uses a specified noise power spectrum in synthesis. For a filter with one training image i, the filter solution is
H= -
g;
1 =(= x X +=T a -lX =) -Iu - 9
(5.4)
and for NT training images, the filter solution is
Its form is the same as in (5.3), however the preprocessing function diagonal matrix with diagonal elements
Ti(u, v) = max[Di(u, v),N(u, v)] and
g iis now a (5.6)
is a diagonal matrix with diagonal elements
T ( u , v )= max[TI(u,v),TZ(U,21),...,TNT(u,v)].
g.
(5.7)
The key step is the choice of the preprocessing function Its elements are chosen separately for each spatial frequency u and v based on the magnitude of the spatial frequencies of the signal D and the noise N. Specifically, if the signal is above the noise at some spatial frequency, we select the signal; otherwise, we use the selected noise level N. This comparison is done separately for each spatial frequency and for
5.2 Optical Pattern Recognition for Computer Vision 881
all training images. This reduces the filter's response at high frequencies and other frequencies (where noise dominates) and hence improves intra-class recognition and performance in noise. This filter has another major advantage of use in our present problem: we can control the filter's recognition and discrimination performance. This is achieved by varying the amount of noise N (through its variance u 2 ) used in filter synthesis. We define the control parameter c = u2/DC
(5.8)
to be the ratio of the noise energy (for white Gaussian noise) to the DC value of the signal energy. Large values of c emphasize lower spatial frequencies and provide filters that are good for detection (intra-class recognition and noise performance). Low c values emphasize higher spatial frequencies and such filters are good for identification. Medium c values prove to be useful for recognition. This provides the MINACE filter with a flexibility not found in other correlation filters which are quite rigid. Specifically, by varying the training set and the control parameter c, the same filter synthesis algorithm yields filters suitable for the three different levels in scene analysis (detection, recognition, and identification). We now provide initial examples of such results to demonstrate the concepts. 5.2. Test Results
Figure 14 provides an example of this multifunctional filter synthesis algorithm. Figure 14(a) shows the input scene. It contains 13 objects as noted in Fig. 14(b) (the values in parentheses indicate the orientation of each object). We formed a MINACE filter trained only on 36 orientations of the ZSU object at 10" intervals with a large c = 0.1 value. This object was chosen since it is the smallest one and we initially desire to only implement detection. Figure 14(c) shows the detection correlation results. We find peaks for all 13 objects and only one false alarm (lower left). Thus, this filter can achieve detection of all interesting ROIs independent of their orientation and in considerable noise. To achieve recognition of only the larger and more dangerous objects (the SCUD and FROG missile launchers), we trained a MINACE filter on only SCUDs and FROGS and used a lower c = 0.05 value. The results (Fig. 14(d)) show correlation peaks at the locations of these five mobile missile launchers. This filter thus achieves recognition of a subclass of objects (large missile launchers) independent of their orientation and in noise. To achieve identification of only the SCUD objects, we form another MINACE filter trained only on SCUDs and with a smaller c = 0.001 value (a lower c provides more discrimination). The results (Fig. 14(e)) locate the three SCUD objects and demonstrate identification. Correlators are well-known to be ideal for detection in noise and when multiple objects are present. This example demonstrates how the same basic filter synthesis algorithm can achieve detection, recognition and identification and hence can solve quite complex scene analysis problems.
882
D. Casasent
(4
(4
(el
Fig. 14. Advanced MINACE distortion-invariant hierarchical correlation filter results. (a) Input, (b) input, (c) detection, (d) recognition (SCUDS/FROGS), (e) identification (SCUDS).
Another noteworthy example of filter performance is now briefly noted. For identification or discrimination between two very similar objects, correlation techniques using all object pixels rather than reduced dimensionality feature space methods are preferable. Figure 15 shows images of the SA-13 and ZSU-23 objects with about 32x12 pixel resolution. As seen, they are quite similar. To identify the SA-13 and discriminate it from the ZSU-23 object when 36 different rotated versions of each object are considered, we used a MINACE filter with c = 0.001 and NT = 19 training images of the SA-13 at 19 of the 36 distorted angles. This filter successfully recognizes all true class SA-13 objects and yields no correlation peaks above 0.5 for any of the 36 false class ZSU-23 objects. Three other properties of the MINACE filter emerge from this example. As we increase c, we can reduce the size NT of the required training set (e.g. NT = 19 not 36 here). Use of a flat spectrum for our MINACE noise model thus also effectively models object distortions (e.g. controlling the spatial frequencies used to recognize an object in noise is similar to controlling
5.2 Optical Pattern Recognition for Computer Vasion 883
Fig. 15. Two similar objects for identification and discrimination (a) SA-13, (b) ZSU-23.
the spatial frequencies to achieve intraclass recognition, as this example has shown). Finally, no false class training images were used (i.e. we could have, but did not, train the filter to produce a zero or low output correlation value for troublesome false class images). This is attractive since one does not generally know every false class object that is possible. In multiple correlation stages of the identification portion of scene analysis, this may be allowable (and necessary) in some cases. 6. Very High-Level Computer Vision (Neural Nets) Many potential applications for neural nets in computer vision have been advanced [28]. These include image enhancement and feature extraction. We find other techniques (Sections 3 and 4) to be preferable and sufficient for these operations. The major reason is the large number of neurons and interconnections required when the neural net input is an iconic (pixel-based) image representation. For example, one can achieve shift invariance in a neural net with N input neurons by the use of N4 interconnections [29]. However, when N = 512’, this is very excessive and since the same property can be achieved with the FT etc., we find such methods to be preferable. When multiple objects are present in the field of view, no neural net can handle all objects in parallel. Conversely, a correlator (Section 5) easily achieves this. A correlator is in fact a most powerful neural net with the filter function being the set of weights applied to input iconic neurons (an image) with the unique property that the weights are applied in parallel to every region of the input scene. Thus, for such cases, we find a correlator using advanced distortion-invariant filters to be preferable. In general, we find the use of such FT-based free-space optical interconnections to be preferable to other neural net approaches which achieve shift invariance with much more hardware with many hard-wired forced interconnections required (and are not easily achieved without optical processing techniques).
884 D. Casasent
6.1. Neural Net Classifier Algorithm In our opinion, one of the major uses of neural nets is their ability to provide an algorithm for determining nonlinear piecewise discriminant surfaces for classifiers. We now highlight our adaptive clustering neural net [30] and how it uses linear discriminant functions (linear classifiers) and neural net techniques to achieve a nonlinear classifier. As the input neuron representation space we use feature space neurons (Section 4) obtained for ROIs from a morphological detection processor. The classic three-layer neural net we use is shown in Fig. 7. The input P1 neurons are a feature space (we use wedge IFT samples in our example). The output P5 neurons indicate the class of the input object. To determine the number of hidden layer neurons, we use standard clustering techniques [31] to select prototypes or exemplars for each class from the full training set. The number of prototypes decided upon is the number of hidden layer neurons N3. We typically use three prototypes per class. We assign each of these to a hidden layer neuron (i.e. we use 3C hidden layer neurons, where C is the number of object classes). Each prototype and hence each P3 hidden layer neuron corresponds to a feature vector or a point in the multidimensional feature space. As the initial weights from P1 to each P3 neuron (e.g. P3 neuron i), we use the feature vector p-a . for prototype i. This results in a set of Pl-to-P3 weights that are classic linear discriminant functions as used in standard pattern recognition. These are only the initial weights. They are then adapted into nonlinear discriminant functions by our neural net algorithm. To achieve this, we add an additional input neuron whose value is minus 0.5 times the sum of the squares of the other weights. Thus, with NF features, we use N1 = NF 1 input P1 neurons with the weights wij (from P1 input neuron j to P3 neuron i) described by
+
where pij is element j of the prototype vector p--2 . . This insures that the hidden layer neuron closest to the input vector at PI will be the most active one. We use a winner-take-all selection of the most active P3 neuron during classification. We now use neural net training to adapt these initial Pl-to-P3 weights (the neural net thus forms weights that are combinations of linear discriminant functions and hence it produces piecewise nonlinear discriminant surfaces as we shall show). To adapt the weights, we present each of the training set of image feature spaces to the neural net and we determine the P3 neuron values for each input g (for P3 neuron i this is simply the vector inner product g T ~ofi the training set input vector g and the weight vector zui from all PI neurons to P3 neuron i). We then calculate the most active P3 neuron i(c) in class c of the input vector and the most active neuron i(Z) in any other class. We denote their weight vectors by wi(.)and For each
5.2 Optical Pattern Recognition for Computer Vision 885
input, we then determine an error E for a perceptron error function as
where S = 0.05 in our case. After each presentation of the training set, we calculate the derivative a E / a a i and use it to adapt the weights using a conjugate gradient algorithm [32]. We then present the training set again, calculate E in (6.2) and continue to adapt the weights until convergence or negligible change occurs. 6.2. Neural Net Classifier Results
To best demonstrate the power of a neural net to produce piecewise nonlinear decision surfaces from linear discriminant functions, we consider the artificial problem in Fig. 16 with 383 samples in three classes (181 in class 1 on the left and bottom represented by a triangle; in class 2 in the center represented by a circle; and 105 in class 3 in the upper right represented by a diamond). We chose this 3-class example because it uses only two features and the results can thus be displayed. We used our ACNN algorithm to solve this problem using Nl = 3 input PI neurons (the two features plus one bias neuron), N3 = 2C = 6 hidden layer neurons and N5 = C = 3 output neurons, the number of data classes. Figure 16 shows the piecewise nonlinear decision boundaries produced (they consist of six straight lines, modified combinations of the six initial linear discriminant functions associated with the six hidden layer neurons). The results obtained gave Pc = 97% correct recognition after only 80 iterations during training (in classification our ACNN algorithm is a one-pass non-iterative algorithm). We compared this neural net performance to that of the standard but very computationally expensive multivariate Gaussian
0 class 2
+ class 3 boundaries I
.c
FEATURE 1 Fig. 16. Discrimination problem showing nonlinear decision surfaces automatically produced by our neural net algorithm (231.
886
D. Casasent
classifier which achieved only PC = 89.5% correct recognition. Thus, as expected a neural net is necessary to solve this problem. This 2-D (two feature) example is instructive to visually demonstrate the ability of a neural net algorithm to easily compute complex decision surfaces for difficult discrimination problems. We now consider a more complex version of the pattern recognition problem in Section 4, one that requires the use of a neural net classifier. Specifically, we consider three not two aircraft (F4, F104 and DC-10) and a larger f85" range of roll and pitch distortions. Figure 17 shows several views of each aircraft. In each set of images, the top center image is top-down with no distortions in roll or pitch. Each row left-to-right corresponds to pitch angles of -80", -40", 0", +40", and +80". From top-to-bottom, they correspond to roll angles of 0", 40" and 80". We attempted to solve this multiclass pattern recognition problem using the linear classifier of Section 4 and obtained poor results (Pc < 60%). We then used our ACNN algorithm with Nl = 33 input neurons (32 wedge IFT( features plus one bias neuron), N3 = 3C = 9 hidden layer neurons and N5 = C = 3 output neurons, one per class. The training set consisted of the feature spaces for 630 distorted objects per class (3 x 630 = 1890 in total). The test set consisted of over 1800 distorted objects at intermediate roll and pitch distortions between those used in training. The results we obtained were excellent (Pc = 98.6% correct recognition). This example vividly demonstrates the advantage of a neural net over a linear classifier for complex discrimination problems. Thus, in our general block diagram (Fig. 9), we show a neural net classifier used on feature space data calculated for regions of interest (ROIs) obtained from the detection portion of our general scene analysis system. 6.3. Production System Neural Net Another useful very high-level function in scene analysis is a production system. In this case the various facts learned about each ROI in the scene must be analyzed to obtain further data. To achieve this, one can write a set of IF-THEN rules, e.g. I F a - + b I F a a n d c a n d f -+g I F b + a IFf andg-+c
where the antecedents are the entries to the left, the consequents are those on the right and the arrow denotes THEN. More complex formulations with predicate calculus can be produced, but this example suffices to show the point. One can implement the above production system rules on a neural net as we now discuss [33]. We use a two-layer neural net (optical matrix-vector multipler) with each fact (antecedent or consequent) assigned to a specific neuron and with an equal number of input and output neurons. We encode the rules in the weights (matrix) as shown in Fig. 18 for the above example.
5.2 Optical Pattern Recognition for Computer Vision 887
Fig. 17. Distorted images of the three aircraft used (a) F4, (b) F104, (c) DC10.
There are seven input and output neurons (a to g) for this simple example. The first matrix-vector multiplication and the neuron outputs after the first iteration indicate new facts learned from the initial input facts. We feed the output neurons back to the input neurons keeping previously activated input neurons (facts) still “on”. These subsequent iterations allow the system to learn new rules not directly encoded in the original rules. The iterations continue until an output object “consequent” neuron has been activated in which case the object identification of the input scene region has been determined. The optical realization of this system is
888
D. Casasent
Outputs (control signals or j u s t feedback)
Inputs (from sensors, correlators or feedback) Fig. 18. Production system neural net [23].
the simple matrix-vector processor of Fig. 6 with feedback (Section 2). One can extend this basic system in many ways such as by using analog neurons proportional to the probabilities of each antecedent fact and by use of predicate calculus rather than propositional calculus formulations. We recently [34]demonstrated this system optically in real time for a set of objects composed of generic object parts (circles, rectangles, horizontal and vertical posts, etc.).
7. Summary We have briefly reviewed the role for optics in four levels of general computer vision. As seen, optical processing has a significant role in each area, and optical hardware to achieve these functions is rapidly maturing. The algorithms described in each area of computer vision are novel and can also be implemented digitally. Our new morphological low-level vision algorithm for detection of regions of interest (ROIs) in a scene is most attractive. Its implementation and the realization of other low-level image enhancement operations on an optical correlator are also most attractive. We use a high-level vision optical correlator with new distortioninvariant filters to further analyze ROIs (and in some cases for detection and even object identification). For the general case, once ROIs have been detected, initially analyzed by a correlator and enhanced (if necessary) by morphological techniques, we perform feature extraction and finally use a neural net (for complex multiclass problems) for object identification of each ROI. References
I11 H. Stark (ed.), Applications
of Optical Fourier Transforms (Academic Press, 1982). I21 S. H. Lee (ed.), Optical Information Processing, Vol. 28, Topics in Applied Physics (Springer-Verlag, 1981). 131 G . I. Vasilenko and L. M. Tsibul’kin, Image Recognition by Holography (Consultants Bureau, 1989). [41 D. Casasent (ed.), Optical Pattern Recognition, PTOC. SPIE, -Val. 201, 1979. P. Schenker and H. K. Liu (eds.), Optical and Digital Pattern Recognition, Proc. SPIE, Vol. 754, 1987.
5.2 Optical Pattern Recognition for Computer Vision 889 [5] D. Casasent and A. Tescher (eds.), Hybrid Image Processing, Proc. SPIE, Vol. 638, 1986. D. Casasent and A. Tescher (eds.), Hybrid Image and Signal Processing, Proc. SPIE, Vol. 939, 1989. D. Casasent and A. Tescher (eds.), Hybrid Image and Signal Processing ZI, Proc. SPIE, Vol. 1279, 1990. [6] B. V. K. Vijaya Kumar (ed.), Optical Engineering, Special Issue on Optical Pattern Recognition 29,9 (1990). [7] D. L. Flannery and J. L. Homer, Fourier optical signal processors, in Proc. IEEE 77 (1989) 1511-1527. [8] D. Casasent, Coherent optical pattern recognition: A review, Optical Engineering 24 (1985) 26-32. (91 G. G. Lendaris and G. L. Stanley, Diffraction-pattern sampling for automatic target recognition, Proc. IEEE 58 (1979) 198-205. [lo] P. Maragos, Tutorial: Advances in morphological image processing and analysis, Optical Engineering 26 (1987) 623-632. [ll] P. C. Lindberg and C. F. Hester, The challenge to demonstrate an optical pattern recognition system, Proc. SPIE, Vol. 1297, Apr. 1990, 72-76. [12] D. A. Gregory, J. C. Kirsch and J. A. Loudin, Optical correlators: optical computing that really works, Proc. SPZE, Vol. 1296, Apr. 1990, 2-19. [13] J. Goodman, A. R. Dias and L. Woody, Fully parallel high-speed incoherent optical method for performing discrete Fourier transforms, Optics Letters 2 (1983) 1-3. [14] J. Ohta, M. Takahashi, Y. Nitta, S. Tai, K. Mitsunaga and K. Kyuma, A new
approach to a GaAs/AlGaAs optical neurochip with three layered structure, in Proc. ZJCNN Int. Joint Conf. on Neural Networks, Washington, D.C., Jun. 1989, Vol. 11, 11-477-11-480. [15] D. Casasent and E. Botha, Optical symbolic substitution for morphological transformations, Applied Optics 27 (1988) 3806-3810. [IS] D. Casasent, R. Schaefer and J. Kokaj, Morphological processing to reduce shading and illumination effects, Proc. SPIE, Vol. 1385, 1990, 152-164. [17] D. Casasent and R. Schaefer, Optical implementation of gray scale morphology, Proc. SPZE, Vol. 1658, Feb. 1992. [18] D. Casasent, R. Schaefer and R. Sturgill, Optical hit-or-miss morphological transform, Applied Optics 31 (1992) 6255-6263. [19] R. Duda and P. Hart, Pattern Classification and Scene Analysis (John Wiley and Sons, New York, 1973). [20] B. Telfer and D. Casasent, Ho-Kashyap optical associative processors, Applied Optics 29 (1990) 1191-1202. [21] D. Clark and D. Casasent, Practical optical Fourier analysis for high-speed inspection, Optical Engineering 27,5 (1988) 365-371. [22] J. Richards and D. Casasent, Real-time optical Hough transform for industrial inspection, Proc. SPZE, Vol. 1192, 1989, 2-21. [23] D. Casasent, Optical processing and hybrid neural nets, Proc. SPIE, Vol. 1469, Apr. 1991. [24] D. Casasent, Unified synthetic discriminant function computational formulation, Applied Optics 23 (1984) 162C1627. [25] B. V. K. Vijaya Kumar and E. Pochapsky, Signal-to-noise ratio considerations in modified matched spatial filters, J. Opt. SOC.Am. A 3 (1986) 777-786. [26] A. Mahalanobis, B. V. K. Vijaya Kumar and D. Casasent, Minimum average correlation energy (MACE) filters, Applied Optics 26 (1987) 3633-3640. [27] G. Ravichandran and D. Casasent, Minimum noise and correlation energy (MINACE) optical correlation filter, Applied Optics 31 (1992) 1823-1833.
890
D. Casasent
[28] H. Wechsler (ed.), Neural Networks for Human and Machine Perception (Academic Press, 1991). [29] C. L. Giles, R. D. Griffen and T. Maxwell, Encoding geometric invariances in higherorder neural networks, in D. Anderson (ed.), Neural Information Processing Systems, Denver, CO (AIP, 1988) 301-309. 1301 D. Casasent and E. Barnard, Adaptive clustering optical neural net, Applied Optics 29 (1990) 2603-2615. [31] T. M. Cover and P. E. Hart, Nearest neighbor pattern classification, IEEE Puns. Inf. Theory 13 (1967) 21-27. [32] M. J. D. Powell, Restart procedures for the conjugate gradient method, Muthematical Programming 12 (1977) 241-254. [33] E. Botha, D. Casasent and E. Barnard, Optical production systems using neural networks and symbolic substitution, Applied Optics 27 (1988) 5185-5193. [34] D. Casasent and E. Botha, Optical correlator production system neural net, Applied Optics 31 (1992) 103&1040. [35] D. Casasent, Optical morphological processors, Proc. SPIE, Vol. 1350, 1990, 380-394.
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 891-924 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
I CHAPTER 5.3 I INFRA-RED THERMOGRAPHY: TECHNIQUES AND APPLICATIONS*
M. J. VARGA and P. G . DUCKSBURY Image Processing and Interpretation, Defence Evaluation and Research Agency St Andrews Road, Malvern, W R l 4 3PS, UK Infra-red (IR) technology is applied in a wide range of application domains, e.g. military, medical and others. All objects, live or dead and of any colour, emit infra-red radiation by virtue of their temperature; the exact degree of radiation is determined by the absolute temperature and the thermal characteristics of the material from which it is made. The radiation is present day or night with or without external illumination. Infra-red technology is concerned with the detection and imaging of the emitted infra-red radiation. Infra-red imaging is therefore a method for producing an image of the heat emitted from any object’s surfaces. A thermogram is a calibrated graphic record of the temperature distribution obtained by thermography.
Keywords: Infra-red thermography, medical imaging, Bayesian networks, Target Deterction.
1. Introduction Infra-red (IR) technology is applied in a wide range of application domains, e.g. military, medical and others. All objects, live or dead and of any colour, emit infra-red radiation by virtue of their temperature; the exact degree of radiation is determined by the absolute temperature and the thermal characteristics of the material from which it is made. The radiation is present day or night with or without external illumination. Infra-red technology is concerned with the detection and imaging of the emitted infra-red radiation. Infra-red imaging is therefore a method for producing an image of the heat emitted from any object’s surfaces. A thermogram is a calibrated graphic record of the temperature distribution obtained by t hermography. In this chapter, both the hardware and some of the processing techniques associated with infra-red technology are discussed. Examples of military and medical applications will demonstrate the versatility of infra-red imagery. *The chapter is published with the permission of the controller of Her Britannic Majesty’s Stationary office @British Crown Copyright 1998/DERA.
891
892
M. J. Varga €9 P. G. Ducbbury
1.1. Infra-Red Wavebands Spectrally infra-red radiation is located between the visible and radio frequencies. Infra-red radiation is generally thought of in three spectral bands: Short Wavelength Infra-red (SWIR) also called near infra-red, lying between 0.7-2.0 pm; Medium Wavelength Infra-red (MWIR) ranging from 3.0-5.0 pm; and Long Wavelength Infra-red (LWIR) between 8.0-14.0 pm. Both the MWIR and the LWIR are strongly absorbed by water and organic compounds. Infra-red sources are either thermal (i.e. emitted by matter in the temperature range 100-3000 K) or electronic (i.e. emitted by high-energy electrons interacting with magnetic fields) [l-31.
1.2. Infra-Red Detectors There are two types of infra-red detectors, those that require cooling (cooled) and those that do not (uncooled). Cooled infra-red detector systems are bigger and more expensive than uncooled IR detector systems, but are more sensitive and can produce higher resolution images. Uncooled IR detectors on the other hand are cheaper, lighter, more compact and hence more portable. At present, however, they tend to be less sensitive and are commonly used for 8-14 pm only. The heart of a cooled thermographic camera is an infra-red photo-detector typically made of indium antimomide (InSb) or cadmium-mercury-telluride. This lies in the focal plane of the camera and is cooled by liquid nitrogen. Uncooled infra-red detectors typically use pyroelectric bolometer techniques. The infra-red radiation from an object is focused on the detector - the focusing system can be based on either refractive or reflective optics. The thermogram can be produced by a n array of detectors which converts the infra-red radiation directly into a n image. Alternatively, a scanning system can be used in which an image is built up by mechanically scanning the image onto a single detector or a linear or two-dimensional array. The signal can be represented as a grey-level or colour coded image. Long linear array detectors can be used with simple scanning mechanisms to generate high performance imaging over wide fields of view. Two-dimensional focal plane arrays increasingly provide the basis of systems which require no scanning and offer high sensitivity. There is a wide and developing range of infra-red focal plane array sensors using different detector technologies. Common to the development of all these arrays is the continual increase in thermal sensitivity. This enhanced sensitivity may be used directly or compromised to provide different operating designs. The available optical system and the required frame rate determine the choice of read-out process required t o achieve a given sensitivity [4].
2. Military Infra-Red Imaging Infra-red technology is applied in a variety of military applications and there is a need for both cooled and un-cooled systems. Cooled IR detectors offer high performance when required, for example in; weapon sights, surveillance systems, remote ground sensors, imaging IR seekers, non-co-operative target recognition,
5.3 Injra-Red Thermography: Techniques and Applications
893
mine sensors, driving aids, fire fighting and rescue. Where lower performance is acceptable, un-cooled infra-red detectors reduce the logistic burden and their low power and compactness is particularly useful for remote and autonomous operation.
3. Military Surveillance: Downward Looking IR Imagery The fundamental task here is generically defined as the location of some region of interest in an image, for example, the fast and automatic detection of urban regions [6]. This could be used as a cueing aid for more detailed processing, such as the detection of road networks or junctions and buildings in order to allow registration of imagery to maps for navigation, for change and target detection, image distortion correction as well as map-update. It could also be an attention cueing device for human image interpreters. This section describes the use of a Pearl Bayes Network (PBN) for the automatic extraction of knowledge about regions from infra-red linescan imagery, i.e. surveillance imagery.
3.1. In*-Red
Linescan Imaging and Correction
The aerial infra-red linescan imagery used in this application is produced by a sensor which has a single detector. Scanning in the x direction is achieved via a rotating mirror which has a uniform angular velocity and gives a 120" field of view. Scanning in the y-direction is achieved by the aircraft motion. The scanner arc introduces a characteristic (sec2) distortion into the imagery at either extreme of the 120" arc. This can be corrected with a relatively simple trigonometric transformation. Figure 1 illustrates the relationship in which h is the height of the aircraft, p is the bank angle of the aircraft, Ax is the ground resolution of a single pixel with A0 being the corresponding swathe angle for that pixel. The equation for the correction is Ax = ABh/ cos2(0 p).
+
r =
h
cos(0
+P 1 AQ
leH Ax Fig. 1. Linescan Sensor Distortion.
3.2. Pearl Bayes (Belief) Networks Pearl Bayes networks [7] are directed acyclic graphs, see Fig. 2. In this graph nodes B, C and E represent different statistical information extracted from the image, whilst node A represents the "belief" in detecting an urban patch. A graph
894
M . J. Varga €4 P. G. Duclcsbury
GBA
Fig. 2. Pearl Bayes Network.
G is a pair of sets (V,A) for which V is non-empty. The elements of V are vertices (nodes) and the elements of A are pairs (x,y) called arcs ( l i n k s ) with z E V and y E v. Consider the simple network that is shown in Fig. 2. Here the symbol 7r represents the causal support ( o r evidence) whilst X represents the diagnostic support (or evidence). GZA and GBA are subgraphs as described in the next section, together with the equations for computing the belief and propagation of information. a 3.2.1. Belief equations Consider the link from node B to A then the graph G consists of the two subgraphs GZA and GBA. These two subgraphs contain the datasets DLA and DBA respectively. From Fig. 2 it can be observed that node A separates the two subgraphs GAA u G z A U GLA and G I F . Given this fact we can write the equation: P(D,FIAi,
D i A , DAA, DLA) = P(DAFIAi)
(3.1)
aThe equations are derived along similar lines to those derived by Pearl in [7] where in his example node A has just two predecessors and two successors.
5.3 Infra-Red Thennogmphy: Techniques and Applications 895 by using Bayes rule while the belief in Ai can be written as
where a is taken to be a normalizing constant. It can be seen that Eq. (3.2) is computed using three types of information: 0 0
0
Causal support T (from the incoming links). Diagnostic support X (from the outgoing links). A fixed conditional probability matrix (which relates A with its immediate causes B , C and E ) .
Firstly the causal support equations:
Secondly the diagnostic support equation is given by
Finally the conditional probability matrix is defined to be
The belief Eq. (3.2) can now be rewritten in order to obtain the belief at node A based on the observations at B, C and E7 e.g. the belief that an urban region is detected:
The belief at nodes B7Cand E can be obtained from the equations (3.9) (3.10) (3.11)
896 M. J. Varga & P. G. Ducksbury
In other words the belief is the resultant product of causal support information, diagnostic support information (belief) and prior knowledge. The propagation equations described below are iterated to support belief of a certain event. 3.2.2. Propagation equations
The propagation equations for the network are derived as follows, firstly the diagnostic support. From a previous analogy with Eq. (3.6) we can write
by partitioning the DBA into its component parts, namely A, DAF, D$A, D i A we can obtain
likewise for X ~ ( c j )and XA(Ek)
and
3.2.3. Causal equations
These are defined using a similar analogy as follows. (3.16)
and from this we then derive the equation
An important point to realise is the fact that Eqs. (3.13)-(3.15) and (3.17) demonstrate that the parameters X and r are orthogonal to each other i.e. perturbation of one will not affect the other. Hence evidence propagates through a network and there is therefore no reflection at boundaries.
5.3 Infra-Red Thermography: Techniques and Applications 897 s1 -s3 : Edges,
Extrema, Distribution type from fine resolution image.
s 4 - s 6 : EdgCS,
fmm coarse resolution image.
B
: Belief
B
: Belief
from tine molution.
Half Resolution Pixel Data
- - - - - - - - - (fine) --statistics
FUl Resolution (raw)Pixel Data Fig. 3. Multi-resolution approach.
3.3. Region Segmentation Using a Pearl’s Bayes Network
The above Pearl’s Belief network approach [7]has been adapted for the detection of urban regions [8,11],using a high powered parallel processing system for improved performance. The belief network is used in a multi-resolution sense to combine statistical measures of texture into the detection ( o r belief) of the required region,b see Fig. 3. The problem is approached by taking several statistical measures from small patches of a n image which are treated as a set of judgements about the content of the patches. These statistics are the number of edges, the number of extrema and grey level distribution type. These statistics are quantised down into a small number of levels. The number of edges and extrema are both reduced to five levels, whilst the distribution type has four possibilities. It is important t o stress that any suitable measure that provides the required textural discrimination could have been used. The statistics are then used to produce a set of judgements; for example, an expert might, upon looking at a particular window issue a report of the form (0.0,0.7,0.9,0.6,0.0). This means that he believes there is a 70% chance that level 2 describes the number of edges, 90% chance that its level 3 will do the same and 60% for level 4. But he believes there to be no chance of it being levels 1 or 5. bThe authors have recently carried out some initial work into using the Belief Network approach at a higher level of abstraction, i.e. for combination of several region finding algorithms.
898 M. J. Varga tY P.
G. Ducbbury
For the purpose of the system described here the Belief at nodes B f , B, and B in Fig. 3 is quantised to three levels, namely (low, medium, high). The fixed conditional probability matrices (i.e. P ( B f I s 1 ,s2,sg) etc), which are the prior information and relate the given node with its causal information, are created along similar lines to the approach used in [9] and [lo]. They are based upon the assumption that the probability of an event at a given node should be greater if its causal information is more tightly clustered together than it should be if the causal information is further apart. For the P ( B ( B fB,) , matrix (which relates the beliefs from the fine and coarse resolutions) slightly more emphasis is given to the causal information received from the coarse resolution belief. P ( Bf i l s l j , s2k, s31) is described formally as
. .
if a = j = k = 1 0.25/a if (i # j = k = 1 ) AND (0 < li - j l 5 C ) 0-75 if ( j = k = 1 ) l.O/p P(Bfilslj,S2k,s31) = AND (max(j, k,1 ) - min(j, k,I ) 5 2 C ) AND (min(j, k,1 ) L i 5 max(j, k,I ) 10.0 otherwise (3.18) such that & k , l P ( B f i I s l j ,S2k, s31) 5 1 vi where c = 1 and i , j , k, range over the number of variables in B f , s l , s2 and s3 respectively. a and p represent the number of different values of i satisfying the constraint. P ( B I B f ,B c ) is defined as
I
1
I
0.9 0.7 0.3 P(Bi1BfjBck) = 0.6 0.1 ( 0.0
if i = j = k if (i = j ) AND (li - kI 5 1 ) if (i = k) AND (li - j l 5 1 ) if (i = j ) AND (li - kl > 1 ) if (i = k) AND (li - j l > 1 ) otherwise
(3.19)
such that x j , k P(Bi1BfjBck) 5 1 V i where i, j , k range over the number of variables in B , Bf and B c respectively.
3A. Performance An example result is shown in Fig. 4, which demonstrates the location of urban regions in a near infra-red image. The image was taken during the night at approximately 4000 feet. An outline has been drawn around the areas labelled as most likely to be urban. It is possible to post-process such a result to remove small isolated regions which are likely to be erroneous classifications, this has not been done here. A probability surface could also have been integrated into the image to produce a smoother transition between the areas of different classification.
5.3 Znfra-Red Thermography: Techniques and Applications
899
Fig. 4. Outline of Potential Urban Region.
This system can be easily adapted to alternate applications of a broadly similar nature, i.e. classifying or clustering regions. The only change necessary may be a different set of statistics which more accurately describe the detail required in the image. In addition, if the number of input nodes alter then the prior knowledge in the fixed conditional probability matrix will need to change. However the set of basic equations given previously can be used to automatically generate this informat ion. This approach has been demonstrated successfully for the texture based segmentation of driveable regions for autonomous land vehicles, and more recently for urban region segmentation in both SPOT, and Russian Satellite imagery in [8,12,13]. The algorithm was developed initially on a SUN workstation prior to implementation on a parallel processor architecture developed within the UK Defence Research Agency [141.
4. Target Detection and Tracking in Forward Looking IR Imagery 4.1. Introduction One of the most powerful features in any modern battlefield surveillance system is the ability to carry out automatic detection and tracking of targets. The amount of information being presented t o human operators is increasing at an alarming
900 M . J. Varga €4 P. G. Ducbbury
rate and needs to be reduced. Any system that can simply filter the data to present results of possible targets without all the intermediate information will be of significant benefit. The task here is generically defined as the detection of small point sized “hot” targets in IR imagery. These potential targets will be within some operator specified sizes, typically 2 x 2 t o 10 x 10 pixels. The resulting detection can be used t o aid subsequent target recognition through cueing for a narrow field of view imager and/or a human operator. The requirement for the system described here was not necessarily to locate all targets or indeed just the targets, but rather t o locate possible areas of interest for further analysis. The wide field of view sensor used produces very small potential targets. These targets have low contrast and low signal-to-noise ratios which makes their detection difficult. 4.2. System Overview
The system combines some “conventional” image processing techniques with morphological analysis t o perform automatic cueing and tracking of small objects/ targets. Most stages of the process have been deliberately chosen because of their suitability for future implementation in special DSP hardware modules. The process is shown schematically in Fig. 5. Only some of the main elements of this system are considered below.
Median Filter + lllmholding __________
Morphological closing Local Morphological Dilation (Redback) L
Display Annotation
Fig. 5. Algorithm Block Diagram.
5.3 Infra-Red Thennography: Techniques and Applications 901 4.2.1. Destriping Sensor imperfections mean that the imagery contains marked fixed pattern stripe artefacts. A destriping stage is therefore required to reduce or even to eliminate the banding effects that occur in this type of imagery. It is likely that future generations of infra-red imagers for this application will produce images of improved quality with imperceptible striping artefacts and hence such a destriping algorithm will become unnecessary. The destriping algorithm (151 removes bias variation between adjacent scan lines. Two adjacent scan lines i and i 1 are equalised by determining the distribution differences between adjacent pixels along the scans. The majority of entries should reflect differences in the baseline between the two scan lines. An additive correction to scan line i 1 is obtained from the median of the difference distribution, the median being used as it is a statistically robust measurement. This process is then repeated using scan lines i 1 and i 2 and so on. The disadvantage of this approach is that a software implementation is relatively slow. So an alternative scheme was developed as an intermediate measure (prior to hardware design and implementation of the full destriper). The approach is basically to model the sensor responses in order t o estimate a set of corrective factors. If it is assumed that the image is uniform over a number of neighbouring scan lines then any differences should be due to the sensor itself. The median of each image row is obtained and the maximum of all medians is taken as the maximum sensor response. The difference of the maximum response and the median of each row can then be used as an additive amount for that row. The analysis can be done on the first frame in the sequence and then at successive frames the corrective amounts are simply added to each row.
+
+
+
+
4.2.2. Median filter The median filter [16,17] is used to remove the salt-and-pepper type of noise. In a small window over an image the pixel values are likely to be homogeneous with only a small number of them being attributable to noislk. These noisy pixels tend to be a t either extreme of the grey level distribution and therefore are unlikely t o be selected as the output from a median filter (the output being the median of the ranked inputs). This filter has the advantage of reducing noise without having a smoothing effect on the image. In this instance since the possible targets are very small only a 2 x 2 median filter" is applied; despite this small window size the filter is successful in removing noise. =An important point to be considered is that a median filter when used for noise suppression can in fact be replaced by gray scale morphology. This point does not really apply to this particular algorithm as the filter we use only has a small kernal. It is however important and worth mentioning. A morphological opening followed by a closing operator can achieve the same effect as a median filter. Morphology has two distinct noise suppression stages, the opening suppresses positive noise impulses whilst the closing suppresses negative noise impulses.
902
M. J. Varga €4 P. G. Ducksbury
4.2.3. Thresholding Thresholding [17,18]is used to segment objects from the background. It is useful, whenever possible, to calibrate detected feature values (e.g. grey level) so that a given amplitude interval represents a unique object characteristic. There are various useful adaptive thresholding schemes, for instance, based on the examination of local neighbourhood histograms or other measures. Unfortunately these approaches can produce a significant amount of noise and in this application such noise would pose a major problem due to the small target sizes. In an effort to minimise the problem for this application the thresholding scheme used is based upon the global mean and variance of the portion of the imagery being processed. The threshold is set at p 30. This proved to be an acceptable level for this application domain, but an option has been provided in the algorithm for this to be varied interactively during run time (see also the section below on local morphological dilation).
+
4.2.4. Morphological closing The term morphology originally comes from the study of forms of plants and animals. In image processing it means the study of topology or structure of objects from their images. Morphological processing refers to certain operations where an object is “hit” with a structuring element and hence “reduced” to a more revealing shape. Most morphological operations are defined in terms of two basic operations, namely erosion and dilation. Erosion is a shrinking operation, whereas dilation is an expansion operation; erosion of an object is accompanied by enlargement or dilation of the background. If X the object and K the morphological structuring element are thought of as sets in two-dimensional Euclidean space, then the erosion of X by K is the set of all points x such that K , is included in X,where K , is the transformation of K so that its origin is located at x. The dilation of X by K is the set of all points x such that K , intersects with X. The morphological closing operator is defined as an erosion followed by dilation. Closing aims at blocking up narrow channels and thin lakes and is ideal for the study of inter-object distance. The reasons for applying a morphological closing operator in this application are twofold. Consider the thresholding of an image, this can and does result in the fragmentation of objects. Firstly, an object which is a target could be fragmented into several parts thus leading to the possibility of several targets being detected instead of one. Secondly, an object which is not a target (perhaps by virtue of its size) could be fragmented into small parts which are then likely to be identified as possible targets. To resolve this problem a morphological closing operator is applied in an attempt to piece the fragments back together. The structuring element kernel use is deliberately kept small to try to avoid merging several genuine targets. A fuller and more detailed description of morphology can be found in numerous papers in the literature, see, for example; [19-231.
5.3 Infra-Red Thermography: Techniques and Applications 903 4.2.5. Connected component labelling
The objective of connected component labelling (CCL) is to take a binary image and to apply a segmentation in order to obtain a set of connected regions, each of these disjoint regions being labelled with a unique identifier. Although this stage is currently performed in software a design and hardware module exists for future use [24]. This fundamental process is important in many applications and can be used as an input to shape recognition tasks. 4.2.6. Target elimination
Once a labelled image has been obtained regions can be discarded according to a number of criteria. The human operator will initially have specified a set of bounds for targets of interest. Regions which have a width, height or total area outside these constraints are discarded. It is not possible to discard objects based upon shape without knowledge of the type of targets, such information is unavailable in this application domain. 4.2.7. l?racking/prediction
Once an acceptable set of regions has been obtained the co-ordinates of the centre points are passed to the tracking process. Tracking introduces an element of temporal consistency into the system. This is used to resolve a number of issues such as false targets (due to segmentation errors or genuine noise in the imagery), targets appearing and disappearing, and overlapping targets. Once these issues have been resolved a prediction stage is performed to estimate the target’s position in the next frame. Targets develop a “history” over n frames and therefore isolated noise which appears for n - 1 frames or less will not be tracked and can be eliminated. The initial part of the tracking is actually an association stage where observations are associated with tracks. This uses a standard assignment optimisation algorithm [25] which was modified by [26] to deal with targets which appear and disappear. It was also modified by the authors to resolve the problem of several observations being identical distances from a given track but outside the permittable (gated) regions for all other tracks. This condition appeared to cause the standard algorithm to fail to converge to an optimum assignment. Kalman filtering [27] is the classical approach for the prediction of a target’s new position. It is the optimal predictor for tracking. It has been shown 128,291 that if the x and y target co-ordinates can be decoupled the Kalman filter can be reduced to the so called CY - ,O filter, which is much simpler and requires no matrix multiplication. 4.2.8. Local morphological dilation (Adaptive threshold feedback)
An important point arising from the thresholding is the difficulty in setting a threshold level at just the correct value for detection of all targets. As has been
904 M. J. Varga €4 P. G. Duclcsbury
mentioned previously it is possible for noise to be included and also for a genuine target (but one which is very small and/or emitting low thermal radiation) to be excluded from the thresholded image. The effect of this has been reduced by incorporating feedback from the tracking algorithm and essentially using the track confidence to adapt the output from the thresholded image. This is achieved simply by performing a local morphological dilation in an area around known targets (targets that developed a history). This attempts to enlarge the thresholded output to a point where it would be accepted by the target elimination stage and effectively reduces the number of dropouts due to weak targets. If a target genuinely disappears then this approach will have no effect. 4.2.9. Performance
A target (or rather areas of possible interest) is indicated by a diagonal line in Fig. 6 , the lower part of which points towards the target centre whilst the upper part has a unique target identifier.
Fig. 6. Target Detection.
The algorithm was initially developed on a SUN workstation prior to implementation on a parallel architecture. In order to increase performance to real-time the algorithm has now been ported onto a version of the architecture which includes a number of dedicated DSP modules.
5. Medical Applications 5.1. Infra-Red Imaging i n Medicine Infra-red thermography has been used clinically for over thirty years. It provides a means of obtaining high quality images of thermal patterns on the surface of the human body. The IR systems used for medical applications should have a sensitivity that varies from 0.01"C to 1°C and respond to temperatures in the range 5OC to 40"C, depending on the particular system and the part of the body to be examined [30]. In these systems any error caused by variation in the detector's response can be calibrated out, for example, by alternatively detecting the radiation from the object and the radiation from a reference source.
5.3 Infra-Red Thermography: Techniques and Applications 905 The correlation between the skin temperature and underlying malignant disease was first realised in the case of breast cancer. This inevitably resulted in initial infrared thermographic studies being concentrated on breast diseases [31]. However, there were problems, for instance, with limited sensitivity t o deep lying tumours and poor control of environmental conditions during examination and recording. The majority of the detected radiation comes from the topmost layers of the skin, i.e. body surface to 300 microns depth. The surface temperature of the skin is affected by both internal and external factors. The internal factors can be pathological or physiological, while the external factors are a function of ambient conditions, such as temperature, humidity and air flow. Indeed ambient air flow is very important in medical thermography, and a uniform environment without any cooling draught, direct warmth of sunlight, or radiators etc., is essential. In general an ambient air temperature of between 18°C and 20°C has been found to be appropriate. Temperatures below 18°C may induce a cold stress response and shivering, resulting in “flicker” thermograms. While temperatures above 20°C may cause sweating and create other anomalies and noise on the image. It is also important for the area of the body under examination to reach a steady state in controlled environmental conditions. In some cases, it is necessary for the patient to partially undress so as to expose the area of the body to be examined directly t o the ambient temperature for a short stabilisation period (10-15 minutes is usually enough). Loose clothing and underwear are required to avoid altering the local blood flow ( a n d thus the overlying skin temperature) through the pressure and restriction caused by tightfitting garments. Dressings, ointment or any other surface moisture will affect, to a certain degree, the infra-red emission from the skin. These must all be eliminated prior to equilibration if thermography is to be used in a controlled manner. This sensitivity is due to the fact that infra-red radiation in the wavelength typically used (i.e. between 3-5 or 8-12 pm) are strongly absorbed by water and organic compounds [32]. If environmental conditions are adequately controlled, heat emission from the skin is largely determined by the underlying blood supply. In the absence of deeper lying organic disease or other factors, which may indirectly alter skin blood flow, the thermographic image of the heat emitted by the skin may be interpreted in terms of the status of the underlying peripheral circulation [33-361. Thermography can therefore be used for detecting peripheral arterial disease, diabetic angiography, Raynaud’s phenomenon and related conditions, inflammatory conditions, and for the determination of amputation level. For deeper-seated pathological conditions radio-isotope imaging, ultrasound or radiography are more suitable. At present IR thermography is most widely used in applications associated with the vascular system [37,38], peripheral and cutaneous circulations as well as relatively superficial tissue lesions. In some cases thermography provides a beneficial preliminary or complementary aid to examination, in others it fills in gaps in the existing armoury of assessments. However, its use in clinical assessment is still considered by some to be controversial, partly due to the wide range of temperatures
906
M. J .
Varga & P. G. Duclcsbury
of lesions or diseases (this is particularly true in breasts) and also due to the lack of understanding of the basic principles of thermography (i.e. its characteristics and limitations). The work reported in Secs. 6 and 7 addresses some of these recognised problems.
5.2. Static and Temporal Sequence of Thermogmms Various methods have been used for the presentation, analysis and classification of thermograms. These include functional images, spatial signatures and texture signatures. Historically these methods have been applied to individual static thermograms for diagnosis. Many conditions, however, are not evident from such single static images. Historically, also, the assessments from a static thermogram would normally be based on an individual patient’s data only, much like the common use of X-ray pictures. Diagnostic results would be in the form of an index or some form of written report, with or without graphical explanation. These approaches do not fully utilise the information available from thermography. It has been found that useful information can be obtained by observing the thermal behaviour of the body over time. In order to do this the technique of temperature stress testing has been developed whereby a temporal thermal response is induced in the body under controlled conditions. A sequence of thermograms is taken over time to record the body’s thermal behaviour. Two diagnostic systems will be described both of which use analysis of the temporal thermal response and wider statistical data for automatic classification and diagnosis. The first, in Sec. 6, is concerned with diagnosing joint disease, namely arthritis and rheumatism as well as Raynaud’s syndrome. It is based on analysing the bodie’s thermal behaviour after a “cold stress”. The second system, described in Sec. 7, is concerned with the diagnosis of breast cancer after a “heat stress”. 5.3. Advantages and Disadvantages of Thermography in
Medical Applications There are numerous advantages t o applying infra-red thermography in clinical and medical investigations. They include:
0
0 0 0
The recording of a patient’s thermogram is inexpensive to perform, although initial equipment costs are high. The technique is simple, and can be repeated at frequent intervals, allowing real time assessment (especially compared with radiological images), and hence results in good patient co-operation. It is non-invasive and involves no radiation hazard. Thermography may indicate lesions too small to be seen on a roentgenogram. Each patient has a unique thermogram, useful in a follow-up analysis. Thermographic images are opaque and do not contain the overlapping objects present in radiographic images.
5.3 Infra-Red Thermography: Techniques and Applications 907 0
0
0
Thermographic images are inherently quantitative in nature due to the direct representation of the physical temperature. Only a rudimentary knowledge of anatomy and pathology is necessary t o interpret a thermogram. Diagnostic criteria involve essentially a measure of the symmetry and relative temperature distribution and pattern. Because the temperature signal is available directly in an electrical form, it is simple to connect the thermographic equipment directly to a computer via an analogue-to-digital converter for automatic image acquisition and storage.
There are, of course, also disadvantages which include: 0
0 0 0 0
0
0 0
Lack of specificity. Provides limited range of image brightness. Low spatial resolution. Not all abnormalities exhibit observable thermal phenomena. Thermal variations do not necessarily have a spatial relationship to the disease investigated. Thermographic signs can occur in benign as well as malignant conditions (e.g. breast thermography) Occasionally, an anatomical aberration gives a false positive reading. Simultaneous, bi-lateral symptoms could be diagnosed as negative.
6. Joint Disease Diagnosis Using Dynamic Infra-Red Thermography
6.1. System Overview The main objective of this work was to automate the analysis and classification of a temporal series of thermograms (recording the response of the hand to a cold stress) into different classes namely, normal, Raynaud’s and inflammatory. Various other quantitative imaging systems have been used for assessment of such conditions, for example: the differential thermistor thermometer, the infra-red thermometer, radiography and arteriography. Their limitations lie in the technical difficulties, expense and/or their invasive nature. In this system statistical pattern recognition techniques are used to analyse the results of a temperature stress test. The temperature stress is induced by immersing the hand in a cold water bath at 20°C for 1 minute (cold stress). The body’s response to the stress is recorded by thermograms taken a t regular two minute intervals over about twenty minutes. In order to study the thermal response of the hand it is, in general, necessary t o track the thermal behaviour over time of every point on the hand. It is therefore necessary to have knowledge of the correspondence between points in each of the series of images. The simplest way t o do this would be to ensure that there was no movement of the hand over the twenty minute period, thus the thermal response of each object pixel could be tracked in a straight forward manner. Unfortunately, it is not possible to restrain the hand in a way that does not affect its thermal response. The patient must simply hold the hand in front of
908
M. J . Varga & P. G. Ducksbury
the camera and the correspondence must be built up by aligning the images of the hand with an aligning and stretching algorithm [39]. The thermograms of hands used in this work were taken at a distance of between 0.75-1.5 metres, depending on the size of hand in question. The thermograms were digitised into a 128 x 128 square image with 256 grey-levels. In this work only the anterior view of the hand (back of the hand) was used, and no attempt was made to analyse thermograms taken from posterior or lateral views. The feature extraction method developed for this application is based on the Kittler & Young transformation. The function of this transformation is to extract those features that appear most important for classification. A 7-Nearest Neighbour (7-NN) classifier, built using the Condensed Nearest Neighbour (CNN) technique, is applied to the features. It was recognised that it would be desirable for the resultant classification (diagnosis) t o be presented as a colour coded diagnostic image showing both the classified disease category and the location of the “hot spots” (inflammatory condition), or “cold spots” (Raynaud’s). The severity of the disease should also be indicated through the intensity of the colour in the diagnostic image. For example, using green colour for normal cases, red for inflammatory cases and blue for Raynaud’s. In the case of uncertainty, the resultant image would have a non-primary colour. The system achieved about 96% accuracy at pixel level.
6.2. Thennographic Colour Coding Schemes 6.2.1. Gradual natural scale The use of grey-scale (Fig. 7) or pseudo-colour in medical images has been the subject of debate. Some believe that colour-coding is artificial and misleading and can create confusion which leads to misinterpretation. While others find that the use of grey-scale in some images makes it difficult to differentiate pathological areas from normal areas. This controversy could partly be due to the use of inappropriate pseudo-colouring systems which are insensitive to the particular information required and the requirement of special expertise t o interpret the colour codes.
Fig. 7. A Grey-Scale Hand Thermogram.
5.3 Infra-Red Thermography: Techniques and Applications 909
Fig. 8. Gradual Natural Scale.
Two different colour coding schemes are considered here [40]. The first, the Gradual Natural coding scheme is shown in Fig. 8. It is a smooth gradual scale ranging from black (cold) to white (hot) with blue, green and red gradually intermixing with one another over the inter-mediate range. Such a scheme conveys the overall temperature range of an image in an easily identified and recognised colour spectrum, giving a general idea of where the temperatures lie. The semi-circular disc shown on the left of the image is the temperature standard. The encoding is based on three primary colours - red, green and blue, and has 8 bits to represent all the colours, i.e. 256 levels. These levels are split into 3 ranges, one for each colour, and within the range the intensity can be varied uniformly and gradually. At the boundary between any two primary colours (e.g. green and red), a gradual mixing of the two colours (e.g. a decrease in green colour intensity accompanied b y an increase in red colour intensity) results in the perception of a non-primary colour (e.g. yellow). This non-primary colour is necessary to create smooth changes over the boundary of the two different colours, thus providing an overall gradual and smooth colour spectrum. 6.2.2. Randomised blocking scale
The second coding scheme, Randomised Blocking, is illustrated in Fig. 9; as its name suggests it uses randomised colour blocks. As before, black and white are at either ends but the intermediate colours are small repetitive blocks with different colours and intensities. Adjacent temperatures are represented by significantly different colours, so that slight temperature differences will be accentuated; such differences would otherwise be undetected. The coding scheme is constructed as follows. The 8-bit control byte is split into three fields, one for each primary colour; red, green and blue. Thus in the example of Table 1, bits 0, 3, and 6 are associated with red (R), bits 1, 4, and 7 for green
910
M. J. Vurga €d P. G. Ducksbury
Fig. 9. Randomised Blocking Scale.
Table 1. Coding Scheme. Bit
7
6
5
4
3
2
1
0
Colour Example
G
R
B
G
R
B
G
1
1
0
1
1
0
1
R 1
(G) and bits 2 and 5 for blue (B). This 3-3-2 combination means that there are 7 possible intensities for both red and green but only 3 for blue. An example is given in Table 1, here level 7 of green is combined with level 7 of red and no blue (e.g. resulting in a yellow colour). This coding scheme is also useful for coding of disease classification categories because in such applications there is not necessarily a uniform continuum of information to be encoded (i.e. just blue of some degree, just red of some degree or just green) but rather, as in this case, any possible combination of red and green or blue and green. 6.3. Observed Hand Thermal Characteristics During the course of data collection, some thermal characteristics or behaviour patterns were noted. It is these patterns that the system must be able to extract, analyse and quantify. 6.3.1. Normal condition It is known that in a normal hand all the fingers remain at similar temperatures (27°C f2"C) and display a positive gradient, i.e. temperature increases towards the finger tips. Most showed hypothermia during the 15-20 minutes stabilisation period, which resulted in an increase in hand temperature and which decreased the temperature differences between a normal and an inflamed hand.
5.3 Infra-Red Thermography: Techniques and Applications 911
Fig. 10. Example of a Normal Hand.
After the cold stress the hand rewarmed quickly and throughout the test the hand maintained its temperature distribution with hotter fingers. Examples of a normal hand is shown in Fig. 10. In some normal cases a diagonal gradient pattern could be noted while in other cases isolated cold fingers were found which did not necessarily relate to the symptoms. 6.3.2. Raynaud’s conditions In Raynaud’s phenomenon (primary and secondary) the mean temperature was always lower than 27°C (approxcimately 22°C - 23°C) and during the cold stress the temperature range across the hand was typically up to 10”Cl whereas in normal hands it was no more than 6°C. The symptoms were characterised by a well banded isothermal pattern with negative gradient (i.e. colder towards the finger tips). In some patients with severe cases of secondary Raynaud’s the gradient could be over as much as 12°C. Similar, but less marked, banded patterns over the hand were found in patients with primary Raynaud’s. The hands of patients with Raynaud’s condition tended to cool down during the stabilisation period prior to the cold stress. In patients with primary Raynaud’s, the hands rewarmed after the stress and in the end the gradient differences reduced. In patients with severe secondary Raynaud’s the hands would cool down further in response t o the cold stress. An example is given in Fig. 11. 6,3.3. Inflammatory conditions
Higher temperature is often recorded (29°C - 34°C) for arthritic hands due to the inflammatory mechanism. Classic symptoms being swelling, slight deformity and the presence of a “hot spot”. The temperature rise on the overlying skin at the affected joint can be up t o 5°C. The precise nature and extent of such hypothermic areas are determined by the underlying pathology. For example, synovitis may cause a localised area of increased temperature, while chronic rheumatoid arthritis
912
M. J. Varga & P . G. Ducksbury
Fig. 11. An Example of a Raynaud’s Hand.
Fig. 12. An Example of an Inflammatory Hand.
may result in a generalised hypothermia over the whole joint. Gout also causes dramatic and characteristic increase in temperature over affected joints. Inflamed areas remained at higher temperatures after cold stress. During the “warm up” period the affected hand warmed up as with the normal hand but less markedly. In fact, due to vasodilation, the temperature difference between the two classes was less prominent in the early stage of cold stress response [38]. An example is given in Fig. 12. 6.4. Application of the Kittler €3 Young Method to Sequences
of Hand Thermograms 6.4.1. The Kittler & Young method
Ideal classification criteria should be based on the discriminatory potential of both the class means and class variances, scaling them according to their relative significance. Such criteria are, however, complex and difficult to formulate. In many practical situations the contribution of the differential ability of class variances is neglected. This simplicity is assumed by all the standard variants of the
5.3 Infra-Red Thermography: Techniques and. Applications
913
Fig. 13. Series of Thermograms.
Karhunen-Loeve transformation method [41]. The Kittler & Young method [42] is a feature extraction method based on two Karhunen-Loeve transformations, and is intended to overcome this problem. 6.4.2. Application of the Kittler
63 Young method
The Kittler & Young method was applied to a series of thermograms (e.g. those in Fig. 23) to compress the differential thermal information, based on both the thermal means and variances (of diflerent classes), of thermograms into a series of transformed “Eigen” images. In order to accentuate the features of the first three resultant transformed Eigen images for visual inspection and for further use as diagnostic images, the following colour coding scheme, based on the RB scale, was applied: 0
The first transformed image (Fig. 14) was coded into 7 different levels of green, where the maximum value corresponded to the darkest green and vice versa. The
Fig. 14. First Transformed Image Coded in Green.
914
M. J . Varga Ed P. G. Ducksbury
Fig. 15. Second Transformed Image Coded in Red.
Fig. 16. Third Transformed Image Coded in Blue.
0
0
7 levels of green are given the following values: 0, 2, 16, 18, 128, 130, 144, 146 respectively. These values coincided with the colour “bits” assignment in Table 1. The second transformed image (Fig. 15) was coded into 7 different levels of red, as described above but given the following values: 0, 1, 8, 9, 64, 65, 72, 73. The third transformed image (Fig. 16) was coded into 3 different levels of blue, 0, 4, 32, 36.
Individually the three colour coded transformed images with different intensities of the corresponding colour accentuate the variations in thermal behaviour of different parts of the hand, with the highest intensity corresponding to the maximum variation. Among them the green image conveys the most discriminatory information about the variation in the corresponding parts of the hand over the time series, as well as within the hand anatomy. The coding matched quite well with the diagnosis of the physician and could be used as a general guidance diagnostic image. In order to make the most of the
5.3 Infra-Red Themography: Techniques and Applications 915
Fig. 17. Composite Transformed Image.
three transformed images, however, a composite transformed diagnostic image was developed (Fig. 17). This is created by summing the three colour-coded Eigenimages together. From this composite image, the difference in thermal responses in the hand could be seen more clearly than in any of the three colour coded transformed images individually. The “hot spots” (in inflammatory conditions) and “cold spots” (in Raynaud’s condition) could be identified more easily than using the three individual transformed images. This composite transformed Eigen-image was found to match most closely the physician’s diagnosis. 6 . 5 . Classification
6.5.1. Classifier training and testing data The available data came from three diagnostic classes; inflammatory, normal and Raynaud’s. Some of the affected areas were localised, though mixed classes commonly occur in the same patient (for example only parts of the hand might be affected by a disease while the rest of the hand is normal). The training data therefore consisted of appropriate classes only and not all the pixels in a classified hand. This meant, for example, that only the inflamed areas of a hand were used in the inflammatory training set, and similarly, for the normal and Raynaud’s classes. The training data thus contained only representative data of its class and was not mixed with other classes. The selection of different class representative vectors was carried out by visual inspection of the diagnostic composite Eigen-images. Square or rectangular areas were located manually on displayed composite Eigen-image and the corresponding regions in the original series of thermograms were then extracted. These selected areas formed the training data. The optimal co-ordinate system were then obtained by applying the Kittler & Young analysis to this data. In order t o reduce the storage and computational requirement for the classification a Condensed Nearest Neighbour (CNN) classifier 1431 was used.
916
M. J. Varga €9 P. G. Ducksbury
6.5.2. Condensed nearest neighbour ( C N N ) classifier
In an ideal minimal subset, the 1-NN classification of any new pattern based on such a subset would be the same as the 1-NN classification with the complete set. Although it is simple to formulate such procedures, the computational complexity involved when the pattern space has a moderately large number of dimensions makes this impractical. As an alternative, Hart [43] described the CNN technique which provided a consistent subset of the original set for the NN rule, and could correctly classify all the remaining points in a sample set. However, the CNN rule will not in general find a minimum consistent subset that is consistent with a minimal number of elements. The CNN algorithm is defined as follows: Assume that the original set is arranged in some order, and two bins named STORE and GARBAGE are set. 0
0
0
0
Place first sample in STORE. Second sample is classified by NN rule, using the current store as the reference set. If the second sample is classified correctly it is placed in GARBAGE, otherwise into STORE. By induction, the ith sample is classified by the current contents of the STORE. If classified correctly it is placed in GARBAGE, otherwise in STORE. After one pass through the original sample set the procedure continues to go through GARBAGE until termination, which occurs when one of the below conditions is encountered. -
0
GARBAGE is exhausted, with all its members transfered to STORE or One complete pass is made through the GARBAGE with no transfer to STORE, as the underlying decision surface is stationary.
The final contents of STORE are used as reference points for the NN rule, the contents of GARBAGE are discarded.
When the Bayes risk is small, i.e. the underlying densities of the various classes have small overlap, the algorithm will tend to select points near the boundary between the classes. Therefore points deeply inside a class need not be transfered to STORE as they will be classified correctly. Consequently, when Bayes risk is high, the STORE will contain most of the points in the original sample set and no substantial reduction in sample would be possible. CNN tends to pick the initial STORE set randomly in the data space, and becomes more selective subsequently when the decision boundary becomes better defined. This results in vectors chosen later in the process tending to lie close to the decision boundaries. 6.5.3. Classification results
Three CNN classifiers were built based respectively on the Eigen images corresponding to the first 2, 3 and 4 Eigen vectors. These were tested using the 7-NN
5.3 Infra-Red Thennography: Techniques and Applications
917
classification technique with a Euclidean distance measure and a majority vote system. The testing samples were transformed using the Eigen-vector matrix derived from the training data set. It was found that the best performance was achieved when a three-dimensional CNN classifier was used giving an error rate of only 4.3% with respect to pixels as classified by a physician. It was found that the majority of errors came form the mis-classification of the Raynaud’s class as normal. This is possibly due t o the similarity in behaviour of a “cold hand complaint” and that of a mild Raynaud’s condition. It can be concluded from the above that only 3 dimensions are needed because the Kittler & Young method has succeeded in compressing most of the discriminatory information into the first three components of the transformed feature vectors, making the rest of the components in the transformed space redundant. The performance of the system compares favourably with a clinician’s diagnosis. Moreover, the majority of the errors came from the 4-3 tie condition of the 7-NN classifier where the correct category was in the 3-minor vote position. Hence, although they were mis-classified their correct classification could still be identified when presented in the colour representation technique developed in this study and described below. 6.6. Presentation of Classification Results for Diagnosis The resultant classified thermal features were used to produce diagnostic images; in these colour coded “classification pixels” were used to replace the corresponding pixels in the thermogram. The degree of confidence in the classification was denoted by the different intensity of the assigned disease class colour. For example, in the 4-3 tie condition of normal-Raynaud’s classification, then fourth degree intensity of green (normal) together with third degree intensity of blue (Raynaud’s) were summed together, giving a “greenish-blue” colour as the final “diagnosis colour” to
Fig. 18. A Diagnostic Image Of Normal Class.
918
M. J. Varga €9 P. G. Ducksbuy
Fig. 19. Classified Raynaud’s Hand.
Fig. 20. A Diagnostic Image of Inflammatory Class.
the pixel in question. In fact this “greenish-blue” colour was found to be associated closely to those patients with complaints of cold hands. While “yellowish” cases were found to be the patients with mild inflammatory conditions, i.e. when there was a tie between normal (green) and inflammatory (red). The resultant diagnostic images indicated the locations of the affected areas on the hands as well as the degree of “ t r u t h f u l n e s ~(severity ~~ and certainty) of classification by means of different colours and varying degrees of intensity Figs. 18-20. The diagnostic results compared extremely well with the physician’s diagnosis. The effect of the error rate was that the exact dimensions of the affected area might not be precisely defined, but the locations were identified. The classification errors were most likely to be at the boundary between two different classes on the hand.
5.3 Infra-Red Thermography: Techniques and Applications 919
7. Breast Cancer Detection Every year 1500 women in the UK alone die of breast cancer. Studies have shown that early detection is crucial t o survival in breast cancer, and it is believed that the use of appropriate image processing can make screening and diagnosis easier. Currently X-ray mammography is the most commonly used imaging technique [45]. Due to their invasive nature X-ray mammograms cannot be acquired regularly. An alternative complementary technique using non-invasive infra-red thermography together with low-level microwave radiation is described here.
7.1. System Overview This work is concerned with the analysis of normal and abnormal infra-red mammograms. The analysis is based on a series of infra-red mammograms of the breast subjected to a warming temperature stress. The temperature stress is induced using a very low level of microwave radiation. This produces a higher heating rate in tumourous tissue than in normal healthy tissue. The technique uses a sequence of mammograms taken during the cooling process. The image analysis and processing is basically the same as that described for the diagnosis of joint diseases in Sec. 6 above.
7 . 2 . Temperature Stress Technique and Data Collection It has been found that the static infra-red mammogram provides insufficient information for diagnosis due t o the limited transmission of IR radiation through fatty tissues. Therefore in this study a temperature stress was applied to induce a thermal response of the breasts t o aid diagnosis. Here the body is exposed to a
Fig. 21. Infra-red Mammograms.
920
M. J.
Varga €9 P. G. Ducksbury
micro-wave heating (frequency: 0.45 Ghz, power density: 80-100 m W/cm2) for two minutes. The irradiation penetration is typically 1.7 cm in muscle and 10 cm in fatty tissue, so the heating process can penetrate the breast fairly deeply. In order t o reduce excessive heating of the subcutaneous fat layer an active convective cooling of the skin is necessary. The use of this cooling means that the ambient temperature control is more relaxed in this application than in the previous study. After the stress there is a temperature transient in the breasts. It is the nature of this transient upon which the analysis is based. Thermograms are taken a t 30 second intervals for about 8 minutes in order to record this transient, see Fig. 21. Tumours appear as hot spots due to differences in the dielectric constants, vascularization, density and specific heat. The result of this process is that small and/or deep tumours can be detected. Furthermore, these infra-red mammograms are easier to interpret than X-ray mammograms; radiographers’ interpretation of X-ray mammograms is known to highly variable. Another advantage of this system is that it takes less time to record the series of thermograms (approximately 8 minutes) than the 30 minutes typically required to set up a single conventional thermogram (allowing time for
acclimatisation etc.). 7.3. Observed Thermal Characteristics of Breast During the course of data collection some thermal characteristics or behaviour patterns were observed. It is these patterns that the system must be capable of extracting, analysing and quantifying [44]. 0
0
0
0
Normal tissues - Temperature does not rise as high as the abnormal tissues. Temperature drops linearly with time. Veins - Temperature rises slightly higher than the abnormal tissues. Temperature drops more rapidly with time. Nipple -Temperature does not rise as high as the abnormal tissues. Temperature drops quickly with time. Tumours - Temperature is generally 0.3”C - 1.5”C higher than normal tissues. Temperature remains high but sometimes drops a little.
7.4. Elcperimental Results
Initial results from this work showed that in some severe cases the “hot spots” (tumours) can be identified using colour coded composite Eigen images from the Kittier & Young transformation. Also, that the vascular pattern can be identified to a certain extent , this can also help diagnosis because it indicates if micro-calcification exists. However, this approach can only be used as a supplementary diagnostic for breast cancer detection and the following-up treatment. At present X-ray mammography still must be used as the main diagnostic aid. It must be stressed that these are only preliminary results. Before such a technique can be applied in practice a far larger evaluation would be required, c.f. the 30 patients used in this study.
5.3 Infra-Red Thermography: Techniques and Applacatzom
921
8. Other Infra-Red Applications Infra-red technology is used in many application domains other than defence and medicine, for example: infra-red security system to detect intruders through their body heat and movement, fire fighting, rescue and driving aids [46].
8.1. Fire Fighting In fire fighting airborne imaging systems can often provide high-resolution data in a more timely fashion than space-based systems. Aircraft with on-board infrared and multi-spectral instruments are used, for example, in fighting forest fires to acquire images and information on the fire's perimeter, its hot spots and direction of travel. Information from this type of imagery can help fire fighters to suppress the fire significantly more efficiently than through the use of space-based imaging systems. This type of infra-red and multi-spectral imaging system can also be applied in law enforcement (e.g. m a ~ j u a n adetection) and forestry (e.g. identification of diseased tress within a forest). Often the collected images or video sequence acquired at the site are transmitted to a home base for use. Image processing, computer vision and pattern processing techniques are then used to analyse the images to develop strategies and action plans. 8 . 2 . Monitoring Radioactive Waste
One way of dealing with radioactive waste is to pour a mixture of molten glass and waste into canisters for storage. An infra-red detection system has been used to monitor the mixture level inside the canisters during filling and thus prevent spills. The system feeds live thermal video to a remote control room where operators can monitor the hot-glass level during the filling operation. The main advantage of using infra-red technology in this application is that it does not require radiationreference sources. This allows non-radioactive start-up testing of the process and facilitates safe worker entry into the vitrification cell before actual processing of the radioactive waste. 9. Future Infra-Red Sensor Technology Most current thermal imaging systems require complex cooling techniques with concomitant penalties in system size and weight and a significant logistic burden. Therefore, much research focuses on un-cooled, compact, low power and cheaper solutions. Ferroelectric detectors are perhaps the most promising devices for uncooled or ambient temperature IR imaging. Good performance can be achieved with large arrays where there is a single ferroelectric detector element for each image pixel. The performance can be further enhanced by reductions in detector noise bandwidth through advanced element and integrated circuit readout design. For the present however, cooled technology is still required for applications which need the highest performance.
922
M. J. Varga €4 P. G. Ducksbury
A major problem associated with current infra-red sensors is the non-uniformity inherent in the detector array. This requires correction prior to any subsequent image processing. At present correction is achieved using off-focal plane processing. However, the continuing advances in silicon integrated circuit technology will now allow more functionality, including non-uniformity correction, to be included within each pixel of the focal plane sensor array itself. Transferring this function onto the focal plane will result in a more cost effective solutions, giving improved performance and reliability together with reduce size and weight. In addition, the background pedestal current can be removed, on a pixel by pixel basis, resulting in systems benefits such as improved range and image quality. Moreover, there are opportunities to implement novel and advanced image processing and pattern recognition techniques on the focal plane array. This will result in new capabilities such as motion detection and clutter rejection which will significantly enhance the performance of IR systems.
Acknowledgements British Crown Copyright 1996/DERA. Published with the permission of Her Britannic Majesty’s Stationary office.
References [I] W. L. Wolfe and G. J. Zissis (eds.), The Infrared Handbook, revised edition, 3rd printing, The Infrared Information Analysis (IRIA) Center, Environmental Research Institute of Michigan, USA, 1989. [2] A. W. Vere, L. L. Taylor, M. Saker and B. Cockayne, Nonlinear Optical Material for the 3-5 p m Waveband - Available Performance and Technical problems, Technical report, unpublished, October 1994. Defence Research Agency, Farnborough, Hants GU14 6TD, UK. [3] L. F. Pau and M. Y. Ei Nahas, An introduction to infra-red image acquisition and clas sification (Research Studies Press Ltd., Letchworth, Hertfordshire, England, 1983). (41 R. A. Ballingall, I. D. Blenkinsop, I. M. Baker and Parsons, Practical design consideration in achieving high performance from infrared hybrid focal plane arrays, Proc. SPIE Vol. 819 Infrared Technology XIII (1987), 239-249. [5] A. J. Myatt, D. A. Spragg, R. A. Ballingall and I. D. Blenkinsop, Flexible electronic control and correction System for use with IR focal plane arrays, S P I E Vol. 891 Infrared Technology XIII (1987), 239-249. [6] P. W. Foulkes, Towards Infrared Image Understanding, Ph.D. Thesis, Engineering Department, Oxford University, 1991. [7] J. Pearl, Fusion, propagation, and structuring in Belief Networks, Artificial Intelligence 29 (1986) 241-288. [8] P. G. Ducksbury, Parallel texture region segmentation using a Pearl Bayes Network, British Machine Vision Conference University of Surrey (1993) 187-196. [9] P. A. Devijver, Real-time modeling of image sequences: based on hidden Markov mesh random field models in decision making in context, A Course on Statistical Pattern Recognition, by P. A. Devijver and J. Kittler (Surrey University, 1989). [lo] P. G. Ducksbury, Evidential Reasoning - A Review and Demonstration, UK DTI IED project 1936: Vision by Associative Reasoning, Report no. VAR-TR-RSRE92-4, July 1992.
5.3 Infra-Red Thermography: Techniques and Applications 923 [ll] J. M. Brady, P. G. Ducksbury and M. J. Varga, Image Content descriptors - Concepts, Unpublished technical report, DRA Malvern, UK, March 1996. [12] P. G. Ducksbury, Driveable region detection using a Pearl Bayes Network, IEE Colloquium on Image Processing for Transport Applications (London, Dec 1993). [13] P. G. Ducksbury, Image Content Descriptors: Feature Detection Stage, Unpublished technical report, DRA Malvern, UK, March 1996. [14] R. W. M. Smith, Conceptual Hierarchical Image Processor (CHIP): System Design, Issue 1.0, Unpublished technical report, DRA Malvern, UK, October 1992. [15] D. M. Booth and C. J. Radford, The Detection of Features of Interest in Surveillance Imagery: Techniques Evaluation, Unpublished technical report, DRA Malvern, UK, Nov 1992. [16] B. I. Justusson, Median filtering: statistical properties, in T. S. Huang (ed.), Two dimensional digital signal processing 11, Top. Appl. Phys. (Springer Verlag, Berlin, 1981) 161-196. [17] A. Rosenfeld and A. C. Kak, Digital Picture Processing, Vols. 1 and 2 (Academic Press, New York, 1982). [18] P. K. Sahoo, S. Soltani, A. K. C. Wong and Y. C. Chen, A survey of thresholding techniques, Computer Vision, Graphics and Image Processing 41 (1988) 233-260. [19] P. Maragos and R. W. Schafer, Morphological systems for multidimensional signal processing, Proc. IEEE, 78, 4 (April 1990). [20] Z. Hussain, Digital Image Processing: Practical Applications of Parallel Processing Techniques (Ellis Honvood, 1991). [21] R. M. Haralick, Mathematical morphology and computer vision, 22nd Asilamer Conf. Signals and Computers, Pacific Grove, CA, US, 31-0ct/2-Nov 88. [22] J. Serra and P. Soille (eds.), Computational Imaging and Vision, Mathematical Morphology and its Applications to Image Processing, Vol. 2 (Kluwer Academic, 1994). [23] F. K. Sun and S. L. Rubin, Algorithm development for autonomous image analysis based on mathematical morphology, Proc. SPIE 845: Visual Communications and Image Processing II(1987). [24] R. W. M. Smith and C. J. Radford, Development of a connected component labeller DSP module for CHIP, Unpublished technical Report, DRA Malvern, UK, Oct. 1993. [25] S. S. Blackman, Multiple Target Trucking with Radar Applications (Artech House, 1986). [26] G. Brown, R. W. M. Smith and C. J. Radford, Target Acquisition and Tracking of a Staring Array Sequence on CHIP, Unpublished technical Report, DRA Malvern, UK, Sep. 1993. [27] B. Kalman, New results in linear prediction and filtering, J. Basic Engin. 83-D (1961) 95-108. [28] D. J. Salmond, The Kalman Filter, The a - p filter and smoothing filters, Royal Aircraft Establishment, report TM-AW-48, February 1981. [29] A. W. Bridgewater, Analysis of 2nd and 3rd order steady state tracking filters, AGARD Conf. Proc. 252 (1978). [30] Yu. Gulyaev, V. Marov, L. G. Koreneva and P. V. Zakharov, Dynamic infrared thermography in humans, IEEE Engineering in Medicine and Biology (November/December 1995), 766-771. [31] J. E. Goin and J. D. Haberman, Automated breast cancer detection by thermography: performance goal and diagnostic feature identification. Pattern Recogn. 16, 2 (1983) 125-129.
924
M. J. Varga €4 P. G. Ducksbury
[32] M. J. Loh (nee’ Varga), Application of Statistical Pattern Recognition techniques to Analysis of Thermograms, 1986, Department of Community Medicine, Cambridge University, 1986. [33] R. N. Lawson, Implications of surface temperatures in the diagnosis of breast cancer. Can. Med. Assoc. J. 75 (1956) 309-301. [34] B. H. Phillips and K. Lloyd-Williams, The clinical use of thermography. Brit. J. Hosp. Med. Equip. Suppl. 1974. [35] L. M. Carter, The clinical role of thermography. J. Med. Eng. Tech. 2 (1978) 125. [36] M. V. Kyle, G. Pam, R. Sallisbury, P. Page-Thomas and B. L. Hazelman, Prostalgandin E l Vasospastic Disease and Thermography, presented at Heberden Round, 1982. [37] P. A. Bacon, A. J. Collins, F. J. Ring and J. A. Cosh, Thermography in the assessment of inflammatory arthritis, Clinical Rheumatology, Dis Vole. 2, M. I. V. Jayson (ed.) Philadelphia, W.B. Saunders and Co., 51-65. [38] C. Rajapakse, D. M. Grennan, C. Jones, L. Wilkinson and M. Jayson, Thermography in the assessment of peripheral joint inflammation - a re-evaluation, Rheumatology and Rehabilitation 20 (1981) 81-87. [39] M. J. Varga and R. Hanka, Dynamic elastic image stretching applied to thermographic images, Special Issues: IEE Proceedings-I, Communications, Speech €4 Vision (1990). [40] M. J. Varga and R. Hanka, Pseudo-colouring systems for thermographic images, 11th European Conf. Visual Perception, Bristol, UK, (1988). [41] P. Devijver and J. Kittler, Pattern Recognition: A Statistical Approach (Prentice-Hall International, 1982). [42] J. Kittler, Mathematical methods for feature selection in pattern recognition, Znt. J. Man. Manch. Stud. 7 (1975) 603-637. [43] P. E. Hart, The Condensed Nearest Neighbour Rule (CNN), ZEEE Trans. Information Theory 14 (1968) 515-516. [44] I. M. Ariel and J. B. Cleary, Breast Cancer Diagnosis and Treatment (McGraw-Hill Book Company, 1987). [45] M. J. Varga and P. De Muynck, Thermal analysis of infra-red mammography, 11th Int. Conf. Pattern Recogn., The Hague, The Netherlands, 1992. [46] R. Highman and J. M. Brady, Model-based image enhancement for infra-red images, IEEE Physics Based Modelling in Computer Vision Workshop, Boston, USA, 1995. [47] J. E. Shamblin and G. T. Stevens, Operations Research : A Fundamental Approach (McGraw-Hill, 1974).
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 925-944 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
1 CHAPTER 5.4 I VIEWER-CENTERED REPRESENTATIONS IN OBJECT RECOGNITION: A COMPUTATIONAL APPROACH
RONEN BASRI Department of Applied Mathematics, The Weizmann Institute of Science Rehovot 76100, Israel Visual object recognition is a process in which representations of objects are used to identify the objects in images. Recent psychophysical and physiological studies indicate that the visual system uses viewer-centered representations. In this chapter a recognition scheme that uses viewer-centered representations is presented. The scheme requires storicgonly a small number of views to represent an object. It is based on the observation that novel views of objects can be expressed as linear combinations of the stored views. This method is applied to rigid objects as well as to objects with more complicated structure, such as rigid objects with smooth surfaces and articulated objects.
Keywords: Alignment, linear combinations, 3-D object recognition, viewer-centered representations, visual object recognition.
1. Introduction Visual object recognition is a process in which images are compared to stored representations of objects. These representations, their content and use, determine the outcome of the recognition process. The features stored in an object’s model determine those properties that identify the abject and overshadow other properties. It is not surprising, therefore, that the issue of object representation has attracted considerable attention (reviews of different aspects of object representations can be found in [l-51). For many objects, shape (as opposed to other cues, such as color, texture, etc.) is their most identifiable property. In shape-based recognition a model contains properties of the object that distinguish it from objects with different shapes. The wide range of possible shape representations is divided into two distinct categories, object-centered representations and viewer-centered ones. Object-centered representations describe the shape of objects using view independent properties, while viewer-centered representations describe the way this shape is perceived from certain views. Recent psychophysical studies indicate that viewer-centered representations are used in a number of recognition paradigms (see details in Section 2). 925
926
R . Basri
Recognition of 3-D objects from 2-D images is difficult partly because objects look significantly different from different views. A common approach to recognition, which received the name alignment, aligns the object’s model t o the image before they actually are compared [5] (see also [6-81). We present a scheme that combines the use of viewer-centered representations with the alignment approach. The scheme, referred to as the “Linear Combinations” scheme (originally developed in [9]) represents an object by a small set of its views. Recognition is performed by comparing the image t o linear combinations of the model views. The scheme handles rigid objects as well as more complicated objects, such as rigid objects with smooth bounding surfaces and articulated objects.
2. Viewer-Centered Representations The issue of object representation is critical to recognition. It determines the information that makes a n object stand out and the circumstances under which it can be identified. In addition, it divides the computational process into its online components, the “recognition” part, and off-line components, the “learning” or “model acquisition” part. Object-centered representations describe the shape of objects using view independent properties. These representations usually include either view-invariant properties of the object (e.g. [10,11]) or structural descriptions defined within some intrinsic coordinate system, such as generalized cylinders [12,13], constructive solid modeling [14],and the vertices and edges in polyhedra [8,15]. Object-centered models in general are relatively concise. A single model is used t o recognize the object from all possible views. Viewer-centered representations describe the appearance of objects in certain views. Typically, a viewer-centered model consists of a set of one or more views of an object, possibly with certain 3-D shape attributes, such as depth or curvature (in a similar fashion to the 2;-D sketch suggested by Marr and Nishihara [13]). Often, a viewer-centered representation covers only a restricted range of views of an object. A number of models is then required to represent the object from all possible views. Viewer-centered representations are in general easier t o acquire, t o store, and t o handle than object-centered ones. For instance, with viewer-centered models there is no need to perform elaborate computations to account for self occlusion, since such occlusion is implicit in the model views. Recent psychophysical and physiological studies indicate that in certain recognition paradigms the visual system uses viewer-centered representations. A number of experiments establish that the response time in recognition tasks varies as a function of the angular distance of the object t o be recognized from either its upright position or a trained view. This effect, known as the mental rotation effect (originally shown in views comparison tasks by Shepard and Metzler [IS]),was found in naming tasks of both natural and artificially made objects [17-231. These effects considerably
5.4 Viewer-Centered Representations in Object Recognition 927 diminish with practice [18,20,23,24]. Namely, as subjects become more familiar with the objects, their response time becomes more and more uniform. Practicing the task on views of one object does not alter the performance for other objects [18,25],indicating that this is not a side effect resulting from the subjects’ learning to perform the experiment better, but that indeed subjects attain richer representations of the objects with practice. Findings by Tarr and Pinker [23,26,27]suggest that massive exposure to different orientations of objects does not necessarily result in the formation of object-centered representations. They showed cases where the response time was linear with the angular separation between the observed object and its closest view in the training set. Additional support to these findings was found in measuring the error rates in naming tasks. A few studies show that the number of incorrect namings increases with the angular separation between tested views and either trained views or the object’s upright position [19,28,29]. Edelman and Bulthoff [28] found that error rates increase not only as a function of distance of the tested view to the training set, but they also depend on the specific relation between the tested view and the trained views. In their experiment subjects were trained on two views of an object. It was found that intermediate views, views that lie within the range between the trained views, were correctly recognized more often than extrapolated views, that is, views that lie outside this range. Interestingly, they also found that, unlike response time, error rates do not diminish with practice [30], indicating that even after practice subjects did not attain complete view-invariant representations. Evidence consistent with the use of multiple viewer-centered descriptions was also found in single-cell activity recordings. Perret et al. [31] have investigated the response properties of face-sensitive cells in area STS of the macaque’s visual cortex. They have found that cells typically respond to a wide range of 3-D orientations, but not to all viewing directions. A face-selective cell that responds to e.g. a face-on view will typically not respond to a profile view, but will respond to a wide range of intermediate orientations. The authors concluded that “High level viewer-centered descriptions are an important stage in the analysis of faces” ([31] p. 314). It is important to remember that these experiments can be interpreted in more than a single way, and that the tested paradigms may not reflect the general recognition process. (See for example [32] where a case of dissociation of mental rotation from recognition is presented.) It seems, however, that a large number of experiments are consistent with the notion of viewer-centered representations.
3. Alignment
A major source of difficulty in object recognition arises from the fact that the images we see are two-dimensional, while the objects we try to recognize are threedimensional. As a result, we always see only one face of an object at a time. The images of the same object may differ significantly from one another even when these
928
R. Ban’
Fig. 1. Deformation of an image following a 1 5 O rotation of a car. An o\ierlaid picture of the car before and after rotation. Although the rotation is fairly small, the discrepancies between the two images are fairly large.
views are separated by a relatively small transformation (see for example Fig. 1). Cluttered scenes introduce additional complexity due to partial occlusion. One approach t o overcome these difficulties is t o first recover the underlying 3-D shape of the observed object from the image (using cues like shading, stereopsis, and motion) and then to compare the result with the 3-D model (e.g. [13,33]) Although in recent years there has been tremendous progress in understanding early visual processes, current shape recovery algorithms still seem to be limited in their ability to provide accurate and reliable depth information. Moreover, people’s ability to recognize objects seem t o be fairly robust t o elimination of depth cues (e.g. [30,34]). The ability t o recognize objects from line drawings, which contain only sparse information about the shape of objects, demonstrates that shape recovery may not be essential for recognition. The alignment approach avoids recovering the underlying 3-D shape of the observed object by comparing the object’s model t o the image in 2-D (“template matching”). To account for orientation differences between the stored model and the observed image these differences are compensated for before the model and the image are compared. The transformation that compensates for these differences is called “the alignment transformation.” Alignment is therefore a two-stage process. First, the position and orientation (pose) of the observed object is recovered, and then the model is transformed to this pose, projected t o the image plane, and compared with the actual image. A large number of studies use alignment-like algorithms to recognize 3-D objects from 2-D images [5-8,35-371. These studies vary in the representations used and the method employed t o recover the alignment transformation. Most of these studies use object-centered representations. When viewer-centered representations are used, the naive approach usually is taken; namely, the system can recognize only the stored views of an object (e.g. [37-391). For example, in [37] an object is modeled by a large number of views (the representation includes a table of 72 x 72 = 5184 views). A view is recognized only if the image is related t o one of these views by a rotation in the image plane, in which case this view and the image share the same appearance.
5.4 Viewer-Centered Representations in Object Recognition 929 In the rest of this chapter we present an alternative to these approaches: an alignment scheme that recognizes objects using viewer-centered representations. The method requires only a small number of views to represent an object from all its possible views. 4. The Linear Combinations (LC) Scheme
The variability and richness of the visual input is overwhelming. An object can give rise to a tremendous number of views. It is not uncommon for humans to forget familiar views, perhaps because the visual system is incapable of storing and retrieving such huge amounts of information. Consequently, the visual system occasionally comes across novel views of familiar objects, whether these views have been forgotten, or they are entirely new. The role of the recognition process when a novel view is observed is to deduce the information that is necessary to recognize the object from its previously observed images. This relationship between the novel and the familiar views of objects is (implicitly) specified by the representation used by the recognition system. The linear combinations (LC) scheme relates familiar views and novel views of objects in a simple way. Novel views in this scheme are expressed by linear combinations of the familar views. This property can be used to develop a recognition system that uses viewer-centered representations: an object is modeled in this scheme by a small set of its familiar views. Recognition involves comparing the novel views to linear combinations of the model views. For such a representation to be feasible, the correspondence between the model views should first be resolved. Correspondence between views of objects is a source for understanding how the objects change between views. This information allows the system to track the location of feature points in the model images and predict their location in novel views of the object. A view in the LC scheme is represented by the locations of feature points (such as corners or contour points) in the image. A model is a set of views with correspondence between the points. As already mentioned, novel views are expressed by linear combinations of the model views. When opaque objects are considered, due to self occlusioli, different faces (“aspects”) of the object appear in different views. A number of models (not necessarily independent) would then be required to predict the appearance of such objects from all possible viewpoints. The LC method applies to rigid objects as well as to more complicated objects, such as objects that undergo affine transformations, rigid objects with smooth bounding surfaces, and articulated objects. In this section we describe the main properties of the LC scheme. A more thorough presentation can be found in [9]. 4.1. Rigid Objects
In this section we show that for rigid objects novel views can be expressed as linear combinations of a small number of views. We begin with the following
930 R . Basri
definitions. Given an image I with feature points, pl = (q,y l ) , . . . ,p, = (z,, y,), a view fi is a pair of vectors x, y E R" where x = ( ~ 1 , .. . ,z,)~and y = (yl, . . . ,Y,)~ contain the location of the feature points, P I , . . . , p , , in the image. A model is a set of views {Vl, . . . , Vb}. The location vectors in these views are ordered in correspondence, namely, the first point in Vl is the projection of the same physical point on the object as the first point in V2, and so forth. The objects we consider undergo rigid transformations, namely, rotations and translations in space. We assume that the images are obtained by weak perspective projection, that is, orthographic projection together with uniform scaling. The proof of the linear combinations property proceeds in the following way. First, we show (Theorem 1) that the set of views of a rigid object is contained in a four-dimensional linear space. Any four linearly independent vectors from this space can therefore be used to span the space. Consequently, we show (Theorem 2) that two views suffice to represent the space. Any other view of the object can be expressed as (two) linear combinations of the two basis views. Next, we show (Theorem 3) that not every point in this 4-D space necessarily corresponds to a legal view of the object. The coefficients satisfy two quadratic constraints. These constraints depend on the transformation between the model views. A third view can be used to derive the constraints.
Theorem 1. The views of a rigid object are contained in a four-dimensional linear space. Proof. Consider an object 0 with feature points pl = ( 2 1 , y1, z l ) , . . . , p , = (z,, y, 2,). Let I be an image of 0 obtained by a rotation R, translation t , and scaling s, followed by an orthographic projection II. Let q1 = (xi,y:), . . . ,qn = (z;, y;) be the projected location in I of the points p l , . . . , p , respectively. For every l < i l n qi = sII(Rpi) t .
+
More explicitly, this equation can be written as
where { r i j } are the components of the rotation matrix, and t,, t , are the horizontal and the vertical components of the translation vector. Since these equations hold for every 1 5 i 5 n, we can rewrite them in vector notation. Denote x = ( X I , . . . , z,)~, y = (y1, ..., Y,)~, z = (21 ,..., z , ) ~ , 1 = (1,..., l)T,x' = (zi,...,z;)~,and y' = (yi, . . .,Y;)~, we obtain that
+ u2y + u3z + a41 y' = blx + b2y + b 3 + ~ b41 x' = a l x
5.4
Viewer-Centered Representations in Object Recognition 931
where a1
= sql
bl
= sr21
a2 = sr12
b2 = sr22
a3 = ~
1b3 3= ~
a4
= t,
~
b4
~
2
3
= t,
The vectors x' and y' can therefore be expressed as linear combinations of four vectors, x, y, z, and 1. Notice that changing the view would result merely in a change in the coefficients. We can therefore conclude that
x', Y' E spanix, Y, z, 1) for any view of 0. Notice that if translation is omitted the views space is reduced to a three-dimensional one. 0
Theorem 2. The views space of a rigid object 0 can be constructed from two views of 0." Proof. Theorem 1 above establishes that the views space of a rigid object is fourdimensional. Any four linearly independent vectors in this space can be used to span the space. The constant vector, 1, belongs to this space. Therefore, only three more vectors remain to be found. An image supplies two vectors. Two images supply four, which already is more than enough to span the space (assuming the two images are related by some rotation in depth, otherwise they are linearly dependent). Let V1 = ( x 1 , y l ) and Vz = (x2,y2) be two views of 0, a novel view V' = (x',y') of 0 can be expressed as two linear combinations of the four vectors X I , y1, x2, and 1. The remaining vector, y2, already depends on the other four vectors. 0 Up to this point we have shown that the views space of a rigid object is contained in a four-dimensionallinear space. Theorem 3 below establishes that not every point in this space corresponds to a legal view of the object. The coefficients of the linear combination satisfy two quadratic constraints.
Theorem 3. The coefficients satisfy two quadratic constraints, which can be derived from three images. Proof. Consider the coefficients a l , . . . ,a4, bl, . . . ,b4 from Theorem 1. Since R is a rotation matrix, its row vectors are orthonormal, and therefore the following equations hold for the coefficients. a:
+ a; + a: = b: + bz + b: albl + a2bz + a& = 0 .
Choosing a different basis to represent the object (as we did in Theorem 2) will change the constraints. The constraints depend on the transformation that separates the model views. Denote by a l l . .. , c q , PI,.. . ,P4 the coefficients that "This lower bound was independently noticed by Poggio 1401.
932
R. Basri
represent a novel view with respect to the basis described in Theorem 2, namely
and denote by U the rotation matrix that separates the two model views. By substituting the new coefficients we obtain new constraints
alp1
+ a 2 P 2 + a 3 P 3 + (alp3 + Q3P1)1L11 + ( a 2 P 3 + a 3 0 2 ) u 1 2 = 0.
To derive the constraints the values of u11 and u12 should be recovered. A third view can be used for this purpose. When a third view of the object is given, the constraints supply two linear equations in ull and u12, and, therefore, in general, the values of u11 and u12 can be recovered from the two constraints. This proof suggests a simple, essentially linear structure from motion algorithm that resembles the method used in [41,42],but the details will not be discussed further here. 0 The scheme is therefore the following. An object is modeled by a set of views, with correspondence between the views, together with the two constraints. When a novel view of the object is observed the system computes the linear combination that aligns the model to the object. The object is recognized if such a combiriation is found and if in addition the constraints are verified. Figure 2 shows the application of the linear combination scheme to an artificially made object.
Fig. 2. Application of the linear combinations scheme to a model of a pyramid. Top: two model pictures of a pyramid. Bottom: two of their linear combinations.
For transparent objects a single model is sufficient to predict their appearance from all possible viewpoints. For opaque objects, due to self occlusion, a number of models is required to represent the objects from all aspects. These models are not
5.4 Viewer-Centered Representations in Object Recognition 933 necessarily independent. For example, in the case of a convex object as few as four images are sufficient to represent the object from all possible viewpoints. A pair of images, one from the “front” and another one from the “back” contains each object point once. Two such pairs contain two appearances of all object points, which is what is required to obtain a complete representation of all object points. Note that positive values of the coefficients (“convex combinations”) correspond to interpolation between the model views, while extrapolation is obtained by assigning one or more of the coefficients with negative values. This distinction between intermediate views and other views is important, since if two views of the object come from the same aspect then intermediate views are likely to also come from that aspect, while in other views other aspects of the objects may be observed. 4.2. Additional Views
In the previous section we have shown that two views of a rigid object are sufficient to represent an object from all possible viewpoints. All other views are linear combinations of the two views. In practice, however, because of noise and occlusion one may seek to use additional views to improve the accuracy of the model. In this section we present a method to build models from more than two views. The idea is as follows. Each view provides two vectors, one for the x-coordinate and the other for the y-coordinate. These vectors can be viewed as points in Rn. The space of views of the object is known to be four-dimensional. The objective, then, is to find the four-dimensional subspace of Rn that best approximates the input views. This subspace can be found using singular value decomposition. More formally, given 1 vectors, v1,.. . ,vl,we denote F = [vl,.. . ,vl]; F is an n x 1 matrix. The best k-dimensional space through these vectors (in a leastsquared sense) is spanned by the k eigenvectors of FFt that corresponds to its k largest eigenvalues. (This is shown in [9], Appendix B.) This method resembles the algorithm used by Tomasi and Kanade [43]to track features in motion sequences, with the exception that in our case the motion parameters do not need to be recovered since we are only interested in finding the linear space from which these views are depicted. A method that approximates the space of views of an object from a number of its views using Radial Basis Functions [44]was recently suggested [45].Similar to the LC method, the system represents an object by a set of its familiar views with the correspondence between the views. The number of views used for this approximation, between 10 to 100, is much larger than the number required under the linear combinations scheme. The system, however, can also approximate perspective views of the objects. 4.3. A B n e Objects
In this section we extend the LC scheme to objects that undergo general affine transformations in space. In addition to the rigid transformations a€Eine
934
R. Basri
transformations include stretching and shearing. They are important since tilted pictures of objects appear to be stretched [46]. This effect is known as the La Gournerie Paradox (see [47]). In order to extend the LC method to include affine transformations the same scheme can be used, but with the quadratic constraints ignored. Namely, the fourdimensional linear space contains all and only the affine views of the object. Two views are therefore sufficient to span the space with no further constraints. 4.4. Rigid Objects with Smooth Surfaces
In this section we extend the LC scheme to rigid objects with smooth bounding surfaces. These objects are considerably more difficult to recognize from their contour images than are objects with sharp edges (such as polyhedral objects). When objects with sharp edges are considered, the contours are always generated by those edges. With objects with smooth bounding surfaces, however, the silhouette (the boundary of the object) does not correspond to any particular edges on the object. That is, the rim (the set of object points that generates the contours) changes its position on the object with viewpoint, and its location therefore is difficult to predict (see Fig. 3).
V
‘d
Fig. 3. The change of the rim of an object with smooth bounding surface due t o rotation. Left: a horizontal section of an ellipsoid. p is a point on the rim. Right: the section rotated. p is no longer on the rim. Instead p’ is the new rim point. The method described in Section 4.4 approximates the position of p’ using the curvature circle at p . (See [48]for details.)
The position change of the rim depends largely on the 3-D curvature at the rim points. When this curvature is high the position change is relatively small. (In the case of a sharp edge, the curvature is infinite and the position change vanishes.) When the curvature is low the position change is relatively large. Following this observation a method to approximate the position change of the rim using the surface curvature was developed [48]. In the original implementation a model contained a single contour image of the object. Each point along the contour was associated with its depth coordinate and its radial curvature (the curvature
5.4 Viewer-Centered Representations in Object Recognition 935 at the section defined by the surface normal and the line of sight). It was shown that a small number of images (at least three) is sufficient to recover this curvature. Using this information the system could approximate the appearance of objects with smooth bounding surfaces for relatively large transformations. In a later paper Ullman and Basri [9] showed that this approximation method is linear in the model views. They concluded that objects with smooth bounding surfaces can be represented by linear combinations of their familiar views. The space of views in this case is six-dimensional (rather than four), and at least three views (rather than two) are required to span the space. Additional quadratic constraints apply to the coefficients of the linear combinations. It should be noted that in order to handle objects with smooth bounding surfaces the definition of correspondence should be modified since contour points no longer represent the same physical points on the object from all views. Under the modified version, silhouette points in one image are matched to silhouette points in the second image that lie along the epipolar line. Ambiguities are resolved in a straightforward manner. Note also that advance knowledge of the type of the object, whether it has sharp edges or smooth bounding surfaces, is not required. The views of a curved object span a larger space than the views of a polyhedral object. Thus, singular value decomposition can be used to distinguish between the two (see Section 4.2). Figure 4 shows the application of the method to real edge images of a car. It can be seen that the predictions obtained are fairly accurate even though the bounding contours are smooth. 4.5. Articulated Objects
An articulated object is a collection of links connected by joints. Each link is a rigid component. It can move independently of the other links when only its joints constrain its motion. The space of views of an articulated object with 1 links is at most (4 x 1)-dimensional. The joints contribute additional constraints, some of which may be linear, and they reduce the rank of the space, others are non-linear, in which case they are treated in the same way the quadratic constraints are treated in the rigid case. Consider, for example, an object composed of two links connected by a rotational joint (e.g. a pair of scissors). The views space of a two-link object is at most eightdimensional (four for each of the links). The rotational joint constrains the two links by forcing them to share a common axis. Denote by p and q two points along this axis, and denote by Ti and T2 the rigid transformations applied to the first and second links respectively, then the following two constraints hold:
936
R. Basri
Fig. 4. Application of the linear combination scheme to a ‘W car. Top: three model pictures o the car. Middle: matching the model t o a picture of the VW car. A linear combination of the three model images (left), an actual edge image (middle), and the two images overlaid (right). The prediction image and the actual one align almost perfectly. Bottom: matching the VW model to an image of another car. A linear combination of the three model images (left), an actual image of a Saab car (middle), and the two images overlaid (right). In this case, although the coefficients of the linear combination were chosen such that the prediction would match the actual image as much as possible, the obtained match is relatively poor.
These two constraints are linear, and therefore they reduce the dimension of the space from eight to six. In addition, there is one quadratic constraint that implies the two links are scaled by the same amount. To summarize, the space of views of an articulated object that is composed of two links connected by a rotational joint is contained in a six-dimensional linear space. Five additional quadratic constraints (two follow the rigidity of each of the two links and one follows the common scaling) apply to the coefficients. As in the case of objects with smooth bounding surfaces, advance knowledge of the number of links and the type of the joints is not required. When sufficiently many views are presented, the correct rank of the views space can be recovered using singular value decomposition. Figure 5 shows the application of the linear combinations scheme to a pair of scissors. The images in this figure were obtained by different rigid transformations as well as articulations. It can be seen that the predictions match the real images also in the presence of articulations.
5. Recognition Using the LC Scheme In the previous section we have presented a viewer-centered representation for object recognition. An object is modeled in this scheme by a small set of its views with the correspondence between the views. Novel views of the object are expressed
5.4 Viewer-Centered Representations in Object Recognition 937 ..............
I
---.
.
.
- - .......... -
........
.
.
...
.
- ........
.
I --- .. Fig. 5. Application of the linear combination method to a pair of scissors. Left: two linear combinations of the model views, Middle: actual edge images, Right: overlay of the predictions with the real images.
by linear combinations of the model views. In addition, the coefficients of these linear combinations may follow certain functional constraints. In this section we discuss how this representation can be used in a recognition system. The task assigned t o the recognition system is t o determine, given an incoming image, whether the image belongs to the space of views of a particular model. In this section we discuss two principal methods t o reach such a decision. The first involves alignment of the model to the image by explicitly recovering the coefficients of the linear combination, and the second involves the application of a “recognition operator”.
5.1. Recovering the Alignment Coeficients The alignment approach to object recognition identifies objects by first recovering the transformation that aligns the model with the incoming image, and then verifying that the transformed model matches the image. In the LC scheme, the observed image is expressed by linear combinations of the model views. The task is therefore to recover the coefficients of these linear combinations. In other words, given a view v’ and a model {vl, .... vk} we geek a set of coefficients for which
holds. (In practice, t o overcome noise, we may seek t o minimize the difference between the two sides of this equation.) To determine the coefficients that align a model to the image, either one of the two following methods can be employed. The first method involves recovering the correspondence between the model and the image, and the second method involves a search in the space of possible coefficients. In the first method correspondence is
938 R. Basri
established between sufficiently many points so as to recover the coefficients. For a model that contains k views, at least k correspondences are required to solve a system of 2k linear equations (k equations for recovering the coefficients for the x-values, and another k equations for recovering the coefficients for the y-values). In this way, for example, four correspondences between model and image points are required to recover the coefficients for a rigid object by solving a linear system. If in addition we consider the quadratic constraints, this number is reduced to three. This is similar to the three-point alignment suggested by Huttenlocher and Ullman (5,7]. Applications of this method usually try to match triplets of model points to all combinations of triplets of image points to guarantee recognition. An alternative approach to determine the coefficients involves a search in the space of possible coefficients. This method does not require correspondence between the model and the image. The idea is the following. Using global properties of the observed object, such as axes of elongation, an initial guess for the values of the coefficients can be made. This initial guess can be then improved by an iterative process. At every step in this process a new set of coefficients is generated. The model is transformed using these coefficients, and the result is compared to the actual image. If the two match, the observed object is recognized, otherwise the process is repeated until it converges. Minimization techniques such as gradient descent may be employed to reduce the complexity of the search. Such techniques, however, involve the risk of converging into a local minimum, which occasionally may be significantly worse than the desired solution. It is interesting to note that the phenomenon of mental rotation seems to be consistent with the idea of search. The evidence for mental rotation suggests that recognition is not attained in an instance, but rather the response time increases with the angular separation between the observed object and its stored representation (see the discussion in Section 2). 5 . 2 . Recognition Operator
A second approach to identify novel views of objects involves the application of “recognition operators” to these views. Such operators are essentially invariants for a given space of views, that is, they return a constant value for all views of the object, and different values for views of other objects. This method does not require the explicit recovery of the alignment coefficients. Still, it does require correspondence between the model and the image. In the LC scheme a view is treated as a point in R”. A view contains the appearance of an object if it belongs to the space of views spanned by the object’s model. A natural way to identify the object would be to determine how far apart the incoming view is from the views space of the object. The result of such a test would be zero if and only if the given view is a possible view of the object. By projecting the given view to the views space of the object we can generate a distance metric between the model and the view to be recognized.
5.4 Viewer-Centered Representations in Object Recognition 939 Let v1, . . . ,vk be the model views. Denote M = [vl, . . . , vk], M is a k x n matrix. Theorem 5 below defines a recognition operator L. L measures the distance of a view v' from the linear space spanned by the model views, v1,. . . , vk, and ignores the nonlinear constraints.
Theorem 4. Let
L=I-MM+ where M+ = (MTM)-lMT denotes the pseudo inverse of M . Then Lv' = 0 if and only if v' is a linear combination of v1, . . . ,Vk.
Proof. Lv' = 0 if and only if v' = MM+v'. M M + is a projection operator; it projects the vector v' onto the column space of M . Therefore, the equality holds if and only if v' belongs to the column space of M , in which case it can be expressed by a linear combination of v1,. . . ,vk. The matrix L is therefore invariant for all views of the object; it maps all its views to zero. 0 Note that L only considers the linear envelope of the views space of the object. It does not verify any of the quadratic constraints. To verify in addition the quadratic constraints a quadratic invariant can be constructed. Weinshall [49] has recently presented a quadratic invariant for four points. This invariant can be modified to handle more points in a straightforward manner, but the details will not be discussed here. The recognition operator can be made associative. The idea is the following. Suppose L is a linear operator that maps all model views to the same single vector, that is, q = Lvl = . . . = Lvk. Since L is linear it maps combinations of the model to the same vector (up to a scale factor). Let v' be a novel view of the object, v' = C:=, aivi, then
q serves as a name for the model, and it can be either zero (in which case we obtain an operator that is identical to the operator in Theorem 4 above) or it can be a familiar view of the object (e.g. vl). Note, however, that a special care should be given to the case that ai vanishes. A constructive definition of the associative operator is given below. Let (v1,. . . ,vn} be a basis for Rn such that the first k vectors are composed of the model views. Denote
940
R. Basri
(We filled the matrix Q with the vectors vk+l l . . . v, so that the operator L would preserve the magnitude of noise if such is added t o the novel view. These vectors can be replaced by any vectors that are linearly independent of 9.) We require that
LP=Q. Therefore
L = QP-’ (Notice that since P is a basis for R” its inverse exists.) We have implemented the associative version of the recognition operator and applied it to the pyramid from Fig. 2. The results are given in Fig. 6. It can be seen that when this operator is applied to a novel view of the pyramid it returns a familiar view of the pyramid, and when it is applied to some other object it returns a n unknown view.
Fig. 6. Top: applying an associative “pyramidal” operator to a pyramid (left) returns a model view of the pyramid (right, compare with Fig. 2, top left). Bottom: applying the same operator to a cube (left) returns an unfamiliar image (right).
Both versions of the recognition operator can be implemented by a simple neural network with one layer [50]. The network contains only input and output layers with no hidden units. The weights are set t o be the components of L (see Fig. 7 ) . The network operates on novel views of some object and returns either zero or a familiar view of the object, according t o the operator it implements. It should be noted that for such an operator to be applicable the correspondence between the image and model must first be resolved. 6. Summary
Visual object recognition is a process in which images are compared to stored representations of objects. While recent psychophysical and physiological studies indicate that the visual system uses viewer-centered representations, most computational approaches t o recognition use object-centered representations. The few existing methods that use viewer-centered representations require a large number of views to represent an object from all possible viewpoints.
5.4 Viewer-Centered Representations in Object Recognition 941
Fig. 7. A neural network architecture that implements the recognition operator L. The input to this net,work is composed of the elements of the novel view v’, and the output is the “name” vector q (up t > a scale factor a).
A scheme was presented in which objects are recognized using viewer-centered models. The scheme is based on the observation that the novel views of an object can be expressed as linear combinations of a small set of its familiar views. An object is modeled by a set of views with correspondence between the views and possibly with some functional constraints. A novel view is recognized if there exists a linear combination of the model views that aligns the model to the image, and if the coefficients of this combination satisfy the functional constraints. ‘The method was applied to rigid objects as well as to objects that undergo affine transformations, rigid objects with smooth bounding surfaces (in which case the method only approximates the appearance of these objects), and articulated objects. The number of views required to represent an object depends on the shape of the object, whether it has sharp edges or smooth surfaces, in the case of a rigid object, and on the type of joints that connect the links in the case of an articulated one. This number can be deduced from the set of views of the object. To recover the alignment coefficients, a small number of points in the image and their corresponding points in the model can be used, or a search can be conducted in the space of possible coefficients. Alternatively, if the complete correspondence between the model and the image can be recovered, a “recognition operator” can be applied to the image. This operator obtains as its input a novel view of the object and returns a constant value, either the zero vector or a familiar view of the object. Furthermore, the operator can be implemented in a neural network with simple structure. Finding the correspondence between the model and the image is the difficult problem in recognition. The phenomenon of apparent motion, however, demonstrates that the visual system can successfully solve the correspondence problem.
942
R. Basri
Acknowledgements I wish to thank Shimon Ullman without whom this work would not have been possible, and to T . D. Alter, S. Edelman, W. E. L. Grimson, T . Poggio, and A. Yuille for helpful comments at different stages of this work. This report describes research done at the Weizmann Institute of Science and at the Massachusetts Institute of Technology within the Artificial Intelligence Laboratory and the McDonnell-Pew Center for Cognitive Neuroscience. Support for the laboratory’s artificial intelligence research is provided in part by the Advanced Research Projects Agency of the Department of Defense under Office of Naval Research contract N00014-915-4038. Ronen Basri is supported by the McDonnell-Pew and the Rothchild postdoctoral fellowships.
References [l] I. Biederman, Recognition by components: a theory of human image understanding, Psychol. Rev. 94 (1987) 115-147. [2] R. T . Chin and C. R. Dyer, Model-based recognition in robot vision, Comput. Sum. 18, 1 (1986) 67-108. [3] P. Jolicoeur, Identification of disoriented objects: A dual-systems theory, Mind and Language 5 (1990) 387-410. (41 S. E. Palmer, Fundamental aspects of cognitive representation, in E. Rosch and B. B. Lloyd (eds.), Cognition and Categorization (Lawrence Erlbaum, Hillsdale, NJ, 1978) 259-303. [5] S. Ullman, Aligning pictorial descriptions: An approach to object recognition, Cognition 32,3 (1989) 193-254. [6] M. A. Fischler and R. C. Bolles, Random sample consensus: A paradigm for model fitting with application to image analysis and automated cartography, Commun. ACM 24,6 (1981) 381-395. [7] D. P. Huttenlocher and S. Ullman, Object recognition using alignment, in Proc. Int. Conf. on Computer Vision (ICCV), London, UK, 1987, 102-111. [8] D. G. Lowe, Perceptual Organization and Visual Recognition (Kluwer Academic Publishers, Boston, MA, 1986). [9] S. Ullman and R. Basri, Recognition by linear combinations of models, IEEE Trans. Pattern Anal. Mach. Intell. 13,10 (1991) 992-1006. (lo] R. C. Bolles and R. A. Cain, Recognizing and locating partially visible objects: The local feature focus method. Int. J. Robot. Res. 1, 3 (1982) 57-82. [ll] M. K. Hu, Visual pattern recognition by moment invariants, IRE Trans. Inf. Theory 8 (1962) 169-187. [12] T. 0. Binford, Visual perception by computer, in Proc. IEEE Conf. on Systems and Control, Miami, FL, 1971. [13] D. Marr and H. K. Nishihara, Representation and recognition of the spatial organization of three dimensional shapes, Proc. Royal Society B200 (1978) 269-291. [14] A. Requicha and H. Voelcker, Constructive solid geometry, Production Automation Project Tm-26, University of Rochester, NY, 1977. [15] L. G . Roberts, Machine perception of three-dimensional solids, in J. T. Tippett et al. (eds.), Optical and Electro-Optical Information Processing (MIT Press, Cambridge, MA, 1965). [16] R. N. Shepard and J. Metzler, Mental rotation of three dimensional objects, Science 171 (1971) 701-703.
5.4 Viewer-Centered Representations in Object Recognition 943 [17] L. A. Cooper, Demonstration of a mental analog to an external rotation, Perception and Psychophysics 1 (1976) 20-43. [18] P. Jolicoeur, The time to name disoriented natural objects, Memory and Cognition 13, 4 (1985) 289-303. [19] P. Jolicoeur and M. J. Landau, Effects of orientation on the identification of simple visual patterns, Canadian J. Psychol. 38, 1 (1984) 80-93. [20] R. Maki, Naming and locating the tops of rotated pictures, Canadian J. Psychol. 40 (1986) 368-387. [21] R. N. Shepard and J. Metzler, Mental rotation: effects of dimensionality of objects and type of task, J. Exper. Psychol.: Human Perception and Performance 14,1(1988) 3-11. [22] S . P. Shwartz, The perception of disoriented complex objects, in Proc. 3rd Conf. on Cognitive Sciences, Berkeley, CA, 1981, 181-183. [23] M. J. Tarr and S. Pinker, Mental rotation and orientation-dependence in shape recognition, Cognitive Psychology 21 (1989) 233-282. [24] M. C. Corballis, Recognition of disoriented shapes, Psychol. Rev. 95 (1988) 115-123. (251 P. Jolicoeur and B. Milliken, Identification of disoriented objects: Effects of context of prior representation, J. of Exper. Psychol.: Learning, Memory, and Cognition 1 5 (1989) 200-210. [26] M. J. Tarr and S. Pinker, When does human object recognition use a viewer-centered reference frame? Psychol. Sci. 1 (1990) 253-256. [27] M. J. Tarr, Orientation Dependence in Three-Dimensional Object Recognition, Ph.D. thesis, Massachusetts Institute of Technology, 1989. [28] S. Edelman and H. H. Bulthoff, Viewpoint-specific representations in threedimensional object recognition, Technical Report A. I. Memo 1239, The Artificial Intelligence Lab., M.I.T., 1990. (291 I. Rock and J. DiVita, A case of viewer-centered object perception, Cognitive Psychology 19 (1987) 280-293. [30] S. Edelman and H. H. Bulthoff, Orientation dependence in the recognition of familiar and novel views of 3d objects, Vision Research 32 (1992) 2385-2400. [31] D. I. Perret, P. A. J. Smith, D. D. Potter, A. J. Mistlin, A. S. Head, A. D. Milner, and M. A. Jeeves, Visual cells in the temporal cortex sensitive to face view and gaze direction, Proc. Royal Society B223 (1985) 293-317. [32] M. J. Farah and K. M. Hammond, Mental rotation and orientation-invariant object recognition: Dissociable processes, Cognition 29 (1988) 29-46. [33] R. J. Douglass, Interpreting three dimensional scenes: A model building approach, Comput. Graph. Image Process. 1 7 (1981) 91-113. (341 J. E. Hochberg and V. Brooks, Pictorial recognition as an unlearned ability: A study of one child’s performance, Am. J. Psychol. 75 (1962) 624-628. [35] C. H. Chien and J. K. Aggarwal, Shape recognition from single silhouette, in Proc. Int. Conf. on Computer Vision (ICCV), London, UK, 1987, 481-490. [36] 0. D. Faugeras and M. Hebert, The representation, recognition and location of 3-D objects, Int. J. Robot. Res. 5, 3 (1986) 27-52. (371 D. W. Thompson and J. L. Mundy, Three dimensional model matching from an unconstrained viewpoint, in Proc. IEEE Int. Conf. on Robotics and Automation, Raleigh, NC, 1987, 208-220. [38] Y. S. Abu-Mostafa and D. Pslatis, Optical neural computing, Sci. Am. 256 (1987) 66-73. [39] P. Van Hove, Model based silhouette recognition, in Proc. IEEE Computer Society Workshop on Computer Vision, 1987.
944
R. Basri
1401 T. Poggio, 3D object recognition: On a result by Basri and Ullman, Technical Report TR 9005-03, IRST, Povo, Italy, 1990. [41] S. Ullman, The Interpretation of Visual Motion, (MIT Press, Cambridge, MA, 1979). [42] T. S. Huang and C. H. Lee, Motion and structure from orthographic projections, IEEE Trans. Pattern Anal. Mach. Intell. 2, 5 (1989) 536-540. [43] C. Tomasi and T. Kanade, Factoring image sequences into shape and motion, in Proc. IEEE Workshop on Visual Motion, Princeton, NJ, 1991, 21-29. [44] T. Poggio and F. Girosi, Regularization algorithms for learning that are equivalent to multilayer aetworks, Science 247 (1990) 978-982. [45] T. Poggio and S. Edelman, A network that learns to recognize three-dimensional objects, Nature 343 (1990) 263-266. [46] D. W. Jacobs, Space efficient 3D model indexing, in Proc. C V P R Conference, Urbana, IL, 1992. (471 J. E. Cutting, Perception with A n Eye for Motion (MIT Press, Cambridge, MA, 1986). [48] R. Basri and S. Ullman, The alignment of objects with smooth surfaces, in Proc. 2nd Int. Conf. Computer Vision, Florida, 1988, 482-488. [49] D . Weinshall, Model based invariants for (3-D) visicn, Int. J. Computer Vision, lO(1) (1993) 27-42. [50] R. Basri and S. Ullman, Linear operator for object recognition, in J. E. Moody, S. J. Hanson and R. P. Lippmann (eds.), Advances in Neural Information Processing Systems 4 (Morgan Kaufmann, San Mateo, CA, 1991).
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 945-977 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
1 CHAPTER 5.51 VIDEO CONTENT ANALYSIS AND RETRIEVAL
HONGJIANG ZHANG Hewlett-Packard Laboratories, 1501 Page Mill Road Palo Alto, Calijornia 94304, USA E-mail:
[email protected]. corn This chapter discusses a number of available techniques and state-of-the-art research issues in video content analysis and retrieval. It focuses on basic algorithms for video structure parsing, content representation, content-based abstraction and applications tools for content-based video indexing, retrieval and browsing.
Keywords: Video content analysis, video partition, visual abstraction, content-based indexing, content-based retrieval and browsing, digital video, multimedia, digital library.
1. Background and Motivations
With rapid advances in communication and multimedia computing technologies, accessing a vast amount of visual data is becoming a reality on information superhighways and in digital libraries. Though the most important investments have been targeted at the information infrastructure, it has been realized that video content analysis and processing are key issues in putting together successful applications [1,2]. The matter of fact is that interacting with multimedia data, video in particular, requests much more than just connecting users with data banks and delivering data via networks to customers’ homes or offices. It is simply not enough to just store and display video as in commercial video-on-demand services. The need, from the point of view of content, is that the visual data resources managed by such systems have to be structured and indexed based on the content before being accessed. However, when it is possible, the human production of such content descriptive data is so time consuming - and thus costly - that it is almost impossible to generate it for the vast amount of visual data available. Further, human-produced descriptive data is often subjective, inaccurate and incomplete. Thus, fundamentally, what we need are new technologies for video content analysis and representation to facilitate organization, storage, query and retrieval of mass collections of video data in a user-friendly way. When text is indexed, words and phrases are used as index entries for sentences, paragraphs, pages or documents. Similarly, video indexing will require partitioning of video documents into shots and scenes and extracting key frames or key sequences as entries for scenes or stories. Therefore, automated indexing of video will require 945
946
H. J. Zhang
the support of tools that can detect such meaningful segments, and extract content features of any video source. Figure 1 shows a system diagram of such a video content analysis process consisting of three major steps.
Parsing: This process will partition a video stream into generic clips of different levels of granularity and extract structural information of video. These clips will be the units for representation and indexing. Content analysis, abstraction and representation: Ideally, individual clips will be decomposed into semantic primitives based on which a clip can be represented and indexed with a semantic description. In practice, the abstraction process will generate a visual abstract of clips and low-level visual features of clips will be used to represent their visual content. Retrieval and browsing: Indices are built based on content primitives or meta- data through, for instance, a clustering process which classifies shots into different visual categories. Schemes and tools are needed to utilize these content representation and indices to query, search and brows large video databases for retrieving desired video clips.
Feature Extraction
Meta-data
Fig. 1. Process diagram for video content analysis and retrieval.
The temporal segmentation process is analogous to sentence segmentation and paragraphing in parsing textual documents, and many effective algorithms are now available for temporal segmentation [4-131, as to be described in detail later in Section 2. Since fully automated visual content understanding or mapping low level visual features to high level semantic content is not feasible in the near future, video content representation will be mostly based on low-level content features. Such retrieval approaches using low-level visual features as content representation have shown a great potential for retrieval of images in large image databases [14]. While we tend to think of indexing for supporting retrieval, browsing is equally significant for video source material since the volume of video data also requires techniques to present information landscape or structure to give an idea of what is out there. The task of browsing is actually very intimately related to retrieval. On the one hand, if a query is too general, browsing is the best way to examine the results. Hence, browsing also serves as an aid to formulate queries, making it easier for the user to just ask around in the process of figuring out the most appropriate
5.5 Video Content Analysis and Retrieval
947
query to pose. A truly content-based approach to video browsing also requires some level of analysis of video content, both structural and semantic, rather than simply providing a more sophisticated view of temporal context. The need for video content analysis tools as summarized above poses many research challenges to scientists and engineers across all multimedia computing disciplines. In this chapter, we will try to discuss the approaches to visual content analysis, representation and their applications and to survey some open research problems. Section 2 covers video content parsing algorithms and schemes, including temporal segmentation, video abstraction, shot comparison and soundtrack analysis. Section 3 presents briefly video content representation schemes in terms of visual features, objects and motions. Section 4 describes tools for content-based video retrieval and browsing and Section 5 reviews some research prototypes and application systems. Section 6 summarizes some current research issues. 2. Temporal Partition of Video Sequences
The basic temporal unit for indexing and manipulation of video is shots, consisting of one or more frames recorded contiguously and representing a continuous action in time and space. A collection of one or more adjoining shots that focus on an object or objects of interest may comprise a scene. Temporal partitioning is the process of detecting boundaries between every two consecutive shots, so that a sequence of frames belonging to a shot will be grouped together. Hence, temporal partitioning is the first step in parsing video, and has been one of the first issues addressed by many researchers in video content analysis. The temporal partitioning problem is also important for applications other than video indexing and editing. It also figures in performing motion compensated video compression, where motion vectors must be computed within segments, rather than across segment boundaries [15]. However, accuracy is not as crucial for either of these applications; for example, in compression, false positives only increase the number of reference frames. On the other hand, in video segmentation for indexing, such false positives would have to be corrected by manual intervention. Therefore, high accuracy is a more important requirement in automating the partitioning process for video content analysis. There are a number of different types of transitions or boundaries between shots. The simplest transition is a cut, an abrupt shot change which occurs between two consecutive frames. More sophisticated transitions include fades, dissolves and wipes, etc. [16]. A fade is a slow change in brightness of images usually resulting in or starting with a solid black frame. A dissolve occurs when the images of the first shot get dimmer and the images of the second shot get brighter, with frames within the transition showing one image superimposed on the other. A wipe occurs when pixels from the second shot replace those of the first shot in a regular pattern such as in a line from the right edge of the frames. A robust partitioning algorithm should be able to detect all these different boundaries with good accuracy.
948 H. J. Zhang
The basis of detecting shot boundaries is the fact that consecutive frames on either side of a boundary generally display a significant change in content. Therefore, what is required is some suitable quantitative measure which can capture the quantitative difference between such a pair of frames. Then, if that difference exceeds a given threshold, it may be interpreted as indicating a shot boundary. Hence, establishing suitable metrics is the hey issue in automatic partitioning. The optimal metric for video partitioning should be able to detect the following three different factors of image change: Shot change, abrupt or gradual; Motion, including those introduced by both camera operation and object motion; Luminosity changes and noise. In this section, a number of algorithms for temporal partitioning of video data either in its original format or in compressed domain representations will be presented .
2.1. Shot Cut Detection Metrics using Original Format of Digital Video Data Figure 2 illustrates a sequence of three consecutive video frames with a clit occurring between the second and third frames. The significant difference in content is readily apparent. If that difference can be expressed by a suitable metric, then a segment boundary can be declared whenever that metric exceeds a given threshold. The major difference among a variety of automatic video partitioning is the difference metrics used to quantitatively measure changes between consecutive frames and schemes to apply these metrics. Difference metrics used in partitioning can be divided into two major types: those based on local pixel feature comparison, such as pixel values and edges, and those based on global features such as pixel histograms and statistical distributions of pixel-to-pixel change. These types of metrics may be implemented with a variety of different modifications to accommodate the idiosyncrasies of different video sources and have been successfully used in shot boundary detection.
Fig. 2. Three frames across a sharp cut.
5.5 Video Content Analysis and Retrieval
949
2.1.1. Pixel value comparison
A simple way to detect a qualitative change between a pair of frames is t o compare the spatially correspondi.ng pixels in the two frames t o determine how many have changed. This approach is known as pair-wise pixel comparison. In the simplest case of monochromatic images, a pixel is judged as changed if the difference between its intensity values in the two frames exceeds a given threshold t . This algorithm simply counts the number of pixels changed from one frame to the next. A shot boundary is declared if more than a given percentage of the total number of pixels (given as a threshold T ) have changed. A potential problem with this metric is its sensitivity to camera movement. For instance, in the case of camera panning, a large number of objects will move in the same direction across successive frames; this means that a large number of pixels will be judged as changed even if the pan entails a shift of only a few pixels. To make the detection of camera breaks more robust, instead of comparing individual pixels, we can compare corresponding regions (blocks) in two successive frames. One such approach applies motion compensation at each block [8]. That is, each frame is divided into a small number (e.g. 12 in [8]) of non-overlap blocks. Block matching within a given search window is then performed t o generate a motion vector and match value, normalized to lie in the interval [0, 11 with zero representing a perfect match. A common block matching approach used in video coding is that for each block in frame j, find the best fitting region in image ( j 1) in a neighborhood of the corresponding block according to the matching function
+
where:
Fj(k, E ) represents the value at pixel (k,1) of a ( m x n) block in the current frame j ; Fj+l(k,l)represents the value at pixel (k,E)of the same ( m x n) block in the next frame ( j+ 1); ( d z , d y ) is a vector representing the search location and the search space is d x = { - p , + p } and dy = { - p , + p } ; F,, is the maximum pixel value of frames and is used to normalized the matching function. The set of (dz,d x ) that produces the minimum value of Diff defines the motion vector from the center of block i in the current frame to the next frame. Then, the difference between two frames can be defined as K i=l
950
H. J. Zhang
where i is the block number in a frame, K is the total number of blocks, Ri is the match value for block i which equals to the minimum Diff in a given searching space, and ci is a set of predetermined weights for each block. To eliminate the effect of noise, the 2 highest and 2 lowest match values are discarded in calculating D,. A cut is declared if D, exceeds a given threshold. 2.1.2. Histogram comparison An alternative t o comparing corresponding pixels or blocks in successive frames is to compare some statistic and global features of the entire image. One such feature is the histogram of intensity levels. The principle behind this algorithm is that two frames having an unchanging background and objects will show little difference in their respective histograms. The histogram comparison algorithm should be less sensitive to object motion than the pair-wise pixel comparison algorithm, since it ignores the spatial changes in a frame. Let H f ( i )denote the histogram value for frame f , where i is one of the G possible pixel levels, then, the difference between frame f and its successor (f 1) may be given by the following X2-test formula,
+
If the overall difference Dh is larger than a given threshold T , a segment boundary is declared. To be more robust to noise, each frame is divided into a number of regions of same size, e.g. 16 regions. That is, instead of comparing global histograms, histograms for corresponding regions in the two frames are compared and the 8 largest differences are discarded to reduce the effects of object motion and noise [4].
2.1.3. Edge pixel comparison Edges in images provide useful information about the image content and changes in edge distributions between successive frames is a good indication of content changes. When a cut or a graduation transition occurs, new intensity edges a p pear far from the locations of old edges and similarly, old edges disappear far from the locations of new edges. Based on this observation, a n effective video partitioning algorithm which can detect both cuts and gradual transitions has been developed as described below [9]. First, we define an edge pixel in a current frame that appears far from (a distance r ) an existing edge pixel in the last frame as a n entering edge pixel, and an edge pixel in the last frame that disappears far from an existing edge pixel in the current frame as an exiting edge pixel. Then, by counting the fraction of the entering edge pixels (pi,) and that of the exiting edge pixels (pout) over the total number of pixels in a frame, respectively, we can detect transitions between two shots. That is, the difference metric between frames f and f 1 can be defined as:
+
5.5 Video Content Analysis and Retrieval 951
D,(f, f + 1) will assume a high value across shot boundaries and generate peaks in the time sequence. Once a peak is detected, it can be further classified as corresponding to a cut or gradual transitions since cuts usually correspond to sharp peaks occurring over 2 or 3 frames, while gradual transitions usually correspond to low but wide peaks over a larger number of consecutive frames. Experiments have shown that this algorithm is very effective in detecting both sharp cuts and gradual transitions. However, since it requires more computation, this algorithm is slower than others. 2.2. Gradual Transition Detection using Original Format of Digital Video Data Cuts are the simplest shot boundary and easy to detect using the difference metrics described above. Figure 3 illustrates a sequence of inter-frame difference resulting from the histogram comparison. It is easy to select a suitable cutoff threshold value (such as 50) for detecting the two cuts represented by the two high pulses. However, sophisticated shot boundaries such as dissolve, wipe, fade-in, and fade-out are much more difficult to detect since they involve more gradual changes between consecutive frames than does a sharp cut. Furthermore, changes resulting from camera operations may be of the same order as that from gradual transitions, which further complicate the detecting. Figure 4 shows five frames from a typical dissolve: the last frame of the current shot just before the dissolve begins, three frames within the dissolve, and the frame in the following shot immediately after the dissolve. The actual dissolve occurs across about 30 frames, resulting in small changes between every two consecutive frames in the dissolve. The sequence of inter-frame differences of this dissolve defined by the histogram calculation is displayed in the inset of the graph shown in Fig. 3, whose values are higher than those of their neighbors but significantly lower than the cutoff threshold. This sequence illustrates that gradual transitions will downgrade the power of a simple difference metric and a single threshold for camera break detection algorithms. 80
70 60 Y
50 e
!
f
40
30 20 10 0
_ - - _ - - _- -2 _'f a_' z , _ 2 _x _ _ N
7
10
w
0
N
Frame Number
Fig. 3. A sequence of histogram based inter-frame differences.
952
H. J. Zhang
Fig. 4. An example of dissolve sequences.
The simplest approach t o this problem would be to lower the threshold. Unfortunately, this cannot be effectively employed, because noise and other sources of changes often introduce the same order of difference between frames as of gradual transitions, resulting in “false positives.” In this subsection, we discuss four video partition algorithms which are capable of detecting gradual transitions with acceptable accuracy while achieving very high accuracy in detecting sharp cuts. They are: Zhang’s twin cornparison algorithm, Agrain’s algorithm based on distribution of pixel change, Zabih’s edge comparison algorithm and Hampapur’s editing model based algorithm. 2.2.1. Twin- comparison approach This algorithm was the first published one which achieves high accuracy in detecting both cuts and gradual transitions. As shown in Fig. 4 it is obvious that the first and the last frame across the dissolve are different, even if all consecutive frames are very similar. In other words, the difference metric with the threshold as shown in Fig. 3 would still be effective as it was applied t o the comparison between the first and the last frames directly. Thus, the problem becomes one of detecting these first and last frames. If they can be determined, then the period of gradual transition can be isolated as a segment unto itself. If we look at the inset of Fig. 3, it can be noticed that the difference values between most of the frames during the dissolve (as well as wipes and fades) are higher, although only slightly, than those in the preceding and following segments. What is required is a threshold value which can detect this sequence and distinguish it from a n ordinary camera shot. Based on this observation, the twin comparison algorithm was developed which introduces two comparisons with two thresholds. The algorithm uses two thresholds: Tb for sharp cut detection in the same manner as was described in the last subsection; and the second and lower threshold Ts is introduced for gradual transition detection. As illustrated in Fig. 5, whenever the
5.5 Video Content Analysis and Retrieval
953
Fig. 5. Twin-comparison approach.
difference value exceeds Tb, a camera cut is declared, e.g. F s in Fig. 5. However, the twin-comparison also detects differences which are smaller than Tb but larger than Ts, Any frame which exhibits such a difference value is marked as the potential start (F,) of a gradual transition. Such a frame is labeled in Fig. 5. This frame is then compared against subsequent frames, which is called a n accumulated comparison. The end frame ( F E )of the transition is detected when the difference between consecutive frames falls below threshold Ts while the accumulated difference exceeds Tb. Note that the accumulated comparison needs only t o be computed when the difference between consecutive frames exceeds Ts. If the consecutive difference value drops below Ts before the accumulated comparison value exceeds Tb, then the potential start point is dropped and the search continues for other gradual transitions. A potential problem with this algorithm is that camera panning and zooming and large object motion may introduce similar gradual changes as gradual transitions, which will result in “false positives”. This problem can be solved by global motion analysis as presented later in Section 2.4. That is, every potential transition sequence detected will be passed t o a motion analysis process to further verify if it is actually a global motion sequence [5]. Experiments show that the twin comparison algorithm is very effective and achieves a very high level of accuracy. 2.2.2. Pixel change classification Another algorithm for detecting both cuts and gradual transitions is derived based on the statistic of pixel value changes between two consecutive frames [6]. It is assumed that the inter-frame pixel values change due t o a combination of sources: First , a small amplitude additive zero-mean Gaussian noise, modeling camera, tape and digitization noise sources; second, changes of pixels resulting from object or camera operation and lighting within a given shot; and third, changes caused by cuts and gradual transitions. According to analytical models for each of these
954 H . J . Zhang
changes, cuts can be found by looking at the number of pixels whose change of value between two consecutive frames falls in the range (128,255). Dissolves and fade to/from whitelblack can be identified by the number of pixels which change in value between two consecutive frames in range {7,40) for %bit coded gray-level images. However, changes of pixel value resulting from wipes are also in the range of (128,255); thus, wipes may not be detected reliably. This may be solved by further looking at the spatial distributions of pixels whose range of changes are in (128,255) since during wipes, each frame will have a portion of the current shot and a portion of the new shot, thus, the changes usually occur in the boundary areas of the two portions. Based on these statistics, sharp cuts and gradual transitions can be detected by examining the difference ranges of corresponding pixels in two consecutive frames. This algorithm is not designed to detect cuts or transitions based on only two consecutive frames, but incorporating a temporal filtering over a sequence of consecutive frames. Also, histogram equalization needs to be applied to frames for wipe and cut detection, which slows down the detection process. 2.2.3. Edge pixel comparison
The edge pixel comparison algorithm as defined by Eq. (2.4) can also be applied in detecting gradual transitions. This is because gradual transitions usually introduce relatively lower but wide peaks of D e ( f ,f') values over a number of consecutive frames, different from sharp and narrow peaks for cuts. Fades and dissolves can be distinguished from each other by looking at relative values of local regions. During a fade in, pi, will be much higher than pout since there should be more entering edge pixels and fewer exiting edge pixels while a new shot progressively appears into the frames from back. In contrast, during fade out, poutwill be much higher than p i , since the current shot progressively disappears into the back frames. A dissolve, on the other hand, consists of an overlapping fade in and fade out: during the first half of dissolve, pi, will be greater while during the second half, pout will be greater. Wipes can be distinguished from dissolves and fades by looking at the spatial distribution of entering and exiting pixels, since frames of a wipe sequence usually have a portion of the current shot and a portion of the new shot. Therefore, if we take the location into the analysis when calculating the fraction of changed edge pixels, wipes can be detected and distinguished from other types of transitions. 2.2.4. Editing model fitting
Hampapur et al. have studied algorithms for detecting different types of gradual transitions by fitting sequences of inter-frame changes to editing models, one for a given type of gradual transition [lo]. However, the potential problem with such model based algorithms is that as more and more different types of editing effects (which still fall in mainly three basic classes: dissolve, wipe and fade) become available, it is hard to model each one of them. Furthermore, transition sequences may
5.5 Video Content Analysis and Retrieval 955 not follow any particular editing model, due to possibly noises and/or combination of editing effects. Such problems may exist in other detection algorithms as well, though this particular algorithm may be more prone to this problem. 2.3. Video Partitioning Algorithm using Compressed Domain Representation
As JPEG, MPEG and H.26X [15] have become industrial standards, more and more video data have been and will continue to be stored and distributed in one of these compressed formats. It would, therefore, be advantageous for the tools we envisage to operate directly on compressed representations, saving on the computational cost of decompression. More importantly, the compressed domain representation of video defined by these standards provide features which may be more effective in detecting content changes. DCT (Discrete Cosine Transform) coefficients are the basic compressed domain features encoded in JPEG, MPEG and H.26X. Another important feature encoded in the latter two standards is motion vectors. These are the two main features to be utilized in algorithms for video partitioning, with the most effective one being the one that combines bot-h features. In this subsection, we discuss in detail three basic types of algorithms for video partitioning using compressed video data. 2.3.1. DCT coeficient-based comparison
In general, compression of a video frame begins by dividing a frame into a set of 8 x 8 pixel blocks [15], as shown in Fig. 6. The pixels in the blocks are then transformed by the forward DCT (FDCT) into 64 DCT coefficients. That is,
where C ( k ) = 1 / a if k = 0 and 1 otherwise. F ( u , w )are DCT coefficients and p(2, y) is value of pixel (2,y) in a block. F(0,O) is the DC term or DC coefficient of a 8x8 block
+ X
Y Original pixels
DCT coefficients
Fig. 6. 8 x 8 block based DCT and zig-zag encoding.
956
H. J. Zhang
block, which is the average value of the 64 pixels, and the remaining 63 coefficients are termed the AC coefficients. These DCT coefficients are then quantized and encoded in a zig-zag order by placing the low frequency coefficients before the high frequency coefficients as shown in Fig. 6 . The coefficients are finally Huffman entropy encoded. The process can then be reversed for decompression. Since the DCT coefficients are mathematically related to the spatial domain and represent the content of each frame, they can be used to detect the difference between two video frames. Based on this idea, the first DCT comparison metric for partitioning JPEG videos was developed by Arman et al. [ll].In this algorithm, a subset of the blocks in each frame (e.g. take out the boundary blocks of a frame and use every other one of the rest) and a subset of the DCT coefficients for each block were used as a vector representation (V’) for each frame. That is,
V’ = (co,c1,. . . , c i , . .. ,cm)
(2.6)
where ci is the ith coefficients of the selected subset. The members of V’, are randomly distributed among all AC coefficients. One way to choose the subset is to use every other coefficient. The members of V’ remain the same throughout the video sequence to be segmented. The difference metric between frames is then defined by content correlation in terms of a normalized inner product:
DDCTC= 1 - IV’ .VS+rpl/lVfl lVj+rpl
(2.7)
where cp is the number of frames between the two frames being compared. It has been observed that for detecting shots boundaries, DC components of DCT’s of video frames provide sufficient information [18]. Based on the definition of DCT, this is equivalent to a low resolution version of frames, averaged over 8 x 8 non-overlap blocks [15]. Applying this idea makes calculation of (2.5) much faster while maintaining similar detection accuracy. Using DC sequences extracted from JPEG or MPEG data also makes it easy to apply histogram comparison. That is, each block is treated as a pixel with its DCT value as the pixel value, then, histograms of DCT-DC coefficients of frames calculated and compared using metrics (2.3). This algorithm has been approved to be very effective, achieving both high detection accuracy and speed in detecting sharp cuts [19]. The DCT based metrics can be directly applied to JPEG video, where every frame is intra-coded. However, in MPEG, temporal redundancies are reduced by applying block-based motion compensation techniques, while spatial redundancies are reduced by block-based intra-coding as in JPEG. Therefore, as shown in Fig. 6, in MPEG, there are three types of frames used: intra(1)-framed, predicted (P) frames and bi-directional predicated and interpolated (B) frames. Only the DCT coefficients of I frames are transformed directly from original images, while for P and B frames DCT coefficients are in general residual errors from motion compensated prediction. This means that DCT based metrics can only be applied in
5.5 Video Content Analysis and Retrieval
957
comparing I frames in MPEG video. Since only a small portion of frames in MPEG axe I frames, this significantly reduces the amount of processing which goes into computing differences. On the other hand, the loss of temporal resolution between I frames will introduce a large fraction of false positives in video partitioning which have to be handled with subsequent processing. 2.3.2. Motion vector based segmentation Apart from pixel value, motion resulting from either moving objects, camera operations or both represents another important visual content in video data. In general, the motion vectors should show continuity between frames within a camera shot and show discontinuity between frames across two shots. Thus, a continuity metric for a field of motion vectors should serve as a n alternative criterion for detecting segment boundaries. The motion vectors between video frames are in general obtained by block matching between consecutive frames, which is a n expensive process. However, if the video data are compressed using either MPEG standards, motion vectors can be obtained from the bit streams of the compressed images. In MPEG data streams, as shown in Fig. 7, there is one set of motion vectors associated with each P frame representing the prediction from the last or next I frame; and there may be two sets of motion vectors associated with each B frame, forward and backward. That is, each B frame is predicted and interpolated from its preceding and succeeding I/P frames by motion compensation. If there is a significant change (discontinuity) in content between two frames, either two B frames, a B frame and an I/P frame or a P frame and an I frame, there will be many blocks in the frame in which the residual error from motion compensation is too high to tolerate. For those blocks, MPEG will not apply the motion compensation prediction but instead intra-code them using DCT. Subsequently, there will be either no or only a few motion vectors associated with those blocks in the B or P frames. Therefore, if there is a Forward Predktion
I t t t l t t t l t t t l Bidirectional Prediction Fig. 7. Frame types in MPEG video.
958
H . J. Zhang
es
500
1500
1000
2000
Fig. 8. Camera break detection based on motion vectors - A sequence of numbers of motion vectors associated with B frames from a documentary video compressed in MPEG.
shot boundary falling in between two frames, there will be a smaller number of inter-coded, but a larger number of intra-coded blocks, due t o the discontinuity of the content between the frames. Based on this observation, we can detect a shot boundary by counting the number of inter-coded blocks in P and B frames [12]: That is, if NinterlNintra
< Tb
is lower than a given threshold, then a shot boundary is declared between the two frames. Figure 8 illustrates values of N i n t e r l N i n t r a for a video sequence. In this case, the camera cuts are accurately represented as valleys below the threshold level. However, this algorithm may fail t o detect gradual transitions because the number of inter-coded blocks in frames during such a transition sequence is often much higher than the threshold. 2,3.3. Hybrid approach Combining the DCT based and motion based metrics into a hybrid algorithm will improve the detection accuracy as well as processing speed in partitioning MPEG compressed video. That is, DCT-DC histograms of every two consecutive I frames are first compared to generate a difference sequence [12]. Since there is a large temporal distance between two consecutive I frames, it is assumed that if we set the threshold relatively low, all shot boundaries, including gradual transitions, will be detected by looking at the points where there is a high difference value. Of course, this first pass will also generate some false positives. Then, B and P frames between two consecutive I frames, which have been detected as potentially containing shot boundaries, will be examined in a second pass using motion-based metrics (2.8). That is, the second pass is only applied to the neighborhood of the potential boundary frames. In this way, both high processing speed and detection accuracy
5.5 Video Content Analysis and Retrieval 959 are achieved at the same time. The only potential problem with this hybrid algorithm is that it may detect false motion sequences as transition sequences, just like the twin-comparison algorithm, which requires a motion analysis based filtering process as discussed later. In summary, experimental evaluations have shown that compression domain feature based algorithms perform with at least the same order of accuracy as those using video data in the original format, though the detection of sharp cuts are more reliable than that of gradual transitions. On the other hand, compression feature based algorithms achieve much high processing speed, which make software only real time video partitioning possible. 2.4. Camera Operation and Object Motion Analysis
Camera operation and object motion are another important source of information and attributes for shot content analysis and classification. It is well known that camera operations or framing of shots are elaborately done by directors/camera operators to present certain scenes or objects and to guide the viewer’s attention. Object motions usually represent human activity and major events in video shots. Detection of camera works and object motion need to be performed in at least the following five processes in video content analysis and representation: 0 0
0
Motion-based temporal partitioning; Filtering false positives result from motion in gradual transitions detection; Recovering global motion to construct salient stills for representing video contents; Selection of key-frames; Video content representation and motion based shot retrieval.
The first two applications have already been discussed in detail in the description of motion based video segmentation algorithms in the last section. The other three applications will be described in the next two sections. The scientific problem of camera work analysis resides in the discrimination between camera work-induced apparent motion and object motion-induced apparent motion, followed by analysis to identify particular camera works and describe object motion. These are classical and unsolved problems in computer vision. However, for our needs in video content analysis and representation, several algorithms can solve this problem with satisfactory accuracy and speed. Camera works include panning and tilting (horizontal or vertical rotation of camera), zooming (focal length change), in which camera position does not change; and tracking and booming (horizontal and vertical transverse movement of camera), and dollying (horizontal lateral movement of camera) in which position of camera does change, as well as combination of these operations. The specific feature which serves to classify camera works is the motion field, as each particular camera operation will result in a specific pattern of motion field [ 6 ] . Based on this, Zhang et al.
960
H. J . Zhang
have developed a simple, yet effective approach to camera operation analysis which distinguishes the gradual transition sequences and classifies camera pan and zoom operations [6]. A more sophisticated quantitative approach to detecting camera operation uses the transformation relation between a point in the space and its coordinates in image space [20]. The transformation is then used to derive the pixel coordinate changes when there is a pan or zoom. A simple zoom is caused by a change in the camera's focal length and there is no camera body movement. Camera pan, on the other hand, is caused by rotating the camera about an axis parallel to the image plane. The combination effects of panning and zooming can be expressed as
u'=fzU+p
(2.9)
where fz is called the zoom factor, and p the pan vector. We can derive f z and p using a motion field calculated from two frames and an iterative algorithm [20]. That is, (2.10)
(2.11)
This approach not only detects camera pans, zooms, and combinations but also describes them quantitatively. The price paid for this information is a significant increase in computation time. We can combine the above two algorithms together, meaning that the first algorithm is used to detect a potential camera operation while the second algorithm is applied only to the potential frames. A limitation of the two algorithms or their combination is that when a sequence of frames is covered by a single moving object, a panning will be detected falsely. More sophisticated motion detection algorithms for video content parsing include those based on discrete tomography for camera work identification [21] and visual icon construction [22]. The distribution of the angles of edges in 3-D tomography images resulting from video can be matched to camera work models, and camera motion classification and temporal segmentation can be obtained directly. Discrimination between pan and lateral traveling and between zoom and booming can be achieved only through a complete projective model including parallax analysis [21].
3. Visual Abstraction of Video Considering the large amount of data video, it is critical to offer means for quick relevance assessment of video documents. How can we spend only few minutes to
5.5 Video Content Analysis and Retrieval
961
view an hour of video and still have a fairly correct perception of its contents? In other words, how can we map an entire segment of video to a small number of representative frames or images? Obviously, what we need is a representation t o present information on landscape or structure of video in a more abstracted manner. We call this the video content abstracting problem. In this section, we discussed three approaches to visual abstraction of video data: key-frames, video icons and skimmed highlights.
3.1. K e y - h m e Extraction Key-frames are still images extracted from original video data which best represent the content of video shots in an abstract manner. Key-frames have been frequently used to supplement the text of a video log [23], but there has been little work done in identifying them automatically. Apart from browsing, key-frames can also be used in representing video in retrieval: video index may be constructed based on visual features of key-frames, and queries may be directed at key-frames using query by image content techniques [25,26]. In some prototype systems and commercial products, the first frame of each shot has been used as the only keyframe to represent the shot content. However, while such a representation does reduce the data volume, its representation power is very limited since it often does not give sufficient clue as what actions are presented by a shot, except for shots with no change or motion. Key-frame based representation views video abstraction as a problem of mapping an entire segment (both static and motion content) to some small number of representative images. The challenge is that the extraction of key-frames needs to be automatic and content based so that they maintain the important content of the video while removing all redundant information. In theory semantic primitives of video, such as interesting objects, actions and events should be used. However, such general semantic analysis is not currently feasible, especially when information from soundtracks and/or close caption is not available. In practice, we have t o rely on low-level image features and other readily available information instead. An approach to key-frame extraction based on low-level video features has been proposed by Zhang et al. [12,26,27]. This approach determines a set of key-frames for each shot according to the following steps.
Segementation: Key-frames will be extracted at shot level based on features and content information of a shot. Given a shot, the first frame will always be selected as the first key-frame; but, whether more than one key-frame needs t o be chosen will depend on the following two criteria.
Color feature based frame comparison: After the first key-frame is selected, following frames in the shot will be compared against the last key-frame sequentially as they are processed, based on their similarities defined by a color histogram. If a significant content change occurs between the current frame and the last
962
H. J . Zhang
key-frame, the current frame will be selected as a new key-frame. Such a process will be iterated until the last frame of the shot is reached. In this way, any significant action in a shot will be captured by a key-frame, while static shots will result in only one key-frame.
Motion based selection: Dominant or global motion resulting from camera operations and large moving objects are the most important source of content changes and thus an important input for key-frame selection. Color histogram representation often does not capture such motion quantitatively, due to its insensitivity to motion. For key-frame extraction, it is necessary to detect sequences involving two types of global motions: panning and zooming. In reality, there are more than just these two types of motion in video sequences, such as tilting, tracking, and dollying. But, due to their similar visual effects, at least in key-frame selection, camera panning, tilting and tracking, as well as horizontal and vertical motion of large objects are treated as one type of motion: panning-like. Similarly, camera zooming and dollying and perpendicular (to the imaging plan) motion of large objects are treated as another type: zooming-like. To select key-frames representing these two types of motion, there are two criteria: For a zooming-like sequence, at least two frames will be selected - the first and last frames since one will represent a global, while the other will represent a more focused view. For a panning-like sequence, the number of frames to be selected will depend on the scale of panning: ideally, the spatial context covered by each frame should have little overlap, or each frame should capture a different, but sequential part of object activities. The overlap ratio can be varied to determine the density of key-frame selection. Figure 9 shows an example in which three key-frames from a tilt-up shot were extracted automatically. It is obvious that from the three key-frames one can see clearly that it is a tilt-up sequence, which is impossible to see from any single keyframe. In this respect, extracting three key-frames is a more adequate abstraction than only a single key-frame. Although choosing any one of the 3 key-frames will capture the visual content of the shot: business district buildings, it will not be able to show the camera movement which is important for users (especially producers and editors) who want to choose some shots from stock footages.
Fig. 9. Examples of key-frame extracted automatically from a shot.
5.5 Video Content Analysis and Retrieval 963 3.2. Video Icon
Another effective way of representing shot content of video for browsing is using static icons, which has attracted much research work, with two major approaches: Construction of a visual icon based on a key-frame, supplemented with pseudodepth for the representation of the duration of the shot, and perhaps arrows and signs for the representation of object and camera motion; Synthesis of an image representing the global visual contents of the shot. The first approach has been favored when the emphasis is on building a global structured view of a video document, fitted for quick visual browsing, such as in the IMPACT system [5]. Some researchers have used icon spacing or image size instead of pseudo-depth for representing the duration of the shot, but this does result in an efficient screen space use. Teodosio and Bender [6] have proposed methods for the automatic construction of an overview image representing the visible contents of an entire shot. Using camera work analysis and the geometrical transformations associated with each camera motion, successive images are mapped into a common frame, and the synthetic image is progressively built. This image is not generally rectangular. Recently, Irani et al. have perfected this type of method on two points [7]: 0 0
They use a more complete projective model, including parallax; They have shown that it is possible to compute what they called dynamic mosaic images with emphasis being given to the moving parts of the image (action) instead of background oriented images.
The resulting images have been termed salient stills [26], mosaic images [27], micon (motion icon) or VideoSpacelcons [22]. 3.3. Video Skimming Video skimming is the scheme for answering the request of abstracting an hour of video, for instance, into 5-minute highlights with a fair perception of the video contents. This is a relatively new research area and requires a high level of content analysis. A successful approach is to utilize information from text analysis of the video soundtrack. Researchers [e.g. [30]] working on documents with textual transcriptions have suggested producing video abstracts by first abstracting the text using classical text skimming techniques and then looking for the corresponding parts in the video. A successful application of this type of approach has been the Informedia project, in which text and visual content information are fused to identify video sequences that highlight the important contents of video [29]. More specifically, low-level and
964 H. J . Zhang
mid-level visual features, including shot boundary, human face, camera and object motion and subtitles of video shots are integrated with keywords, spotted from text obtained from close caption and speech recognition, using the following procedures: 0 0
0 0
Keyword selection using the well-known TF-IDF technique to skim audio; Sequence characterization by low-level and mid-level visual features; Selecting a number of keywords according to the required skimming factor; Prioritizing image sequences located in close proximity to each selected keyword: (1) Frames with faces or text; (2) Static frames following camera motion; (3) Frames with camera motion and human faces or text; (4) Frame at the beginning of the scene;
0
Composite a skimmed highlight sequence with selected frames.
Experiments using this skimming approach have shown impressive results on limited types of documentary video which have very explicit speech or text (close caption) contents, such as education video, news or parliamenary debates [31]. However, satisfatory results may not be achievable using such a text (keyword) driven approach to other videos with a soundtrack containing more than just speech, or stock footage without soundtrack. 4. Video Content Representation and Similarity
After partitioning and abstraction, the next step in video analysis is to identify and compute representation primitives, based on which the content of shots can be indexed, compared, and classified. Ideally these should be semantic primitives that a user can employ to define interesting or significant events. These semantic primitives include constituent objects’ names, appearance and motion, as well as relationships among different objects at different times and the contributions of all these attributes and relationships to the story being presented in a video sequence [32]. However, automatic extraction of such primitives is not feasible; so that we have to build content representation based on low-level features, such as color, texture and motion statistics of shots. The first set of low-level visual primitives for video content representation should be extracted from key-frames. However, such a representation alone will be insufficient to support event-based classification and retrieval, since key-frame based features capture most the spatial information while motion is an essential and unique feature of video. Therefore, the second type of primitives should be based on temporal variation and motion information in shots. With these two types of representations, we can then index video shots and define shot similarity used in video retrieval.
5.5 Video Content Analysis and Retrieval
965
4.1. Key-Frames Based Features for Shot Content Representation
Key-frames based representation of video content uses the same features as those for content based still image retrievals. These features include color, texture, and shape, which may be defined in different formats and extracted by different operations [3,14,31].
Color features: Color has been one of the first choice for image content representation and similarity measuring since it has excellent discrimination power in measuring image similarity (See Chap. 2.4 for a detailed discussion): It is very rare that two images of totally different objects will have similar colors [32]. The popular representation schemes for color include histogram [14,32,33],dominant colors [32,33], and statistical moments [34]. To make the representation effective and invariant to illumination conditions, different color spaces have been evaluated. It is concluded that L*u*v* color space tends t o be the best one [33]. Also, it is noticed that a small number of color ranges capture the majority of pixels in most of the images, thus, a few dominant colors lead to a good approximated representation of color distribution. Texture features: Texture has long been recognized as being as important a property of images as is color, if not more so, since textural information can be conveyed as readily with gray-level images as it can in color. A detailed discussion on definition and calculation of a variety of texture measures is given in Chap. 2.1. Among many alternatives, the most popular and effective texture models used in image retrieval are: Tamura features (contrast, directionality, and coarseness) [35] and the Simultaneous Auto-regressive (SAR) model [36]. To define a rich perceptive space, Picard and Liu [37] have shown that it is possible t o do so by using the Wold decomposition of the texture considered as a luminance field. One gets three components (periodic, evanescent and random) corresponding t o the bi-dimensional periodicity, mono-dimensional orientation, and complexity of the analyzed texture. Shape features: Dominant objects in key-frames represent important semantic content and are best represented by their shapes, if they can be identified by either automatic or semi-automatic spatial segmentation algorithms. A proper definition of shape similarity calls for the distinctions between shape similarity in images (similarity between actual geometrical shapes appearing in the images) and shape similarity between the objects depicted by the images, i.e. similarity modulo a number of geometrical transformations corresponding to changes in viewing angle, optical parameters and scale. In some cases, one wants to include even deformations of non-rigid bodies [38]. Even for the first type of similarity, it is desirable to use shape representations which are scale independent, based on curvature, angle statistics and contour complexity. Systems such as QBIC [14] use circularity, eccentricity, major axis orientation (not angle-independent) and algebraic moment.
As for color and texture, the present schemes for shape similarity modeling face serious difficulties when images include several objects or background. A
966
H. J. Zhang
preliminary segmentation as well as modeling of spatial relationships between shapes is then necessary (are we interested in finding images where one region represents a shape similar to a given prototype or to some spatial organization of several shapes?). 4.2. Temporal Features for Representation of Shot Content
Though a set of key-frames will represent temporal content of shots to some extent, more precise representation of shot content should incorporate motion features, apart from features of static images discussed in above. In response to such a requirement, a set of statistical measures of motion features of shots has been proposed and applied in news anchor detection and shot clustering for browsing and annotation [3,39]. However, defining more quantitative measures of shots similarity that capture the nature of motion video still remains a challenging research topic.
Temporal variation and camera operations: The means and variances of average brightness and a few dominant colors calculated over all frames in a shot may be used to present quantitative temporal variations of brightness and colors. An example is that such temporal variations have been used to classify news video clips into anchorperson shots and news shots [41]. Motion information resulting from algorithms presented in Section 3 can be used to classify video sequences into static and motion sequences. Statistic motion features: Since motion features have to roughly match human perception and it is still not clear how humans describe motions, one may have to base the motion representation on statistical motion features, rather than object trajectories [3]. More specifically, these features include directional distributions of motion vectors and average speed in different directions and areas, which may be derived from optical flow calculated between consecutive frames [3,39].
To obtain localized motion information, we can also calculate the average speed and its variance in blocks uniformly divided in frames. That is, instead of calculating average speeds in M directions for the entire frame, we calculate a set of motion statistics in M blocks of each frame. Then, the motion based comparison of shot contents will be based on the motion statistics comparison of corresponding blocks in consecutive frames. 4.3. Shot Similarity
The visual features presented above provide content representations of shots, but the goal is to define shot similarity based on these representations, to enable shot comparison or clustering for video retrieval and browsing, as discussed in Section 5. When key-frames are used as the representation of each video shot, we can define video shot similarity based on the similarities between the two key-frame sets. If two shots are denoted as Si and S j , their key-frame sets as Ki = { f i , m , m = 1 , . . .,M }
5.5 Vadeo Content Analysis and Retrieval
967
and Kj = { f j , n , n = 1 , . . . , N } , then the similarity between the two shots can be defined as
where 8 k is a similarity metric between two images defined by any one or a combination of the image features; and there are totally M x N similarity values, from which the maximum is selected. This definition assumes that the similarity between two shots can be determined by the pair of key-frames which are most similar, and it will guarantee that if there is a pair of similar key-frames in two shots, they are considered similar. Another definition of key-frame based shot similarity is: 1
Sk(Si,S j ) = M
c M
m=[Sk(fi,m, fj,l), s k ( f i , m , f j , 2 ) , . . .
m=l
7
S k ( f i , m , fj,”
.
(4.2)
This definition states that the similarity between two shots is the sum of the most similar pairs of key-frames. When only one pair of frames match, this definition is equivalent to the first one. The key-frame based similarity measure as defined above can be further combined with that of motion feature based to make the comparison more meaningful for video. 4.4. Summary
This section has discussed a set of visual features for shot content representation and similarity computation for shot comparison. It should be noted that each of the features presented above represents a particular property of images or image sequences and is effective in matching those in which the particular feature is the salient one. Therefore, it is important to identify salient features for a given image or shot and to apply the appropriate similarity metric. Developing such algorithms remains a long term research topic. Also, how to integrate the different features optimally into a feature set that has the combined representation power of each feature is another challenging problem. Finally, how to link the low level features to high level semantics (presence of object and faces, and their actions) remains and will continue to remain for a long time an open problem in computer vision. 5. Video Scene Analysis and Shot Clustering There can be hundreds of shots in one hour of a typical video program. Thus, the production of a synoptical view of the video contents usable for browsing or for quick relevance assessment calls for the recognition of meaningful time segments of longer duration than a shot, or for grouping similar shots. In media production, the level immediately higher than shots is called sequence or scene, a series of consecutive shots constituting a unit from the narrative point of view, because they are shot
968 H. J. Zhang
in the same location, or they share some thematic visual content. The process of detecting these video scenes is analogous to paragraphing in text document analysis and requires a higher level content analysis. 5.1. Scene Analysis
Two different kinds of approaches have been proposed for the automatic recognition of sequences of programs. Aigrain et al. have used rules formalizing medium perception in order to detect local (in time) clues of macro-scope change [40]. These rules refer to transition effects, shot repetition, shot setting similarity, apparition of music in the soundtrack, editing rhythm and camera work. After detection of the local clues, an analysis of their temporal organization is done in order to produce the segmentation in sequences and to choose one or two representative shots for each sequence. Zhang et al. have used structure models of specific types of programs such as TV news [41]. They recognize specific shot types such as shots with an anchor person, and then use the model to analyze the succession of shot types and produce a segmentation in sequences. Such model or knowledge based approaches can also be applied to, for instance, sport video parsing [42]. However, when we extend the application domain, we are facing the same difficulties as in computer vision. In summary, video scene analysis requires higher level content analysis and one cannot expect that it can be fully automated based on visual content analysis using current image processing and computer vision techniques. Fusion of information from video, audio and close caption or transcript text analysis may be a solution and a successful example is the I n f o m e d i a Project [29]. 5 . 2 . Shot Clustering
Clustering shots into groups, each of which contains similar content, are essential to building an index of shots for content-based retrieval and browsing. There are mainly two types of clustering approaches: partitional clustering arranges data in separate clusters; and hierarchical clustering leads to a hierarchical classification tree [44]. For the purpose of clustering a large number of shots to allow classbased video indexing and browsing with different levels of abstractions, partitional methods are more suitable since they are capable of finding optimal clustering at each level and are more suitable to obtain a good abstraction of data items. An approach of such type was proposed for video shot grouping, which is very flexible such that different feature sets, similarity metrics and iterative clustering algorithms can be applied at different levels [39]. One implementation of this approach is to use an enhanced K-means clustering algorithm incorporating fuzzy classification, which allows assignment of data items at the boundary of two classes to both of them according to membership function of the data item to all the classes. This is useful especially at higher levels of hierarchical browsing, where users expect all similar data items to be under a smaller number of nodes. The fuzzy clustering algorithm is as follows.
5.5 Video Content Analysis and Retrieval 969 (1) Get N classes using the K-means algorithm. (2) For every data item vi, i = 1 , . . . ,M
calculate its similarity with each class k(k = 1,. . . ,N ) as
where dij is the distance between data item vi and the reference vector of class j ; 4 is the fuzzy exponent (4 > 1.0). if S i k 2 p where p is the threshold set by users (0 < p < l ) ,add item vi into class k. if vi is not assigned to any class in above step, assign it to the miscellaneous class. The clustering can be based on key-frames and motion features of video shots.
A comprehensive evaluation of this approach using color and motion features can be found in [3,39]. Implementation and evaluation of this clustering approach using the Self-organization Map (SOM) method can be found in [39,45]. The advantage of SOM is its learning ability without prior knowledge and good classification performance, which have been shown by many researchers. Another benefit of using SOM is that the similarities among the extracted classes can be seen directly from the two-dimensional map. This will allow horizontal exploring as well as vertical browsing of the video data, which is very useful when we have a large number of classes at lower levels. 6. Content-Based Video Retrieval and Browsing Tools Once a video sequence has been segmented and a scheme for representation of video content has been established, tools for content-based retrieval and browsing can be built upon that representation. This section discusses briefly a set of such tools.
6.1. Retrieval Tools With the representation and similarity measures described in Section 4, querying a video database to retrieve shots of interest can be performed based on the metadata, including key-frame features and motion features, or a combination of the two. The retrieval process using these features needs to be interactive and iterative, with the system accepting feedback to narrow or reformulate searches or change its link-following behavior, and to refine any queries that are given. In case of using key-frame based features for retrieval, the index schemes and query and retrieval tools developed for image databases, such as QBIC [14], can be applied directly to video retrieval. That is, each shot is linked to its key-frames, and search of particular shots becomes a matter of identifying those key-frames from the
970 H. J. Zhang
database which are similar to the query, according to the features and similarities defined in Section 4. Similar to QBIC image database, to accommodate different user requests, three basic query approaches should be supported: query by template manipulation, query by object feature specification, and query by visual examples. Also, the user can specify a particular feature set to be used for a retrieval [3,14,15]. The retrieved video shots in such systems will be visually represented by their key frames. The user may then view the actual shots by clicking a “Video” button, as implemented in the SWIM system [3]. That is, a video player, or other types of viewers, will be initialized and play the selected segment to allow the user verifying the retrieval results. Retrieval may thus be followed by browsing as a means to examine the broader context of the retrieved key-frame. On the other hand a query can also be initiated from browsing. That is, a user may select an key-frame while browsing and offer it as a query: a request to find all key-frames which resemble that image. Figure 10 shows an example of sketch based retrieval of key-frames and a player window launched by clicking a key-frame to allow the users to view the shot represented by the first key-frame.
(b) Fig. 10. Example of key-frame-based retrieval: (a) A sketch based query; (b) Top five candidates from a database of key-frames.
Shot based temporal features provide another set of primitives that captures the motion nature of video and are used in our system as another retrieval tool to improve the retrieval performance. For detailed examples of shot based retrieval, see [3]. It should be pointed out that shot example based video retrieval is far less reliable than image example based image retrieval. This fact calls for more robust and effective representation schemes for video. Though there have been many research efforts, the development of image feature based video and image retrieval algorithms is still in its early stage and will need much more progress before they can be applied to the full search of a complete video archival. Among other things, developing efficient indexing schemes based on
5.5 Vzdeo Content Analysis and Retrieval 971 similarity features for managing large data volume is a critical problem for retrieval. It has been shown that traditional database indexing techniques like R-trees, etc. fail in the context of content based image search, and currently there is no technique that allows retrieval of similar objects in multi-dimensional space. Ideas from statistical clustering, multi-dimensional indexing, and dimensionality reduction may be useful in this area. 6.2. Browsing Tools
Interactive browsing of full video contents is probably the most essential feature of new forms of interactive access to digital video. Content-based video browsing tools should support two different nonlinear approaches to accessing video source data: sequential and random access. In addition, these tools should accommodate two levels of granularity, overview and detail, along with an effective bridge between the two levels. Such browsing tools can only be built by utilizing structure and content featured obtained in the video parsing, abstraction and feature extraction processes as discussed in the previous sections. There are mainly five types of browsing tools built based on different structural and content features of video as follows, (1) time-line display of frames; (2) content-based video player; (3) light table of video icons; (4) hierarchical browser; (5) graph based story board.
Time-line based browsers have been favored by users in video production and editing systems, for which time-line interfaces are classical. Some browsers rest on a single shot-based image component line [4,46]; but the multidimensional dimension character of video, calling for multi-line representation of the contents has been stressed by researchers working in the frame of the Muse toolkit [47,48]. This has been systematized in the strata models proposed by Aguierre-Smith and Davenpoprt [49]. The limitation of time-line browsers is that, since it is difficult to zoom out while keeping a good image visibility, the time-scope of what is actually displayed at a given moment on screen is relatively limited. Another sequential browser takes an improved VCR-like interface illustrated in Fig. 11. Apart from the conventional video play functions, this browser makes use of the meta-data of video structure and content extracted from the parsing process [26]. That is, a user can select to view key-frames at a selected rate, skip between shots such as “go to the next similar shot”, and play a skimmed version of the original video. In this way, the user is no longer constrained to linear and sequential viewing such as fast forward available in most conventional VCR’s. The light-table kind of video browser is often called clip window, in which a video sequence is spread in space and represented by video icons which function rather like a light table of slides [4,42]. In other words the display space is traded for time
972
H. J. Zhang
Fig. 11. Smart video player.
to provide a rapid overview of the content of a long video. The icons are constructed with methods presented in Section 3.3. A window may contain a sequentially listed shots or scenes from a video program, a sequence of shots from a scene, or a group of similar or related shots from a stock archival. A user can get a sense about the content of a shot from the icon, especially when salient stills are used. They can also be used to display the results of retrieval operations. The clip window browser can also be constructed hierarchically just like the window file systems used in P C operating systems nowadays. That is, each icon in a clip window can be zoomed in to open another clip window, in which each icon represents a next level and finer segment of video sequences [42]. A first attempt at building hierarchical browsers - called the Video Magnifier [50] simply used successive horizontal lines each of which offered greater time detail and narrower time scope by selecting images from the video program. To improve the content accessibility of such browsers, the structural content of video obtained in video parsing are utilized (3,261. As shown in Fig. 12, videos are accessed as a tree. At the top of the hierarchy, an entire program is represented by five key-frames, each corresponding to a sequence consisting of an equal number of consecutive shots. Any one of these segments may then be subdivided to create the next level of the hierarchy. As we descend through the hierarchy, our attention focuses on smaller groups of shots, single shots, the representative frames of a specific shot, and finally a sequence of frames represented by a key-frame. We can also move to more detailed granularity by opening a video player to view sequentially any particular segment of video selected from this browser at any level of the hierarchy. As shown in Fig. 12, icons displayed at the top level of the hierarchy are selected based on a clustering process using shot similarities. That is, programs or clips will be first clustered into groups of similar shots or sequences as described in Section 5.2; then, each group is represented by an icon frame determined by the centroid of the group, which is then displayed at the higher levels of the hierarchical browser. As one can see from this example, with such a similarity based hierarchical browsing,
5.5 Video Content Analysis and Retrieval 973
LEVEL 3 Mct
Fig. 12. Similarity-based hierarchical browser.
Fig. 13. Similarity-based hierarchical browser.
974
H. J. Zhang
the viewer can get a sense of the content of all shots in a group roughly even without moving down to a lower level of the hierarchy [2]. An alternative approach to hierarchical browsers is the class based transition graph, proposed by Yeung et al. (511. Using the clustering of visually similar shots, a directed graph whose nodes are clusters of shots is constructed, as shown in Fig. 13. Cluster A is linked to cluster B if one of the shots in A is immediately followed by a shot in B. The resulting graph is displayed for browsing, each node being represented by a key-frame extracted from one of the shots in the node. The graph can be edited for simplification by a human operator. The drawbacks of this approach lie in the difficulty of the graph layout problem, resulting in poor screen space use, and in the fact that the linear structure of the document is no longer perceptible. With the browsing tools, another advantage of using key-frames is that we are able to browse the video content down to the key-frame level without necessarily storing the entire video. This is particularly advantageous if our storage space is limited. Such a feature is very useful not only in video databases and information systems but also to support previewing in VOD systems. What is particularly important is that the network load for transmitting small numbers of static images is far less than that required for transmitting video. Through the hierarchical browser, one may also identify a specific sequence of the video which is all that one may wish to LLdemand”.Thus, the browser not only reduces network load during browsing but may also reduce the need for network services when the time comes to request the actual video.
7. Video Content Analysis - A Challenging Area
In this chapter, we have discussed a number of techniques for video content analysis for computer-assisted video parsing, abstraction, retrieval and browsing. A majority of these techniques address problems in recovering structure and low level information of video sequences, though these techniques are basic and very useful in facilitating intelligent video browsing and search. Successful efforts in extracting semantic content are still very limited and there is a long way to go before we can achieve our goal of automatic video content understanding and content-based retrieval. Many of the bottlenecks and challenges in video content analysis are the classic ones in pattern recognition and computer vision. On the other hand, it is believed that low level feature-based retrieval algorithms and tools will provide us a bridge to a more intelligent solution and will improve our productivity in image and video retrieval, though the current tool set is far from satisfactory. When we strive for visual content analysis and representation, it should be pointed out that integration of different information sources, such as speech, sound and text is as important as visual data itself in understanding and indexing visual data. Keywords and conceptual retrieval techniques are and will always be an important part of visual information systems.
5.5 Video Content Analysis and Retrieval
975
An application-oriented approach is critical to the success of visual data representation and retrieval research and will prevent us from being too theoretical. By working on strongly focused applications, the research issues reviewed in this chapter can be addressed in the context of well-defined applications and will facilitate the applications; while achieving general solutions remain a long term research topic. On the other hand, it should be pointed out that, one of the most important points is that video content analysis, retrieval and management should not be thought of as a fully automatic process. We should focus on developing video analysis tools to facilitate human analysts, editors and end users to manage video more intelligently and efficiently. It is clear that the number of research issues and their scope are rather large and expanding rapidly with advances in computing and communication [1,2]. As a result, visual data is becoming the center of multimedia computing and image analysis research, and more and more researchers from different fields are attracted and have started to explore these issues. References [l] R. Jain, A. Pentland and D. Petkovic (eds.), Workshop report: NSF-ARPA workshop on visual information management systems, Cambridge, Mass., USA, Jun. 1995. [2] P. Aigrain, H. J. Zhang and D. Petkovic, Content-based representation and retrieval of visual media: A state-of-the-art review, Int. J. Multimedia Tools and Applications 3, 3 (1996). (31 H. J. Zhang, J. H. Wu, D. Zhong and S. W. Smoliar, Video parsing, retrieval and browsing: an integrated and content-based solution, Pattern Recogn., 1996. (41 A. Nagasaka and Y. Tanaka, Automatic video indexing and full-search for video appearances, in E. Knuth and I. M. Wegener (eds.), Visual Database Systems (Elsevier Science Publishers, Vol. 11, Amsterdam, 1992) 113-127. [5] H. Ueda, T. Miyatake and S. Yoshisawa, IMPACT: An interactive natural-motionpicture dedicated multimedia authoring system, Proc. CHI’91, ACM, 1991, 343-350. [6] H. J. Zhang, A. Kankanhalli and S. W. Smoliar, Automatic partitioning of full-motion video, Multimedia S y s t e m , ACM-Springer 1, 1 (1993) 10-28. [7] P. Aigrain and P. Joly, The automatic real-time analysis of film editing and transition effects and its applications, Computers €4 Graphics 18,1 (1994) 93-103. [8] B. Shahraray, Scene change detection and content-based sampling of video sequences, SPIE Proceedings Digital Video Compression: Algorithm and Technologies, San Jose, February, 2419 (1995) 2-13. [9] R. Zabih, K. Mai and J. Miller, A robust method for detecting cuts and dissolves in video sequences, Proc. ACM Multimedia’95, San Rancisco, Nov. 1995. [lo] A. Hampapur, R. Jain and T. E. Weymouth, Production model based digital video segmentation, Multimedia Tools and Applications 1, 1 (1995) 9-46. [ll] F. Arman, A. Hsu and M. Y. Chiu, Feature management for large video databases, Proc. SPIE Conf. Storage and Retrieval for Image and Video Databases I , SPIE, 1908,Feb. 1993, 2-12. [12] H. J. Zhang, C. Y. Low, Y. Gong and S. W. Smoliar, Video Parsing Using Compressed Data, Proc. SPIE’94 Image and Video Processing 11, San Jose, CA, USA, Feb. 1994, 142-149.
976
H. J. Zhang
[13] J. Meng, Y. Juan and S.-F. Chang, Scene change detection in an MPEG compressed
video sequence, IS&T/SPIE’95 Digital Video Compression: Algorithm and Technologies, San Jose, 2419,Feb. 1995, 14-25. [14] M. Flickner et al., Query by image and video content, IEEE Computer, Sept. 1995, 23-32. [15] B. Furht, S. W. Smoliar and H. J. Zhang, Image and Video Processing in Multimedia Systems (Kluwer Academic Publishers, 1995). [16] D. Bordwell and K. Thompson, Film Art: A n Introduction (McGraw-Hill, New York, 1993). (171 Dynamic Vision, in Computer Vision: Principles, R. Kasturi and R. Jain (eds.) (IEEE Computer Society Press, Washington) 469-480. [18] B.-L. Ye0 and B. Liu, A unified approach to temporal segmentation of motion JPEG
and MPEG compressed video, Proc. IEEE Int. Conf. Multimedia Computing and Networking, Washington D.C., May 1995, 81-88. [19] I. S. Sethi and N. Patel, A statistical approach to scene change, Proc. SPIE Conf. Storage and Retrieval for Video Databases III, San Jose, CAI USA, Feb. 1995. [20] Y. T. Tse and R. L. Baker, Global zoomjpan estimation and compensation for video compression, 1991, May, Proc. ICASSP’91, Vol. 4. [21] A. Akutsu and Y. Tonomura, Video tomography: An efficient method for camerawork extraction and motion analysis, Proc. ACM Multimedia Conference, San Francisco, Oct. 1993. [22] Y. Tonomura, A. Akutsu, K. Otsuji and T. Sadakata, VideoMAP and VideoSpaceIcon: Tools for anatomizing video content, Proc. InterChi’9.9, ACM (1994) 131-136. [23] B. C. O’Connor, Selecting key frames of moving image documents: A digital environment for analysis and navigation, Microcomputers for Information Management 8 , 2 (1991) 119-133. [24] H. J. Zhang, S. W. Smoliar and J. H. Wu, Content-based video browsing tools,
Proc. IS& T/SPIE’95 Multimedia Computing and Networking, San Jose, 2417, Feb. 1994. [25] H. J. Zhang, C. Y. Low, S. W. Smoliar and J. H. Wu, Video parsing, retrieval and
browsing: an integrated and content-based solution, Proc. ACM Multimedia’95, San Francisco, Nov. 1995, 15-24. [26] L. Teodosio and W. Bender, Salient video stills: Content and context preserved, Proc. ACM Multimedia’93, Anaheim, CA, USA, Aug. 1993. [27] M. Irani, P. Anandan, J. Bergen, R. Kumar and S. Hsu, Mosaic based representations of video sequences and their applications, Image Communication special issue on Image and Video Semantics: Processing, Analysis and Application, 1995. [28] A. Takeshita, T. Inoue and K. Tanaka, Extracting text skim structures for multimedia browsing, in Mark Maybury (ed.), Working Notes of IJCAI Workshop on Intelligent Multimedia Information Retrieval, Montreal, Aug. 1995, 46-58. [29] A. G. Hauptmann and M. Smith, Text, speech and vision for video segmentation: The informedia project, Working Notes of IJCAI Workshop on Intelligent Multimedia Information Retrieval, Montreal, Aug. 1995, 17-22. [30] M. Davis, Media streams: An iconic visual language for video annotation, Proc. Sym. Visual Languages, Bergen, Norway, 1993. [31] A. Pentland, R. W. Picard and S. Sclaroff, Photobook: Content-based manipulation of image databases, Proc. Storage and Retrieval for Image and Video Databases II, 2185,San Jose, CA, USA, Feb. 1994. [32] M. J. Swain and D. H. Balllard, Color indexing, Int. J. Computer Vision 7 (1991) 11-32.
5.5 Video Content Analysis and Retrieval
977
[33] H. J. Zhang et al., Image retrieval based on color features: An evaluation study, Proc. SPIE Phonics East, Conf. Digital Storage €9 Archiving, Philadelphia, Oct. 1995. [34] M. Stricker and M. Orengo, Similarity of color images, Proc. SPIE Conf. Storage and Retrieval for Image and Video Databases III, 1995, February, SPIE Conf. Proc. 2420, San Jose, CA, USA, Feb. 1995, 381-392. [35] H. Tamura, S. Mori and T. Yamawaki, Texture features corresponding to visual perception, IEEE Trans. Syst. Man Cybern. 6, 4 (1979) 460-473. [36] J. Mao and A. K. Jain, Texture classification and segmentation using multiresolution simultaneous autoregressive models, Pattern Recogn. 25, 2 (1992) 173-188. [37] R. Picard and Fang Liu, A new word ordering for image similarity, Proc. Int. Conf. Acoustic Signals and Signal Processing, Adelaide, Australia, 5, Mar. 1994, 129. [38] S. Sclaroff and A. Pentland, Modal matching for correspondence and recognition, IEEE R a m . Pattern Analy. Mach. Intell. 17, 6 (1995) 544-561. [39] D. Zhong, H. J. Zhang and S.-F. Chang, Clustering methods for video browsing and annotation, Proc. Storage and Retrieval for Image and Video Databases I V , San Jose, CA, USA, Feb. 1995. (401 P. Aigrain and P. Joly and V’eronique Longueville, Medium-knowledge-based macrosegmentation of video into sequences, Working Notes of IJCAI Workshop on Intelligent Multimedia Information Retrieval, Montreal, Aug. 1995, 5-14. [41] H. J. Zhang, S. Y. Tan, S. W. Smoliar and Y. Gong, Automatic parsing and indexing of news video, Multimedia Systems, 2, 6 (1995) 256-265. [42] Y . Gong, L. T. Sin, H. C. Chuan, H. J. Zhang and M. Sakauchi, Automatic parsing of TV soccer programs, Proc. Second IEEE Int. Conf. Multimedia Computing and Systems, Washington D.C., May 1995, 167-174. [43] H. J. Zhang and S. W. Smoliar, Developing power tools for video indexing and retrieval, Proc. SPIE’94 Storage and Retrieval for Video Databases, San Jose, CA, USA, Feb. 1994. [44] A. K. Jain and R. Duber, Algorithms f o r Clustering Data (Prentice Hall, 1988). [45] H. J. Zhang and D. Zhong, A scheme for visual feature based image indexing, Proc. IS&T/SPIE Conf. Image and video Processing III, San Jose, CA, 1995, 36-46. [46] P. Aigrain and V. Longueville, A connection graph for user navigation in a large image bank, Proc. RIA0’91, Barcelona, Spain, 1991, 1, 67-84. (471 M. E. Hodges and R. M. Sassnett and M. S. Ackerman, A construction set for multimedia applications, IEEE Software (1989) 37-43. [48] W. E. Mackay and G. Davenport, Virtual video editing in interactive multimedia applications, Communications of the ACM 32, 9, July, 1989. [49] T. G . Aguierre-Smith and G. Davenport, The stratification system: A design environment for random access video, Proc. 3rd Int. Workshop on Network and Operating System Support for Digital Audio and Video, La Jolla, CA, USA, Nov. 1992, 250-261. [50] M. Mills, J. Cohen and Y . Y . Wong, A magnifier tool for video data, Proc. INTERCHI’92, 1992, ACM, May, 93-98. [51] M. M. Yeung, B.-L. Yeo, W. Wolf and B. Liu, Video browsing using clustering and scene transitions on compressed sequences, IS& T/SPIE’95 Multimedia Computing and Networking, San Jose, 2417, Feb. 1995, 399-413.
Handbook of Pattern Recognition and Computer Vision (2nd Edition), pp. 979-1002 Eds. C. H. Chen, L. F. Pau and P. S. P. Wang @ 1998 World Scientific Publishing Company
1 CHAPTER 5.6 I VLSI ARCHITECTURES FOR MOMENTS AND THEIR APPLICATIONS TO PATTERN RECOGNITION
HENG-DA CHENG, CHEN-YUAN WU and JIGUANG LI Computer Science Department, Utah State University, Logan, Utah 84321, USA Moment is one of the most popular techniques for image processing, pattern classification and computer vision, and has many applications. In this chapter, we apply moments t o extract the features of breast cancer biopsy images and pavement distress images. The resulting features are input into neural networks for classification. The satisfactory outcomes demonstrate the usefulness of moments. However, the high computational time complexity of the moment limits its application for pattern recognition, especially, for real-time tasks. The rapid advance of hardware technology makes VLSI implementation of pattern recognition algorithms more feasible and attractive. We propose a one-dimensional and two two-dimensional systolic arrays for computing regular and central moments. The operations of the proposed architectures which are much faster than the existing ones and their computational time complexities are studied. The important issue of VLSI design, algorithm partition, is also discussed. The basic idea of this article can be easily extended to design VLSI architectures for other kinds of moments as well.
Keywords: Moments, breast cancer detection, pavement distress detection, VLSI, algorithm partition.
1. Introduction
The task of recognizing an object independent of its position, size, or orientation is very important for many applications of pattern recognition, image processing, and computer vision. In every aspect of developing a pattern recognition system, we should always carefully determine and extract the characteristics of the pattern to be recognized. When the pattern undergoes rotation, translation, or scaling, the extracted features are more crucial for the recognition result. Many methods have been proposed to describe and extract the features of digital images [I],and among them, moment is one of the most popular techniques for extracting rotation-scalingtranslation-invariant features. In the early 1960’s, Hu [2] published the first paper on moment invariant for two-dimensional pattern recognition based on the method of algebraic invariants. Since then, many researchers have applied the moment invariants for pattern classification, image processing, and image description successfully. Teh and Chin [3] 979
980 H.-D. Cheng, C.-Y. W u & J. La
evaluated a number of moments for pattern recognition, such as regular moments, Legendre moments, Zernike moments, pseudo-Zernike moments, rotational moments, and complex moments. Teague [4] summarized some well-known properties of the zeroth-order, first-order and second-order moments, discussed the problems of image reconstruction from the inverse moments, and suggested using the orthogonal moments to recover an image. Abu-Mostafa and Psaltis [5] discussed the image recognitive aspects of moment invariants, and focused on the information loss, suppression, and the redundancy encountered in the complex moments. Dudani et al. [6] used the same set of moment invariants to recognize different types of aircrafts. Chou and Chen [7] proposed a two-stage pattern matching method called “moment-preserving quantization” that reduced the complexity of computation with quite a good accuracy, and proposed a low cost VLSI implementation. In [8], Wong and Hall used a set of moments which are invariant to translation, rotation, and scale changes to perform the scene matching of radar t o optical images. Casey [9] used the second-order moments to specify the transformation of the coordinate wave forms. By using moments, the original pattern was mapped into a variation-free domain, and the linear pattern variation of the hand printed characters was removed. Cash and Hatamian [lo] used two-dimensional moments to extract the pattern features of the optical character, and showed that the pattern features extracted from the moments provided good discrimination between characters, and 98.5% t o 99.7% recognition rates were achieved for the tested fonts. Ghosal and Mehrotra proposed a subpixel edge detection method based on a set of orthogonal complex moments called Zernike moments of the image [ll]. Khotanzad and Hong [12] also proposed a n invariant image recognition method by using Zernike moments. Saradana et al. [13] applied the second-order moments t o extract the feature vectors that can describe objects efficiently in an n-dimensional space. Liu and Tsai [14] proposed a corner detection method based on the moments. They showed that the moments of images were big factors choosing the threshold value. Belkasim et al. gave a detail study of the efficiencies of different moment invariants in pattern recognition applications [15]. They proposed a new method for deriving Zernike moments with a new normalization scheme, and obtained a better overall performance even when the data contained noise. Reeves et al. [16] presented a procedure using moment-based feature vectors t o identify a three-dimensional object from a two-dimensional image recorded at an arbitrary angle and range. They compared two methods used for this task: moments and Fourier descriptors, and proved that the standard moment gave a slightly better result than the Fourier descriptors. Though moments have been widely applied to pattern recognition and image processing, the high computational complexity limits their applications, especially for the tasks requiring real-time processing. VLSI technology provides a feasible, effective and efficient solution by implementing algorithms on hardware chips which utilize the space and time concurrencies extensively and expedite the computation greatly.
5.6 VLSI Architectures for Moments and . . . 981
2.
VLSI Architectures for Moments
Many VLSI architectures have been developed to implement algorithms for image processing and pattern recognition 117-281. By using VLSI architecture, we can speed up the computation tremendously. In order to improve the performance, using both parallelism and pipelining, a VLSI architecture must have the following characteristics [23,32]:
(1) There are only a few different types of processing elements, and the communication between them is local, simple and regular. (2) Data flow is simple and regular. In the best case, the data flow is linear (only one direction). In this chapter, we will study VLSI architectures for computing regular and central moments, and the important issue for VLSI design, algorithm partition. 2.1. Regular Moments and Central Moments The regular or geometric two-dimensional moments of order ( p
+ q ) of an area
A , for a continuous image f(z,y), is defined as F
F
where p , q E { 0 , 1 , 2 , . . .}. For a digital image of area A , the moments are: MPP
=
c
In order to obtain translation invariance, we can first locate the centroid of the image, such that
We can define the central moments that are translation invariant as follows:
From above, we can see that it takes a lot of additions and multiplications to compute the regular and central moments. Several efficient methods have been proposed to speed up the computation. Some researchers 129,301 discussed the superiority of boundary-based computation and proposed a simpler algorithm based on Green’s theorem. They converted the double integrals into a linear integral around the boundary of the polygon. Both algorithms are only suitable for binary images, i.e. f(z,y) is either 0 or 1. Jiang and Bunke [29] did not include the time for finding the edge (vertices) which could make up the majority of the total computation time. Li and Shen [30] gave the consideration of the entire process of the
982
H.-D. Cheng, C.-Y. Wu €9 J . Li
computation and proposed the use of an upper triangular systolic structure to speed up the computation, but did not include the time for calculating the lower triangular matrix required by the algorithm. Chen [31] developed a parallel algorithm for computing moments based on decomposing a two-dimensional moment into vertical and horizontal moments. He used a so-called cascade-partial-sum method to compute the summation of the partial results. The time complexity for calculating the moments is O ( ( p+ l)(q + 1 + cl)Zog(n) + (q + 1 c 2 ) n ) for a linear array, where c1 and c2 are the times spent for intermediate summations of the cascade-partial-sum method, and the values will be increased irregularly when p and q increase. Chen [31] assumed the data were preloaded in the processor array. It would take at least n time units to load the data which were not included in the time analysis. The cascade-partial-sum method will take more time for the intermediate summations of the higher-order moments; the memory of each processing element had t o be varied for different orders of moments. Furthermore, the data flow and control flow are irregular and complicated causing difficulty in achieving correct timing and designing the array structure. Therefore, this algorithm is not suitable t o be implemented using VLSI architecture [23,32].
+
2.2. One-Dimensional
VLSI Architecture for Computing
Regular Moments F'rom Eq. (2.2), for a n n x n image, we can calculate the moments of order ( p + q ) using the following algorithm: Given that i) f ( z , y ) is the gray level of the image; ii) a: and y are corresponding coordinates of the pixel of the image; iii) Mp,qare the moments of order ( p q ) ,
+
let z := 1; M,, := 0 while (x < n) begin y := 1 O U t p r e ( Z , y) := 0 while (y < n ) begin
+ zP x y q x f(x, Y) + 1) := O U t n e z ( Z ,Y)
O U t n e z ( 2 ,Y) := O U t p r e ( Z , Y) O ~ t p r e (Y ~1
y:=y+1
end /* while(y) y:=y-1 Mpq := M p q
*/
+ OUtnez(Z, Y)
a::=a:+l end /* while(x)
*/
...
5.6 VLSI Architectures for Moments and
983
The sequential algorithms with loops can be implemented using VLSI architectures [23,32]. Based on the above algorithm, we can use a one-dimensional systolic array which has two types of processing elements to calculate the regular moments. The first type of processing elements has: Five inputs -- p , q , f(z,y),control signal, and the output from the left processing element; Four outputs - p , q, the current output to the right processing element, and a signal which controls each processing element to start the calculation; Three multipliers - one for calculating zp,one for calculating yQ, and another for computing x p x yQ x f(z,y); Two registers - to store the corresponding coordinates of the processing element; One adder - to add the output from the left neighbor processing element and the currently calculated value; Some logical gates which are necessary for controlling the timing of calculation. Figure 1 shows the structure of the processing element. The symbolic representation for the processing element is shown in Fig. 2. From Fig. 1, it is clear that within a processing element, the operation for calculating zp x yQ x f(z,y) Outpre must be done concurrently with other operations. The computation time needed
+
M.1 proc...or
P
.t&
to tho noxt procomaor
lia Out,,One
ttmr unrr delay
lrpvrl Multiplexer
Tho initid vducm that stored in x' md'y UC1. will Bcloct tho data. At tho initi.lt h o . it will scloct p m d q.
LEI
And M o r that. it sclocts tho data torn tho register. Check 2 0 or not ~~t
Fig. 1. Structure of PE for One-Dimensional VLSI Architecture.
984 H.-D. Cheng, C.-Y. Wu €4 J. Li
I
f&Yl p'-p with o m t h o unit delay. 9'-q with on0 timo unit delay. #tut'-#tut with ouo timo unit dolay.
Fig. 2. Symbolic Representation of Processing Element.
+
for each processing element is max(p, q ) 2. The second kind of processing element is an accumulator which is the (n 1)th processing element.
+
2.2.1. The operations of PEs Each processing element will receive p , q, f(2,y), and the output from its previous processing element as the inputs. The data, f ( z , y ) , will be input to the one-dimensional array row by row (or column by column). At the first time unit, processing element (1,l) will receive p and q, store these values in the registers, and start to calculate 1P and 19. At the same time, p and q will be sent to the delays connected with the next processing element. Since 1 P and 1 9 can be computed simultaneously, it takes max(p, q ) time units to finish the computation. Meanwhile, the start signal takes max(p,q) time units delay to input f ( 1 , l ) and to calculate Outnez = Outp,., l p x 1 9 x f ( 1 , l ) . It takes two more time units to perform the multiplication and addition (refer to Fig. 1). Thus, the total time to calculate Out,,, = Out,,., 1 P x 1 9 x f ( 1 , l ) is max(p,q) 2 time units. The start signal will be input to processing element (1,2) after one time unit, i.e. at the time unit max(p, q ) + 1. Note for PE(1, l ) , the value of Out,,, is zero. At time unit max(p, q ) + 2, processing element (1,l) will produce the output and pass it to processing element (1,2). The values of p and q will arrive at processing element (1,2) at the second time unit, which starts to compute 1 P and 29. It also takes max(p,q) time units to calculate 1P and 29. At the (max(p, q ) 1)th time unit, the calculation for 1 P and 29 is finished, and f(1,2) is input to processing element (1,2). Therefore, at time unit max(p, q ) + 1, the required data for for processing element (1,2) are ready, and the start signal arrives to initiate the calculation. When the output from processing element ( l , l ) , Outpre,is received by processing element (1,2) at the next time unit, processing element (1,2) will perform the multiplication and addition, and produce the output Out,,, = Outpre 1 P x 29 x f ( l , 2 ) at the (max(p, q ) 3)th time unit. According to the above description, the adjacent data of the same row needs one time unit delay in order to match the timing requirement. Now let us consider the data of the next row. In order to calculate Out,,, = Outpre 2P x 1 9 x f(2, l ) , the 2-coordinate of processing element (1,l) has to be
+
+
+
+
+
+
+
5.6 VLSZ Architectures for Moments and
...
985
ca
t A
ON8 nm. un:t d d a y
U
Fig. 3. One-Dimensional VLSI Architecture.
increased by one after l p is calculated. At the pth time unit, x will be increased from 1 t o 2. Because max(p,q) time units are needed to calculate 2P and 1 9 , max(p,q) time units delay is necessary t o input the first data of the second row. Here we want to indicate that the data, f ( x , y), in the same column have the same y4 value, which was computed for f(1,y). We can input f ( x , y) without computing yq again, where x = 2 , 3 , . . . ,n. However, we have to make a trade-off between the regularity and uniformity of the processing elements and the simplicity of the structure. Since for most VLSI designs, the regularity is more important, we adapt this principle here. Figure 3 shows the VLSI structure and the data arrangement. For a n n x n image we need n 1 processing elements to form the one-dimensional structure. The operations of this structure are summarized as follows:
+
(1) p and q will be input t o and stored in the first processing element at the first time unit. Then they are passed to and stored in the next processing element after one time unit, and so on. (2) A start signal with max(p,q) time delay is needed for inputting the first data of the first row, and it will be passed to the next processing element after one time unit. (3) A max(p, q ) time delay is needed to input data of adjacent rows. (4) The adjacent data in the same row will be input after one time unit delay, i.e. the data are skewed. (5) The x-coordinate of each processing element will be increased by one after the xp is computed according to the corresponding data arrangement, and the ycoordinate is fixed. Certainly, if we want to input data column by column, the roles of 2 and y will be switched.
986
H.-D. Cheng, C.-Y. Wu & J . La
Based on the above description, the time complexity is O(max(p, q ) + 2 + n 1 max(p, q ) x (n- 1) 1) = O(max(p, q ) x n n 2) for computing the ( p q)th moment of the entire image. If there are k images, the time complexity will be k[max(p,q ) x n n 21.
+
+ +
+
+
+ +
2.3. Two-Dimensional VLSI Architecture for Computing Regular Moments
+
Although One-Dimensional VLSI architecture needs only n 1 processing elements to compute the moments of order ( p q ) for an n x n image, it needs max(p, q ) x n + n + 2 time units to complete the entire calculations. We can use a two-dimensional VLSI architecture to perform the calculation of the moments with a smaller time complexity. The basic algorithm for the two-dimensional architecture is as follows:
+
Let Mpq := 0 while not finished if start = 0 then for y := n to 1 do for x := 1 to n do begin store f ( x , y) in the register of the (x,y)th processing element calculate x p and yq end /*for x */ end /*for y */ if start = 1 then x := 1 while (x < n) begin y:= 1 Outpre(x,y) := 0; while (y < n) begin O u t n e z ( x ,Y) := O u t p r e ( 2 , Y) + x p x 'Y x f(x,Y) O u t p r e ( 2 , Y 1) := Outnez(x, Y) y:=y+l end /*while y */ y:=y-1 M p q := Mpq + Outnez(x,Y) x:=x+1 end /* while x */
+
+
For this architecture we need n x (n 1) processing elements. The structure of each processing element is about the same as the one in one-dimensional VLSI
5.6 VLSZ Architectures f o r Moments and . . . 987
4 2P' 6i n
a
w
; e B
$
start -
star#*
to the next processor
MulnpNar Mulnplexar The initid values that stored in 9 a d y' YO 1. (AYD( The ffx,y) in the register will bo output to the next proceiior when start u 0. tively. A n d will iekct the data. At tho initid time. it will ickct p Md 4. re# After that. it sekoti tho data from the corrciponding registers. Chack 2 0 or not N o t c _
fl
Tz="
Fig. 4. Structure of P E for Two-Dimensional VLSI Architecture.
fcsc.Yl out,,
start
f,fX.Yj
,, *
q'
arart'
out".? atart
p ' - p with one time unit delay. 4 ' - 9 with one time unit delay. start'-start with one time unit dclay.mdI'fx,y)-f~s.y) with one time unit delay.
Fig. 5 . Symbolic Representation of Processing Element.
architecture, but more outputs and inputs are needed. Figure 4 shows the structure of each processing element, and Fig. 5 is the symbolic representation of Fig. 4. 2.3.1. The operations of PEs In this architecture, we need t o input the data from the last column (or row) to make sure that when the calculation is started, the data are in the corresponding processing elements. A start signal is needed to initiate the calculation, and is issued until f ( 1 , l )is input to the ( 1 , l ) t h processing element. It takes n time units for f(1,l)to arrive at processing element (1,l). Meanwhile, p , q, and f(z,y) will be input to and stored at the corresponding processing elements. Once the processing element receives p , q, and the corresponding f(s, y), it starts t o calculate z p , yQ, and stores f(z, y) in the register.
988 H.-D. Cheng, C.-Y. Wu 6 J. Li
After the start signal is issued, processing element (1,l)will need two more time units to perform the calculation of Out,,, = Outpre xp x yQ x f(z,y). In the next time unit, the start signal will be passed to the next processing element at its right hand side, i.e. processing element (1,2), and to the processing element below, processing element (2,1), and so on. Since the processing elements will calculate x p and yQ immediately after p and q arrives, when the start signal is input, both x p and yQ have been calculated. The processing elements will then perform the calculation and produce the output using f(z,y) stored in the register. It is clear that after the start signal arrives, the time delay for the output data is equal to one time unit. To calculate the moments for an n x n image, we need n time units delay for issuing the start signal. Then, another 2 n (n - 1) 1 = 2n + 2 time units are needed to complete the entire calculation. Totally, it takes 3n + 2 time units to calculate the moments of order ( p q ) . Figure 6 shows the entire two-dimensional VLSI architecture. From Fig. 6 we can see that the last column of data, f(1, n),f ( 2 , n), . . .,f(n, n), are input first while the first column of data, f(1, l),f ( 2 , l ) ,. . . ,f(n, l ) , are input last. To ensure the correct data flow and timing, the data are skewed as shown in Fig. 6. Cheng and Tong [22] have a brief description about how to arrange the data to ensure the correct data flow and timing.
+
+ +
+
+
Tho start rignd wiU bo initialized with mu(p.p)+n time delay, u shown in tho figuro. Assumo tho initid vduo of M , u zoro and tho data travel between processing elements OIL0
timo
onit.
Fig. 6. Two-Dimensional VLSI Architecture.
2.4. Two-Dimensional
VLSI Architecture for the Computation of
Central Moments We propose a two-dimensional VLSI architecture for calculating the central moments. From Eq. (2.3), for calculating Z and jj, we have to compute MOO,Mol
5.6 VLSZ Architectures for Moments and
...
989
and M ~ oFrom . the definition of moments we know:
let ~ o o ( x:= ) Mol(x) := Mio(x) := O let Moo := Mol := Mi0 := 0 for y := n to 1 do for x := 1 to n do begin Moo(x) := Moo(x) + f(GY) MOl(2) := M01(x) + Y x f ( x , y ) MlO(2) := MlO(2) + x x f(x7 Y) end /* for x*/ end /* for */ for x := 1 to n do begin Moo := Moo + Moo(x) MOl := MOl MOl(X) MlO := MlO + MlO(Z) end /* for */
+
A two-dimensional VLSI structure that will calculate central moments shown in Fig. 7 consists of two subsystems: Subsystem A is to calculate z and jj and Subsystem B is to calculate the central moments. Subsystem A consists of 3 x n processing elements. The first column is to compute MOO,and the structure of the processing element is shown in Fig. 8. The second column is to compute M I O ,and the structure of the processing element is shown in Fig. 9. The value of x will be pre-stored in each processing element. The third column is for computing Mol, and the structure of the processing element is shown in Fig. 10. We need to store the value of y, which will be decreased by one after each calculation, in each processing element of this column. In order to calculate the central moments, we can combine this architecture with a two-dimensional architecture similar to the one in Section 2.3 to form a new architecture as shown in Fig. 7.
990 H.-D. Cheng, C.-Y. W u F3 J . Li
m
3n+3 time
PP.
@
max(p.q) +Zn+J time uniin dday
unit# delay Smbmyrtem B of the two-dimensional VLSI structltre for cakuktmg central moment
Fig. 7. Two-Dimensional VLSI Structure for Central Moment.
Fig. 8. Structure of PE of the First Column of Subsystem A.
Fig. 9. Structure of PE of the Second Column of Subsystem A.
5.6 VLSI Architectures for Moments and
...
991
Check 2 1 or not
Fig. 10. Structure of PE of The Third Column of Subsystem A.
2.4.1. The operations of PEs Data f ( x , y ) will be input in a column-major manner. At the first time unit, the data will be input to subsystem A, and at the third time unit, the data will be input to subsystem B. In order to perform the calculation correctly, we have to skew the input data to meet the time requirement. The data will arrive at the corresponding processing elements of subsystem B at the (n 3)th time unit, be stored in the registers and wait for the start signal to perform the calculation. According to the data arrangement and Fig. 7, at the 2nth time unit, Moo will be calculated and sent to the next column to calculate z. At the next time unit, M10 will be calculated and used for computing ??, MOOwill be sent to the next column to perform the calculation of jj. At the next time unit, the (2n + 2)th time unit, Mol will be calculated and used for computing 8. Then z and jj will be input to subsystem B. We can start to calculate (x - z)P and (y - j j ) q at the (2n 3)th time unit, and it takes max(p, q ) time units to calculate them. We have to make the data and the start signal ready at the (271 3 max(p,q))th time unit. Hence, the start signal will initiate the calculation of subsystem B at the (2n 3 max(p, q ) 1)th time unit. The structure of the processing element of subsystem B and its symbolic representation are shown in Fig. 11and Fig. 12, respectively. From the data arrangement and Fig. 7, we can see that it takes 2n+3+max(p, q ) time units for issuing the start signal, and 2n - 1 2 1 = 2n 2 time units to perform the moments computation. Thus, it takes a total of 2n 3 max(p, q ) 2n 2 = 4n max(p, q ) 5 time units to finish the entire calculation.
+
+
+ +
+ +
+ + +
+ +
+ +
+
+
+
2.5. Algorithm Partition When a computational task size is larger than the VLSI architecture size, we have to partition the task into smaller subtasks to calculate the moments on a fixed-size VLSI architecture. If we have an image with the size k x 1 and the twodimensional VLSI architecture with size m x n, and k, 1 are dividable by m and n respectively, we can easily partition the image and calculate the moments using
992
H.-D. Cheng, C.-Y. Wu €9 J. La
the next pmcesior
Mdtiplerar The initial v&ci that stored in 9 and fl u c 1. Th&y.) in the register will be output to the next proceisor when s l u t u 0. WIU i e b c t the data. At tho initid time. it will sebct p and q. reipectiveiy. Not After that. it i o b c t i the data fiom the corresponding reguterr.
Fig. 11. Structure of P E of Subsystem B.
cia
PkYj
