Cognitive Techniques in Visual Data Interpretation

Lidia Ogiela and Marek R. Ogiela Cognitive Techniques in Visual Data Interpretation

Studies in Computational Intelligence, Volume 228 Editor-in-Chief Prof. Janusz Kacprzyk Systems Research Institute Polish Academy of Sciences ul. Newelska 6 01-447 Warsaw Poland E-mail: [email protected] Further volumes of this series can be found on our homepage: springer.com Vol. 206. Ajith Abraham, Aboul-Ella Hassanien, André Ponce de Leon F. de Carvalho, and Václav Snášel (Eds.) Foundations of Computational Intelligence Volume 6, 2009 ISBN 978-3-642-01090-3 Vol. 207. Santo Fortunato, Giuseppe Mangioni, Ronaldo Menezes, and Vincenzo Nicosia (Eds.) Complex Networks, 2009 ISBN 978-3-642-01205-1 Vol. 208. Roger Lee, Gongzu Hu, and Huaikou Miao (Eds.) Computer and Information Science 2009, 2009 ISBN 978-3-642-01208-2 Vol. 209. Roger Lee and Naohiro Ishii (Eds.) Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, 2009 ISBN 978-3-642-01202-0 Vol. 210. Andrew Lewis, Sanaz Mostaghim, and Marcus Randall (Eds.) Biologically-Inspired Optimisation Methods, 2009 ISBN 978-3-642-01261-7 Vol. 211. Godfrey C. Onwubolu (Ed.) Hybrid Self-Organizing Modeling Systems, 2009 ISBN 978-3-642-01529-8 Vol. 212. Viktor M. Kureychik, Sergey P. Malyukov, Vladimir V. Kureychik, and Alexander S. Malyoukov Genetic Algorithms for Applied CAD Problems, 2009 ISBN 978-3-540-85280-3 Vol. 213. Stefano Cagnoni (Ed.) Evolutionary Image Analysis and Signal Processing, 2009 ISBN 978-3-642-01635-6 Vol. 214. Been-Chian Chien and Tzung-Pei Hong (Eds.) Opportunities and Challenges for Next-Generation Applied Intelligence, 2009 ISBN 978-3-540-92813-3 Vol. 215. Habib M. Ammari Opportunities and Challenges of Connected k-Covered Wireless Sensor Networks, 2009 ISBN 978-3-642-01876-3 Vol. 216. Matthew Taylor Transfer in Reinforcement Learning Domains, 2009 ISBN 978-3-642-01881-7 Vol. 217. Horia-Nicolai Teodorescu, Junzo Watada, and Lakhmi C. Jain (Eds.) Intelligent Systems and Technologies, 2009 ISBN 978-3-642-01884-8

Vol. 218. Maria do Carmo Nicoletti and Lakhmi C. Jain (Eds.) Computational Intelligence Techniques for Bioprocess Modelling, Supervision and Control, 2009 ISBN 978-3-642-01887-9 Vol. 219. Maja Hadzic, Elizabeth Chang, Pornpit Wongthongtham, and Tharam Dillon Ontology-Based Multi-Agent Systems, 2009 ISBN 978-3-642-01903-6 Vol. 220. Bettina Berendt, Dunja Mladenic, Marco de de Gemmis, Giovanni Semeraro, Myra Spiliopoulou, Gerd Stumme, Vojtech Svatek, and Filip Zelezny (Eds.) Knowledge Discovery Enhanced with Semantic and Social Information, 2009 ISBN 978-3-642-01890-9 Vol. 221. Tassilo Pellegrini, S¨oren Auer, Klaus Tochtermann, and Sebastian Schaffert (Eds.) Networked Knowledge - Networked Media, 2009 ISBN 978-3-642-02183-1 Vol. 222. Elisabeth Rakus-Andersson, Ronald R. Yager, Nikhil Ichalkaranje, and Lakhmi C. Jain (Eds.) Recent Advances in Decision Making, 2009 ISBN 978-3-642-02186-2 Vol. 223. Zbigniew W. Ras and Agnieszka Dardzinska (Eds.) Advances in Data Management, 2009 ISBN 978-3-642-02189-3 Vol. 224. Amandeep S. Sidhu and Tharam S. Dillon (Eds.) Biomedical Data and Applications, 2009 ISBN 978-3-642-02192-3 Vol. 225. Danuta Zakrzewska, Ernestina Menasalvas, and Liliana Byczkowska-Lipinska (Eds.) Methods and Supporting Technologies for Data Analysis, 2009 ISBN 978-3-642-02195-4 Vol. 226. Ernesto Damiani, Jechang Jeong, Robert J. Howlett, and Lakhmi C. Jain (Eds.) New Directions in Intelligent Interactive Multimedia Systems and Services - 2, 2009 ISBN 978-3-642-02936-3 Vol. 227. Jeng-Shyang Pan, Hsiang-Cheh Huang, and Lakhmi C. Jain (Eds.) Information Hiding and Applications, 2009 ISBN 978-3-642-02334-7 Vol. 228. Lidia Ogiela and Marek R. Ogiela Cognitive Techniques in Visual Data Interpretation, 2009 ISBN 978-3-642-02692-8

Lidia Ogiela and Marek R. Ogiela

Cognitive Techniques in Visual Data Interpretation

123

Lidia Ogiela AGH University of Science and Technology Faculty of Management ul. Gramatyka 10 30-067 Krakow Poland E-mail: [email protected]

Marek R. Ogiela AGH University of Science & Technology Institute of Automatics 30 Mickiewicza Ave. PL-30-059 Krak´ow Poland E-mail: [email protected]

ISBN 978-3-642-02692-8

e-ISBN 978-3-642-02693-5

DOI 10.1007/978-3-642-02693-5 Studies in Computational Intelligence

ISSN 1860-949X

Library of Congress Control Number: Applied for c 2009 Springer-Verlag Berlin Heidelberg  This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typeset & Cover Design: Scientific Publishing Services Pvt. Ltd., Chennai, India. Printed in acid-free paper 987654321 springer.com

We dedicate this book to our great Parents in gratitude for their great parental effort, limitless love, the support and the great hope they gave us To our Mom, Izydora and in memory of our Dad, Julian

Contents Contents

1

Introduction…………………………………………………………….

2

Traditional pattern recognition technigues and latest image interpretation approaches………………………………………….. 2.1 Characteristics of pattern recognition algorithms………………….. 2.2 Basic stages in the pattern classification process…………………... 2.3 Basic methods of pattern recognition……………………………… 2.3.1 Metrics-based methods……………………………………... 2.3.2 Methods based on function series…………………………... 2.3.3 Methods based on probability distributions………………… 2.4 Structural methods for describing and recognising images………... 2.5 Methods of determining image semantics…………………………. 2.5.1 Algorithms for determining image semantics………………. 2.5.2 Methods based on formal languages for determining image meanings………………………………………….

3 Cognitive aspects performed in the human mind……......................... 3.1 Brain science………………………………………………………... 3.2 General brain structure……………………………………………… 3.3 Brain functions……………………………………………………… 3.4 Information processing in the cerebral cortex and perception models……………………………….............................................. 3.4.1 Mapping dependent on categories…………………………… 3.4.2 Mapping dependent on features……………………………... 3.4.3 Mapping dependent on processes……………………………. 3.5 Cognition levels……………………………………………………... 4

The fundamentals and development of Cognitive Informatics……... 4.1 Development of cognitive subjects in science …………………….. 4.2 Theoretical aspects and formal models of cognitive informatics….. 4.2.1 IME Model – Information-Matter-Energy………………….. 4.2.2 IME-I Model – Information-Matter-Energy-Intelligence…... 4.2.3 LRMB Model – Layered Reference Model of the Brain…… 4.2.4 OAR Model – Object-Attribute-Relation............................. 4.2.5 The model of consciousness and machine cognition……….. 4.3 Cognitive science in categorisation tasks…………………………..

1

7 7 13 14 14 15 17 18 20 21 24 29 29 32 34 36 36 37 37 38 41 41 49 51 52 53 55 55 57

VIII 5

Contents

Cognitive information systems………………………………………... 5.1 Types and functions of cognitive information systems……………. 5.2 Definition and characteristics of cognitive categorization systems………………………………………………………….. 5.3 Properties of computer cognitive categorisation systems…………..

63 65

6

Understanding-based image analysis systems………………………...

75

7

UBIAS systems in cognitive interpretation of medical visualization…………………………………………………………. 7.1 UBIAS systems for semantically interpreting foot visual data…….. 7.1.1 Analysis of foot images in the dorsoplanar projection……... 7.1.2 Analysis of foot images in the lateral projection…………… 7.1.2.1 External lateral projection…………………………. 7.1.2.2 Internal lateral projection………………………….. 7.2 Cognitive systems for supporting bone fracture therapy…………...

79 79 80 85 85 89 95

Summary………………………………………………………………...

103

References……………………………………………………………….

107

Index……………………………………………………………………..

113

8

70 73

1 Introduction The extremely rapid progress of science dealing with the design of new computer systems and the development of intelligent algorithmic solutions for solving complex problems has become apparent also in the field of computational intelligence and cognitive informatics methods. The progress of these new branches of informatics has only started a few years ago, but they are already making a very significant contribution to the development of modern technologies, and also forming the foundations for future research on building an artificial brain and systems imitating human thought processes. We are already able to build robots with basic machine intelligence, which can sometimes perform complex actions and also operate by adapting to changing conditions of their surroundings. This very impressive development of intelligent systems is manifested in the creation of robotic devices which use artificial intelligence algorithms in their operations, movements, when solving difficult problems or communicating with humans. It is also evidenced by the introduction of new methods of reasoning about and interpreting objects or events surrounding the system. One of the fields in which the need to deploy such modern solutions is obvious are cognitive vision systems used both in mobile robots and in computer systems which recognise or interpret the meaning of recorded signals or patterns. Years ago, such applications inspired the authors of this book to undertake research on the design of computer systems, later called cognitive, which would allow the semantic meaning of registered patterns to be determined. This subject turned out to be extremely interesting, scientifically inspiring and appealing to different groups of scientists because, as became apparent later, it contributes to broad groups of varied, open scientific problems in which algorithms from this field can be used. Examples of such problems include: determining the ontology and meanings of text phrases in web searching, supporting strategic decisions (in business) and diagnostics (e.g. in medical practice), or modelling the operation of brain structures which in the future will allow the secrets of the human psyche to be revealed and might also allow computer models of the brain to be designed. The authors' research concerned mainly creating formal foundations for designing various classes of cognitive computer systems. These classes were oriented towards various interpretation tasks, e.g. identifying people, analysing the meaning of economic and marketing data, systems for diagnosing system errors, voice recognition systems, and in particular systems for image data interpreting. In this last case, the authors’ intention was to develop systems which interpret the semantics of images and thus can determine their meaning. It seems that this has been achieved for some special classes of medical images. Further in this book. we will present details of research conducted in this field. It is already known that the proposed methods are quite universal and promising. They offer a wide potential for applying similar solutions to new types of images not researched yet. This research continues. However, the most fascinating feature of science is that it keeps L. Ogiela and M.R. Ogiela: Cognitive Techniques in Visual Data Interpretation, SCI 228, pp. 1–6. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

2

1 Introduction

developing and will always present us with new challenges. But even before cognitive informatics methods become widespread and are used for various interpretation tasks, the authors already have the pleasure now to share their observations and achievements with interested readers, scientists and students, who may contribute even more to the development and popularisation of those methods in the future. Our goal is to present the most recent achievements in designing computer systems working by executing cognitive processes and oriented towards the semantic interpretation of image data. However, before we begin, this subject should be briefly outlined first. Computer data analysis processes, used for a number of years, are now moving away from tasks of simply interpreting and classifying the analysed data (e.g. using feature vectors) and are focusing mainly on reasoning about and understanding the meaning of this data. It is precisely for activities aimed at understanding the analysed data that a special class of intelligent decision-support or information systems called cognitive categorisation systems has been developed. Such systems execute not just a simple data analysis, but mainly aim at revealing the semantic information found in that data, understanding this data and reasoning based on they semantic information it contains. This process was made possible by using formalisms of linguistic perception and understanding of data for automatic reasoning processes, which are executed by following the example of biological processes of interpretation, analysis, understanding and reasoning taking place in the human brain. Analysis, interpreting and reasoning processes were used to design and describe new classes of cognitive categorisation systems aimed at the in-depth analysis of data and reasoning about its meaning based on the information about its occurrence. The idea of the presented cognitive categorisation systems has been described, among other works, in the following publications of the authors of this book [43, 54]. However, the research on using cognitive analysis methods for interpretation, understanding and reasoning tasks executed by IT systems has been and still is conducted by a scientific team made up of researchers currently employed by the AGH University of Science and Technology, Krakow. During the study of such subjects concerning systems that analyse data using cognitive data analysis methods, an idea came up to design and describe a new type of cognitive analysis systems imitating the processes of purely human, cognitive perceptory analysis aimed at categorising data according to its meaning. As a result of following those ideas about conducting cognitive categorisation, new classes of cognitive interpretation systems have been proposed. These systems will be characterised in subsequent chapters. Within the set of such systems, six classes of systems designed for analysing various types of data were distinguished, described and characterised. Then classes of systems were selected using which in-depth semantic analysis was conducted. A novel approach to data interpretation and analysis was mainly proposed within the UBIAS (Understanding Based Image Analysis Systems) class, which is used for the cognitive categorisation of image-type data

1 Introduction

3

with particular emphasis on selected types of medical visualisations. Examples of such solutions described further in this book are systems dedicated to interpreting two selected types of medical diagnostic images, namely those showing foot bones, acquired in three different projections, and images of long bone fractures in extremities. The results of research work conducted by the authors of this book for other classes of medical visualisations have also been presented in the following publications: [68, 69, 95]. In our research work, when formulating concepts and ideas of cognitive data analysis systems and when detailing those ideas, we used the experience from previous teamwork to characterise an innovative class of UBMSS systems, proposed and designed by Professor Ryszard Tadeusiewicz [89, 90, 96]. These systems were proposed as ones capable of performing a cognitive analysis of data used in processes of taking economic decisions, particularly those strategic for an enterprise. This type of systems was developed as a kind of supplement to the UBIAS class and was to show that cognitive analysis processes can be used not only for image patterns, but just as well for interpreting and understanding economic data. However, it must be emphasised that UBMSS systems, although they turned out to be a solution very novel from the scientific point of view, have not yet been fully implemented in practice because of difficulties with obtaining representative economic data. Still, work in that direction continues. Notwithstanding the above, the proposal of these systems elicited wide interest and significant recognition from scientists dealing with this subject of research. However, the main topic of this monograph will be to characterise new and only recently developed classes of information systems for interpreting and analysing image data as well as reasoning about it using the cognitive categorisation concept. The concept of cognitive categorisation systems allows us to present their key purposes, characteristics and - very importantly - the novel elements these systems include. This will be done by presenting a selected class of cognitive categorisation systems, namely UBIAS systems which analyse patterns recorded as images. The operation of such cognitive categorisation systems is founded on adapting the specific course of thought, cognitive and reasoning processes taking place in the human mind, which ultimately allow the meaning of patterns presented to be understood. The most important element in the presented analysis and reasoning process is that functions based on cognitive categorisation which leads to semantic reasoning occur during both the human cognitive/thinking process and the system's information/reasoning process that conducts an in-depth interpretation and analysis of data. It should be added that this process is based on the cognitive resonance phenomenon which occurs during the though process - which phenomenon forms the starting point for the process of data understanding - consisting in extracting the semantic information and the meaning contained in the analysed type of data, e.g. images [41].

4

1 Introduction

Cognitive resonance is an attempt to compare and distinguish certain similarities and differences between the set of features of analysed data and the set of features represented by a knowledge base. The set of analysed data is used for its broadly-understood analysis (the analysis of the form, the contents, the meanings, the shape etc.), as a result of which it also becomes possible to extract certain significant features or values of the analysed data. At the same time, the collected and possessed set of information or knowledge about the specific objects is used to generate expectations about the analysed data by referring to the knowledge resource held by the system. Those expectations are compared to the features of the analysed data extracted during the analysis process. Then, as a result of comparing the features and expectations, cognitive resonance occurs, which consists in indicating the similarities that appear between the analysed dataset and the generated set of expectations as to the possible consequences of the knowledge acquired by the system. The similarities are revealed during the comparative analysis conducted by the system, in the course of which the analysed data is subjected to the phenomenon of understanding. The reasoning process which constitutes the result of the understanding process is an indispensable factor for the correct data analysis, as if it did not occur, it would become impossible to forecast and reason as to the future of the phenomenon being studied. So conducting the analysis without the reasoning process could actually lead to impoverishing the entire analysis process, as it would be limited only to understanding the reasons for the occurrence of the analysed phenomenon, but without a chance of determining its further development. The ability to conduct a complete analysis of image-type data arose as a result of transferring cognitive and thought processes occurring in biological structures (e.g. in the human brain) to information systems, as a result of which information systems can analyse data using the foundations of cognitive categorisation. The description of cognitive categorisation processes was made possible by theoretical foundations taken from psychology and philosophy. The advantage of these processes is that they can be applied in a universal and interdisciplinary way and that it is possible to design the correct execution of the aforementioned cognitive processes. The cognitive categorisation subject is now very intensively developed and used in research, but we must be aware that this subject is founded not only on biology or social studies, but also science. Cognitive categorisation originates from cognitive science, in which the notions of thought, cognitive or analysis processes have been unambiguously defined. These known notions of cognitive processes will be used in this book to present and discuss information systems analysing image data, using processes referred to in psychology and philosophy as cognitive processes. The essence and the novelty of the approach, presented further in this book, to data interpreting and analysis as well as to the process of reasoning based on the analysed data is the structure of cognitive categorisation systems designed to run decision-making processes using semantic data categorisation methods. What is

1 Introduction

5

also novel is the presentation of a new class of cognitive categorisation systems UBIAS - designed for analysing, interpreting and categorising data which has not been subjected to such an analysis before. The authors of this work will present the results of their research on the cognitive categorisation of image data using two types of medical images. These will be images portraying various types of foot bone deformations recorded by X-ray examinations in three different projections (dorsoplanar, internal lateral and external lateral). The second type of images are those of long bone fractures. Meaning-based analysis of this type of data has not been researched by other authors before, so the results obtained represent original scientific achievements illustrating the wide potential for using the UBIAS system class we developed for the computer interpretation of image data. The cognitive analysis of image data and the semantic interpretation of the analysed data aimed at executing computer processes of data understanding and categorisation can be performed using formalisms of linguistic perception and semantic reasoning based on graph and sequential syntactic methods [20, 83]. Advanced graph formalisms were used in UBIAS systems analysing foot bone images, while sequential formalisms were applied for the cognitive categorisation of long bone fractures. The presented approach to the subject of broadly-understood data analysis originated from a newly defined branch of science called cognitive informatics. We describe cognitive informatics as a newly-defined scientific field on purpose, because even though this branch of informatics has been developed by many scientists and researchers in the past years, its formal definition appeared only relatively recently as an attempt to formalise the tasks that this field of science deals with. This is because informatics has always been treated as a branch of sciences about learning (particularly in the philosophical approach), and therefore of cognitive science. However, it was only recently that the role and significance that cognitive informatics plays in cognitive processes was recognised. The following chapters of this book will present key issues concerning the neurophysiological aspects of executing cognitive thought processes and the basics of cognitive informatics and new proposals of UBIAS systems dedicated to the meaning-based analysis of selected types of medical images. In particular, to structure the considerations of pattern classification methods, Chapter 2 will discuss traditional image recognition techniques and algorithms from the simplest methods based on metric spaces up to methods that use the paradigms of computer image understanding. Chapter 3 will deal with the cognitive aspects of brain function. Information from this chapter will allow us, in a latter part of this book, to show functional analogies between the operation of biological systems and computer implementations. Chapter 4 will provide a short compendium of knowledge about the new branch of informatics which formally describes thought processes, namely cognitive informatics. The introduction to subjects of cognitive processes analysed by

6

1 Introduction

cognitive informatics will then allow us to introduce new classes of computer systems executing cognitive resonance processes. The following Chapter 5 will allow us to define a new class of information systems using cognitive resonance processes. This chapter will review several proposals of various classes of cognitive categorisation systems put forward by the authors. Chapter 6 will contain a broader discussion of the UBIAS system class which the authors proposed for the meaning-based analysis of medical images. Then, Chapter 7 will discuss in detail two examples of UBIAS systems built for the semantic classification of foot bone X-rays and images of long bone injuries in extremities. Chapter 8, the last, will compile and summarise information on creating cognitive vision systems designed for the semantic classification of patterns. So we present this book to Readers in the hope that it will stir their fascination with the scientific aspects of creating new generation computer systems which imitate thought processes and can determine the meaning of complex image patterns. Lidia Ogiela & Marek R. Ogiela

2 Traditional pattern recognition technigues and latest image interpretation approaches This chapter briefly describes aspects of the traditional approach to pattern recognition and classification. Among the number of known methods developed over the years, we can distinguish structural analysis and pattern classification algorithms which the authors have used in their research work for many years to study selected classes of medical images [69]. This chapter will provide a brief summary of key information about traditional approaches to image analysis, and present the methods, developed in recent years, of semantically classifying selected patterns [69]. Structuring information about these subjects will help us, in the following chapters, to raise the possibility of using cognitive and associational techniques for a deeper, meaning-based classification of images.

2. 1 Characteristics of pattern recognition algorithms Fundamental works on pattern recognition methods [17, 95] define pattern recognition as an attempt to identify whether the patterns analysed or objects shown in them belong to certain predefined classes. Such recognition can be performed using various classification methods which, when correctly selected, will allow the specific characteristics of the image to be used to interpret it properly or to allocate it to a defined class of objects. Executing such tasks is usually simple and necessitates completing the stage of image pre-processing (e.g. using simple image algebra procedures), and then analysing it in order to determine certain material distinctive features, which will then be used to classify that image and will allow the object studied to be unanimously recognised (in the sense of assigning it to a given class). By comparing our considerations to the steps taken by persons recognising new patterns we can risk saying that after the material characteristics have been identified, recognising the object boils down to implementing an algorithm which uses the same method by which an expert selects the right class. A very pertinent example confirming this is the computer analysis of medical images. This is because in this application, for many diseases you can try to define certain standards which allow the image of a healthy patient to be distinguished from an image suggesting that some pathologies are present. It has to be added, however, that in the case of medical images recognition based on the application of strict rules of interpreting certain traits of the image can only be used for simple (mainly from the diagnostic point of view) images. In the case of many complicated types of images which can be acquired in medical diagnostics there is no a'priori information about the rules whereby these images belong to specific classes that would indicate, for instance, the type or progress of disease processes. In such a situation the information available to the recognition algorithm for using can be contained in the so-called training set L. Ogiela and M.R. Ogiela: Cognitive Techniques in Visual Data Interpretation, SCI 228, pp. 7–27. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

8

2 Traditional pattern recognition technigues and latest image interpretation approaches

formed of cases diagnosed and confirmed earlier [36]. The ability to refer to a training set is very desirable and interesting from the point of view of improving the efficacy of recognising subsequent difficult and complicated images. This is done by searching for certain similarities to cases already known and forming certain associations based on the closeness (similarity) of new patterns to those already known. Pattern recognition has, from the very beginning, constituted one of the most interesting artificial intelligence applications due to the frequent lack of strictly defined rules determining the nature of the images recognised. In practice this means that the computer, just like a person who does not know certain types of patterns, must learn to recognise new cases based on examples presented previously. Such software keeps improving systematically and as the number of interpreted cases grows, the effectiveness of the classification performed rises. Fortunately, in the medical pattern recognition considered here, it is generally quite easy to collect the requisite set of examples (e.g. from diagnostic studies or clinical databases etc.), together with recognitions verified by the therapy that followed them and its results. The key to training pattern recognition algorithms is generalisation, which means that such an algorithm can not only collect knowledge based on the patterns provided to it, but after the completed training it can recognise completely new cases which had not been a part of the training sequence. If such images are in some sense similar to the representatives from the training sequence, then defined classification functions will be used to classify them. However, if they are completely different or dissimilar to the training items in the sense of the adopted membership rules, we should suppose they will form the beginning of a new class of patterns whose name has not been defined yet. This way, image recognition methods can be used to classify any pattern, including the aforementioned medical images, whose correct interpretation can become significant when diagnosing further patients. If relatively simple images (e.g. showing a single object and two-dimensional) which represent objects of limited possible variety are recognised, it is possible to recognise them by comparing them to the pattern already stored, which method is called template matching. In the case of medical images, for many types of images the varied nature of objects which can be recognised necessitates using such a recognition process in which the preliminary stage is to define certain material features, which facilitate the unanimous characterisation of the classified patterns, e.g. changes to the morphology of an organ or changes to its structure. To summarise this general information about pattern recognition processes we can say that such jobs are most often executed as multi-stage algorithms during which it becomes necessary to significantly reduce the quantity of data contained in the original image in order to determine a set of selected features which allow the new object to be unanimously assigned to one of known classes. Data in the form of an image should have a large size and take a lot of storage volume, as an image recorded with a high spatial resolution and a high depth of grey levels or colours can have dozens of megabytes. As we have already mentioned, the infor-

2. 1 Characteristics of pattern recognition algorithms

9

mation which will be used at the classification stage is significantly reduced during image pre-processing. This reduction is possible due to the use of a number of algorithms classified as image pre-processing, whose result is to determine the vector of features which define crucial parameters describing the recognised object. The upshot of this is that a new representation, describing the contents of the image and telling us what can be seen in the image, is created. These contents are recorded in the feature vector created, which forms the foundation of subsequent analysis stages, and is in particular used for the recognition. The recognition process itself also leads to a significant reduction of the information volume, because the completed classification leads to assigning only the label of one of the recognised classes to the object. The final recognition (classification) therefore replaces the previously determined vector of analysed features which requires a high precision of value representation. Example medical images are shown in Fig. 2.1.

A)

B)

C)

10

2 Traditional pattern recognition technigues and latest image interpretation approaches

D)

E) Fig. 2.1. Example medical images acquired during various diagnostic tests. (A) An X-ray of the cartoid artery; (B) A USG image of the gallbladder; (C) An MCG image of the heart. Left: a visualisation using an MFM contour map. Right: the corresponding PCDM map; (D) Images of the p-CT brain perfusion (from the left: CBF, CBF and TTP, respectively); (E) A spatial recognition of the coronary vascularisation from a spiral tomography test.

As below we will try to present the opportunities to use cognitive computer science for the intelligent semantic interpretation of images, here we can try to systematise the subject of pattern recognition starting with the historic assumptions and preliminarily classifying the methods considered. The development of pattern recognition methods started in the pioneering period of artificial intelligence method development in the 1950s. First practical implementations of pattern recognition techniques were based on neural network systems introduced by Rosenblatt in 1968 under the name Perceptron. These systems were founded on a structure of several levels composed of electronic artificial neurons which processed computational data including premises supporting the recognition, and output circuits which indicated final decisions. So the Perceptron was the first technical implementation of the operating model of fundamental elements of the brain, e.g. neurons. Since then, intensive work has continued to imitate thinking and interpretation processes that take place in the brains of living organisms. Early experiments in artificial intelligence and pattern recognition point to two basic conclusions which have speeded up further research:

2. 1 Characteristics of pattern recognition algorithms

11

1. Artificial intelligence algorithms must be able to learn, and thus recognise new, patterns not considered previously. 2. Natural intelligence is mainly based on recognising and perceiving patterns, and only to a negligible extent (i.e. approximately 1%) on reasoning processes [87]. This second characteristics was a strong driver accelerating the progress of research on computational intelligence and cognitive computer science (including machine image perception), as apart from scientists intending to solve specific problems (like those of recognising biometric or behavioural patterns), the subject of recognition was undertaken by many people interested in the theoretical foundations of artificial intelligence. This is why a broad spectrum of various methods of image recognition and pattern classification was developed and studied, of which many were employed for interesting identification jobs and are still used today. In line with the development of pattern classification and interpretation methods, the first that should be listed are the so-called classical pattern recognition methods. These mainly use minimum distance techniques. Their forerunner was the nearest neighbour (NN) method [6, 87, 95]. It is very intuitive and notionally simple, but can fail in certain cases, particularly if elements from the training sequence are wrongly classified. This led to the appearance of its modifications: algorithms not susceptible to errors in the training sequence, such as the NN or the jNNN methods. Although these were the forerunners of all other pattern recognition methods, they are still frequently used today, as they have a high precision of object recognition. Their drawback is their high time consumption and memory complexity. Further types of so-called classical pattern recognition methods were approximation and probabilistic methods. They represent a completely different way of creating an object classification algorithm, which, however, in many cases yields similar results and a similar recognition effectiveness as minimum distance methods, albeit with a much smaller load on the recognition system, both in terms of the memory used and the calculation time. As we have mentioned above, the aforementioned methods have successfully solved many practical problems, from automatic reading of vehicle license plates, handwriting or printed character recognition, through the biometric identification of people (fingerprints, faces etc.) to Voice Recognition Systems. Many such methods were applied in medicine to interpret diagnostic data and also to detect lesions or to model anatomic structures. So we can see that such methods are invaluable when creating modern solutions to support diagnostic and therapeutic processes. Classical methods have turned out to be very effective when classifying patterns in the form of vectors of numerical data or containing characteristics specified in a qualitative way. However, with reference to the broadest class of medical data currently found, i.e. image-type data obtained from various imaging modalities (e.g. CT, NMR, PET, USG, MCG, RTG etc.), classical pattern recognition methods are generally not very successful as they cannot recognise image pat-

12

2 Traditional pattern recognition technigues and latest image interpretation approaches

terns consisting of many objects (e.g. bones of limbs or CT images of the abdominal cavity). However, this problem turned out to be solvable as syntactic (structural) methods appeared which use mathematical linguistic formalisms. What is interesting with regard to this subject is that syntactic methods developed in 1960s and oriented towards the computer modelling and description of natural languages have been used for many years only to program computers. The ability to use such languages to describe natural grammar turned out to be very restricted. This meant that the work in cognitive science aimed at discovering the mechanism by which people learn to use a language could not use mathematical linguistics formalism to a wide extent. Only the work on new algorithms of computer image understanding [69, 95] allowed these techniques to be used and put research on the track of developing new methods (linguistic ones) to determine the semantics, i.e. the meaning of the image pattern analysed. Such methods are strictly connected with the development of machine cognitive science and research aimed at developing computer science models of reasoning and perception. Such studies are the prerogative of cognitive computer science which will be characterised in the following chapters. Coming back to the development of pattern classification methods, apparently 1980s saw a huge growth in studies of neural networks and the ability to use them to recognise images. However, it turned out that the suitability of neural networks for analysing and classifying medical images is limited as the image as such (particularly one of very high resolution) cannot be delivered to the input of a neural network, while attempts to train pattern recognition networks using image features (determined at the preliminary analysis stage) have not shown any significant advantage of neural networks over techniques based on previously mentioned classical methods. This statement seems rather obvious considering that the simplest neural network models can be associated with simple implementations of approximation methods, for which the basic functions are defined identically as the neuron activation functions. The development of all the above methods means that now there are many varied ways of identifying or interpreting different types of patterns, including medical ones. Obviously such methods are constantly developed and improved. Their number keeps increasing as newer and newer types of images appear and there is a need to develop new systems capable of analysing and recognising them. This proves how current and scientifically important the automation of classification procedures and computer recognition is. However, the multitude of these methods also shows that no single one is completely universal and suitable for every pattern recognition problem. Yet the variety of those methods means that we now have a significant range from which to select the optimum algorithm allowing even very complicated images and patterns to be recognised. Examples of such applications will be presented in subsequent chapters. In order to clearly structure further considerations, Fig. 2.2 shows a general classification of pattern recognition methods and methods that support the semantic interpretation of image patterns, which will be discussed in greater detail in subsequent chapters.

2.2 Basic stages in the pattern classification process

13

Fig. 2.2. General classification of pattern recognition methods

2.2 Basic stages in the pattern classification process As the main purpose of recognising images is to develop an algorithm allowing the analysed patterns to be finally classified based on some features distinguished in the image, we can picture the entire recognition process as a kind of mapping which assigns the name or the label of the recognised class to the input image. So this operation will be defined on a set of objects to be recognised, e.g. recorded by a camera or a diagnostic apparatus. The effect of this operation will be some name belonging to a set of classes of recognised objects. Obviously, the recognition system must account for cases in which it will not be able to classify a new pattern, and therefore sometimes no decision will be taken, which is called a neutral recognition, if the recognition system cannot complete the classification and cannot name the recorded objects. So pattern recognition denotes the entire process of input image analysis, starting from its recording until its final classification. However, for this classification to be possible, certain stages have to be completed in the entire process, ending in the specific object being recognised. In particular, the recognition system must complete the following tasks: 1. Perception – which consists in determining the most important features of the registered patterns. Such features should be distinctive characteristics supporting the unanimous identification of the image. 2. Determining the degree of membership or similarity – at this stage the system must use the features distinguished to define to what extent new objects resemble objects belonging to particular recognised classes; 3. Correct classification – this consists in unanimously assigning the new image to one of the known classes or in generating a message that this classification is impossible. In accordance with the introduced classification function, a decision is taken that the object belongs to the class to which its degree of membership or similarity is the greatest. All these stages are detailed in book [17]. The most simplified diagram of the recognition process is presented in Fig. 2.3.

14

2 Traditional pattern recognition technigues and latest image interpretation approaches

Fig. 2.3. Stages in medical pattern recognition

The most important stage in the recognition is to map all the objects considered, which are to be recognised, as points in the so-called feature space. This space is created when a certain number of distinctive features, whose values can be defined qualitatively or measured quantitatively, can be identified in every object being recognised. Features identified in that way allow certain groupings defining areas occupied by objects belonging to individual classes to be identified in the space of those features.

2.3 Basic methods of pattern recognition This sub-chapter lists and briefly characterises basic methods used to classify images. These methods are called classical as they were developed in the early stages of scientific research on computer image analysis.

2.3.1 Metrics-based methods Some of the first pattern recognition methods were those based on metrics defined in the considered feature spaces and called minimum distance methods. In these methods, the mapping which defines the degree to which a new pattern belongs to specific classes is connected with the notion of distance calculated in the space of features representing the given objects. Obviously, the feature space must be equipped with the correctly defined metric. This metric can be any mapping which fulfils certain axiomatic conditions [17]. Of course, some of the most popular metrics frequently used in practical pattern recognition are the Euclidean and Czebyszew metrics.

2.3 Basic methods of pattern recognition

15

The classification of unknown patterns using minimum distance methods consists in selecting, as the correct identification, the number or the name of the class to which the closest pattern belongs. Distances between these patterns are obviously measured using the previously selected metric. Examples of such methods are the nearest neighbour (NN) and the α-nearest neighbour (α-NN) methods. The nearest neighbour (NN) method is the simplest metric-based method in which the decision making rule assumes that the unknown object will be classified to the class to which the object of the training sequence located closest to it (in the sense of the defined metric or the similarity function) in the feature space belongs. This method is frequently used to classify and group objects on satellite or radiotelescope photographs, as well as when analysing medical images. These methods are very simple and intuitive, but their correct operation depends on the ability to use the training sequence. A modification of this method is its extension into a variety which uses the information about the α closest neighbours from the training sequence to take the decision about the membership. This is the α-NN – Nearest Neighbours method. It is easy to perceive that the general case of the NN method was very susceptible to potential classification errors that could occur in the training sequence. In the classical method, for such incorrectly classified elements, also their entire neighbourhood (that is the set of elements located the closest) will be incorrectly classified and assigned to the same group as the incorrectly identified element from the training sequence. In order to make such wrong classifications less likely, a modification is introduced in the NN method so that the decision is taken based on a greater number of objects from the training sequence (i.e. α elements). The objects considered are located the closest to the recognised object of a still unknown membership. The final classification process in this method is executed in such a way that after determining the α number of objects for the new pattern, its distances from all elements contained in the training sequence are calculated. At the end, the new pattern is classified to the class which is most numerously represented among the α considered elements closest in the entire training sequence. When generally characterising methods based on metrics we can state that they are frequently used in practice due to their simplicity and reliability as well as high effectiveness. However, their disadvantage lies in the high computational and storage complexity of those algorithms, which is due to the need to allocate the entire training sequence in the form of known patterns and their corresponding classifications.

2.3.2 Methods based on function series Another class of pattern recognition methods are those based on function series, also known as approximation methods. In these methods the function of member-

16

2 Traditional pattern recognition technigues and latest image interpretation approaches

ship (similarity) of the recognised pattern in (to) particular classes is calculated by expanding a defined family of base functions into a function series. Assuming the notation where the membership function of a new object marked as x is represented by a function marked C, we can generally write the function as:

m i i C ( x ) = ∑ Vv ϕ v ( x ) v =0 where i is the number of the class for which this membership function is calculated and ϕ denotes selected base functions. With this approach it is easy to notice that once the forms of base functions for this expansion have been selected, it only becomes necessary to determine the i i value of V v expansion coefficients which allow the value of subsequent C (x) membership functions to be determined. Such weights are determined in the learning process using patterns from a given training series and the correct diagnoses known for them. The problem of selecting base functions has been discussed at length in [17]. As in some special cases of base function forms the above expansion may have the form of a linear function [17], the recognition rule can then be defined as a membership function linearly separating object classes (Fig. 2.4). In practical recognition jobs, due to the varied type of object features analysed, we can never be sure whether the classes considered are linearly separable using linear hyperplanes, so it is reasonable to try to use an expansion with non-linear functions if the classifier generates too many errors (Fig. 2.4).

Fig. 2.4. An example of separating classes of gallbladder diseases. (A) Linearly separable classes determining a hydrocele and a fold/turn of the gallbladder; (B) linearly inseparable classes determining the fold/turn and a tumour of the gallbladder

The advantage of using approximation methods is that the recognition method itself is relatively easy to execute and the number of elements that must be stored is small compared to the size of the training sequence. This is because it is enough i to store the determined V v weight coefficients and the expansion into series with-

2.3 Basic methods of pattern recognition

17

out the need to store the entire training sequence use to define the values of expansion weights. If for the case of a linear separation of classes membership functions do not generate satisfactory recognition results, it becomes necessary to modify them into the form of non-linear functions separating classes of any shape in the feature space.

2.3.3 Methods based on probability distributions Classical pattern classification also includes probabilistic methods. Here the recognition algorithms can be divided into supervised and unsupervised methods. Recognition is supervised when it refers to patterns preset previously. Unsupervised recognition takes place without reference to preset patterns, which are not available. Such a situation occurs if there are no patterns from the training sequence. In such classification methods the recognised patterns are included in the most probable classes. Decision-making rules can be created in many ways. Usually the maximum probability values or the minimum values of the classification error are used. Recognising images by statistical methods assumes that for every class, the probabilities of objects from particular classes occurring and the conditional probabilities of objects occurring in particular classes P(wi|x) and P(wi) are known. One of the most popular and simplest methods of this type is the method based on the Bayes' rule. The method using the calculation of the Bayes’ a posteriori probability is the basic statistical method applied to classify images. This technique is based on the assumption that the decision-making problem is formulated in a probabilistic way and that significant probabilistic parameters are preset, like the frequency of occurrence of elements from particular classes or probability density functions. The best way to illustrate this will be to use two classes, c1,c2, as examples. The a priori probabilities of these classes P(c1) i P(c2) are known from calculations of the available training set. We also know the probability density distribution functions p ( xi | ci ), i = 1,2 .

Functions p ( xi | ci ) can also be called the functions of the conditional probability that elements xi will occur in class ci. p ( x | ci ) P( ci ) A reference to the Bayes’ rule yields P( ci | x ) = p( x ) where p(x) is the total probability determined by the formula: 2 p ( x ) = ∑ p ( x | ci ) P( ci ) i =1

The Bayes’ classification rule for the case of two classes c1, c2, is as follows:

18

2 Traditional pattern recognition technigues and latest image interpretation approaches

If P(c1|x) > P(c2|x), then x is recognised as element c1 If P(c1|x) < P(c2|x), then x is recognised as element c2 Of the known classification methods, statistical ones are considered to be of poor precision and frequently yield large recognition errors. Of course, this depends on the specific data classification case. However, they are often useful to verify the regularity of results obtained using methods other than statistical algorithms, e.g. approximate or metrics-based methods. Even though not very efficient, these methods are frequently used to classify medical patterns. This is because diagnostic processes are frequently based on population observations. One of the main applications of such methods are to diagnose the cancer of various organs. A number of interesting applications is presented in [87].

2.4 Structural methods for describing and recognising images The methods characterised in previous chapters, regardless of their many practical applications, numerous advantages due to their simplicity, and sometimes effective operation in selected cases, may turn out to be insufficient or not very effective in others. This is particularly noticeable when analysing image-type patterns (e.g. photos and images) that are highly complex. In practice, this also requires a large number of recognised classes. In such situations the methods described previously would require considering feature vectors of many components. Such limitations do not apply to syntactic methods of pattern recognition which use formalisms of mathematical linguistics [95]. When characterising the way in which these methods operate we can see that in the syntactic approach, a complex image is reduced into simpler sub-images, and the next step is an attempt at recognising those by treating them as independent units. If the sub-images are still highly complex (many objects, complicated shapes or spatial relationships), the splitting operation is continued until it yields indivisible picture components referred to as picture primitives. Picture primitives are elements which we can assume exist and can be easily defined for a given image class (e.g. as a result of decomposing the original image). So as you can see, for syntactic methods, before you can start the recognition proper, you have to isolate picture primitives and point out relations between them. The most frequent method of distinguishing picture primitives is eliminating excessive information and decomposing the shape by image pre-processing. Pre-processing can consist, for instance, in segmenting or detecting edges using gradient operators or pattern matching operators. By executing such procedures, we eliminate immaterial information like the image background from the original image and at the same time we can move towards creating its new representation, e.g. in the form of edges of objects that interest us. Pattern identification is an extremely important stage because, when recognising images (in this case picture primitives), we treat them as independent sub-images. Consequently, the correct composition of the entire im-

2.4 Structural methods for describing and recognising images

19

age from sub-images determines the possibility and the effectiveness of its subsequent correct analysis. Apart from the need to distinguish picture primitives, another important element in the syntactic approach to recognising a given image is to identify the relations between such primitives. This is because these relations determine the correct composition of the image, and thus the right recognition of its individual elements. The character of relations is also a very important criterion for the basic classification of syntactic pattern recognition methods. In the case of multi-object images, two relational sets are frequently defined. One of them is used to describe the basic component structure distinguished in the input image and independently recognised. The second is used to define mutual relations between the recognised objects, all of which make up the input image. Syntactic pattern recognition methods are classified into three main groups. The first and earliest developed group are sequence methods based on parsing formal languages, most frequently context-free ones. More complex methods are tree methods and graph methods. These three groups of methods will be briefly characterised below. In terms of the simplicity of formalisms, sequence methods are the first group. In these methods the image is represented by a sequence. This means that only one kind of relationship that exist between picture primitives can be distinguished. This relation is the concatenation of subsequent elements so that together they form one sequence. Sequential languages are suitable for recognising and describing single (frequently very complex) objects in an image. Examples of such languages: languages based on chain codes (Freeman ones), PDL picture description languages and LSFD languages for shape feature description. It should be noted that such languages do not support describing images containing many different objects. This capacity is however offered by tree or graph languages. Sequence, tree and graph formalisms have been successfully used to analyse satellite images and examine medical images. Detailed cases of recognising various complicated shapes of organs in the abdominal and chest cavities can be found in [69]. Somewhat different analysis capabilities are provided by tree- and graph-based methods. Tree methods are constructed based on formalisms that define mathematical trees. Two applications of such methods are most widespread: scene analysis and texture analysis. Scene analysis is a process for recognising a 2D or 3D image in which objects that make it up have been distinguished and identified and relations between these objects have been described. Here we distinguish two sets of picture primitives. The first is the set of objects making up the image, and the second is the description of relations between these objects. Defining relations between objects allows the image to be interpreted as a whole apart from recognising individual elements. It should be noted that graph grammars are the most robust tools used to recognise images. This is why literature makes many mentions of using graphs to describe 2D or 3D images. However, their use for recognising medical images is not so widespread, as the syntactic analysis of graph grammars is very difficult. In subsequent chapters dealing with cognitive systems for image analysis we will

20

2 Traditional pattern recognition technigues and latest image interpretation approaches

present specific examples of using graph methods to recognise complicated medical images and interpret their meaning. This will, in particular, concern the semantic description with the use of graph grammars that model lesions shown on foot images. Such images are very complicated, mainly due to the large number of bones they show, and also the need to analyse them in various projections. Such an interesting task will be the subject of a subsequent chapter. In addition, we will present the opportunities for using ordinary sequence grammars on the example of an analysis of injury lesions of long bones of extremities.

2.5 Methods of determining image semantics When analysing and recognising image patterns, we usually strive to recognise some material characteristics or elements of the image and classify (name) the patterns processed. However, there are image classes that have major layers of semantics, i.e. meaning. This is why recent years have seen a lot of focus on developing intelligent methods that could determine the semantics of images. Obviously, not all types of images can be interpreted with a view to the meaning of the information they contain. The broadest class of patterns which have semantic significance are medical images portraying various kinds of lesions, acquired during various diagnostic tests. The authors have already proposed procedures for the automatic analysis and understanding of image semantics for a number of image classes [95]. However, it should be noted that automatic image understanding is still a new research concept founded on the use of advanced techniques of computational intelligence. In this concept, we take advantage of cognitive reasoning mechanisms, which are very distant from all pattern recognition techniques described above [95]. These techniques represent a significant novelty when compared to all known classical image classification methods as they support automatic machine perception which reaches much deeper into the meaning of the information contained in the image. In the automatic understanding of medical images we try to reach the semantic significance of all details detected in the specific image as a result of analysing it. This probing into the contents of the image leads to the attempt to understand the nature of disease processes which have led to this and no other appearance of the examined organ in the image. This makes it possible to identify the disease entities to which the medical problem under consideration should be classified and to support therapeutic and diagnostic processes for that case. As a result, automatic understanding can be a very valuable tool supporting diagnostic processes. However, understanding the nature of the disease means more than simply classifying it, i.e. assigning the name of the visible disease to the image. It can be said that automatic medical image understanding brings significant new content and offers new opportunities in the artificial intelligence field. Lower down in this chapter we will present our methodology connected with methods of image understanding and determining their semantics. Then, further chapters will present the directions in which such techniques can

2.5 Methods of determining image semantics

21

later develop based on models of the cognitive perception and understanding that takes place as part of the thinking processes in brain structures.

2.5.1 Algorithms for determining image semantics It has been said previously that the development of vision systems and pattern classification methods has allowed image analysis systems to be turned to finding and identifying image semantics. Such capacities appeared as computer technologies and algorithms modelling thinking processes developed. These algorithms are not perfect yet, but we can suspect that as time passes, such methods will become more and more effective along with the improving ability to imitate perception processes taking place in the human mind. However, as we know from the development of information methods and artificial intelligence algorithms, the attempt to entrust more and more of our information activities to computers have frequently led to a situation where an action apparently obvious and self-evident for a human (often done spontaneously without thinking much about its nature) became extremely hard to define algorithmically and execute using computers. It is worth noting that typical activities such as making calculations, processing information, collecting results, communicating and transferring data did not cause such difficulties, and employing computers for them actually streamlined them and made human activity easier. So computers are a useful tool for information management. However, we need to note that progress in medical imaging, for instance, means that we now have a huge number of various medical images available, which we need not only to correctly process, annotate or send to another institution, but no less often we want to correctly interpret their meaning, i.e. understand the import of the lesions shown in such images. Present day medical physics not only provides the physician with data showing images of internal organs and disease processes visible in them, but also allows these visualisations to be improved and classified. Medical imaging technologies currently offer as many as three basic types of methods related to analysing the acquired images. These methods, in the chronological order, are: • Preprocessing – whose purpose is to extract significant image elements or improve the legibility of details observed in the image. Here it is possible to detect edges, count elements, segment or change the tone curve to reveal selected ranges of grey levels. • Image analysis – aimed at determining the feature vector for pattern classification. We have described classification methods based on vectors of distinctive features in previous chapters. It is at this stage that the values of such features are determined. They may take various forms, from numerical values (e.g. location coordinates, through shape coefficient values, morphometric features etc.) to qualitative features.

22

2 Traditional pattern recognition technigues and latest image interpretation approaches

• Pattern classification – this recognition stage is aimed at identifying and classifying certain elements, regions or entire images to certain predefined classes. This classification is performed using the feature vector containing quantitative or qualitative data for analysed patterns. Figure 2.5 contrasts the capabilities of the above techniques, which represent the capabilities of classical recognition procedures, with those of semantic analysis techniques.

Fig. 2.5. Interdependencies between classical elements in the image interpretation process and image understanding techniques

It is just worth noting that the three above sets of computer image processing and classification methods have very many applications. In particular, in medical diagnostics, these methods now form very useful tools supporting and streamlining physicians’ work. Their use now gives the diagnostician a much better view of the details of examined organs and the physiological or disease processes occurring in them which are of interest to him/her. This allows him/her also to take a better, more reliable diagnostic decision. Also due to the results of all analyses of medical image characteristics and the objects shown in that image that are accessible with computers, the physician can base his/her reasoning on premises much more reliable and measurable than just the visual assessment of the image, which makes his/her actions more effective and gives him/her more certainty and security. Finally, the increasingly popular techniques of automatic recognition and classification of biological objects distinguished in medical images can help make the right diagnosis and formulate therapeutic plans. All such techniques of the computer acquisition and recognition of significant changes evidencing disease processes do not exhaust the full range of tasks and problems that may appear due to employing an image as a source of useful diagnostic information. This is because the most important and the hardest thing in this whole affair is to correctly interpret such images and determine their meaning. This is of particular importance to the patient and the prognosis for the possible further improvement of his/her health. Semantic interpretation (medical image understanding) methods, just like specialists in the specific area of medicine, not only determine how the shapes of organs and their lesions caused by disease processes can be not better described or shown in the image, but also try to define their

2.5 Methods of determining image semantics

23

medical meaning and identify the actual significance of these and not other lesions being visible in the image. In the practice to-date, the process of isolating and defining image characteristics which accompanied the further analysis and attempts at the automatic classification only provided certain premises for the reasoning process. As such, the process of reasoning about the significance of the lesions observed was (and still is) conducted by the physician, who used his/her knowledge, experience and also intuition. Without such content interpretation of the analysed images, the classification results are almost worthless, as for the diagnostic and therapeutic activities, the value of a specific parameter and the class to which this or that object visible in the image can be assigned is of no practical importance. What is important is what it all means for the current condition of the patient, the disease he/she suffers from, and for defining the best therapy or rehabilitation plans and assessing their success. All the present-day technology that delivers many various medical images and the entire IT technology of the classical Image Vision & Interpretation methods depend on the main factor which is the process by which a human understands the examined image. If you account for the possible interpretation problems caused by the constantly developing, but therefore also constantly changing modern medical visualisation, and for how many different organs are examined and in how many different ways they can be deformed by the disease process, you will not be surprised that image diagnosis needs urgent support from modern IT technologies. Consequently, as computer technologies and new medical imaging forms (like the MCG magnetocardiograms - measurements of the heart magnetic vector) develop, the demand grows for better and better techniques of computer assistance for the process of analysing, interpreting, classifying and recognising such new images. This is significant, as at the current IT development, computer systems can easily modify and interpret the form of an image (e.g. change its geometry or find patterns by reference to the colour properties of the image), but the meaning of the image still remains completely inaccessible for the automation sphere. The authors have shown in their previous publications [69] that in medical diagnostics images of a completely different form can have the same contents. This is illustrated by Figure 2.6, which shows a selected medical problem in the form of completely dissimilar images recorded for different patients, yet which lead to the same medical diagnosis.

Fig. 2.6. Examples of pancreas USG images in which pancreatic cancer can be identified. Optically, these images are completely different from one another

24

2 Traditional pattern recognition technigues and latest image interpretation approaches

Yet if the form of an image is not connected with its contents, we urgently need tools that could determine the meaning of this image obscured by its complex form. And semantic image interpretation techniques automatically extract a certain significant meaning contained in the image, but not directly visible in it.

2.5.2 Methods based on formal languages for determining image meanings One known method that makes the meaning analysis of various classes of medical images possible is an approach which uses formal languages and methods of syntactic pattern recognition enhanced with the capacity of defining special procedures determining semantic parameters. As a result of the conducted analysis, such procedures facilitate not just recognising the lesion visible in the image, but also interpreting it to determine the disease entity, its advancement, as well as to try to identify recommended therapeutic directions. Using such an approach is extremely convenient as linguistic formalisms are universal, so various classes of medical images in which morphology changes of the examined images are shown can be interpreted. The procedure of the semantic interpretation of an image using image languages can be divided into the following stages: 1. Defining regions of interest (ROI) by isolating the examined structure from the image. This is usually done by the pre-processing operation. 2. Introducing a new linguistic representation for the analysed object using a linguistic description. This description allows the shape (the morphology) of the examined structure and the lesions caused by progressing disease processes to be defined. 3. Defining the linguistic formalism in the form of a selected image grammar which will describe the possible lesions. 4. Implementing a reasoning system in the form of a syntactic and a semantic analyser recognising new images using elements of the introduced grammar. This analyser should classify new patterns and generate a so-called semantic record containing information on the type and number of recognised lesions; it should also determine their medical significance. Depending on the nature and complexity level of the image, images showing either one medical structure (examined organ) or many objects can be subjected to meaning interpretation. The class of complex patterns which can also be interpreted using grammar rules can also include 3D spatial reconstructions of internal organs obtained by spiral tomography. Methods of applying graph grammars to specific tasks of the semantic description and analysis of complex images will be presented in subsequent chapters. Be-

2.5 Methods of determining image semantics

25

low, we will just present a general methodology leading to interpreting a complex image containing numerous objects. It consists of executing the following analysis stages: 1. As the potential image contains many independent objects, it should be analysed with graph methods which support the simple definition of the location of every object relative to others shown in the image. 2. In order to create a new representation using grammar rules for a given image, you should define the set of basic objects present in the image and certain mutual relations between them. 3. Then, having derived relational sets, you can replace the input image with a new representation having the form of a graph described using picture primitives and the relations defined for them. 4. The new representation will be analysed to detect significant lesions in the image. However, before doing this, you can use the appropriate methods of converting graphs to the bracket notation to obtain the final representation of the analysed image as a series. This series will be analysed by the analyser. 5. The last step is to implement the recognition procedure, which will compare the representation of the new object with the rules of the grammar derived earlier using a representative set of analysed cases (e.g. a training series). What is characteristic in the automatic determination of the meaning of images is the stage at which selected features of the analysed pattern are compared to the set of expected features if that image may have a certain meaning of interest to us. So when developing systems for automatically determining image semantics we must include in their structure a significant resource of information (knowledge) which can be used to formulate such expectations. The method of using collected knowledge in processes of reasoning out the meaning of new patterns is schematically presented in Figure 2.7. This diagram gives a very simplified picture of how the semantics determination process runs, i.e. how the computer attempts to understand the contents of the input image.

Fig. 2.7. Semantic reasoning process of intelligent vision systems

The diagram shows that the examined input image is analysed with methods briefly characterised in sub-chapters above. As a result, the system gets a reasona-

26

2 Traditional pattern recognition technigues and latest image interpretation approaches

bly precise vector of information (or features) concerning the components making up the original image, as well as information on their mutual relations. Most frequently such an analysis yields a representation in the form of a sequence of characters describing significant features of the object shown in the image. This description is written in a specific image description language defined by the introduced grammar applied in the system discussed. The second important source of information indispensable for defining the meaning of the analysed pattern in this procedure is the system’s internal resource of knowledge. It can have the form of a database, formal rules or knowledge recorded in a different representation. Such a knowledge resource is most frequently empirically acquired from experts who can interpret patterns of classes considered by the system. In the case of the automatic understanding of medical images, such information comes from experienced diagnosticians who, in their professional practice, have acquired the skills of understanding the medical meaning of various categories of medical images. Such an internal resource of knowledge in the analysing system serves as the source of certain semantic expectations with which features detected in the input image should be consistent. These two streams of information (expectations from the knowledge base and the actual features) are compared and interfere as part of a process called the cognitive resonance. Models of psychological resonance processes taking place in the human mind are discussed in subsequent chapters. However, we can already briefly observe that the interaction runs as follows: let us imagine that as a result of analysing and describing a new image we have a sequence of symbols generated by the procedure executing the linguistic (in the sense of the derived image grammar) image analysis. This description of the contents of the analysed image in the computer representation can have the form of any sequence written with special symbols [69]. A set of new symbols (called the set of terminal symbols of the grammar) is derived according to the methodology of using mathematical linguistics methods. Recognition procedures can quickly interpret such notation and detect in it significant elements responsible for the appearance of some special strings (e.g. evidencing lesions). Using an analogy from molecular biology we could say that this notation can be compared to the process of finding strings of nucleotides that determine the presence of certain genes in DNA chains. The operation of the recognition system consists in detecting special strings that describe some specific patterns defined by the internal knowledge source within such sequences. Obviously, there are certain expectations here as to the form of lesions detected. Let us imagine that the system wants to check if there are symptoms of lesions typical for cancer in the analysed sequence describing an image. The system then generates the pattern of such a cancer lesion. Then, this pattern can be found anywhere in the entire examined image (that is in the sequence of symbols that describe it) by sequentially matching the string looked for to subsequent fragments of the complete description of the analysed image. In this simple way, the location of the lesion looked for is found. In the case of medical images, this lesion may have the nature of a pathology and a certain significance for the patient.

2.5 Methods of determining image semantics

27

In the described procedure, reconciling the searched-for pattern resulting from the given hypothesis with the linguistic description of the image does not yet prove the hypothesis to be right. Obviously the internal source of knowledge may generate many hypothetical meaning descriptions of the analysed image, every of which hypotheses has its related patterns necessary to verify it. However, generating and verifying subsequent hypotheses and their related code fragments which should appear in the input description of the analysed image means that some hypotheses result in a positive resonance (the significance of the specific feature is amplified when it is used to determine the meaning) while others expire and in a sense dampen the specific premise, excluding it from the stage at which the meaning of the entire image is determined. A meaning interpretation formulated using this mechanism is the more reliable the stronger the amplifying resonance. Consequently, in some circumstances, the meaning of the analysed image can be understood precisely and finally, while in others we only get weak hypotheses which are difficult to confirm finally. However, we should remember that a person, when he/she tries to understand a novel situation (or define the meaning of an unknown pattern) also does not always reach full certainty. Details of the processes described here, which use cognitive reasoning and brain function models, are of course complicated and will be enlarged upon in subsequent chapters. In their scientific research, the authors have dealt with recognising and the simple computer determination of the meaning of medical images for many different diagnostic images. Details of this research are presented in publications [69, 95]. In subsequent chapters we will present techniques for determining the meaning of and for the semantic reasoning about new classes of medical images. In particular, those will be images of foot bones and long bones of extremities. For those classes, we will propose new models of cognitive reasoning and present capabilities of interpreting them using image languages.

3 Cognitive aspects performed in the human mind 3.1 Brain science Since time immemorial man has been fascinated by the structure of his mind and has striven to learn the secrets of its operation. The human image of the world, all that we believe in and perceive at a given moment, the feeling of happiness or its lack, are the result of the operation of the nervous system, and in particular the brain in which our mind functions. Until recently, complex psychological phenomena were extremely difficult to describe and research in detail. This is why for a long time there was even no proper term describing all the subjects related to the psychological life of individual people. Such subjects include both the biological foundations, i.e. the structure of the nervous system, particular senses and the brain, but also mathematical models of brain and mental process operation. In addition, they cover the philosophy of the mind, cognitive psychology, linguistics, anthropology and even introspection and spiritual life. It was only quite recently that the integration of many different fields of science offered a possibility of understanding the operating principles of the mind. This group of scientific fields was called cognitive science after the Latin word cognito, i.e. knowledge. Cognitive science deals with all phenomena related to the mind, and in particular with subjects concerning the method by which stimuli are perceived and the brain interacts with the external world. Usually, several fields of science with a fundamental significance for understanding the mind are listed: those are certain branches of psychology, artificial intelligence, psycholinguistics, brain science and cognitive philosophy. Cognitive neuropsychology should be added to this group. Cognitive neuropsychology studies the biological mechanisms of the brain which form the basis for human mental functions. The term "cognitive neuroscience" was coined by Michael S. Gazzaniga in late 1970. Gazzaniga defined this field very simply by saying that it is about learning "how the brain creates the mind". This science uses a set of different methods (like scanning the brain or computer modelling) and analyses psychological phenomena, thus discovering where, why and how the brain causes them to appear. Cognitive: neuropsychology is neither classical neurology - which researches the basic biology of the brain - nor psychology, which explores the internal life of humans. Cognitive neuropsychology represents a certain view of the mind according to which specific fundamental elements and principles, working step by step, create the conscious experience and activity of humans. L. Ogiela and M.R. Ogiela: Cognitive Techniques in Visual Data Interpretation, SCI 228, pp. 29–39. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

30

3 Cognitive aspects performed in the human mind

Neuroscience refers to the trend, prevailing for at least a dozen years, to consolidate knowledge about the nervous system coming from different fields of science. Neuroscience tries to compile knowledge provided by many specialised fields that study the functional structure of the nervous system. Every one of those fields offers many facts. For instance, neurobiology offers knowledge about gene interactions, histology, micro- and macroscopic anatomy, physiology at the level of single neuron interactions as well as of systems. Neurology, neurosurgery, psychiatry and neuropsychology provide descriptions of the effects of brain injuries, but also the dynamics of processes taking place in injured brain and mechanisms for compensating for and rehabilitating disturbed information processing operations. An important role in learning the mechanisms of perception is also played by the development of mathematical, physical and informatics methods. The use of computational methods allows us to construct advanced theoretical models that lead to hypotheses about complex processes of transforming elementary data into more complicated information. Such processes take place and are executed in the brain. The combination of all these scientific fields allows one great body of knowledge about the nervous system to be collated. Cognitive neuropsychology deals with everything concerning the relationships between the brain and the mind. It is therefore an interdisciplinary area grouping various scientists traditionally assigned to separate fields. Neuropsychology studies the consequences of brain damage, so cognitive psychologists were able to examine the function of brain in sufferers of various neurological disorders which damaged some specific part of their brains. By comparing how healthy and disabled individuals execute various tasks, psychologists were able to verify ideas about cognitive processes studied. This enabled psychologists to examine the reactions of the nervous system to the analysed phenomena which involve the attention or memory. The results of this research can confirm the validity of specific concepts of cognitive processes. In recent years this field employed the most modern technologies for brain functional diagnostics, so called neuroimaging methods. These methods include: • MRI (magnetic resonance imaging) - allows the live brain to be imaged with high precision but safely and non-invasively, provides any number of crosssections along selected planes of the brain. MRI also provides a number of other data used in medical diagnostics (Fig. 3.1) and makes it possible to measure the volume of the gray and white matter.

3.1 Brain science

31

Fig. 3.1. Magnetic resonance imaging results

• fMRI (functional magnetic resonance imaging) - the imaging of the functional magnetic resonance, allows local changes in the concentration of oxygenated blood and blood without oxygen to be captured, which indirectly illustrates the activity of neurons (Fig. 3.2).

Fig. 3.2. Example results of an fMRI examination during car and face recognition

• PET (positron emission tomography) - the tomography of positron emission, supports precise measurements and imaging of local changes in the blood flow, oxygen and glucose consumption as well as the activity of specific chemical transmission systems inside the brain (Fig. 3.3).

32

3 Cognitive aspects performed in the human mind

Fig. 3.3. Magnetic resonance results combined with PET results (source: Internet)

3.2 General brain structure It is widely known that the nervous system is made up of nerve cells called neurons and glia cells which perform support functions (nutrition, insulation, physical support) [34]. The structure of neurons can be simplified to two elements: dendrites and the body of the cell which contains the cellular nucleus and several other internal cellular structures, as well as the axon which branches off from the cell body. Concentrations of cell bodies are called the gray matter, and concentrations of axons are the white matter. Axons transfer information from the cell body to other nerve cells or executive organs, while dendrites transfer stimuli to the body of a nerve cell. Individual nerve cells are interconnected over synapses which mediate in information transmission. The nervous system is composed of the central and the peripheral nervous system. The central system can be generally split into the spinal cord and the brain, which in turn can be divided into: – cerebellum – brain stem, composed of the medulla oblongata, olfactory bulb, pons, mesencephalon; – the brain, in which the diencephalon and the telencephalon are distinguished. The telencephalon (endbrain) is the decision-making centre of the brain which supervises the majority of physical and mental actions. It is divided into two

3.2 General brain structure

33

clearly distinguished brain hemispheres and much less distinctly into the frontal, occipital, parietal and temporal lobes. The diencephalon is relatively small, but it serves as the nerve and hormonal coordination centre. It influences the adjustment of the blood pressure, maintaining body temperature and is responsible for causing the feelings of thirst, hunger etc. The cerebellum controls the function of muscles. People who have this structure damaged are not agile and easily lose their balance. This damage also causes difficulties in shifting attention from aural to visual stimuli and back. The medula oblongata contains nerve centres responsible for reflex functions: centres of respiratory, motion, heart, sucking and swallowing actions. Damage to the medulla oblongata causes a serious risk of death. The telencephalong and cerebellum are built of a relatively thin external layer of gray matter covering white matter. This layer of gray matter is referred to as the cerebral cortex and the cerebellar cortex, respectively. Gray matter is where neurons execute various computational tasks, while white matter is made up of nerve fibres through which various parts of the brain communicate. Various parts of the cerebral cortex are related to specific human functions. The visual cortex is the part of the occipital lobe in the rear part of the brain which is responsible for receiving and interpreting visual signals. The right brain hemisphere controls almost exclusively the behaviour of the left side of the body, while the left hemisphere deals with the right side. For this reason, almost all nerves which enter or leave the telencephalon must cross from one side to the other. The situation is more complicated in the case of the visual cortex, whose right side does not control the left eye, but rather the left part of the field of vision of both eyes. Figure 3.4 presents individual areas of the cerebral cortex corresponding to individual senses. Sound signals also come from ears to the opposite sides of the brain. The auditory cortex on the right, belonging to the right temporal lobe, receives signals mainly from the left ear. Smell seems to be an exception to this general rule. The right olfactory cortex, located in the front of the telencephalon in the frontal lobe, is connected with the right nostril, and the left with the left. In humans and other mammals, the cerebral cortex contains up to six separate layers of nerve cell bodies lying parallel to the cortex surface and separated one from another by bands of nerve fibres. The thickness of these layers changes significantly in various regions of the cortex, and what is more, some layers can disappear completely. In the cerebral cortex, cells also form columns which are perpendicular to layers. These are made up of neurons of similar characteristics. Cells belonging to one column have similar properties and there are many connections between them. For example, if one cell in a given column reacts to the touch of the left hand, then other cells in that column will also react to the touch of the left hand.

34

3 Cognitive aspects performed in the human mind

Fig. 3.4. Cerebral cortex areas responsible for particular senses

The sense of touch is connected with parietal lobe areas called the somatosensory cortex. This centre is situated right next to the border between the frontal and the parietal lobes. Particular parts of body surface correspond to specific areas of the somatosensory cortex.

3.3 Brain functions Modern neuroimaging methods have made it possible to locate brain areas responsible for higher psychological actions, such as speech, thinking, planning and associating (Fig. 3.5).

3.3 Brain functions

35

Fig. 3.5 Location of psychological activities in particular brain regions

The most important of these activities are executed in the following regions of the brain: • The occipital lobe - receives visual signals, so the rear part of the occipital lobe is called the primary visual cortex or the striate cortex. Damaging any fragment of the visual cortex causes cortical blindness in the part of the field of vision corresponding to that fragment. For instance, extensive damage to the visual cortex in the right hemisphere causes cortical blindness in the left field of vision. People with this type of complaint also lose visual imagination, which is not observed in those whose eyeballs were injured. • The parietal lobe - parts of this lobe receive information about touch and the body position, and this is where representations of the eye, head and body location are created. They constitute the information about what the eyes are looking at, how the head is positioned and in which direction the body is bent. From here, it is sent to brain areas which control motion activities. If this part of the cerebral cortex is damaged, attention becomes indivisible, eyesight cannot be focused and spatial orientation becomes difficult. • The temporal lobe - contains primary auditory areas. In addition, in humans, this structure (usually the left lobe) is of key importance in understanding speech. It houses the complex analysis of certain aspects of visual information: the perception of motion and facial recognition. If a brain tumour forms within this structure, it frequently causes complicated auditory or visual hallucinations. A tumour in the occipital lobe only causes simple observations of flashes of light. The temporal lobe is also responsible for emotional and motivational behaviours.

36

3 Cognitive aspects performed in the human mind

• The frontal lobe - the rear part of this lobe contains the cortex portion specialised in controlling precise movements, e.g. of individual fingers. If this part of the brain is damaged, it causes an inability to move parts of the body and to plan sequences of movements. It can also lead to emotional instability (the left lobe is responsible for depression, the right for the feeling of satisfaction).

3.4 Information processing in the cerebral cortex and perception models The ease with which we react to outside stimuli obscures the incredible complexity of processes which take place in the brain and which really make it possible to classify these stimuli or patterns at all. So a question springs to mind at what stage in processing the information recorded by our senses a decision on classifying the given stimulus or pattern is taken. One of the answers to this question is offered by perception models which determine the methods whereby the brain interprets stimuli. One of such models is the model at the level of neurons. When information is transmitted by the nervous system, neurons react to increasingly complex characteristics. Consequently, it should be possible to identify neurons specialised in recognising specific objects. The main argument refuting the correct operation of this model is that large parts of the brain can be damaged without causing major changes to behaviour. This characteristics of the brain and the ability to have the function of a damaged brain part taken over by another suggest that brain functions are not unambiguously positioned. These characteristics also suggest that brain functions are distributed within populations of neurons interconnected into nets. So different approaches of cognitive neuropsychology to category representation in the brain can be distinguished. One assumes that there are structures in the brain assigned to various object categories. The second says that classification is based on distributed topologies corresponding to features. The third assumes that correct classifications of patterns depend on processes which may be connected with them.

3.4.1 Mapping dependent on categories There are several possible hypotheses according to which the brain is divided into areas responsible for recognising various objects. One method to determine these areas is to distinguish sets of neurons responsible for recognising specific categories of objects. There are areas of the brain which are activated when the analysed pattern belongs to the selected category. In addition, in this model the brain areas

3.4 Information processing in the cerebral cortex and perception models

37

assigned to one category will not react to representatives of other categories. So the brain area which corresponds to category A does not react to category B and the other way around. It is easy to see that in such a model, if the information about a category is selectively represented in brain structures, destroying one of these structures should cause the knowledge about the selected category to be lost. Disturbances in the ability to recognise objects in which patients show selective problems in finding representatives of categories or features are linked with brain injuries causing various diseases. However, some researchers question the existence of areas linked to a specific category. They propose an alternative explanation of the brain activation in response to representatives of categories, based on distributed networks of neurons corresponding to specific features.

3.4.2 Mapping dependent on features More and more research suggests that pattern categories may be represented in the brain by smaller systems of neurons reacting mainly to the features of objects recognised. For example, the occipital/temporal cortex may not be composed of areas corresponding to particular categories, but of groups of neurons representing object features. Objects belonging to one category stimulate areas corresponding to the same (or similar) features. As it turns out, there are cells that react to patterns like strips and edges, so their activation seems necessary to perceive a strip, line or edge. These cells are feature detectors: neurons whose reactions indicate that a given feature is present. Cells in more distant regions of the cortex react to more complex features. The legitimacy of this hypothesis is supported by the fact that looking at a specific, selected feature for a long time reduces sensitivity to it, as though the appropriate receptors were getting fatigued. The assignment of certain features to distributed brain structures is also supported by the fact that the same areas of the brain react when patterns of different categories are registered. On the other hand, this approach does not explain why brain structures react when specific objects are present, but do not react to the presence of objects with similar features. We can suppose that objects which have similar features should cause a similar activation of the same areas of the brain, but this is not always the case.

3.4.3 Mapping dependent on processes In this model, the division of the brain into specialised areas is based neither on categories nor features of the object, but on the features of the process necessary

38

3 Cognitive aspects performed in the human mind

to analyse that object. The mental response to information about a pattern belonging to a given category is based not just on the category itself or its characteristic features, but on the process associated with that category. Particular areas of the visual cortex specialise in executing various actions. Objects belonging to different categories cause various areas of the brain to be activated depending on how the information is analysed and on the previous experience of the brain. Various areas of the brain are activated depending, for instance, on the ease of recognising these objects. For instance, the response of the brain to faces may seem different from its response to objects, as recognising faces requires capturing multiple subtle details, which is more complicated than recognising simple objects. The above research on category representation provides three possible models telling us how information about particular categories may be represented in the brain. Although these models differ from one another, they do not exclude one another completely. It is possible that various areas of the brain represent different categories in different ways. In addition, every category representation model must explain the processes of learning new categories and also support methods of distinguishing elements within categories. It is also quite likely that category representation may differ depending on the brain hemisphere, whereas the representation in one hemisphere is more category-dependent, and in the other more dependent on the feature topology.

3.5 Cognition levels The methods of studying and modelling cognition levels in various cognitive science fields differ, but their common feature is that they imitate nature. Of course, none of the perception models presented so far covers all the capabilities of human and animal cognitive processes in their natural environment. However, science presents various models of learning at individual evolution levels, and we will briefly discuss one of those. Cognitive systems of animals at every rung of the evolution ladder are more capable than those of organisms from the rung below. Primitive organisms have basic sensory/motion mechanisms combined with a small number of neurons. At this level, the appropriate model could be a neural network with several intermediate layers. The brain of a fish is small compared to that of mammals, but it already has a complex structure which receives impulses from highly varied sensory mechanisms and sends the signal to no less complicated and varied muscular mechanisms. The precise operating principle of this process is unknown, but research suggest that there is a split into perception mechanisms interpreting input signals and motion mechanisms controlling the activity. Obviously, there must be numerous interactions between both mechanisms in this process, and then the simple neural network is no longer sufficient as a model imitating such processes. The next evolutionary level of organisms are animals having cerebral cortices with specific areas responsible for sensing, which cooperate with sensory and mo-

3.5 Cognition levels

39

tion mechanisms. New abilities at this level include the ability to perceive similarities. Finally, at the human level, new thought functions like induction, abduction and deduction appear. The ability to reason can be defined as a special method of using analogies, while all thought mechanisms cooperate with those more primitive. Intelligence and neural processes operate at many levels. The immediate reaction to pain caused by the touch of a hot object is controlled by primitive mechanisms found in even the simplest organisms, long before the brain receives the information that such a stimulus appeared. Although muscles can be controlled using conscious attention, they work more effectively if the details of their control are left to more basic parts of the brain. Modelling and imitating such reactions is now also within the domain of artificial intelligence systems. However, a number of difficulties arises here as there are problems, not completely solved, related to brain operation methods. One of such interesting, open problems in computer cognitive science is language acquisition. No contemporary software can compete in this field (i.e. acquiring and using language) even with small children. Chomsky and his followers believed that children acquire language so quickly because the human brain has an inborn module to ease acquiring the syntax. However, if language were based only on this inborn module, its computer simulation would allow machines to quickly understand and learn language just as children do. So where is the problem? It turned out that it is not the syntax that is the most difficult for computers, nor is it crucial for language understanding by people. People speaking a foreign language can be understood even if they use incorrect syntax, while in such a case computers have problems to complete a correct syntactic analysis of a phrase understandable to humans. This confirms the hypothesis that symbols and meanings are more important than the syntax in the language. In the acquisition process, you first learn words symbolising certain things and then acquire patterns of their use. Such tasks are still difficult to execute quickly and effectively using computers, although further in this book we present methods of creating systems capable of determining the meaning of complex image patterns. The next chapter will present basic knowledge about cognitive informatics. Particular attention will also be paid to perception models introduced by this field of informatics for the purpose of building computer systems that imitate thought processes.

4 The fundamentals and development of Cognitive Informatics Cognitive informatics (CI) is a discipline which combines the subjects of both the cognitive science and informatics based on information mechanisms and processes taking place in the human brain. So cognitive informatics uses natural intelligence merged with computer applications broadly applied in interdisciplinary research and science. Cognitive informatics thus covers not only the aspects of using mathematical theories to describe and analyse data and information presented in large knowledge bases but also engineering fields with informatics, cognitive sciences, neuropsychology, system sciences, cybernetics, computer engineering, knowledge engineering and computational engineering. Theoretical foundations of cognitive informatics are strictly linked with mathematical notions, and the formal models describing theories of cognitive informatics are based on computational intelligence, both of humans and machines. Cognitive informatics can be applied in such areas as so-called cognitive computers, cognitive knowledge bases, cognitive models and simulations of human brain function, autonomic agent systems and computational intelligence. The term “cognitive informatics” was coined the beginning of interdisciplinary research at the border of informatics, cybernetics, cognitive science, neuropsychology, knowledge engineering, computational intelligence and sciences dealing with human life. Such a combination of research was proposed by authors of the notion of cognitive informatics in their works [99-103]. Numerous publications of the authors of this book also contributed to this field [42-68]. All research problems coming up in the area of cognitive informatics aim at understanding the operating mechanisms of human intelligence and at modelling cognitive processes taking place in a person's brain. Understanding how those mechanisms work makes it possible to apply them to solve technical and engineering problems. For many years, cognitive issues have been analysed not just by philosophy and psychology, but also informatics.

4.1 Development of cognitive subjects in science Cognitive subjects are of constant interest to various scientific disciplines coming from various theoretical and methodological traditions. Cognitive science is very often understood as the science of learning and is therefore identified with the cognitive theory. Cognitive theories started developing with the introduction of the philosophical cognitive theory – epistemology [9]. Representatives of epistemology focused in their work on fundamental problems of human learning, which included the sources and the nature of knowledge acquired as well as the theory of truth. L. Ogiela and M.R. Ogiela: Cognitive Techniques in Visual Data Interpretation, SCI 228, pp. 41–61. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

42

4 The fundamentals and development of Cognitive Informatics

Cognitive theory was at the beginning developed as a speculative science supported by logic and everyday observations. However, as time passed, it increasingly started to use the achievements of philosophy, psychology, medicine, linguistics and also informatics. Cognitive science, as the science of learning, started focusing on a special field of science which was the philosophy of the brain, which then considered the classical psycho-physical problem originally defined as the relationship between the tangible body and the intangible soul, later understood as the relationship between the brain and conscience. At the time, cognitive theory concentrated on matters explaining how the brain can generate conscious conditions of the psyche and on determining the function and the origin of qualia qualitatively individual, subjective states of consciousness. In addition, in-depth observations were made of the expression of emotions in people and the phenomenon of emotions occurring from the perspective of the human evolution. Foundations were laid for the development of a scientific discipline based on researching the phenomenon of emotions in a fashion that would open the way to study it experimentally, and scientists wrote about the significance and the role of emotions for psychopatology. The issue of emotions and the entire phenomenon connected with them had a huge impact on the appearance of two existence models of mental processes - the neurophysiological and the perception model. Each one of them is present in the human brain to a greater or smaller extent and is based on emotional states triggered earlier, starting at the initial phases of the cognitive analysis process. For many years, at many scientific centres world-wide, research went on about the phenomenon of emotional states occurring and their link to basic functions of human mental processes based on attention, perception, memory or language [1]. It th is worth noting that the 20 century saw a critical reflection about the development of cognitive science, namely that work aimed at studying the impact of emotions has been totally neglected. As a result, this somewhat neglected area became the subject of intense research. It is notable how small the group of scientists was who researched emotions on the ground of cognitive science, but who laid the foundations for an extremely important analysis of this phenomenon and at the same time characterised many components and tremendously important details of the researched phenomenon. They were: Paul Ekman [19], Jerome Kagan [25], Richard S. Lazarus [31], George Mandler [33], Robert B. Zajonc [111], Steven Schachter and Joel Singer [78]. Emotional processes studied by the above scientists formed firm ground for learning the functions and properties of a psychogenetic area called the limbic system. The links between that system and the emotions that appear are of great importance because emotions are compared to other functions connected with other brain structures. The difference between emotions and the cognitive acquisition of new information is due to the duality of structures and functions between the limbic system and the neocortex. A new element in the research on human emotions was the presentation, by many researchers studying neurophysiological cases, of extremely important links between human emotional states and the right brain hemisphere [8, 12, 75].

4.1 Development of cognitive subjects in science

43

Psychology distinguished the following stages in the process of understanding any information received by a human and subjected to cognitive analysis: • The recording, memorizing and coding stages of the information obtained. Recording may boil down to a single perception cycle, but it may also take a complex form which assumes an explorative activity of the individual. Memorising may consist in the simple fixing in memory of detailed information, i.e. facts, patterns of objects, methods of action, but in more complex situations may consist in creating a universal memory trace (a gnostic unit) capable of generating features needed later to use the knowledge possessed in the process of understanding new situations being analysed. Useful knowledge, which among other functions facilitates generalising the information learned, usually has this second form. • Storage stage – this is the latent stage of mental processes which is not accessible to direct research, whose nature and course must be reasoned out from the last phase. There are reasons to believe that storage is not just a passive process of keeping information, but consists in creating subsequent versions of more and more refined internal representations of the knowledge possessed and systems of its internal links. • Retrieval stage – covers remembering, recognising, understanding and relearning specific skills. Retrieval is the moment at which the acquired knowledge and wisdom resulting from it may manifest in the individual’s behaviour. The retrieval stage is the measure of the human memory processes: the remembering, recognising, understanding and possibly learning anew. Another differentiation consists in linking memory as a characteristic of the nervous system with individual analysers. Memory dependent on the type of analyser can be called a peripheral ability. For the essence of mental processes it is important that memory also exists as a general ability, working when complex stimuli act, achieved as a result of the operation of many analysers. The most important features of memory used to assess this mental process are: – – – –

–

permanence - a criterion related to the storage stage; speed - defined as the ease of recording new facts and links between them; this criterion belongs to the memorizing stage; accuracy or reliability - defines the relations between the retrieved information and what constituted the contents of the memorizing stage; readiness - adequate to the retrieval stage - determines whether remembering occurs without major problems or whether additional activating stimuli are necessary; range or capacity - refers to the coding phase.

Recognition, which is one of the indications that memory works, is inseparably linked to perception and decision-making. When perceiving objects in his/her surroundings, the individual either recognizes them as known (he/she can then name them and associate them with a specific action), or finds that he/she does not rec-

44

4 The fundamentals and development of Cognitive Informatics

ognize them, which also plays an important role as the so-called detector of novelty. Both if the specific object is recognised and if it is detected as a novelty, the human brain assigns a specific mental category to this perception situation, and thus takes the decision whether the stimulus acting upon it is old or new, known to it or unknown. This mechanism is of fundamental importance in the entire psychology of perception. The sensory threshold - susceptibility and sensitivity - is determined by reference to the recognition process. A threshold stimulus is in 50% of cases received or recognized as a stimulus, and in 50% of cases is not. Similarly, the difference threshold is set at that value of the stimulus which in 50% of cases is recognized as equal in value to the standard stimulus, and in 50% as different. The neurophysiological model of the cognitive analysis is based on the operation and behaviour of the brain which can be described by studying attractors resulting from the dynamics of great groups of neurons composed of millions of cells. These attractors are defined in the stimulation space of such neurons. Even though there is ambiguity in interpreting surface states of the brain (e.g. their activity examined using the EEG), in the dynamics of its function we can find certain constant relations between “practical attractors” which are relatively stable, set neurophysiological states that can be identified in the brain behaviour. What is more, dynamic brain states, if they are properly handled mathematically, are characterised by certain deeper relations which may have a simple logical representation. The correspondence between the mental states and forms of brain activity does not concern only surface states which are of an ephemeral (transitory and shortlasting) character, but also attractor states which exhibit some stability. However, internal representations, which according to model research must be more stable, do not translate to similarly stable surface representations, which makes their empirical study difficult. In addition, small disturbances in the electro-chemical structure of the brain cause significant changes at the mental phenomenon level, e.g. bring about mental illnesses. These are frequently the consequence of the global brain dynamics, dependent on the condition of all its structures, an in most cases no single, well defined anatomical structure responsible for the specific form of pathology can be pinpointed. In the brain, a holistic principle can generally be observed which consists in all structures contributing to a greater or smaller extent to all functions. This leads, to a fact (among others), known for many years, that the brain has no areas in the anatomic sense which are interconnected so weakly that they could be eliminated (e.g. excised) without changing its function, although sometimes those changes may be difficult to detect. This justifies the claim by John R. Searl [79] that although the neurophysiology of the brain does determine the character of the mind, this does not mean that intelligence or some form of a mind could not be created by the interaction of elements based on silicone or other, non-carbon compounds. The development of computer science, and in particular artificial intelligence, has proven that intelli-

4.1 Development of cognitive subjects in science

45

gent information processing is also possible using silicone compounds (of which electronic systems are built), and the neurochemical states determining human mental impressions can be approximated with a satisfactory precision using digital devices. However, such “minds” have a structure different from that of human minds, which means that the way in which they “experience the world” is different because different brain structures must bring about different minds. So the basic features that a given system should exhibit so that its function can be equated to that of a mind include intentionality, understanding and conscience. At first sight, these features seem to describe only biological systems, but the evolution in the design of robots and IT systems has made them of interest to technology as well. We now have many examples of systems controlled by software that executes “brain-like” methods of information processing. Another feature of these systems is also that only their general properties are programmable, which allows them to be considered as moving towards intentionality. We currently observe the appearance of a number of technical solutions with an increasing autonomy of operation and it seems that the convergence limit of this series of better and better models may be intentional systems of increasingly complex structures and purposes of operation. The progress in neurophysiological knowledge increasingly convinces us that it is the structures of brains that are the reason and the drivers of the ways in which minds operate. Those structures demonstrates the obvious dependency of the complexity of mind forms on the complexity of the brain itself. What is more, the increasing complexity of the mind and the brain is evidenced by complex forms of behaviour. It seems indisputable that the increasingly advanced computer models created using neurophysiological data contribute to a more and more detailed explanation of aspects of the mind's operation. Every self-organising system currently exhibits intentional behaviour due to pursuing general values and needs of that system which result from its biological structure or technical design. In the process of developing such models of information systems which allow information to be processed and understood in a way modelled on the human brain function, we should now expect increasingly complex forms of behaviour. The human brain is the location of real neurophysiological processes which cause real, conscious impressions. Many of the above processes are executed by areas of the brain that have now been well located and can be observed using various techniques, combining morphological imaging with functional monitoring, for instance positron emission tomography. This allows us to study processes during which several sensory modalities must cooperate and references must be made to episode memory layers located in a specific area of the cerebral cortex. The most frequent ones are analyses of sensory data, such as segmenting an image. Components kept in the episodic memory contain the aspects of a specific modality that cannot be executed locally. What becomes necessary for the whole process is a mechanism for distributing information and combining results obtained from various areas of the brain into one whole. This integrating operation of the brain means that perception processes use fragments of mental representations supplied by the senses and internal stimulations of areas processing sensory information, or

46

4 The fundamentals and development of Cognitive Informatics

a holistic representation of a complex situation containing all the fragments currently stimulated. This is a perceptual meta-representation of this complex situation, probably produced by the global dynamics of bioelectric discharges in the brain, which can be observed as waves on the EEG. Cognitive science concepts are useful as far as they approximate models describing the brain's operation at the neurophysiological level. If some regularities can be found in neurophysiological observations, an attempt is made to express them as certain rules of logical procedure which can then be mapped to cognitive science models. Consequently, in some models of information systems it is convenient to use the notions of classical cognitive science, primarily logical rules utilised to explain mind states in order to simplify their biophysical models. This paradigm can be related to many levels of the description of neural structures and functions, which include: – – – – – –

the sub-molecular, or the genetic, level; the cell structure level; the level of neurochemical phenomena; the level of single neuron activity; the level of activity of neural nuclei and other, anatomically distinguishable areas of the brain; the level of the global dynamics of bioelectric phenomena in the brain which correlates with states of mind.

It cannot be disputed that for a long time yet the simulated mind will not be identical to the biological one, among other reasons because of the technical difficulties in this type of simulations. Currently, the accuracy limits of simulating models of information systems on computers are not completely clear. The majority of models broadly used in neuroinformatics are very simplified, like the popular neural networks of widespread applications. However, attempts to model neural systems realistically (e.g. de Schutter’s Genesis project) prove that computer techniques can also map neural systems with a precision and fidelity which had until recently seemed completely unachievable. The naturalistic solutions presented above lead to various empirical predictions and in the future may form the foundations for developing a satisfactory theory of the brain. The development of cognitive subjects concerned not only social sciences like philosophy or psychology: they developed no less successfully in the fields of informatics and mathematics, where definitions of mathematical concepts, notation and symbols played a significant role. Artificial intelligence, which leads to designing computer systems capable of executing individual cognitive actions, represents a special approach to the study of cognition. Artificial intelligence started focusing on engineering sciences, where intelligent systems were developed for practical use. Their designers strove to enable these systems to imitate people's cognitive processes very effectively. The success of systems built was assessed by reference to their practical utility

4.1 Development of cognitive subjects in science

47

and the possibility of deploying them to conduct basic research on cognition. The term “computer modelling” appeared and was accepted as one of the basic research methods used in cognitive psychology. In the light of the extremely interdisciplinary research conducted, people started talking of a new field of research, not just on cognition, but also on understanding and reasoning - the cognitive science. In 1978, Sloane wrote a report in which he stated that cognitive science was an attempt to combine subjects followed by six basic, traditional scientific disciplines [84]: – philosophy; – psychology; – linguistics; – informatics; – anthropology: philosophical, biological and sociological; – neural science. All the above disciplines cooperate within cognitive science in line with the scientific links between them, presented in Fig. 4.1.

Fig. 4.1. Sciences making up cognitive science and links between them. Source: developed on the basis of [4]

The history of cognitive science has seen the gradual introduction of new problems represented by novel mathematical formalisms into this science [5, 72, 108].

48

4 The fundamentals and development of Cognitive Informatics

The use of mathematical tools in cognitive science boiled down to two categories which could be called analytical and denotational mathematics [98, 100, 101]. Mathematical structures based on analyzing variable functions and operations were used to build system architectures analysing data in dynamic processes. The analysis of this type of events led to distinguishing a new branch on the border of cognitive informatics, computational intelligence, software engineering, knowledge engineering and the application of mathematical forms of definitions of the analysed problems. In recent years, Wang has called this field denotational mathematics [100, 101]. The most important forms and structures of descriptive mathematics are: – – – –

the notion of algebra; the system of algebra; the algebra of real time processes; the semantic algebra.

These four basic types of mathematical structures are used in practice to execute tasks not just in the area of cognitive informatics, but also in computational intelligence or knowledge engineering. Apart from work illustrating the use of cognitive methods based on mathematic formalisms in computer sciences, work is also conducted to use cognitive subjects for the computer analysis of data. This type of application of cognitive structures leads largely to the structural and semantic analysis of data under consideration. Structural analysis is based on applying cognitive methods to the information about the structure, the form and the shape of the analyzed object. On the other hand, semantic analysis uses the information contained within the data: the semantic information (meaning). The author’s examples of using cognitive methods in informatics will be described in subsequent chapters of this book. However, here it seems reasonable to also indicate other areas of informatics in which cognitive informatics is now present. These undoubtedly include the James Anderson's work to build a system resembling the structure of the human brain [2, 3]. This project calls for executing three basic processes dealing with the following areas: preliminary hardware design, programming techniques and the application of the software generated. The part proposed in the project and related to selecting and building the hardware architecture is based on the structure of the neocortex in mammals, which can be compared to the 2D connection of elements within the CPU, which is then connected to the system memory. In order to introduce cognitive software into the above project, it is necessary to have new programming techniques based on topographic data representations, data transmission to the outside, data acquisition from the outside and the use of connection modules to complete the appropriate calculations. Software applications require the simultaneous joining of various elements, which include: –

the natural language;

4.2 Theoretical aspects and formal models of cognitive informatics

– – – – –

49

programming languages; cognitive data analysis; information processes taking place; decision-making; and knowledge management.

Another example of using cognitive methods in computer science is described by Jean-Claude Latombe in his work [30]. He presents the use of the traffic process of automatic robots on the example of a new traffic planning method utilising the plan of a probabilistic traffic map. Examples of using cognitive methods in computer science are very often connected with data analysis, and in particular image analysis. Such work was conducted in the field of analyses of medical data showing various types of lesions found within an examined organ [67, 68]. Other work in cognitive computer science concerns face, speech, speaker, language and movement analyses. Some of it is described in the following publications: [102, 107, 112]. This kind of work was also conducted to build intelligent mobile robots and unmanned vehicles [1, 37], as well as to adapt psychological and philosophical methods to computer science, computational intelligence, creating intelligent IT systems, automatic and machine learning and signal analysis [102, 107, 112].

4.2 Theoretical aspects and formal models of cognitive informatics IT theories, information theories, various ways of acquiring information, of perceiving and analysing it, and of obtaining objects necessary for the analysis processes have all been researched from the time when the information theory was introduced by contemporary informatics until today, when cognitive informatics is developed. The beginnings of cognitive informatics were connected with the introduction of what is called the contemporary informatics, which studies information as an element distinguished (acquired) from the reality around us, which can be either a representative of a given group of information, or a completely new element or exception, unknown and never before analysed. Such a perspective on information has shown that processes of information analysis and interpretation are not simple and obvious. On the contrary, this analysis and interpretation should be based on some of the most difficult processes currently known to science world-wide. These are processes taking place in the human brain, because they allow us to understand the essence of the process of reasoning using the information possessed and also lead to the correct analysis of that information.

50

4 The fundamentals and development of Cognitive Informatics

Such a presentation of cognitive informatics allowed it to be defined in a way which unambiguously specifies the place of cognitive informatics among sciences [99]. The authors propose introducing the following definition that describes the essence of cognitive informatics. Cognitive informatics may be understood as the combination of cognitive sciences and informatics to study the mechanisms by which information processes operate in the human mind, which processes are treated as elements of natural intelligence and applied for engineering and technical tasks using an interdisciplinary approach. The system architecture presented in cognitive informatics for building theoretical networks of cognitive informatics is based on models introduced by Wang, which include [99]: – – – – – – – – – –

the Information-Matter-Energy (IME) model, the Layered Reference Model of the Brain (LRMB); the Object-Attribute-Relation (OAR) model; the model of information representation in the human brain; the model of cognitive informatics based on the human brain; the natural intelligence (NI) model; the neuron informatics model (NeI); the human perception process mechanism; cognitive computer models; cognitive machine models.

Theoretical aspects of cognitive informatics are considered within two categories: • the first is the application perspective of the computer science, computer techniques and cognitive research problems like: memory, learning, reasoning, drawing conclusions, analysis; • the second category is the use of cognitive theories to solve problems in computer science, knowledge engineering, software engineering and computational intelligence. These problems can be solved by attempting to establish the theoretical foundations of processes taking place in the human brain, such as information acquisition, selecting the representation of information, memory, recovering lost information, generating communication and its process. Selected models of cognitive informatics will be discussed in subsequent subchapters.

4.2 Theoretical aspects and formal models of cognitive informatics

51

4.2.1 IME Model – Information-Matter-Energy The first presented model of the cognitive approach is called Information-MatterEnergy (IME), in which information is recognised as one of three complementary components of the natural world, the other two being the matter and energy that surround us. This combination became possible under the assumption that apart from the basic elements making up the contemporary world - matter and energy information is treated as its third main component, because the processes running in the human brain are founded on basic functions connected with information processes. Such a model perspective of the cognitive approach means that the natural world (NW) is divided into components: the world described by physics, the world of the tangible - the physical world (PW); and the world of the abstract, the world of the perception - the abstract world (AW), in which both matter and energy are used to describe information (Fig. 4.2.).

Fig. 4.2. The IME cognitive information model. Source: developed on the basis of [103]

The essence of the presented cognitive information model is to show information as an important link between two worlds of event description. It is a kind of process of recognising the relations between the biological world operating within the realm of the world of physics, and knowledge in the form of accumulated bases of information taken from the abstract world. The natural world, described by information, matter and energy, is inseparably connected to the abstract world by information learned and perceived using individual, frequently unique tech-

52

4 The fundamentals and development of Cognitive Informatics

niques dependent on the perception methods and the mental context of the person identifying the specific information. So in the IME model, the definition of information depends on individual ways of human perception, which also serve as the element linking the physical and the abstract world, and which exert a significant impact on the creation of knowledge bases used by cognitive information systems. As research on using the IME model for the cognitive analysis of data constantly progresses, an improved model which additionally contains one more element - intelligence - has been created. This model is called the InformationMatter-Energy-Intelligence (IME-I) Model.

4.2.2 IME-I Model – Information-Matter-Energy-Intelligence Adding the component called intelligence to the IME model represents an extremely important stage in work on building a cognitive information model. The perceived physical world and the invisible world of the abstract are linked not just by information, but also by intelligence which lies at the border of these two, completely different realms. This is because intelligence is described as a part of the human nature, but is also a notion defined in a very concrete way, so it is a notion coming from the physical world. The difficulty in unambiguously defining and placing intelligence in the natural world is a problem that cannot be solved as there is ambiguity in defining the areas of the abstract and the physical world. The analysed IME-I cognitive information model presented in Figure 4.3 points to the phenomenon of two-stage data analysis using the available intelligence, both human and automatic, the latter founded only and exclusively on structures of human intelligence models.

4.2 Theoretical aspects and formal models of cognitive informatics

53

Fig. 4.3. The IME-I cognitive information model. Source: developed on the basis of [103]

Intelligence, presented in Figure 4.3., is understood as the ability of humans or systems to interpret and analyse information and to automatically combine information elements that come from the abstract world, such as: data, information, knowledge, experience. It serves as a connection through which data is transmitted and transferred between information, matter and energy.

4.2.3 LRMB Model – Layered Reference Model of the Brain The cognitive information model presented in this chapter makes use of basic and fundamental mechanisms that occur in the human brain and of processes of natural, human intelligence. As processes running in the human brain which are referred to as cognitive processes and are therefore of interest to the cognitive informatics, psychology, cognitive sciences, neural sciences and neuropsychology are greatly varied, the cognitive analysis process requires that all the cognitive processes occur and also that a network is created within which all the aforementioned cognitive processes are included. The proposed LRMB model describes thirty-nine cognitive processes split into seven main elements, namely (Figure 4.4): – –

sensations, feelings, impressions; memory;

54

4 The fundamentals and development of Cognitive Informatics

– – – – –

perception; action, process; meta-cognitive; meta-reasoning; highest cognitive layers.

Fig. 4.4. The LRMBcognitive information model. Source: developed on the basis of [99]

The layers of the LRMB system presented in Figure 4.4 are mutually dependent and complement one another. Each one of them, as a group of life functions, is a component of the entire system regardless of whether those functions are conscious or unconscious. The operation of cognitive data analysis processes is based on initiating cognitive processes as well as other human life functions, which will serve as the foundation for building an intelligent information system.

4.2 Theoretical aspects and formal models of cognitive informatics

55

4.2.4 OAR Model – Object-Attribute-Relation The OAR model combines methods of representation, recording and analysis of knowledge and information which take place in the human brain. It describes levels of the human memory with particular emphasis on the long-term kind, as well as the level of memory at which a dependency may appear between various metaphors introduced. The OAR model demonstrates relationships between human memory and knowledge using synaptic and neural junctions. This model is very frequently used to describe mechanisms and processes taking place in cognitive systems based on the occurrence of the above cognitive processes. The operation of the OAR model is possible thanks to defining the notion of neuronal informatics (NeI) understood as the new, biological/psychological, interdisciplinary perspective on knowledge and information representation taking place in the human brain at the neural level, illustrated using mathematical models. Neural informatics is a branch of cognitive informatics which identifies human memory seen as the basis and foundation of intelligence, whether natural or artificial. Apart from neural informatics, the OAR model also includes the Cognitive Memory Model (CMM). The CMM model works on the basis of five different models of human memory, which should include: – – – – –

the SBM model – Sensory Buffer Memory; the STM model – Short-Term Memory; the CSM model – Conscious-Status Memory; the LTM model – Long-Term Memory; the ABM model – Action-Buffer Memory.

The results of the operation of the above models can be seen in the following applications of cognitive informatics and neural informatics models, in which the LTM (long-term memory) model plays an important role: – – – – –

using the LTM model for subconscious processes; the operation of long-term memory during sleep; the operating mechanism of the LTM model during sleep; the operation of the LTM model with reference to the OAR model; the role of the visual function and the perception function (including daydream creation) in the LTM model.

4.2.5 The model of consciousness and machine cognition The theoretical model of cognitive informatics presented in this chapter, based on the model of human consciousness and cognition, has been used for machines to execute the above cognitive tasks.

56

4 The fundamentals and development of Cognitive Informatics

The analysis of cognitive processes, including the analysis of consciousness, dates back to the times of Aristotle, when work started to analyse the human body and brain [14, 28, 32, 86, 106]. Consciousness was treated as the basis element describing and characterising the life and mind of humans, so in cognitive informatics publications is has been assigned features connected not only with human cognitive aspects, but also machine elements. So consciousness is understood as one of life functions that works within the LRMB model (described previously) which represents conscious and subconscious life functions. The presented model of consciousness and machine cognition frequently uses the learning theory of both the human and the machine learning process. This is because learning is a cognitive process of acquiring, obtaining and transferring knowledge and behavioural experience. The psychological variety of learning processes is impressive and has been described at great length in the literature of the subject [32, 85, 88]. In cognitive science, learning is synonymous with the appearance of relatively permanent changes in the cognitive and thinking processes as well as in perception, understood as the results of certain impressions and experiences gained during the entire learning process [16, 24, 73, 76, 85, 106]. Cognitive informatics borrowed certain items from the cognitive learning process characteristic for the human mental process to adapt learning processes for their execution by machines. This is why cognitive informatics defines machine learning processes as cognitive processes taking place at the level of cognitive meta-processes described for the LRMB model. These processes influence one another as part of the following tasks: – – – – – – – –

object identification; abstracting; searching; introducing a notion (defining); understanding; comprehending; remembering; and retrieving and testing memory.

The most important element of the learning process is memory, in particular the long-term kind (the LTM cognitive model). All machine learning processes found in cognitive informatics are based on the LTM and OAR cognitive models, where natural minds form the foundation for creating all the possible relationships occurring in human minds and then adapting them to machine learning processes.

4.3 Cognitive science in categorisation tasks

57

4.3 Cognitive science in categorisation tasks This sub-chapter presents selected applications of cognitive science, in particular the use of cognitive informatics for categorisation jobs. Categorisation processes refer to collecting objects into certain groups whose joint feature is a set of common characteristics. The ease of indicating common characteristics found within the distinguished group makes it possible to introduce a verbal label, which is the linguistic representation of that group (colloquially: the group name). The linguistic representation does not always have common features with the conceptual representation to which the specific name has been assigned. Conceptual representations depend on the context in which they are acquired and used [35, 39], whereas the lack of stimuli in the process of learning about a new representation does not allow the correct conceptual representation to be created. The above presentation of the categorisation process confirms Ockham’s concept of representationalism which states that deep representations may function in isolation from the experience of real objects, but frequently experiencing those objects is necessary in the process of acquiring those representations. This concept split psychologists into two camps. One consisted of supporters of associationism1who agreed with the Ockham theory, the other were supporters of gestaltism2 who claimed that it was possible for imageless concepts that do not come from perceptional experience to exist. This trend of the psychological presentation of concepts leading to conceptual categorisation gave birth to two fundamentally different types of concepts - the classical concept and the natural concept: • Classical concept - a concept clearly determined and defined. Such a concept constitutes the cognitive representation of a finite number of characteristics common to all designata, which accrue to all specimens of the concept to the same extent. Classical concepts are called scientific or Aristotelian concepts because it was Aristotle who started defining objects by indicating the closest type and the specific difference. It is worth noting that the closest type is a category superior to the object being defined, so it is the closest in terms of the abstraction level, but too general. The second important component is the spe1

Associationism appeared in psychology as a concept according to which the entire psychological life mechanism is subordinate to the operation of laws of associating images and ideas. As a result of defining the associationism concept, a theory called the associative theory of perception was proposed, according to which observation is the simple sum of all available impressions and is formed by combining them [82]. 2 In psychology, gestaltism was a school which adopted the idea that psychological life is not a complex of psychological elements, but is composed of certain entireties which have their individual forms. In addition, those entireties have characteristics which cannot be boiled down to the sum of characteristics of components making up those wholes [13].

58

4 The fundamentals and development of Cognitive Informatics

cific difference, that is the feature distinguishing the object being defined from other representatives of that category. • Natural kind concept - a concept determined less precisely, of a more individual nature. It is the cognitive representation of a finite number of characteristics which accrue in different ways to individual designata of the concept. Because of this, every natural kind concept has more and less typical designata. The situation is similar when we take into account characteristics that define the given conceptual representation, as these characteristics may be more or less typical. These concepts dominate in the creation of the commonplace representation of the world in the human mind, so sometimes they are also changed as a result of contacts with other designata of a given concept. It is worth noting that both the variability and the dependence on experience and the situational context clearly distinguishes natural kind concepts from classical concepts, although frequently a classical concept for a given group can become a natural kind concept for another group. This division of concepts introduced and determined certain frameworks for defining a given thing, structure etc. If various concepts are introduced, they can be categorised on the basis of identification, perception, analysis and description processes. This type of categorisation, aimed at gathering objects into groups sharing certain characteristics, is possible only thanks to the occurrence of cognitive processes. Hence cognitive science has become an indispensable element of the categorisation process. Both the real and the virtual world are described using a cognitive categorisation which supports the process of identifying the phenomenon considered. In the entire categorisation process, the process of cognitive categorisation aimed at collecting objects into groups with common cognitive characteristics has been distinguished. So the process itself of describing a given concept belonging to a certain conceptual group has been extended to cover the process of including in that group concepts connected only with the semantic presentation of the object. This process is possible only when, apart from the simple naming of a given concept, it is also possible to determine the meaning to which that concept refers. Because the essence of the concept (the semantics of the concept) was distinguished, it became possible to deploy cognitive science in categorisation problems and tasks. This is because the cognitive categorisation process itself is not just about the simple nomenclature or the correct definition of the concept. It is a process based on a certain analysis of the concept: a semantic analysis of what the object, data, information or structure under consideration means. This is why this type of categorisation can be applied for deeper analysis tasks - computer analysis, automatic analysis - where the appropriate computer technology combined with cognitive categorisation processes makes it possible to execute in-depth data analysis processes which consist in describing this data, analysing it, interpreting it and reasoning from its semantic content, and as a result classifying that data according to its meaning. The process of creating a category aimed at the cognitive categorisation is founded on two basic rules of category building:

4.3 Cognitive science in categorisation tasks

59

• The first rule stems from the function of category systems: the job of categorisation systems is to deliver the maximum quantity of information with the smallest possible cognitive effort; • The second rule stems from the structure of category systems: the structure of information supplied supports the observation that the perceived world reaches us more often as structured information, and much more seldom as a set of arbitrary or unpredictable attributes The above categorisation rules allow us to say that the main feature of the categorisation process is that it generates the most information with the least cognitive effort if categories map the structure of the perceived world extremely precisely. This process also occurs when it is possible to fit categories to established structures of characteristics, and also when it is possible to define or redefine characteristics for the appropriate case of structuring the given sets of categories. The principle of cognitive economy very often applies in categorisation processes. This principle says that a stimulus/object can be categorised if not only its similarity to other stimuli from the same category but also its difference from the stimuli from outside that category are analysed for the purpose of completing that categorisation process. It is worth noting that in this perspective, such cognitive economy could lead to creating a huge number of category sets if there were small differences between categories. At the same time, in the categorisation process it would be possible to limit the infinite number of differences between the perceived stimuli to a number that is behaviourally and cognitively useful. So not differentiating stimuli is beneficial only if the diversification of the stimuli is immaterial for achieving the specific goal. The cognitive economy rule is sometimes confronted with the world structure perception rule which states that unlike the set of stimuli presented in laboratory tests, the perceived world is not a structured set of all characteristic occurring contemporaneously with equal likelihood. This is because material objects are perceived as having a correlational structure. Hence it is extremely important to realise that the characteristics which will be perceived depend on many factors related to the functional needs of the perceiving person, who interacts both with his/her physical and social environment. What influences how given characteristics may be determined is the existent system of categories. The presented categorisation rules may impact both the level of category abstraction and the internal layout of the previously built categorisation structures. So categorisation processes can be presented in two basic dimensions: the vertical and the horizontal. – –

The vertical dimension refers to the content level of a given category. The horizontal dimension concerns the division of categories at the same content level.

The vertical categorisation dimension created by cognitive data analysis processes is of crucial importance at the stage of describing and analysing the following areas:

60

4 The fundamentals and development of Cognitive Informatics

– –

– –

images - it is possible to create a mental image which corresponds to the appearance of specimens of a given class as a whole; perception - objects may be first perceived or recognised as specimens of their basic categories, and then additional processing can be used to identify them as representatives of superior or subordinate categories; development - basic objects constitute the first categorisations of specific objects; language - the basic level of a category is very frequently coded with a single sign, but there may be no names for superior and subordinate categories.

In contrast, the horizontal categorisation dimension is of crucial importance in describing and analysing somewhat different areas, including the following: –

– –

–

the processing speed - the reaction time if processing measures are established by observation in research on semantic memory conducted during the information processing process; the speed of learning artificial categories, that is errors; the sequence and probability of units occurring - units coming as reactions on the output are very often treated as certain aspects of the category storage, retrieval or searching; the influence of previous information on the execution of the task, i.e. attitude, precedence - the degree to which an item is a prototype determines whether the previously held information about the category name eases or hinders reactions in matching tasks.

Cognitive categorisation requires not only that the categorisation dimension is determined correctly, but also that the nature of perceived features is determined in cognitive categorisation processes. After a specific categorisation system has been adopted, the features characteristic for the described and interpreted types of data are defined in the most economical and logical way. However, if this system is remote from the limitations that exist, it will be modified to make it consistent with the occurring features by redefining features. Cognitive categorisation processes for data determine to a significant extent the kind of analysis that must be conducted to adopt the correct categorisation method. This is because these processes can, to a various extent, determine the quantity of data collected in the conducted data analysis process and can impact the accuracy of interpreting this data. Subsequent chapters of this book will present examples of semantic categorisation jobs for selected classes of images. Such jobs will be executed by a new class of information systems - UBIAS - designed for understanding the semantics of image data. Already at this point, however, it is worthwhile to mention that the potential for classifying images using their semantic meaning is much greater than that offered by traditional image recognition techniques. This increased potential will be obvious when classifying patterns that are completely different in terms of

4.3 Cognitive science in categorisation tasks

61

the image content (or for instance also the geometry of the shapes) to the same semantic classes determined as a result of cognitive categorisation processes.

5 Cognitive information systems It became possible to develop cognitive information systems on the foundation of intelligent information systems whose purpose was not just the simple analysis of data consisting in recording, processing and interpreting it, but primarily an analysis by understanding and reasoning about the semantic contents of the processed data (e.g. images, biometric patterns, economic data etc.). Every information system which analyses a selected type of patterns and data using certain features characteristic for them keeps in its database some knowledge – indispensable for executing the correct analysis – which forms the basis for generating the system's expectations as to all stages of the interpretation it conducts. As a result of combining certain features found in the analysed type of data with the expectations - generated using this knowledge - about the existing semantic contents of the data or information, cognitive resonance occurs as described previously. Cognitive information systems use methods which determine semantic/structural reasoning techniques serving to interpret patterns [57]. A system which executes a cognitive data analysis very often analyses not just text information or numerical data, but frequently also image data. In this last case, the structure of the image being analysed is compared during the analysis process to the structure of the image which serves as a type of pattern. This comparison is obviously conducted using strings of derivation rules which enable the pattern to be generated unambiguously. These rules, sometimes referred to as productions, are established in a specially derived grammar, which in turn defines a certain formal language, called an image language. Examples of such languages will be presented in subsequent chapters describing example systems for interpreting selected medical images. An image (or information) thus recognised is assigned to the class to which the pattern that represents it belongs. Cognitive analysis used in information systems very frequently uses a syntactic approach which employs certain strictly defined functional stages for the meaning analysis and interpretation of the image [13, 17]. The input image first of all undergoes preprocessing, which comprises: – – –

coding the image with terminal components of the introduced language; approximating the shapes of the analysed objects; and also filtering and preprocessing the input image.

The completion of these stages represents the image anew in the form of hierarchical structures of a semantic tree and subsequent steps of deriving this representation from the initial symbol of the grammar [17, 37]. At the stage of preprocessing image data, an intelligent cognitive recognition system must (in most cases) segment the image, identify picture primitives and also determine the relations between them. The classification proper, combined with the semantic interpretation, consists in recognising whether the specific representation of the input image belongs to the class of images generated by the L. Ogiela and M.R. Ogiela: Cognitive Techniques in Visual Data Interpretation, SCI 228, pp. 63–74. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

64

5 Cognitive information systems

formal language defined by one of the grammars that can be introduced – a sequential, tree or graph grammar – which are used for recognition processes executed during the syntactic analysis performed by the system [17, 37]. The most recent studies of intelligent information systems indicate that only recognising (that is classifying) the analysed image is no longer sufficient, because researchers increasingly frequently propose to employ these systems also for automatic, computer understanding of the image, which means determining its semantics as well. This applies in particular to image data, which can contain deep layers of meaning. Such image data certainly includes medical images. This is why pattern interpretation systems have mainly developed in medical patterns in recent years. In order to support machine semantic reasoning for a selected class of patterns, artificial intelligence techniques are used. These techniques, apart from the simple recognition of the image identified for analysing, can also extract significant semantic information which forms its semantic interpretation, and this in turn makes its full understanding possible. This process applies only to cognitive information systems and is much more complex than just recognition, as the information flow in this case is clearly twoway [53, 57, 61]. In this model, just as in brain structures, the stream of empirical data contained in the subsystem whose job it is to record and analyse the image interferes with the stream of expectations generated. A certain type of interference must occur between the stream of expectations generated by the specified hypothetical meaning of the image and the stream of data obtained by analysing the image currently under consideration. This interference means that some coincidences (of expectations and features found in the image) become more important, while others (both consistent and inconsistent) lose importance. This interference, whose action leads in consequence to achieving cognitive resonance, confirms one of the possible hypotheses (in the case of an image whose contents can be understood), or justifies a statement that there is a non-removable inconsistency between the image currently perceived and all gnostic hypotheses which have an understandable interpretation. The second case means that the attempt to automatically understand the image has failed [49]. So cognitive information systems use cognitive resonance, which is characteristic only for these systems of computer reasoning and distinguishes them from other intelligent information systems. The use of such systems may be varied, as today's science offers them broad possibilities. However, the greatest opportunities for using cognitive information systems are currently found in medicine, as more and more pathological entities occur in disease processes which can affect individual organs of the human body. Such systems may significantly contribute to improving the detectability and recognisability of such entities. Medical images are among the most varied data and have extremely deep and significant meaning interpretation. Cognitive information systems could certainly also help in many other fields of science and everyday life if an attempt were made to add the process of understanding the analysed information and data to in-

5.1 Types and functions of cognitive information systems

65

telligent information systems in the fields of economy, marketing, management, logistics, and defence. The authors of this book have proposed an interesting example of using cognitive systems to analyse economic phenomena [90]. In the next subsection, such systems will be classified to the appropriate group of systems that use resonance processes in data interpretation.

5.1 Types and functions of cognitive information systems Cognitive analysis based on processes of learning about and understanding the studied phenomenon became the foundation for developing intelligent information systems. Such systems are employed in many fields which use data analysis. The most general class of information systems using cognitive analysis for semantic interpretation and reasoning are cognitive categorisation systems, among which an important role is played by systems for computer understanding of the meaning of various types of information as well as data analysis and interpretation systems (Fig. 5.1).

Fig. 5.1. Cognitive system division. Source: developed based on [91, 93]

UBCCS cognitive categorisation systems include UBDSS decision systems developed to take decisions and reason by reference to decisions already taken before. In the meaning of the classical decision theory, decision systems are based on a description of the behaviour of an ideal, perfectly rational decision-maker (a person or a system) in circumstances which require one of a number of well-known options to be chosen [18]. A cognitive system taking a decision follows the utility

66

5 Cognitive information systems

principle and considers all the options which somehow contribute to taking the right decision. The second principle which a cognitive decision system should follow in its operation is the principle of probability which rejects options (decisions) that are not realistic enough. The classical decision theory makes several important assumptions, mainly that the cognitive decision system should have the complete knowledge of all options available (to the system). It is nothing new that a great majority of decisions are taken in conditions of some risk, where it is uncertain what type of possible decisions can be considered, and in particular what the consequences of taking a specific decision may be. Cognitive decision systems are therefore developed to conduct analyses during which all the available options will be considered and their suitability will be compared in every important regard. This can happen only if the system has the resources it needs, which include time and computing capacity. With regard to these two inputs it should emphasised that they which can cause the decisions to be taken within the optimum time or can cause the system to conduct the decision analysis over a time exceeding commonsense usefulness. In that case, the decision taken will become useless, because the time of taking a suitable decision counts in the decision-making process. A cognitive decision system analyses not only based on the possible time of completing (undertaking) the job, but also on the ability to cognitively process all the available options including the consequences of adopting every one of them. A situation in which we have all the cognitive resources necessary to consider every option may be true in large, strong units, e.g. business ones. Cognitive decision systems also follow the rule of the adopted criterion materiality. It is well known that all decision-making criteria cannot always be assigned the same weight, as some of them are more material than others, even when this does not always seem the best. In addition, some decision-making criteria may operate interactively, so they cannot be considered in isolation. Cognitive decision systems analyse and then execute decision-making processes using the classical decision theory, so they try to eliminate new approaches to the subject of decision-taking. These approaches are based on the current decision-making strategies associated with humans. The most important of them is the satisficing strategy which consists in reviewing the available options in a random order and selecting the first one which is sufficiently satisfactory to the decisionmaker [80, 81]. Human decision-making processes are not the best, as people taking decisions do not compare all the available options in every possible regard to select the best one. For a human, this is very often impracticable because of the shortage of time, knowledge, cognitive resources or computing capacity. According to Simon [81], such limitations are due to the bounded rationality of the human mind. The satisficing strategy is a very poor decision-making strategy, as it very frequently does not lead to expected, i.e. beneficial, results. Another strategy followed by people when taking decisions, which is avoided by using by cognitive decision systems, is the elimination by aspects strategy [97],

5.1 Types and functions of cognitive information systems

67

which consists in determining a series of criteria and then eliminating those decisions which do not meet subsequent criteria. In this strategy, an option once rejected is no longer taken into account in subsequent considerations, even if some aspects of it are very appealing (from the perspective of the final decision). Due to the above defects of the human decision-making process, cognitive UBDSS systems make use of only those decision-making processes that can be considered optimum ones. These processes must be quick in terms of the time taken to make the decision, and economical in terms of the complexity and difficulty of cognitive operations necessary in the decision-making process. Examples of such heuristics have been proposed by psychology, and comprise [23]: • The "follow what is most important" rule – decisions are taken in accordance with the rule of following certain criteria, not all of which are of the same importance. The essence of this method is selecting the most important criterion, and then comparing individual options in pairs, every time rejecting the one whose value is lower or unknown in the light of the selected recommendation. • The "follow what has worked recently" rule – decisions are taken based on the criterion that was tried and worked in the most recent attempt of the same kind. The presented decision-making criteria do not always allow the best decision to be taken, and what is more often entail quite a high risk and uncertainty. This is why the concept increasingly frequently considered to absolutely correctly describe the decision making process is one which takes into account the perspective theory [26]. The perspective theory is based on the framing effect, i.e. the impact of the mental representation of the decision-making problem on the content of decisions taken. The cognitive representation of a problem may change completely or to a great extent as a result of the words used to describe it or of taking into account the broader context in which the decision is taken. A person taking a decision makes a comparison to some extent by overlaying a kind of frame on the considered situation, which leads to a specific presentation of the problem and in consequence to taking a decision consistent with the framing process [38]. The operation of cognitive decision systems depends on many significant components of the human decision-making process. However, not all human aspects of decision-making processes can be applied in systems without restrictions, because, as mentioned above, not all the rules mentioned constitute optimum decision-making rules. Cognitive decision systems operate using significant elements of psychological theories transferred to the field of information systems. The cognitive decision-making process itself is based on a complex human psychological process made up of cognitive and motivational elements. Cognitive categorisation systems, in addition to cognitive decision systems, also include UBMSS information management systems built for taking strategic decisions, for instance in enterprises. Cognitive systems of this type place significant weight on the information aspect. This aspect does not only apply to the information acquisition phase, but also to its correct processing and interpretation. UBMSS systems, just as other classes of cognitive categorisation systems, work by using cognitive resonance, but their additional benefit is that the cognitive sys-

68

5 Cognitive information systems

tem created within the information system is founded on the semantic network found in computer information systems. This is why in this type of models the concepts connected with decision systems found in enterprises are kept permanently in the form of a hierarchical network structure made up of nodes and relations between them that link them. In particular nodes of the network, various concept representations are coded, and in addition features characterising these concepts are identified. These features are assigned to concept representations at the lowest possible level of the generality hierarchy. Introducing this type of concept representation meets the proposal of cognitive operation economy in which the economy of the network structure can often be associated with the cost of its operation. The analysis, interpretation and reasoning processes executed by UBMSS systems facilitate the economical (from the financial point of view) use of data contained in the available knowledge base in which concept representations are coded. In the presented UBMSS system, the semantic network described contains, in addition to the concepts coded in the system and presented as network nodes, also the semantic relations that link particular nodes. In this case, the semantic relationship between the present concept representations is expressed as the sum of all links between their designations and features. Concepts that are closely linked are characterised by a value of their mutual links in the network, which value corresponds to a certain community containing all their features. These links, i.e. network paths, are differentiated in the UBMSS system with regard to their weight, which means that the stronger the link between two concept representations, the greater the weight assigned to the path that links them. This allows an easier mutual activation in the data processing operation. The duality of semantic relations occurring between concepts has been taken into account in building UBMSS systems. These relations comprise: – –

Relations built on positive links, and Relations built based on negative paths.

Semantic relations determined based on negative paths are significantly beneficial in the process of taking strategic decisions, because they demonstrate that the greater the semantic distance between concept representations, the faster the decision can be taken that the sentence which positively links the represented concepts is false. The presented models of semantic networks found in UBMSS systems allow semantic relations occurring in specific concept representations to be identified. These models can also be used to develop the next class of cognitive categorisation systems – UBPAS personal authentication systems - which include biometric identification systems, for instance. UBPAS systems are based on two basic groups of memory examination methods, which include direct and indirect methods. In direct methods, the system retrieves from its memory everything that it knows (so this is a knowledge retrieval method) or it recognises elements occurring in the previously applied or learned material in the material now presented to

5.1 Types and functions of cognitive information systems

69

it (this is a recognition method). In the recognition method another possibility is assumed, namely that the system should classify the analysed data, should interpret it and reason about it. This approach to analysis processes in UBPAS systems means that the system should execute a set of actions which cannot be taken without previously acquired knowledge. So UBPAS systems operate using two sets of knowledge – overt and secret knowledge. This is because the system analyses not only the sets known to it before, but also elements that meet definition criteria not known to it, without disclosing the definition itself, but at the same time classifying and independently identifying elements that fulfil the definition or do not fulfil it. This method of operation of UBPAS systems to some extent confirms the presence of secret knowledge based on: • The exclusivity criterion which means that the analysis method allows accessing only the knowledge which is used and necessary to execute a given task. In this case, the processes of data analysis and interpretation used are limited only and solely to those of them which are to some extent independent of other elements not connected with completing the specific task. • The sensitivity criterion, which means that the type of analysis conducted is exhaustive enough to disclose the complete information contained in the knowledge base held in the system and of which the system is in a sense "aware". The essence of this approach is that if the system uses only a part of the overt knowledge about the analysed phenomenon, this knowledge, which has not been collected in the system (because the knowledge could not be accessed), should not be considered secret. In the case of UBPAS systems, just like other classes of cognitive categorisation systems, we cannot accept the thesis of the complete separation of the overt and secret knowledge. It should be emphasised that bases of overt knowledge definitely dominate in these systems, but we cannot completely rule out some contribution of secret knowledge when the knowledge bases are built by expert teams. This approach to overt knowledge means that we can agree with Cleeremans [10] that knowledge is secret when it influences the processing, interpretation and analysis of information without being aware of its features. Regardless of the lack of its awareness, secret knowledge is represented in the memory of UBPAS systems, so it can be retrieved from it and used in analysis, interpretation and reasoning processes. The next class of UBCCS subsystems are UBDAS systems used for varied signal analysis, and UBACS systems which are automatic control systems. These two types of systems are only mentioned in this study, as their role and significance has been described in varied scientific and research publications by other researchers [3, 15, 27, 29, 71, 103, 107, 110, 112]. Apart from the UBDSS, UBMSS and UBPAS systems presented above, in the following chapters the authors will propose UBIAS cognitive systems aimed at analysing image-type data and used to analyse various types of image patterns, in

70

5 Cognitive information systems

particular medical images [41, 48, 54, 61]. UBIAS systems are the fruit of several years of research and their characteristic is the leading subject of chapters 6 and 7. All the types of cognitive categorisation systems presented before have wideranging applications, from economics, sociology and philosophy to technical and defence sciences, medicine or natural science. All the cognitive categorisation system types presented above uses methods of cognitive analysis in their operation to extend the capabilities of classical data analysis techniques in order to reason based on the semantics (meaning) of that data or the analysed information.

5.2 Definition and characteristics of cognitive categorisation systems Both the ongoing scientific research work and the robust development of information systems allow a new class of systems – cognitive categorisation systems – to be introduced for analysing and interpreting data. In this study the authors propose defining computer cognitive categorisation systems as follows: ‘Cognitive categorisation systems’ shall describe intelligent information systems designed for conducting in-depth data analysis based on the semantic contents of this data. Semantic analysis is conducted with algorithms for describing this data based on expert information possessed (for example in the form of knowledge bases) and the processes of machine (computer) perception and understanding of data occurring with the use of computational intelligence methods, for instance mathematical linguistics. In cognitive categorisation systems, the interpreted data, due to its semantics, will also undergo description, analysis and reasoning, which may mean that the analysed data will not only be correctly processed, but also learned and understood. The detailed description of the data analysis process taking place in cognitive categorisation systems is presented in Figure 5.2.

5.2 Definition and characteristics of cognitive categorisation systems

71

Fig. 5.2. Data analysis process in cognitive categorisation systems. Source: own development

Cognitive categorisation systems run processes of data analysis based on biological cognitive processes in which various information processing stages can be distinguished. In the case of cognitive categorisation systems, information is not the only type of data that can be subjected to cognitive categorisation processes which borrow the methods of executing processes composed of various operations from classical information processing. Thanks to these processes, the course of the stimuli received by the system (e.g. outside data) is optimised during the process executed. This process is associated with the assumption of the economical course of processes taking place (necessary in the analysis process). This is why significant cognitive tasks (for the analysis process) are determined for the data undergoing analyses, which leads to the correct description of the cognitive process. During the description of the correct cognitive process, an information overload can take place. This overload can be eliminated only if a stage of selecting data that is immaterial (form the point of view of the analysis conducted) is introduced. This selection, which is possible because the system has a king of ‘attention’ directed at eliminating superfluous information and because it repeats the same actions (operation) many times, leads to choosing increasingly stable and flexible cognitive structures.

72

5 Cognitive information systems

The stage of information overload and superfluous data elimination can be equated with the stage of data preprocessing during which the information contained in the analysed data is reduced in such a way that significant information searched for is not lost, but at the same time the final effect of this process is a new data representation significant for the analysis process conducted. Unlike the defined cognitive processes, cognitive structures are relatively permanent in the process of defining cognitive categorisation systems, and the system can use those structures multiple times in various conditions and situations. Cognitive categorisation systems operate not just depending on the regularity of the cognitive processes executed to correctly classify the data type, but also based on a defined notion apparatus contained in the appropriate type of expert knowledge bases. This type of knowledge bases are created by expert teams using various knowledge elements, convictions and cognitive schemes, and are then utilised in cognitive data analysis and interpretation processes. The information collected by the system is used to confront (compare) the characteristic features it distinguished during the data analysis process with certain expectations which the system has generated based on expert knowledge. This comparison is possible at various layers of data processing, which means that the same data can be analysed with various intensities and care, i.e. at various layers of data processing. The extent to which our data will be analysed depends on the demands of the situation (e.g. the external one) or the cognitive problem formulated. Multi-layer processing and analysis of data is quite widespread, as it reflects one of the elementary features of the human mind. Regardless of the selected data processing and analysis layer, the comparison of expectations and features characteristic for the analysed data leads to cognitive resonance, which causes the analysed data to be understood. Data understanding processes are based on the phenomenon of understanding the semantic contents of the analysed data, so at this operating stage of the cognitive categorisation systems, semantic reasoning takes place (about the meaning conveyed by the analysed data). The semantic reasoning stage is parallel to the cognitive categorisation process, as a result of which objects are gathered into certain groups whose common characteristic is the set of common features assigned a label (e.g. a verbal one) which serves as the representation (e.g. a linguistic one) for the specific group. The linguistic representation found in cognitive categorisation processes may come from a natural language, and when it is applied in system processes, it also comes from the computer language. Semantic reasoning and cognitive categorisation processes become the starting point for determining the directions of subsequent treatment of the analysed data (i.e. taking a decision for the future).

5.3 Properties of computer cognitive categorisation systems

73

5.3 Properties of computer cognitive categorisation systems Referring to the definition, presented in the previous chapter, of cognitive categorisation systems used for data analysis, interpretation and reasoning based on its semantic contents, it should be emphasised that cognitive categorisation systems have various properties, of which the most important are as follows: • A broad spectrum of analysed data - cognitive categorisation systems can analyse various data, because the cognitive methods based on cognitive resonance which are implemented in the system are universal; • Wide opportunities to use cognitive categorisation systems in various scientific fields; • The use of informatics formalisms and tools which support a semanticallyoriented cognitive reasoning with the use of structural artificial intelligence techniques; • The use of the rule of the unity of cognition and action - the idea behind the operation of cognitive systems is to implement human cognitive processes which are used by categorisation systems, but at the same time combining those cognitive processes with the action which cognitive categorisation systems execute at their last operating stage, that is at the stage of determining the directions of subsequent activity; • The ability to freely select the categorisation method depending on the chosen representation of the analysed data - the existing categorisation methods (listed below) can be applied in cognitive categorisation systems equally well: – – – –

comparing sets of defining features; comparing sets of all features and establishing the proportion of defining features; determining the semantic distance, i.e. comparing to the adopted standard or prototype; comparing to the first specimen found;

• The use of knowledge bases built from the appropriate expert knowledge collected; • Accounting for the chronometries of systems which are based on the chronometry methods of the mind, which consist in determining the reaction time for analysing a given phenomenon in relation to the stage of transmitting information about the analysed data to the system, the stage at which the system processes this information and the stage at which the cognitive system programs the reaction and executes it; • The ability to deploy cognitive categorisation systems taken from the scientific research domain in the practical world.

74

5 Cognitive information systems

Cognitive categorisation systems are used to understand and analyse data. Their significant feature is that as a result of the broadly-understood data analysis conducted by the system, an in-depth semantic interpretation of this data, and therefore also its cognitive categorisation, become possible. This type of data classification and description allows semantic information to be extracted, which makes it possible to reason at the stage of the data analysis conducted.

6 Understanding-based image analysis systems The purpose of the subject discussed in this book, namely systems for the cognitive interpretation of image data, is to present new classes of information and vision systems called UBIAS (Understanding Based Image Analysis Systems). These systems were developed to support analysing, interpreting and reasoning about the semantic meaning of images. UBIAS systems are one of six classes of cognitive categorisation systems presented in the previous chapter (Fig. 5.1), which are designed not just to simply describe the analysed data, but also conducting its detailed analysis, and primarily at facilitating its meaning interpretation, data understanding as well as indicating and defining the forecasting opportunities. Cognitive categorisation systems have been developed based on intelligent information systems combined with cognitive systems in which cognitive resonance and cognitive analysis occur [41-54]. What makes these systems innovative is that UBIAS systems as a sub-class of cognitive categorisation systems are based on an approach to the analysis, processing, interpreting and reasoning stages characteristic for human cognitive processes. These processes run in the human brain and follow the modular nature of the human mind. The modularity of the mind was proposed by Jerry Fodor [22], to whom the human mind is made up of three types of individual elements, which include receptors, central systems and modules. In this modular approach, receptors acquire information from the environment and transform it into a form accessible to the remaining cognitive mechanisms. Central systems are responsible for reasoning processes, and therefore for thought and cognitive processes. In contrast, the main activity of modules is mediating between receptors and central systems. The operation of cognitive categorisation systems, including UBIAS, is founded on the assumption that every system type can only work on a distinguished class of stimuli, while a cognitive system based on human thinking, cognitive and understanding processes is implemented using a universal data processing system. Its main features include perception and motor activity control. However, the modular approach to data analysis subjects in UBIAS systems on its own could pose the risk that the analysis processes conducted would not be completed, as biological models of the mind also operate based on network models. The information processing functions are carried out at the cognitive process stage by the activity of very numerous units which form a whole network whose nodes, identical with all the units, are activated at the same time. This process is possible if we acknowledge that cognitive processes are executed as parallel and distributed information processing. This type of information processing is founded on an analysis of an individual type of observations made during the process of understanding the investigated image-type data. This process includes a stage at which analysed objects are identified, which is possible by following the concept of models or the concept of features. Both the model and the feature concepts describe (though differently) the L. Ogiela and M.R. Ogiela: Cognitive Techniques in Visual Data Interpretation, SCI 228, pp. 75–78. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

76

6 Understanding-based image analysis systems

categorisation process consisting in assigning the perceived (analysed) object to a superior object category. Consequently, in the model concept, the analysed object (perceived by the system) is compared to the model of the pattern representing the superior category. In this case, the model may be a specific object, that is the first representative of the given pattern encountered by the system, or the most frequent object among the patterns encountered by the system so far. In contrast, in the feature concept, the decision to recognise a pattern is the result of comparing the features of the categorised object and the standard features. Both types of concepts presented are successfully used in categorisation processes executed by UBIAS systems, where the categorisation process takes place at the time of recognising the object as representing (capable of being assigned to) defined patterns, and this process lasts the longer the more categories must be checked by the cognitive system in the analysis process. During the data analysis process, every cognitive system starts its operation from the stage of identifying the object, consisting of several phases, comprising: –

– –

–

–

data reception – perception processes lead to forming an object representation in the cognitive system being built in a form dependent on the analysed object; creating the data representation as a kind of reflection of the stimulation process taking place at the data reception phase; perceptive classification of the object – the object, and in particular its structure, is compared to the data structures of objects collected within the cognitive system; semantic classification of the object – recalling the data about functions played by the analysed object and data originating from other objects closely connected to those undergoing analysis; object classification – the object is assigned the label or class associated with it.

The object identification stage comes first in the data analysis process executed by UBIAS systems. After the data identification stage, the process of the data analysis proper starts, and during it significant information about the form, shape, structure, content, size etc. of the examined object is identified. At this stage of the cognitive categorisation process, it is tremendously important to define the set of appropriate semantic dimensions (e.g. size, width, length). Every dimension should be allocated the appropriate weight which defines how important this dimension is for defining the specific object undergoing analysis. Every unit is characterised by the distribution of possible values along every appropriate dimension, and this distribution should correspond to experts' weights of relative frequencies of dimension values for the entire class of elements of the analysed object. The significance of the analysed object can be presented as a list of values together with the weights assigned to them, and in addition with a sample of values for the given dimension.

6 Understanding-based image analysis systems

77

The semantic categorisation process is based on the previously mentioned cognitive analysis process, in the course of which the comparative process stage is executed in the following basic phases: –

–

– – – –

–

identifying a list of features of the analysed object and of the category containing features originating from characteristic dimensions and from defining dimensions; comparing the distinguished features with regard to their total similarity, and forming pairs of feature consistency with the expectation of the following links: object feature – cognitive category – expectation; comparing link elements obtained during the second phase and identifying the consistency between them; comparing link elements obtained during the second phase and identifying the inconsistency between them; determining whether a given inconsistency can be eliminated or not; if it is impossible to eliminate the existent inconsistencies observed at the previous phase, rejecting pairs between which those inconsistencies occur (as pairs not representative for the conducted analysis process); accepting only and solely those links in which a full consistency of links was observed for the further analysis process.

The stage of the data analysis proper ends when consistency appears in cognitive resonance. The appearance of this consistency becomes the starting point for executing the data understanding process using the links between object features, the expectations generated using the expert knowledge base possessed and the indications of the appropriate cognitive category. At this stage, significant semantic contents of the analysed object are distinguished and their importance for the conducted analysis process is identified. This is because data understanding is based on the process of semantic data interpretation according to its semantic contents. The data understanding process ends in the reasoning phase during which significant forecasting elements are found that indicate the directions in which the studied phenomenon many develop or further data changes that can take place some time after the completion of the data analysis process. The novelty which sets cognitive categorisation systems, including UBIAS ones, apart is that they are understood as those systems that can signal their readiness to answer (the question formulated or an interpretation problem) in a not fully determined environment, in conditions of uncertainty, when the right behaviours cannot be determined algorithmically, and at the same time they are the systems that make success the most likely. Intelligence understood in this way is built on many levels (intelligence levels) determined by three parameters: – –

the computing power and the high memory capacity of the system; the automatic searching for data and the automatic selection of its processing routines when the system is used to find solutions to problems which are not completely known at the time the system is built. In this context, the intelligence of the system boils down to the process of learning and

78

6 Understanding-based image analysis systems

–

adapting, of modelling the worlds, of generating behaviours, assessing opinions and communicating (e.g. linguistically); The quality and quantity of information collected in the system (both in terms of its semantics and its quantity resulting from information theory propositions.

Cognitive categorisation systems are designed not just to solve indicated problems, but also to foresee changes which may take place in the future in order to anticipate them and take preventive steps. Generally, these systems are characterised by action aimed at maximising the likelihood that the system will achieve complete success, whereas success may be defined in very different ways depending on the circumstances.

7 UBIAS systems in cognitive interpretation of medical visualization The UBIAS systems described in the previous chapter, designed for analysing medical images, are developed quite rapidly as research has been conducted on analysing and cognitively categorising images of various lesions occuring within the central nervous system [41, 46]. What is novel, however, is research on UBIAS systems for analysing and interpreting images showing complex multi-object structures. One example of such images are views of foot bones. Such images can be acquired in three different projections: dorsoplanar, external lateral or internal lateral, which makes their computer analysis even harder. UBIAS cognitive categorisation systems for the above images of foot bones will be presented in subsequent sub-chapters.

7.1 UBIAS systems for semantically interpreting foot visual data The cognitive analysis of images showing foot bones has been conducted using formalisms of the linguistic description, analysis and interpretation of data, which include such formalisms as graph grammars as well as by identifing and intelligently understanding the analysed X-ray images of bones of the foot. In order to perform a cognitive analysis aimed at understanding the analysed data showing foot bone lesions, a linguistic model was proposed in the form of an image grammar, the purpose of which is to define a language describing the possible cases of foot bone layouts which are within physiological norms and the possible lesions of foot bones. The purpose of the research work conducted was to determine the appropriate formalisms and to check their utility for executing cognitive analysis of the foot bone images considered using this class of medical information systems. This utility will be measured by the effectiveness of executing a job whereby the system detects lesions indicating the presence of selected disease units, among which the following have been distinguished: fractures, deformations, bone displacements and the appearance of an additional bone. The lesions described can be further divided into: – – – – –

various types of foot bone fractures; degenerations leading to skeleton deformation; bone displacements; the appearance of an additional bone among foot bones; the appearance of hematomas, calcifications and various irregularities in the structure of foot bones.

L. Ogiela and M.R. Ogiela: Cognitive Techniques in Visual Data Interpretation, SCI 228, pp. 79–101. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

80

7 UBIAS systems in cognitive interpretation of medical visualization

Cognitive reasoning methods were used in this project to detect all the above groups of pathological phenomena related to foot bones. The results achieved, as will be proven, confirm the suitability of the cognitive approach, although the unanimous identification of all disease units turned out to be extremely difficult due to slight changes in the input data (images) which were used to take the decision to classify the case under consideration to a specific disease entity. Cognitive analysis methods were initially used for foot X-rays to process images taken in the dorsoplanar projection. This is one of the possible projections in which images are acquired in diagnostic examinations of the foot, and it is the one most frequently used in injury radiography. However, before foot visualisations underwent interpretation analysis, it was necessary to execute a sequence of pre-processing operations which helped to extract all bones comprising the foot skeleton from the X-ray image. This stage corresponds to executing the information overload stage during the cognitive resonance process as it allows a significant quantity of data that has no impact on the final diagnosis to be eliminated. To complete this stage, it was necessary to segment the image, label the detected bones, determine their centres of gravity, which centres would then be represented by the apexes of graph descriptions introduced. After its proper segmentation, the image showing bones was subjected to median filtration to smooth out minor irregularities of the contour. Such operations are described in one of the authors' publications [67, 68] in which the same operations were used to segment wrist bones. Completing the necessary preprocessing operations yielded binary images showing contours of all bones. Further analysis was conducted for images so processed to create the graph representation for all the projections considered later.

7.1.1 Analysis of foot images in the dorsoplanar projection The analysis of foot bones in the dorsoplanar projection led to defining a graph used for a model description of the foot bone skeleton (Figure 7.2) which utilises known anatomical regularities of this part of the lower extremity (Figure 7.1).

Fig. 7.1. Names of bones for the dorsoplanar projection of foot images. Source: own development

7.1 UBIAS systems for semantically interpreting foot visual data

81

Fig. 7.2. A graph describing the foot bone skeleton in the dorsoplanar projection. Source: own development

For a so defined, spanned graph describing the foot bone skeleton in the dorsoplanar projection, topographic relationships were introduced to describe the location of particular structures relative to one another, as well as the possible pathological changes within the foot (e.g. bone dislocations). These relations are shown in Figure 7.3.

Fig. 7.3. A relation graph for the dorsoplanar projection of the foot. Source: own development

Introducing spatial relations and the graph representation of the foot bone skeleton served to define the input graph in which all the adjacent foot bones were marked as appropriate for the investigated dorsoplanar projection (Figure 7.4.). This graph shows foot bones, already numbered, to which labels were assigned in accordance with the strategy to search the graph across (bfs/wfs-wide first search). The proposed representation describes foot bones using an IE graph [20,

82

7 UBIAS systems in cognitive interpretation of medical visualization

82] which is an ordered and oriented graph for which the syntactic analysis will start with the distinguished apex numbered 1. The IE graph obtained for a foot bone image is presented in Figure 7.4.

Fig. 7.4. A graph with numbers of adjacent bones marked based on the relation graph for the dorsoplanar foot projection. Source: own development

The definition of such IE graphs is aimed at presenting the topological interrelations which may occur between particular elements of the foot bone structure represented as the nodes and edges of the graph. Those relations and the method of labelling the apexes of the graph spanning foot bones makes it possible to write a linguistic description of the semantics of the lesions searched for and to recognise pathological conditions. For the purpose of the analysis conducted, a formal definition of an EDG (expansive) graph grammar was introduced, which took into account the formulated linguistic description of correct connections between foot bones shown in Figure 7.4. This grammar had the following form:

G1 = ( N , Σ, Γ, ST , P )

where: 1. The set of non-terminal labels of apexes is defined as follows: N= {ST, CALCANEUS, OS NAVICULARE, OS CUBOIDEUM, OS CUNEIFORME MEDIALE, OS CUNEIFORME INTERMEDIUM, OS CUNEIFORME LATERALE, M1, M2, M3, M4, M5} 2. The set of terminal apex labels (also presented in Figure 7.5) is defined as follows: ∑= {c, on, oc, ocm, oci, ocl, m1, m2, m3, m4, m5} Γ= {r, s, t, u, v, w, x, y, z} – the set of edge labels (also shown in Figure 7.3) The ST start symbol P – a finite set of productions shown in Fig. 7.6.

7.1 UBIAS systems for semantically interpreting foot visual data

83

Fig. 7.5. A graph defining interrelations between particular elements of the structure of foot bones for the dorsoplanar projection. Source: own development

The introduction of the above grammar to UBIAS systems for image data analysis oriented towards identifying the semantics of images showing lesions in foot bones showed what the model (in the sense of the mathematical description) structure of a regular foot should be like. This also helped to define a set of grammar rules describing the correct structure of foot bones for the analysed dorsoplanar projection. These rules are shown in Figure 7.6.

Fig. 7.6. A set showing the healthy structure of foot bones including numbers introduced for them. Source: own development

Our analysis of image-type data was aimed at facilitating the computer understanding of the semantics of analysed images, and in this case also of specific le-

84

7 UBIAS systems in cognitive interpretation of medical visualization

sions shown as incorrect connections between neighbouring bones, their dislocations or defects. Figure 7.7 shows the opportunities for describing and diagnosing various disease cases obtained by expanding the set of linguistic rules to include additional grammar rules. Such rules are written next to analysed disease cases.

Fig. 7.7. Examples of using the automatic understanding of foot bone lesions detected by the UBIAS system in the dorsoplanar projection. Source: own development

The presented examples of a cognitive analysis describing the lesions appearing in foot bones show four possible cases, namely: fractures, deformations, bone displacements and the appearance of an additional bone. The types of foot bone lesions detected by an UBIAS system are presented using a selected type of projection - the dorsoplanar projection – if which foot bones are imaged. Obviously, similar solutions can be proposed for the remaining projection types, i.e. the external lateral and internal lateral projections. Such a presentation can also serve to more precisely and completely understand the analysed lesions by cognitively reasoning about them.

7.1 UBIAS systems for semantically interpreting foot visual data

85

7.1.2 Analysis of foot images in the lateral projection In the analysis of complicated medical images containing many structures which may obscure one another, it is necessary to consider various projections in which such images can be acquired and analysed to correctly interpret the lesions which may appear in them. In the case of foot images, apart from the dorsoplanar projection described in the previous chapter, another projection that can be analysed is the lateral one. Usually two types of the lateral projection are considered: the external lateral and the internal lateral projections. Each of these two types of the lateral projection will be discussed below.

7.1.2.1 External lateral projection So another example of using cognitive interpretation of image-type data to analyse data depicting foot bone pathologies is an analysis of images acquired in the lateral projection. We will first present solutions for the external lateral projection. For this projection, the appropriate set of foot bone names shown in Figure 7.8 was adopted and a new graph description of the skeleton of foot bones corresponding to the healthy anatomy of this part of the lower extremity in the external lateral projection was introduced (Figure 7.9).

Fig. 7.8. Names of bones for the external lateral projection of foot images. Source: own development

86

7 UBIAS systems in cognitive interpretation of medical visualization

Fig. 7.9. Definition of an EDG graph describing the foot bone skeleton in the external lateral projection. Source: own development

In order to analyse X-rays of foot bones in the external lateral projection, it was also necessary to define the graph representation of these bones showing numbers consistent with the adjacency relations between these structures. Such a definition can be formulated based on a description of a graph identifying topological relations between particular elements, i.e. foot bones. The graph of spatial relationships is show in Figure 7.10.

Fig. 7.10. Mutual spatial relationships between individual foot bones visible in the exterior lateral projection. Source: own development

The use of a graph of such topological relationships to build a graph containing the numbers of adjacent foot bones for the external lateral projection is shown in Figure 7.11.

7.1 UBIAS systems for semantically interpreting foot visual data

87

Fig. 7.11. A graph with numbers of adjacent bones created based on the graph of spatial relationships for the external lateral projection. Source: own development

The next stage in conducting a substantively correct cognitive analysis is to define the appropriate formalism for the linguistic analysis and interpretation of data. In the case of the external lateral projection, the proposed graph grammar has the following form of a set consisting of five elements:

G2 = ( N , Σ, Γ, ST , P )

where: 1. A set of non-terminal labels of apexes: N={ST, TIBIA, FIBULA, TALUS, CALCANEUS, OS NAVICULARE, OS CUBOIDEUM, OS CUNEIFORME INTERMEDIUM, OS CUNEIFORME LATERALE, M1, M2, M3, M4} 2. A set of terminal labels of apexes (presented in Figure 7.12): ∑={t, f, ta, c, on, oc, oci, ocl, m1, m2, m3, m4} Γ={r, s, t, u, v, w, x, y, z} – a set of edge labels (Figure 7.10) ST - the start symbol P – a finite set of productions shown in Fig. 7.13.

88

7 UBIAS systems in cognitive interpretation of medical visualization

Fig. 7.12. Mutual relations between particular elements of the foot bone structure for the external lateral projection. Source: own development

Defining such elements of the grammar is aimed at specifying a set of grammatical rules allowing all cases of images showing the correct structure of foot bones to be interpreted. It should be kept in mind that this is a different set of grammatical rules for every projection. Figure 7.13 shows a set of graphs defining the correct structure of foot bones visible in the external lateral projection.

Fig. 7.13. A set showing the healthy structure of foot bones including their numbers for the external lateral projection. Source: own development

Determining the correct relationships and the structure of foot bones (seen in the external lateral projection) in the presented way makes it possible to conduct a meaning analysis in UBIAS systems, which, for such projections of foot images, can yield the results of reasoning about and interpreting selected types of fractures

7.1 UBIAS systems for semantically interpreting foot visual data

89

and pathological conditions illustrated in Fig. 7.14. This figure shows additional grammatical rules allowing various kinds of foot skeleton irregularities seen in images in the external lateral projection to be interpreted.

Fig. 7.14. Automatic understanding of foot bone lesions detected by an UBIAS system for the external lateral projection. Source: own development

The presented type of UBIAS system for the automatic understanding and analysis of foot bone X-rays in the external lateral projection allows various kinds of fractures, irregularities of the foot structure, and also lesions in the form of haematomas to be detected. 7.1.2.2 Internal lateral projection Another example of using the approach presented is the semantic analysis of foot bone fractures in the internal lateral projection. The appropriate set of foot bone names (Figure 7.15) was adopted and a linguistic description representing the foot bone skeleton in accordance with Figure 7.16 was introduced for this projection.

90

7 UBIAS systems in cognitive interpretation of medical visualization

Fig. 7.15. Names of bones for the internal lateral projection of foot images. Source: own development

Fig. 7.16. EDG graph definition for the internal lateral projection of the foot. Source: own development

Just as in the previous cases, the definitions of topological relations between particular bone types (Figure 7.17) were used to define a graph on which the numbers of the present, adjacent foot bones in the internal lateral projection were marked (Figure 7.18).

Fig. 7.17. Diagram of spatial relationships between foot bones for the internal lateral projection. Source: own development

7.1 UBIAS systems for semantically interpreting foot visual data

91

Fig. 7.18. A graph with numbers of adjacent bones created based on the graph of spatial relationships. Source: own development

In the case of the internal lateral projection, the proposed graph grammar with derivation rules has the following form:

G3 = ( N , Σ, Γ, ST , P )

1. A set of non-terminal labels of apexes: N={ST, TIBIA, FIBULA, TALUS, CALCANEUS, OS NAVICULARE, OS CUNEIFORME INTERMEDIUM, OS CUNEIFORME MEDIALE, M1, M2} 2. A set of terminal labels of apexes (shown in Figure 7.19): ∑={t, f, ta, c, on, oc, oci, ocm, m1, m2} Γ={r, s, t, u, v, w, x, y, z} – a set of edge labels (Figure 7.17) ST - the start symbol P – a finite set of productions shown in Fig. 7.20.

Fig. 7.19. Interrelations between particular elements of the structure of foot bones in the internal lateral projection. Source: own development

92

7 UBIAS systems in cognitive interpretation of medical visualization

Introducing the above grammar elements serves to define a set of derivation rules which presents the healthy structure of foot bones shown in Fig. 7.20 as a set of graphs specifying the correct structure of foot bones in the internal lateral projection.

Fig. 7.20. A set showing the healthy structure of foot bones including their numbers in the internal lateral projection. Source: own development

Defining the correct spatial relationships and the healthy structure of foot bones makes it possible to analyse the meaning of foot images in the internal lateral projection. Figure 7.21 shows example results of automatic reasoning about and an interpretation of selected types of fractures and selected pathological cases appearing in the structures of foot bones in the internal lateral projection. Such diagnoses became possible because additional rules specified in Figure 7.21 were added to the grammar rule set.

7.1 UBIAS systems for semantically interpreting foot visual data

93

Fig. 7.21. Automatic understanding of foot bone lesions detected by an UBIAS system in the internal lateral projection. Source: own development

In the cognitive analysis of X-ray images of foot bones for the purpose of determining the semantics of the analysed foot bone data, images showing such pathologies as bone fractures, deformations and dislocations underwent the interpretation process. To summarise the above considerations about the cognitive analysis of foot bone images, it can be said that the three types of foot projections described above became the basis for introducing definitions of grammars and formal descriptions that are to support the in-depth analysis, interpretation and the semantic description of analysed image data. The analysis and reasoning conducted were not about the simple identification of the pathology, but mainly about identifying the type of that pathology and interpreting its meaning based on the semantics contained in the image. Such an interpretation requires determining the type of lesion present, its location, extent, nature etc. This information can be used to determine the medical significance of the detected lesion. One example may be to detect an additional structure which may indicate that injury lesions have occurred, or detecting a condition with bone loss, which may indicate metabolic irregularities or synostoses. These two cases involve completely different diagnostic conclusions and recommendations

94

7 UBIAS systems in cognitive interpretation of medical visualization

In the presented examples, the attempt to automatically understand the data analysed in the UBIAS system was made using the cognitive analysis and interpretation of selected medical images depicting various types of foot bone deformations. These images are very difficult to analyse because of their complex nature and the variety of disease or injury cases that can occur. Consequently, it became necessary to introduce lesion description and classification formalisms robust enough, and also to consider three independent projections of the analysed images. The depth of analysis conducted confirms the legitimacy of using not just one, but as many as three projections of the foot, as there may be cases of lesions that are visible in only one of them. The use of sufficiently robust formalisms of linguistic description and meaning analysis of data makes it possible to conduct a complete analysis of foot bone images, comprising semantic reasoning and the identification of a specific type of a pathology, but can also serve to indicate specific therapeutic recommendations in the diagnostic/treatment process conducted by a specialist physician (obviously taking into consideration additional information from the patient's disease history, treatment progress so far etc.). The examples presented earlier showed the results of a semantic, meaningbased interpretation of analysed lesions found in the area of foot bones in various projections. The efficacy of applying a cognitive analysis to such problems is presented in a table to compare the results obtained by cognitive analysis with results which can be considered to represent correct diagnoses (Tab. 7.1). Table 7.1. Efficacy of applying cognitive analyses for the meaning-based recognition of selected foot bone disorders Lesion analysed

Number of images Number of images (le- Cognitive analysis efanalysed sions) diagnosed cor- ficacy [%] rectly

Bone fractures

12

10

83,3

Bone displacements

20

18

90,0

Deformations, degenerations

16

14

87,5

Incorrect bone structure

20

16

80,0

Appearance of an additional 10 bone

8

80,0

Total

66

84,6

78

These results were achieved by using graph linguistic methods in a cognitive analysis conducted by semantic reasoning modules of parsers forming part of the proposed diagnostic system. The semantics could be identified by defining additional semantic procedures subordinate to structural rules of introduced graph grammars. The analysed types of foot bone deformation images are identified using such parameters as the type of lesion analysed, its size, location, the scale and number of lesions observed.

7.2 Cognitive systems for supporting bone fracture therapy

95

It seems that the proposed approach has the characteristics of a scientific novelty, as there are no other reports in the literature of similar algorithmic solutions and analyses of highly complex, i.e. multi-object, images. The authors of this book have also completed work on a similar subject, namely the analysis of wrist bones using a graph approach [67,68]. Practical applications of this approach should be used in medical diagnostic support systems. The research conducted by the authors in the field of foot bone image analysis has shown that a cognitive analysis of image-form data may significantly enhance the capabilities of modern information systems and systems supporting medical diagnostics. This research has particularly shown that a correctly built grammar makes it possible to run an analysis and describe selected diagnostic cases precisely, isolating significant semantic information from them about the nature of processes progressing and lesions occurring in foot bone structures. It is worth emphasising that the recognition results described here were obtained by applying cognitive processes which imitate the way in which a specialist reasons: he/she sees the deformation of the organ shown in the medical image made and tries to understand the pathological process which caused the observed deformation to appear, he/she does not just make a simple classification (e.g. of the location or the number of bones) whose sole purpose would be to identify the pattern most similar to the pathological image. In addition, this research has shown that linguistics based on graph image grammars can be used to try and run cognitive analyses of lesions within foot bones.

7.2 Cognitive systems for supporting bone fracture therapy In this chapter we will present cognitive categorisation systems designed for analysing images of long bone fractures. These systems are also classified as UBIAS cognitive systems and are used to interpret and support the diagnostics of various types of long bone fractures [52, 55, 58, 59, 60, 91]. What is crucial for a correctly operating information system supporting the diagnostics of medical images is to develop a method for the cognitive analysis of disease units and types of fractures of long bones. Long bone fractures can be divided into simple ones (only the bone is damaged), complicated ones (both the bone and tissues are damaged) and comminuted fractures (the bone is damaged in several places). By reference to the course of the fissure in the broken bone, fractures are divided into: – – – – –

oblique fractures; transverse fractures; spiral fractures; longitudinal fractures; comminuted fractures; and

96

7 UBIAS systems in cognitive interpretation of medical visualization

–

fractures after which a part of the bone has not returned to the correct position relative to the other part - a displaced fracture.

Examples of long bone fractures are shown in Figure 7.22

Fig. 7.22. Examples of long bone fractures in extremities

A cognitive analysis with the use of the UBIAS system built is aimed at proposing an automatic method of correctly interpreting these extremely complex medical images acquired by imaging fragments of long bone fractures. An attributed grammar in the following form was proposed for analysing fractures of long bones:

G4 = (VN ,VT , SP, STS )

where: VN={DEFORMATION, FRACTURE, FISSURE, TRANSVERSE, SPIRAL, ADHESION, DELAYED_UNION, DISPLACED_M1, DISPLACED_M2, DISPLACED1, DISPLACED2, LONGITUDINAL, A, B, C, D, E, F, G, H} – is a set of non-terminal symbols; VT={'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'} – is a set of terminal symbols where individual elements have been defined as follows: a∈[-10°, 10°], b∈(10°, 70°], c∈(70°, 110°], d∈(110°,170°], e∈(170°, -170°), f∈(-110°, -170°], g∈(-70°, -110°], h∈(-10°, -70°] (Figure 7.23.).

7.2 Cognitive systems for supporting bone fracture therapy

97

Fig. 7.23. Definitions of individual elements of the set of terminals VT. Source: own development

STS=DEFORMATION – the start symbol of the grammar; the SP set is a set of productions shown in Table 7.2. The proposed grammar is used not just for simply analysing the image, but also becomes the starting point for conducting the semantic reasoning about the analysed fractures. The images of long bone fractures presented in this chapter have undergone descriptive analysis leading to the formulation of specific medical diagnoses, and in addition an attempt has been made to reason about the topical and the semantic contents of the analysed image showing a selected long bone fracture. To illustrate the operating method of the UBIAS system, we have presented below automatically generated diagnostic descriptions of detected lesions for several example images of long bone fractures. Particular figures (Figures 7.24-7.27) show example results obtained during the examinations of several selected cases of long bone fractures. All the results presented were obtained by applying an attribute grammar and are an example of a cognitive approach to analysing the medical data (images) considered. The fracture type was identified using semantic procedures of the proposed grammar. Suggested diagnoses were obtained by equipping the system with the appropriate knowledge base built on the basis of qualifications, knowledge and practical experience in the considered classes of long bone fractures possessed by a team of specialists. Figure 7.24 shows an automatic, diagnostic description of a spiral bone fracture, Figure 7.24 of a longitudinal bone fracture, and Figure 7.26 of an extremity fracture after some time of union together with the periosteum forming around it. The UBIAS system recognised the presented lesion as a bone fracture at the stage of forming hard bone matter. The image shows the fracture, but the analysed area is partially filled with the periosteum. Figure 7.27 shows an example of a spiral fracture of the humeral bone with a dislocation.

98

7 UBIAS systems in cognitive interpretation of medical visualization

Table 7.2. Set of derivational rules of the SP grammar

Lesion

Grammar rules

Fissure fracture DEFORMATION → FRACTURE → A FISSURE A FISSURE → H B | H A B

Transverse fracture

DEFORMATION → FRACTURE → A H E | A TRANSVERSE E | A G F E | A TRANSVERSE H E TRANSVERSE → H G | H F

Spiral fracture DEFORMATION → FRACTURE → A SPIRAL A SPIRAL → ADHESION F | ADHESION G F | ADHESION F E | ADHESION F G |HF|GF|FG|FH|F ADHESION → B A H | B H

Displaced frac- DEFORMATION → FRACTURE → DISPLACED_M1 F | DISPLACED1 F ture | DISPLACED_M2 D | DISPLACED2 D DISPLACED_M1 → B A | B G | B H DISPLACED_M2 → H G | H F | H E DISPLACED1 → B A H G | B A H | B A G | B A G H GE

DISPLACED2 → H G F E | H

Delayed union DEFORMATION → FRACTURE → A DELAYED_UNION A fracture DELAYED_UNION → ADHESION ADHESION | ADHESION A ADHESION | ADHESION G ADHESION | ADHESION C ADHESION | ADHESION G A ADHESION | ADHESION G C ADHESION | ADHESION G A C ADHESION | ADHESION A C ADHESION | ADHESION B C ADHESION ADHESION → B A H | B H

Longitudinal fracture Adhesion

DEFORMATION → FRACTURE → A LONGITUDINAL E LONGITUDINAL → TRANSVERSE TRANSVERSE | TRANSVERSE E TRANSVERSE | TRANSVERSE E H | H E TRANSVERSE | H E H DEFORMATION → FRACTURE → A ADHESION E ADHESION → B A H | B H

Elements of the A → 'a' A | 'a' B → 'b' B | 'b' C → 'c' C | 'c' D → 'd' D | 'd' E → 'e' E | 'e' F → 'f' F | 'f' detected lesions G → 'g' G | 'g' H → 'h' H | 'h'

7.2 Cognitive systems for supporting bone fracture therapy

Fig. 7.24. A diagnostic description of a spiral bone fracture. Source: own development

Fig. 7.25. A diagnostic description of a longitudinal bone fracture. Source: own development

99

100

7 UBIAS systems in cognitive interpretation of medical visualization

Fig. 7.26. A diagnostic description of a bone fracture at the stage of hard bone matter generation. Source: own development

Fig. 7.27. A diagnostic description of a spiral bone fracture with dislocation. Source: own development

The UBIAS systems presented in this chapter and designed for analysing images of bone fractures in lower and upper extremities can analyse various types of fractures, including spiral, longitudinal and displaced fractures, fractures at the bone reconstruction (restructuring) stage and the hard bone matter generation stage. Cognitive analysis was used to analyse lesions and pathologies within long bone fractures, and the presented cognitive analysis and interpretation system for image data uses this analysis to reason and analyse based on the semantic information contained in the analysed image.

7.2 Cognitive systems for supporting bone fracture therapy

101

The above examples show a semantic analysis, i.e. an automatically executed meaning-based interpretation of the analysed and detected lesions occurring in long bone fracture cases. It is worth emphasising that in the considered case, the semantic approach was strictly linked with a syntactic analysis, as the presented results were obtained using cognitive (semantic) analysis executed in semantic reasoning modules of the proposed system, but the right semantic actions were selected based on actions assigned to structural rules activated during the parsing of linguistic structures which described (in the selected grammar) the contour shape of the considered organ. The semantic parameters indicated by the system during the cognitive categorisation process can identify the fracture or union shape, its height, width, smoothness of the bone edge (e.g. in case of spiral fractures), the number of visible elements, the frequency of occurrence of the given lesion, the number of deformations within the analysed fracture etc. Investigations conducted on analyses of images showing fractures (lesions) of long bones have shown that cognitive data analysis can be a factor enhancing the capabilities of current information systems and in the case of medical patterns, of systems for medical diagnostic support. During our cognitive data analysis ending in categorising this data, not only is the image quality improved (during its processing) but additional image features are distinguished, thus classifying - if necessary - objects detectable in the image and moving into the domain of the medical significance of the detected deformations, which leads to the complete meaning interpretation of the considered image. During this research it was shown that a correctly built image grammar makes it possible to run an analysis and describe selected medical images precisely, isolating significant semantic information from them about the nature of the processes progressing and lesions occurring. The examples of cognitive data analysis and categorisation presented in this chapter were achieved by a complex cognitive process which imitates the way in which a specialist reasons: he/she sees the deformation of the organ revealed in the medical image used and tries to understand the pathological process which caused the observed deformation to appear, he/she does not just make a mechanical classification whose sole purpose would be to name the disease by finding the pattern most similar to the pathological image in a completely textbook manner.

8 Summary This book presents selected aspects of the development and implementation of cognitive vision systems. The introduction of such systems is the natural consequence of the development of artificial intelligence methods and such scientific disciplines as cognitive informatics. These disciplines now make it possible to design sophisticated, intelligent systems whose job is not just to acquire or collate various categories of data, but also to facilitate their in-depth semantic interpretation. Attempts to define semantics are also frequently made for data in the form of texts. However, if we only consider image data in the form of various types of photographs or visualisations, there has been no wide scale research on their semantic interpretation yet, so one can hardly expect the limited work that has to have already yielded concrete, satisfactory results. The authors of this book are filling this scientific gap by trying to show how cognitive computer systems capable of interpreting images will develop in the near future. Obviously such considerations are presented only using selected image classes, in particular chosen types of medical diagnostic images. However, it seems that the presented approach will have much broader applications and in the future will allow other, no less important patterns to be interpreted. To summarise all the interesting aspects presented in this book, we can observe that computer cognitive categorisation systems generally used for data interpreting and analysis imitate reasoning processes based on human cognitive processes. Such systems have been developed for several years in various directions. The broadest class of cognitive categorisation systems is designed to handle medical images. These images are of great importance because of the results of their analysis, but they also constitute patterns that are difficult to interpret in a simple way. The have one advantage however: they are easy to acquire with equipment that is generally available. Medical images are interpreted by cognitive categorisation systems, particularly the UBIAS class systems described in this book. For a very long time they have been treated as a very interesting type of data, frequently subjected to various types of analyses, from tests of pre-processing methods through methods of analysing the structure of pathological features to classification stages. This is why the authors and their team, who have been conducting very intense research on cognitive data analysis [41, 60, 61, 68, 90], have developed this type of cognitive categorisation systems. The novelty presented in this book are UBIAS systems for the cognitive categorisation of images of foot bones and long bone fractures. The authors have mainly illustrated their presentation of the methodology of executing cognitive analyses on new-generation information systems with examples related to medical image analyses, as in addition to their popularity and availability, it is hard to imagine other image data that would convey such deep layers of meaning (semantic) information. In the case of medical visualisations, the appearance of some lesion or pathology is always of great importance for the patient. So it is not enough to recognise the visible lesion, we should also strive to develop L. Ogiela and M.R. Ogiela: Cognitive Techniques in Visual Data Interpretation, SCI 228, pp. 103–105. © Springer-Verlag Berlin Heidelberg 2009 springerlink.com

104

8 Summary

intelligent solutions that can reason about the nature of visible irregularities in the same way as specialised diagnosticians do. Cognitive analysis applied in the context of the image interpretation problem presented in this book improves and enhances classical image recognition methods, thus offering opportunities to extend them to include deeper semantic interpretations and extracting the contents from the patterns examined. Methods of cognitive data analysis and categorisation have been used to interpret image-type data, which has helped to enhance and develop medical information systems and diagnostic support systems for selected types of lesions and deformations of foot bone structures and for analysing long bone fractures in extremities. It this regard, the type of analysis executed can lead to UBIAS systems generating results which can be of practical utility and assessed in terms of it. In addition, cognitive categorisation systems, particularly UBIAS ones, make us realise certain very significant opportunities offered by cognitive analysis treated as a tool for obtaining not just simple data subjected to analysis processes, but also valuable knowledge components generated by modern information systems. In the design of cognitive systems for data categorisation and analysis it has been empirically proven that modern artificial intelligence methods cross the barrier between the form of data collected in an information system and its contents which serve to understand its meaning. Cognitive data analysis requires various kinds of data pre-processing, various procedures of data extraction and analysis, and also varied formalisms that generate descriptions of the meaning of this data and later support extracting its meaning. This is why using the results of this project in a broader context requires solving many very difficult detailed problems. The presented computer methods of cognitive analysis used to design cognitive data categorisation systems, and thus cognitive information systems, can constitute an additional, very useful and precise tool that can also serve many other purposes. In the case of medical systems, its purpose may be to support the early diagnostics of irregularities found in the examined organ or progressive lesions within it. We can also consider using cognitive categorisation systems in medical data archiving and transmission systems, referred to as PACS, as well as in hospital information systems (HIS) and radiology information systems (RIS). The research conducted and data analysis processes executed by cognitive categorisation systems allow us to claim that cognitive categorisation processes can be used in the future for practical categorisation tasks in other, broader classes of UBCCS systems. This is because the developing methods of cognitive informatics offer a hope for its broad application to analyse various types of data, e.g. biometric or economic. The great computational effectiveness of the presented image analysis algorithms makes the proposed cognitive categorisation methods very useful from a practical perspective. Scientific research may also offer a practical tool for medical practice, where it can be used to extract, recognise and understand diagnostic features of analysed image data.

8 Summary

105

Semantic information about the disease factor or the lesion extracted during syntactic reasoning is used mainly for reasoning about the correct diagnosis, but it can also find other uses, in particular: – –

to determine the progress of therapeutic processes and to predict the future condition of the patient; to build descriptions that index image data in distributed medical databases and to improve processes of context-based searching for semantic image data [94].

The cognitive categorisation systems presented here, designed for intelligent analyses of visual data and semantic reasoning about it, based on human cognitive processes, represent a new proposal that charts the directions of the future development of intelligent analysis and understanding of image data. These methods also contribute to enhancing formalisms and practical applications in the cognitive informatics field.

References 1. Albus, J.S., Meystel, A.M.: Engineering of Mind – An Introduction to the Science of Intelligent Systems. John Wiley & Sons Inc., Chichester (2001) 2. Anderson, J.R.: The Architecture of Cognition. Harvard Univ. Press, Cambridge (1983) 3. Anderson, J.R.: ICCI 2005 Proc. 7th International Conference on Cognitive Informatics. IEEE CS 17 Press, Stanford University (2005) 4. Bechtel, W., Abrahamsen, A., Graham, G.: The live of cognitive science. In: Bechtel, W., Graham, G. (eds.) A companion of cognitive science, Blackwell Publishers, Oxford (1998) 5. Bender, E.A.: Mathematical Methods in Artificial Intelligence. IEEE CS Press, Los Alamitos (1996) 6. Bodzioch, S., Ogiela, M.R.: New Approach to Gallbladder Ultrasonic Images Analysis and Lesions Recognition. Computerized Medical Imaging and Graphics 33, 154–170 (2009) 7. Bodzioch, S., Ogiela, M.R.: Effective filtration techniques for gallbladder ultrasound images with variable contrast. Journal of Signal Processing Systems 54, 127–144 (2009) 8. Borod, J.C.: Interhemispheric and Intrahemispheric Control of Emotion: A Focus on Unilateral Brain Damage. Journal of Consulting and Clinical Psychology 60, 339–348 (1992) 9. Branquinho, J. (ed.): The Foundations of Cognitive Science. Clarendon Press, Oxford (2001) 10. Cleeremans, A.: Principles for implicit learning. In: Berry, D.C. (ed.) How implicit is implicit learning? Oxford University Press, London (1997) 11. Cohen, H., Lefebvre, C. (eds.): Handbook of Categorization in Cognitive Science. Elsevier, The Netherlands (2005) 12. Davidson, R.: Prolegomenon to Emotion: Gleanings from Neuropsychology. Cognition and Emotion 6, 245–268 (1992) 13. Davis, L.S.: Foundations of image understanding. Kluwer Academic Publishers, Dordrecht (2001) 14. Dennett, D.C.: Consciousness Explained, Little, Brown and Co., Boston (1991) 15. Dreyfus, H.: What Computers Still Can’t Do. MIT Press, Cambridge (1992) 16. Driscoll, M.: Psychology of Learning for Instruction. Allyn and Bacon (1991) 17. Duda, R., Hart, P., Stork, D.: Pattern Classification, 2nd edn. Wiley-Interscience Publication, Hoboken (2001) 18. Edwards, W.: The theory of decision making. Psychological Bulletin 51, 380–417 (1953) 19. Ekman, P.: Facial Expressions and Emotion. American Psychologist 48, 348–392 (1993) 20. Flasiński, M.: On the Parsing of Deterministic Graph Languages for Syntactic Pattern Recognition. Pattern Recognition 26, 1–16 (1993) 21. Flasiński, M.: Inference of Parsable Graph Grammars for Syntactic Pattern Recognition. Fundamenta Informaticae 80, 379–413 (2007) 22. Fodor, J.A.: The modularity of mind. MIT Press, Cambridge (1983) 23. Gigerenzer, G., Todd, P.M. (eds.): Simple heuristics that make us smart. Oxford University Press, Oxford (1999) 24. Gray, P.: Psychology. Worth Publishers, Inc., New York (1994) 25. Kagan, J.: Unstable Ideas: Temperament, Cognition and Self. Harvard University Press (1989) 26. Kahneman, D., Tversky, A.: Prospect theory: An analysis of decision under risk. Econometrica 47, 263–293 (1979) 27. Kephart, J., Chess, D.: The Vision of Autonomic Computing. Computer 26(1), 41–50 (2003) 28. Kihlstrom, J.F.: The Cognitive Unconscious. Science 237, 1445–1452 (1987) 29. Kinsner, W.: Towards Cognitive Machines: Multiscale Measures and Analysis. In: Proc. 5th International Conference on Cognitive Informatics (ICCI 2006), pp. 8–14. IEEE CS Press, Los Alamitos (2006)

108

References

30. Latombe, J.C.: Probabilistic Roadmaps: A Motion Planning Approach Based on Active Learning. In: Proc. 5th International Conference on Cognitive Informatics (ICCI 2006). IEEE CS Press, Los Alamitos (2006) 31. Lazarus, R.S.: Emotion and Adaptation. Oxford University Press, New York (1991) 32. Leahey, T.H.: A History of Psychology: Main Currents in Psychological Thoughts. Prentice Hall, Upper Saddle River (1997) 33. Mandler, G.: Mind and Body: Psychology of Emotion and Stress. Norton, New York (1984) 34. Marieb, E.N.: Human Anatomy and Physiology. The Benjamin/Cummings Publishing Co., Inc., Redwood City (1992) 35. McClosey, M., Glucksberg, S.: Natural categories: Well-defined or fuzzy sets? Memory and Cognition 6, 462–472 (1978) 36. Meyer-Base, A.: Pattern Recognition for Medical Imaging. Elsevier Academic Press, Amsterdam (2004) 37. Meystel, A.M., Albus, J.S.: Intelligent Systems – Architecture, Design, and Control. Wiley & Sons, Inc., Canada (2002) 38. Minsky, M.L.: A framework for representing knowledge. In: Winston, P.H. (ed.) The psychology of computer vision. McGraw-Hill, New York (1975) 39. Murphy, G.L., Medin, D.L.: The Role of Theories in Conceptual Coherence. Psychological Review 92, 289–315 (1985) 40. Najarian, K., Splinter, R.: Biomedical Signal and Image Processing. CRC Press, Boca Raton (2006) 41. Ogiela, L.: Cognitive Understanding Based Image Analysis Systems (UBIAS) of the Diagnostic Type. In: IEEE International Workshop on Imaging Systems and Techniques – IST 2007, Krakow, Poland, May 4-5 (2007) 42. Ogiela, L.: UBMSS (Understanding Based Managing Support Systems) as an Example of the Application of Cognitive Analysis in Data Analysis. In: CISIM 2007-6th International Conference on Computer Information Systems and Industrial Management Applications, Ełk, Poland, June 28-30, pp. 77–80 (2007) 43. Ogiela, L.: Cognitive systems for medical pattern understanding and diagnosis. In: Lovrek, I., Howlett, R.J., Jain, L.C. (eds.) KES 2008, Part I. LNCS (LNAI), vol. 5177, pp. 394– 400. Springer, Heidelberg (2008) 44. Ogiela, L.: Syntactic approach to cognitive interpretation of medical patterns. In: Xiong, C.-H., Liu, H., Huang, Y., Xiong, Y.L. (eds.) ICIRA 2008. LNCS (LNAI), vol. 5314, pp. 456–462. Springer, Heidelberg (2008) 45. Ogiela, L.: Cognitive Computational Intelligence in Medical Pattern Semantic Understanding. In: 4th International Conference on Natural Computation ICNC 2008, Jinan, China (October 2008) 46. Ogiela, L., Gądek, A.: The Artificial Intelligence Computer Algorithms in Interpretation of Selected CNS Visualization. Bio-Algorithms and Med-Systems Journal 2, 13–18 (2006) 47. Ogiela, L., Tadeusiewicz, R., Ogiela, M.R.: Cognitive analysis in diagnostic DSS-type IT systems. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L.A., Żurada, J.M. (eds.) ICAISC 2006. LNCS (LNAI), vol. 4029, pp. 962–971. Springer, Heidelberg (2006) 48. Ogiela, L., Tadeusiewicz, R., Ogiela, M.R.: Cognitive Approach to Visual Data Interpretation in Medical Information and Recognition Systems. In: Zheng, N., Jiang, X., Lan, X. (eds.) IWICPAS 2006. LNCS, vol. 4153, pp. 244–250. Springer, Heidelberg (2006) 49. Ogiela, L., Tadeusiewicz, R., Ogiela, M.R.: Graph-Based Structural Data Mining In Cognitive Pattern Interpretation. In: Sakakibara, Y., Kobayashi, S., Sato, K., Nishino, T., Tomita, E. (eds.) ICGI 2006. LNCS (LNAI), vol. 4201, pp. 349–350. Springer, Heidelberg (2006) 50. Ogiela, L., Tadeusiewicz, R., Ogiela, M.R.: Cognitive Computing in Intelligent Medical Pattern Recognition Systems. LNCIS, vol. 344, pp. 851–856 (2006)

References

109

51. Ogiela, L., Tadeusiewicz, R., Ogiela, M.R.: Cognitive Informatics In Automatic Pattern Understanding. In: Hang, D., Wang, Y., Kinsner, W. (eds.) Proceedings of the Sixth IEEE International Conference on Cognitive Informatics, ICCI 2007, Lake Tahoe, CA, USA, August 6-8, pp. 79–84 (2007) 52. Ogiela, L., Tadeusiewicz, R., Ogiela, M.R.: Cognitive Computing In Analysis of 2D/3D Medical Images. In: The 2007 International Conference on Intelligent Pervasive Computing – IPC 2007, Jeju Island, Korea, pp. 15–18. IEEE Computer Society, Jeju Island (2007) 53. Ogiela, L., Tadeusiewicz, R., Ogiela, M.R.: Cognitive Categorization in Modeling Decision and Pattern Understanding. In: Torra, V., Narukawa, Y., Yoshida, Y. (eds.) Modeling Decisions for Artificial Intelligence, 4th International Conference, MDAI 2007, Kitakyushu, Japan, pp. 69–75 (2007) 54. Ogiela, L., Tadeusiewicz, R., Ogiela, M.R.: Cognitive Methods in Medical Image Analysis and Interpretation. In: The 4th International Workshop on Medical Image and Augmented Reality, MIAR 2008, August 1-2. The University of Tokyo, Japan (2008) 55. Ogiela, L., Tadeusiewicz, R., Ogiela, M.R.: Cognitive Modeling in Medical Pattern Semantic Understanding. In: The 2nd International Conference on Multimedia and Ubiquitous Engineering MUE 2008, Busan, Korea, pp. 15–18 (2008) 56. Ogiela, L., Tadeusiewicz, R., Ogiela, M.R.: Cognitive techniques in medical information systems. Computers in Biology and Medicine 38, 502–507 (2008) 57. Ogiela, L., Tadeusiewicz, R., Ogiela, M.R.: Al-Cognitive Description in Visual Pattern Mining and Retrieval. In: The Second Asia Modeling & Simulation AMS, Kuala Lumpur, Malaysia, May 13-15, pp. 885–889 (2008) 58. Ogiela, L., Tadeusiewicz, R., Ogiela, M.R.: Cognitive Categorization in Medical Structures Modeling and Image Understanding. In: Li, D., Deng, G. (eds.) 2008 International Congress on Image and Signal Processing, CISP 2008, Sanya, Hainan, China, May 27-30, vol. 4, pp. 560–564 (2008) 59. Ogiela, L., Tadeusiewicz, R., Ogiela, M.R.: Cognitive Approach to Medical Pattern Recognition, Structure Modeling and Image Understanding. In: Peng, Y., Zhang, Y. (eds.) First International Conference on BioMedical Engineering and Informatics, BMEI 2008, Sanya, Hainan, China, May 27-30, pp. 33–37 (2008) 60. Ogiela, L., Tadeusiewicz, R., Ogiela, M.R.: Cognitive Categorizing in UBIAS Intelligent Medical Information Systems. In: Sordo, M., Vaidya, S., Jain, L.C. (eds.) Advanced Computational Intelligence Paradigms in Healthcare 3. Studies in Computational Intelligence (SCI), vol. 107. Springer, Heidelberg (2008) 61. Ogiela, L., Tadeusiewicz, R.: Categorisation in cognitive systems. AGH University Press, Kraków (in Polish) (2009) 62. Ogiela, M.R., Tadeusiewicz, R., Ogiela, L.: Content-based reasoning in intelligent medical information systems. In: Fifth International Conferences on Enterprise Information Systems ICEIS 2003, Angers, France, pp. 503–506 (2003) 63. Ogiela, M.R., Tadeusiewicz, R., Ogiela, L.: Cognitive reasoning in Intelligent Medical Information Systems – Visual Data Perception Algorithms in Medical Decision Support Systems. In: ICEIS 2004, Sixth International Conference on Enterprise Information Systems, Porto, Portugal, April 14-17, pp. 222–225 (2004) 64. Ogiela, M.R., Tadeusiewicz, R., Ogiela, L.: Medical Visualisation Intelligent Content Analysis and Understanding. In: WIAMIS 2004 - 5th International Workshop on Image Analysis for Multimedia Interactive Services, Lisboa, Portugal (2004) 65. Ogiela, M.R., Tadeusiewicz, R., Ogiela, L.: Intelligent Semantic Information Retrieval In Medical Pattern Cognitive Analysis. In: Gervasi, O., Gavrilova, M.L., Kumar, V., Laganá, A., Lee, H.P., Mun, Y., Taniar, D., Tan, C.J.K. (eds.) ICCSA 2005. LNCS, vol. 3483, pp. 852–857. Springer, Heidelberg (2005) 66. Ogiela, M.R., Tadeusiewicz, R., Ogiela, L.: Cognitive Visio Techniques in Medical Image Processing and Analysis. In: Healthcom 2005 - 7th International Workshop on Enterprise Networking and Computing in Healthcare Industry, Busan, Korea, June 23-25, pp. 120– 123 (2005)

110

References

67. Ogiela, M.R., Tadeusiewicz, R., Ogiela, L.: Graph image language techniques supporting radiological, hand image interpretations. Computer Vision and Image Understanding 103, 112–120 (2006) 68. Ogiela, M.R., Tadeusiewicz, R., Ogiela, L.: Image languages in intelligent radiological palm diagnostics. Pattern Recognition 39, 2157–2165 (2006) 69. Ogiela, M.R., Tadeusiewicz, R.: Modern Computational Intelligence Methods for the Interpretation of Medical Images. Springer, Heidelberg (2008) 70. Pavlidis, T.: Algorithms for Graphics and Image Processing. Computer Science Press, Rockville (1982) 71. Patel, D., Patel, S., Wang, Y. (eds.): Cognitive Informatics: Proc. 2nd IEEE International Conference (ICCI 2003). IEEE CS Press, Los Alamitos (2003) 72. Pedrycz, W. (ed.): Granular Computing: An Emerging Paradigm. Physica-Verlag, Heidelberg (2001) 73. Pinel, J.P.J.: Biopsychology. Allyn and Bacon, Needham Heights (1997) 74. Pratt, W.K.: Digital Image Processing. Wiley & Sons, New York (1991) 75. Rapcsak, S.Z., Comer, J.F., Rubens, A.B.: Anomia for Facial Expressions: Neuropsychological Mechanisms and Anatomical Correlates. Brain and Language 45, 233–252 (1993) 76. Reisberg, D.: Cognition. In: Exploring the science of the mind, 2nd edn. Norton &Company (2001) 77. Ritter, G.X., Wilson, J.N.: Computer Vision Algorithms in Image Algebra. CRC Press, Boca Raton (1996) 78. Schachter, S., Singer, J.: Cognitive, Social and Physiological Determinants of Emotional State. Psychological Review 63, 379–399 (1962) 79. Searle, J.: Mind, brains and programs. Behavioral and Brain Sciences 3, 417–424 (1980) 80. Simon, H.A.: Administrative behavior. Macmillan, New York (1947) 81. Simon, H.A.: Information-processing theory of human problem solving. In: Estes, W.K. (ed.) Handbook of learning and cognitive processes: Human information. Lawrence Erlbaum, Oxford (1987) 82. Skomorowski, M.: Use of random graph parsing for scene labeling by probabilistic relaxation. Pattern Recognition Letters 20, 949–956 (1999) 83. Skomorowski, M.: Syntactic recognition of distorted patterns by means of random graph parsing. Pattern Recognition Letters 28, 572–581 (2007) 84. Sloane, P., MacHale, D.: Perplexing Lateral Thinking Puzzles. Sterling, New York (1997) 85. Smith, R.E.: Psychology. West Publishing Co., St. Paul (1993) 86. Solso, R.L. (ed.): Mind and Brain Science in the 21st Century. MIT Press, Cambridge (1999) 87. Sonka, M., Fitzpatrick, J.M.: Medical Imaging. In: Medical Image processing and Analysis, vol. 2. SPIE Press (2000) 88. Sternberg, R.J.: Search of the Human Mind. Harcourt Brace & Co., Orlando (1998) 89. Tadeusiewicz, R., Ogiela, L., Ogiela, M.R.: Cognitive Analysis Techniques in Business Planning and Decision Support Systems. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L.A., Żurada, J.M. (eds.) ICAISC 2006. LNCS (LNAI), vol. 4029, pp. 1027–1039. Springer, Heidelberg (2006) 90. Tadeusiewicz, R., Ogiela, L., Ogiela, M.R.: The automatic understanding approach to systems analysis and design. International Journal of Information Management 28, 38–48 (2008) 91. Tadeusiewicz, R., Ogiela, L.: Selected Cognitive Categorization Systems. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2008. LNCS (LNAI), vol. 5097, pp. 1127–1136. Springer, Heidelberg (2008) 92. Tadeusiewicz, R., Ogiela, L.: Modern Methods for the Cognitive Analysis of Economic Data and Text Documents and Their Application in Enterprise Management. In: IEEE Proceedings 7th International Conference Computer Information Systems and Industrial Management Applications, June 26-28, pp. 11–23. The Czech Republic, Ostrava (2008)

References

111

93. Tadeusiewicz, R., Ogiela, L.: Categorization in Cognitive Systems. In: Svetoslav, N. (ed.) Fifth International Conference of Applied Mathematics and Computing FICAMC 2008, Plovdiv, Bulgaria, August 12-18, vol. 451 (2008) 94. Tadeusiewicz, R., Ogiela, M.R.: Artificial intelligence techniques in retrieval of visual data semantic information. In: Meenasaivas, E., et al. (eds.) AWIC 2003. LNCS (LNAI), vol. 2663, pp. 18–27 (2003) 95. Tadeusiewicz, R., Ogiela, M.R.: Medical Image Understanding Technology. Springer, Heidelberg (2004) 96. Tadeusiewicz, R., Ogiela, M.R., Ogiela, L.: New Approach for Computer Support of Strategic Decision Making in enterprises by means of new Class Understanding Based Management Support Systems. In: IEEE Proceedings 6th International Conference CISIM 2007 – Computer Information Systems and Industrial Management Applications, Ełk, Poland, June 28-30, pp. 9–13 (2007) 97. Tversky, A.: Elimination by aspects: A theory of choice. Psychological Review 79, 281– 299 (1972) 98. Wang, Y.: The Real-Time Process Algebra (RTPA). The International Journal of Annals of Software Engineering 14, 235–274 (2002) 99. Wang, Y.: On Cognitive Informatics. Brain and Mind: A Transdisciplinary Journal of Neuroscience and Neurophilosophy 4(2), 151–167 (2003) 100. Wang, Y.: On Concept Algebra: A Denotational Mathematical Structure for Knowledge and Software Modeling. International Journal of Cognitive Informatics and Natural Intelligence 2(2), 1–19 (2008) 101. Wang, Y.: On System Algebra: A Denotational Mathematical Structure for Abstract System modeling. International Journal of Cognitive Informatics and Natural Intelligence 2(2), 20–42 (2008) 102. Wang, Y., Johnston, R., Smith, M. (eds.): Cognitive Informatics: Proceedings 1st IEEE International Conference (ICCI 2002). IEEE CS Press, Calgary (2002) 103. Wang, Y., Kinsner, W.: Recent Advances in Cognitive Informatics. IEEE Transactions on Systems, Man, and Cybernetics (Part C) 36(2), 121–123 (2006) 104. Wang, Y., Wang, Y., Patel, S., Patel, D.: A Layered Reference Model of the Brain (LRMB). IEEE Transactions on Systems, Man, and Cybernetics (Part C) 36(2), 124–133 (2006) 105. Wang, Y., Zhang, D., Latombe, J.C., Kinsner, W. (eds.): Proc. 7th IEEE International Conference on Cognitive Informatics (ICCI 2008). IEEE CS Press, Los Alamitos (2008) 106. Wilson, R.A., Keil, F.C.: The MIT Encyclopedia of the Cognitive Sciences. MIT Press, Cambridge (2001) 107. Yao, Y., Shi, Z., Wang, Y., Kinsner, W. (eds.): Proc. 5th IEEE International Conference on Cognitive Informatics (ICCI 2006). IEEE Computer Society Press, Los Alamitos (2006) 108. Zadeh, L.A.: Fuzzy Sets and Systems. In: Fox, J. (ed.) Systems Theory. Polytechnic Press, Brooklyn (1965) 109. Zadeh, L.A.: Fuzzy logic, neural networks, and soft computing. Communications of the ACM 37(3), 77–84 (1994) 110. Zadeh, L.A.: Toward human level machine intelligence–Is it achievable? In: Proc. 7th International Conference on Cognitive Informatics (ICCI 2008). IEEE CS Press, Los Alamitos (2008) 111. Zajonc, R.B.: On the Primacy of Affect. American Psychologist 39, 117–123 (1984) 112. Zhang, D., Wang, Y., Kinsner, W. (eds.): Proc. 6th IEEE International Conference on Cognitive Informatics (ICCI 2007). IEEE CS Press, Los Alamitos (2007)

Index

A Artificial brain, 1 Artificial intelligence, 8, 10, 39, 46, 73 Associationism, 57 B Bayes' rule, 17 Behavioural patterns, 11 Biometric identification, 11 Bone fractures, 95, 100 Brain, 32 areas, 45 dynamics, 44 frontal lobe, 36 function, 44 gray matter, 32 hemispheres, 33 occipital lobe, 35 operation, 46 parietal lobe, 35 temporal lobe, 35 white matter, 32 C Cerebral cortex, 33 Character recognition, 11 Cognition, 47, 55 levels, 38 Cognitive analysis, 44 categorisation, 2, 57, 58, 60, 67, 70, 103 computer, 49 decision systems, 66 economy, 59 informatics, 1, 5, 41, 48, 49, 55, 56, 103

information systems, 63, 64 neuropsychology, 29, 36 neuroscience, 29 processes, 56 psychology, 47 science, 12, 29, 41, 46, 48, 56, 58 vision systems, 1, 103 Cognitive resonance, 4, 26, 64, 73, 75 Computational intelligence, 1, 11, 41, 48 D Denotational mathematics, 48 DNA chains, 26 E Emotional processes, 42 Epistemology, 41 F fMRI (functional magnetic resonance imaging), 31 Foot bone dorsoplanar projection, 80 external lateral projection, 85 fractures, 79 images, 79 internal lateral projection, 89 skeleton, 81 Formal languages, 19 Freeman codes, 19 G Gestaltism, 57 Graph grammars, 19

Index

114

I

N

IE graph, 82 Image analysis, 21 classification, 13, 22 perception, 11, 13 preprocessing, 9, 21, 63 similarity, 13 understanding, 12, 20 vision, 23 Information coding, 43 management, 21 memorizing, 43 recording, 43 retrieval, 43 storage, 43 Information system, 46, 64 Information-Matter-Energy (IME) model, 50

Natural intelligence (NI) model, 50 Natural language, 48 Nervous system, 29, 32 Neural networks, 10 Neural system, 46 Neuroimaging methods, 30, 34 Neuron activation functions, 12 Neuroscience, 30

L Languages for shape feature description - LSFD, 19 Layered Reference Model of the Brain (LRMB), 50 Linguistic image representation, 24 M Mathematical linguistic, 12 Medical diagnostics, 7 images, 9, 12, 24 imaging, 23 visualisations, 3 Memory, 30 Action-Buffer, 55 Conscious-Status, 55 Long-Term, 55 Sensory Buffer, 55 Short-Term, 55 Mental process, 29 Mind, 29, 44 MRI (magnetic resonance imaging), 30

O Object-Attribute-Relation (OAR) model, 50 Ontology, 1 P Pattern classification, 7, 12, 21 Pattern recognition, 11, 12 α-NN – Nearest Neighbours method, 15 approximation methods, 11, 15 minimum distance methods, 14 minimum distance techniques, 11 nearest neighbour (NN) method, 11, 15 probabilistic methods, 17 Perception, 42 models, 36, 38, 39 Perceptron, 10 PET (positron emission tomography), 31 Picture description languages - PDL, 19 Picture primitives, 18, 63 R Representationalism, 57 S Semantic classification, 76 information, 48

115

Index

interpretation, 2, 5, 10, 12 reasoning, 5, 72 Spatial resolution, 8 Structural analysis, 7 Syntactic methods, 12 Syntactic pattern recognition, 19, 24

UBCCS systems, 65, 104 UBDSS systems, 67 UBIAS systems, 3, 5, 70, 75, 79 UBMSS systems, 3, 67 UBPAS systems, 68 Unmanned vehicles, 49

T

V

Template matching, 8 Thinking processes, 21 Training set, 7

Vision systems, 21 Voice recognition systems, 1, 11 X

U UBACS systems, 69

X-ray images, 79