Information systems and data compression

INFORMATION SYSTEMS AND DATA COMPRESSION

INFORMATION SYSTEMS AND DATA COMPRESSION

by Jerzy A. Seidler Universitdt Salzburg

\N\UK? ARCHIEF

KLUWER ACADEMIC PUBLISHERS Boston/Dordrecht/London

Distributors for North America: Kluwer Academic Publishers 101 Philip Drive Assinippi Park Norwell, Massachusetts 02061 USA Distributors for all other countries: Kluwer Academic Publishers Group Distribution Centre Post Office Box 322 3300 AH Dordrecht, THE NETHERLANDS

Library of Congress Cataloging-in-Publication Data

A CLP. Catalogue record for this book is available from the Library of Congress.

Copyright © 1997 by Kluwer Academic Publishers All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher, Kluwer Academic Publishers, 101 Philip Drive, Assinippi Park, Norwell, Massachusetts 02061. Printed on acid-free paper. Printed in the United States of America

To Mam ani ?awd

CONTENTS PREFACE

XV

NOTATION

xix

PART 1 BASIC CONCEPTS 1 BASIC FUNCTIONS AND STRUCTURES OF INFORMATION SYSTEMS

1

1.1 FUNDAMENTAL CONCEPTS 1.1.1 Systems pursuing goals 1.1.2 The concept of information 1.1.3 Information systems:Fundamentals 1.1.4 Prototypes of basic information systems

1 1 3 5 7

1.2 ACQUISITION OF INFORMATION 1.2.1 Basic types of states 1.2.2 Information sources

12 12 15

1.3 STRUCTURE OF A CONCRETE INFORMATION 1.3.1 1.3.2 1.3.3 1.3.4

Basic concepts Scalar elementary information and scalar identifier Structured elementary information and/or structured identifier Classification of fundamental structure types

1.4 THE 1.4.1 1.4.2 1.4.3

17 17 19 21 25

SET OF POTENTIAL FORMS OF INFORMATION Basic concepts The discrete set of potential forms of information The continuous set of potential of potential forms of information 1.4.4 Weights of potential forms of information

26 26 28 29

1.5 TRANSFORMATIONS OF INFORMATION 1.5.1 Basic types of information transformations 1.5.2 Reversible transformations 1.5.3 Irreversible transformations 1: basic concepts 1.5.4 Irreversible transformations 2:transformations compressing continuous information 1.5.5 Indeterministic transformations

35 35 37 39 42

1.6 OPTIMIZATION OF TRANSFORMATIONS OF INFORMATION 1.6.1 Indicators of performance of an information transformation 1.6.2 The optimization problem

52 52 55

1.7 INTELLIGENT INFORMATION SYSTEMS 1.7.1 Design information 1.7.2 Intelligent operation in a slowly varying environment

57 57 59

34

51

Vlll

1.8 RELATIONSHIPS BETWEEN THEORY OF INFORMATION SYSTEMS AND OTHER SCIENCES

64

REFERENCES

69

2 EXAMPLES OF INFORMATION SYSTEMS

71

2.1 DATA TRANSMISSION SYSTEMS 2.1.1 Transmission of a single binary information 2.1.2 Transmission of a block of binary pieces of information 2.2 INTELLIGENT DATA TRANSMISSION SYSTEMS 2.2.1 Information about the state of the channel 2.2.2 The partner information

72 72 82 89 90 93

2.3 MULTIPLE ACCESS SYSTEMS 2.3.1 Systems using a common channel 2.3.2 Information transmission networks

94 96 100

2.4 INFORMATION STORAGE SYSTEMS 2.4.1 Primary information about the state of a hierarchical system 2.4.2 Secondary information about the state of objects and systems

105 107 109

2.5 A SYSTEM SIMPLIFYING THE STRUCTURE OF IMAGE INFORMATION

112

2.6 SUBSYSTEMS ASSEMBLING TRAINS OF INFORMATION BLOCKS 2.6.1 Basic concepts 2.6.2 Segmentation by a comma 2.6.3 Segmentation based on length information 2.6.4 Segmentation based on specific structure of the code book 2.6.5 Compression of trains of blocks interleaved with pauses

119 120 121 122 123 125

REFERENCES

127

PART 2 STATES OF SYSTEMS 3 CONCRETE STATE OF A SYSTEM

129

3.1 THE EXTERNAL STATE OF A SYSTEM 3.1.1 Basic descriptions of states 3.1.2 Rough description of a state 3.1.3 The course of an external state in time 3.1.4 The set of potential forms of an external state

130 130 131 136 137

IX

3.2 THE INTERNAL STATE OF A SYSTEM 1: RELATIONSHIPS BETWEEN CONTINUOUS EXTERNAL STATES 3.2.1 Terminal interconnected systems 3.2.2 Relationships between time-continuous states of two-terminal systems 3.2.3 Relationships between time-continuous states of a network of two-terminal systems 3.2.4 Relationships between time-discrete states of a network 3.2.5 A classification of relationships between external states of systems

154

3.3 THE INTERNAL STATE OF A SYSTEM 2: RELATIONSHIPS BETWEEN DISCRETE EXTERNAL STATES 3.3.1 Relationships described by equations 3.3.2 The relationships described by logical expressions 3.3.3 Set of potential forms of an internal state

156 156 159 165

REFERENCES

166

4 STATISTICAL STATE OF A SYSTEM

167

4.1 FREQUENCIES OF OCCURRENCES OF DISCRETE STATES 4.1.1 Basic concepts 4.1.2 The frequencies of joint occurrences of states 4.1.3 Generalizations

168 169 170 173

4.2 FREQUENCIES OF OCCURRENCES OF CONTINUOUS STATES 4.2.1 The discrete approximation of a continuous process 4.2.2 The density of occurrences of potential forms of a continuous state

174 175

138 140 141

144 149

178

4.3 STATISTICAL REGULARITIES 4.3.1 Statistical regularities in trains of discrete states 4.3.2 Statistical regularities in trains of continuous states 4.3.3 Statistical regularities in assembles of systems 4.3.4 Testing the existence of statistical regularities and estimation of probability distribution

180 180 183 185 185

4.4 THE AXIOMATIC APPROACH TO STATISTICAL REGULARITIES 4.4.1 The axioms of probability theory 4.4.2 The random variables 4.4.3 The statistical average 4.4.4 Correlation coefficients and correlation matrix

188 188 189 191 193

4.5 PROTOTYPE PROBABILITY DISTRIBUTIONS 4.5.1 The uniform probability distribution 4.5.2 The gaussian probability distribution

195 196 198

4.6 THE FUNDAMENTAL PROPERTY OF LONG TRAINS OF RANDOM VARIABLES

201

4.7 THE GENERALIZED STATE AND A UNIVERSAL CLASSIFICATION OF STATES AND INFORMATION

205

REFERENCES

208

5 STATISTICAL RELATIONSHIPS

209

5.1 THE ROUGH DESCRIPTION OF STATISTICAL RELATIONSHIPS BY PARAMETERS 5.1.1 The correlation coefficient 5.1.2 The entropy and amount of statistical information 1: discrete states 5.1.3 The entropy and amount of statistical information 2: continuous states

215

5.2 PROTOTYPE STATISTICAL RELATIONSHIPS 5.2.1 Poisson process and derived processes 5.2.2 Gaussian processes

218 219 221

5.3 MARKOV PROCESSES

229

210 210 211

5.4 THE RELATIONSHIPS BETWEEN A STATE AND ITS INDETERMINISTIC TRANSFORMATION 5.4.1 The basic model 5.4.2 Calculation of probability distribution of the transformed state when the noiseless component is exactly known 5.4.3 Calculation of probability distribution of the transformed state when the noiseless component depends on unknown parameters 5.4.4 The rough descriptions of the transformations performed by a communication channel

240

5.5 THE MODELS OF INDETERMINISM OF A STATE RELATIVE TO AVAILABLE INFORMATION

244

REFERENCES

250

233 234 236 238

PART 3 DATA COMPRESSION AND OPTIMIZATION OF INFORMATION SYSTEMS 6 LOSSLESS COMPRESSION OF INFORMATION 6.1 THE VOLUME OF DISCRETE INFORMATION AND ITS COMPRESSION 6.1.1 The indicators of resources needed to process structured discrete information 6.1.2 The effect of transformations of structured discrete information on its volume

251 252 253 259

XI

6.2 EXAMPLES OF LOSSLESS COMPRESSION OF TRAINS OF STRUCTURED INFORMATION 6.2.1 Compression of the train of blocks 1: the potential forms of blocks are known 6.2.2 Arithmetic coding 6.2.3 Compression of a train of blocks 2: the potential forms of blocks are not known

272

6.3 REAL TIME COMPRESSION OF A TRAIN OF BLOCKS WITH IDLE PAUSES 6.3.1 The model of the primary information 6.3.2 The volume of a train of blocks interleaved by idle pauses 6.3.3 The compression indicator

275 275 277 279

6.4 COMPRESSION OF INFORMATION EXHIBITING STATISTICAL REGULARITIES 6.4.1 The basic concepts 6.4.2 The effect of overflow 6.4.3 The indicators of statistical compression

280 281 282 285

261 261 268

6.5 EXAMPLES OF COMPRESSION OF INFORMATION EXHIBITING STATISTICAL REGULARITIES 287 6.5.1 Compression of the trains of blocks separated by idle pauses 287 6.5.2 The minimum statistical volume 292 6.5.3 The choice of size of segments compressed by Huffman algorithm 294 6.6 TRANSFORMATIONS UTILIZING THE STRUCTURE OF CONTINUOUS INFORMATION TO COMPRESS ITS VOLUME 6.6.1 The volume of the prototype continuous information 6.6.2 The volume of structured continuous information and its compression 6.6.3 The statistical volume of continuous information

296 296 300 304

REFERENCES

307

7 DIMENSIONALITY REDUCTION AND QUANTIZATION

309

7.1 SPECTRAL REPRESENTATIONS OF VECTOR INFORMATION 7.1.1 Review of fundamental concepts of K dim geometry 7.1.2 Defmition of the spectrum of block information 7.1.3 Some important properties of spectral transformations

310 310 313 319

Xll

7.2 DECORRELATING SPECTRAL REPRESENTATIONS OF VECTOR INFORMATION 7.2.1 Basic concepts 7.2.2 The eigen vectors of the correlation matrix 7.2.3 The decorrelation based on eigen vectors

321 321 323 325

7.3 REDUCTION OF DIMENSIONALITY OF VECTOR INFORMATION 7.3.1 A study case 7.3.2 The algorithm for dimensionality reduction

327 327 338

7.4 SPECTRAL REPRESENTATIONS AND REDUCTION OF DIMENSIONALITY OF FUNCTION INFORMATION 7.4.1 Basic concepts 7.4.2 Harmonic spectral representations 7.4.3 Deterministic reduction of dimensionality of function-information 7.4.4 Statistical reduction of dimensionality of function-information

342 343 345 349 354

7.5 QUANTIZATION 7.5.1 The recovery of the primary continuous information from quantized information 7.5.2 Quantization of vector information 7.5.3 The current quantization of information

359 365 371

REFERENCES

377

8 STRUCTURES AND FEATURES OF OPTIMAL INFORMATION SYSTEMS

379

8.1 INDICATORS OF INFORMATION SYSTEMS PERFORMANCE 8.1.1 Indicators of systems performance in a concrete situation 8.1.2 Indices characterizing the performance of an information transformation as a whole 8.1.3 Indicators of the performance of an ultimate information transformation 8.1.4 Indicators of the performance of a preliminary information transformation

359

380 380 384 385 388

Xlll

8.2 METHODS OF SOLVING OPTIMIZATION PROBLEMS 8.2.1 Reduction of the minimization problem to search in a set of solutions of an auxiliary equation 8.2.2 Numerical finding of the zero point: the samples of the function are exactly known 8.2.3 Numerical fmding of the zero point: only distorted samples of the function are available 8.2.4 Finding the point of minimum 8.3 OPTIMAL RECOVERY OF DISCRETE INFORMATION 8.3.1 General solution of the optimization problem 8.3.2 Structures of optimal subsystems recovering discrete information in an open system 8.3.3 Structures of optimal subsystems recovering discrete information in a system with feedback 8.4 PERFORMANCE OF OPTIMAL INFORMATION RECOVERY 8.4.1 The general method of calculating the statistical performance indices 8.4.2 Performance of binary information recovery 8.4.3 The performance of optimal recovery of discrete information whenL>2

390 390 394 399 403 408 409 412 421 427 428 431 435

8.5 OPTIMAL RECOVERY OF CONTINUOUS INFORMATION 8.5.1 The solution of the optimization problem 8.5.2 Optimal character of linear recovery rules 8.5.3 Optimal character of intelligent recovery rules with independent state parameter estimation 8.5.4 Universal performance estimations of optimal recovery rules

440 440 443

8.6 OVERALL OPTIMIZATION OF INFORMATION SYSTEMS 8.6.1 The overall optimization of the prototype information system 8.6.2 The optimization of the subsystem providing information about the state of main systems environment

453 453

REFERENCES

466

INDEX

467

445 450

461

PREFACE

The systems for information acquisition, transmission, storage, and processing are crucial for functioning and growth of today's society. Typical of such systems are • measurement and identification systems acquiring information about properties (mechanical, thermal, electrical, optical, chemical, economical, etc.) of objects; • systems for transmission of information (data, speech, music, images); • systems for information storage (primary magnetic, optic storage systems, systems for storage of structured information, data banks); • systems for simplification of structured information (information compression, extraction of features), and • systems transforming information according to given algorithms. The specific character of primary information, of the tasks, and of the technologies used, caused these information systems to develop quite independently in their early stages. However, during the last two decades a strong trend has emerged for integration of those systems on operational, implementation, and design levels. First, there is a tendency to physically integrate various information systems into one large system. Typical examples are integrated systems for data acquisition, transmission, and storage, integrated systems for data, voice, and image processing, surveillance, and remote sensing systems. Second, there is a trend to unify the implementation. The progress of solid state technology has allowed standardized devices for digital signal processing. Third, the transfer of concepts developed primarily for specific types of information systems has become very intensive. Interest has been growing in all areas of information processing and transmission in such concepts as optimization, adaptation, learning, and resources sharing. This in turn makes the design philosophies to become similar (coupling of analytical and simulation techniques, computed aided design, and countermeasures against indeterminate events). This book is concerned with the third trend. It has two objectives. The first is to analyze the concept of information and to develop a universal methodology of intelligent information system design. The second objective is to provide a solid basis for the design of systems for compression of digital data, processes, and images. The combination of the two objectives has two purposes. First, since almost all basic techniques of information processing are used in systems compressing information, they are a representative example of information systems. Second, in all types of information systems we have information compressing subsystems. Often these subsystems are essential for the overall performance of the whole system.

XVI

The basic idea of this book is to look at an information system as a system imbedded between a superior system, for which the information system renders its services, and the environment in which the superior system operates (see Figure 1.1, page 5). Information is used by the superior system to match its operation to the state of its environment. An information system itself is in a similar situation as the superior system. The efficiency of information system's operation can be improved without increasing its material resources, if in the processing of information destined to the superior system, the information system uses auxiliary information about its environment. Such information systems are called intelligent and their analysis is the central topic of this book. A consequence of considering the information system as a component of a larger system is to emphasize the relationships between the methods of designing information systems with methods used in other sciences. The localization of topics presented in this book on the map of sciences is shown in Figure 1.28, page 65. The book consists of three parts. The first part (Chapter 1 and 2) presents the basic ideas in an informal and concise manner. Part two (Chapters 3, 4, and 5) offers a physics-like description of the environment of an information system and of the transformations of information. The third part (Chapters 6 , 7 , and 8) discusses optimization of an information system and matching its operation to the state of environment. Chapters 6 and 7 concentrate on systems compressing information. Methodology of formulating the optimization problems, the methods of their solutions, and methods of analyzing the performance of information systems are presented in the synthesising Chapter 8. Much effort was made to present both the heuristic and analytic aspects of information systems design. The informal interpretations are formulated as separate comments. To present the analytic considerations in a rigorous but readable form, and to keep the size of the book within reasonable limits several techniques have been used: uniform terminology and notation, frequent use of geometrical interpretation, introducing new concepts with simple models and then generalizing them. Because the range of problems covered by the book is broad, a list of related publications would be very long. Therefore, for each topic only few selected publications are cited, usually in this sequence: introductory publications, textbooks, detailed studies, in particular collections of articles. As references for concrete procedure I have identified sources of relevant computer programs so that the reader can experiment with the algorithms. A course emphasizing the relationships between the information system, its environment, and the superior system to which the information system renders its services is missing in most curriculums. Also, common features of various types of information systems are rarely discussed. This book can help to bridge such gaps in two ways. A synthesising course for undergraduate students of electronic engineering, communications, and computer sciences can be based on the first part of the book. All of the material can be used for a course on theory of information systems for graduate students of mentioned specializations. This book can also be interesting for students working on their theses in the area of information compression or intelligent information system design.

xvu The synthesizing approach makes the book useful for system analysts and engineers developing the various types of information systems listed on page xv, especially for those interested with transfer of concepts developed for various types of information systems. The material is presented in an autonomous way so that the reader can go through the book without reaching for several other books. However, to profit fully from the book the reader should be familiar with the fundamental concepts of system theory and its basic mathematical tools as optimization theory or probability theory. Some knowledge of decision theory would be also desirable. ACKNOWLEDGEMENTS Several concepts presented here emerged in discussions with students and colleagues during seminars in the Aloha Systems Laboratory at the University of Hawaii led by Dr Norman Abramson, in the Communication Systems Laboratory at University of Kansas led by Dr David Frost, and in the Mathematical Institute of the Salzburg University directed by Prof Dr Peter Zinterhof. I am also very thankful to Dr J.M. Zurada, director of the VLSI Circuit Design Laboratory at the University of Louisville, for his help in preparing this book. A preliminary review of the material presented in Chapter 1 and in Section 8.6.2 has been published by the Austrian Computer Society'. The comments of Prof Dr Veith Risak from Siemens Research Laboratories in Vienna on this publication contributed to the present form of this material. I am also greatly indebted to Prof Dr Karl Josef Parisot from the Salzburg University for his advice and help in preparing a camera ready copy of the manuscript. Finally, I owe a special debt of gratitude to my publisher and editor.

Jerzy A. Seidler May 13, 1997

' J.Seidler, A Systematic Approach to Intelligent Information System Design, Schriftenreihe der Ostereichischen Computer Gesellschaft, Band 79, R. Oldenbourg, Wien Miinchen 1995.

XIX

NOTATION GENERAL PRINCIPLES. In text, italic indicates a newly appearing or emphasized term or statement, such as "This operation we call quantization.''. In equations, the following typeface conventions are used: Italic roman for a numerical variable such as h,x,Ay\ Italic lowercase bold characters for a string or a vector variable, such as h,x\ Italic uppercase bold characters for an array variable, such as U,X, also for a function assigning a vector, string, or an array to a variable, such as y(w); Script-type uppercase characters for a set, event, or object, such as ^ , 0 ; Shadowed lower case characters for a random variable, such as s, X, also for a stochastic process, such as s(/); Shadowed upper case characters for a multidimensional random variable such as S; Upright standing upper case bold character (Albertus font)-operations acting on non-numerical variables, such as Ess ,L (_>^) . COMPOSITE SYMBOLS A symbol with a bar such as Q denotes a statistical average. A symbol with a wave, such as Q denotes the arithmetical average. A symbol with a hat such as u denotes a reference object in the next neighbor transformation. A symbol with a prime such as x' denotes a number related to a but never denotes differentiation. The elements of a set or components of a structured object are listed within braces such as ^ = {a,Z?,c} or M = {W(1), W(2), W(3)}. Symbols for functions of several variables: a variable is identified by its number in parentheses and written in the same line as the symbol of function, such as x[w(l),w(2)]. The exception is the notation for potential forms of a state or an information. The integer identifying the potential form is written in the subscript; for example jc, is the Ith potential form of the vector information x. A closed interval with the left end x^ and right end x^ is denoted by <x^,x^> . If an interval is opened at one of the ends soft braces are used. A symbol of a function with a dot instead of an argument for example, s{'), V(-) denotes a function, a rule of a transformation considered as a whole. For a more detailed explanation see equation (1.3.2). OP x,Q I ^is a shorthand notation for the problem of finding the x^ maximizing (minimizing) the criterion Q on constraints C. For a more detailed explanation see equation (1.6.6). MNEMONIC ABBREVIATIONS The mnemonic abbreviations characterizing modified variables are written as upright characters in the subscript, and the characters used as abbreviation are printed bold in the accompanying definition in text; for example, "• • and we denote by Jt^ux the auxiliary information.".

XX

Frequently used mnemonic abbreviations: b-bloc, binary buf-buffer c-contmuous, current d-decision DES-design e-error

ma-maximal mi-minimal

pt-pomt r-recovered

T-transmitter th-threshold

mx-matrix

R-receiver

tr-train

nty-non typical

sb-structure blind

ty-typical

seg-segment

vc-vector

sh-shaping

w-white (noise)

n-noiseless, normalized o-optimal

DIM-dimension, NNT-next neighbor transformation; for definition see page 41, info-infromation (only on some figures), D-end of an example. UNIVERSAL SYMBOLS HAVING THE SAME MEANING THROUGHOUT THE BOOK Jl aggregation set A operation of arithmetical averaging b, b side parameters c^^{m,n) the correlation coefficient of random variables x(m) and 2S(m) Cxx correlation matrix with elements c^Jjn.n) C capacity d{x',x") distance between points {x' and x") D operation removing the dependence on details D{X^,) duration of the train {X^,) E operation of statistical averaging, h{n) coefficients determining a linear transformation h{t) function determining a time continuous linear transformation (impulse response) H matrix describing a linear transformation H operation assigning to random variable its entropy H value of entropy / index numbering elements of a train / total number of elements of a train I operation assigning to a pair or random variables the amount of statistical information which the one variable delivers about the other. / index numbering potential forms of discrete information (usually as subscript) L number of potential forms of discrete information i.{X) operation assigning to a discrete set X the number of its elements n index numbering components of a vector, of a string A^ number of components of a vector (its dimensionality), of a string (its length) p{x) density of probability in point x p{x\ C) density of conditional probability in point x on condition C

XXI

P operation assigning to an event its probability q(x, jc*) index of distortions in a given situation, Q index of performance of an information transformation, r, r information retrieved after the fundamental transformation (output of a transmission channel, information retrieved from a mass storage) R rate of information transmission (rate of information delivery) i, s state of an object, of a system S set of potential forms of a state t time T('), T(') transformation of information T^i^ck) structural type of information for defintion see pages 25, 26 w, u information delivered to subsystem of the informatin system U set of potential forms of w,w V, V information produced by a subsystem of the information system V set of potential forms of v,v KO, y(') transformation producing information v,v; if it is necessary the transformation is identified by two subscripts, e.g. V^,y(-), the subscripts are standard symbols used to denote the particular primary and the processed information, v(ji:) volume of information x (resources needed to process it) V(/V) volume (resources needed to process) any potential form of information w, w information put into fundamental infomation processing subsystem (communication channel, storage channel) A', jc, X information delivered by the information source into the information system (primary, working information) X set of potential forms of the primary information y, y information about the state of the environment of the working information system (state information) z, z components of the state of environment influencing the operation of the information system (side factors, noise)

BASIC FUNCTIONS AND STRUCTURES OF INFORMATION SYSTEMS This chapter introduces a framework of concepts that allows to describe a great variety of information systems in the same terms. In particular, it permits a uniform treatment of systems for information acquisition, transmission, storage, and compression, for feature extraction, and for the execution of given algorithms. The concepts introduced in this chapter are illustrated with typical examples of information systems presented in the next chapter. In the last section of this chapter the relationships between the theory of information systems, the sciences providing the tools and sciences using the results of the theory of information are discussed (see Figure 1.28). This helps to localize the topics of this book and shows their roots. This and the next chapter constitute the first part of this book. This part plays a double role: it gives an informal but a global look at the problems considered in the book, and provides material for a systematic, analytic approach to the design of information compression systems and of intelligent information systems discussed in the third part of the book (Chapters 6, 7, and 8). In the first part of the book the formal side is kept as simple as possible but often heuristic arguments are used. Therefore, only a general background is needed to read this and the next chapter. Most concepts that are introduced in this chapter in a concise form are analyzed in detail in the second and third part of the book. The number of the corresponding section is given and in this section the related publications can be found. Only publications related to the last section are cited here.

1.1 FUNDAMENTAL CONCEPTS The information system is a service system that provides to the user information about the environment in which the user operates. The principal task of the information system is to transform available primary information about the environment into ultimate information that can be directly applied by the user to pursue more efficiently his goals. This consideration of information systems therefore begins with a description of users of information. 1.1.1 SYSTEMS PURSUING GOALS Characteristic of living organisms is their drive to survive. This fundamental urge induces a great variety of secondary goals. Thus, the activities of living organisms, especially of humans can be characterised as purposeful and the living organisms may be called biological systems pursuing goals.

2

Chapter 1 Basic Functions and Structures of Information Systems

Human goals range from basic existential to sophisticated intellectual goals. To realize a more complicated goal, humans join in a group equipped with tools and organize a system that as a whole, pursues the goal. Examples of such systems are production, power, transponation, economic, administration, and educational systems. Information systems such as postal or telephone systems, also belong to this class. Humans project their goals onto automatic devices and design them so that they adjust their operation to the changing state of the environment in which they operate. We also may say that such devices perform purposeful actions and call them technical systems pursuing goals. Examples of such systems are the plethora of control systems, from simple mechanical controllers to sophisticated navigation systems. A purposeful activity is performed in an environment by special acting subsystems that can directly influence the environment. These subsystems are controlled by a subsystem making decisions about actions. In the case of human purposeful activities the decisions may have a hybrid character: some classes of them are made by technical devices, and others by people. The resources of an acting subsystem, in particular the available energy that it can utilize, are limited. Therefore, it is essential to adjust decisions about the actions to the state of the environment, or more precisely, to the components of the state of the environment that influence the outcome of the activity. These components are called relevant state components. The relevant state components are usually not directly accessible for making decisions. However, it is often possible to acquire some features of these components that allow better decisions to be made about the purposeful actions. Such features of state components are called information. Information can be presented in two fundamental forms: static and dynamic. Static information is presented as a configuration of some standard objects located on a carrier. Static information is suitable for storage or mechanical transportation. Typical technical carriers of static information are paper, films, magnetic carriers, and electronic solid state devices. Examples of carriers of static biological information are hormones, DNA molecules, and special blood cells. Often the standard objects are ordered either on a string (lineal information) or on a plane (planar information). The information presented in this text is in principle a string information; the standard objects are the characters. Genetic information carried by DNA can also be considered as lineal information. The standard objects are the four bases: adenine, cytosine, guanine, and thymine, which are arranged in a helix. Information presented as a time process is called dynamic information. This form is suitable both for its transportation and for transformations. The fundamental carriers of dynamic information about the state of distant objects are the waves, particularly electromagnetic or acoustic waves. For example, visual information is delivered by electromagnetic waves in the optical range. The disadvantage of presenting information in the wave form is that only simple transformations of information in this form (such as transformations by lenses) are now feasible.

1.1 Fundamental Concepts

3

Most important carrier of dynamic information over short ranges is electrical current. The fundamental advantage of presenting information in such a form is that even very complicated transformations of information can easily be performed. If primary information is presented in a non electrical form (such as mechanical or chemical), either static or dynamic, it is not difficult to convert it into the electrical form using very efficient transducers that are now available. Therefore, most technical processing is now performed on information that has the form of an electric process. Such processing also occurs in biological information systems (nervous subsystems). 1.1.2 THE CONCEPT OF INFORMATION The word "information" was previously used in the intuitive sense. To formalize the considerations on information its workable definition must be introduced. The concept of information may be placed in importance third after the concepts of matter and energy. Matter is the carrier of all events, and energy is related to the states of matter, in particular to its motion and its changes of form. The concept of information is associated with purposeful activities of living organisms, especially of people. Being a fundamental concept like matter or energy, information cannot be defined by means of other basic concepts but rather by the purpose it serves: Information is a function of relevant components of the state of environment, which can be used to improve the quality of a purposeful action performed in the environment.

(1-1.1)

To make this definition more concrete the meaning of the term "state" must be specified. This is a basic concept not only of physics but of several sciences concerned with description of various aspects of the real world. The basic types of states are described in Section 1.2.1. The detailed analysis of the concept of state is the subject of the second part of the book (Chapters 3 , 4 , and 5). The environment in which a purposeful action is taken usually consists of several interacting objects. In other words, the environment is structured. This causes the state to consist of interrelated components, so that the state is also structured. Since information is a function of the state, it too is structured. Its structure usually reflects the structure of the states of the environment. A review of the structures of information is presented in Section 1.3. Definition (1.1.1) stresses that information is related to di purposeful action. Therefore, information is relative. For example, the sound of the engine of a plane may be only a nuisance for passengers but is important information for the pilot. The relationship between information and a purposeful activity is essential for two reasons. First, it determines the ultimate form of information so that it is suitable for making decisions about actions. The second reason is deeper. Definition (1.1.1) is based on the assumption that there is an intention to improve the quality of a purposeful action. The resources available to realize a goal are usually limited and it is essential to evaluate the costs of actions taken to achieve the goal. Therefore, an indicator of quality of a purposeful activity is usually defined.

4

Chapter 1 Basic Functions and Structures of Information Systems

The purpose of the information is to make the improvement of the quality of superior systems actions possible. Therefore, the indicators of quality of information supplied to the superior system must be based on indicators of quality of the purposeful activity utilizing the information. The indicators of quality of information can be in turn used to introduce indicators of performance of a transformation of primary information into ultimate information. Such indicators are necessary to compare information systems, to introduce the concept of an optimal information system and of an intelligent information system. Thus, the criteria for design of an information system are determined by features of its superior system. In Section 1.6.1 the indicators of the quality of information and indicators of performance of information transformations are defined; in Section 8.1 they are analyzed in detail. In Section 1.6.1 the problem of optimization of an information system is formulated and in Section 1.7 the concept of intelligent information systems is discussed. In Chapters 6 and 7 examples of optimization of information compressing systems are presented. The structures of basic optimal information systems are derived systematically in Sections 8.3 and 8.5. The performance of optimal and intelligent information systems is considered in Sections 8.4 to and 8.6. THE RELATIONSHIPS BETWEEN INFORMATION SCIENCE AND OTHER SCIENCES Life is the process of an organism adjusting to its environment. Therefore, every living organism utilizes information about the state of its environment. Consequently the concept of information presented in definition (1.1.1) is of primary importance for all sciences concerned with life. Information plays a crucial role in all human activities, from simple everyday activities to social and political activities. Consequently, mutual relationships exist between information science and all sciences concerning systems that realize goals set by people, such as economics and the social sciences. Information is of central importance to the intellectual human activities. Therefore, of all other technical sciences, information science has closest links with human sciences. In particular, mutual relationships exist between philosophy and information sciences. Since the concept of purposeful action does not occur in natural sciences concerned with inanimate matter, such as physics or chemistry, the concept of information defined by (1.1.1) essentially does not occur in these sciences. The exceptions are the areas of these sciences concerned with the utilization of physical or chemical phenomena for some purpose, such as the transformation of unorganized thermal motion into organized motion (thermodynamics), or for measurements. The mentioned relationships between information science and other sciences are discussed in more detail in Section 1.8. See in particular. Figure 1.28 illustrating the relationships. Several references are also cited there.

1.1 Fundamental Concqpts

5

1.1.3 INFORMATION SYSTEMS: FUNDAMENTALS The previous considerations on information are illustrated in Figure 1.1. The system which uses information to pursue goals is called superior system. The properties of the superior system determine the form of the information that the system can accept and determine the indicators of performance of information transformations. The available information about the relevant components of the state of a superior systems environment is called primary information. The form of this information depends on the properties of the environment and of the transducers transforming the state of the environment into the primary information.

-h--

SUPERIOR SYSTEM ACTING SUBSYSTEMS

DECISIONS ABOUT ACTIONS

INFORMATION SYSTEM PRIMARY INFORMATION ABOUT THE STATE OF THE ENVIRONMENT

It I

ULTIMATE INFORMATION

ACTIONS

I

ENVIRONMENT OF THE INFORMATION SYSTEM

ENVIRONMENT OF THE SUPERIOR SYSTEM

Figure 1.1. The basic model of a superior system using information to improve the quality of its purposeful actions. Since they are determined by different factors, the forms of primary information and of information that can be used by the superior system usually do not match. The set of devices transforming primary information into ultimate information that can be directly used by the superior system is called information system. The type of required transformation determines the type of the information system. The fundamental types of transformations needed to match the information delivered by an information source to a superior system are transformations • Of a primary information acquired at a place into information available at another remote place, • Of a primary information acquired at a time interval into information available at a later time interval, • Of a primary structured information into information having a simpler structure, and • Of structured data and a given algorithm into the result of applying the algorithm to the data.

6

Chapter 1 Basic Functions and Structures of Information Systems The corresponding fundamental basic information systems are • Communication systems in panicular, remote measurement systems such as radar or sonar systems, • Information storage systems in particular data banks and hypertext systems, • systems simplifying the structure of information, • information processing systems.

The performance of an information system can be improved by using auxiliary information about the state of the environment of the information system. This auxiliary information is called state information and the system processing it is called state information system. This system renders its services to superior information system, which in turn serves to his superior system. Such a pair of information systems is called hierarchical information system. If the superior information system uses the information about the state of its environment to process optimally the information for its superior system, then the hierarchical information system is called intelligent information system. The components of an information system often operate as autonomous information subsystems and they may be intelligent too. The environment of a subsystem consists of the environment of the superior information system and of the other subsystems of the superior system. Correspondingly, the state information which an intelligent information subsystem may use consists of two components: the information about the state of environment of the superior information system (called external state information) and information about the state of cooperating subsystems of the superior information system (called internal or partner state information). Chapter 2 presents several examples of information systems with intelligent subsystems using both external and internal state information. In general the superior system in not an information system and the analysis of relationships between the superior system and serving information system is complicated. In a hierarchical information system both systems have a similar character. This greatly simplifies the analysis of the relationships between a superior system and its information system. Such an analysis is one the main topics of this book. THE PROTOTYPE STRUCTURE OF INFORMATION SYSTEMS A survey of technical information systems shows, that the simplest information system has the chain structure shown in Figure 1.2. Such a system is called the prototype {information) system. Other, more complicated information systems, such as those with feedback or multiple access systems considered in Chapter 2 can be decomposed into prototype information systems. The first component of the prototype system is the interface coupling the information system with the environment of the superior system. It transforms the relevant components of the state environment of the superior system into xht primary information (also called message in communications, primary data in data processing, observation in measurements). This interface is called information source.

1.1 Fundamental Concepts

E N V I INFO PRELIMINARY FUNDAMENTAL ULTIMATE SUPERIOR R TRANSFORMATION TRANSFORMATION SOURCE TRANSFORMATION SYSTEM "*" X V O s tate r JC* N preprocessed prim ary info processed info ultimate info M (message, info, (received signal. (recovered E primary data, (transmitted retrieved data. message. N observation) signal. record, composed recovered info, T record, info unit info block.) decision simplified info,) conclusion)

E N \r

^ I ^ R actions O I

N

M E N T

^^^

Figure 1.2. The prototype structure of information systems. The names used in communications, databases, and other areas are given in parentheses. For examples of the prototype structure, see Figures 1.3 and 1.4.

The properties of primary information and the basic information processing resources are usually predetermined. Therefore, the type and structure of the primary information must be changed to be suitable for subsequent processing by the information system. Such an information is cMtd preprocessed information and the transformation converting primary information into preprocessed information is cdlltd preliminary transformation. The main part of processing is usually performed by a component sub-system, which is usually costly and standardized. This process is called fundamental information transformation, ihtsnbsysitmptrfonmngiifundamental subsystem, and the produced mform2X\on processed information. The last component of the prototype system is the interface coupling the information system and the superior system. Its task is to transform the information delivered by Uie fundamental information processing system into information that can be used directly by the superior system. This transformation is called ultimate transformation. If a human is the superior system this transformation should display the information in the best perceivable form. In some cases primary information is suitable for direct utilization by the superior system, but its structure had to be changed to match it to the properties of a given fundamental information processing subsystem. Then the task of the ultimate transformation is to revert the preliminary transformation. Often the information system is required to take over some responsibilities of the decision-making subsystem of the superior system. Then the ultimate transformation must not only match the structure of produced information to the properties of the superior system, but it also must extract from the primary information the features relevant to the superior system. In most cases, this implies a simplification of the structure of the primary info. 1.1.4 PROTOTYPES OF BASIC INFORMATION SYSTEMS The basic types of the overall transformation which an information system performs and of the corresponding basic information systems were discussed previously. Now the concrete forms of the prototypes of these systems are presented.

8

Chapter 1 Basic Functions and Structures of Information Systems

COMMUNICATION AND INFORMATION STORAGE SYSTEMS Figure 1.3a shows the structure of a prototype conununication system. The fundamental information processing subsystem is the communication channel. The preliminary transformation performed by the transmitter includes queuing of irregularly arriving pieces of primary info, coding and modulation. The ultimate transformation is performed by the receiver. Often the primary information ( in communications terminology the "message") is structured, and its structure is simplified before the information is fed into the transmitter or receiver. Such preprocessing is discussed in Sections 1.5.3 and 1.5.4. Sections 2.1 till 2.3 describe models of communication systems and cite several related publications. The optimization of communication systems is discussed in Sections 8.3, 8.5, and 8.6. preliminary transformation

fundamental transformation

ultimate transformation

INFO SOURCE

INFO TDAMQK/TTTPO L COMMUNICATION RECEIVER DESTINATION TRANSMnTERh-rCHANNEL I w\ IX received recovered transmitted message signal signal signal

a)

preliminary transformation DATA SOURCE

fundamental transformation

ultimate transformation

REPORT INFO GENERATION DESTINATION r report retrieved record

MASS RECORD 1 1 FORMATTING ^ ^ STORAGE pnmaiy data

record

SEARCH SUBSYSTEM]

SOURCE OF REQUESTS

b) Figure 1.3. Basic structures of (a) a communication system and (b) a data bank. Information storage systems are another class of very important systems. In these systems the primary information consisting usually of many pieces of elementary information (a file, a database) is stored. After some time, on a request, the system should retrieve a desired single primary piece of information (a record) or it should provide a simplified version of several pieces of information, called report. Such a system is called data bank. The fundamental information processing subsystem is a mass storage device. The structure of the prototype data bank is shown in Figure 1.3b. Another type of information storage system is a hypertext system. Its task is to assemble several pieces of stored information (also called information units) into a larger entity, called composed block. This requires some linking information that may be obtained by a simplifying transformation of the information unit. Section 2.4 describes a simple model of a data bank and cites references.

1.1 Fundamental Concepts

9

SYSTEMS SIMPLIFYING THE STRUCTURE OF INFORMATION The state of the real world, and consequently, primary information about it, has a highly complicated structure. It may be a time process, or a two dimensional or three dimensional static or moving image. In addition, the information often has a complicated macro structure. Therefore, the simplification of the structure of primary information is essential for the transmission, storage, and ultimate transformation of information into decisions made by the superior system. A great variety of systems simplifying the structure of information is used.They operate either as specialized subsystems of the previously described communication and information storage systems or as almost autonomous, universal systems. The systems simplifying the structure of information can be classified into two categories. If the simplification of information is performed so that it is possible to recover exactly or almost exactly the primary information from the simplified information (to decompress the simplified info) the transformation is called information-compression. Special types of information compression are loss-less data compression, discretization (quantization), anddimensionality reduction, inparticular sampling. The prototype compression and decompression systems are shown in Figure 1.4a.

INFO SOURCE

PRELIMINARY TRANSFORMATION

FUNDAMENTAL TRANSFORMATION

ULTIMATE TRANSFORMATION

INFO COMPRESSION

ICOMM. CHANNEL MASS STORAGE

RECOVERY OF PRIMARY INFO

compressed info

structured primary info

retrieved info

INFO DESTINATION

recovered primary info

a) PRELIMINARY TRANSFORMATION INFO SOURCE

FUNDAMENTAL TRANSFORMATION

SAMPLING

DECORELATION w

train (array) of continous samples

contmous function info

DIMENSONALFFY REDUCTION

train (array) of decorelated continous samples

QUANTIZATION OF SINGLE COMPONENTS

train (array) of continous samples with reduced dimensionality

train (array) of quantized samples

b) PRELIMINARY TRANSFORMATION INFO SOURCE

->

FUNDAMENTAL TRANSFORMATION

EXTRACTION OF FEATURES

X

V

structured primary info

features

CLASSIFICATION

INFO DESTINATION

indentifier of class, pattern, prototype

C)

Figure 1.4.Systems simplifying the structure of information: (a). Typical configuration of an information compression and the complementary decompression system, (b). Basic structure of an image compression system, (c) Basic structure of a two level pattern recognition system.

10

Chapter 1 Basic Functions and Structures of Information Systems

Often a complex object or a state can be considered as a modification of a prototype, and for the superior system the modifications are not essential. The second category of systems simplifying information are systems rejecting the non essential details and identifying the prototype. Such a system is called pattern recognition (also classification) system. Thus, contrary to information compression, in the case of pattern recognition it is not required, that the primary information can be exactly recovered from the processed information. Figure 1.4c shows a simplified diagram of a pattern recognition system. Let us discuss in more detail the systems simplifying the structure of information. INFORMATION COMPRESSING SYSTEMS If the structure of the primary information is complicated, the compression is realized in a chain of component-simplifying transformations. A typical example is the compression of an image. The first component compression is sampling. It transforms the image into an array of samples (called pixels),Thost samples are usually interrelated. To simplify the subsequent operations, those interrelations are removed. A typical transformation of this type is decorrelation. The next simplifying transformation is removal of the least significant elements of the set of decorrelated samples. It is called dimensionality reduction.In the last step each retained decorrelated sample is quantized. The structure of the described system is shown in Figure 1.4 b. In this system sampling plays the role of the preliminary transformation, quantization the ultimate. Since it usually requires the most computing power, we may consider the decorrelating subsystem as the fundamental subsystem. If the chain compression is used, then the primary information usually is recovered in corresponding steps but in a reversed sequence. Section 2.6 describes a concrete application of the described procedure for image compression. PATTERN RECOGNITION SYSTEMS The task of a pattern recognition system is to identify the prototype by rejecting irrelevant details of an available information about a modification of the prototype. The recognition (in pattern recognition terminology extraction) of the features plays the role of the preliminary transformation, and the identification of the prototype the role of the fundamental transformation. Often the prototype consists of lower-ranking prototypes; thus, it has a hierarchical structure. For example, a typed character consists of bars and semicircles. The highest-ranking prototype is called pattern or class. The lower-ranking prototypes are cdll&d features. The modification of a structured prototype consists then usually of modifications of lower-ranking prototypes. Then the basic structure of a pattern recognition system is the chain shown in Figure 1.4c. An example of pattern recognition is automatic recognition of an English character presented in a handwritten form. Then the character typed in a standard font plays the role of the prototype, and the number of the character for example the ASCII code is the identifier of die prototype.

1.1 Fundamental Concepts

11

Another example of pattern recognition is the recovery of primary information (message) from a distorted received signal (output of the channel; see Figure 1.3a). For many channels the noiseless signal plays the role of the prototype. The noiseless signal is defined as the hypothetical received signal which would be produced by the transmitted signal carrying a given message, on the assumption that no external distortions occur. The message carried by the noiseless signal plays the role of the identifier of the prototype. Section 2.1 gives concrete examples of such an application of pattern recognition for distorted information recovery. A closer review of transformations performed by irreversible information compression systems and by pattern recognition systems shows that almost all those transformations can be considered as modifications of a basic information transformation called next-neighbor transformation. It is briefly described in Section 1.5.3, several examples are given in Chapters 6 and 7, and its optimal character is discussed in Section 8.3. DECISION MAKING SYSTEMS Often the task of an information system is to transform primary information directly into a decision about actions of the superior system so that the performance of the superior system is possibly good. Usually the structure of a decision is much simpler than of the primary information, but similarly as in the case of pattern recognition we do not require that primary information could be recovered back from a decision. Therefore most decision rules (the rules transforming the primary information into the decision) are information-simplifying transformations. On other hand, information-simplifying transformations can be interpreted as decision rules. For example, the rule of recognizing the pattern may be called the rule of making decisions about the pattern, and the rule according which the receiver in a communication system recovers the message as the rule of taking decisions about the information. The relationships between information theory and decision theory are discussed in Section 1.8.

BIOLOGICAL INFORMATION SYSTEMS Besides human-made information systems there are also biological information systems. Higher living organisms have a hierarchy of such systems. As technical intelligent subsystems, the biological information systems use external and internal state information (see Section 1.1.3). The senses: hearing, vision, feeling, smell, and taste are the biological external information systems. They have usually a chain structure, similar to the prototype structure of technical information systems. The external information subsystems deliver processed information to the brain, which plays for them the role of the superior system. The brain processes all information obtained from the external subsystems, compresses it, and stores it in the memory.

12

Chapter 1 Basic Functions and Structures of Information Systems

The brain combines also all available information about the current state of the environment with information stored in the past and works out decisions for the motoring system of the organism. The structure of brain is a spaced network structure much more complicated than the simple chain structure. Final commands are worked out in the cortex. Thus, we may say that the motoric system of the organism is the superior system for the brain. In view of the enormous complexity of the higher organisms, it is natural that they have developed several subsystems for acquiring the information about the state of organs. These are the biological internal information systems. The pain subsystem provides information about damages or malfunctioning to the brain. Several other internal information subsystems provide information for other then the brain organs. These are, for example, systems providing information needed for control of several metabolic processes. This information is delivered in form of specific biochemical substances. The system of carrying the genetic info by the DNA molecules is extremely complicated. This system may be classified as an information system rendering its services not only to the organism (production of new cells) but also to the species to which the organism belongs (production of new organisms). Studies of biological information systems have a growing impact on the theory of technical information systems. The relationships between those two areas are discussed in Section 1.8, which also cites relevant publications.

1.2 ACQUISITION OF INFORMATION In the next two sections the properties of information are discussed in more detail. The properties of information, in particular its structure, are determined by: 1) the properties of the states the information is about and 2) the rule according to which the information source transforms the state into information. Therefore, in considering the properties of information we start with a brief discussion of the concept of the state. Next, the basic types of information sources are described. 1.2.1 BASIC TYPES OF STATES The environment of a superior system is usually a set of many interacting components that are called objects. The set of features of the environment is called its state. The subset of features directly responsible for interactions between the objects is called external state. The external state consists usually of many components, such as the position of an object in space, its velocity, the temperature of the object, its electrical potential, and the intensity of acoustic, electromagnetic, and gravitational fields. Some components of the external state are related by some general relationships, which are inherent features of the environment. Since they manifest themselves only through the external state, the general relationships are called internal state. The detailed discussion of external and internal states is the subject of Chapter 3. Only some general concepts related to the states, that are needed for the following considerations about information are presented here.

1.2 Acquisition of Information

13

EXTERNAL STATE The structure of the external state reflects the structure of the environment. This causes that often both structures are similar. Also, many concepts having their roots in description of a complicated environment are applicable for description of structured states. Therefore, the structure of a typical environment of a superior system is considered first. A typical object of the environment usually consists of lower-ranking objects and so on. In other words, the environment has a multilevel hierarchical structure. However, starting with a hierarchical level the superior system cannot utilize properties of too small objects either because its acting subsystems are not precise enough to handle them or because the information system cannot provide accurate information about them. Such objects are considered as indivisible/7