Computing with Instinct: Rediscovering Artificial Intelligence

Lecture Notes in Artificial Intelligence Edited by R. Goebel, J. Siekmann, and W. Wahlster Subseries of Lecture Notes i...

Author: Yang Cai

130 downloads 1108 Views 5MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes in Artificial Intelligence Edited by R. Goebel, J. Siekmann, and W. Wahlster

Subseries of Lecture Notes in Computer Science

5897

Yang Cai (Ed.)

Computing with Instinct Rediscovering Artificial Intelligence

13

Series Editors Randy Goebel, University of Alberta, Edmonton, Canada Jörg Siekmann, University of Saarland, Saarbrücken, Germany Wolfgang Wahlster, DFKI and University of Saarland, Saarbrücken, Germany Volume Editor Yang Cai Carnegie Mellon University CYLAB - Instinctive Computing Lab CIC-2218, 4720 Forbes Avenue, Pittsburgh, PA 15213, USA E-mail: [email protected]

ISSN 0302-9743 e-ISSN 1611-3349 ISBN 978-3-642-19756-7 e-ISBN 978-3-642-19757-4 DOI 10.1007/978-3-642-19757-4 Springer Heidelberg Dordrecht London New York Library of Congress Control Number: 2011922330 CR Subject Classification (1998): H.5, I.2, F.1.1, I.6, K.4 LNCS Sublibrary: SL 7 – Artificial Intelligence

© Springer-Verlag Berlin Heidelberg 2011 This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)

Cover Photo. The autonomous vehicle was tested at Carnegie Mellon University, Pittsburgh campus, where the Instinctive Computing Workshop was held on June 9–10, 2009. The vehicle is designed to sense roads and avoid collisions instinctually.

Preface

Simplicity in nature is the ultimate sophistication. Honey bees are not able to play chess or solve the Tower of Hanoi puzzle; however, they do know how to build, defend, forage, navigate, and communicate for survival. They can even learn to recognize human letters independent of size, color, position, or font. Instinct is an inherited behavior that responds to particular environmental stimuli. In his book On the Origin of Species, Darwin pointed out that no complex instinct can possibly be produced through natural selection, except by the slow and gradual accumulation of numerous, slight, yet profitable, variations. Darwin also concluded that no one would dispute that instincts are of the highest importance to every animal species. The world’s magnificence has been enriched by the inner drive of instincts, perhaps the most profound drive of our everyday life. Instinctive Computing is a computational simulation of biological and cognitive instincts, which influence how we see, feel, appear, think, and act. If we want a computer to be genuinely secure, intelligent, and to interact naturally with us, we must give computers the ability to recognize, understand, and even to have primitive instincts. We aim to understand the instinctive common sense of living creatures, including the specialties of individual species as well. Instinctual systems will learn from insects, marine life, animals, and children, to broaden and develop their essential primitive thinking. “Computing with instincts” must be conceived as a meta-program, not a violent attack on traditional artificial intelligence. Instead, this is an aggressive leap for a robust, earthy, and natural intelligence. In the summer of 2009, the first Instinctive Computing Workshop (ICW 2009) was hosted at Carnegie Mellon University, Pittsburgh, USA, jointly sponsored by the National Science Foundation, Cylab, and Google. The two-day workshop aimed to explore the transformational developments in this area, including the building blocks for instinctive computing systems and potential applications in fields such as security, privacy, human–computer interaction, next-generation networks, and product design. The workshop was organized to engage in in-depth dialogue in a small group with multidisciplinary minds, returning to the origin of workshops to focus on ideas. This book, Computing with Instinct, comprises the proceedings of ICW 2009. It is the first state-of-the-art book on this subject. This book consists of three parts: Instinctive Sensing, Communication, and Environments. Part I. Instinctive Sensing. For many years, cyborg pioneer Warwick has explored neural behavior with bi-directional interactions between the brain and implanted devices, which he calls “Implantable Computing.” In this book, Warwick and his colleagues present their new experiments with culturing biological neurons in vitro for the control of mobile robots. Inherent operating characteristics of the cultured neural network have been trained to enable the physical

VIII

Preface

robot body to respond to environmental stimuli such as collisions. The 100,000 biological neurons are grown and trained to act as the brain of an interactive realworld robot – thereby acting as hybrid instinctive computing elements. Studying such a system provides insights into the operation of biological neural structures; therefore, such research has immediate medical implications as well as enormous potential in computing and robotics. This keynote chapter provides an overview of the problem area, gives an idea of the breadth of present ongoing research, details the system architecture and, in particular, reports on the results of experiments with real-life robots. Warwick envisioned this as a new form of artificial intelligence. Sound recognition is an invaluable primitive instinct for mammals. A recent archeological discovery suggested that, for over 120 million years, animals have developed an elaborated auditory system for survival. In the modern era, it is the most affordable diagnostic sensory channel for us, ranging from watermelon selection, car diagnosis to using a medical stethoscope. Cai and Pados explore an auditory vigilance algorithm for detecting background sounds such as explosion, gunshot, screaming, and human voices. They introduce a general algorithm for sound feature extraction, classification, and feedback. It is concluded that the new algorithm reaches a higher accuracy with available training data. This technology has potential in many broader applications of the sound recognition method, including video triage, healthcare, robotics, and security. About half of our brain cells are devoted to visual cognition. A texture provides instinctual cues about the nature of the material, the border, and the distance. The visual perception of texture is key to interpreting object surfaces. In Vernhes and Whitmore’s study, images of textured surfaces of prototype art objects are analyzed in order to identify the methods and the metrics that can accurately characterize slight changes in texture. Three main applications are illustrated: the effect of the conditions of illumination on perceived texture, the characterization of changes of objects due to degradation, and the quantification of the efficiency of the restoration. Part II. Instinctive Communication. Visual abstraction enables us to survive in complex visual environments, augmenting critical features with minimal elements – words. Cai et al. explore the culture and esthetic impacts on visual abstraction. Based on everyday life experience and lab experiments, they found that the factors of culture, attention, purpose, and esthetics help reduce the visual communication workload to a minimum. These studies involve exploration into multi-resolution, symbol-number, semantic differentiation, analogical and cultural emblematization aspects of facial features. To learn a genre is to learn the instinctual and cultural situations that support it. This dominant thinking overlooks critical aspects of genre that appear to be based in deep clusters within natural language lexicons that seem instinctual and cross-cultural. Hu et al. present a theory of lexical clusters associated with critical communication instincts. They then show how these instincts aggregate to support a substrate of conventional English writing genres. To test the crosscultural validity of these clusters, they tested Chinese students in rural China

Preface

IX

with limited training in native English writing and limited exposure to native English cultural situations. Non-verbal communication such as gestures and facial expressions is a major part of fundamental interaction among people. Sonntag views intuition as instinctive dialog. To allow for an intuitive communication, multimodal taskbased dialog must be employed. A concrete environment, where an intuition model extends a sensory-based modeling of instincts, can be used to assess the significance of intuition in multimodal dialog. Part III. Instinctive Environments. Rapidly growing virtual world technologies permit collaboration in a distributed, virtual environment. In a real-world environment, distributed teams collaborate via face-to-face communication using social interactions, such as eye contact and gestures, which provide critical information and feedback to the human decision maker. The virtual environment presents unique challenges in this regard. Yvonne and Aguiar focus on how we evaluate human performance and various levels of expertise, strategies, and cognitive processes of decision makers within the virtual environment. Their exploitations include accurate and time-critical information flow, cognitive workload, and situational awareness among team members. We are not living in the forest anymore. Modern living environments enable us to maximize comfort zones; however, they also introduce new problems associated with those artifacts. Garcia et al. study how to enable end-users to manage their preferences in personal environments. The system uses rules and modularizing agents, paying special attention to end-user programming issues and the natural hierarchies present in the environment. Furthermore, O’Grady et al. propose an intelligent middleware framework as a means for harnessing the disparate data sources necessary for capturing and interpreting implicit interaction events. The manifesto for ubiquitous computing was released in early 1990. Ten years later, ambient intelligence was envisioned. Today, how to implement networked intelligent artifacts remains an open issue. Human–computer interaction tries to combine psychology, computing, and design into a science. However, prevailing usability-centric studies have had little impact in real-world products or interactions. We need new genes, new dimensions, and new approaches. The goal of this book is to rethink the origin of human interactions, to define instinctual components, and to demonstrate the potential of such a new computing paradigm. We believe that “computing with instinct” is a solution for fundamental problems in ambient intelligence, such as situation awareness, understanding, learning, and simplicity. On behalf of the workshop committee and editing crew, I would like to thank all of the authors for their support for the book. Many thanks to Sylvia Spengler of the National Science Foundation, Pradeep Khosla, Adrian Perrig, Virgil Gligor, Howard Lipson, Richard Noland, Kristopher Rush, Willian Eddy, David Kaufer, Mel Siegel, and Richard Stafford of Carnegie Mellon University for their support. The Instinctive Computing Workshop was generously supported by the National Science Foundation, Google, and Cylab of Carnegie Mellon University.

X

Preface

The related projects have been in part sponsored by the US Army Research Office, Center for Emergency Response Team (CERT), and Air Force Research Lab in Rome, NY. However, the concepts in this book do not necessarily reflect the policies or opinions of any governmental agencies. Yang Cai

Organization

Organizers Yang Cai Sylvia Spengler Howard Lipson

Carnegie Mellon University National Science Foundation, USA CERT, Carnegie Mellon University

Program Committee Julio Abascal Xavier Alaman Jose Bravo Andrew Cowell David Farber Virgil Gligor Fabian Hemmert Michael Leyton Xiaoming Liu Yvonne Masakowski Adrian Perrig Mel Siegel Brenda Wiederhold Mark Wiederhold Brian Zeleznik

University of the Basque Country, Spain Autonomous University of Madrid, Spain Universidad de Castilla-La Mancha, Spain Pacific Northwestern National Laboratory, USA Carnegie Mellon University, USA Carnegie Mellon University, USA Deutsche Telekom Labs, Germany Rutgers University, USA GE Research Center, USA NAVY, USA Carnegie Mellon University, USA Carnegie Mellon University, USA Interactive Media Institute, Belgium Virtual Reality Medical Center, USA Carnegie Mellon University, USA

Editor Yang Cai

Editing and Design Assistant Emily Durbin

Coordinator Samantha Stevick

Table of Contents

Part I: Instinctive Sensing Experiments with an In-Vitro Robot Brain . . . . . . . . . . . . . . . . . . . . . . . . . . Kevin Warwick, Slawomir J. Nasuto, Victor M. Becerra, and Benjamin J. Whalley

1

Sound Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yang Cai and K´ aroly D. Pados

16

Texture Vision: A View from Art Conservation . . . . . . . . . . . . . . . . . . . . . . Pierre Vernhes and Paul Whitmore

35

Part II: Instinctive Communication Visual Abstraction with Culture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yang Cai, David Kaufer, Emily Hart, and Yongmei Hu

47

Genre and Instinct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yongmei Hu, David Kaufer, and Suguru Ishizaki

58

Intuition as Instinctive Dialogue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Daniel Sonntag

82

Part III: Instinctive Environments Human Performance in Virtual Environments . . . . . . . . . . . . . . . . . . . . . . . Yvonne R. Masakowski and Steven K. Aguiar

107

Exploitational Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Manuel Garc´ıa–Herranz, Xavier Alam´ an, and Pablo A. Haya

119

A Middleware for Implicit Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M.J. O’Grady, J. Ye, G.M.P. O’Hare, S. Dobson, R. Tynan, R. Collier, and C. Muldoon

143

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

163

Experiments with an In-Vitro Robot Brain Kevin Warwick1, Slawomir J. Nasuto1, Victor M. Becerra1, and Benjamin J. Whalley2 1

School of Systems Engineering, School of Chemistry, Food Biosciences and Pharmacy, University of Reading, UK {K.Warwick,S.J.Nasuto,V.M.Becerra,B.J.Whalley}@reading.ac.uk 2

Abstract. The controlling mechanism of a typical mobile robot is usually a computer system either remotely positioned or in-body. Recent research is on-going in which biological neurons are grown and trained to act as the brain of an interactive real-world robot – thereby acting as instinctive computing elements. Studying such a system provides insights into the operation of biological neural structures; therefore, such research has immediate medical implications as well as enormous potential in computing and robotics. A system involving closed-loop control of a mobile robot by a culture of neurons has been created. This article provides an overview of the problem area, gives an idea of the breadth of present ongoing research, details our own system architecture and, in particular, reports on the results of experiments with real-life robots. The authors see this as a new form of artificial intelligence.

1 Introduction In the last few years, considerable progress has been made towards hybrid systems in which biological neurons are integrated with electronic components. As an example, Reger [1] demonstrated the use of a lamprey brain to control a small wheeled robots movements; meanwhile, others were successfully able to send control commands to the nervous system of cockroaches [2] or rats [3] as if they were robots. These studies can inform us about information processing and encoding in the brains of living animals [4]. However, they do pose ethical questions and can be technically problematic since access to the brain is limited by barriers such as the skin and skull, and data interpretation is complicated due to the sheer number of neurons present in the brain of even the simplest animal. Coupled with this, approaches which involve recording the activity of individual neurons or small populations of neurons are limited by their invasive, and hence destructive, nature. As a result, neurons cultured under laboratory conditions on a planar array of non-invasive electrodes provide an attractive alternative with which to probe the operation of biological neuronal networks. Understanding neural behaviour is certainly extremely important in establishing better bi-directional interactions between the brain and external devices. On top of this, for neurological disorders, establishing improved knowledge about the fundamental basis of the inherent neuronal activity is critical. A robot body can potentially move around a defined area and the effects within a biological brain, which is Y. Cai (Ed.): Computing with Instinct, LNAI 5897, pp. 1–15, 2011. © Springer-Verlag Berlin Heidelberg 2011

2

K. Warwick et al.

controlling the body, can be witnessed. This opens up the possibility of gaining a fundamental appreciation and understanding of the cellular correlates of memory and resultant actions based on learning and habit. Research has recently been focused on culturing networks of some tens of thousands of brain cells grown in vitro [5]. These cultures are created by enzymatically dissociating neurons obtained from foetal rodent cortical tissue and then culturing them in a specialised chamber, in doing so providing suitable environmental conditions and nutrients. An array of electrodes is embedded in the base of the chamber (a Multi Electrode Array; MEA), providing an electrical interface to the neuronal culture [6-9]. The neurons in such cultures begin to spontaneously branch out and, within an hour of placement, even without external stimulation, they begin to re-connect with other nearby neurons and commence electrochemical communication. This propensity to spontaneously connect and communicate demonstrates an innate tendency to network. Studies of neural cultures demonstrate distinct periods of development defined by changes in activity which appear to stabilise after 30 days and, in terms of useful responses, last for at least 2-3 months [10, 11]. The cultures of neurons form a monolayer on the MEA, making them both amenable to optical microscopy and accessible to physical and chemical manipulation [9]. The specific aim of the ongoing project described here is to investigate the use of cultured neurons for the control of mobile robots. However, in order to produce useful processing, we postulate that disembodied biological networks must develop in the presence of meaningful input/output relationships as part of closed loop sensory interaction with the environment. This is evidenced by animal and human studies which show that development in a sensory-deprived environment results in poor or dysfunctional neural circuitry [13, 14]. To this end, the overall closed loop hybrid system involving a primary cortical culture on an MEA and a mobile robot body must exist within a sufficiently rich and reasonably consistent environment. This then constitutes an interesting and novel approach to examining the computational capabilities of biological networks [15]. Typically, in vitro neuronal cultures consist of thousands of neurons generating highly variable, multi-dimensional signals. In order to extract components and features representative of the network’s overall state from such data, appropriate preprocessing and dimensionality reduction techniques must be applied. Several schemes have till now been constructed. Shkolnik created a control scheme for a simulated robot body [16] in which two channels of an MEA were selected and an electrical stimulus consisting of a +/-600 mV, 400 μsecs biphasic pulse was delivered at varying inter-stimulus intervals. Information coding was formed by testing the effect of electrically-induced neuronal excitation with a given time delay termed the InterProbe Interval (IPI) between two stimulus probes. This technique gave rise to a characteristic response curve which formed the basis for deciding the robot’s direction of movement using basic commands (forward, backward, left and right). In one experiment with a simulated rat [32] as the embodiment, this moving inside a four-wall environment included barrier objects. Meanwhile, physical robots were used in an experiment [16] wherein one of the robots was able to maintain a constant distance from a second robot, which was moving under pseudo-random control. It was reported that the first robot managed to successfully approach the second and maintain a fixed distance from it. Information on the spontaneous activity of the

Experiments with an In-Vitro Robot Brain

3

culture was sent to a computer which then made the binary decisions as to what action the robot should take. The culture itself was not directly controlling the Koala through a feedback loop and no learning effect was reportedly exploited. In contrast with these experiments, both closed-loop control and learning are central aims in our own study. DeMarse and Dockendorf investigated the computational capacity of cultured networks by implementing the control of a “real-life” problem, namely controlling a simulated aircrafts flight path (e.g. altitude and roll adjustments) [17]. Meanwhile, Shahaf and Marom [18] reported one of the first experiments to achieve desired discrete output computations by applying a simple form of supervised learning to disembodied neuronal cultures. Recently, Bull & Uroukov [19] applied a Learning Classifier System to manipulate culture activity towards a goal level using simple input signals. In both of these latter experiments, the desired result was only achieved in about one third of the cases, indicating some of the difficulties in achieving repeatability. But this is a field of study very much in its infancy. There are bound to be difficulties; however, there is much to be learnt. It is apparent that, even at such an early stage, such re-embodiments (real or virtual) have an important role to play in the study of biological learning mechanisms and neurological behaviour in general. Our physical embodied robots provide the starting point for creating a proof-of-concept control loop around the neuronal culture and a basic platform for future – more specific – reinforcement learning experiments. The fundamental problem is the coupling of the robot’s goals to the culture’s input-output mapping. The design of the robot’s architecture discussed in this paper therefore emphasises the need for flexibility and the use of machine learning techniques in the search of such coupling. In the section which follows, the general procedure for laying out the neural culture (the biological component) is described. This is followed by a description of the main elements of the closed loop control system, including the culture as an important element in the feedback loop. Details of the current systems architecture are given in section 3. Section 4 includes a description of our initial tests and preliminary results. Section 5 meanwhile provides an explanation of the Machine Learning (ML) context, and Section 6 concludes with an overview of current progress. Finally, Section 7 discusses new, ongoing research and planned future extensions.

2 Culture Preparation To realise the cultured neural network, cortical tissue is dissected from the brains of embryonic rats and neuronal cells enzymatically dissociated before seeding onto planar Multi Electrode Arrays (MEAs). The cells are restricted to lie within the recording horizon of the electrode array by means of a template placed on the MEA prior to seeding and removed immediately after cells have settled (~ 1 hour). The MEA is also filled with a conventional cell culture medium containing nutrients, growth hormones and antibiotics, of which 50% is replaced twice weekly. Within the first hour after seeding, neurons appear to extend connections to nearby cells (even within the first few minutes this has been observed) and, within 24 hours, a thick mat of neuronal extensions is visible across the seeded area.

4

K. Warwick et al.

The connectivity between seeded cells increases rapidly over subsequent days. After 7 days, electrical signals are observed in the form of action potentials which, in the ‘disembodied culture’ (not connected within the closed loop), transform into dense bursts of simultaneous electrical activity across the entire network over the following week. This bursting feature subsequently continues through to maturity (30 days in vitro and onwards). It is not well understood what the bursting actually means and how much it is part of normal neural development. However, such continued behavior, after this initial development phase, may subsequently be representative of an underlying pathological state resulting from impoverished sensory input and may differ from the activity of a culture developing within a closed loop [20]. This is something which remains to be studied further. Cultures usually remain active until approximately 3 months of age. During this time, they are sealed with Potter rings [21] to maintain sterility and osmolarity and are maintained in a humidified, 37oC, 5% CO2 incubator. Recordings are undertaken in a non-humidified 37oC, 5% CO2 incubator for between 30 minutes and 8 hours dependent on environmental humidity and the resulting stability of activity.

3 Experimental Arrangements The multi-electrode array enables voltage fluctuations in the culture (relative to a reference ground electrode outside the network) to be recorded in real-time at 59 sites out of 64 in an ‘8x8’ array (Figure 1). This allows for the detection of neuronal action potentials within a 100 µm radius (or more) around an individual electrode. By using spike-sorting algorithms [12], it is then possible to separate the firings of multiple individual neurons, or small groups of neurons, as monitored on a single electrode. As a result, multi-electrode recordings across the culture permit a picture of the global activity of the entire neuronal network to be formed. It is possible to electrically stimulate via any of the electrodes to induce focused neural activity. The multielectrode array therefore forms a functional and non-destructive bi-directional interface to the cultured neurons. Electrically-evoked responses and spontaneous activity in the culture (the neuronal network) are coupled to the robot architecture, and hence on to the physical robot via a machine-learning interface, which maps the features of interest to specific actuator commands. Sensory data fed back from the robot is associated with a set of appropriate stimulation protocols and is subsequently delivered to the culture, thereby closing the robot-culture loop. Thus, signal processing can be broken down into two discrete sections: (a) ‘culture to robot’, in which an output machine learning procedure processes live neuronal activity, and (b) ‘robot to culture’, which involves an input mapping process, from robot sensor to stimulus. It is important to realise that the overall system employed in this experiment has been designed based on a closed-loop, modular architecture. As neuronal networks exhibit spatiotemporal patterns with millisecond precision [22], processing of these signals necessitates a very rapid response from neurophysiological recording and robot control systems. The software developed for this project runs on Linux-based workstations that communicate over the Ethernet via fast server-client modules, thus providing the necessary speed and flexibility required.


5

In recent years, the study of neuronal cultures has been greatly facilitated by commercially available planar MEA systems. These consist of a glass specimen chamber lined with an 8x8 array of electrodes, as shown in Figure 1. It is just such one of these MEAs that we have employed in our overall robot system.

Fig. 1. a) An MC200/30iR-gr MEA (NMI, Reutlingen, Germany, UK), showing the 30µm electrodes which lead to the electrode column–row arrangement b) Electrode arrays in the centre of the MEA seen under an optical microscope (Nikon TMS, Nikon, Japan), x4 magnification c) An MEA at x40 magnification, showing neuronal cells in close proximity to an electrode, with visible extensions and inter-connections

A standard MEA (Figure 1a) measures 49 mm x 49 mm x 1 mm and its electrodes provide a bidirectional link between the culture and the rest of the system. The associated data acquisition hardware includes a head-stage (MEA connecting interface), 60 channel amplifier (1200x gain; 10-3200Hz bandpass filter), stimulus generator and PC data acquisition card. To this point, we have successfully created a modular closed-loop system between a (physical) mobile robotic platform and a cultured neuronal network using a MultiElectrode Array, allowing for bidirectional communication between the culture and the robot. It is estimated that the cultures employed in our studies consist of approximately (on average) 100,000 neurons. The actual number in any one specific culture depends on natural density variations in proliferation post-seeding and experimental aim. The spontaneous electrochemical activity of the culture realising signals at certain of the electrodesis used as input to the robots actuators and the robots (ultrasonic) sensor readings are (proportionally) converted into stimulation signals received by the culture, effectively closing the loop. We are using a versatile, commercially available, Miabot robot (Figure 2) as our physical platform. This exhibits accurate motor encoder precision (~0.5 mm) and has a maximum speed of approximately 3.5 m/s. Hence it can move around quite quickly in real time. Recording and stimulation hardware is controlled via open-source

6

K. Warwick et al.

MEABench software [23]. However, we have also developed our own custom stimulator control software, which interfaces with the commercially available stimulation hardware with no need for hardware modification [23]. The overall closed-loop system therefore consists of several modules, including the Miabot robot, an MEA and stimulating hardware, a directly linked workstation for conducting computationally expensive neuronal data analyses, a separate machine running the robot control interface, and a network manager routing signals directly between the culture and the robot body. The various components of the architecture communicate via TCP/IP sockets, allowing for the distribution of processing loads to multiple machines throughout the University of Reading’s internal network. The modular approach to the problem is shown in more detail in Figure 3. The Miabot is wirelessly controlled via Bluetooth. Communication and control are performed through custom C++ server code and TCP/IP sockets and clients running on the acquisition PC which has direct control of the MEA recording and stimulating software. The server sends motor commands and receives sensory data via a virtual serial port over the Bluetooth connection, while the client programs contain the closed loop code which communicates with and stimulates the MEA culture. The client code also performs text logging of all important data during an experiment run.

Fig. 2. The Miabot robot with a cultured neural network


7

Fig. 3. Modular layout of the robot/MEA system

This modular approach to the architecture has resulted in a system with easily reconfigurable components. The obtained closed-loop system can efficiently handle the information-rich data that is streamed via the recording software. A typical sampling frequency of 25 kHz of the culture activity demands large network, processing and storage resources. Consequently, on-the-fly streaming of spike-detected data is the preferred method when investigating real-time closed-loop learning techniques.

4 Experimental Results Firstly, an existing appropriate neuronal pathway was identified by searching for strong input/output relationships between pairs of electrodes. Suitable input/output pairs were defined as those electrode combinations in which neurons proximal to one electrode responded to stimulation of the other electrode at which the stimulus was applied (at least one action potential within 100 ms of stimulation) more than 60% of the time and responded no more than 20% of the time to stimulation on any other electrode. An input-output response map was then created by cycling through all preselected electrodes individually with a positive-first biphasic stimulating waveform (600 mV; 100 µs each phase, repeated 16 times). By averaging over 16 attempts, it was ensured that the majority of stimulation events fell outside any inherent culture bursting that might have occurred. In this way, a suitable input/output pair could be chosen, dependent on how the cultures had developed, in order to provide an initial decision-making pathway for the robot.

8

K. Warwick et al.

To be clear about this initialisation process: In the initially developed culture, we found, by experimentation, a reasonably repeatable pathway in the culture from stimulation to response. We then employed this to control the robot body as we saw fit – for example, if the ultrasonic sensor was active, then we wished the response to cause the robot to turn away from the ultrasonically-located object being located ultrasonically in order to keep moving without bumping into anything. In the set-up, the robot followed a forward path within its corral confines until it reached a wall, at which point the front sonar value decreased below a threshold (set at approximately 30 cm), triggering a stimulating pulse as shown in Figure 4. If the responding/output electrode registered activity following the input pulse, then the robot turned to avoid the wall. Essentially, activity on the responding electrode was interpreted as a command for the robot to turn in order to avoid the wall. It was apparent that, in fact, the robot turned spontaneously whenever activity was registered on the response/output electrode. The most relevant result for the experiment was the occurrence of the chain of events: wall detection–stimulation–response. From a philosophical and neurological perspective, it is of course also of interest to speculate why there was activity on the response electrode when no stimulating pulse had been applied. The typical behaviour in the cultures studied was generally a period of inactivity (or low-frequency activity) prior to stimulus, followed by heightened network activity induced almost immediately (within few miliseconds) after stimulus, which decayed (typically after ~100 ms) to baseline pre-stimulus activity. The study opens up the possibility of investigating response times of different cultures under different conditions and how they might be affected by external influences such as electrical fields and pharmacological stimulants [24]. At any one time, we typically have 25 different cultures available, hence such comparative developmental studies are now being conducted. With the sonar threshold set at approx. 30 cm from a wall, a stimulation pulse was applied to the culture, via its sensory input, each time this threshold was breached – effectively, when the robots position was sufficiently close to a wall. An indication of the robots typical activity during a simple wall-detection/right-turn experiment is shown in Figure 4. The green trace indicates the front sonar value. Yellow bars indicate stimulus pulse times and blue/red bars indicate sonar timing/actuator command timing. As can be witnessed, these response events (single detected spike) may occur purely spontaneously or due to electric stimulation as a result of the sensor threshold being breached. Such events are deemed ‘meaningful’ only in the cases when the delay between stimulation and response is less than 100 ms. In other words, such an event is a strong indicator that the electric stimulation on one electrode caused a neural response on the recording electrode. The red vertical lines indicate the time that a rotation command is sent to the robot. These events are always coupled (the first one starts the right-turn rotation and the second simply ends the rotation). Only the second signals of each pair can be clearly seen here, as the rotation initiation commands are overlaid by the yellow electrode firing bars (a result of electrode firing which instantly initiates a rotation command). A ‘meaningful’ event chain would be, for example, at 1.95 s, where the sonar value drops below the threshold value (30cm) and a stimulation-response subsequently occurs.


9

90 80 70 60 50 40 30

Fig. 4. Analysis of the robots activity during a simple wall-detection/right turn experiment

Table 1 contains typical results from a live culture test in comparison with a “perfect” simulation. If the live culture acted “perfectly,” making no mistakes, then the two columns would be identical. Of course, this raises the question as to what a “perfect” response actually is. In this case, it could be regarded as a programmed exercise – which some might refer to as “machine-like.” In a sense, therefore, the culture is asserting its own individuality by not being “perfect.” To explain Table 1 further, ‘total closed loop time’ refers to the time between wall detection and a response signal witnessed from the culture. ‘Meaningful turns’ refers to the robot turning due to a ‘wall detection-stimulation-response’ chain of events. A ‘wall to stimulation’ event corresponds to the 30 cm threshold being breached on the sensor such that a stimulating pulse is transmitted to the culture. Meanwhile, a ‘stimulation to response’ event corresponds to a motor command signal, originating in the culture and being transmitted to the wheels of the robot to cause it to change direction. It follows that, for the culture, some of the ‘stimulation to response’ events will be in ‘considered’ response to a recent stimulus – termed meaningful. In contrast, other such events – termed spontaneous – will be either spurious or in ‘considered’ response to some thought in the culture about which we are unaware. Table 1. Basic statistics from a wall avoidance experiment

Results Wall -> Stimulation event Stimulation -> Response event Total closed loop time Run time Meaningful turns Spontaneous turns

Simulation

Live Culture

100% 100% 0.075 s

100% 67% 0.2 - 0.5 s

240 s 41 41

140 s 22 16

10

K. Warwick et al.

By totalling the results of a series of such trials carried out (over 100), considerable differences (as typically indicated in Table 1) are observed between the ratio of expected and spontaneous turns between the simulation and the live culture. Under the control of the simulation 95 ± 4% (Mean ± SD) meaningful turns were observed whilst the remaining spontaneous turns (5 ± 4%) were easily attributable to aspects of thresholding spike activity. In contrast, the live culture displayed a relatively low number of meaningful turns (46 ± 15%) and a large number of spontaneous turns 54 ± 19% as a result of intrinsic neuronal activity. Such a large number of spontaneous turns was perhaps only to be expected in an uncharacterised system; current work aims to both quiet the level of ongoing spontaneous activity, reminiscent of epileptiform, present in such cultures and to discover more appropriate input sites and stimulation patterns. As a follow-up closed-loop experiment, the robots individual (right and left separately) wheel speeds were controlled by using the spike-firing frequency recorded from the two chosen motor/output electrodes. The frequency is actually calculated by means of the following simple principle: A running mean of spike rate from both the output electrodes was computed from the spike detector. The detected spikes for each electrode were separated and divided by the signal acquisition time to give a frequency value. These frequencies were linearly mapped (from their typical range of 0-100 Hz) to a range of 0-0.2 m/s for the individual wheel linear velocities. Meanwhile, collected sonar information was used to directly control (proportionally) the stimulating frequency of the two sensory/input electrodes. The typical sonar range of 0-100 cm was linearly re-scaled into the range 0.2-0.4 Hz for electrode stimulation frequencies (600 mV voltage pulses). The overall setup can be likened to a simple Braitenberg model [25]. However, in our case, sensor-to-speed control is mediated by the cultured network acting as the sole decision-making entity within the overall feedback loop. One important aspect being focused on is the evocation of Long Term Potentiation (LTP), i.e. directed neural pathway changes in the culture, thereby effecting plasticity between the stimulating-recording electrodes. Although this was not a major initial target in carrying out this part of the experiment, it has been noted elsewhere that a high frequency burst time can induce plasticity very quickly [27], [28]. As a result, we are now investigating spike-timing-dependent plasticity based on the coincidence of spike and stimulus.

5 Learning Inherent operating characteristics of the cultured neural network have been taken as a starting point to enable the physical robot body to respond in an appropriate fashion – to get it started. The culture then operates over a period of time within the robot body in its corral area. Experimental duration, e.g. how long the culture is operational within its robot body, is merely down to experimental design. Several experiments can therefore be completed within a day, whether on the same or differing cultures. The physical robot body can, of course, operate 24/7. In our studies thus far, learning and memory investigations are at an early stage. However, we were able to observe that the robot appeared to improve its performance over time in terms of its wall avoidance ability. We are currently investigating this


11

and examining whether it can be repeated robustly and subsequently quantified. What we have witnessed could mean that neuronal structures/pathways that bring about a satisfactory action tend to strengthen purely though a process being habitually performed – learning due to habit. Such plasticity has been reported on elsewhere, e.g. [29], and experimentation has been carried out to investigate the effects of sensory deprivation on subsequent culture development. In our case we are monitoring changes and attempting to provide a quantitative characterisation relating plasticity to experience and time. The potential number of confounding variables, however, is considerable, as the subsequent plasticity process, which occurs over quite a period of time, is (most likely) dependent on such factors as initial seeding and growth near electrodes as well as environmental transients such as feed rate, temperature and humidity. On completion of these first phases of the infrastructure setup, a significant research contribution, it is felt, lies in the application of Machine Learning (ML) techniques to the hybrid system’s closed loop experiments. These techniques may be applied in the spike-sorting process (dimensionality reduction of spike data profiles, clustering of neuronal units); the mapping process between sensory data and culture stimulation, as well as the mapping between the culture activity and motor commands; and the application of learning techniques on the controlled electrical stimulation of the culture, in an attempt to exploit the cultured networks’ computational capacity.

6 Conclusions We have successfully realised a closed-loop adaptive feedback system involving a (physical) mobile robotic platform and a cultured neuronal network using a MultiElectrode Array (MEA), which necessitates real-time bidirectional communication between the culture and the robot. A culture being employed consists of approximately 100,000 neurons, although at any one time only a small proportion of these neurons are actively firing. Trial runs have been carried out with the overall robot and comparisons have been made with an “ideal” simulation which responds to stimuli perfectly as required. It has been observed that the culture on many occasions responds as expected; however, on other occasions it does not, and in some cases it provides a motor signal when it is not expected to do so. The concept of an ‘ideal’ response is difficult to address here because a biological network is involved, and it should not be seen in negative terms when the culture does not achieve such an ideal. We know very little about the fundamental neuronal processes that give rise to meaningful behaviours, particularly where learning is involved; we therefore need to retain an open mind as to a culture’s performance. The culture preparation techniques employed are constantly being refined and have lead to stable cultures that exhibit both spontaneous and induced spiking/bursting activity which develops in line with the findings of other groups, e.g. [15] and [21]. A stable robotic infrastructure has been set up, tested, and is in place for future culture behaviour and learning experiments. This infrastructure could be easily modified in order to investigate culture-mediated control of a wide array of alternative robotic

12

K. Warwick et al.

devices, such as a robot head, an ‘autonomous’ vehicle, robotic arms/grippers, mobile robot swarms and multi-legged walkers. In terms of robotics, this study and others like it, show that a robot can have a biological brain to make its ‘decisions’. The 100,000 neuron size is due to present day limitations – clearly this will increase. Indeed, it is already the case that 3-dimensional structures are being investigated [19]. Simply increasing the complexity from 2 dimensions to 3 dimensions (on the same basis) realises a figure of 30 million neurons (approx.) for the 3-dimensional case. The whole area of research is therefore a rapidly expanding one as the range of sensory inputs is expanded and the number of cultured neurons encapsulated rises. The potential capabilities of such robots, including the range of tasks they can perform, therefore needs to be investigated. Understanding neural activity becomes a much more difficult problem as the culture size is increased. Even the present 100,000 neuron cultures are far too complex at the moment for us to gain an overall insight. When they are grown to sizes such as 30 million neurons and beyond, clearly the problem is significantly magnified, particularly with regard to neural activity in the centre of a culture volume, which will be (effectively) hidden from view. On top of this, the nature of the neurons may be diversified. At present, rat neurons are employed in our studies. Potentially, however, any animal neurons could be used; even human neurons are not out of the question from a technical viewpoint. The authors wish to record our feelings here that it is important to stress the need for ethical concerns to be paramount in such circumstances.

7 Future Research There are a number of ways in which the current research programme is being taken forward. Firstly, the Miabot is being extended to include additional sensory devices such as extra sonar arrays, audio input, mobile cameras and other range-finding hardware, such as an on-board infrared sensor. This will provide an opportunity to investigate sensory fusion in the culture and perform more complex behavioural experiments, possibly even attempting to demonstrate links between behaviour and culture plasticity, along the lines of [29], as different sensory inputs are integrated. Provision of a powered floor for the robots corral will provide the robot with relative autonomy for a longer period of time while different learning techniques are applied and behavioural responses monitored. For this, the Miabot must be adapted to operate on an in-house powered floor, providing the robot with an unlimited power supply. This feature, which is based on an original design for displays in museums [30], is necessary since learning and culture behaviour tests will be carried out for hours at a time. Current hardcoded mapping between the robot goals and the culture input/output relationships can be extended by using learning techniques to eliminate the need for an a priori choice of the mapping. In particular, Reinforcement Learning techniques can be applied to various mobile robot tasks, such as wall following and maze navigation, in an attempt to provide a formal framework within which the learning capabilities of the neuronal culture will be studied. To increase the effectiveness of culture training beyond the ~30% success rate seen in previous work, biological experiments are currently being performed to identify


13

physiological features which may play a role in cellular correlates of learning processes. These experiments also investigate possible methods of inducing an appropriate receptive state in the culture that may allow greater control over its processing abilities and the formation of memories [26] involving specific network activity changes which may allow identification of the function of given network ensembles. In particular, in terms of cholinergic influences, the possible effect of acetylcholine (ACh) [33] in coordinating the contributions of different memory systems is being investigated. A further area of research is to identify the most suitable stage of development at which to place cultures within the closed loop and whether a less pathological (epileptiform), therefore more effectively manipulated, state of activity is achieved when cultures are allowed to undergo initial development in the presence of sensory input. The learning techniques employed and the results obtained from the culture need to be benchmarked. In order to achieve this, we are developing a model of the cultured neural network based on experimental data about culture density and activity. In doing so, we hope to gain a better understanding of the contribution of culture plasticity and learning capacity to the observed control proficiency. Presently, we are investigating Hidden Markov Models (HMMs) as a technique for uncovering dynamic spatiotemporal patterns emerging from spontaneously active or stimulated neuronal cultures. The use of Hidden Markov Models enables characterisation of multi-channel spike trains as a progression of patterns of underlying discrete states of neuronal activity. Acknowledgements. This work is funded by the UK Engineering and Physical Sciences Research Council (EPSRC) under grant No. EP/D080134/1, with further financial support being provided by the Royal Society. The team wishes to thank the Science Museum (London), and in particular Louis Buckley, for their housed display explicitly on this work from October 2008 onwards. We also wish to thank New Scientist for their general popular coverage of our robot system in operation [31]. Finally, we wish to extend our gratitude to other members of the University of Reading team, namely Mark Hammond, Simon Marshall, Dimi Xydas, Julia Downes and Matthew Spencer.

References 1. Reger, B., Fleming, K., Sanguineti, V., Simon Alford, S., Mussa-Ivaldi, F.: Connecting brains to robots: An artificial body for studying the computational properties of neural tissues. Artificial Life 6, 307–324 (2000) 2. Holzer, R., Shimoyama, I., Miura, H.: Locomotion control of a bio-robotic system via electric stimulation. In: Proceedings of International Conference on Intelligent Robots and Systems, Grenoble, France (1997) 3. Talwar, S., Xu, S., Hawley, E., Weiss, S., Moxon, K., Chapin, J.: Rat navigation guided by remote control. Nature 417, 37–38 (2002) 4. Chapin, J., Moxon, K., Markowitz, R., Nicolelis, M.: Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nature Neuroscience 2, 664–670 (1999)

14

K. Warwick et al.

5. Bakkum, D.J., Shkolnik, A., Ben-Ary, G., DeMarse, T., Potter, S.: Removing Some ‘A’ from AI: Embodied Cultured Networks. Lecture Notes in Computer Science, pp. 130–145 (2004) 6. Thomas, C., Springer, P., Loeb, G., Berwald-Netter, Y., Okun, L.: A miniature microelectrode array to monitor the bioelectric activity of cultured cells. Exp. Cell Res. 74, 61–66 (1972) 7. Gross, G.: Simultaneous single unit recording in vitro with a photoetched laser deinsulated gold multimicroelectrode surface. IEEE Transactions on Biomedical Engineering 26, 273– 279 (1979) 8. Pine, J.: Recording action potentials from cultured neurons with extracellular microcircuit electrodes. Journal of Neuroscience Methods 2, 19–31 (1980) 9. Potter, S., Lukina, N., Longmuir, K., Wu, Y.: Multi-site two-photon imaging of neurons on multi-electrode arrays. In: SPIE Proceedings, vol. 4262, pp. 104–110 (2001) 10. Gross, G., Rhoades, B., Kowalski, J.: Dynamics of burst patterns generated by monolayer networks in culture. In: Neurobionics: An Interdisciplinary Approach to Substitute Impaired Functions of the Human Nervous System 1993, pp. 89–121 (1993) 11. Kamioka, H., Maeda, E., Jimbo, Y., Robinson, H., Kawana, A.: Spontaneous periodic synchronized bursting during the formation of mature patterns of connections in cortical neurons. Neuroscience Letters 206, 109–112 (1996) 12. Lewicki, M.: A review of methods for spike sorting: the detection and classification of neural action potentials. Network (Bristol) 9(4), R53 (1998) 13. Saito, S., Kobayashik, S., Ohashio, Y., Igarashi, M., Komiya, Y., Ando, S.: Decreased synaptic density in aged brains and its prevention by rearing under enriched environment as revealed by synaptophysin contents. Journal Neuroscience Research 39, 57–62 (1994) 14. Ramakers, G.J., Corner, M.A., Habets, A.M.: Development in the absence of spontaneous bioelectric activity results in increased stereotyped burst firing in cultures of dissociated cerebral cortex. Exp Brain Res 79, 157–166 (1990) 15. Chiappalone, M., Vato, A., Berdondini, L., Koudelka-Hep, M., Martinoia, S.: Network Dynamics and Synchronous Activity in cultured Cortical Neurons. International Journal of Neural Systems 17(2), 87–103 (2007) 16. Shkolnik, A.C.: Neurally controlled simulated robot: applying cultured neurons to handle an approach / avoidance task in real time, and a framework for studying learning in vitro, in Mathematics and Computer Science. Masters Thesis, Dept. of Computer Science, Emory University: Georgia (2003) 17. DeMarse, T.B., Dockendorf, K.P.: Adaptive flight control with living neuronal networks on microelectrode arrays. In: Proceedings of IEEE International Joint Conference on Neural Networks, IJCNN 2005, pp. 1549–1551 (2005) 18. Shahaf, G., Marom, S.: Learning in networks of cortical neurons. Journal Neuroscience 21(22), 8782–8788 (2001) 19. Bull, L., Uruokov, I.: Initial results from the use of learning classifier systems to control in vitro neuronal networks. In: Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation (GECCO), pp. 369–376. ACM, London (2007) 20. Hammond, M., Marshall, S., Downes, J., Xydas, D., Nasuto, S., Becerra, V., Warwick, K., Whalley, B.J.: Robust Methodology For The Study Of Cultured Neuronal Networks on MEAs. In: Proceedings 6th International Meeting on Substrate-Integrated Micro Electrode Arrays, pp. 293–294 (2008) 21. Potter, S.M., DeMarse, T.B.: A new approach to neural cell culture for long-term studies. Journal Neuroscience Methods 110, 17–24 (2001)


15

22. Rolston, J.D., Wagenaar, D.A., Potter, S.M.: Precisely Timed Spatiotemporal Patterns of Neural Activity in Dissociated Cortical Cultures. Neuroscience 148, 294–303 (2007) 23. Wagenaar, D., DeMarse, T.B., Potter, S.M.: MEABench: A Toolset for Multi-electrode Data Acquisition and On-line Analysis. In: Proc. 2nd Int. IEEE EMBS Conf. Neural Eng., pp. 518–521 (2005) 24. Xydas, D., Warwick, K., Whalley, B., Nasuto, S., Becerra, V., Hammond, M., Downes, J.: Architecture for Living Neuronal Cell Control of a Mobile Robot. In: Proc. European Robotics Symposium EURO 2008, Prague, pp. 23–31 (2008) 25. Hutt, B., Warwick, K., Goodhew, I.: Emergent Behaviour in Autonomous Robots. In: Bryant, J., Atherton, M., Collins, M. (eds.) Information Transfer in Biological Systems. Design in Nature Series, vol. 2, ch. 14. WIT Press, Southampton (2005) 26. Hasselmo, M.E.: Acetycholine and learning in a cortical associative memory source. Neural Computation Archive 5, 32–44 (1993) 27. Cozzi, L., Chiappalone, M., Ide, A., Novellino, A., Martinoia, S., Sanguineti, V.: Coding and Decoding of Information in a Bi-directional Neural Interface. Neurocomputing 65/66, 783–792 (2005) 28. Novellino, A., Cozzi, L., Chiappalone, M., Sanguinetti, V., Martinoia, S.: Connecting Neurons to a Mobile Robot: An In Vitro Bi-directional Neural Interface. In: Computational Intelligence and Neuroscience (2007) 29. Karniel, A., Kositsky, M., Fleming, K., Chiappalone, M., Sanguinetti, V., Alford, T., Mussa-Ivaldi, A.: Computational Analysis In Vitro: Dynamics and Plasticity of a NeuroRobotic System. Journal of Neural Engineering 2, S250–S265 (2005) 30. Hutt, B., Warwick, K.: Museum Robots: Multi-Robot Systems for Public Exhibition. In: Proc. 35th International Symposium on Robotics, Paris, p. 52 (2004) 31. Marks, P.: Rat-Brained Robots Take Their First Steps. New Scientist 199(2669), 22–23 (2008) 32. DeMarse, T., Wagenaar, D., Blau, A., Potter, S.: The Neurally Controlled Animat: Biological Brains Acting with Simulated Bodies. Autonomous Robots 11, 305–310 (2001) 33. Chang, Q., Gold, P.: Switching Memory Systems during Learning: Changes in Patterns of Brain Acetylcholine Release in the Hippocampus and Striatum in Rats. Journal of Neuroscience 23, 3001–3005 (2003)

Sound Recognition Yang Cai and Károly D. Pados Carnegie Mellon University [email protected]

Abstract. Sound recognition has been a primitive survival instinct of early mammals for over 120 million years. In the modern era, it is the most affordable sensory channel for us. Here we explore an auditory vigilance algorithm for detecting background sounds such as explosion, gunshot, screaming, and human voice. We introduce a general algorithm for sound feature extraction, classification and feedback. We use Hamming window for tapering sound signals and the short-term Fourier transform (STFT) and Principal Component Analysis (PCA) for feature extraction. We then apply a Gaussian Mixture Model (GMM) for classification; and we use the feedback from the confusion matrix of the training classifier to redefine the sound classes for better representation, accuracy and compression. We found that the frequency coefficients in a logarithmic scale yield better results versus those in linear representations in background sound recognition. However, the magnitude of the sound samples in a logarithmic scale yields worse results versus those in linear representations. We also compare our results to that of the linear frequency model and the Melscale Frequency Cepstral Coefficients (MFCC)-based algorithms. We conclude that our algorithm reaches a higher accuracy with available training data. We foresee broader applications of the sound recognition method, including video triage, healthcare, robotics and security. Keywords: audio, sound recognition, event detection, sound classification, video analytics, MFCC, sound spectrogram.

1 Introduction The middle ear is perhaps one of the most sensitive organs in all 5400 known mammals. Separated from the jawbone, the middle ear enables mammals to sense surroundings while chewing food. Sensitive hearing made it possible for early mammals to coexist with the dinosaurs; it was literally a matter of life and death. To hunt small insects, the mammalian middle ear is sensitive to high-pitched noises like a mosquitos whine [33]. To avoid day-hunting dinosaurs, the mammalian ear is tuned for detecting very quiet sounds at night. Even today, most mammals prefer to come out after dark. A recent archeological discovery suggested that, for over 120 million years, having an elaborated auditory system that was well-adapted has been fundamental for mammals for their survival [34-35]. Sound recognition is a primitive instinct for mammals. In the modern era, it is the most affordable sensory channel for us, ranging from watermelon selection and car Y. Cai (Ed.): Computing with Instinct, LNAI 5897, pp. 16–34, 2011. © Springer-Verlag Berlin Heidelberg 2011

Sound Recognition

17

diagnosis to using a medical stethoscope. Taking asthma diagnosis as an example, the sound generated by asthma patients breathing is widely accepted as an indicator of disease activity [20-21]. Digital devices may revolutionize the patient’s care by monitoring airway diseases in real-time, including recording, transmission and recognition of tracheal breath sounds [22-23]. Robotics researchers have tried to simulate natural auditory vigilance in robots. For example, the robotic head can turn to the auditory source. In some video surveillance systems, the cameras can also pan, tilt and zoom to the auditory source at night. We call this listening-only mode ‘passive sensing’. Wu and Siegel, et al [19] developed a system that can recognize vehicles based on their sound signatures. They recorded sounds of various vehicles, built feature vectors based on spectrograms and Principle Component Analysis, and classified the vectors by defining a Euclidean distance measure to the center of each known class. On the other hand, to make the auditory vigilance more effective, we emit a sound to generate echo patterns. We call this “active sensing”; for example, an ultrasound sensor array is used on autonomously driving cars to detect nearby obstacles in real-time. Many animals use active sensing based on sound echoes, so called echolocation. Bats, dolphins and killer whales all use echolocation. It allows these animals to accurately determine not only the direction of the sound, but also the distance of the source by measuring the elapsed time between the signals transmission and the echoes reception. This inspires a renaissance of artificial ethology: What if we were capable of sensing like a bat? Online music retrieval is another motivation for sound recognition. Query by tapping (QBT) is a novel method based on the rhythm of songs [24]. The system captures the essential elements of rhythm, allowing it to be represented in textual form, which has well-established algorithms to tolerate tempo variations and errors in the input. The beauty of this invention is that it doesn’t require any special hardware or software - the input device is simply the space bar on the computer keyboard. Audio genre classification is also explored in [13], where the feature vector is built using statistical features to represent the “musical surface” of the audio; they also incorporate a discrete wavelet transform. They use the final 17-dimensional feature vector to classify the audio genre. Perhaps the more challenging task here is to understand, annotate, or search the ever-growing number of digital videos based on content. In many cases, we only need to know a rough class of the clips, called “video triage” before going down to visual and textual details, e.g., explosion, gunshot and screaming, which we annotate as dangerous, alarming or scary scenes. In addition, we only need to know the gender and a rough age range to classify persons in the video, so called “soft biometrics”. In this Chapter, we would like to focus on how to extract auditory features and classify them for video triage.

2 Our Algorithm Our algorithm contains three processes: feature extraction, classification and feedback. The feature extraction process includes sampling raw audio signals and transforming the auditory data into frequency domain feature vectors and compressing them into lower feature dimensions. The classification process includes a machine

18

Y. Cai and K.D. Pados

learning model. Just like humans, computers must be taught what sounds they should recognize. To be able to classify sounds in an unsupervised manner, we have to carefully select and prepare a training dataset and train the model with feature vectors that were extracted in the previous step. Finally, we must adjust the definition of classes according to the feedback from the training results. Figure 1 shows an overview diagram of our approach. In the following sections, we present the solution we have chosen for each of the above steps. In the end, we present our results. Training and Testing Input Signal

Normalization, Windowing

Feature Construction

Feature Compression

Feedback Class Regrouping

Statistical Classification

Output Fig. 1. Overview of our algorithm

3 Feature Extraction There are many methods for auditory feature extraction. For example, frequency analysis, wavelet decomposition, the popular Mel-frequency Cepstral Coefficients (MFCC), more complex feature vectors with additional statistical (such as spectral centroid and variance), and psychoacoustic (sharpness, pitch, loudness, etc.) components [5, 13]. Here we only use analytical features derived from the spectrum of audio frequency and strength. In this section, we focus on signal sampling, transformation and dimensional reduction. 3.1 Human-Like Auditory Sensing The outer ear leads sound pressure waves to the eardrum and in turn to the middle ear. The middle ear consists mainly of ear bones. It protects the inner ear by dampening sounds that are too loud, while at the same time it amplifies some frequencies up to a factor of 20, acting as a mechanical impedance matcher. Amplification is needed, since sound waves must enter the fluid contained in the inner ear after the ear bones and a large portion of the original energy gets lost at the interface of the liquid and air. The inner ear is where “hearing” actually occurs. Besides containing the organ for balance, the inner ear also houses the snail-shaped cochlea. The basilar membrane

Sound Recognition

19

inside the cochlea has different stiffnesses along its length and contains thousands of hair cells on its surface. The various stiffness levels of the basilar membrane cause different parts of it vibrate to different sound frequencies, bringing the hair cells to vibration at those places. For each vibration cycle, each hair cell emits a small pulse to the brain. The sum of all pulses, which are directly related to the frequencies contained in the waveform, is interpreted and perceived as a ‘sound’ by the brain. To summarize, one of the functions of the inner ear is to act as an organic frequency analyzer, directing different sound frequencies to specific receptors. In the end, human sound receptors react to frequencies instead of directly to the amplitude of the sound waves [29,30]. Also, according to Weber-Fechner’s law, the relationship between a stimulus physical magnitude and its perceived intensity is logarithmic, which can be described by the equation

p = k ⋅ ln

S S0

(1)

Here, S is the physical stimuli, p is its perceived intensity, k a context-dependant multiplicative factor and S0 is the threshold below which nothing is perceived. This relationship is valid for many of our sensory systems, such as for the feeling of weight, for vision, and, of course, hearing. We incorporate the above facts into our framework by analyzing the frequencies in the audio signal and scaling their magnitude logarithmically. Knowing WeberFechner’s law, taking the logarithm of the Fourier-transform of a signal simulates human hearing relatively closely. The goal of simulating human hearing is to make sure that a machine extracts the same kind of information from a sound wave that the human organ would, helping to build the same classes of sound that the human brain would interpret. 3.2 Signal Sampling

To represent an auditory signal, the first thought would be to use the Fourier transform. However, it assumes the signal has infinite length, which is not true in our case. The transform treats finite signals by infinitely repeating the same signal all over, which most often creates discontinuities in the signal as interpreted by the transform. In the resulting frequency analysis, this leads to unwanted effects, like the Gibbs phenomenon, which manifests as overshoot and ripples that surround the discontinuity during reconstruction. Ultimately, the problem is trying to represent a non-bandlimited signal using a finite number of infinite-length basis functions. Therefore, tapering or windowing is necessary to sample the signal. A smooth windowing function will make the signals ends connect smoothly by band-limiting the signal, minimizing the unwanted artifacts of the transform [29]. Here we use the Hamming window to sample the auditory signal slices. To somewhat compensate for the information loss caused by the Hamming window, there is an overlap of 128 samples between each time slice. All of those slices together result in approximately 0.5 sec of audio. Our empirical tests show that, as reference [6] has stated, taking approximately half a second of audio in total results in the best

20


*

=

Fig. 2. Hamming window minimizes unwanted artifacts (a) untapered signal; (b) Hammingwindow; (c) tapered signal. Note that the ends connect more smoothly than in (a).

performance and that the success rate saturates then declines with longer periods of data. Since all our data has been resampled to 44100Hz, 1024 samples for a single slice represent approximately 23ms of audio, which is in accordance with the works of others [1, 4, 5]. We construct the Hamming window as defined by [17] as eq. (2). The next step is to transform each of these slices of audio.

w(n) = 0.54 + 0.64cos(

π ⋅ 2n ) N −1

(2)

3.3 STFT Spectrogram

The spectrogram describes a piece of audio using its frequency spectrum over time. It is basically the frequency analysis of the audio, taken many times on different but adjacent time segments [18]. Because the Fourier transform is taken on short periods of data, a smooth windowing function is necessary. This we have already done in the previous step using the Hamming window. Taking the Fourier transform of windowed data of short periods is often referred to as the short-time Fourier transformation (STFT). Thus, the discrete STFT is defined as F(u), where w(n) is the windowing function as described previously [4]. F(u) =

N −1

∑

w(n)f(n)e

− π ⋅ 2j⋅ u ⋅ n N

(3)

n =0

A spectrogram is a collection of STFTs of adjacent audio data. Spectrograms are often visualized using a 2D plot, with the horizontal axis as the time and the vertical axis as the frequency. The magnitude of a frequency at a specific time is represented by the intensity or color of that point. The following illustrations show the spectrogram of the first three seconds of a representant of each class in our training database. The axes are linearly scaled from 0 to 3 seconds and from 0 to 22050Hz, while the magnitudes are normalized and scaled logarithmically. Brighter colors signify frequency components of higher magnitudes. The preliminary feature vector is then the concatenation of the logarithmically scaled Fourier coefficients of each time slice in the spectrogram for 0.5 seconds.

Sound Recognition

Fig. 3. Spectrogram of an explosion sound

Fig. 4. Spectrogram of multiple gunshots

Fig. 5. Spectrogram of a female speaking

Fig. 6. Spectrogram of a male speaking

21

22


Fig. 7. Spectrogram of a male screaming

3.4 Compressing the Feature Vector

After calculating the spectrogram, our feature vector has 24 slices, each with 512 elements. Thus the total length of our feature vector is 12288. Unfortunately, working with such huge features would require us to use a training database that is orders of magnitude larger. According to the Hughes phenomenon [8], the ratio between the number of training samples and the length of the feature vector must be high enough for a statistical classifier to function properly. As such, an optimal length for the final feature vector has to be chosen. If it is too low, it cannot represent the features of the classes well enough, but if it is too high, noise or other statistically unimportant features will have too much influence. We have heuristically found that a feature vector length of 24 works well for our data. We have also used exactly the same number of time slices, but these two numbers are not related. The reduction of the size of the feature vector is done in two steps. For each time slice of Fourier coefficients, we first compress each slice into 24 floating point numbers. This is done by tiling the frequency axis into 24 equidistant bins and summing up each coefficient into the corresponding bin. To further reduce the dimensionality across time segments, we implement principal component analysis (PCA). PCA transforms a data set into a coordinate system where the first component (axis) of the data set corresponds to most of the variance in the data [10]. Each succeeding component accounts for the remaining variability in a decreasing manner. PCA thus decorrelates the data and can be used to reduce the dimensionality by keeping only the first few components. This implies information loss, but in such a way that the statistically most important features are still kept [2,10]. In our implementation, we keep only the very first component, called the principal component, and transform our 24×24 vector elements into a single set of 24 elements using this component. This is our final feature vector. Assume we have data set X represented by a M × N matrix containing N sets of M observations, and we want to perform principal component analysis on X. Our data is first mean-adjusted across each dimension. N

xm =

∑

1 ⋅ X m, n N n =1

(4)

Sound Recognition

A m, n = X m, n − x m

for all m, n

23

(5)

Then we construct the M × M covariance matrix C, with each element Ci,j containing the covariance of the data in dimensions i and j of A. Let I and J denote the vectors of data in the appropriate dimensions. M

cov(I, J) =

∑ (I k =1

k

− I)(J k − J )

(6)

M −1

From the covariances another matrix V that contains the eigenvectors of C is computed. These eigenvectors are sorted in descending order based on the associated eigenvalues. Eigenvalues can be found by Reighley quotient iteration and are in turn used to get the eigenvectors using Gaussian elimination. The eigenvector with the highest eigenvalue is called the principal component of our data. Since we are only interested in the principal component in our implementation, we keep only this component and transpose and normalize it to unit length. Treating it as a 1-row matrix V’, we finally use it to transform our data into a single M-element vector. This mathematically projects our data set into the space represented by the principal component, formulated as: Y = V' × A

(7)

where A is the mean-adjusted data set, V’ is the principal component, and × is matrix multiplication. Here is a pseudo code of the algorithm for extracting features of audio: 1. Tile audio signal data into slices with overlap 2. Zero offset 3. Normalize by dividing by the greatest magnitude 4. For each slice a. Apply Hamming-window b. Compute logarithm of power spectrum c. Compress using frequency binning 5. Concatenate processed slices 6. Compress using principal component

The resulting feature vector can be used to train a classifier in the training phase. Alternatively, after training, the feature vector can be used to classify new audio samples.

4 Sound Classification There are many kinds of classifiers for sound recognition. Support Vector Machines [25] are a common choice, as well as k-Nearest Neighbor schemes [19], neural networks [31] or Hidden Markov Models [3]. Not all of these methods return a probability value for the result of the classification. It is advantageous not only to be able to classify sound data, but to be able to detect whether a pattern we are interested in is

24


present or not. Gaussian Mixture Models are also widely used in signal processing. They calculate the probability value as a confidence indicator. Based on this indicator, we should be able to reject a feature vector and tell that it does not belong to any of the predefined and trained classes if this value is too low. Here we chose a Gaussian Mixture model for our implementation. 4.1 The Classification Model

Classification problems are often referred to as part of clustering. The goal is, given a point in high-dimensional space (e.g., a feature vector), to find the class of known points it belongs to where a class of points is represented by potentially-multiple clusters. Clusters are also called components and are described by their statistical properties [12]. A single cluster is mathematically represented by its parameterized statistical distribution, most commonly with a Gaussian distribution in the continuous case. The entire data in a class is thus represented as a mix of these Gaussian distributions, hence the name Gaussian mixture model.

Class 1 Class 2 f(Y) = f1(Y) + f2(Y) Component 1 f1(Y)

Component 2

f2(Y)

Feature space

Density function of a class

Fig. 8. (a) A set of points in 2D space generated by a Gaussian mixture. Each cluster can be composed of multiple components. (b) The probability density function f of a 2-component Gaussian mixture.

Assume that Z is a set of feature vectors and d is the dimensionality of a single vector Y (Y ∈ R d ) all belonging to the same class, built as described above. If there are K components, component k can be parameterized by its mean µ k and its covariance matrix C k [12].

f k (Y) = ϕ (Y | μ k , C k ) =

1 (2π ) d C k

e

− (Y − μ k ) T C k−1 (Y − μ k ) 2

(8)

Then the mixture density, where each component’s weight ak is described in reference [11].

Sound Recognition

25

K

f(Y) =

∑a f

k k (Y)

(9)

k =1

To find the parameters of the clusters, given the feature vectors from the training set and the classes they belong to, is the job of the training procedure. The goal is to find the parameters of the unknown distributions for each class so that they maximize the probability of the data of the same class [11]. Formally, for data set Z and parameter set θ find

θ ' = arg max p(Z | θ ) = arg max θ

θ

n

∏ p(Y

p

|θ)

(10)

p =1

The expectation maximization (EM) algorithm iteratively finds an approximate of θ given the above criteria and it can be used for other kinds of distributions too. In each iteration, the algorithm alternately performs either the E-step (expectation) or the Mstep (maximization). In the E-step, each data is given some probability of belonging to each of the clusters. In the M-step, the cluster parameters are recomputed based on all the data belonging to them, weighted by the probability of the data at hand belonging to a specific cluster. The process then starts over. It has been shown that the EM-algorithm converges [11] to a maximum. Initialization can be done randomly in simpler cases or with other algorithms like the k-Means algorithm. Initialization is important to let the algorithm converge to a global maximum. For more information on the EM-algorithm, see references [7,8,11]. The reference [12] has information about the EM-algorithm that is specific to Gaussian Mixture models. The implementation used by our application is described in reference [16]. 4.2 Classification Procedure

Classification is achieved by evaluating the probability of a feature vector for each of the clusters with trained parameters. The feature vector is assumed to belong to the cluster which produces the highest probability. For practical reasons and because of limitations of digital computers, the natural logarithm of probabilities is used in calculations instead of the true probabilities themselves. Multiplying multiple probability values together, which are smaller than 1.0 by definition, would quickly result in underflows. The property of the logarithm that reduces multiplication to addition helps resolve this problem. Detection is achieved by letting the application classify the test samples and also output the log-likelihood of the data belonging to the determined class. If this probability value is too low, the data is rejected and will not be considered as one of the known classes. However, care has to be taken when determining the threshold for the probability value, as there is no common global value suited for every case. The optimal threshold should be determined by hand and depends heavily on the clusters, the training database and the exact algorithms used.

26


5 Feedback Model In many cases, we don’t know the acoustic correlations of sound classes. The classification results from machine learning may provide valuable feedback about the acoustic correlation. When possible, we can regroup the highly correlated classes to a new class with a common name. Audio classification results are often presented in the form of a confusion matrix, a table where the header of every row is the actual class of the audio and the header of every column is the class that the audio has been detected as. Given a reasonable classification procedure, such a confusion matrix can be used as a guiding means to collapse two classes if they are found positioned too close to each other in the chosen feature space. This is useful to recognize falsely assumed dissimilarity between those two classes. In a confusion matrix, the number of correctly classified samples accumulates in the matrix diagonal. Falsely classified ones will be found outside of the diagonal. If two classes indexed i and j are being processed separately although they wield very similar features, the majority of the errors will be found in j for i, and in i for j. Misclassification for any of those two against any other third class will be comparatively low. Using this method, we can specify a margin we call a collapsing threshold that, when reached, will cause the two classes to be collapsed into one. Let i and j be the indices of two arbitrary classes after a complete classification procedure. Furthermore, let R be the confusion matrix of n × n dimensions for n number of audio classes. The sum of row l of R can be defined as n

Sl =

∑R

(11)

l, k

k =1

Then we can define a Boolean expression B that, when it evaluates to true, causes classes i and j to be collapsed.

R j,i ⎧ R i, j ≥t ∧ ≥t ⎪ B i, j = ⎨ (Si − R i,i ) (S j − R j, j ) ⎪ Else ⎩

True

(12)

False

With t being the collapsing threshold, the above expression is to be evaluated for each class pair i and j, where i ≠ j.

6 Experiment Design We have set out to be able to classify five classes of audio: screams, explosions, gunshots, human female speech and human male speech. For each of these classes, we have collected at least a hundred audio files and, for each of the audio files, a single feature vector was constructed based on the beginning of the data. Data came from various sources in various formats and in general was not noise-free. This was to make sure that the classifier is trained on general data that can be used in practice. Sources include audio collection libraries, audio streams of videos from the video

Sound Recognition

27

sharing website YouTube, and audio streams of videos that were shot using common (unprofessional) handheld cameras. However, most of the speech samples originate from the TIMIT database [27]. The formats include compressed and uncompressed data, ranging from low-quality 8 bit, 11025 Hz sources to high-fidelity 16 bit, 48 kHz sources. To be able to decode all these formats we used the open-source FFmpeg libraries [28]. During the processing of the sound files, all were transformed and resampled into a common 64 bit floating point, 441001 Hz mono format to lower the complexity of the later stages in the application. This relieved us from having to code multiple code paths for processing and interpreting multiple sample formats. The conversion was done using the FFmpeg audio conversion API. Table 1 provides an overview of the number of files we use in each class. Of all the samples collected, roughly 80% from each class was used for training, and the remaining 20% was used for obtaining the classification results. The training and testing sets do not overlap. Table 1. Number of sound files in the database No. of files for training

No. of files for testing

Explosion

184

45

Gunshot

110

25

Female

199

55

Male

203

55

Scream

95

23

7 Results The classification results are presented on Table 2. We show confusion matrices. At the end of each row, the success rate for the detection of that class is summarized. Table 2. Spectrogram-based results

1

Scream

Explosion

Gunshot

Male

Female

Success

Scream

22

0

1

0

0

95.65%

Explosion

2

37

6

0

0

82.22%

Gunshot

5

7

13

0

0

52.00%

Male

2

1

0

52

0

94.54%

Female

2

0

1

0

52

94.54%

44.1kHz is a common sampling rate to represent sounds perceivable by most humans.

28


Screams have an almost perfect classification rate. Differentiation between male and female speech is very good, with only six samples being misidentified out of 110. Explosions also get classified well, except in some cases where an explosion is mistaken for a gunshot. Gunshot is the class with the lowest overall success in all of our tests; the most frequent error was mixing them up with explosions, but on occasion they were mistaken for screams. However, without knowing the context it is often hard or impossible even for humans to differentiate explosions from gunshots. These two classes are very similar in their features, which is not surprising as both are the result of principally the same chemical and physical reactions. Using the feedback model from Section 5, we can revise the table such that explosions and gunshot form a new class called “blast”. Table 3. Results using collapsed classes Scream

Blast

Male

Female

Success

Scream

22

1

0

0

95.65%

Blast

7

63

1

0

90.00%

Male

2

1

52

0

94.54%

Female

2

1

0

52

94.54%

8 Comparisons with Other Methods 8.1 Log vs. Linear

As mentioned earlier, one step in building the feature vectors was taking the logarithm of the power frequency magnitudes. In this experiment we conclude that logarithmic scaling does indeed produce better results when compared to linear scaling, as can be seen from Table 4. Table 4. Spectrogram-based results with linear magnitude scaling Scream

Explosion

Gunshot

Male

Female

Success

Scream

21

1

0

0

1

91.30%

Explosion

2

17

25

1

0

37.78%

Gunshot

1

7

13

4

0

52.00%

Male

0

1

2

48

4

87.27%

Female

1

2

1

15

36

65.45%

Logarithmically scaling the frequencies produces better results, because it resembles the way humans hear more closely. Knowing Weber-Fechner’s law, taking the logarithm of the Fourier-transform of a signal simulates human hearing relatively

Sound Recognition

29

Table 5. Spectrogram-based results with logarithmic magnitude scaling Scream

Explosion

Gunshot

Male

Female

Success

Scream

21

1

0

0

1

91.30%

Explosion

1

16

14

14

0

35.56%

Gunshot

2

7

11

5

0

44.00%

Male

1

2

0

50

2

90.91%

Female

0

2

1

19

33

60.00%

closely. The goal of simulating human hearing is to make sure that a machine extracts the same kind of information from a sound wave that the human organ would, helping to build the same classes of sound that the human brain would interpret. In a separate experiment, we also tried scaling the samples logarithmically instead of the power frequency. These results are also inferior to those listed in Table 2. 8.2 MFCC vs. Spectrogram-Based

For speech genres, MFCC is often used for constructing a feature vector. A great amount of research already went into inspecting for what purposes MFCC is adequate and determining the best way to perform the transformation. The study in reference [1] uses the discrete cosine transform as an approximation of the KL-transform to decorrelate the elements of a feature vector that is also MFCC-based for music. This study shows that MFCCs are not only suited for speech but also for music modeling. Although it does not claim that MFCCs are optimal for music, it does conclude that they are at least not harmful. The study in reference [9] tries to determine music similarity and additionally compares different MFCC implementation techniques. They conclude that, “with MFCCs based on fixed order, signal independent LPC, warped LPC, MVDR, or warped MVDR, genre classification tests did not exhibit any statistically significant improvements over FFT-based methods”. This leads to the conclusion that it is preferable to use FFT for the spectral estimation in MFCC for music similarity because of its performance advantage. The study in [4] also compares MFCCs but in the context of MP3 encoding quality. Their results show that the filter bank implementation of the MFCC is only an issue at low bitrates. Since MFCCs seem to perform well for both speech and music spectra, we extend this idea and try to use them for general sound pattern modeling too. The study in reference [5] explores the classification of not only some music genres, but also speech, noise and crowd noise comparing multiple feature sets. In [14], different sound patterns, like explosions and cries, are categorized using spectral properties, and a correlation model is used as the classifier. The study in reference [15] describes a content-based retrieval system for various sounds. As an alternative to the method described in Section 3, we produced an implementation based on Mel-scale Frequency Cepstral Coefficients (MFCC). MFCCs have been widely adopted in the field of describing features of human speech, speech

30


recognition and voice identification as well as for geological purposes. It has been shown that they are even suitable for music classification [1,13]. Inspired by these results, we will see whether MFCC is adequate for the sound classes in our tests too. First, we define the cepstrum of a sound as the magnitude of the Fourier transform of the logarithmically scaled magnitude-spectrum of the original audio [1,4]. MFCC(n) = FFT(log FFT(w(n)f(n)) )

(13)

The Mel-scale cepstrum is given by transforming the frequency axis after the first FFT operation into Mel-scale, which is commonly given by [3,4].

ϕ = 2595 ⋅ log10 (

f + 1) 700

(14)

For the conversion into Mel-scale, different implementations exist and the exact conversion is most often only approximated. The common method is to define a set of band-pass filters, where the distance between the centers of each filter is based on the Mel-scale. As studies [9,4] show, implementations differ in many aspects, including the exact shape of the filters, the lower and upper bound frequencies, the magnitude of the filter centers and the number of bands in the filter bank. Some implementations place the bands in the lower frequency range on a linear estimation of the Mel-scale. In our implementation, all bands have equal height; they range from 20Hz to 22050Hz and we define 24 bands. This way, there is no need for explicitly binning the frequencies, as with the spectrogram approach, because that step is automatically done when converting to the Mel-scale. Table 6. MFCC-based results Scream

Explosion

Gunshot

Male

Female

Success

Scream

20

1

1

0

1

86.96%

Explosion

9

30

5

1

0

66.67%

Gunshot

2

14

9

0

0

36.00%

Male

0

2

1

51

1

92.72%

Female

0

0

1

2

52

94.54%

The use of Mel-scale Frequency Cepstral Coefficients resulted in marginally better results for human speech for clear samples only, giving considerably worse classification rates for the other classes. When we added noisy samples into the male and female classes, the results got slightly worse as can be seen from the tables. We conclude that MFCC is a useful tool when used appropriately, but it does not generate feature vectors suitable for all audio classes. For the classes tested in this work, our own feature vector implementation produces significantly better results.

Sound Recognition

31

9 Background Noise Issues In our context, we refer to unwanted sounds as noise. Noise is unwanted because it negatively affects the processing of our target auditory data. In general, automated processing equipment like computers cannot tell noise from the useful data, unless we have some knowledge about the nature of the noise itself. For classifications, for example, if the noise component of the data is too strong, there is a high possibility that the feature extraction process will construct features of the noise, or that the classifier will be trained to the nature of the unwanted data instead of the intended audio component. Samples collected from audio sources are in general not noise-free. Noise comes from many different sources, and it is possible to classify noise sources based on different criteria. One possible classification scheme, and maybe the most intuitive one, is the origin of the noise, like environmental noise (e.g., a crowd, wind), transmission channel noise (FM-broadcast), data corruption (digital files), recording equipment (imperfections of the hardware) or observation-inherent (sampling, quantization). Not all of these are relevant in every case: with current technology for example, quantization noise is rarely an issue nowadays. If the noise is statistically of a different nature than the audio that we originally target, it is also possible to filter out even relatively strong noise. A random background noise can be easily filtered out from periodic waveforms, for example. On the other hand, trying to filter out strong crowd noise from a single human’s speech can be very challenging. For sound classifier applications, if it can be foreseen that future data to be tested will not be noise-free in general and that the noise cannot be removed or is not practical to remove, it is important to also use noisy audio samples for the training procedure. This allows the noise to be trained into the classification framework, making it more immune and somewhat tolerant to the noise. In general, it is good practice to train the system on a relatively large number of sound samples even if it will only differentiate between a few classes. This is because of the huge variability of the audio samples, whose two largest contributors are variability in the source itself and noise. The amount of noise that can be tolerated will be highly case and- implementationdependant. It will depend on the number and kind of training samples used (and so indirectly on the class definitions also), the feature extraction implementation, and the exact classifier in the classification framework. The study in [25] does non-speech human sound classification and is tolerant to noise if the Signal-To-Noise Ratio (SNR) reaches approximately 50 dB. On the other hand, the speech recognition framework studied in [3] tolerates noise even with an SNR as low as 10 dB in their measurements. The study in [26] designs a noise-robust FFT-based feature vector that achieves success rates better than 90% with an SNR of 10 dB.

10 Conclusions Sound recognition is a basic instinct for mammals. In this Chapter, we explore how to enable auditory vigilance computationally. We use Hamming window to taper sound signals, the short-term Fourier transform (STFT) to extract the acoustic features and

32


use principal component analysis (PCA) to compress the feature vector. Then we apply a Gaussian mixture model (GMM) with the expectation maximization (EM) algorithm for classification. Based on 203 test samples and 791 training samples, we successfully recognize blast (explosion or gunshot), screaming, human female voices and human male voices with accuracy over 90.00%. We believe that our results surpass previous studies in terms of high accuracy and robustness (real-world data). Second, to improve the sound representation and classification, we add a feedback channel from the confusion matrix of the training classifier so that we can collapse sound classes for better accuracy and compression. For example, by collapsing explosion and gunshot classes into one ‘blast sound,’ we not only reduce the number of classes, but also increase the accuracy from 52% to 90.00%. This also indicates that sound classes could be hierarchic. The more accurate the sound classifier, the better distinguished the resolution of lower level sounds. Our feedback model provides a measurable way to refine the class definitions in a hierarchy. We found that the frequency in a logarithm scale yields better results in background sound recognition, which is consistent to Weber-Fechner Law. However, the magnitude in a logarithm scale doesn’t yield better results versus linear representation. We also compare our results with other sound recognition methods, such as MFCC. We found that MFCC is good for representing human sounds and music. However, in this case, since we have a broader range of sound spectrum, our result is better than MFCC. Finally, we believe that sound recognition can go beyond auditory vigilance of anomalous sounds. It has potential in video triage, healthcare, robotics and security. Using a passive sensing mode, sound recognition is definitely an affordable watchdog.

Acknowledgement This research was supported by Center for Emergency Response Team at Carnegie Mellon and CyLab at Carnegie Mellon under grants DAAD19-02-1-0389 and W911NF-09-1-0273 from the Army Research Office. The authors would like to thank William Eddy and Emily Durbin for their comments and editing, Mel Siegel and Huadong Wu for their inspiring work on vehicle sound recognition and Rafael Franco for his outstanding rapid prototype that led to this project.

References 1. Logan, B., et al.: Mel Frequency Cepstral Coefficients for Music Modelling. Cambridge Research Laboratory (2000) 2. Lindsay, I., Smith, A.: Tutorial on Principal Components Analysis (2002) 3. Shannon, B.J., Paliwal, K.K.: A Comparative Study of Filter Bank Spacing for Speech Recognition. In: Microelectronic Engineering Research Conference (2003) 4. Sigurdsson, S., et al.: Mel Frequency Cepstral Coefficients: An Evaluation of Robustness of MP3 Encoded Music, Technical University of Denmark (2006)

Sound Recognition

33

5. Breebaart, J., McKinney, M.: Features for Audio Classification, Philips Research Laboratories (2008) 6. Spina, M.S., Zue, V.W.: Automatic transcription of general audio data: Preliminary analysis. In: Proc. 4th Int. Conf. on Spoken Language Processing, Philadelphia, PA (1997) 7. Dellaert, F.: The Expectation Maximization Algorithm, College of Computing, Georgia Institute of Technology (2002) 8. Hsieh, P.-F., Landgrebe, D.: Classification of High Dimensional Data, Purdue University School of Electrical and Computer Engineering, ECE Technical Reports (1998) 9. Jensen, J.H., et al.: Evaluation of MFCC Estimation Techniques for Music Similarity 10. Shlens, J.: A Tutorial on Principal Component Analysis: Derivation, Discussion and Singular Value Decomposition (2003) 11. Bengio, S.: An Introduction to Statistical Machine Learning - EM for GMMs -, Dalle Molle Institute for Perceptual Artificial Intelligence (IDIAP) slides (2004) 12. Li, J.: Mixture Models, Department of Statistics slides, The Pennsylvania State University (2008) 13. Tzanetakis, G., Cook, P.: Musical genre classification of audio signals. IEEE Trans. Speech Audio Processing 10, 293–301 (2002) 14. Pfeiffer, S., Fischer, S., Effelsberg, W.: Automatic audio content analysis. Tech. Rep. No. 96-008, University of Mannheim (1996) 15. Foote, J.: Content-based retrieval of music and audio. Multimedia Storage and Archiving Systems II, 138-147 (1997) 16. Bouman, C.A.: CLUSTER: An Unsupervised Algorithm for Modeling Gaussian Mixtures, School of Electrical Engineering, Purdue University (2005) 17. Window function, http://en.wikipedia.org/wiki/Window_function (retrieved on 07/30/2010) 18. Spectrogram, http://en.wikipedia.org/wiki/Spectrogram (retrieved on 07/30/2010) 19. Siegel, M., et al.: Vehicle Sound Signature Recognition by Frequency Vector Principal Component Analysis. IEEE Trans. on Instrumentation And Measurement 48(5) (October 1999) 20. Spiteri, M.A., Cook, D.G., Clark, S.W.: Reliability of eliciting physical signs in examination of the chest. Lancet. 2, 873–875 (1988) 21. Pasterkamp, H., Kraman, S.S., Wodicka, G.R.: Respiratory sounds:advances beyond the stethoscope. American Journal of Respiratory Critical Care Medicine 156, 974–987 (1997) 22. Anderson, K., Qiu, Y., Whittaker, A.R., Lucas, M.: Breath sounds, asthma, and the mobile phone. Lancet. 358, 1343–1344 (2001) 23. Cai, Y., Abascal, J.: Ambient Intelligence in Everyday Life. LNCS (LNAI), vol. 3864. Springer, Heidelberg (2006) 24. Peter, G., Cukierman, D., Anthony, C., Schwartz, M.: Online music search by tapping. In: Cai, Y., Abascal, J. (eds.) Ambient Intelligence in Everyday Life. LNCS (LNAI), vol. 3864, pp. 178–197. Springer, Heidelberg (2006) 25. Liao, W.-H., Lin, Y.-K.: Classification of Non-Speech Human Sounds: Feature Selection and Snoring Sound Analysis. In: Proc. of the 2009 IEEE Int. Conf. on Systems, Man and Cybernetics (2009) 26. Chu, W., Champagne, B.: A Noise-Robust FFT-Based Spectrum for Audio Classification, Department of Electrical and Computer Engineering. McGill University, Montreal (2006) 27. TIMIT Acoustic-Phonetic Continuous Speech Corpus, Linguistic Data Consortium, University of Pennsylvania, http://www.ldc.upenn.edu/Catalog/ CatalogEntry.jsp?catalogId=LDC93S1

34


28. FFmpeg, http://ffmpeg.org 29. Smith, S.W.: The Scientist & Engineer’s Guide to Digital Signal Processing. California Technical Pub. (1997) ISBN 0966017633 30. Hearing Central LLC, How the Human Ear Works, http://www.hearingaidscentral.com/howtheearworks.asp (retrieved on 10/25/2010) 31. Lee, H., et al.: Unsupervised feature learning for audio classification using convolutional deep belief networks. Stanford University, Stanford (2009) 32. Forero Mendoza, L.A., Cataldo, E., Vellasco, M., Silva, M.: Classification of Voice Aging Using Parameters Extracted from the Glottal Signal. In: Diamantaras, K., Duch, W., Iliadis, L.S. (eds.) ICANN 2010. LNCS, vol. 6354, pp. 149–156. Springer, Heidelberg (2010) 33. Angier, N.: In Mammals, a Complex Journey. New York Times (October 13, 2009) 34. Ji, Q., Luo, Z.X., Zhang, X.L., Yuan, C.X., Xu, L.: Evolutionary Development of the Middle Ear in Mesozoic Therian Mammals. Science 9 326(5950), 278–281 (2009) 35. Martin, T., Ruf, I.: On the Mammalian Ear. Science 326(5950), 243–244 (2009)

Texture Vision: A View from Art Conservation Pierre Vernhes and Paul Whitmore Art Conservation Research Center, Department of Chemistry, Carnegie Mellon University, 700 Technology Drive, Pittsburgh, PA 15219 {pvernhes,pw1j}@andrew.cmu.edu

Abstract. The appreciation of many works of visual art derives from the observation and interpretation of the object surface. The visual perception of texture is key to interpreting those surfaces, for the texture provides cues about the nature of the material and the ways in which the artist has manipulated it to create the object. The quantification of texture can be undertaken in two ways: by recording the physical topography of the surface or by analyzing an image that accurately portrays the texture. For most art objects, this description of texture on a microscopic level is not very useful, since how those surface features are observed by viewers is not directly provided by the analysis. For this reason, image analysis seems a more promising approach, for in the images the surfaces will naturally tend to be rendered as they would when viewing the object. In this study, images of textured surfaces of prototype art objects are analyzed in order to identify the methods and the metrics that can accurately characterize slight changes in texture. Three main applications are illustrated: the effect of the conditions of illumination on perceived texture, the characterization of changes of object due to degradation, and the quantification of the efficiency of the restoration.

1 Introduction The appreciation of many works of visual art derives from the observation and interpretation of the object’s surface. The visual perception of texture is key to interpreting those surfaces, since the texture provides cues about the nature of the material and the ways in which the artist has manipulated them to create the object. But surface textures are not immutable qualities of an object. The surfaces can be portrayed differently depending on how they are illuminated during exhibition. Deterioration and damage can lead to alteration or loss of an object’s surface. Cracks may emerge on painted surfaces, metals may corrode, textiles may pill, stone may become worn or granular. These surface alterations threaten the aesthetic message of the artist. For this reason, conservators are often asked to restore a surface to some earlier condition or to stabilize the current state. Even the most delicate of treatments can further alter the surface. Some of the most extreme interventions, such as consolidation (the infusion of an adhesive in order to stabilize a very friable surface) can lead to a profound change in the appearance of the surface. A continuing challenge for the art conservation field is to develop treatments that are effective with minimal or acceptable changes to surface texture. This effort is made more difficult by the lack of Y. Cai (Ed.): Computing with Instinct, LNAI 5897, pp. 35–46, 2011. © Springer-Verlag Berlin Heidelberg 2011

36

P. Vernhes and P. Whitmore

an analytical method to quantitatively measure the appearance of surface texture and the changes induced by aging and treatment. Defining texture is not an easy task. As a matter of fact, there is no strict definition of texture, and each research field appropriates the word in a different way. Texture is often classified as complex visual patterns and subpatterns that have characteristic properties such as color, size, brightness, etc. [1]. Descriptions of the visual perception of texture are expressed in terms of roughness, regularity, glossy vs. matte, granulation, and so on. In addition, the textural properties of a surface are deeply related to the illumination conditions. Texture may be enhanced by grazing angle illumination, while diffuse illumination will tend to de-emphasize the surface topography. The quantification of texture can be undertaken in two ways: by recording the physical topography of the surface or by analyzing an image that accurately portrays the texture. For the former, 3D mapping of a surface using an optical profilometer or AFM is usually used when the surface quality requirements are extremely precise. For most art objects, this description of texture on a microscopic level is not very useful, since how those surface features are observed by viewers is not directly provided by the analysis. For this reason, image analysis seems a more promising approach, for in the images the surfaces will naturally tend to be rendered as they would when viewing the object. There exist numerous ways to perform image analysis on a collection of data. The four main categories of approaches to texture analysis are structural, statistical, model-based, and transform [2]. The choice of the set of analytical tools needed depends on the object and on the aim of the study. In this study, images of textured surfaces of prototype art objects were analyzed in order to identify the methods and the metrics that can accurately characterize slight changes in texture. As a case study, we investigated the effect of a dry-cleaning treatment on unpainted canvas. Three main applications are illustrated: the effect of the conditions of illumination on perceived texture, the characterization of changes of an object surface due to degradation, and the quantification of the texture changes resulting from conservation treatment.

2 Material and Methods 2.1 Experimental Setup The experimental device presented in Figure 1 allows for the control of the inclination of illumination. A digital camera is positioned normal to the surface of the sample to be examined (the camera is fixed). The sample itself is located on a rotating stage. Since the sample stage rotates, it is possible to fully describe the response of a sample to light according to both the inclination and the azimuthal position. Five inclination angles were chosen, 20°, 30°, 45°, 60° and 75°, respectively. The measurement of these angles is done by comparison to the position of the camera (ie: compared to the normal). A typical measurement involves 40 pictures, corresponding to 5 inclinations and 8 azimuthal positions of the lamps. The images were captured using the experimental setup presented in Figure 1. Image sizes were 2896 x 1944 pixels. The images were converted to gray scale in order to extract textural properties. The different algorithms and data treatments were coded using routines written with MATLAB V.8.

Texture Vision: A View from Art Conservation

Directional light spot

Digital camera

30°

45°

37

20°

60° 75°

Sample Rotating stage

Fig. 1. Experimental setup and the corresponding angular position of the lamps

2.2 Image Analysis Methods Non-destructive texture and surface quality analyses have applications for a large variety of material and fields. That is why the number of analytical tools aiming to describe surface texture is huge. Among the most popular are histogram analysis, autocorrelation, discrete image transform, ring/wedge filtering, Gabor filtering, and gray level co-occurrence matrices. Sonka et al [3] classify the different approaches by defining two families: statistical analysis and syntactic analysis. In a more elaborate categorization, Turceryan and Jain [1] distinguish four different approaches to texture analysis: statistical, geometrical, model-based, and signal processing approaches. 2.2.1 First-Order Histogram-Based Features The most natural way to analyze an image is to calculate its first-order histogram, from which parameters such as the mean, the variance, the standard deviation, the skewness or the kurtosis can be extracted. Despite their simplicity, histogram techniques demonstrate their usefulness in various applications. They are insensitive to rotation and translation. Although the first-order histogram does not carry any information on how the gray levels are spatially related to each other within the image, recent studies show that this simple method is capable of distinguishing matte from glossy textures. Motoyoshi et al. [4] demonstrated the close relationship between the asymmetry of the luminance distribution (the skew) and the perceived gloss or lightness of an image. 2.2.2 Gray Level Co-occurrence Matrix Another approach analyzes the second-order histogram of the image, the gray level co-occurrence matrix (GLCM) [5]. Co-occurrence probabilities provide a secondorder method for characterizing texture features. These probabilities represent the

38


conditional probabilities of all pair-wise combinations of gray levels in the spatial window of interest given two parameters: the interpixel distance or lag (δ) and the orientation (θ). Hence, the probability measure can be defined as: Pr( x ) = {C ij (δ , θ )}

(1)

where Cij is the co-occurrence probability between gray levels i and j and is defined as: Cij =

Pij G

(2)

∑P

i , j =1

ij

where Pij represents the number of occurrences of gray levels i and j within the image for a given pair (δ,θ), and G is the quantized number of gray levels. In order to reduce both the computing time and the noise of the signal the value of G is typically 8 or 16. Statistical parameters can be calculated for the GLCM matrix aiming to describe various textural properties. For example, the texture features include randomness, coarseness, linearity, periodicity, contrast and harmony. In this study, we mainly focused on the GLCM contrast, defined as: Contrast = ∑ Cij (i − j ) 2

(3)

Perceptually, an image is said to have a high contrast if areas of different intensity level are clearly visible. Hence, both the spatial frequency of change and the intensity difference between neighboring regions will affect the contrast. 2.2.3 Autocorrelation Function and Regularity Calculation The regularity of a pattern is of fundamental importance in the characterization of some textures, such as those of textiles or wood. A patterned texture has two meanings: the spatial relationship between pixel intensities and the repeat distance of repetitive units [6]. Several approaches have been developed to analyze the regularity of patterned structure. The GLCM was applied to image retrieval and defect detection ([7], [8]). Image subtraction [9], Gabor filtering [10] and Hash function were also investigated [11]. Spatial domain filtering is also a common tool for texture analysis. In this work, we focus on the autocorrelation function calculated from the Fourier transform in order to extract the regularity, following the work of Chetverikov ([12], [13]). Chetverikov developed a regularity calculation algorithm. For a given periodic structure, it is possible to calculate the regularity between 0 and 1 (1 describing a perfectly regular structure while 0 indicates a random texture). This work also demonstrated a close relationship between the definition of regularity and human perception. For extensive explanations of the different parts of the algorithm, see Chetverikov’s work ([12], [13]). Here we present only the main parts of the calculations. First, the power spectrum of the considered image is needed. The power spectrum

S (u, v) =

F (u, v) M2

2

where F is the Fourier transform of the image.

(4)


39

The power spectrum is related to the areal autocorrelation function (AACF) thanks to the Wiener theorem, which states that the AACF and the power spectrum form a pair.

AACF(m, n) = IFFT[ FFT[ I (m, n) × FFT[ I (m, n)]]

(5)

where IFFT is the inverse transform of FFT (Fast Fourier Transform). To allow the calculation in the alternate direction, the AACF is normalized and then converted into a polar representation. The regularity is composed of two contributions: the intensity regularity (Rint) and the position regularity (Rpos). Rpos represents the periodicity of the layout of the elements composing the pattern, while Rint indicates the regularity and stability of the intensity of the elements. For each angular direction, the normalized AACF is extracted. According to the position of the extrema and their intensity, Rint is calculated. The distance between the various extrema allows for the quantification of Rpos. The regularity is then defined as: Regularity=max{(Rpos x Rint)2}

(6)

Figure 2 briefly summarizes the different stages of the calculation.

(a)

(b)

(c)

(d)

Fig. 2. The various steps of the calculation of the regularity. From the raw image (a), the Fourier spectrum is calculated (b). Then a polar and normalized AACF representation is calculated (c), from which the regularity position and intensity (d) are obtained by applying the Chetverikov algorithm.

40


3 Texture Variation Analysis of Canvas Due to Cleaning The range of application of texture analysis in the field of art conservation is great. In order to illustrate its potential and its usefulness, we selected a particular case study: the cleaning of unpainted canvas. As a result of aging or damage, unpainted canvas can become stained or discolored. Several techniques exist to clean stained textiles, such as unpainted canvas, without using liquids, including soft and hard sponge, eraser, and eraser crumb [14]. These dry treatments involve a certain amount of abrasion of the surface, which causes slight changes in the canvas texture. Two commercial cotton duck canvases were selected (referenced as Canvas 1 and 2). Their surfaces were scrubbed with a fine sandpaper to simulate the cleaning process at two different stages. The first stage was a gentle scrubbing while the second stage was stronger, reaching the point where the canvas threads were visibly damaged. 3.1 Effect on Regularity A careful visual inspection of the canvas surfaces showed a decrease in the regularity of the weave pattern due to the scrubbing. The physical strain due to the scrubbing on the surface tended to affect the arrangement of the fibers. As a result of the gentle scrubbing, the tension of the weave was loosened and we also noticed the emergence of slub. With stronger scrubbing, defects in the structure, such as holes and snagged fibers, were observed. In order to quantify these variations in pattern regularity, we computed the algorithm described in Part 2.2.3 for the different samples considered. Figure 3 presents the results. 0.8

Canvas 1 Canvas 2

0.7

Regularity

0.6 0.5 0.4 0.3 0.2 0.1 0

Untreated

Gently scrubbed

Strongly scrubbed

Fig. 3. Pattern regularity function of the cleaning stage for canvas 1 and 2

For both sets of samples, the regularity calculated using the Chetverikov algorithm ([12], [13]) decreases with the intensity of the scrubbing. Before cleaning, canvas 1 possesses a higher regularity than canvas 2. This difference is the reason why gentle scrubbing created a greater decrease in the pattern regularity for canvas 1 than for canvas 2. Furthermore, both canvases see their regularity decrease almost linearly with more abrasion. Both trends are in accord with results observed by visual


41

inspection. Hence, it is possible to conclude that the regularity calculation proposed is suitable to characterize slight variations in pattern regularity. The next step of this study is to explore the variation of texture strength and variation of appearance of the canvas due to the cleaning. 3.2 Effect of Illumination and Cleaning Stage on Canvas Texture Most real surfaces exhibit a non-Lambertian behavior. In fact, the apparent brightness of the surface to an observer is strongly dependent on the position of the illumination (and the position of the viewer). For a highly patterned surface such as a textile, the effect of the illumination on the perceived brightness and pattern must be examined in detail. 3.2.1 Effect on Brightness The various sets of samples were imaged for the 40 positions of illumination. For each position, the average brightness was calculated (corresponding to the mean gray level of the image when converted to gray scale). One possibility to examine the effect of cleaning is to make a polar 3D plot of the variation of luminance according to the position of illumination (see Figure 4).

Fig. 4. Variation of the luminance due to gentle scrubbing for canvas 1 according to the position of illumination

42


The anisotropy of the structure is shown, the mean luminance depending on both the elevation and azimuthal position of the source (the brightness of an isotropic structure would have been invariant to the azimuthal position of the illumination). The lighter part of the graph corresponds to the position of the illumination where the scrubbing increased the brightness, while the darker part corresponds to a loss of luminance. Hence, we can notice a general increase of the mean luminance for high elevation angle from scrubbing, while the mean luminance for grazing angle illumination is decreased. This result has a simple origin. One of the obvious effects of scrubbing is to make the fibers more fuzzy and pilled. As a result, part of the weave structure that was empty becomes filled with fibrils. Consequently, a portion of the light that was traveling through the structure of the untreated canvas is now reflected back to the camera. On the other hand, scrubbing diminishes the height of the weave structure, and therefore the surface becomes flatter, with fewer asperities to reflect light incident at grazing angle. The main limitation of this method is that it does not actually give clues about the variation of texture or perceived texture. To overcome this limitation, the effect of illumination on the GLCM was investigated. 3.2.2 Effect on GLCM Contrast Figure 5 shows the variation of GLCM contrast as a function of the lateral offset for canvas 1 according to the elevation of the illumination. 2

GLCM Contrast

1.6 75°

1.2

60° 45° 30° 20°

0.8

0.4

0 0

10

20

30

Offset (in pixels)

40

50

Fig. 5. GLCM contrast of untreated canvas 1 for various elevation angles for the illumination

By computing the GLCM contrast as a function of the lateral displacement, the weave pattern is exhibited. The offset is chosen in pixels rather than in mm or micrometer for sake of simplicity, but the width of the undulation of the GLCM contrast corresponds precisely to the actual width of the threads. These results also demonstrate that illumination incident near grazing angle (larger elevation angle) tends to enhance the pattern of the canvas texture. For the elevation angle of 20°, very near normal, the undulation of the weave is barely noticeable. These results are again in accordance with the visual perception of the canvas surface. As with brightness, the


43

GLCM contrast is deeply related to the position of the light source. In this context, the full characterization of this contrast was needed to determine the best geometry position to perform the measurement of texture change from cleaning. Figure 6 presents a 3D polar plot of the maximum GLCM contrast as a function of the illumination position.

Fig. 6. Maximum GLCM contrast on untreated canvas 2 according to the position of illumination

The lighter region corresponds to the position of the light source producing the largest values of GLCM contrast. These regions are expected to be more sensitive to the effect of cleaning. This quantification of the effect of illumination allows for the choice of the optimum lighting geometry for detecting texture changes due to cleaning. The figure indicates that the higher values of contrast are obtained for a large elevation angle (75°) and an azimuthal angle of 0° or 180°. However, in order to avoid masking and shadowing effects, we selected an elevation angle slightly less than optimal (60°). Having chosen the lighting geometry for the two canvases, Figure 7 presents the effect of the cleaning on the GLCM contrast for an elevation angle of 60° and the azimuthal position of 0°. The effect of the cleaning on the GLCM contrast is evident. For both canvas 1 and 2 we noticed a sharp decrease of the contrast after gentle scrubbing. Further contrast decrease following stronger scrubbing of the samples was slight. In addition, the undulation characteristics of the pattern are also attenuated due to the cleaning. As a result of the pilling of the threads the pattern is attenuated and the surface seems more

44


GLCM Contrast

0.8

0.4

Untreated Gently scrubbed Strongly scrubbed 0 0

10

20

30

40

50

Offset (in pixels)

Fig. 7. Effect of cleaning on the GLCM contrast as a function of the offset for canvas 1 (top) and canvas 2 (bottom)

homogeneous and has lower contrast. In order to provide a single metric to characterize the contrast variation, which carries the information about the texture pattern, we may consider the maximum amplitude variation of the GLCM contrast (see Figure 8). This parameter is defined as the difference between the maximum and minimum of the GLCM contrast.


45

0.9

Canvas 1 Canvas 2

0.8

GLCM contrast amplitude

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

Untreated

Gently scrubbed

Strongly scrubbed

Fig. 8. Effect of cleaning on the GLCM contrast amplitude variation

Using the statistics obtained from the co-occurrence matrix of images seems promising for quantifying the slight changes in perceived texture that can result from cleaning.

4 Conclusion This study focused on the extraction of textural properties of artworks from analysis of their visual properties. An experimental device was developed to study the effect of the illumination direction. As an example, the effect of surface modification induced by the cleaning of unpainted canvas on the visual perception was investigated. Three metrics were selected to characterize key features of the perceived canvas texture, namely the brightness, the regularity, and the contrast. As the cleaning of the surface increases the fuzziness of the fibers while decreasing its roughness, the observed brightness is modified. The mean luminances of the samples increase for illumination angles close to the normal, although the loss of roughness decreases diffuse reflection and leads to a decrease of the brightness for grazing angle illumination. The regularity calculation showed a progressive decrease for the two canvases with a greater degree of cleaning. The contrast was evaluated using the GLCM contrast and computed for different offset distances. In a first approach, we investigated the effect of the illumination direction on the contrast value. The illumination position giving the maximum contrast was selected. From the images captured under these lighting conditions, it was possible to quantify the effect of the scrubbing on both canvases. It appeared that the contrast dropped dramatically after the gentle scrubbing and then did not decrease significantly with further scrubbing. By combining these three metrics, it is therefore possible to precisely describe the effect of the cleaning on the perception of these surfaces. The application of image analysis tools devoted to art conservation science seems promising. Study of textural variation of particular materials such as stone, wood, or ceramic holds great potential in evaluating the effects of conservation treatments as well as tracking the changes produced during the degradation of object surfaces.

46


References 1. Tuceryan, M., Jain, A.K.: In: Chen, C.H., Pau, L.F., Wang, P.S.P. (eds.) Handbook of Pattern Recognition and Computer Vision, pp. 207–248. World Scientific, Singapore (1998) 2. Jain, A., Robert, P., Mao, J.: Statistical pattern recognition: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 4–37 (2000) 3. Sonka, M., Hlavac, V., Boyle, R.: Image Processing, Analysis, and Machine Vision. Brooks/Cole, Pacific Grove (1998) 4. Motoyoshi, I., Nishida, S., Sharan, L., Adelson, E.H.: Image statistics and perception of surface qualities. Nature 447, 206–209 (2007) 5. Haralick, R.M.: Statistical and structural approaches to texture. Proc. IEEE 67, 786–804 (1979) 6. Ngan, H.Y.T., Pang, G.K.H., Yung, S.P., Ng, M.K.: Wavelet based methods on patterned fabric defect detection. Int. J. Pattern Recogn 38, 559–576 (2005) 7. Lin, H.C., Wang, L.L., Yang, S.N.: Regular-texture image retrieval based on textureprimitive extraction. Image Vis. Comput. 17, 51–63 (1999) 8. Kuo, C.F.J., Su, T.L.: Gray relational analysis for recognizing fabric defects. Textile Res. J. 73, 461–465 (2003) 9. Sandy, C., Norton-Wayne, L., Harwood, R.: The automated inspection of lace using machine vision. Mech. J. 5, 215–231 (1995) 10. Kumar, A., Pang, G.: Defect detection in textured materials using Gabor filters. IEEE Trans. Ind. Applicat. 38(2), 425–440 (2002) 11. Baykal, I.C., Muscedere, R., Jullien, G.A.: On the use of hash functions for defect detection in textures for in-camera web inspection systems. In: Proc. IEEE Int. Symp. Circuits Systems, ISCAS, vol. 5, pp. 665–668 (2002) 12. Chetverikov, D.: Pattern regularity as a visual key. Image Vis. Comput. 18, 975–985 (2000) 13. Chetverikov, D., Hanbury, A.: Finding defects in texture using regularity and local orientation. Pattern Recognit. 35, 2165–2180 (2002) 14. Esmay, F., Griffith, R.: An Investigation of Cleaning Methods for Untreated Wood. Postprints of the Wooden Artifacts Group of the American Institute for Conservation of Historic and Artistic Works, AIC, Washington, DC, pp. 56–64 (2004)

Visual Abstraction with Culture Yang Cai1, David Kaufer1, Emily Hart1, and Yongmei Hu2 1

Carnegie Mellon University, USA 2 Guizhou University, China

Abstract. Visual abstraction enables us to survive in complex visual environments by augmenting critical features with minimal elements – words. In this chapter, we explore how culture and aesthetics impact visual abstraction. Based on everyday life experience and lab experiments, we found that the factors of culture, attention, purpose and aesthetics can help reduce visual communication to a minimal footprint. As we saw with the hollow effect, the more familiar we are with an object, the less information we need to describe it. The Image-Word Mapping Model we have discussed allows us to work toward a general framework of visual abstraction in two directions, images to words and words to images. In this chapter, we present a general framework along with some of the case studies we have undertaken within it. These studies involve exploration into multi-resolution, symbol-number, semantic differentiation, analogical, and cultural emblematization aspects of facial features.

1 Introduction As ubiquitous computing enables sensor webs to become increasingly interconnected, the volume and complexity of information grows exponentially. Information overload becomes a constant problem. To address this overload, there has been an everincreasing demand for intelligent systems to navigate databases, spot anomalies, and extract patterns from seemingly disconnected numbers, words, images and voices. Visual information is a growing contributor to this overload, calling for intelligent systems that can make principled visual abstractions over mountains of visual data. Visual abstraction can be defined as a minimal description of an object, its relationship to other objects or to its dynamics. Visual abstraction must often be guided by culture and aesthetics as well as human psychology. Written language is the ultimate visual abstraction of how human verbal communication mediates mind and culture. If a picture is worth 10,000 words [11], can a word be worth 10,000 images? The answer is yes. Many referential expressions are abstract but still convey visual information. It would take many millions of digital bytes to register the look of disbelief on a human face. It takes a writer only a single referential phrase, “raised eyebrows,” to capture this look through a linguistic abstraction. In our everyday life, we detect, recognize and retrieve images instantly with words. The linguistic retrieval of images dramatically reduces the representational overhead of the communicative transaction. Sometimes simple pictorial similarities between single alphabetic letters (e.g., “T” or “X”) and complex visual images (e.g., a traffic intersection) supply effective visual abstractions as well. The Roman rhetoricians circa 90 BC were the first to formalize Y. Cai (Ed.): Computing with Instinct, LNAI 5897, pp. 47–57, 2011. © Springer-Verlag Berlin Heidelberg 2011

48

Y. Cai et al.

how mnemonic word patterns could serve as prompts to organize one’s memory and retrieve abstractions about prior situations that called for wisdom. They understood how seeding one’s memory with image-rich proverbs, maxims, adages, commonplaces, tropes and figures gives the orator a quick advantage in classifying previously unclassified situations, characters and norms. For better or worse, there are no slimmed-down algorithms to guide us toward optimal visual abstraction. As mentioned above, visual abstraction must often be guided by considerations of culture and aesthetics, matters far beyond the provenance of traditional computer science. By describing a traffic intersection with a letter ‘T’, or ‘X’, we compress a dense image (e.g. 1 megabyte) to a sparse letter (e.g. 1 byte). The plan for the rest of this chapter is to examine in greater detail how context and culture affect visual abstraction, how visual abstraction can be encoded, and the role that technology can play in facilitating visual abstraction, such as face recognition and search over a massive video database.

2 The ‘Hollow Effect’ Most of the information in our daily life is redundant. Studies [20] show that photos normally provide much more information than we need. This redundancy can be as high as ninety percent [15]. In facial communication, dramatic reductions in the spatial resolution of images can be tolerated by viewers [3]. From the point of view of psychology and economics, the level of detail in data communication can be greatly reduced. For example, photos in newspapers normally only have two values for each dot (pixel): with or without ink. With grid screen processing, the size of the smallest pixels is increased so that the number of dots per area can be reduced greatly. However, the picture is still recognizable. Increasing the level of details of the grid screen can make the image more attractive but not more recognizable nor comprehensible.

Fig. 1. We can often recognize everyday objects by their contours

The Russian psychologist Yarbus [28] used an eye tracking system to study the gazing path of the human visual search process. One of his significant discoveries was that human visual searching is guided by a context or purpose. Humans selectively look at things that interest them. Furthermore, humans anticipate things that are

Visual Abstraction with Culture

49

familiar. Fig.1 illustrates examples of daily objects that can be identified easily from their contours. The capacity to recognize these objects instantly depends on the level of experience the viewer has with particular objects. We call this information reduction phenomenon the ‘hollow effect.’ The more we know about the context of the object, the less information we need to recognize it. We conducted experiments to study the robustness of the hollow effect under varying conditions. We designed a lab experiment to understand: 1) the average minimal number of pixels of various images (face, indoors, outdoors, etc.) that subjects require for basic recognition; 2) the effect that guided questions can play in reducing the minimal number of pixels subjects require for face recognition; 3) the effect of age on recognition; and 4) the differences in recognition using black and white or color images. Ten unique images in both color and black and white formats were randomly chosen to cover four picture categories: (1) faces, (2) indoor scenes, (3) outdoors scenes, and (4) complex images, such as oil paintings. These images were also randomly ordered and presented individually at timed intervals by a simple computer program. We hypothesized that, given a set of randomly selected images, those containing human faces would be recognized at smaller resolutions, followed by simple, commonly known objects, and then by more complex indoor and outdoor scenes. Regarding facial recognition, we hypothesized that simple recognition of a generic face would require the least resolution, while gender identification and recognition of a well-known individual (i.e., former President Bill Clinton) would require more pixels. We further hypothesized that the subject’s age would have no effect on required image size, and that an image being in black and white or color would make a negligible difference — though with a slight advantage toward color images. Our initial prompt to subjects was simple. We simply asked subjects, “What is this?” after presenting each of the photos of face-only, indoors, outdoors, figures, and complex scenes (oil paintings). Subjects adjusted the size of the images until they could recognize the image. Facial recognition required significantly fewer pixels than human figures, indoor scenes, and outdoor scenes. As expected, complicated scenes required the largest number of pixels for identification. Finding that generic faces are recognized quickly, we next tested a set of photos of faces. We asked subjects three questions: “Who’s this?” “What is this?” and “Male or female?” respectively. The results show that answering the question “Who’s this?” required the fewest pixels (17 x 17 pixels). Subjects needed more resolution (32 x 32 pixels) to answer “What is this?” They needed even more pixels to identify gender (35 x 35 pixels). To some extent, the number of pixels associated with answers to each of these questions reflects the difficulty of the cognitive task. We found that face recognition needs far fewer pixels than we originally thought, especially if the subject knows what he/she is looking for. These results are consistent with previous findings that show human visual attention fluctuates according to the purposes of seeing, and that demand for visual information resources varies from task to task (Zabrodsky, 1997). When we think about our faces, we realize they are wellstructured compared to other objects. Humans are also well-experienced at identifying and discriminating between faces, which can make generic recognition of faces straightforward. Nonetheless, there are questions about face recognition that remain unsettled, such as whether humans have special “wired connections” to recognize a face.

50

Y. Cai et al.

Despite our experiments uncovering that image recognition can be task and context dependent, our experiments established a general relationship in the order of complexity in which objects are recognized. The order, ranging from fewer to more pixels, is “faces”, “outdoors”, “figure”, “indoors”, and “complex scenes.” Complex scenes, such as impressionist oil paintings, contain more vague objects that confuse viewers and can make recognition challenging even at high pixel resolutions. Our experiments also revealed that a “pixel” can serve as a legitimate numerical measurement of visual information processing. This is a controversial finding. For decades, cognition scientists have relied on other measures: reaction time, number of entities, error rate, and other quantities to measure visual information processing. However, we found that a pixel is a simple way to capture and compute visual processing within a normal human-computer interaction environment. It is simple but not without problems. Because of redundant pixels from extraneous sources, the pixels of an image may not always provide an accurate measure of visual information. To eliminate this problem, we found we had to preprocess the images to eliminate extraneous sources of pixel redundancy. For example, for face recognition tasks, we cut off the background that is outside the face outline. We also used a square image to simplify the measurement. Curiously, subjects needed slightly fewer pixels to recognize things with black and white images than color images. However, those differences were not statistically significant.

3 Image-Word Mapping Model Cognitive scientists have developed models to uncover the relationship between words and images. CaMeRa [21], for example, is a computational model of multiple representations, including imagery, numbers and words. However, the mapping between the words and images is linear and singular, which lacks flexibility. The Artificial Neural Network model, handling this greater flexibility, is proposed to understand images as complex as oil paintings [19], where Solso remarks that the hidden layers of the neural network enable us to map the words and visual features more effectively and efficiently. The claim is that, through hidden layers, we need fewer neurons to represent more images. However, what’s in the hidden layers of the neural network remains shrouded in mystery. Images consist of two- or three-dimensional structures in contrast to language’s one-dimensional construction. Thus, the mapping between words and images is a challenging task. In order for this one-dimensional construction to be able to work in tandem with the visual field, it must maintain the ability to go beyond one-dimensionality. Arnheim asserts that, through abstraction, language categorizes objects. Yet it is language that permits humans to see beyond mere shape [2]. This is not because language moves us closer to the physics of the world, but because language accommodates human interests and purposes by way of categorizing the physical world according to human affordances. Pinker [2] has explored these human affordance features of language in great depth. He notes that English conceives of water lines as a 2D boundary (over water, underwater) within a 3D space in part because the 2D difference matters to human survival. Surfaces can be presented as 1D, 2D, or 3D, depending on how we want to focus on motion relative to them. When we want to emphasize trajectory and direction without texture or resistance, we can


51

describe the surface as one-dimensional, with a 1D referent, like path. We can say, “he walked along a path through the field” to indicate a linear trajectory. The preposition “along” cues a 1D trajectory. Should we want to emphasize the 2D texture of the field, we can reference the field directly with the preposition “across” – “she walked across the field.” If we want to emphasize resistance through a 3D space, we can say, “he walked through the field,” suggesting some stepping, some stomping, and some resistant brush. A man described as walking on or over the snow is understood to have an easier walk than one described as walking through the snow. As Pinker [16] characterizes it, the tense system is designed to help humans segment episodic time between that which is knowable and experienced (the past), that which is unchartered (the future), and that which is emergent and unclassified (the present). He shows that the language of causation is associated with what is foreseeable, over which one takes responsibility. Pinker holds that language provides affordable access to the needs of human categorization because it evolved that way. A language apparatus that could serve human classification needs and interests would, according to Pinker, have evolutionary advantages over language systems that could not. Pinker’s observations complement our own project, which is to study how far we can get by viewing language as a computational device for abstracting key features of non-linguistic types of information. By virtue of language, humans are inherently trained to go beyond object shape and explore further textures, dimensions, and sub-shapes; these further explorations seem to be the only method we have to satisfactorily describe a human subject. Along these lines, Roy developed a computerized system known as Describer that learns to generate contextualized spoken descriptions of objects in visual scenes [17]. Roy’s work illustrates how a description database can be useful when paired with images in constructing a composite image. Roy’s findings suggest that significant variation in the language used to describe an image results in significant variations in the images retrieved. Such variation can be reduced by organizing more constrained words and phrasal patterns into the computer as input. Roy’s work nicely illustrates how words are abstractions of images and images are extensions of words.

4 Descriptions for Humans In our project, we have focused on various aspects of the mapping between words and images for human features. For the rest of this paper, we describe these studies. We were intrigued by the rich mapping between words and faces both because of the cultural importance of faces and because facial descriptions are encoded in the literatures of the world. So comprehensively have faces been described in world literatures that language references have been compiled to record the various ways in which human faces can be rendered. For example, the Description Dictionary is a collection of descriptive words and phrases about human features from literatures from around the world. 4.1 Multiple Resolution Descriptions Human descriptions are classifiers for shape, color, texture, proportion, size and dynamics in multiple resolutions. For example, one may start to describe a person’s

52

Y. Cai et al.

figure, then hairstyle, face, eyes, nose, and mouth. Human feature descriptions have a common hierarchic structure. For example, figure, head, face, eye, et al. Like a painter, verbal descriptions can be built in multiple resolutions. The words may start with a coarse description and then ‘zoom’ into finer-grained sub-components. We have collected over 100 entries of multi-resolution descriptions from literature. Due to the limitation of space, we only enlist a few samples, where the underlined sections represent the global levels of description, and the bolded show the component-based descriptions. The italicized sections are the details: • •

•

“For a lean face, pitted and scarred, very thick black eyebrows and carbon-black eyes with deep grainy circles of black under them. A heavy five o’clock shadow. But the skin under all was pale and unhealthy-looking [6]”. “Otto has a face like very ripe peach. His hair is fair and thick, growing low on his forehead. He has small sparkling eyes, full of naughtiness, and a wide, disarming grin which is too innocent to be true. When he grins, two large dimples appear in his peach blossom cheeks [30]”. “Webb is the oldest man of their regular foursome, fifty and then some- a lean thoughtful gentleman in roofing and siding contracting and supply with a calming gravel voice, his long face broken into longitudinal strips by creases and his hazel eyes almost lost under an amber tangle of eyebrows [23]”.

4.2 Semantic Differential Representation The Semantic Differential method measures perceptual and cognitive states in numbers or words. For example, the feeling of pain can be expressed with adjectives, ranging from weakest to strongest. Figure 2 shows a chart of visual, numerical and verbal expressions of pain in hospitals: No Hurt (0), Hurts Little Bit (2), Hurts Little More (4), Hurts Even More (6), Hurts Whole Lot (8) and Hurts Worst (10).

Fig. 2. Expressions of pain in pictures, numbers and words

The physical feeling can be quantified with mathematical models. When the change of stimulus (I) is very small, we won’t detect the change. The minimal difference (ΔI) at which sensation is just noticeable is called perceptual threshold and it depends on the initial stimulus strength I. At a broad range, the normalized perceptual threshold is a constant, ΔI/I = K. This is Weber’s Law. Given the perceptual strength E, as the stimulus I changes by ΔI, the change of E is ΔE. We have the relationship ΔE = K*ΔI/I. Let ΔI be dI and ΔE be dE, thus we have Weber-Fechner’s Law:


E = K * ln(I) + C

53

(1)

where C is constant. K is Weber Ratio. I is stimulus strength and E is perceptual strength. Weber-Fechner’s Law states that the relationship between our perceptual strength and stimulus strength is a logarithmic function. People remember better with cartoon-like figures because cartoons show exaggerated features. This result appears to be an application of Weber’s Law running instinctively in our visual memory. 4.3 Symbol-Number Descriptions In many cases, numbers can be added to give more granulites. For example, the FBI’s Facial Identification Handbook [9] comes with a class name, such as bulging eyes, and then a number to identify specific levels and types. The FBI has created a manual for witnesses, victims, or other suspect observers to use in identifying possible suspect features. The Catalog presents several images per page under a category such as “bulging eyes”; each image in such a category has bulging eyes as a feature, and the respondent is asked to identify which image has bulging eyes that most closely resemble the suspect’s. This book is an extremely efficient and effective tool for both forensic sketch artists and police detectives. It is most commonly used as to help a witness or victim convey the features of the suspect to the sketch artist in order to render an accurate composite sketch. 4.4 Analogical Descriptions Analogy is a coarse descriptor. Instead of describing features directly, people often refer to a feature through a stereotype, for example, a movie star’s face. The analogical mapping includes structural mapping (e.g. face to face), or component mapping (e.g. Lincoln’s ear and Washington’s nose). Children often use familiar things to describe a person, for example, using ‘cookie’ to describe a round face.

Fig. 3. Analogical description of noses and face type in Chinese

Analogies are culture-based. In the Western world, nose stereotypes are named according to historical figures. Many analogies are also from animal noses or plants. Fig. 3 illustrates examples of the nose profiles as described above. We use simple line drawings to render the visual presentation. Analogy is triggered by experience, which involves not only images, but also dynamics. The third sketch in Fig. 3 shows a ‘volcano nose’, which triggers readers’ physical experiences, such as pain, eruption, and

54

Y. Cai et al.

explosion. In this case, readers not only experience these senses, but also predict the consequences. Therefore, this is an analogy of a novel physical process that remains below the visible surface. Given a verbal description of the nose, how do we visually reconstruct that nose profile with minimal elements? In this study, we use a set of 5 to 9 ‘control points’ to draw a profile. By adjusting the relative positions of the control points, we can reconstruct many stereotypes of the nose profiles, and many other points in between. To smooth the profile contour, we apply the Spline curve-fitting model [26]. The cultural basis of analogy leads people to improvise analogies from daily objects. Similarly, different cultures build analogies from objects that are salient to them. For example, Chinese use an ‘onion’ shape to describe short snub noses and a ‘sunflower seed’ shape to describe slim faces. Chinese language has evolved from a pictorial language. Today, many Chinese characters are still used to describe people’s face shapes. Based on similarities between the face shapes and Chinese character shapes, face shapes are divided into eight types: tian (field), you (due), gui (country), yong (use), mu (eye), jia (jia), feng (wind), shen (apply). The last three sketches in Fig. 3 show the characters and correspondent shapes [14]. All languages are symbols. Understanding the cultural influences on analogical descriptions enables us to unleash the power of languages for visual abstraction. It took hundreds of years for the world to use Arabic numerical symbols. With crosscontinental telegraphy, it took days for the world to accept Winston Churchill’s V-sign. Networked computing is increasingly helping people share analogical descriptions across culture barriers. The unique descriptions from Chinese or Indian cultures can conceivably be used in other parts of the world, in unexpected applications, such as video search engines and in archeological reconstruction.

5 Decontextualization, Recontextualization and Emblemization Facial images can reveal much about psychological states but they do so imperfectly and with much error. In order to over-determine the accuracy of what facial images mean for the description of mood, the media often relies on a process called decontextualization. Decontextualization means to take an image out of context and use that image to reinforce verbal interpretations of a person’s internal state in contexts where there is less direct support. The coverage of Hillary Clinton provides an excellent example of this media phenomenon. From August 1998 through June 2000, NBC news ran ten broadcasts of facial images of Hillary Clinton from a memorial service honoring American victims of terrorist bombings where her face is shown in isolated profile with a shiny eye that looks tear-stained. For nine of these broadcasts, NBC never mentioned that the images came from a memorial service and instead “recontextualized” the image to offer support for their verbal commentary on two unrelated matters: the “strained” state of her marriage with Bill Clinton and the “pressure” she was under to decide whether or not to run for the Senate in 2000. During the release of the Starr Report hearings, NBC consistently used the image in profile to emblematize Hillary Clinton’s personal anguish as a betrayed spouse. During her Senate run, NBC hauled out the same image in profile to emblematize the pressure she was under to decide on a Senate run.


55

Such uses of de-contextualization, re-contextualization and emblematization to create visual “mood bites” are ubiquitous in the media and add weight to undersubstantiated verbal descriptions of psychological states. Yet why “mood bites” work and the ethical boundaries between fair representation and distortion are still poorly understood. Based on the facial imaging environments we have created for other tasks, we will create a simulated user-controlled “news broadcast” that allows a user to manipulate the following two parameters along a continuous scale: the visibility of decontextualization (if de-contextualization becomes too visible, the image will seem “staged” and lose contextual credibility); and the exact placement in the verbal voiceover of where the image is shown. The idea is that, the longer the image is shown across multiple clauses of a verbal voiceover, the more the image is used to capture a general mood rather than a specific point. By way of contrast, when the image is displayed just over a specific clause and then removed, it functions more definitely and emblematically as evidence for a definite point expressed in the clause. By letting users manipulate these parameters on a stock news story, then fill out a survey covering the “information value” and “fairness” of the visual display, we believe we can make significant progress in understanding the role and excesses of the de-contextualized visual in media.

6 Interactive Facial Reconstruction We developed a prototype of the interactive system for facial reconstruction on a computer. In the system, a user selects the feature keywords in a hierarchical structure. The computer responds to the selected keyword with a pool of candidates that are coded with labels and numbers. Once a candidate is selected, the computer will superimpose the components together and reconstruct the face. As we know, a composite sketch of a suspect is often done by professionals. Our system enables inexperienced users to reconstruct a face with a menu-driven interaction. In contrast, the reconstruction process is reversible, so it can be used for facial description studies, robotic vision and professional training.

Fig. 4. Interactive front facial reconstruction based on image components

56

Y. Cai et al.

7 Conclusions In this chapter, we have explored the impacts of culture and aesthetics on visual abstraction. Based on everyday life experiences and lab experiments, we found that the factors of culture, attention, purpose, and aesthetics can help reduce visual communication to a minimal footprint. As we saw with the hollow effect, the more we are familiar with an object, the less information we need to describe it. The Image-Word Mapping Model we have discussed allows us to work toward a general framework of visual abstraction in two directions, images to words and words to images. In this chapter, we have simply overviewed that general framework and presented some of the small studies we have undertaken within it. These studies involve exploration into multi-resolution, symbol-number, semantic differentiation, analogical and cultural emblematization aspects of facial features. None of the studies we conducted could have been started and completed solely through algorithms. Matters of context and culture reared their head at every turn. Despite some substantial progress, this research remains at an early phase. We have successfully identified many puzzle pieces teaching us how visual abstraction works through language and other mediators. But we have yet to find all the pieces we need even to understand the larger puzzle frame. We need the puzzle frame to understand how best to fit together the pieces we have identified and to learn about the pieces still hidden from view.

Acknowledgement We are also in debt to Emily Durbin and Brian Zeleznik for their assistance.

References 1. Allport, A.: Visual Attention. MIT Press, Cambridge (1993) 2. Arnham, R.: Visual Thinking. University of California Press, Berkeley (1969) 3. Bruce, V.: The Role of the Face in Communication: Implications for video phone design. Interaction with Computers 8(2), 166–176 (1996) 4. Cai, Y.: How Many Pixels Do We Need to See Things? In: Sloot, P.M.A., et al. (eds.) ICCS 2003. LNCS, vol. 2657. Springer, Heidelberg (2003) 5. Chen, J.L., Stockman, G.C., Rao, K.: Recovering and Tracking Pose of Curved 3D Objects from 2D Images. In: Proceedings of IEEE Comput. Vis. Patt. Rec. Publisher, New York (1993) 6. Doctorow, E.L.: Loon Lake. Random House, New York (1980) 7. Duchowski, A.T., et al.: Gaze-Contingent Displays: A Review, Cyber-Psychology and Behavior (2004) 8. Fan, T.J., Medioni, G., Nevatia, R.: Recognizing 3-D Objects Using Surface Descriptions. IEEE Trans. Patt. Anal. Mach. Intell. 11, 1140–1157 (1989) 9. FBI Facial Identification Catalog (November 1988) 10. Larkin, J.H., Simon, H.A.: Why a Diagram Is (Sometimes) Worth 10,000 Words. Cognitive Science 11, 65–100 (1987) 11. Lowe, D.G.: The Viewpoint Consistency Constraint. Int. J. Comput. Vision 1(1), 57–72 (1987)


57

12. Luo, J., et al.: Pictures Are not Taken in a Vacuum. IEEE Signal Processing Magazine (March 2006) 13. Majaranta, P., Raiha, K.J.: Twenty Years of Eye Typing: Systems and Design Issues. In: Eye Tracking Research and Applications (ETRA) Symposium. ACM, New Orleans (2002) 14. Mou, F., Li, Z.: Modern Surgery and Techniques. New Time Press, Beijing (2003) ISBN 7-5042-0851-5 15. Petersik, J.T.: The Detection of Stimuli Rotating in Depth Amid Linear Motion and Rotation Distractors. Vision Research 36(15), 2271–2281 (1996) 16. Pinker, S.: The Stuff of Thought: Language as a Window into Human Nature. Viking Press, New York (2007) 17. Roy, D.: Learning from Sights and Sounds: A Computational Model. Dissertation. Media Arts and Sciences. MIT, Cambridge (1999) 18. Silberberg, T.M., Davis, L.S., Harwood, D.A.: An Iterative Hough Procedure for ThreeDimensional Object Recognition. Pattern Recognition 17(6), 621–629 (1984) 19. Solso, R.L.: Cognition and the Visual Arts. MIT Press, Cambridge (1993) 20. Stroebel, L.D., et al.: Visual Concepts for Photographers. Focal Press, New York (1980) 21. Tabachneck-Schijf, H.J.M., Leonardo, A.M., Simon, H.A.: CaMeRa: A Computa-tional Model of Multiple Representations. Cognitive Science 21, 305–350 (1997) 22. Ullman, S., Basri, R.: Recognition by Linear Combinations of Models. IEEE Trans. Patt. Anal. Mach. Intell. 13, 992–1006 (1991) 23. Updike, J.: Rabbit is Rich. Ballantine Books, New York (1996) 24. Viola, P., Jones, M.: Rapid Object Detection Using a Boosted Cascade of Simple Features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, December 8-14, vol. 1, pp. 511–518. IEEE Computer Society Press, New York (2001) 25. Chiariglione, http://www.chiariglione.org/MPEG/standards/mpeg-7/ mpeg-7.htm 26. Wikipedia (2007), http://en.wikipedia.org/wiki/Spline_mathematics 27. Web Exhibits, http://www.webexhibits.org/colorart/ag.html 28. Yarbus, A.L.: Eye Movements during Perception of Complex Objects. Plenum, New York (1967) 29. Zabrodsky, R., Peleg, S., Avnir, D.: Symmetry as a Continuous Feature. IEEE Trans. Pattern Analysis and Machine Intelligence 17(12), 1154–1166 (1995) 30. Isherwood, C.: Berlin Stories. Random House (1952)

Genre and Instinct Yongmei Hu1, David Kaufer2, and Suguru Ishizaki2 1 2

College of Foreign Languages, Guizhou University Department of English, Carnegie Mellon University

Abstract. A dominant trend relates written genres (e.g., narrative, information, description, argument) to cultural situations. To learn a genre is to learn the cultural situations that support it. This dominant thinking overlooks aspects of genre based in lexical clusters that appear instinctual and cross-cultural. In this chapter, we present a theory of lexical clusters associated with critical communication instincts. We show how these instincts aggregate to support a substrate of English genres. To test the cross-cultural validity of these clusters, we gave three English-language genre assignments to Chinese students in rural China, with limited exposure to native English and native English cultural situations. Despite their limited exposure to English genres, students were able to write English papers that exploited the different clusters in ways consistent with native writers of English. We conclude that lexical knowledge supporting communication instincts plays a vital role in genre development.

1 Introduction This chapter examines the learning of written genres (e.g., narrative, information, description, argument) as partly an instinctual process, in the manner of Chomsky’s innateness hypothesis for language learning. Pinker [1], defending that hypothesis, challenged various assumptions about language education and the learning of writing in school, particularly at the sentence level. Pinker argued that descriptive grammar was innate and “instinctual” and developed independently of experience, schooling, or even learning. Descriptive grammar, according to Pinker, provides a native substrate for the more superficial and culturally dependent prescriptive grammar taught and learned in school. Much prescriptive grammar, Pinker argued, is empirically suspect, and, even when not, at best marginally improves on a writer’s innate skill rather than embodies or essentializes that skill. This chapter takes up the question of whether the notion of written genre might itself be based on universal “instincts” behind the formulation of “lexical clusters” and aggregations of these clusters to construct “reader experience,” or whether genre depends on rule-based learning subject to cultural variation. Stating the matter as an all or nothing “either-or” between instinct and rule is almost certainly, in our view, too strong. Pinker’s (and Chomsky’s) innateness hypothesis remains controversial and the subject of much discussion [2]. In addition, the notion of “instinct” is itself ambiguous, ranging between a strict notion of “closed instinct,” which polarizes instinct and experience, and a notion of “open instinct,” which fashions instinct and learned experience into permeable concepts, allowing that Y. Cai (Ed.): Computing with Instinct, LNAI 5897, pp. 58–81, 2011. © Springer-Verlag Berlin Heidelberg 2011

Genre and Instinct

59

instincts are not learned but better honed through learning and experience [3]. We adopt the latter position that written genres are largely based on open instincts about lexical clusters and the way they co-occur to produce reader experience. This means that we acknowledge that knowledge of written genres develops in part along with writing experience and an increasingly refined knowledge of the cultural situations that genres support. But it also means that the teaching and learning of genres from a cultural perspective can improve the writer’s performance, at best, only at the margins. The dominant North American pedagogies that cover genre emphasize the cultural determinants of genre, with genre variation largely explained as a manifestation of variation in cultural situations [4-7]. This cultural emphasis toward genre is well-motivated and, for the most part, well-executed. However, it has overlooked its own limitations by ignoring or leaving understudied aspects of genre knowledge that work instinctually and remain independent of one’s cultural positioning or experience. In this chapter, we review what the cultural-based pedagogies ignore. We also show what the cultural-based pedagogies would not predict, namely that second language writers with limited exposure to cultural situations in English show key rudiments of genre knowledge by virtue of the basic English fluency they claim. The driving engine of genre performance, in our view, is the deployment of open instincts about lexical clusters in language and the way these clusters aggregate to produce reader experience. These lexical clusters and their aggregation potential are implicit and never formally learned nor taught either to first language or secondlanguage writers. Yet, these clusters and aggregation mechanisms seem available to first or second language writers as they acquire basic fluency in a language. In the next section, we describe the work we have done to identify these instinctual clusters. In the section following that, we describe some basic aggregation rules that help define reader experience across various genres. In the section after that, we show how Chinese students are able to reproduce some major genre differentiations, despite being asked to do so in an impromptu setting, with no prior notification or instruction, with no formal schooling in English language genres and undergoing limited exposure to the cultural situations of the native English speaker.

2 Clusters of Lexical Experience Constituting Genre In this section, we briefly run through seventeen major clusters of lexical experience constituting genre. A monograph length treatment of these clusters in slightly different form is available [8]. 2.1 First Person Reference/Attribution This cluster allows for self-reference and self-attribution through the use of first person singular pronouns (e.g., I, me, my, myself). This cluster permits authors or characters to express their own involvement in texts.

60

Y. Hu, D. Kaufer, and S. Ishizaki

2.2 Third Person Reference/Attribution This cluster shifts the perspective to the third person through pronouns like “he” and “she” with various predicates indicating the involvement of third parties. This cluster stresses pronouns more than adjective + noun combinations (e.g., the old man), because the latter may be a one-shot reference; whereas the interest of third-person attribution to whole text experience is when the third person reference persists [9] to create a sense of narrative experience about a character. A succession of co-referential third person pronouns is correlated with narrative experience. Moreover, narrative experience is much more a trademark of fiction than non-fiction texts. If one does a statistical tabulation of third person pronouns (e.g. he, she) across the Brown Corpus, as we did, one will find a statistically significant higher proportion in fiction over non-fiction entries. 2.3 First Person Private Register This cluster of words creates the experience of a “private” register, a first person narrator giving voice to thoughts that are not normally publicly known or circulated. This cluster is created by aligning words of self-reference to private cognition verbs indicating personal thought or feeling (e.g., I think, I feel, I believe); or reluctance (e.g., I regret that, I am sorry that, I'm afraid that). An autobiographical effect is created with the use of the aspectual “have” or “used to” (e.g., I have, I used to). 2.4 Generalized Private Register This cluster deepens private experience from the first or third person perspective and can be conveyed through free floating cognitive verbs (e.g., believe, feel, conjecture, speculate, pray for, hallucinate); verbs and adverbs expressing subjectivity (e.g., seems, seemingly, tentative, one way to think about), subjective time (e.g., seems like only yesterday, in a flash), markers of epistemic stance such as confidence (e.g., completely true, assuredly so) and uncertainty (e.g., maybe, perhaps). This cluster is also activated through the use of confessional verbs (e.g., confess, acknowledge that, admit, let on that, let it slip that), adjectives and adverbs expressing intensity, (e.g., very, fabulously, really, torrid, intensely, amazingly), temporal adverbs expressing immediacy (e.g., right now, now, just then). 2.5 Narrative This cluster marks the development of a story unfolding one event after another in time. It becomes active through the use of verbs that convey witnessed action (e.g., came, saw, conquered), temporal clauses (when she was young), temporal verb phrases (would never again) temporal prepositional phrases (in her youth) denoting biographical time and time intervals (for two years, over the last month), temporal adverbs indicating time shifts (e.g., next week, next month) and time/date information (e.g., on June 5, 2000).

Genre and Instinct

61

2.6 Interactivity This cluster indicates interactivity between the writer and reader or between characters in a text. It is activated by a multitude of English subclusters, such as curiosity raising, stimulating another mind to formulate questions arising along a common path of inquiry (e.g., what then shall we make of?); requests (e.g., request, ask of); direct address, the mental equivalent of hailing or summoning another’s attention (e.g., let us, I advise you, I urge, I urge you to, I recommend, I favor) or making implicit acknowledgement of an interlocutor (e.g., you, trust me, okay now); questions (Wh-questions – what do you think? how may I help?); quantitatively oriented survey questions (e.g., how often do you; with what frequency do you?)and conventional follow-ups (e.g., in response to your latest memo, per your last message). This cluster also involves grounded feedback of various sorts (e.g., okay, good, yes, go on, alright). 2.7 Reasoning This cluster denotes paths of inquiry that the writer and reader share and traverse together under the writer’s direction. Subclusters consist of constructive reasoning that builds paths either in a forward direction (premise-conclusion) (e.g., thus, therefore) or a backward direction (conclusion-premise) (e.g. because, owing to the fact, on the grounds that, for the reason that). Constructive reasoning also includes initiating reasoning to launch an inquiry (e.g., suppose that, imagine that) and legitimating the link between premise and conclusion (e.g., as evidence for, in support of). A second subcluster is reasoning under contingency (if…then, it depends). A third is oppositional reasoning, which is used not to construct the reasoning of oneself but to acknowledge or block the reasoning of another. Oppositional reasoning itself consists of subclusters, such as denials/disclaimers (e.g., not the case, not do it, am not a crook), concessives (e.g., although, even if, it must be acknowledged), and resistance, which evokes the tension between competing ideas, events, forces, or groups (e.g., resistant, veto, counterargument, filibuster, went into battle against). 2.8 Emotional Valence This cluster indicates emotionally tinged language, expressing positive (e.g., wonderful, marvelous, great, fantastic) and negative emotion (e.g., mistreatment, criminal, despised). Our investigations suggest that negative emotion is more elaborated in English than positive emotions and that negative emotions can be subcategorized into large subclusters of anger (e.g., irritable, rancorous, indignant), fear (afraid, scared, worried), and sadness (inconsolable, distraught, sorrow, depressed). 2.9 Interpersonal Relations This cluster structures interpersonal relationships between texts, again between author and reader or between characters in a text. The relationships can be primarily positive, such as promising (e.g., I promise), reassuring (e.g., don't worry about), reinforcing (e.g., good job), acknowledging (e.g., thank you), and inclusive (e.g. let us all work as a team). There are further negative relationships, primarily threats and confrontations (e.g., What were you thinking of? I'll kick your butt).

62


2.10 Past Orientation This cluster is active when the writing references a time prior to the time the writing was created. The narrative cluster (1.5 above) also references time past. However, this cluster contains structures of English that convey past orientation that are not on the main event path of a narrative flow of events, but that can elaborate that flow or that can appear in a text as a stand-alone past reference. These include non-narrative expressions that signal a mental leap to the past (e.g., used to, have been, had always wanted, would have liked to), or a future-in-the-past that adds critical perspective to a narrative (e.g., Lincoln was to look for the general who could win the war for him; Mom was to be home by now). 2.11 Future Orientation This cluster is active when the text orientation makes a casual reference to the future (e.g., in order to, look forward to, will be in New York) or actively predicts the future (e.g., By 2020, the social security fund will be bankrupt). As we will see below, one of the genres that makes most active use of this cluster are instructions. In this genre, the future is regularly used to guide audiences by telling them what they will see if they take correct or incorrect actions (e.g., turn right at the corner and you will see a gas station). 2.12 Description This cluster is active when the text refers to input from the five senses: the sights, sounds, aromas, touches ("the warm embrace") and tastes ("the salty crunch of bacon") of experience. Description conjures concrete spaces, scenes, and objects with lively, colorful properties in the reader’s mind. It also includes words that reference visible objects (e.g., table, chair), spatial relations (e.g., near to, seated with), motions (e.g., run, jump), dialogue cues (e.g., “….,” he barked.) and oral speech features (e.g., uh huh). 2.13 Public Register This cluster helps constitute important aspects of public or institutional genres, namely words and phrases that reference institutions (e.g., judicial, electoral, senatorial) and groups and roles authorized by them (republicans, democrats, chairmen, official). There are many subclusters, including words invoking institutions, groups, and roles functioning as commonplace authorities (e.g., founding fathers, Emirs, parliamentary course of action, duly authorized), precedents (e.g. widely accepted), received points of view (e.g., some hold that; others maintain that), or with established thought that is now being rehearsed or confirmed (e.g., I recognize that, I agree with). 2.14 Values This cluster, often aggregating with public registers, is active when public values are invoked. The values may be positive (e.g., justice, happiness, fairness, the good) with one strand of positive values referencing innovation (e.g., breakthrough, cutting-edge,

Genre and Instinct

63

state-of-the-art). They may also reference values that are negative (e.g., injustice, unhappiness, unfairness, bad). 2.15 Information This cluster, also often aggregating with public registers, includes methods of exposition that indicate traversal of an information space. Unlike narrative structures, which move temporally from event to event, information is structured around hierarchically organized “points” and “sub-points.” Subclusters include generalization (e.g., all, every), examples and illustrations (e.g., for example, to illustrate), comparison (e.g., more than, fewer than), resemblance (e.g., resembles, looks like), specification (e.g., in particular, more specifically), exceptions (e.g., an exception, the only one to, sole), and definition (e.g., is defined as, the meaning of the term). 2.16 Reportage This cluster, often co-occurring with information structures, is active when verbs are used to report states of affairs (e.g., is carried by, is housed in), events (e.g., established, instituted, influenced), processes (e.g., make, change, transform), updates (e.g., the latest, late-breaking, announcing a new) and sequences (e.g., first, second, third). Reportage verbs differ from narrative verbs mainly by the register with which they are associated. Narrative verbs most strongly correlate with private registers, reporting verbs with public registers. 2.17 Directives Directive clusters are active when the reader of a text or a character within the text is directed to believe or do something. Subclusters include imperatives (e.g., come to; stop what you are doing); procedures, (e.g., fold on the dotted line, do not bend, fold, or mutilate), moving the body (e.g., clasp, grab, twist, lift up); error-recovery (e.g., should you get lost...), and insistence (e.g., you need to come; you need to consider). Hallmarks of insistence are also conveyed in the modals "must," "should," "need," and "ought.”

3 Clusters Composed into Prototype Genres We were able to treat our 17 clusters as multivariate objects that aggregate and divide to create seven prototype genres. [10] By prototype genres, we mean genres that form a deep lexicon providing building blocks of many historically specific genres. 3.1 Self-portraits Self-portraits underlie English genres of self-expression, ranging from diaries and personal journals to autobiographies, first person blogs, and cover letters. Selfportraits tend to aggregate First Person Reference/Attribution [cluster 1.1], First Person Private Register [cluster 1.3], Generalized Private Registers [cluster 1.4] and Narrative [cluster 1.5]. When used for business cover letters, self-portraits tend to

64


aggregate with Public Registers [cluster 1.13] and clusters variously associated with public registers [clusters 1.15 -1.16, namely information and reportage]. The writer of the business cover letter is constructing an image of him or herself, but that image is narrowly constrained to employment interests and trajectories. 3.2 Observer Portraits Observer portraits, such as bibliography, third person memoir, and personal and professional profiles, underlie English genres. These portraits range from the lighthearted and subjective profile of a teen to the staid and consequential profile of a CEO for an annual report or a candidate for elective office. Observer portraits take as their foundation third person reference/attribution [cluster 1.2], description [1.12] and values [cluster 1.14] as their foundation. The genre, at base, describes another person and his or her values from the third person perspective. Observer portraits also tend to employ a generalized personal register [cluster 1.4] and narrative [cluster 1.5] if the portrait is personal, and a public register and its associate clusters [information, and reportage] clusters 1.15-1.16] if the profile is public. Like self-portraits, observer portraits can be literary and range across all aspects of life in and outside of a professional resume. However, observer portraits can also be professionally focused (a CEO profile) and limited in the observations they make about the subject of the portrait. 3.3 Scenic Writing Scenic writing underlies sensory writing found in poetry, literature, and fiction writing for the leisure market. It is also a component of genres where visual information is necessary and not just nice (e.g. observer portraits, field writing among geographers and anthropologists, science writing for lay audiences, and instructions). Further, scenic writing enhances the exposition in any form of writing where the visual is an optional dimension. Scenic writing relies on description [cluster 1.2] and narrative [cluster 1.5], where the challenge is to tell stories that naturally emerge from the close observation of spaces. Scenic writing often requires the writer’s discipline to remain within description, so as to keep the reader as an eyewitness to what the writer sees and hears, rather than what the writer thinks. 3.4 Narrative History Narrative history underlies the genres of history, biography, autobiography, and literature told from a narrative perspective. The key clusters are narrative [cluster 1.5], a past orientation [cluster 1.10] and interpersonal relations [cluster 1.9], as most writing with a narrative basis deploys shifting input about interpersonal relationships as a way of keeping the reader involved. The narrative may remain within the private register [1.4] as a piece of personal history, or within the public register [cluster 1.13], as a piece of institutional history. It may also be launched from the first [cluster 1.1] or third person [cluster 1.2]. Narrative histories seek to help a reader recover a world that is no more, with the vividness of yesterday.

Genre and Instinct

65

3.5 Information Information writing underlies all genres that seek to supervise learning for a reader. Unlike experiential writing in the leisure market, where the reader’s learning is unsupervised, information writing tends to contractually lay out for the reader (through purpose statements and other devices) what to learn. The reader is a client of the writing and not simply a patron. Unsurprisingly, the information cluster [cluster 1.15] is the fundamental cluster driving information genres; but information genres also tend to rely a great deal on interactivity [cluster 1.6] in order to establish a relationship of trust and personal connection with the reader as learner. Information genres tend to rely on public rather than private registers and so, when the information is event driven, it will rely on reportage [cluster 1.16] – and sequence if the events reported occur in sequence; and it will turn to narrative [cluster 1.5] only if personal stories are seen as a good way to illustrate the main points being conveyed. What distinguishes information writing is that the reader is promised some durable learning (e.g., “points to retain”) meant to survive the reading experience itself. 3.6 Instruction Instructional genres are evident in product instructions, procedures, regulatory forms, and any other document seeking to build manual skill or to achieve compliance. The writing underlying these genres depends upon directives [cluster 1.17] as a way to guide the reader through a manual task or a task space of requirements and constraints. Instructional writing also tends to be dependent upon spatial description [cluster 1.12] and even a future orientation [cluster 1.11] insofar as readers are often instructed how the future will appear (e.g., you will see) as a way of offering visual confirmation of correct action to the instruction taker. Instructions must accommodate the reader’s interest in solving spatial problems or achieving compliance with regulations. 3.7 Argument Argument writing underlies persuasive writing across the disciplines, the professions, and civic life, from legal briefs and appellate decisions to academic journal articles, petitions, memoranda of professional lobbyists, candidate press releases and letters to the editor. A hallmark of such writing is a focus on reasoning [cluster 1.7] along with a focus on a public register [cluster 1.13] and values [cluster 1.14]. In truth, argument is a situational and dynamic genre, meaning that the writer anticipates what clusters are most needed to break down the resistance to the message being offered and then deploys those clusters. In this sense, no cluster is out of bounds when it comes to argument.

4 Genre and Instinct The lexical clusters elaborated above are never explicitly taught in school, neither to native nor non-native speakers. They seem to be part of the natural equipment of using language in the oral register prior to schooling. They seem, as we have asserted,

66


more the product of instinct than rule. Moreover, because these clusters combine in various combinations to support genre, genre itself seems to have an instinctual inheritance in language apart from convention or culture. Genre [5] is a recognized category of writing that shares a common form, purpose or content. Student writing classes and assessment protocols are often organized into genre tasks. A common assessment genre in the United States is the five paragraph essay [11] and in China the two or three paragraph essay [12]. Yet, however formularized a genre may become for evaluation purposes, it defies exhaustive specification through rules, and students who typically construe the task as nothing more than applying and memorizing rules do poorly. Formulas can help writers adjust to the overwhelming demands of an assignment. They can be a good place for writers to start, but writers would not be writers if formulas were a good place to end. Knowing we are involved in one genre or another as a reader or writer is an instinctual judgment that eludes formulas. We have no trouble recognizing the discrete sentences of a detective novel or cookbook, but we have a much harder time describing the different experiences in which these sentences are embedded. This instinctual aspect of genre makes it especially challenging for evaluators of student writing to make assessments like “the student’s writing is appropriate to the writing task.” Insofar as the writing task is associated with an evaluation of genre appropriateness, this criteria calls upon two instinctual judgments – the genre under evaluation and the achievement of appropriateness within this genre through the discrete actions of the student writer. The writing assessor may think the student has hit or missed the bulls-eye by either a little or a lot; but, in either case, both the target to be hit (the genre) and the extent to which it is hit (appropriateness) at least begin in gut-level judgments that may never move beyond that. Our instincts about a text’s appropriateness or genre (or both) are difficult for serial readers to elaborate on because serial readers pay inordinate attention to the text segments they are currently processing [10], whereas analyses of genre must take into account the collective weight of the whole text at once. Instinctual judgment for the serial reader of text is not necessarily immediate judgment. Often, a reader must sample widely across a text to determine if his or her first instincts survive or are overthrown by secondary and later instincts. This fact significantly increases the labor of assessing writing, particularly judgments of appropriateness (of task or genre). If a paper misses the ballpark of genre appropriateness, line by line commentary can be fruitless and futile. The student will need to rethink the task representation rather than fix broken patterns. Yet teachers can waste vast amounts of time reading and scanning to form a conclusive instinctual judgment about appropriateness.

5 Computational Recognition of Lexical Clusters Fitting Genre Computers are able to weigh multiple passages at once and so can, in principle, settle on instinctual judgment more cost-effectively than humans. To automate instinctual judgments of genre identification for various purposes, including writing assessment, Kaufer and Ishizaki created a computational environment, DocuScope [8, 10, 13], that makes judgments about a text’s genre characteristics from student writing. The program provides an environment for tagging student writing samples through pre-defined dictionaries, including 200 million patterns comprised of individual

Genre and Instinct

67

words and word sequences created through incremental coding and testing of hundreds of texts across dozens of genres. By handing coding words and short sequences, typically between 2 and 6 words in length, into functional categories, DocuScope classifies a text through human-informed local judgments, and these judgments in turn can be aggregated to generate an impression of the whole. As a whole, the patterns are designed to span the variation in English prose that one can find by reading across a spectrum of American English prose, from fiction to non-fiction, from everyday informal texts to erudite academic treatises. The patterns found are then statistically analyzed by multivariate methods in order to determine which family of patterns are used centrally (prototypically) in a text or corpus of texts and which are used only at the margins. DocuScope has been used in writing education in the states, but mainly as a tool for textual researchers. At the heart of the DocuScope program is a text visualization and analysis environment specifically designed to carry out rhetorical research with language and text. The program permits human knowledge workers, through computer-aided visual inspection and coding, to harvest and classify strings of English, primarily 1-5 contiguous word sequences. These are strings that, without conscious effort, speakers and writers use and reuse as part of their vast repertoire of implicit knowledge relating to language and the audience experience of the clusters constituting genre. We have chosen a knowledge-based, expert-system-like approach for our language measures because we were especially interested in the analysis and discovery of textual genres. Genres lie at the interaction of language and culture as tools to perform situated work [3]. To capture them requires a breakdown of texts that considers how texts structure experience for human readers. Employing an implementation of the Knuth-Morris-Pratt string matching algorithm [14], the DocuScope environment has the ability to uniquely discriminate patterns in a textual stream of any arbitrary length. For example, the string matcher knows that the 9 word string “the cat jumped over the table to get food” is non-identical to, and can be classified independently of, the 10 word string “the cat jumped over the table to get food yesterday.” This flexibility provides the capacity to separate small but important functional differences in the textual stream with contextual nuance. For example, we could discriminate the shorter string “smeared her,” which could be a negative category, from the longer string “smeared her with soap in the tub,” which indicates physical motion more than negativity. Aggregating functional categories like negative affect, physical motion and dozens of others functional categories into frequencies and comparing the frequencies against different text types has shown DocuScope to be a useful tool for distinguishing major genres of English. To find and classify strings of reader experience in English texts, we employed iterative hand coding methods. We first coded a list of categorized strings and then observed how they matched on a set of sample texts. We used our string-matcher on new texts to test our prior codings for accuracy and completeness. When we discovered our string-matcher making incorrect (i.e., that is, ambiguous, misclassified, or incomplete) matches on the new texts, we would use this information to elaborate the strings our string matcher could recognize. By repeating a cycle of coding strings on training texts and then testing and elaborating strings based on how well they explained priming actions on new texts, we were able to grow our catalog of strings systematically and consistently. Let’s now watch this process in action. Imagine

68


reading a newspaper containing “smeared the politician” as a verb phrase. This first inspection would prompt the generalization that the string “smeared the politician” conveys the idea of negative affect. We would code it thus in our dictionaries. We would input into our dictionary many syntactic variations that also conveyed this negative idea (e.g., smeared him, smeared them, smeared them). Now these dictionaries would be applied to new collections of uncoded texts, allowing us to find the limits of our own initial coding assumptions. For example, the coding of “smeared him” with a negative effect would seem incorrect in the longer string environment of “smeared him with soap.” Although errors of this type were ubiquitous, particularly in our early generations of coding, the software made it relatively easy for us to spot the mistakes and revise the underlying dictionaries accordingly. We thought of this rapid revision process as “improving the eyesight” of the dictionaries by putting human readers in to assist it. Over three years, we repeated this process of adding, testing, and differentiating strings of English over thousands of texts. We stayed with this process until we had nearly 150 categories that seemed robust and stable and that could differentiate, in principle, millions of strings. As our coding of strings evolved, we were able to derive formal decision criteria for classifying strings into one of 17 overall clusters. The string matcher can match any literal string of English of any length. For efficiency of coding, the software allowed us to run the string matcher on up to 500 texts at a time and over any number of user-defined categories of different strings. When the string matcher found a matching string in any of the target texts, it tagged it by name and color. The visualizer made it relatively easy for the research team to study the performance of the string-matcher and to improve it rapidly based on errors in its performance. The visualizer made it possible to build a very large and consistently classified inventory of priming strings in a relatively short amount of time. Where did we find the speech and texts to look for priming strings? We sampled the Lincoln/Douglas debates [15], texts associated with description, narrative, exposition, reporting, quotation, dialogue and conversational interaction. We also relied on three “seed” text collections. The first was a 120 text digital archive of short stories and fictional work. The second was a database of 45 electronic documents associated with a software engineering project, including proposals to the client, software design specifications, meeting minutes within the design team, meeting minutes between the design team and the client team, software documentation, focus group reports, public relation announcements, and feature interviews. We constructed a third archive from the Internet: the Federalist papers, the Presidential Inaugurals, the journals of Lewis and Clark, song lyrics from rappers and rockers, the clips of various syndicated newspaper columnists, the Web page welcomes of 30 university presidents, Aesop fables and the Brother’s Grimm, the writings of Malcolm X, the 100 great speeches of the 20th century, 10 years of newspaper reporting on the Exxon Valdez disaster, and movie reviews. We sampled 200 texts from this collection and made sure that we had multiple instances of each type of writing so that each type could be divided into training and test runs as we cycled through test and improvement cycles. On a weekly basis over a three-year period, we also coded strings from New Yorker magazine and from the editorials, features, and news pages of The New York Times. To capture data from speech, we coded for 2 to 4 hours every week the

Genre and Instinct

69

strings we identified heard over radio stations that focused on news, talk, or sports. The visualization environment allowed us to visually inspect and test new samples in our archive. To further control quality, we built a collision detector that would warn us if we assigned the same string to multiple categories. This helped us locate debugging inconsistencies and ambiguities in the string data. The visualization environment we taught with [10] also became a centerpiece in one of our graduate writing courses. As part of their continuing training in close reading, students were asked to keep semester logs of the matched strings they found in their own writing and in the writing of their peers. They were asked to keep systematic notes about whether the strings matched by the software were ambiguous or incorrect. In cases where they found errors, students proposed changes to the software’s internal dictionaries as part of their log-work. If their proposals were verified by the course instructors, the internal dictionaries were changed to reflect them. It is beyond the scope of this paper to say more about these methods, but further discussion about these techniques is available elsewhere [8, 13].

Fig. 1. Separating genres in a 2D projection of N dimensions by selecting and aggregating lexical clusters on an X and Y axis. This is a snapshot of one of the DocuScope interfaces used to separate genre through the selection and aggregation of specific lexical clusters. In this figure, the user has selected clusters (past, description, narrative) on the Y axis associated with reminiscences. The user has selected clusters (first person, first person personal register, personal register, and interactivity) on the X axis associated with letters. The interface confirms that these features are relevant to defining similarities and differences between these genres by separating reminiscences high on the Y axis and letters to the far right on the X axis.

70


The relevant point for now is why we needed software in the first place to capture the communication instincts evident in a text. The web of consideration behind the “instinct,” that a paper is appropriate or not to genre, is vast, so much so that it is easier for a computer, monitoring hundreds of lexical clustering patterns at once, to keep track of. Figure 1 displays a visual snapshot of one of DocuScope’s interfaces, which is used to separate genres visually according to the lexical clusters they make active. In Figure 1, 30 civil war letters and 30 reminiscences were drawn from the Internet. Letters are expected to be based more in first person, personal, and interactive lexical clusters than reminiscences. Reminiscences, by contrast, are expected to alert the reader to recalled situations from the past that are more deeply spatially and temporally elaborated than the situations of letters. Figure 1 shows how the DocuScope interface can confirm these hypotheses. The user clicks on different lexical clusters on a 2-D (X, Y) projection of N dimensional space to isolate and combine lexical clusters into genre groupings. In Figure 1, the user has selected clusters on the Y axis (viz., past, descriptive, narrative) favoring reminiscences and clusters on the X axis (viz., first person, first person personal register, personal register, and interactivity) favoring letters. As one can see, the selection of these clusters causes a visual separation of the actual letters and reminiscences. The reminiscences, coded in orange, dominate the top (Y) axis, while the letters, coded in red, dominate the (X) axis to the right.

6 Instinct and Rule in the Assessment of Writing Instincts are an important criterion for assessing student writing, but not the only criterion. Assessment criteria involve a mixed assortment of judgments across various grain sizes of text. The verbal criteria typically begin, we have argued, with coarsegrain judgments that one might call “instinctual.” These judgments depend upon making a gut reaction about the document’s overall “appropriateness” to the writing task or genre. Instinctual judgments can’t be localized to certain passages but implicate the reader’s overall impressions of the text. Instinctual judgments rely on combinations of judgments across many different passages. They further rely on the rater’s history of experience with previous tests and one’s responses to these as appropriate or inappropriate to the task or genre. There is no descriptive vocabulary ready at hand to characterize these whole-text instinctual judgments. They tend rather to be captured in a prescriptive shorthand to register that the text passes threshold (“is appropriate to the genre”) or not (“is not appropriate to the genre). In marked contrast to instinctual judgments in writing assessment, there are also what one might call “discrete” judgments of defective patterns. These are the finegrained judgments that detect errors in the surface text. These errors are “visual” in the sense that they are associated with enumerable defective patterns. These defects can be taught and remedied through perceptual training because they are spatially circumscribed in the textual stream. These defective patterns can furthermore be individuated, tallied and graphically displayed. Raters’ judgments of discrete criteria are rule-governed rather than instinctual. A human rater can detect, and often state a rule for verifying, the presence of a grammatical error at a specific location of text [5]. Further, the existence of the error is independent of what’s happening with other

Genre and Instinct

71

passages across the real estate of the text. This property of independent distribution guarantees that grammatical mistakes and other errors spotted through perception can be arithmetically summed to get a scalable assessment of overall errors in a text. By contrast, instinctual judgments of genre appropriateness are not additive in this way. What makes a text inappropriate to certain genre requirements typically has less to do with raw counts of “defective patterns” in the surface stream and more to do with detecting deviations in the underlying task representations behind the patterns chosen. To judge writing as appropriate or inappropriate to a genre is a judgment about the writer’s failure to meet important, culturally situated requirements that stand behind the words. Halfway between the realm of instinct and the realm of rule in writing assessment are the mid-tier questions of organization and style. Rating these aspects of student writing can be executed from the top-down, the residue of instinctual judgments about appropriateness and coherence, or from the bottom up, the residue of rule-governed judgments about grammaticality, correctness, and cohesiveness. Although this mesolevel of writing assessment is very important, we won’t discuss it further here. Noteworthy for present purposes is that the notion of instinct has been problematic for the institutions of assessment to address. Instinct, after all, seems a flimsy notion on which to base assessment. Whatever the validity of instinct, it seems diametrically at odds with the staid institution of assessment. Assessment relies on elaborated and consensual reasons and accepted rules of validation. Instinct relies on neither. Formal assessment may begin with the gut, but it is not supposed to stop there. Yet, rather than halt assessment in its tracks, writing teachers/evaluators for generations ignored the qualitative differences between judging that a writer had produced an “inappropriate” text and judging that the writer had split an infinitive. Both phenomena, on this longheld commonplace view, could simply be tabulated as “errors.” Dramatic change in this thinking occurred in the 1970s because of the pioneering work of Shaughnessy [16] and her many followers who taught basic writers of English [17, 18]. Through her work, Shaughnessy demonstrated that student errors were more epiphenomenal than phenomenal, that the defective patterns in the surface stream of text were not self-evident aberrations but rather socio-cultural curiosities that were rationally tied to the student’s background of experience and acculturation. For the teacher to help correct the defective patterns, she would need to recover the student’s hidden rationality based on the student’s limited cultural experience and then seek to augment that rationality by extending the student’s practice. The virtue of Shaughnessy’s approach, now dominant among composition researchers in the U.S., was to delve into the black box of appropriateness judgments through the window of defective patterns at the surface. While this approach has strengths, it is not without limitations. One drawback is that, while it can lead to insightful and rigorous analysis, this approach is hard to produce without considerable time and training, and impossible to deploy within the real time constraints of formal assessment. A second, subtler and unintended consequence is that, by making the text epiphenomenal, this approach can potentially sever all relationships between the judgment of textual appropriateness and textual patterns. Some in composition have openly championed this very implication, arguing that “good writing” should be defined by “good teaching” rather than “good texts” [18]. While this view has more merit than it may seem at first, it does, taken to the extreme, deny

72


the text as an evidentiary source of quality writing, which is counter-intuitive to the very feasibility of written assessment. More important for our purposes, this view discourages looking to see whether instinctual judgments of textual “appropriateness” can be rooted in deep patterns of text.

7 Is Genre Appropriateness Cross-Cultural? To gain fluency in a language is to acquire the instincts through which communicative intentions are realized and genres are enacted. As we have argued, these instincts are buried deep within the lexical clusters of a language and not simply learned through the externalized rules of situations in a culture. We have suggested that learning such rules can improve the performance of genres at the margins. But the essence of genre performance lies less in these externalized rules per se than in the lexical instincts at the core of meaning-making in a language. However, these propositions remain unproved conjectures. To test these propositions, we sought a student population of non-native speakers who had little face-to-face contact with native speakers and little face-to-face exposure to cultural situations in the west. Students taking English courses but who were not English majors at Guizhou University in the remote Guizhou province of China seemed a good pilot population for such a study. In the university level English curricula in China, there is a bifurcation between the curriculum for English majors and non-majors. The major in English is relatively small among the population of Chinese college students and so classes for English majors tend to be small (under 30 and often under 20), comparable to many US English language classrooms for majors or non-majors. Curriculum coverage is also similar, with Chinese English majors getting work in literature and advanced writing, including work in English modes and genres. In standard national tests of student writing in both China [12] and the United States [19, 20], human raters are asked to judge student essays on a multiple-point scale, often 1-6, where a “6” paper is rated highest and a “1” rated lowest [11]. Verbal criteria are given to help raters discriminate high, medium, and low ranking essays along this number system. English majors must pass the Test for English Majors (TEM), which also has a writing component and comes in two varieties. TEM 4 is taken at the end of the sophomore year and must be passed for the student to receive the English major. TEM8 is taken in the senior year and can help determine whether the student is recommended for graduate work in English abroad [12]. Non-majors in China learn English under very different circumstances. China has the largest population in the world and the most aggressive English language learning programs. At the University level, all enrolled students who are not majoring in English must take two China English [Language] tests (CET for short) (CET4 and CET6). Students who fail CET4 are not allowed to graduate. Students who fail CET6 may not be recommended for graduate training or work in other countries [12]. Non-majors congregate in classrooms that can range between 30 and 80 students, making assignments in production (speaking and writing) far less frequent than assignments in listening or reading. This is unfortunate because, whether they get writing practice or not, their performance will be measured on the high-stakes CET tests. Even more

Genre and Instinct

73

importantly, non-English majors are enrolled in a variety of professions where proficiency in English across all modalities will be assumed, including proficiency in writing and professional genres.

8 Prototypical vs. Peripheral Student Drafts We introduce the notion of the prototypical and the peripheral text. A prototypical text is defined in terms of genre characteristics. A genre can be defined as a clustering of lexical clusters that address the recurrent situational needs of the reader [4, 5, 7]. Different genres call for different lexical clusters depending on the situation of the writing and the relationship that the writer seeks to establish with the reader. A prototypical text is always relative to a genre of interest. An information text that exhibits the lexical clustering appropriate to sharing information with a reader is a prototypical information text. A narrative text that exhibits the lexical clustering appropriate to telling a story is a prototypical narrative text. By way of contrast, a text that fails to exhibit the lexical clustering of a target genre can be called peripheral to that genre. The text simply falls “out of range” of what the text is specified to do in the rhetorical situation. We have already presented a technological method for analyzing student texts for their lexical clustering. This technology allows a teacher in a large class of English non-majors to assign a text in a specified genre and to assess, in a fraction of the time it would take to read and mark the texts, the texts as prototypical or peripheral to the assigned genre. Students who are writing texts that are peripheral to the genre likely lack the English lexical clustering patterns to differentiate the genre they are writing in from other genres. Apart from theory testing, a pedagogical extension of our method is that it extracts and isolates the successful patterns of the students able to write prototypical texts. These patterns can then be taught directly to the less successful students. After further training, these students can be invited to revise the assignment, and the technology can reassess whether their text is now moving closer to prototypical status. A second important feature of our approach is that the teacher can assess and provide feedback to students on these important characteristics of genre writing without the labor of line-by-line reading and marking. The teacher instead runs a computer program and statistical methods that take seconds and minutes to run rather than hours and days of tedious labor.

9 Experiments Our research question was, can Chinese students with no training in English genres create prototypical genres of English? To answer this question, we applied the DocuScope technology to three sections of a required course for non-majors taught by the first author. Each section contains students from a different major, and each section was asked to write a 500 word English text in a different genre. The first section, consisting of 38 telecommunication majors, was asked to write a description of their dorm room. The second section, consisting of 27 physics majors, was asked to write a narrative recounting what they did during the week of China’s national 60th birthday

74


celebration. The third section, consisting of 36 economic majors, was asked to write an information paper which needed to “teach their reader” something. Except for these scant instructions, no further handouts were distributed or instructions given about what it meant to describe, narrate, or inform in English prose. Our interest in this study was to employ these methods as a diagnostic tool. We were primarily interested in seeing whether Chinese students with limited exposure to native speaker cultural situations could produce genre-appropriate (or prototypical) texts. We were also interested, for pedagogical purposes, whether the patterns produced by the capable students could form the basis of instruction for students who did not produce prototypical texts. We must caution at this point that a prototypical text is neither a perfect nor even a grammatically correct text. It simply means that it contains evidence of lexical clustering in the surface text that is expected and appropriate to the situational demands of the writing. A text that is prototypical must still undergo considerable revision to be an acceptable finished draft. At the same time, that revision toward a finished product will be wasted work if the draft to be revised does not exhibit prototypical features of acceptability. To identify the English patterns used by the 101 student texts across the three sections, we parsed them with the DocuScope technology. These patterns were then statistically analyzed to determine how the patterns clustered, whether the clusters resembled predicted lexical clusters of the genres assigned, which students were responsible for these prototypical texts, and which students, writing peripheral texts, created lexical clusters “out of range” of the assigned genres. Prototypical and peripheral texts lie on a continuum. However, for convenience, we focused our analysis on locating the most prototypical text in each classroom and the most peripheral.

10 Results Our main results suggested that rural Chinese students, even with limited exposure to native speaker situations and no training in English genres per se, were able, on average, to make key separations between information, narrative, and descriptive writing. 10.1 Separation between Information and Description Genres Figure 2 reveals one of the main results of the analysis, the separation between information and descriptive writing. Figure 2 shows this separation by plotting the first two factors when we factor-analyzed the patterns for co-occurrence regularities. The horizontal axis of Figure 1 shows the pattern clustering more typical of information writing. Information writing consists of patterns of direct interaction with the reader (e.g., “you will now learn”), stimulating the readers’ curiosity (e.g., “the puzzle to solve is”) and moving along in a sequence (e.g., “firstly”, “secondly,” “thirdly”). The vertical axis of Figure 2 shows the dominant clusters of descriptive writing. Among the students in our sample, descriptive writing is restricted to the present tense and is marked by the use of visual nouns (e.g., book, horse) and adjectives (blue, cuddly) and especially visual phrases (e.g., under the cabinet). The circles represent descriptive writing, the squares narrative writing, and the diamonds information writing. The reader will note that the diamonds (information writing) dominate the upper region of

Genre and Instinct

75

the chart. The circles (descriptive writing) dominate the lower center region. The separation appears in both factors, and both are statistically significant (see Figure caption). We boxed off both clusters as a visual guide to the reader. We were able to isolate one student in the economics section by the name of Li as having written the most prototypical information assignment. Note that Li’s paper is farthest to the right of any other information paper. We were able to isolate another student in the telecommunication section by the name of Qian as having written the most prototypical description assignment. Please note that Qian’s paper is lower than any other descriptive paper. These two students were able to produce a paper most “in range” with the restrictions of the genre assigned. Conversely, for each prototype paper, we could identify a corresponding peripheral paper, a paper whose lexical clustering was most out of range for the assignment. Youfei, an economics student assigned an information paper, wrote a text that clustered as a narrative piece. Peng, a telecommunications student assigned a descriptive paper, wrote a text with few descriptive features and also with relatively few markers of the other genres. -Separation of Information and Descriptive Writing - 3 < e g a u g n a L l a u s i V d e s a B t n e s e r P : 2 F

Pro totype Region Info

2 1

Peng

Description Narrative Information

Li

Youfei

0 -1 Qian -2 Pro totype Region Desc rip

-3 -3 -2 -1 0 1 2 3 F1: Gui ding the R eader Seque ntially Th rough Curi osity ----- >

Fig. 2. Plotting Factor 1 vs. Factor 2. Factor 1 isolates student texts that include guiding the reader, sequence, prescriptive, and curiosity. Factor 1 significantly distinguishes (MANOVA, F = 24.11; p

Computing with Instinct: Rediscovering Artificial Intelligence

Distributed Computing and Artificial Intelligence

Artificial Intelligence and Soft Computing

Artificial intelligence and soft computing

Artificial Intelligence with Uncertainty

Artificial Intelligence with Uncertainty

Artificial Intelligence

Artificial Intelligence

Artificial Intelligence

Artificial Intelligence

Artificial Intelligence

Practical Artificial Intelligence Programming With Java

Tools and Applications with Artificial Intelligence

Artificial general intelligence

Artificial Intelligence Through Prolog

Handbook of Artificial Intelligence,

Handbook of Artificial Intelligence,

Encyclopedia of Artificial Intelligence

Handbook of Artificial Intelligence,

Artificial Intelligence: Modern Approach

Applied Artificial Intelligence

Artificial Intelligence in Medicine

Artificial Intelligence Illuminated

Artificial Intelligence eBook

Artificial Intelligence: Modern Approach

Artificial Intelligence, Genetic Programming

Artificial Intelligence: Modern Approach

Artificial Intelligence Today

Artificial General Intelligence

Artificial Intelligence: Modern Approach

Artificial Intelligence today

Computing with Instinct: Rediscovering Artificial Intelligence

Distributed Computing and Artificial Intelligence

Artificial Intelligence and Soft Computing

Artificial intelligence and soft computing

Artificial Intelligence with Uncertainty

Artificial Intelligence with Uncertainty

Artificial Intelligence

Artificial Intelligence

Artificial Intelligence

Artificial Intelligence

Artificial Intelligence

Practical Artificial Intelligence Programming With Java

Tools and Applications with Artificial Intelligence

Artificial general intelligence

Artificial Intelligence Through Prolog

Handbook of Artificial Intelligence,

Handbook of Artificial Intelligence,

Encyclopedia of Artificial Intelligence

Handbook of Artificial Intelligence,

Artificial Intelligence: Modern Approach

Applied Artificial Intelligence

Artificial Intelligence in Medicine

Artificial Intelligence Illuminated

Artificial Intelligence eBook

Artificial Intelligence: Modern Approach

Artificial Intelligence, Genetic Programming

Artificial Intelligence: Modern Approach

Artificial Intelligence Today

Artificial General Intelligence

Artificial Intelligence: Modern Approach

Artificial Intelligence today

Recommend Documents