Design and Control of Intelligent Robotic Systems (Studies in Computational Intelligence, Volume 177)

Dikai Liu, Lingfeng Wang, and Kay Chen Tan (Eds.) Design and Control of Intelligent Robotic Systems

Studies in Computational Intelligence, Volume 177 Editor-in-Chief Prof. Janusz Kacprzyk Systems Research Institute Polish Academy of Sciences ul. Newelska 6 01-447 Warsaw Poland E-mail: [email protected] Further volumes of this series can be found on our homepage: springer.com Vol. 155. Hamid R. Tizhoosh and M. Ventresca (Eds.) Oppositional Concepts in Computational Intelligence, 2008 ISBN 978-3-540-70826-1 Vol. 156. Dawn E. Holmes and Lakhmi C. Jain (Eds.) Innovations in Bayesian Networks, 2008 ISBN 978-3-540-85065-6 Vol. 157. Ying-ping Chen and Meng-Hiot Lim (Eds.) Linkage in Evolutionary Computation, 2008 ISBN 978-3-540-85067-0 Vol. 158. Marina Gavrilova (Ed.) Generalized Voronoi Diagram: A Geometry-Based Approach to Computational Intelligence, 2009 ISBN 978-3-540-85125-7

Vol. 166. Constantinos Koutsojannis and Spiros Sirmakessis (Eds.) Tools and Applications with Artificial Intelligence, 2009 ISBN 978-3-540-88068-4 Vol. 167. Ngoc Thanh Nguyen and Lakhmi C. Jain (Eds.) Intelligent Agents in the Evolution of Web and Applications, 2009 ISBN 978-3-540-88070-7 Vol. 168. Andreas Tolk and Lakhmi C. Jain (Eds.) Complex Systems in Knowledge-based Environments: Theory, Models and Applications, 2009 ISBN 978-3-540-88074-5 Vol. 169. Nadia Nedjah, Luiza de Macedo Mourelle and Janusz Kacprzyk (Eds.) Innovative Applications in Data Mining, 2009 ISBN 978-3-540-88044-8

Vol. 159. Dimitri Plemenos and Georgios Miaoulis (Eds.) Artificial Intelligence Techniques for Computer Graphics, 2009 ISBN 978-3-540-85127-1

Vol. 170. Lakhmi C. Jain and Ngoc Thanh Nguyen (Eds.) Knowledge Processing and Decision Making in Agent-Based Systems, 2009 ISBN 978-3-540-88048-6

Vol. 160. P. Rajasekaran and Vasantha Kalyani David Pattern Recognition using Neural and Functional Networks, 2009 ISBN 978-3-540-85129-5

Vol. 171. Chi-Keong Goh, Yew-Soon Ong and Kay Chen Tan (Eds.) Multi-Objective Memetic Algorithms, 2009 ISBN 978-3-540-88050-9

Vol. 161. Francisco Baptista Pereira and Jorge Tavares (Eds.) Bio-inspired Algorithms for the Vehicle Routing Problem, 2009 ISBN 978-3-540-85151-6 Vol. 162. Costin Badica, Giuseppe Mangioni, Vincenza Carchiolo and Dumitru Dan Burdescu (Eds.) Intelligent Distributed Computing, Systems and Applications, 2008 ISBN 978-3-540-85256-8 Vol. 163. Pawel Delimata, Mikhail Ju. Moshkov, Andrzej Skowron and Zbigniew Suraj Inhibitory Rules in Data Analysis, 2009 ISBN 978-3-540-85637-5 Vol. 164. Nadia Nedjah, Luiza de Macedo Mourelle, Janusz Kacprzyk, Felipe M.G. Fran¸ca and Alberto Ferreira de Souza (Eds.) Intelligent Text Categorization and Clustering, 2009 ISBN 978-3-540-85643-6 Vol. 165. Djamel A. Zighed, Shusaku Tsumoto, Zbigniew W. Ras and Hakim Hacid (Eds.) Mining Complex Data, 2009 ISBN 978-3-540-88066-0

Vol. 172. I-Hsien Ting and Hui-Ju Wu (Eds.) Web Mining Applications in E-Commerce and E-Services, 2009 ISBN 978-3-540-88080-6 Vol. 173. Tobias Grosche Computational Intelligence in Integrated Airline Scheduling, 2009 ISBN 978-3-540-89886-3 Vol. 174. Ajith Abraham, Rafael Falc´on and Rafael Bello (Eds.) Rough Set Theory: A True Landmark in Data Analysis, 2009 ISBN 978-3-540-89886-3 Vol. 175. Godfrey C. Onwubolu and Donald Davendra (Eds.) Differential Evolution: A Handbook for Global Permutation-Based Combinatorial Optimization, 2009 ISBN 978-3-540-92150-9 Vol. 176. Beniamino Murgante, Giuseppe Borruso and Alessandra Lapucci (Eds.) Geocomputation and Urban Planning, 2009 ISBN 978-3-540-89929-7 Vol. 177. Dikai Liu, Lingfeng Wang and Kay Chen Tan (Eds.) Design and Control of Intelligent Robotic Systems, 2009 ISBN 978-3-540-89932-7

Dikai Liu Lingfeng Wang Kay Chen Tan (Eds.)

Design and Control of Intelligent Robotic Systems

123

Dikai Liu

Kay Chen Tan

ARC Center of Excellence for Autonomous Systems (CAS) Faculty of Engineering Level 6, Building 2 University of Technology, Sydney P.O.Box. 123 Broadway, NSW 2007 Australia Email: [email protected]

Department of Electrical and Computer Engineering National University of Singapore 4 Engineering Drive 3 Singapore 117576 Singapore Email: [email protected]

Lingfeng Wang Department of Electrical and Computer Engineering Texas A&M University 214 Zachry Engineering Center College Station, TX 77843-3128 USA Email: [email protected]

ISBN 978-3-540-89932-7

e-ISBN 978-3-540-89933-4

DOI 10.1007/978-3-540-89933-4 Studies in Computational Intelligence

ISSN 1860949X

Library of Congress Control Number: 2008942028 c 2009 Springer-Verlag Berlin Heidelberg  This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typeset & Cover Design: Scientific Publishing Services Pvt. Ltd., Chennai, India. Printed in acid-free paper 987654321 springer.com

Preface

With the increasing applications of intelligent robotic systems in various fields, the design and control of these systems have increasingly attracted interest from researchers. This edited book entitled “Design and Control of Intelligent Robotic Systems” in the book series of “Studies in Computational Intelligence” is a collection of some advanced research on design and control of intelligent robots. The works presented range in scope from design methodologies to robot development. Various design approaches and algorithms, such as evolutionary computation, neural networks, fuzzy logic, learning, etc. are included. We also would like to mention that most studies reported in this book have been implemented in physical systems. An overview on the applications of computational intelligence in bio-inspired robotics is given in Chapter 1 by M. Begum and F. Karray, with highlights of the recent progress in bio-inspired robotics research and a focus on the usage of computational intelligence tools to design human-like cognitive abilities in the robotic systems. In Chapter 2, Lisa L. Grant and Ganesh K. Venayagamoorthy present greedy search, particle swarm optimization and fuzzy logic based strategies for navigating a swarm of robots for target search in a hazardous environment, with potential applications in high-risk tasks such as disaster recovery and hazardous material detection. K.N. Krishnanand and D. Ghose, in Chapter 3, address the applications of robotic swarms in localization of multiple signal sources such as light, sound, fire-origins in bush fire, oil spills, etc. A modified glowworm swarm optimization algorithm is proposed and implemented in real-time experiments. Seiji Yamada and Daisuke Katagami, in Chapter 4, describe approaches to simulationbased behavior learning with a genetic algorithm and direct teaching with a learning classifier system (LCS) to enable a robot to learn sufficiently fast in real environments. In Chapter 5, D. Wang, N. M. Kwok, et al. present a ranked Pareto particle swarm optimization approach and its application in the design of force fields for motion planning and collision avoidance in multi-robot systems. In chapter 6, R. Barber, V. Egido, et al. investigate emotional intelligence for robot path planning by taking into account emotional factors, as human beings do, in the decision making process. An example of an emotion inspired planner is demonstrated.

VI

Preface

In Chapter 7, Jim Pugh and Alcherio Martinoli discuss how varying sensor offsets and scaling factors affect parallel swarm-robotic learning of obstacle avoidance behavior using both genetic algorithms and particle swarm optimization, in designing effective behavioral controllers for mobile robots in multi-robot systems. Patrick M. McDowell, Brian S. Bourgeois et al., in Chapter 8, investigate a memorybased learning for robot control in dynamic environments. A neural network which uses sensor inputs and action outputs is used as the basic controller, and genetic algorithms are applied to develop the network weights and transfer functions. In Chapter 9, Dongbing Gu, Jindong Liu, et al. present a behavior based control and learning system for autonomous robots. A Markov decision process is used to model the coordination mechanism, and a Q-learning approach to learn the behavior coordination. The learning system is implemented in two different types of robots. In Chapter 10, Yohannes Kassahun, Jan Hendrik Metzen, et al. present a method, evolutionary acquisition of neural topologies, for evolving the structures and weights of neural networks to find minimal neural structures for solving a given learning task. This method is applied to control a crawling robotic insect to learn and adapt to the environment. Terrence P. Fries, in Chapter 11, presents a fuzzy genetic algorithm approach to robot motion planning in real-time without the limitations of monotone paths. This approach employs a novel chromosome encoding scheme. The imprecision and uncertainty of terrain data are modeled using fuzzy linguistic variables and included in the path planning decision. In Chapter 12, Sukumar Kamalasadan and Adel A. Ghandakly present an intelligent adaptive control approach to precise tracking of autonomous robots. It consists of a neuro-controller that monitors the system’s closed loop dynamics and an adaptive controller for controlling the dynamics. This approach is verified by both simulation and real-time implementation with a mobile wheeled robot. In Chapter 13, Gary B Parker presents a punctuated anytime learning method for robot control. This method uses periodic tests on a robot to improve the accuracy of training done in simulation, and is applied in the learning of a robot control program. Nathan Wiebe and John Anderson, in Chapter 14, address the communication problems in a robot team. An approach is presented to allow a group of robotic agents to develop consistent shared groundings for useful locations in an environment using only local communication and interaction. This approach is demonstrated to be effective in communication in a multi-robot setting. Daniel C. Asmar, Samer M. Abdallah et al., in Chapter 15, develop an inertial-vision Simultaneous Localization and Mapping (SLAM) system that uses real natural objects as landmarks with the objective of creating a purposeful map. In Chapter 16, Florian Hoppe and Gerald Sommer present a programmed robotic system, i.e. a 6 DOF robot arm with two cameras, to learn to solve a shape-sorter puzzle. A new software architecture for artificial cognitive systems is proposed to handle data with different degrees of abstraction and realize nested perception-action cycles. Results from experiments demonstrate the performance of the system and that the system is able to compensate for variations in camera position.

Preface

VII

In Chapter 17, Jacky Baltes and John Anderson describe the work of developing a minimalist humanoid robot, Tao-Pie-Pie, including its walking gait and balancing reflexes. Issues in evaluating humanoid robots are also discussed in this chapter. The design, modeling and control of a four-link redundant parallel ankle rehabilitation robot are presented in Chapter 18 by Yun-Ho Tsoi, Shane Q. Xie et al. Robot kinematic parameters are selected based on singularity and workspace analysis. Controller design is facilitated by the modeling of human ankle (based on the biaxial joint ankle model) and the manipulator. Ankle stiffness and damping parameters are estimated by applying a recursive least squares algorithm. Results from simulations show the performance of the developed controller. In Chapter 19, Hector Hugo Aviles-Arriaga, L. Enrique Sucar, et al. describe a general framework which can be used to develop various applications of service robots. Modules for common tasks are developed and coordinated by a Markov Decision Process (MDP) based planner. With this research, a new application can be created by defining a new MDP whose solution provides an optimal policy that coordinates different behaviors for performing a task. Results from experiments in four different service robot tasks demonstrate the effectiveness of the proposed approach. In Chapter 20, Tetsuyou Watanabe, ZhongWei Jiang, et al. formulate an optimization problem to design desirable grasp operations of a robot hand using defined acceleration and equilibrium-force sets. The fundamental theory and the algorithm for finger configuration and for determining contact positions in grasping and manipulating an object are detailed. The proposed method is validated by simulations. In the last chapter, Gourab Sen Gupta, Chris Messom et al. address the problem of accurate motion control of fast moving wheeled robots by using a triangular targeting algorithm. Results from experiments with soccer robots demonstrate that the algorithm greatly improves the performance of the robot when compared to commonly used methods such as PD controllers. The intended readers of this edited book are engineers, researchers, senior undergraduates, and graduate students who are interested in the field of design and control of intelligent robotic systems. We hope that the readers would find some chapters in this book interesting and useful in their research as well as in practical engineering work in the area of robotics. We would like to thank all the authors who have contributed their work to this book. We are grateful to all the reviewers who have generously devoted their time to review the manuscripts. Special thanks go to Dr. N.M. Kwok for his time and effort in compiling the book. September 2008

D.K. Liu L.F. Wang K.C. Tan

Contents

1

2

Computational Intelligence Techniques in Bio-inspired Robotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M. Begum, F. Karray 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Bio-inspired Robotics and Tools of Computational Intelligence: The Historic Synergy . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 The CI Tools Most Commonly Used in Bio-inspired Robotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Bio-inspired Robotics: An Emerging Research Field . . . . . . . . . . . 1.4 Cognitive Modeling for Autonomous Robots: CI-Based Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.1 Cognition: Robot and Human . . . . . . . . . . . . . . . . . . . . . . . . 1.4.2 Value System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.3 Visual Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.4 Social Cognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Swarm Intelligence for Collective Robotic Search . . . . . . . . . . . Lisa L. Grant, Ganesh K. Venayagamoorthy 2.1 Collective Robotic Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Navigation Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Greedy Search - A Conventional Method . . . . . . . . . . . . . . 2.2.2 Particle Swarm Optimization . . . . . . . . . . . . . . . . . . . . . . . . 2.2.3 Fuzzy Logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Single Target Obstacle-Free Environment . . . . . . . . . . . . . . . . . . . . 2.4 Multiple Target/Obstacle Environment . . . . . . . . . . . . . . . . . . . . . . 2.4.1 Obstacle Avoidance Method . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.2 Case Study 1: Single Swarm . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.3 Case Study 2: Multiple Swarms . . . . . . . . . . . . . . . . . . . . . .

1 1 2 2 3 4 7 7 9 12 14 19 20 29 29 31 31 33 34 38 40 40 41 44

X

Contents

2.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

4

A Glowworm Swarm Optimization Based Multi-robot System for Signal Source Localization . . . . . . . . . . . . . . . . . . . . . . K.N. Krishnanand, D. Ghose 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Prior Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 The GSO Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 GSO Algorithm Description . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 A Simulation Example to Illustrate Basic Working of GSO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Algorithmic Modifications for Robotic Implementation . . . . . . . . 3.4.1 Obstacle Avoidance Model . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Implementation Models for Leapfrogging Behavior . . . . . . 3.5 Kinbots: Multi-robot Platform for Real-Robot-Experiments . . . . 3.5.1 Relative Localization of Neighbors . . . . . . . . . . . . . . . . . . . . 3.5.2 Luciferin Glow/Reception . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Real-Robot-Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.1 Experiment 1: Basic GSO Behavior . . . . . . . . . . . . . . . . . . . 3.6.2 Experiment 2: Leapfrogging Behavior . . . . . . . . . . . . . . . . . 3.6.3 Experiment 3: Localization of a Single Light Source . . . . . 3.7 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Evolutionary Robotics: From Simulation-Based Behavior Learning to Direct Teaching in Real Environments . . . . . . . . . Seiji Yamada, Daisuke Katagami 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Simulation-Based Behavior Learning Using Genetic Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Task: Action-Based Environment Modeling . . . . . . . . . . . . 4.2.3 States, Actions and Environment Vectors . . . . . . . . . . . . . . 4.2.4 Applying GA to Acquire Behaviors . . . . . . . . . . . . . . . . . . . 4.2.5 Simulation Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Directly Teaching for Robots in Real Environments . . . . . . . . . . . 4.3.1 Supervised Learning and Interactive Evolutionary Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.2 Teacher View and Learner View . . . . . . . . . . . . . . . . . . . . . . 4.3.3 Interactive Classifier System . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.4 Robot Experiments in the Real World . . . . . . . . . . . . . . . . 4.3.5 Teaching Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

45 46 49 49 51 52 52 54 55 56 56 57 59 60 60 60 63 65 67 67 69 69 70 71 72 73 75 78 83 85 86 86 87 89 93 94 94

Contents

5

6

7

Ranked Pareto Particle Swarm Optimization for Mobile Robot Motion Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D. Wang, N.M. Kwok, D.K. Liu, Q.P. Ha 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Ranked Pareto Particle Swarm Optimization . . . . . . . . . . . . . . . . . 5.2.1 Key Concepts in MOP s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.2 Particle Swarm Optimization . . . . . . . . . . . . . . . . . . . . . . . . 5.2.3 Ranked Pareto Particle Swarm Optimization . . . . . . . . . . . 5.3 F 2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.1 Definition of a Force Field . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.2 Attractive Force . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.3 Reaction Force . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.4 V SF 2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Using RPPSO for Multiobjective Optimization of F 2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.1 Case 1: Single Robot and One Obstacle . . . . . . . . . . . . . . . 5.4.2 Case 2: Two Robots Navigating in a Corridor . . . . . . . . . . 5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Path Planning Inspired on Emotional Intelligence . . . . . . . . . . R. Barber, V. Egido, M. Malfaz, M.A. Salichs 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 The Path Planning Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Emotions in Robotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Topological Representation of the Environment . . . . . . . . . . . . . . . 6.5 Path Planning Algorithm Inspired on Emotions . . . . . . . . . . . . . . 6.5.1 Environment Representation: The Navigation Chart . . . . 6.5.2 Topological Planning Algorithm: Guided Dijkstra . . . . . . 6.6 How the Guided Dijkstra Algorithm Works . . . . . . . . . . . . . . . . . . 6.6.1 Planning in the AD Control Architecture . . . . . . . . . . . . . . 6.6.2 Deliberative Planner Skill . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6.3 An Example of the Emotional Inspired Planning . . . . . . . 6.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . An Exploration of Online Parallel Learning in Heterogeneous Multi-robot Swarms . . . . . . . . . . . . . . . . . . . . . . . . . Jim Pugh, Alcherio Martinoli 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Parallel Swarm-Robotic Learning Case Study: Obstacle Avoidance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Heterogeneity from Sensor Offsets . . . . . . . . . . . . . . . . . . . . . . . . . .

XI

97 97 99 99 100 100 106 107 108 109 109 111 112 115 116 117 119 119 119 120 121 121 122 123 125 125 127 128 131 131

133 133 135 136 138 139 140

XII

Contents

7.4.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5 Heterogeneity from Sensor Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6 Diversity in Evolving Robot Swarms . . . . . . . . . . . . . . . . . . . . . . . . 7.6.1 Euclidean Diversity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6.2 Entropic Diversity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.8 Conclusion and Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

9

Robot Control in Dynamic Environments Using Memory-Based Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Patrick M. McDowell, Brian S. Bourgeois, Frederick E. Petry 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 ROV/UUV Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3 Memory Based Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4 Genetic Algorithm Training of Neural Network Controllers . . . . . 8.5 Reactive Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6 Memory Based Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A Behavior Based Control and Learning Approach to Real Robots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dongbing Gu, Jindong Liu, Huosheng Hu 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Layered Control Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.1 Locomotion Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.2 Behavior Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.3 Cognitive Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 Behavior Design Using Fuzzy Logic . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Q-Learning in the Cognitive Layer . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5 Robotic Fish Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6 Sony Quadruped Robot Experiments . . . . . . . . . . . . . . . . . . . . . . . . 9.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

141 141 142 143 143 145 145 145 146 148 150 150

153 153 154 156 159 161 162 164 168 169

171 171 173 174 174 175 176 178 180 183 185 186

10 Incremental Acquisition of Neural Structures through Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 Yohannes Kassahun, Jan Hendrik Metzen, Mark Edgington, Frank Kirchner 10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

Contents

10.2 Review of Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.1 Direct Genetic Encoding Approaches . . . . . . . . . . . . . . . . . 10.2.2 Indirect Genetic Encoding Approaches . . . . . . . . . . . . . . . . 10.3 Evolutionary Acquisition of Neural Topologies (EANT) . . . . . . . 10.3.1 Common Genetic Encoding (CGE) . . . . . . . . . . . . . . . . . . . 10.3.2 Genetic Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.3 Evaluating a Linear Genome . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.4 Generating the Initial Population . . . . . . . . . . . . . . . . . . . . . 10.3.5 Exploitation and Exploration of Structures . . . . . . . . . . . . 10.4 Practical Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4.1 Crawling Robotic Insect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4.2 Pole Balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Evolutionary Terrain-Based Navigation of Autonomous Mobile Robots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Terrence P. Fries 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Evolutionary Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Motion Planning Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.1 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.2 Representation of the Environment as a Grid . . . . . . . . . . 11.3.3 Encoding the Chromosome . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.4 Initial Population . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.5 Genetic Operators and Parameters . . . . . . . . . . . . . . . . . . . 11.3.6 Fuzzy Terrain Conditions in the Fitness Function . . . . . . . 11.3.7 Operation in a Dynamic Environment . . . . . . . . . . . . . . . . . 11.4 Test Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 A Novel Intelligent Control Approach for Precise Tracking of Autonomous Robots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sukumar Kamalasadan, Adel A. Ghandakly 12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2 Formulation of NNPAC Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.1 NNPAC Design Based on Feedback Error Learning . . . . . 12.2.2 Direct Model Reference Adaptive Control . . . . . . . . . . . . . 12.2.3 Generic RBFNN Architecture and Design . . . . . . . . . . . . . 12.3 Theoretical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.1 Development of the RBFNN Controller . . . . . . . . . . . . . . . 12.3.2 Mathematical Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.3 Controller Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.4 Stability Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

XIII

189 189 190 191 191 193 195 198 198 199 200 202 206 206

209 209 213 214 214 214 215 216 217 217 218 218 223 224

227 227 229 230 231 233 234 234 235 236 236

XIV

Contents

12.3.5 Modifications of Adaptive Laws Based on NNPAC Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.6 Online Growing RBFNN Design and Its Properties . . . . . 12.3.7 NNPAC Features and Algorithmic Development . . . . . . . . 12.4 Application to Position Tracking of Robots . . . . . . . . . . . . . . . . . . 12.4.1 Simulation Results and Discussions . . . . . . . . . . . . . . . . . . . 12.5 Real-Time Implementations and Experimental Results . . . . . . . . 12.5.1 Design Details and Algorithmic Flow Chart . . . . . . . . . . . . 12.5.2 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Punctuated Anytime Learning to Evolve Robot Control for Area Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gary B. Parker 13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2 Area Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.1 Previous Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.2 Research Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3 The ServoBot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.1 Gait Cycles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.2 Production of Turns in Gait Cycles . . . . . . . . . . . . . . . . . . . 13.3.3 Simulated Robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.4 Cyclic Genetic Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.4.1 CGAs Applied to Area Coverage . . . . . . . . . . . . . . . . . . . . . 13.4.2 Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5 Punctuated Anytime Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.1 Anytime Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.2 An Extension to Anytime Learning . . . . . . . . . . . . . . . . . . . 13.5.3 Fitness Biasing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.4 The Co-evolution of Model Parameters . . . . . . . . . . . . . . . . 13.6 Learning Area Coverage Using the Co-evolution of Model Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.6.1 Partial Recombination Used in the Evolution of Model Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.6.2 Punctuated Anytime Learning in Simulation . . . . . . . . . . . 13.6.3 Punctuated Anytime Learning on the Actual Robot . . . . 13.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

237 237 241 242 244 248 248 253 253 254

255 255 257 257 258 259 259 260 261 262 263 264 265 265 266 267 268 270 271 272 274 275 276

14 Local Methods for Supporting Grounded Communication in Robot Teams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 Nathan Wiebe, John Anderson 14.1 Introduction: Grounded Communication in Multi-robot Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 14.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283

Contents

XV

14.3 Developing Grounded Spatial References . . . . . . . . . . . . . . . . . . . . 14.3.1 Grounding Locations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.3.2 Sharing Groundings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.3.3 Consistency among a Group of Agents . . . . . . . . . . . . . . . . 14.4 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.4.1 Location Grounding Behaviours . . . . . . . . . . . . . . . . . . . . . . 14.4.2 Sharing Groundings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

284 285 286 286 288 289 291 294 299 300

15 Vision SLAM Maps: Towards Richer Content . . . . . . . . . . . . . . . Daniel C. Asmar, Samer M. Abdallah, John S. Zelek 15.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.2 Inertial Navigation System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.3 Computer Vision System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.3.1 Visual Environment Recognition . . . . . . . . . . . . . . . . . . . . . 15.3.2 Tree Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.3.3 Tree Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.3.4 Tree Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.4 Vision SLAM System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.5 Experiments and Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.5.1 Run-Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

303

16 Learning of Hierarchical Perception-Action Cycles . . . . . . . . . . Florian Hoppe, Gerald Sommer 16.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.1.1 Challenges and State of the Art Solutions . . . . . . . . . . . . . 16.1.2 Machine Learning Techniques in Robotics . . . . . . . . . . . . . 16.1.3 Structure of the Chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.2 Software Architecture for Artificial Cognitive Systems . . . . . . . . . 16.2.1 Overview of the Software Architecture . . . . . . . . . . . . . . . . 16.2.2 Concrete Visual Robotic System . . . . . . . . . . . . . . . . . . . . . 16.2.3 Dynamic of the Demonstrator . . . . . . . . . . . . . . . . . . . . . . . . 16.3 Example Submodules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.3.1 Internals of the Perception-Action Module . . . . . . . . . . . . . 16.3.2 Adaptive Visual Servoing Scheme . . . . . . . . . . . . . . . . . . . . 16.3.3 Learning the Long Term Goals of the Puzzle Game . . . . . 16.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.4.1 Validation of the Adaptive Visual Servoing Scheme . . . . . 16.4.2 Validation of the Shape Sorter Puzzle Player . . . . . . . . . . . 16.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

331

303 305 307 308 309 311 311 312 315 321 322 322 328

331 332 333 334 334 335 336 337 337 337 339 342 346 346 348 351 352

XVI

Contents

17 Advancing Artificial Intelligence through Minimalist Humanoid Robotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jacky Baltes, John Anderson 17.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.2 Tao-Pie-Pie . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.3.1 Viki . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.3.2 Pino and Morph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.3.3 Robo Erectus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.3.4 KAIST Robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.3.5 Johnny Walker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.3.6 Development of Walking Gaits . . . . . . . . . . . . . . . . . . . . . . . 17.4 Design Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.5 Mechanical Design of Tao-Pie-Pie . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.6 Development of Walking Gaits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.6.1 Inverse Kinematics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.6.2 Bezier Curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.6.3 Pattern for Straight Walk . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.6.4 Turning Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.6.5 Kicking Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.7 Balancing Reflexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.7.1 Sensor Feedback in Detecting a Fall . . . . . . . . . . . . . . . . . . . 17.7.2 Balancing Reflexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.8 Evaluation of Balancing Reflexes . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.9 Moving Forward: The HuroCup Competition . . . . . . . . . . . . . . . . . 17.10 Discussion: From Tao-Pie-Pie to Blitz . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Design, Modeling and Control of an Ankle Rehabilitation Robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yun-Ho Tsoi, Shane Q. Xie, Andrew E. Graham 18.1 Rehabilitation Robots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.2 Ankle Rehabilitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.3 Robot Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.4 Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.5 Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.5.1 Impedance Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.5.2 Kinematic Self Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.5.3 Environment Parameter Estimation . . . . . . . . . . . . . . . . . . . 18.6 Controller Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

355 355 357 358 358 359 359 359 359 359 360 360 362 362 363 364 365 365 366 368 368 370 372 375 376

377 377 378 379 383 386 386 389 392 394 397 397

Contents

19 Markovito: A Flexible and General Service Robot . . . . . . . . . . H´ector Hugo Avil´es-Arriaga, L. Enrique Sucar, Eduardo F. Morales, Blanca A. Vargas, Josu´e S´ anchez, Elva Corona 19.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.2 Software Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.3 Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.3.1 Map Building and Localization . . . . . . . . . . . . . . . . . . . . . . . 19.3.2 Navigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.3.3 Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.3.4 Speech Synthesis and Recognition . . . . . . . . . . . . . . . . . . . . 19.3.5 Visual Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.4 Coordinator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.4.1 Markov Decision Processes . . . . . . . . . . . . . . . . . . . . . . . . . . 19.4.2 Task Coordination Based on MDPs . . . . . . . . . . . . . . . . . . . 19.5 Implementation Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.5.1 Hardware and Software Platforms . . . . . . . . . . . . . . . . . . . . 19.5.2 General Structure and Operation . . . . . . . . . . . . . . . . . . . . . 19.6 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.6.1 Navigate through Three Places and Return to the Starting Position . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.6.2 Lost and Found . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.6.3 Follow a Person under User Request . . . . . . . . . . . . . . . . . . 19.6.4 Deliver Messages and Objects between People . . . . . . . . . . 19.7 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Task Based Hybrid Closure Grasping Optimization for Autonomous Robot Hand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tetsuyou Watanabe, ZhongWei Jiang, Tsuneo Yoshikawa 20.1 Grasping by a Robot Hand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.2 Target System and Kinematics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.2.1 Target System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.2.2 Nomenclature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.2.3 Kinematic Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.2.4 Statics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.2.5 Frictional Constraint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.3 Mechanics of Hybrid Closure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.4 Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.5 Concept of Required External Force Set . . . . . . . . . . . . . . . . . . . . . 20.6 Formulation of Grasping Optimization Problem Using Required External Force Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.7 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.7.1 Subproblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.7.2 Procedure of the Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 20.8 Numerical Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.9 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

XVII

401

401 402 403 404 406 408 408 409 413 413 413 414 415 415 416 416 417 418 418 420 421

425 425 428 428 429 430 431 431 431 433 434 436 438 439 440 443 446

XVIII

Contents

20.10 Remark for Gravitational Term of Fingers . . . . . . . . . . . . . . . . . . . 446 20.11 Proof for That Subproblem1 Can Be Solved by Solving Subproblem2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449 21 Triangular Targeting Algorithm (TTA) for Accurate Motion Control of Fast Mobile Robots . . . . . . . . . . . . . . . . . . . . . . Gourab Sen Gupta, Chris Messom, Serge Demidenko 21.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Classical PID Control for Robot Positioning and Orientation . . . 21.3 Robot Orientation Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.4 Robot Position Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.5 Advanced Robot Positioning Algorithm . . . . . . . . . . . . . . . . . . . . . 21.5.1 Pre-processing Inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.5.2 Re-targeting System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.6 Triangular Targeting Algorithm (TTA) . . . . . . . . . . . . . . . . . . . . . . 21.7 Proximity Positioning Algorithm (PPA) . . . . . . . . . . . . . . . . . . . . . 21.8 Velocity Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.8.1 Angular Velocity Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.8.2 Linear Velocity Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.8.3 Cross-Effects Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.9 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.9.1 Angular Motion Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.9.2 Linear Motion Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.9.3 Cross Effects Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . 21.9.4 Testing Triangular Targeting Algorithm . . . . . . . . . . . . . . . 21.10 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

453 453 454 455 456 458 458 459 460 464 464 465 465 466 467 468 468 469 474 474 475

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477

List of Contributors

Momotaz Begum Department of Electrical and Computer Engineering University of Waterloo 200 University Avenue West Waterloo, Ontario, Canada N2L 3G1, [email protected]

Department of Electrical and Computer Engineering Missouri University of Science and Technology 301 W 16th Street, Rolla, MO 65409, USA, [email protected]

Fakhri Karray Department of Electrical and Computer Engineering University of Waterloo 200 University Avenue West Waterloo, Ontario, Canada N2L 3G1, [email protected]

K.N. Krishnanand Mobile Robotics Laboratory (GCDSL), Department of Aerospace Engineering, Indian Institute of Science, Bangalore, [email protected]

Lisa Grant Real-Time Power and Intelligent Systems Laboratory Department of Electrical and Computer Engineering Missouri University of Science and Technology 301 W 16th Street, Rolla, MO 65409, USA, [email protected] Ganesh Kumar Venayagamoorthy Real-Time Power and Intelligent Systems Laboratory

D. Ghose Mobile Robotics Laboratory (GCDSL), Department of Aerospace Engineering, Indian Institute of Science, Bangalore, [email protected] S. Yamada National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda, Tokyo, Japan, [email protected] D. Katagami Interdisciplinary Graduate School of Science and Engineering, Tokyo Institute of Technology,

XX

List of Contributors

J2-53 4259 Nagatsuta-chou, Midori, Yokohama, Japan, [email protected] D. Wang ARC Centre of Excellence for Autonomous Systems, Faculty of Engineering, University of Technology, Sydney, Broadway, NSW 2007, Australia, [email protected] N.M. Kwok ARC Centre of Excellence for Autonomous Systems, Faculty of Engineering, University of Technology, Sydney, Broadway, NSW 2007, Australia, [email protected] D.K. Liu ARC Centre of Excellence for Autonomous Systems, Faculty of Engineering, University of Technology, Sydney, Broadway, NSW 2007, Australia, [email protected] Q.P. Ha ARC Centre of Excellence for Autonomous Systems, Faculty of Engineering, University of Technology, Sydney, Broadway, NSW 2007, Australia, [email protected]

M. Malfaz Universidad Carlos III de Madrid, Avda de la Universidad 30, 28911-Leganes, Madrid, SPAIN, [email protected] M.A. Salichs Universidad Carlos III de Madrid, Avda de la Universidad 30, 28911-Leganes, Madrid, SPAIN [email protected] Jim Pugh Distributed Intelligent Systems and Algorithms Laboratory ´ Ecole Polytechnique F´ed´erale de Lausanne, Lausanne, Switzerland, [email protected] Alcherio Martinoli Distributed Intelligent Systems and Algorithms Laboratory ´ Ecole Polytechnique F´ed´erale de Lausanne, Lausanne, Switzerland, [email protected] Patrick M. McDowell Southeastern Louisiana University, Computer Science Department, Hammond, LA 70402, USA, [email protected]

R. Barber Universidad Carlos III de Madrid, Avda de la Universidad 30, 28911-Leganes, Madrid, SPAIN, [email protected]

Brian S. Bourgeois Naval Research Laboratory, Code 7440.5, Stennis Space Center, MS, USA 39529, [email protected]

V. Egido Universidad Carlos III de Madrid, Avda de la Universidad 30, 28911-Leganes, Madrid, SPAIN, [email protected]

Frederick E. Petry Naval Research Laboratory, Code 7440.5, Stennis Space Center, MS, USA 39529, [email protected]

List of Contributors

XXI

Dongbing Gu Department of Computing and Electronic Systems, University of Essex, United Kingdom, [email protected]

Faculty 3-Mathematics /Computer Science, University of Bremen, Robert-Hooke-Straße 5, D-28359, Bremen,Germany, [email protected]

Jindong Liu Department of Computing and Electronic Systems, University of Essex, United Kingdom, [email protected]

Terrence P. Fries Department of Computer Science, Coastal Carolina University, Conway, SC 29526, USA, [email protected]

Huosheng Hu Department of Computing and Electronic Systems, University of Essex, United Kingdom, [email protected]

Sukumar Kamalasadan Department of Engineering and Computer Technology University of West Florida, Pensacola, FL-32514, USA, [email protected] Adel A. Ghandakly Department of Electrical and Computer Engineering California State University, Chico, CA- 95929, USA, [email protected]

Yohannes Kassahun Robotics Lab, Faculty 3-Mathematics /Computer Science, University of Bremen, Robert-Hooke-Straße 5, D-28359, Bremen, Germany, [email protected] G.B. Parker Computer Science Jan Hendrik Metzen Connecticut College German Research Center for Artificial 270 Mohegan Ave, New London, CT, Intelligence (DFKI), USA, Robert-Hooke-Strae 5, [email protected] D-28359, Bremen, Germany, jan [email protected] N. Wiebe Department of Computer Science Mark Edgington University of Manitoba Robotics Lab, Faculty Winnipeg, Manitoba, Canada 3-Mathematics/Computer Science, R3T2N2, University of Bremen, [email protected] Robert-Hooke-Straße 5, D-28359, J.E. Anderson Bremen, Germany, [email protected] Department of Computer Science University of Manitoba Frank Kirchner Winnipeg, Manitoba, Canada German Research Center for Artificial R3T2N2, Intelligence (DFKI), Robotics Lab, [email protected]

XXII

List of Contributors

D.C. Asmar Department of Mechanical Engineering, American University of Beirut, Riad El Solh 1107 2020, P.O. Box 11-0236, Beirut, Lebanon, [email protected] S.M. Abdallah, Deloitte, Al Fahad & Co. for Consulting W.L.L. Fahad Al-Salem Street? Salhia Complex, Entrance 2 P.O. Box 23049, Safat 13091? Kuwait City, Kuwait, [email protected] J.S. Zelek Department of Systems Design Engineering, University of Waterloo, 200 University Ave. W., Waterloo, Ontario, Canada, N2L 3G1, [email protected] F. Hoppe Cognitive Systems, Institute of Computer Science, Faculty of Engineering, Christian-Albrechts-University, Kiel, Ohlshausenstr. 40, 24098, Germany, [email protected]

J.E. Anderson Department of Computer Science University of Manitoba Winnipeg, Manitoba, Canada R3T2N2, [email protected] Y.H. Tsoi Medical and Rehabilitation Robotics Research Group, Department of Mechanical Engineering, The University of Auckland, New Zealand, [email protected] S.Q. Xie Medical and Rehabilitation Robotics Research Group, Department of Mechanical Engineering, The University of Auckland, New Zealand, [email protected] A.E. Graham Medical and Rehabilitation Robotics Research Group, Department of Mechanical Engineering, The University of Auckland, New Zealand, [email protected]

G. Sommer Cognitive Systems, Institute of Computer Science, Faculty of Engineering, Christian-Albrechts-University, Kiel, Ohlshausenstr. 40, 24098, Germany, [email protected]

Hector Hugo Aviles-Arriaga Departamento de Ciencias de la Computaci´ on. Instituto de Investigaciones en Matem´aticas Aplicadas y en Sistemas (IIMAS) Universidad Nacional Aut´ onoma de M´exico (UNAM) Mexico, [email protected]

J. Baltes Department of Computer Science University of Manitoba Winnipeg, Manitoba, Canada R3T2N2, [email protected]

L. Enrique Sucar Instituto Nacional de Astrofisica, Optica y Electronica Computer Science Department. Sta. Ma. Tonantzintla, Mexico, 72840, [email protected]

List of Contributors

Eduardo F. Morales Instituto Nacional de Astrofisica, Optica y Electronica Computer Science Department. Sta. Ma. Tonantzintla, Mexico, 72840, [email protected] Blanca A. Vargas Instituto Nacional de Astrofisica, Optica y Electronica Computer Science Department. Sta. Ma. Tonantzintla, Mexico, 72840, [email protected] Josue Sanchez Instituto Nacional de Astrofisica, Optica y Electronica Computer Science Department. Sta. Ma. Tonantzintla, Mexico, 72840, [email protected] Elva Corona Instituto Nacional de Astrofisica, Optica y Electronica Computer Science Department. Sta. Ma. Tonantzintla, Mexico, 72840, [email protected] Tetsuyou Watanabe Graduate School of Natural Science & Technology, Kanazawa University, Kakuma-machi, Kanazawa, 920-1192, Japan, [email protected]

XXIII

ZhongWei Jiang Graduate School of Science and Engineering, Yamaguchi University, Ube, 755-8611, Japan, [email protected] Tsuneo Yoshikawa College of Information Science and Engineering, Ritsumeikan University, Kusatsu, 525-8577, Japan, [email protected] Gourab Sen Gupta School of Engineering and Advanced Technology (SEAT), Massey University, Private Bag 11 222, Palmerston North, New Zealand 4442, [email protected] Chris Messom Institute of Information and Mathematical Sciences (IIMS), Massey University, Private Bag 102-904, Auckland, New Zealand 0745, [email protected] Serge Demidenko School of Engineering and Advanced Technology (SEAT), Massey University, P.O. Box 756, Wellington, New Zealand 6140, [email protected]

1 Computational Intelligence Techniques in Bio-inspired Robotics M. Begum and F. Karray Department of ECE, University of Waterloo, Canada {m2begum,karray}@uwaterloo.ca

Abstract. Biologically inspired robotics is a rapidly emerging research area standing at the cross-cut of biology, artificial intelligence, and robotics. The structural and functional perfection of the biological creatures achieved through thousands of years of evolution in nature have made them a great source of inspiration for designing complex autonomous systems. Accordingly, robotics researchers have initiated designing artificial machines while considering the characteristics of biological systems as the benchmark knowledge. ‘Biomimetics’, an emerging research area, is a consequence of such initiatives. Bio-inspired robotics is actually a subset of biomimectics focusing mostly on the design of autonomous robots while mimicking the intelligence, cognition, and structural properties of the primates and other lower order species. Computational intelligence techniques developed in the past several years are related closely to bio-inspired robotics as both of them have originated from the common inspiration of mimicking human intelligence, although their development have followed different paths. Bio-inspired robotics have used computational intelligence techniques to mathematically model ‘human-like intelligence behavior’ for autonomous robots. This chapter provides an overview on the tools of computational intelligence and their applications in bio-inspired robotics. It highlights the recent progress made in bio-inspired robotics while giving a special emphasis on the usage of computational intelligence tools to design human-like cognitive abilities in the robotic systems.

1.1 Introduction Biologically inspired robotics is a multidisciplinary research area motivated by the inspiration of using nature as the benchmark in designing complex autonomous systems and, at the same time, benefiting nature with the improved system autonomy. Biological creatures, with their adaptability, structural sophistication and varieties in behavior, have always been a great source of inspiration for the researchers in robotics. But substantial research in bio-inspired robotics has started only a few years back. The reasons behind such a delayed start is mainly due to our limited understanding of nature, and to the lack of technological sophistication to mimic biological systems, both at the structural as well as the intellectual level. During the past two decades we have achieved remarkable improvement in closely pertinent areas of bio-robotics such as nano-technology, artificial intelligence (AI), and neuroscience. Achievements in these three, apparently disjoint, disciplines have placed us in a position to perform reverse engineering on biological systems. Increased computing power in gradually diminishing sized computer chips are showing the glimpse of possibility of having computers D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 1–28. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

2

M. Begum and F. Karray

comparable to the primates brain (could be lower order primates) in terms of processing power, complexity, and compactness. Our steadily developing understanding of living elements, especially the mechanism of brain functions, theory of interaction among mind, body, and brain, is opening up possibilities of mimicking them in a more robust manner. The AI and related disciplines have already earned remarkable improvement in developing sophisticated computational tools, industry-strength algorithms for machine learning, image, and speech processing which may help to successfully model the functionality of biological systems. At the midst of resonance in development among the related disciplines, bio-inspired robotics has become a topic of significant interest. A substantial amount of research work has been reported during the current decade in various sub-disciplines of bio-inspired robotics and the flow is continuously rising. This chapter provides a brief survey on the trends of bio-inspired robotics while giving a special focus on how its development has been influenced by the approaches rooted in computational intelligence (CI). The rest of the chapter is organized as follows. Section 1.2 sheds light on the history of AI, bio-inspired robotic, and CI techniques, their common origin and underlying goals. Section 1.3 provides a brief discussion on bio-inspired robotics as an emerging research area and highlights existing research directions in bio-inspired robotics. Section 1.4 focuses on one of the most appealing branch of bio-inspired robotics namely, cognitive modeling for autonomous robots. It focuses on different emerging areas of cognitive modeling for autonomous robots, provides a brief survey on the state-of-the art of these areas and sheds light on the role of CI techniques in designing artificial cognition for autonomous robots. Finally, section 1.5 draws some concluding comments.

1.2 Bio-inspired Robotics and Tools of Computational Intelligence: The Historic Synergy 1.2.1

Overview

Computational intelligence techniques, e.g. neural networks, fuzzy logic, genetic algorithms, probabilistic algorithms, are well known to be biomimetic computational tools designed from the inspiration of mimicking the structural and functional elegance of the human to perceive the environment and act accordingly while applying their intelligence. On the other hand, the dream of a human-like machine (in terms of structure and intelligence) was there since the birth of AI. The CI techniques and AI, therefore, should have been complementing each other in the course of their own development as well as in building a complete model of human-like intelligence. In reality, however, a different scenario has taken place. The development of CI techniques followed a different path than that of the AI [1]. The CI techniques have experienced tremendous theoretical growth over the past few decades and have also earned popularity in application areas e.g. control, search and optimization, data mining, knowledge representation, signal processing, and robotics. Specifically in AI robotics, a good number of robotic systems uses CI techniques for robust control, planning, and decision making [2, 3, 4, 5]. Unfortunately, these applications do not reflect the vision with which the CI techniques were originally developed for: modeling human intelligence [6]. The AI, on the other

1 Computational Intelligence Techniques in Bio-inspired Robotics

3

hand, ruled by the norms of GOFAI(R) (Good Old Fashioned AI (and Robotics)) [7, 8] for an extended period of time, failed in many ways to utilize the power of CI techniques in designing human-like intelligence. Thus the long-expected synergy between AI and CI was delayed until the AI revolution of 1990s [9] which replaced the static notion of “ ‘mind-as-computer’ in a symbol-based representation of the world” with the dynamic idea of ‘embodied cognition’. Embodied cognition advocates the notion of bidirectional active interaction among body, mind, and environment as the building block of developing intelligence, both natural and artificial [9, 10, 11, 12, 13]. The current AI robotics, therefore, is focusing on modeling the highly complex, non-linear interaction among mind, body, and environment, the self-motivation to build knowledge base, the automated reasoning, and decision making. The ideas of embodied cognition places CI techniques back in the stage of AI. The CI tools blend naturally with these concepts as they share the common principles of biology. The CI techniques, therefore, are very efficient tools to design bio-inspired intelligence in autonomous robots. 1.2.2

The CI Tools Most Commonly Used in Bio-inspired Robotics

The CI tools developed in the past several years have been widely used in different branches of bio-inspired robotics. New research challenges with higher complexities are arising as we are approaching toward the goal of designing human-like intelligent machines. To meet these challenges new CI tools are being developed as well as modification of the existing tools are being performed. For instance, artificial immune algorithm [14] has been developed based on the inspiration of human immune systems and type 2 fuzzy logic [15] has been proposed as an extension of tradition fuzzy logic to include probabilistic uncertainty in fuzzy reasoning. A brief introduction on the CI tools most commonly used in the existing bio-robotics literature is provided here. 1.2.2.1 Artificial Neural Network Artificial neural network (ANN) is one of the most commonly used CI tools in bioinspired robotics [16]. ANNs mimic the highly parallel structural connectivity and nonlinear operating principle of the brain cells (known as neuron) of biological creatures [17]. The structure of a single artificial neuron (also known as perceptron) is very simple: input terminal(s), one output terminal, and a function defining the mapping from input(s) to the output. The simple perceptrons connect with each other in layered architectures to form different kind of neural networks which are generally capable to handle even very high degree of non-linearity. One elegant property of ANNs is they allow the exact pattern of non-linearity to emerge within themselves through the process of training. There are algorithms to conduct efficient training of ANNs. The theories of neural network have gone through several facets of development in the past decades resulting in varieties of ANNs suitable for different application areas, e.g. data mining, robotics, communication. 1.2.2.2 Fuzzy Reasoning Fuzzy reasoning is another CI tool which mimics the capacity of human mind to process partial truth in observed phenomena [18]. Fuzzy logic, the mathematics of fuzzy

4

M. Begum and F. Karray

reasoning, advocates that the traditional crisp (or binary) logic which deals with the two extreme ends (‘yes’and ‘no’) of the decision paradigm is just a special case of fuzzy logic. Fuzzy reasoning, therefore, is governed by the norms of partial membership and assigns a degree belief in favor of a proposition. Such degree of belief may take any value between zero to one. The elegance of fuzzy logic lies in its simplicity to model even extremely complex, high-dimensional decision problems. This results in its widespread application in areas such as control, pattern recognition, artificial intelligence, and robotics. 1.2.2.3 Evolutionary Algorithms Evolutionary algorithms (EA) are generally considered as search and optimization technique inspired from the principle of natural selection. They provide a structured way to locate the most suitable solution in a highly complex search landscape. EAs generally perform a series of operations (selection, mutation, crossover) to iteratively optimize a fitness function. Starting form the simple form as proposed in [19], the evolutionary algorithms have went through numerous modifications to suit with the increasing complexities of todays search problems. A number of new search and optimization algorithms has been developed in recent years based on the inspiration from nature, e.g. artificial immune algorithm [14], swarm intelligence [20], ant-colony optimization [21]. The application of EAs in bio-inspired robotics is still relatively less appealing due to its high time complexity. 1.2.2.4 Probabilistic Algorithms Probabilistic algorithms have been quite useful in several areas of mobile robotics including bio-inspired robotics. This is because of their capacity to deal with the uncertainty inherent in real-world robotic problems [22]. The probabilistic algorithms used in bio-inspired robotics are generally related to Bayesian inference, Bayes network and different variants of Bayes filter [23, 24, 25]. The mechanism of posterior formation based on prior knowledge and observation likelihood makes Bayes theorem a suitable candidate in modeling intelligence behavior for robotics systems. Besides, the mathematical structures of Bayes network and Markov model are well suited for state based robotic systems.

1.3 Bio-inspired Robotics: An Emerging Research Field Bio-inspired robots are a new kind of research platforms developed to fulfill the twopronged desire of humans: enhancing the quality of human life by incorporating artificial systems in it and, at the same time, improving the system design through integrating biology with it. The major driving forces behind the research on bio-inspired robotics, therefore, can be summarized as follows. • Robots to assist human: During the last decade mobile robotics has experienced a significant forward shift toward real-world applications from the laboratory based research. The Mars rover mission [26], winning of DARPA grand challenge 2005

1 Computational Intelligence Techniques in Bio-inspired Robotics

5

[27], robotics application in surgery [28] and health care [29] are among few examples. The bio-inspired robots are expected to top the current successes of mobile robots with their human-like sophistication in perception, reasoning, decision making and locomotion. These robots will be able to adapt autonomously in changing environment, will apply learned knowledge and reasoning capacity to solve new problems, and above all will be able, to a large extent, to take care of themselves [30]. This sort of machines have potential to drastically improve the quality of human life through providing assistance in several areas such as search and rescue, mining, space-applications, military- surveillance, entertainment, surgery, and health care. • Robots to understand human: The other inspiration of bio-inspired robotics is to have a better understanding of the capacities that the humans are blessed with [31]. Until recently, the flow of knowledge between biology and robotics was unidirectional, from biology to robotics. During the last decade a bi-directional link has been established between biology and robotics based on the realization that lifelike machines provide a wonderful platform to quantitatively test and analyze the computational models and theories from neuroscience [13, 32], psychology, developmental study [33], and human locomotion [34, 35]. As mentioned in [36], the bio-inspired robots act as the ‘mirror to reflect our humanity back at us as we interact with them’. Designing a robot with human-like sophistication in structure and intelligence is a crossdisciplinary endeavor and requires intense collaboration among disciplines such as biology, neuroscience, psychology, linguistics, robotics and AI. The efforts made in this endeavor can be categorized into four distinct categories. • Biomimetic perception: Perception is the key requirement for survival of the living elements [37]. The pre-requisites of primates- like reasoning and decision making is to have primates- like perception. The perceptual system, therefore, is one of the most significant aspects of bio-inspired robot design. There is a steady progress in the development of biomimetic sensors and actuators technology. Such sensors and actuators enable a robot to perceive and interact with the environment in primateslike fashion (more specifically, human like). In other words, biomimetic sensors and actuators help the robots to perceive the same representation of the world as human does and thereby ease the blending of robots in the human environment. The log-polar camera for human like vision [38], haptic devices to mimic touch sensitivity [39], micro electro-mechanical systems (MEMS)[40] and elector-active polymer (EAP) to model the biological muscles [41] are among the many examples of biomimetic sensors and actuators. The possibility of integrating soft tissue to construct flexible and deformable robots are on the way to become full-fledge areas of research and applications [42]. A large group of robotics researchers are currently working on reverse engineering the performance characteristics of biological sensors to formalize control mechanisms of biomimetic sensors as well as their power requirements and efficiency. • Locomotion and control: Locomotion of autonomous robots based on the motion primitives or imitation of biological creatures is a rapidly emerging research area in bio-inspired robotics. The major focus is to mimic lower level sensory-motor

6

M. Begum and F. Karray

skills of biological creatures (ranging from insect to human) in autonomous robots. Bipedal walking by humanoid robots [43, 44], humanoid upper body control [45, 46], design and control of articulated robots e.g. reconfigurable robots [47, 48], crawling and jumping robots [49, 50, 51] are only a few examples of widespread research in bio-inspired locomotion control in robotics. • Control architectures and learning methodologies: These are the sub-discipl-ines of bio-inspired robotics which have been using bio-inspired mechanisms for a long period of time. Majority of the existing control architectures used in autonomous robotics are bio-inspired. The traditional reactive [52], deliberative, and hybrid [53] controllers are inspired from insect locomotion control mechanism whereas the relatively newer behavior-based controller [54] obtains inspiration from human cognition. Similarly, the learning methodologies commonly used in robotic, e.g. reinforcement learning, associative learning, classical conditioning, automated animal like (AA)-learning [55], neural network and statistical learning, are designed based on the learning process of biological creatures ranging from insect to human. • Cognitive modeling: Cognitive modeling deals with implanting human-like cognition, albeit reduced complexity, in autonomous robots with the hope that the cognitive capacities, as they work in human and other biological creatures, will also assist the new generation robots to be coherent, self-motivated, social, persistent in behavior, and aware of themselves and their environments. Cognitive modeling, therefore, requires integration of neuroanatomy, cognitive neuroscience, cognitive psychology, developmental study, linguistics, and AI with robotics. A rapidly increasing number of publications on different aspects of cognitive modeling indicates the growing interest in this research area, although a complete model of robot cognition does not yet exist [56, 57]. The latest findings of neuroscience and psychology are continuously modulating the research on cognitive modeling for autonomous robots. Apart from few general theories on robot cognition (e.g. [9, 46, 58, 59, 60, 61]), the majority of the literature is focused on developing models for discrete cognitive abilities, e.g. attention, learning, value system, reasoning, and decision making. The research on cognitive modeling for autonomous robots, therefore, is still in its infancy and there is a long way to traverse before having a full fledge autonomous robot with human-like cognition. A large number of university research laboratories around the globe is performing research on different areas of bio-inspired robotics: the biologically inspired robotics group at Ecole Polytechnique, the biorobotics laboratories at Harvard University, University of Washington, and MIT, the cognitive robotics labs at Vanderbilt University, Technical University of Munich, University of Genova, Michigan State University, University of Southern California, and Idaho National Laboratory. A growing number of conferences (e.g. EpiRob, ICDL) and journals (e.g. International Journal of Humanoid Robotics, special issues on bio-robotics of the IEEE Transaction on Robotics and Connection Science) are now fully dedicated to the findings from bio-inspired robotic research. The rest of this chapter focuses on the emerging use of CI techniques in cognitive modeling for autonomous robots. It provides a survey on the state-of-the-art of

1 Computational Intelligence Techniques in Bio-inspired Robotics

7

cognitive modeling in pursuit of human-like intelligent machine and investigates on how the CI tools have been invoked in modeling different aspects of cognition. It also advocates the idea that the features of cognition emerges naturally in CI techniques, when properly utilized. The CI tools, therefore, offer a very convenient way to model artificial cognition in autonomous robots.

1.4 Cognitive Modeling for Autonomous Robots: CI-Based Approaches With the tremendous advancement of computer systems, both in speed and storage capacity, AI has achieved significant strides during the past decades. Several AI techniques have emerged as a consequence of the enriched computational power. The advancements in AI robotics, to a large extent, is the consequence of this enriched computational power. However, today’s computers with their enormous speed and storage capacity essentially act as mindless intelligence, executing complex routines with extreme efficiency but showing minimal awareness about the tasks they are performing. AI researchers, therefore, have started working on developing cognitive computers which will be able to learn from its experience and apply the learned knowledge to deal with anomalies [62]. Initiatives taken by AI robotics researchers are one step ahead of this effort. Their idea is to develop embodied physical agents namely, cognitive robots, which are expected to act, behave, and think like humans while interacting with natural human environments. In pursuit of such an ideal goal, the only target device to mimic has always been the human brain. With the advent of new technologies, such as fMRI (functional Magnetic Resonance Imaging) [63], PET (Positron Emission Tomography) [64], neuroscience has been experiencing a significant forward shift toward a better understanding of human brain, yet major part of it is still unknown [37]. The advancements in neuroscience have introduced a remarkable change in the perception of the psychologists about human cognition and have encouraged the thought of explaining cognitive developments in terms of the changes in physical structure of the brain [65, 66, 67, 68]. 1.4.1

Cognition: Robot and Human

Since antiquity cognition has been a major research area in psychology and philosophy. It is probably one of the areas where modern neuroscience and psychology have performed the most significant improvement, although there is still a lot more to know [37, 68]. Elucidating cognition in a precise manner is difficult, yet all of us are generally aware of our cognitive abilities. In simple words, ‘cognition’ of a living element refers to its capacity to process perceptual information and thereby manipulating its behavior. Human cognition encompasses a large collection of processes occurring in human mind. The cognitive capabilities of humans are generally manifested in the following [10, 11, 37, 66, 67, 68]. • • • •

Self awareness, perception, learning, knowledge,

8

M. Begum and F. Karray

• reasoning, • planning and decision making. Generally, action execution is not considered as a part of cognition, although cognitive development is largely influenced by our capability to act upon the environment. Besides, action is the only explicit means onto which we reflect our cognitive skills. Designing an artificial agent with some or all of these powerful capabilities is the new grand challenge of AI [30, 62]. A number of attempts have been made over the past few years to meet this challenge and gave birth to a number of new research areas including ‘developmental engineering’ [46], ‘epigenetic robotics’ [69], ‘developmental robotics’ [58], and ‘developmental cognitive robotics’ [60]. All of them share the common vision of designing human-like cognitive agents, irrespective of their names and the methodologies applied. This chapter addresses these ‘similar vision’ efforts as ‘cognitive modeling for autonomous robots’. Based on the recent neuro-biological and psycho-physical findings about executive cognitive functions of human being (e.g. attention, learning, perception, social

Fig. 1.1. Example robotic systems based on bio- inspired cognitive models (a) Kismet (MIT), (b) ISAC (Vanderbilt University), (c) SAIL robot (Michigan University) (d) Babybot (University of Genova)

1 Computational Intelligence Techniques in Bio-inspired Robotics

9

communication), an impressive amount of research work has been carried out during the current decade in the following sectors of cognitive modeling: 1. Value system 2. Visual attention 3. Social cognition The rest of this chapter will provide a brief survey on the state-of-the-art of each of these sectors of cognitive modeling. Two criteria have been defined for a research work to be included in the survey presented in this chapter. • The research has to apply computational intelligence (CI) techniques to deal with the problem in the associated research area. • The research has to demonstrate the performance of the designed intelligence framework using any physical robotic system. The representative works in different sectors of cognitive modeling (value system, visual attention, and social cognition) are summarized and presented in tabular form for convenience in comparison. 1.4.2

Value System

The value system refers to the capacity of a biological brain to increase the likelihood of neural response to an external phenomena provided that there exists a prior experience of having the similar phenomena [70]. The combined action of internal perception, reasoning and decision making capacity contributes in development of values in the primates. Value is not hard-wired in human since birth, rather it grows developmentally while mediated by the plasticity of human brain [10, 11, 37]. Accordingly, the value with which a biological entity is born with is termed as ‘innate value’ whereas the value that it achieves through experience during the course of development is called ‘acquired value’. The plasticity of human brain, which makes the development of acquired value possible, is a recent discovery in neuroscience. The acquired value of a biological entity is reflected in its behavioral adaptivity. The robotic counterpart of value refers to the capacity of an autonomous robot to plan action upon detection of salient stimuli. A robot with value system performs this action planning after analyzing its internal and external context [13, 71]. Internal context of a robot is the knowledge-base developed through past experiences whereas external context refers to the current environmental situation in favor of a planned action. A value-based robot, therefore, is aware of its own action. Value system is a crucial requirement of developing human-like intelligence in robotic systems. The success of designing truly developmental robot depends largely on the design of a value system. The basic idea of the artificial value system for autonomous robots is to design a mechanism which can reflect the effect of the past experience in the future behaviors (e.g. in planning of specific action, perceptual categorization, conditioning, autonomous exploration) of the robots. Artificial value system essentially tries to modulate the behavior of an autonomous system (e.g Fig. 1.2 shows the modulating effect of value system in the SAIL robot cognitive architecture [72]).

10

M. Begum and F. Karray

Fig. 1.2. Modulating role of value system in the cognitive architecture of SAIL robot in university of Michigan (adapted from [72])

Considering value system in the primates as benchmark [73, 74], the properties expected in an artificial value system for the autonomous robots can be summarized as follows. • Prediction: Predictive power is an important requirement of a value system. The system should be able to infer the consequences of its own action before actually executing it. This sort of predictive action planning requires the capacity to apply reasoning and knowledge-base to analyze perceptual information. • Task non-specific: An artificial value system must be general and should operate in a task non-specific manner. It must be unsupervised and should enable a robot to learn by itself (from the consequence of its action) without any help from human instructor. Instructional scaffolding is a very popular mechanism in the primates (specially, human being) to boost value system development during the very early stage but the supervision obtained through scaffolding is expected to be decayed gradually in order to facilitate the natural cognitive development [80]. In a similar manner, limited support is allowed during the early development of artificial value system in robotic agents but, gradually, the system should be capable of evolving in a an autonomous manner based on the interaction with its environments. • Developmental: Similar to the value system in biological entity, the artificial value system might have innate and acquired components. Innate value, in artificial systems, is modeled through assigning prior determined importance to different actions. The acquired value, on the other hand, should generate developmentally. An artificial value system must have the plasticity to continuously encode the feedback that it receives from its interaction with environment. This encoding of experience should be in such a way that facilitate emergence of adaptive behavior. • Value-based Learning: Learning is the prime mechanism which get modulated through the activities of value system. An artificial values system, therefore, should be capable of generating modulating signals to adapt various learning parameters, e.g bias (positive or negative), learning rate.

1 Computational Intelligence Techniques in Bio-inspired Robotics

11

Table 1.1. Research on artificial value system for autonomous robots

Lead research

Synopsis

CI technique

Robot

Value system [13]

Development of a neuromodulatory system based on conditioning and categorization

NN (continuous) firing rate model

Khepera (Monad)

Value system Development of Q-learning for developmental motivation and robot action planning [71, 72, 75] through value system

HR (SAIL robot)

Intelligent Adaptive Curiosity [76, 77]

Intrinsic motivation NN or generation for any standard action planning learning algorithm

PR (AIBO)

Adaptive behavior [78]

Development of a self-modulating value system

NN

MR (DARWIN V)

Value system Value guided for categorization exploration [79] and perceptual categorization

NN

Khepera

PR: Pet Robot

HR: Humanoid Robot

MR: Mobile Robot

Table 1.1 summarizes the research on value system design in autonomous robots. The initial attempts to design artificial value system were made by a group of researchers using robotic systems as a testbed to simulate the theories of primates nervous system [59, 81, 82, 83]. The efforts on designing artificial value system with the goal of developing human-like intelligence in robotic systems, however, are very limited. With the advancement of developmental robotics, the value system design for autonomous robot is gradually gaining attention of the bio-inspired robotics researchers [13, 71, 72, 75, 76, 77, 78, 79]. Artificial Neural networks generally dominate as a design tool in the development of artificial value system. The reason behind this, partly, is our current understanding about the operation of the primates value system. The findings on developmental plasticity suggest that experience drastically changes the ease with which neurons interact with each other [10, 11, 37]. The architecture of artificial neural networks provides a suitable way to model this synaptic modulation as adaptive weight adjustment among connecting neural processing elements.

12

1.4.3

M. Begum and F. Karray

Visual Attention

Attention, in a biological system, refers to the process of selecting a set of relevant information for further processing. It plays a key role in the survival and normal operation of the biological entities as the gate-keeping activity of attention saves the information processing unit (generally the brain) from being flooded with enormous amount of information. Computational modeling of the primates attention mechanism (more accurately, visual attention mechanism) has become a very popular research area in the recent years due to it potential applications in computational neuroscience, computer vision, automated video surveillance, and AI robotics. The purpose of attention mechanism, when applied on the artificial systems, is essentially the same as that in the biological systems: focusing on behaviorally relevant information for further processing. A properly designed attention system provides a task-executing robot with the capacity to blend with human in natural human environment. Extensive research has been performed on the visual attention mechanism of the primates, both in psychology [65, 68, 84, 85, 86, 87, 88, 89] and neurobiology [90, 91, 92, 93, 94, 95]. Based on these research a good number of bio-inspired computational model of visual attention has been proposed in computer vision [96, 97, 98, 99, 100, 101, 102, 103, 104] and in computational neuroscience [105, 106, 107, 108, 109]. The models in computer vision mostly focus on the technical aspects of the attentional mechanism while the goal of the models in computational neuroscience is to simulate the findings from the primates behavioral data. In the recent years, with the emergence of bio-inspired robotics, the robotic researches have started working on developing artificial attention mechanism for embodied physical robots. Accordingly, a number of attention models has been proposed in robotics literature which are dealing with embedding artificial attention mechanism in robotic systems [103, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121]. This chapter categorizes the existing literature on visual attention into three different groups. • Theoretical model for visual attention: This group includes the theoretical models of visual attention which are based on CI technique and have been implemented in robotic systems for attention modeling. The most notable work in this category is the neuromorphic vision tool-kit (NVT) [99, 122](Fig. 1.3). NVT is an extension of the first computational model of visual attention proposed in [97]. It uses winnertake-all type neural network (WTA-NN) for selective attention modeling. The WTA NN have earned huge popularity in the later years and is a key tool for attention modeling in majority of the existing visual attention models, although it has been considerably modified from its original form as proposed in [97]. A good number of works in robotics uses different variants of WTA NN and NVT to implement visual attention [111, 113, 123, 124, 125]. • Robotic model of visual attention: The visual attention models developed specifically for robotic systems are included in this category [111, 116, 126, 127, 128, 129, 130]. A considerable number of model in this category are based on NVT and WTA-NN. Apart from that, competitive NN, radial basis function (RBF), self organizing map (SOM), arrays of neural processing elements (PE) are popular CI tools used for visual attention modeling.

1 Computational Intelligence Techniques in Bio-inspired Robotics

13

Fig. 1.3. Neuromorphic vision toolkit: an architecture of visual attention mechanism (adapted from [99])

• Task specific visual attention model: Focusing attention on an object of interest is the pre-requisite of learning skill development. A substantial number of robotic system, therefore, use visual attention as the basic mechanism to achieve different cognitive skills. These systems generally develop task-specific models of visual attention where the task in hand might be object manipulation [131, 132, 133, 134], different forms of human robot interaction (HRI) [61, 135, 136, 137, 138, 139], and robot navigation [125]. For quick reference, tables 1.2, 1.3, and 1.4 summarize the leading researches under each of the above mentioned categories, respectively. The existing models of visual attention have achieved significant success addressing the issues of ‘what to attend’, ‘when to attend’, and ‘how to attend’[103]. The majority of the visual attention models for the autonomous robots use different kind of artificial neural networks. The reason behind this, partly, is that the earlier computational models of visual attention were NN-based. The another reason is that the enriched formulation of artificial neurons act as a very strong tool to mimic (albeit reduced complexity) the neural circuity of visual cortex dedicated to attention selection. As the goal of bio-inspired robotics is to mimic the functionality of biological system rather than the exact mechanism, there are possibilities for other computational techniques to be used in developing visual attention models. For instance, fuzzy reasoning could play a substantial role in visual search modeling because of its tolerance

14

M. Begum and F. Karray Table 1.2. Theoretical models of visual attention

Lead research

Synopsis

CI technique

Adaptive Resonance theory [96]

A neural network Neural network based on attention mechanism. Combines top-down and bottom -up attention

Koch’s Model [98], Computational model Winner-take-all Neuromorphic of visual attention. neural network vision toolImplements saliency (WTA NN) kit (NVT) map and inhibition [99, 122] of return (IOR) Selective tuning model [101]

Conncectionist model Neural network of visual attention

Feature-gate model [102]

Conncectionist model Spatially distributed, of visual attention. hierarchical neural Combines feature network integration theory [84] and guided search [85]

to partial truth [56]. Further, statistical learning methodologies offer great versatility in the learning of attention based cognitive skill. Specifically the probabilistic architecture provides the most suitable way to handle the perceptual and decision uncertainty inherent in robotics applications. We have not yet found a general purpose model of visual attention capable to tackle all the existing challenges. Integration of different CI techniques might be a very intriguing way to walk toward this goal since it may capture different functionalities that can not be handled by an isolated CI or AI approach. 1.4.4

Social Cognition

Social cognition development in autonomous systems is a widely explored research area in bio-robotics. The research on socially interactive robots has been experiencing a steady development since the birth of biologically inspired robotics. A number of significant achievements have been made, but researchers are interested about very high-level social interaction conducted by robotic agents. A number of different classifications and taxonomies are available for socially interactive robots ([33, 36, 158, 159] are suitable for extensive survey). This document focuses on the body of research dedicated to design task non-specific social cognition in robotic agents as a part of fulfilling the goal of designing human-like cognition in robotic systems. Such research works generally involve the theories of social development of the human proposed in psychology, ethology, linguistics, neuroscience and developmental cognitive neuroscience. The target robotic systems for these approaches may possess physical embodiment but

1 Computational Intelligence Techniques in Bio-inspired Robotics Table 1.3. Robotic model of visual attention

Lead research

Synopsis

CI technique

Distributed attention [113]

Multi-processor attention model

WTA NN 7 DOF (Koch’s model) HH

Object-based attention [111]

Considers object WTA NN as elemental unit (NVT) for attention. Evaluates objectbased saliency map.

Attention network [140]

Efficient visual search by robotic system

Neural network CH (Feature gate model)

Overt attention [110]

Implements space -based saccade and IOR

Competitive NN HR [141]

Attention and Implements space saccade generation -based saccade [123]

WTA NN CH (Koch’s model)

Autonomous mental development (AMD) [58, 142] [143, 144, 145]

Hierarchically HR connected neural (SAIL robot) processing elements (PE)

A developmental design of visual attention mediated by subsumption [54]

Robot

HR (Babybot)

NeuroBotic Object classifier system based on task[116, 126, 127, 128] relevance

Hierarchical MR radial basis (Peoplebot) function (RBF)

MirrorBot project [129, 130]

A distributed model of spatial attention

Continuum neural field theory

MR (peoplebot)

Multi modal attention [146, 147]

Directing visual attention to respond stimuli arising through different modality

NN

CRA

Brain based devices [32, 59] [148]

Attention-based network of perceptual mean firingcategorization rate neurons and visual binding

MR (Darwin VII)

HH: Humanoid Head

HR: Humanoid Robot

CRA: Camera on Robot Arm

CH: Camera Head

15

16

M. Begum and F. Karray Table 1.4. Task-specific model of visual attention

Lead research

Synopsis

CI technique

Robot

Attention for manipulation and reasoning [131]

visual attention Coupled dynamic MR helps to perform neural field manipulation task and plan task activities

Attention guided visual attention object manipulation mechanism help by robotic hand a robot to focus [132] on target object for grasping task

NN

CH

Visual attention for mobile robots [133, 134]

A connectionist attention model for task executing robots

Spiking and mean firingrate neurons (selective tuning model)

MR (Khepera)

Attention guided robot navigation [125]

attention mechanism is used to calculate the orientation of a robotic system

WTA NN and RBF NN

MR

Intelligent machine architecture for HRI [61, 135] [136, 137]

RBF associated with sensory egosphere implements visual attention

NN

HR (ISAC)

not necessarily human-like morphology or face, although these two have been proved to facilitate the social interaction between robots and humans. This type of socially interactive robot has been termed in the literature as ‘sociable robot’ [159]. The studies in psychology and neuroscience on social cognition development in humans identify the active interaction with other human as well as with the environment as the key criteria of social cognition development in human child [10, 80, 160]. The AI robotics research on social cognition development in autonomous systems, therefore, is primarily focusing on developing mechanisms used by the robots to interact with humans. The efforts in this area can be categorized into the following two groups. • Joint attention: Shared or joint attention refers to the ability to intentionally attend to an object/region of mutual interest. This is the ability that human infants start to achieve as early as six months [161]. Joint attention is an extremely complex process demanding the capacity to understand other peoples perspective and intention. It is a fundamental requirement for developing social cognition. An impressive body

1 Computational Intelligence Techniques in Bio-inspired Robotics

17

Table 1.5. Research on joint visual attention

Lead research

Synopsis

CI technique

Active learning of Joint attention [149, 150]

Learning joint RBF NN attention through pointing and reaching by robots

Robot HR (NICO)

Learning Passive learning Multilayer joint attention of joint attention NN, [151, 152, 153] through bootstrap Q-learning mechanism

HH

Attention for social communication [124, 154] [155, 156]

HH (Kismet)

Joint attention WTA NN between human (NVT) and robots based on eye contact

Join attention Joint attention Self organizing MR learning for HRI between human map (SOM) [138, 139] and robots based on gaze following Joint attention for imitation [157]

Imitation is Recurrent achieved through NN (RNN) joint attention

HR (Sony QRIO)

of research work has been reported in the past few years on developing the joint attention capability in robotic systems. Table 1.5 provides a brief summary of the research on joint attention. Gaze following technique has been used in a number of works to identify the intent of the partner in a social engagement [124, 138, 139, 154, 155, 156]. Using the embodiment of the robots to attract the attention of partner (e.g. through pointing fingers or waving hand) is another popular way of achieving joint attention [149, 150]. • Social imitation: Learning through imitation has gained much popularity in bioinspired robotics. Imitation provides a powerful way of developing social behavior. More specifically, the sensori-motor skills required to generate different types of social cues (e.g. pointing, reaching, waving) and task sequence learning are the two major areas where imitation based interaction between humans and robots plays a very significant role. The success of imitation based learning, however, depends on the integration with other cognitive abilities of a robotic system, e.g. attention, values system. The task of skilled imitation, therefore, has been termed as a ‘hard problem’ in [162] subjected to the issues ‘what to imitate’, ‘when to imitate’, ‘how to imitate’, and ‘what is the evaluation criteria of imitation’. Table 1.6 presents a brief summary of research on imitation based learning as a means of developing social cognition in autonomous robots.

18

M. Begum and F. Karray Table 1.6. Research on social imitation

Lead research Synopsis

CI technique

Robot

Imitation learning [163]

Motion sequence generated by a human teacher to perform simple tasks are observed and imitated

Bayesian HR learning, (Fujitsu Hidden Markov HOAP-2) model (HMM)

Learning motor skills through imitation [164]

A training phase trains the robot with different motor skills. The robot imitate learned skills in future engagements

RNN

Imitation behavior [165]

Attention guided imitation of hand posture

Reinforcement HR learning, HMM (Kenta)

HR (Sony QRIO)

Learning Manipulation task through is learned through demonstration imitating the task [166, 167] sequence executed by a human teacher

HMM, probabilistic analysis

HR (HOAP-2)

Early imitation ability [168]

Developmental approach of imitation. Learning imitation through self exploration

NN, causal learning

HR (H3)

Gaze shift learning by imitation [169]

Learning of gaze shifting inspired by saliency through imitating caregiver

HMM

MR (Pioneer)

Learning by imitation [170, 171]

Architecture NN for learning motion primitives through viewpoint transformation and visuo-motor mapping

HR

1 Computational Intelligence Techniques in Bio-inspired Robotics

19

Fig. 1.4. The humanoid robot HOAP-2 engaged in an imitation task (adapted from [163])

Similar to value system and visual attention, the research on social cognition is influenced by different types of artificial neural networks [164, 170, 171], although probability-based algorithms have earned much popularity in the design of social imitation capacity [163, 165, 166, 167, 169].

1.5 Conclusion The ultimate goal of bio-inspired robotics is to build human-like machine, in terms of morphology as well as functionality. The solemn purpose of developing human-like machine is the hope that their human-like intelligence will help them to blend naturally with humans and be a great assistance in human’s everyday operations. For instance, intelligent life-like machines will tremendously improve the quality of human life through their assistance in industry, health care, search and rescue, cleaning and household works, space-operation, education and entertainment. Design of human-like machine remains a challenge for the next generation autonomous robotics. Mimicking the mechanism of intelligence development in the biological creatures (specially in human) is currently seen to be a better option in resolving this challenge as compared to the earlier efforts of ‘hand-coding’ the intelligence in machines. In this endeavor one important issue, as mentioned in [172], is mimicking the intelligent behavior of a biological creatures does not guarantee an automatic emergence of intelligence in a target machine. Rather we have to find the underlying principle that governs the emergence of intelligent behavior. Mimicking the principle, rather than the outcome, might lead us to our goal of designing life-like machine. At this point, the CI techniques show us a relatively better hope. Majority of the CI techniques are developed based on different

20

M. Begum and F. Karray

elegant aspects of the biological creatures, e.g. morphological structure, thinking process, evolutionary selection. The CI techniques, therefore, blend naturally with the goal of bio-inspired robotics. This chapter have summarized some efforts to reach the goal of designing humanlike machine mediated by different CI techniques. The major focus was on the research on cognitive modeling for autonomous robots. Majority of the research in cognitive modeling is intended to model the small pieces of cognition, e.g. attention, value system, social interaction. Different kind of neural networks have gained more popularity and acceptability in this endeavor as compared to the other CI tools, e.g. fuzzy reasoning, genetic algorithm and probabilistic reasoning. The integration of neural network with other powerful CI tools might improve the scenario significantly, although such efforts are still in the early stages. After designing the small pieces of cognition, the most crucial part is to integrate them in a single architecture and have them interact with each other in a meaningful way so that we can see human-like manifestations of cognition as their integrated outcome. Such an architecture is yet to be developed but we hope that the CI techniques will contribute in its design, the same way they have been contributing in modeling the discrete cognitive abilities of autonomous robots.

References 1. Karray, F., Silva, C.D.: Soft Computing And Intelligent Systems Design. Addison Wesley, Pearson (2004) 2. Nolfi, S., Floreano, D.: Evolutionary Robotics: The Biology, Intelligence, and Technology of Self-Organizing Machines. Bradford Books (2000) 3. Jain, L.C., Fukuda, T.: Soft Computing for Intelligent Robotic Systems. Physica- Verlag (1998) 4. Watanabe, K., Hashem, M.M.A.: Evolutionary Computations: New Algorithms and Their Applications to Evolutionary Robotics. Springer, Heidelberg (2004) 5. Teshnehlab, M., Watanabe, K.: Intelligent Control Based on Flexible Neural Network. Kluwer Academic Publishers, Dordrecht (1999) 6. Watanabe, K., Izumi, K.: Biomimetic machines: Motivations to the research. In: SICE Annual Conference, pp. 1098–1103 (2004) 7. Haugeland, J.: Artificial Intelligence: The Very Idea. MIT Press, Cambridge (1985) 8. Newell, A.: Unified Theories of Cognition. Harvard University Press (1990) 9. Brooks, R.A.: Intelligence without representation. Artificial Intelligence 47, 139–160 (1991) 10. Johnson, M.H.: Developmental Cognitive Neuroscience. Blackwell Publishing, Malden (1997) 11. Nelson, C.A., Hann, M., Thomas, K.M.: Neuroscience of cognitive development: the role of experience and the developing brain. John Wiley & Sons, New Jersey (2006) 12. Pfeifer, R.: Robots as cognitive tools. International Journal of Cognition and Technology 1, 125–143 (2002) 13. Sporns, O., Alexander, W.H.: Neuro-modulation and plasticity in an autonomous robot. Neural Networks 15, 761–774 (2002) 14. de Castro, L.N., Timmis, J.: Artificial Immune Systems: A New Computational Intelligence Approach. Springer, Heidelberg (2002) 15. Karnik, N.N., Mendel, J.M., Liang, Q.: Type-2 fuzzy logic systems. IEEE Transactions on Fuzzy Systems 7, 643–658 (1999)

1 Computational Intelligence Techniques in Bio-inspired Robotics

21

16. Ziemke, T., Sharkey, N.E.: Artificial neural networks for robot learning. Autonomous Robots 7 (1999) 17. Haykin, S.: Neural Networks: A Comprehensive Foundation. Prentice Hall, Englewood Cliffs (1998) 18. Yen, J., Langari, R.: Fuzzy Logic: Intelligence, Control, and Information. Prentice Hall, United States (1998) 19. Goldberg, D.E.: Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley Professional, Reading (1989) 20. Engelbrecht, A.P.: Fundamentals of Computational Swarm Intelligence. Wiley, Chichester (2006) 21. Dorigo, M., St¨utzle, T.: Ant Colony Optimization. MIT press, Cambridge (2004) 22. Thrun, S., Burgard, W., Fox, D. (eds.): Probabilistic Robotics. MIT Press, Cambridge (2005) 23. Wrinkler, R.L.: An introduction to Bayesian inference and decision. Wiley, New York (2001) 24. Tanner, M.A.: Tools for statistical inference: methods for the exploration of posterior distributions and likelihood functions. Springer, New York (1996) 25. Pearl, J.: Causality: Models, Reasoning, and Inference. Cambridge University press, UK (2001) 26. Mishkin, A.H., Morrison, J.C., Nguyen, T.T., Stone, H.W., Cooper, B.K., Wilcox, B.H.: Experiences with operations and autonomy of the mars pathfinder microrover. In: IEEE Aerospace Conference, vol. 2, pp. 337–351 (1998) 27. Thrun, S., et al.: Stanley: The robot that won the DARPA grand challenge. Journal of Field Robotics 23, 661–692 (2006) 28. Krupa, A., Gangloff, J., Doignon, C., Mathelin, M., Morel, G., Leroy, J., Soler, L., Marescaux, J.: Autonomous 3-d positioning of surgical instruments in robotized laparoscopic surgery using visual servoing. IEEE Transactions on Robotics and Automation 19, 842–853 (2003) 29. Miyawaki, F., Masamune, K., Suzuki, S., Yoshimitsu, K., Vain, J.: Scrub nurse robot system-intra operative motion analysis of a scrub nurse and timed-automata-based model for surgery. IEEE Transactions on Industrial Electronics 52, 1227–1235 (2005) 30. Brachman, R.J.: More than the sum of its parts. AI Magazine 27, 19–34 (2006) 31. Webb, B., Consi, T.R.: Biorobotic. MIT Press, Cambridge (2001) 32. Krichmar, J.L., Seth, A.K., Nitz, D.A., Fleischer, J.G., Edelman, G.M.: Spatial navigation and causal analysis in a brain-based device modeling cortical-hippocampal interactions. Neuroinformatics 3, 197–222 (2005) 33. Lungarella, M., Metta, G., Pfeifer, R., Sandini, G.: Developmental robotics: a survey. Connection Science 15, 151–190 (2003) 34. Beer, R.D., Chiel, H.J., Quinn, R.D., Ritzmann, R.E.: Biorobotic approaches to the study of the motor system. Current Neurobiology 8, 777–782 (1998) 35. Iida, F., Rummel, J., Seyfarth, A.: Bipedal walking and running with compliant legs. In: IEEE International Conference on Robotics and Automation, pp. 3970–3975 (2007) 36. Bar-Cohen, Y., Breazeal, C.: Biologically Inspired Intelligent Robots. SPIE Press (2003) 37. Drubach, D.: The brain explained. Prentice Hall Health, New Jersey (2000) 38. Wodnicki, R., Roberts, G.W., Levine, M.D.: A log-polar image sensor fabricated in a standard 1.2 ASIC CMOS process. IEEE Journal of Solid-State Circuits 32, 1274–1277 (1997) 39. Payeur, P., Pasca, C., Cretu, A., Petriu, E.M.: Intelligent haptic sensor system for robotic manipulation. IEEE Transactions on Instrumentation and Measurement 54, 1583–1592 (2005)

22

M. Begum and F. Karray

40. Motamed, M., Yan, J.: A review of biological, biomimetic and miniature force sensing for microflight. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3939–3946 (2005) 41. Bar-Cohen, Y.: Biologically inspired intelligent robots using artificial muscles. In: IEEE International Conference on MEMS, NANO and Smart Systems, pp. 2–8 (2003) 42. Matsuyama, Y., Hirai, S.: Analysis of circular robot jumping by body deformation. In: IEEE International Conference on Robotics and Automation, pp. 1968–1973 (2007) 43. Hirai, K.: The HONDA humanoid robot: development of future perspective. Industrial robots 26, 260–266 (1999) 44. Doi, M., Hmegawa, Y., Fukuda, T.: Passive dynamic autonomous control of bipedal walking. In: IEEE/RAS International Conference on Humanoid Robots, vol. 8, pp. 811–829 (2004) 45. Jenkin, O.C., Mataric, M., Weber, S.: Primitive-based movement classification for humanoid imitation. In: IEEE/RAS International Conference on Humanoid Robots (2000) 46. Sandini, G., Metta, G., Konczak, J.: Human sensori-motor development and artificial systems. In: Proceedings of the Int. Symp. on AI, Robotics, and Intellectual Human Activity Support for Applications, pp. 303–314 (1997) 47. Shen, W., Salemi, B., Will, P.: Hormone-inspired adaptive communication and distributed control for conro self-reconfigurable robots. IEEE Transaction on Robotics and Automation 18, 700–712 (2002) 48. Yim, M., Duff, D.G., Roufas, K.D.: Polybot: A modular reconfigurable robot. In: IEEE International Conference on Robotics and Automation, pp. 514–520 (2000) 49. Sugiyama, Y., Hirai, S.: Crawling and jumping by a deformable robot. International Journal of Robotics Research 25, 603–620 (2006) 50. Sangbae, K., Asbeck, A., Cutkosky, M., Provancher, W.: SpinybotII: climbing hard walls with compliant microspines. In: International Conference on Advanced Robotics, pp. 601– 606 (2005) 51. Spenko, M., Cutkosky, M., Majidi, C., Fearingn, R., Groff, R., Autumn, K.: Foot design and integration for bio-inspired climbing robots. In: Proceedings of SPIE: Unmanned Systems Technology VIII, 6230 (2006) 52. Rosenschein, S., Kaelbling, L.P.: The synthesis of machine with provable epistemic properties, pp. 83–98. Morgan Kaufman, Los Altos (1986) 53. Arkin, R.: Toward the unification of navigational planning and reactive control. In: In Proceedings of AAAI, pp. 1–5 (1989) 54. Brooks, R.A.: A robust layered control system for a mobile robot. IEEE Transaction on Robotics and Automation 47, 139–160 (1991) 55. Weng, J.: The developmental approach to artificial intelligence: Concepts, developmental algorithms and experimental results. In: NSF Design and Manufacturing Grantees Conference (1998) 56. Albus, J.S., Meystel, A.M.: Intelligent Systems: Architecture, Design, and Control. WileyInterscience, Hoboken (2001) 57. Hendler, J.: Computers play chess; humans play go. IEEE Intelligent Systems 21, 2–3 (2006) 58. Weng, J., McClelland, J., Pentland, A., Sporns, O., Stockman, I., Sur, M., Thelen, E.: Autonomous mental development by robots and animals. Science 291, 599–600 (2001) 59. Krichmar, J.L., Edelman, G.M.: Brain-based devices: intelligent systems based on principles of the nervous system. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 940–945 (2003)

1 Computational Intelligence Techniques in Bio-inspired Robotics

23

60. Asada, M., MacDormanb, K.F., Ishigurob, H., Kuniyoshi, Y.: Cognitive developmental robotics as a new paradigm for the design of humanoid robots. Robotics and Autonomous Systems 37, 185–193 (2001) 61. Kawamura, K., Noelle, D.C., Hambuchen, K.A., Rogers, T.E.: A multi-agent approach to self-reflection for cognitive robots. In: IEEE International Workshop on Robot and Human Interactive Communication, pp. 568–575 (2003) 62. Brachman, R.J.: Systems that know what they’re doing. IEEE Intelligent Systems 17, 67–71 (2002) 63. Belliveau, J., Kennedy, D., McKinstry, R., Buchbinder, B., Weisskoff, R., Cohen, M., Vevea, J., Brady, T., Rosen, B.: Functional mapping of the human visual cortex by magnetic resonance imaging. Science 254, 716–719 (1991) 64. Strauss, L.G., Conti, P.: The applications of PET in clinical oncology. Neural Medicine 32, 623–648 (1991) 65. Posner, M.I.: The attention system of the human brain. Annual Reviews, Neuroscience 13, 25–42 (1990) 66. Cohen, S.B., Flusberg, H.T., Cohen, D.J.: Understanding other minds: perspectives from developmental cognitive neuroscience. Oxford University Press, New York (2000) 67. Geary, D.C.: The origin of mind: evolution of brain, cognition and general intelligence. American Psychological Association, Washington (2002) 68. Koch, C.: The quest for consciousness. Roberts & Company, Colorado (2004) 69. Zlatev, J., Balkenius, C.: Introduction: Why epigenetic robotics. In: International Workshop on Epigenetic Robotics (2001) 70. Friston, K.J., Tononi, G., Reeke, G.N., Sporns, O., Edelman, G.M.: Value dependent selection in the brain: Simulation in a synthetic neural model. Neuroscience 59, 229–243 (1994) 71. Huang, X., Weng, J.: Value system development for a robot. In: IEEE International Conference on Neural Networks, pp. 2883–2888 (2004) 72. Huang, X., Weng, J.: Novelty and reinforcement learning in the value system of developmental robots. In: Proceedings of International Workshop on Epigenetic Robotics, pp. 47–55 (2002) 73. Schultz, W.: Multiple reward signal in the brain. Nature Reviews, Neuroscience 1, 199–207 (2000) 74. Edelman, G.M.: Group selection and phasic reentrant signaling: a theory of higher brain function, pp. 51–100. MIT Press, Cambridge (1978) 75. Zhang, Y., Weng, J.: Action chaining by a developmental robot with a value system. In: IEEE International Conference on Development and Learning, pp. 53–60 (2002) 76. Oudeyer, P.-Y., Kaplan, F., Hafner, V., Whyte, A.: The playground experiment: Taskindependent development of a curious robot. In: AAAI Spring Symposium Workshop on Developmental Robotics (2005) 77. Oudeyer, P.-Y., Kaplan, F.: Intelligent adaptive curiosity: a source of self-development. In: International Workshop on Epigenetic Robotics (2004) 78. Sporns, O., Alm´assy, N., Edelman, G.M.: Plasticity in value systems and its role in adaptive behavior. Adaptive Behavior 12, 129–148 (2000) 79. Scheier, C., Lambrinos, D.: Categorization in a real-world agent using haptic exploration and active perception. In: Proceedings of the International Conference on Simulation of Adaptive Behavior, pp. 65–75 (1996) 80. Vygotsky, L. (ed.): Thought and Language. MIT Press, Cambridge (1962) 81. Almassy, N., Edelman, G.M., Sporns, O.: Behavioral constraints in the development of neuronal properties: a cortical model embedded in a real-world device. Cerebral Cortex 8, 346–361 (1998)

24

M. Begum and F. Karray

82. Edelman, G.M., Reeke, G.N., Gall, E., Tononi, G., Williams, D., Sporns, O.: Synthetic neural modeling applied to a real-world arttifact. Proceedings of National Academic Science 89, 7267–7271 (1992) 83. Krichmar, J.L., Snook, J.A., Edelman, G.M., Sporns, O.: Experience dependent perceptual categorization in a behaving real-world device, pp. 41–50. MIT Press, Cambridge (2000) 84. Treisman, A.M., Gelade, G.: A feature intigration theory of attention. Cognitive Psychology 12, 97–136 (1980) 85. Wolfe, J.M.: Visual search in continuous naturalistic stimuli. Vision Research 34, 1187– 1195 (1994) 86. Eriksen, C.W., Yeh, Y.Y.: Allocation of attention in the visual field. Journal of Experimental Psychology: Human Perception and Performance 11, 583–597 (1985) 87. Posner, M.I.: Orienting of attention. Journal of Experimental Psychology 32, 3–25 (1980) 88. Posner, M.I., Rothbart, M.K.: Educating the human brain. American Psychological Association, Washington (2007) 89. Bundense, C.: A theory of visual attention. Psychologycal Review 97, 523–527 (1990) 90. Desimone, R.: Visual attention mediated by biased competition in extrastriate visual cortex. Philosophical transactions of the Royal Society of London, Series B, Biological sciences 353, 1245–1255 (1998) 91. Chelazzi, L., Duncan, J., Miller, E.K., Desimone, R.: Responses of neurons in inferior temporal cortex during memory guided visual search. Journal of Neurophysiology 80(6), 2918– 2940 (1998) 92. Kastner, S., Ungerleider, L.G.: The neural basis of biased competition in human visual cortex. Vision Research 40, 1489–1506 (2000) 93. Duncan, J.: Selective attention and the organization of visual information. Journal of Experimental Psychology 113, 501–513 (1984) 94. Luck, S., Chelazzi, L., Hillyard, S.A., Desimone, R.: Neural mechanisms for directed visual attention. Journal of Neurophysiology 77, 24–42 (1997) 95. Desimone, R., Duncan, J.: Neural mechanisms of selective visual attention. Annual Reviews of Neurscience 18, 193–222 (1995) 96. Carpenter, G.A., Grossberg, S.: ART2: Self-organization of stable category recognition codes for analog input patterns. Applied Optics 26(23), 4919–4930 (1987) 97. Itti, L., Koch, C.: Computational modelling of visual attention. Nature Reviews: Neuroscience 2, 194–203 (2001) 98. Koch, C., Ullman, S.: Shifts in selective visual attention: toward the underlying neural circuitry. Human Neurobiology 4, 219–227 (1985) 99. Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Transaction on Pattern Analysis and Machine Intelligence 20, 1254–1259 (1998) 100. Itti, L., Rees, G., Tsotsos, J.K. (eds.): Neurobiology of Attention. Elsevier Academic Press, Amsterdam (2005) 101. Tsotsos, J.K., Culhane, S.M., Winky, Y.K.W., Yuzhong, L., Davis, N., Nuflo, F.: Modeling visual attention via selective tuning. Artificial Intelligence 78, 507–545 (1995) 102. Cave, K.R.: The featuregate model of visual selection. Psychological Research 62, 182–194 (1999) 103. Sun, Y., Fisher, R.: Object-based visual attention for computer vision. Artificial Intelligence 146, 77–123 (2003) 104. Frintrop, S.: VOCUS: A Visual Attention System for Object Detection and Goal-Directed Search. LNCS (LNAI), vol. 3899. Springer, Heidelberg (2006)

1 Computational Intelligence Techniques in Bio-inspired Robotics

25

105. Deco, G.: Biased competition mechanisms for visual attention in a multimodular neurodynamical system. In: Wermter, S., Austin, J., Willshaw, D. (eds.) Emergent Neural Computational Architectures Based on Neuroscience. LNCS (LNAI), vol. 2036, pp. 114–126. Springer, Heidelberg (2001) 106. Deco, G., Rolls, E.T.: A neurodynamical cortical model of visual attention and invariant object recognition. Vision Research 44, 621–642 (2004) 107. Rolls, E.T., Deco, G. (eds.): Computational Neuroscience of Vision. Oxford University Press, Oxford (2002) 108. Lanyon, L.J., Denham, S.L.: A biased competition computational model of spatial and object-based attention mediating active visual search. Neurocomputing 58-60, 655–662 (2004) 109. Lanyon, L.J., Denham, S.L.: A model of active visual search with object-based attention guiding scan paths. Neural Networks 17, 873–897 (2004) 110. Vijayakumar, S., Conrad, J., Shibata, T., Schaal, S.: Overt visual attention for a humanoid robot. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2332– 2337 (2001) 111. Orabona, F., Metta, G., Sandini, G.: Object-based visual attention: A model for a behaving robot. In: IEEE Conference on Computer Vision and Pattern Recognition (2005) 112. Scassellati, B.M.: Foundations for a Theory of Mind for a Humanoid Robot. PhD thesis, Electrical Engineering and Computer Science, MIT (May 2001) 113. Ude, A., Wyart, V., Lin, L.-H., Cheng, G.: Distributed visual attention on a humanoid robot. In: IEEE-RAS International Conference on Humanoid Robots, pp. 381–386 (2005) 114. Driscoll, J.A., Peters, R.A., Cave, K.R.: A visual attention network for a humanoid robot. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1968–1974 (1998) 115. Breazeal, C., Edsinger, A., Fitzpatrick, P., Scassellati, B.: Active vision for sociable robots. IEEE Transaction on System, Man, and Cybernetics, Part A 31, 443–453 (2001) 116. Fay, R., Kaufmann, U., Knoblauch, A.: Combining Visual Attention. In: Object Recognition and Associative Information Processing in a Neurobotic System, pp. 117–142. Springer, Heidelberg (2005) 117. Vitay, J., Rougier, N.P., Alexandre, F.: A Distributed Model of Spatial Visual Attention, pp. 54–72. Springer, Heidelberg (2005) 118. Ban, S.-W., Niitsuma, H., Lee, M.: Bilogicaly inspired autonomous mental development model based on visual selective attention mechanism. In: AAAI Symposium on Developmental Robotics (2005) 119. Stasse, O., Kuniyoshi, Y., Cheng, G.: Development of a biologically inspired real-time visual attention system. In: IEEE International Workshop on Biologically Motivated Computer vision, pp. 150–159 (2000) 120. Manfredi, L., Maini, E.S., Dario, P., Laschi, C., Girard, B., Tabareau, N., Berthoz, A.: Implementation of neurophysiological model of saccadic eye movements on an anthropomorphic robotic head. In: IEEE-RAS International Conference on Humanoid Robotics, pp. 438–443 (2006) 121. Aziz, M.Z., Mertsching, B., Shafik, M.S., Stemmer, R.: Evaluation of visual attention models for robots. In: IEEE International Conference on Computer Vision Systems, pp. 20–26 (2006) 122. Itti, L., Koch, C.: A saliency based search mechanism for overt and covert shift of visual attention. Vision Research 40, 1489–1506 (2000) 123. Clark, J.J.: Spatial attention and saccadic camera motion. In: IEEE International Conference on Robotics and Automation, pp. 3247–3252 (1998)

26

M. Begum and F. Karray

124. Breazeal, C.F.: A motivational system for regulating human robot interaction. In: National conference of AAAI, pp. 54–61 (1998) 125. Baccon, J.C., Hafemeister, L., Gaussier, P.: A context and task dependent visual attention system to control a mobile robot. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 238–243 (2002) 126. Kaufmann, U., Fay, R., Schwenker, F., Palm, G.: Neural networks for visual object recognition based on selective attention. In: International SenseMaker Workshop on Life-Like Perception Systems (2005) 127. Fay, R., Kaufmann, U., Schwenker, F., Palm, G.: Learning object recognition in a neurobotic system. In: Workshop on Self Organization of AdaptiveBehavior SOAVE, pp. 198– 204 (2004) 128. Palm, G., Kaufmann, U., Fay, R.: A two-stage scheme for visual object recognition based on selective attention. In: European Conference on Visual Perception (ECVP) (2004) 129. Rougier, N.P., Vitay, J.: Emergence of attention within a neural population. Neural Networks 19, 573–581 (2006) 130. Vitay, J., Rougier, N.P., Alexandre, F.: A Distributed Model of Spatial Visual Attention, pp. 54–72. Springer, Heidelberg (2005) 131. Erlhangen, W., Bicho, E.: The dynamic neural field approach to cognitive robotics. Journal of Neural Engineering 3, 36–54 (2006) 132. Heidemann, G., Rae, R., Bekel, H., Bax, I., Ritter, H.: Integrating context-free and contextdependent attentional mechanisms for gestural object reference. Machine Vision and Applications 16, 64–73 (2002) 133. Scheier, C., Egner, S.: Visual attention in a mobile robot. In: Proceedings of the IEEE International Symposium on Industrial Electronics, vol. 1, pp. 48–52 (1997) 134. Egner, S., Scheier, C.: Feature binding through temporally correlated neural activity in a robot model of visual perception. In: Gerstner, W., Hasler, M., Germond, A., Nicoud, J.-D. (eds.) ICANN 1997. LNCS, vol. 1327, pp. 703–709. Springer, Heidelberg (1997) 135. Kawamura, K.: Cognitive approach to a human adaptive robot development. In: IEEE International Workshop on Robot and Human Interactive Communication, pp. 210–216 (2005) 136. Kawamura, K., Peters, R.A., Bodenheimer, R., Sarkar, N., Park, J., Spratley, A., Hambuchen, K.A.: Multi agent based cognitive robot architecture and its realization. International Journal of Humanoid Robotics 1, 65–93 (2004) 137. Ratanaswasd, P., Gordon, S., Dodd, W.: Cognitive control for robot task execution. In: IEEE International Workshop on Robot and Human Interactive Communication, pp. 440– 445 (2005) 138. Hasoda, K., Sumioh, H., Morita, A., Asada, M.: Acquisition of human-robot joint attention through real-time natural interaction. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2867–2872 (2004) 139. Nagai, Y., Hosoda, K., Morita, A., Asada, M.: A constructive model for the development of joint attention. IEEE Intelligent Systems 15, 211–229 (2003) 140. Driscoll, J.A., Peters, R.A., Cave, K.R.: A visual attention network for a humanoid robot. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (1998) 141. Amari, S., Arbib, M.A.: Competition and cooperation in neural nets, pp. 119–165. Academic Press, London (1977) 142. Zhang, N., Weng, J.: A developing sensory mapping for robots. In: IEEE International Conference on Development and Learning, pp. 13–20 (2002) 143. Weng, J., Zhang, Y.: Developmental robots: A new paradigm. In: International Workshop on Epigenetic Robotics (2002) 144. Weng, J.: Developmental robotics: Theory and experiments. International Journal of Humanoid Robotics 1, 199–234 (2004)

1 Computational Intelligence Techniques in Bio-inspired Robotics

27

145. Weng, J., Hwang, W.S.: From neural networks to the brain: Autonomous mental development. IEEE Computational Intelligence Magazine 1, 15–31 (2006) 146. Rucci, M., Bajcsy, R.: Learning visuo-tactile coordination in robotic systems. In: IEEE International Conference on Robotics and Automation, pp. 2678–2683 (1995) 147. Colombo, C., Rucci, M., Dario, P.: Attentive behavior in an anthropomorphic robot vision system. Journal of Robotics Autonomous Systems 12, 121–131 (1994) 148. Fleischer, J.G., Szatm´ary, B., Hutson, D.B., Moore, D.A., Snook, J.A., Edelman, G.M., Krichmar, J.L.: A neurally controlled robot competes and cooperates with humans in segway sooccer. In: IEEE International Conference on Robotics and Automation, pp. 3673– 3678 (2006) 149. Doniec, M.W., Sun, G., Scassellati, B.: Active learning of joint attention. In: IEEE/RAS International Conference on Humanoid Robots, pp. 34–39 (2006) 150. Doniec, M.W., Sun, G., Scassellati, B.: A demonstration of the efficiency of developmental learning. In: IEEE International Joint Conference on Neural Networks, pp. 5226–5232 (2006) 151. Nagai, Y., Hosoda, K., Asada, M.: Joint attention emerges through bootstrap learning. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 168–173 (2003) 152. Nagai, Y., Asada, M., Hosoda, K.: A developmental approach accelerates learning of joint attention. In: IEEE International Conference on Development and Learning, pp. 277–282 (2002) 153. Nagai, Y., Hosoda, K., Morita, A., Asada, M.: A constructive model for the development of joint attention. Connection Science 15, 211–229 (2003) 154. Breazeal, C., Fitzpatrick, P., Scassellati, B.: Active vision systems for sociable robots. IEEE Transaction on System, Man, and Cybernetics: Part A 31, 443–453 (2001) 155. Brooks, R.A., Breazeal, C.F., Marjanovic, M., Scassellati, B., Williamson, M.: The cog project: Building a humanoid robot. In: Nehaniv, C.L. (ed.) CMAA 1998. LNCS (LNAI), vol. 1562, pp. 52–87. Springer, Heidelberg (1999) 156. Scassellati, B.: Foundations for theory of mind for a humanoid robot. PhD thesis, MIT, Cambridge, MA (2001) 157. Tani, J., Ito, M.: Joint attention between a humanoid robot and users in imitation game. In: IEEE International Conference on Development and Learning, pp. 277–282 (2004) 158. Fong, T., Nourbakhsh, I., Dautenhahn, K.: A survey of socially interactive robots: concepts design and applications. Technical Report CMU- RI- TR- 02- 29, Robotic Institute, Carnegi Mellon University (2002) 159. Breazeal, C.: Toward sociable robots. Robotics and Autonomous Systems 42, 167–175 (2003) 160. Piaget, J.: The origins of intelligence. Routledge, New York (1953) 161. Butterworth, G., Jarret, B.: What minds have common in space: spatial mechanisms serving joint visual attention in infancy. British Journal of Developmental Psychology 9, 55–72 (1991) 162. Breazeal, C., Scassellati, B.: Robots that imitate humans. Adaptive Behanior 8, 49–74 (2002) 163. Shon, A.P., Storz, J.J., Rao, R.P.N.: Towards a real-time bayesian imitation system for a humanoid robot. In: IEEE International Conference on Robotics and Automation, pp. 2847– 2852 (2007) 164. Ito, M., Tani, J.: On-line imitative interaction with a humanoid robot using a dynamic neural network model of a mirror system. Adaptive Behavior 12, 93–115 (2004)

28

M. Begum and F. Karray

165. Yoshikai, T., Otake, N., Mizuuchi, I., Inaba, M., Inoue, H.: Development of an imitation behavior in humanoid kenta with reinforcement learning algorithm based on the attention during imitation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, vol. 2, pp. 1192–1197 (2004) 166. Calinon, S., Guenter, F., Billard, A.: On learning, representing and generalizing a task in a humanoid robot. IEEE Transactions on Systems, Man, and Cybernetics–Part B: Cybernetics 36, 286–298 (2007) 167. Calinon, S., Guenter, F., Billard, A.: Goal-directed imitation in a humanoid robot. In: IEEE International Conference on Robotics and Automation, pp. 299–304 (2005) 168. Kuniyoshi, Y., Yorozu, Y., Inaba, M., Inoue, H.: From visuo-motor self learning to early imitation - a neural architecture for humanoid learning. In: IEEE International Conference on Robotics and Automation, pp. 3132–3139 (2003) 169. Belardinelli, A., Pirri, F., Carbone, A.: Bottom-up gaze shifts and fixations learning by imitation. IEEE Transactions on Systems, Man, and Cybernetics–Part B: Cybernetics 37, 256–271 (2007) 170. Lopes, M., Victor, J.S.: Bottom-up gaze shifts and fixations learning by imitation. IEEE Transactions on Systems, Man, and Cybernetics–Part B: Cybernetics 35, 438–449 (2005) 171. Lopes, M., Victor, J.S.: A developmental road map for learning by imitation in robots. IEEE Transactions on Systems, Man, and Cybernetics–Part B: Cybernetics 37, 308–321 (2007) 172. Brooks, R.: Robot: The future of flesh and machines. Penguin Books, London (2003)

2 Swarm Intelligence for Collective Robotic Search Lisa L. Grant1 and Ganesh K. Venayagamoorthy2 1

2

Real-Time Power and Intelligent Systems Laboratory, Missouri University of Science and Technology, Rolla, MO 65409, USA [email protected] Real-Time Power and Intelligent Systems Laboratory, Missouri University of Science and Technology, Rolla, MO 65409, USA [email protected]

Abstract. This chapter presents three strategies for the navigation of a swarm of robots for a target search application in a hazardous environment. The strategies explored include greedy search and two computational intelligence techniques—particle swarm optimization and fuzzy logic. Results for the collective search are presented for simulated environments containing single and multiple targets, with and without obstacles. The proposed navigation strategies can be further developed and applied to real-world applications such as aiding in disaster recovery, detection of hazardous materials, and many other high-risk tasks.

2.1 Collective Robotic Search The use of robots to remove the need for human involvement in dangerous tasks has been an area of great interest [2, 6, 15, 21, 26]. For high-risk applications such as landmine and radioactive material detection, the robots need to be inexpensive and dispensable. A multi-robot search using simple or specialized robots has a higher chance for success than a search utilizing a single more complex and expensive robot attempting the task alone, by reducing the impact of robot loss, equipment failure, and communication failure on the system. One approach is to have a group of robots searching in parallel, but individually with little to no communication or interaction with the group, adding redundancy and fault tolerance to the system. Another common method to control a multi-robot system is the centralized approach where a central server or controller coordinates the robots actions for optimal performance in completing the specified task. This method can prove computationally intensive and vulnerable if the central controller becomes disabled or destroyed. Introducing coordination among the robots in an equally distributed system allows for a more efficient search and the ability to implement cooperative task completion. When performing a group-oriented search, the robot’s independent behavior is mainly based on its locally available sensory information allowing for adaptation to changing and hazardous environments, where utilization of the available swarm information increases the efficiency of the overall search. Performing a multiple robot based search also has the advantage of greater coverage of the search environment and simpler implementation [28]. In Collective Robotic Search (CRS), a group of small, inexpensive, and dispensable mobile robots collectively explore a dangerous environment in order to locate and D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 29–47. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

30

L.L. Grant and G.K. Venayagamoorthy

Global Level

Ld Neighborhood level Ld

Pd

Local Swarms

Fig. 2.1. Random deployment of robots in the search space

converge on a known target [7, 9]. For the CRS application in this study, a three-tier architecture is proposed for the implementation of a collective robotic search as shown in Fig. 2.1. The robots are randomly deployed in the search space. Since CRS is targeted to applications where human intervention is undesirable, deployment of the robots into an area of interest can be carried out by dropping the robots from an aircraft. As seen in Fig. 2.1, the search area can be divided into neighborhoods to restrict the robots motion to a smaller area based on a divide and conquer scheme [27]. The robots within each neighborhood form a local team or swarm. Each swarm operates individually of the swarms in the other neighborhoods. In environments with extreme terrain, this division of the search area ensures that the entire search space is explored and the targets are independently pursued. In some cases not all robots are able to access the entire search area due to natural boundary restrictions such as a large body of water or a mountain range. The division allows the robots to perform a fast, efficient and parallel search. The goal of CRS is to identify target locations and the targets can be of different kinds from one application to the other. The simulated environment in this study assumes that most desired targets have an identifiable gradient and uses light sources as the target goal. The robots are equipped with light sensors to detect the total light intensity at their current location relative to the location and number of target light sources. A diagram of the robot’s light intensity sensor is provided in Fig. 2.2. The swarm pursues the target closest to the robot with the highest overall fitness reading. The target fitness is based on the relative intensity, RI, given in Eq. (2.1) which is calculated using the intensity (I) read from each of the n targets in relation to the distance d from the robot to each target. RI =

In I1 + ...+ 2 (d1 ) (dn )2

(2.1)

2 Swarm Intelligence for Collective Robotic Search

31

Target with Intensity In

Target with Intensity I1 dn d1

Robot

Fig. 2.2. Intensity of targets measured at the robot

2.2 Navigation Strategies There are three strategies studied for navigation of the robots in the proposed CRS problem. A traditional approach known as greedy search is compared with two Computational Intelligence (CI) techniques, namely particle swarm optimization and fuzzy logic. These strategies are described in the following subsections. 2.2.1

Greedy Search - A Conventional Method

The greedy search algorithm is a conventional method used to follow a local gradient. When applied to CRS, a robot implementing a greedy search chooses its path based solely on the positions that orient it closest to its goal. The algorithm is called greedy because it grabs the best solution without calculating long-term cost. The robot moves to the position with highest intensity after sampling the intensities at positions around its current location. For the greedy search, essentially only one robot is needed to find the target because each robot acts as an individual agent. In a search utilizing a swarm of greedy robots, no communication between robots is necessary since each robot can locate the target on its own. Since the algorithm relies on little to no communication between robots, it is fairly robust against communication and robot loss [4]. Similar strategies have been proposed to produce agent convergence where global information and communication are not required, but global group behavior is still observed [5, 14, 18, 25]. The version of the greedy search algorithm used in this study is the ordinal greedy search method [24]. In order to find its next position, each robot maps out four possible positions, one in each of the ordinal directions (NE, SE, SW, and NW) from its current position. As the robot visits each of the four search points, it evaluates the target fitness at every position and ultimately chooses the search point with the best fitness or the shortest straight-line distance from the robot to the goal for its next position. The process repeats for every iteration, or position change. The robot is basically used as a mobile sensor. A detailed example of the mapping of one robot’s search points for a number of iterations using the greedy ordinal method is provided in Fig. 2.3. A flowchart of the greedy algorithm applied to a CRS problem is illustrated in Fig. 2.4.

32

L.L. Grant and G.K. Venayagamoorthy

Target (t+4)th move Pos. 4(t+4)

Pos. 1(t+4)

(t+3)th move Pos. 4(t+3) Pos. 1(t+3)

r2

(t+2)th move

(Next Pos. (t+4)) Pos. 4(t+2)

Pos. 1(t+2) Pos. 3(t+3)

(t+1)th move

(Next Pos. (t+3)) Pos. 4(t+1)

Pos. 1(t+1) Pos. 3(t+2)

(Next Pos. (t+2))

tth move

Pos. 4(t) (Next Pos. (t+1))

Pos. 3(t+1)

Pos. 1(t) Original Pos. (t)

Pos. 3(t)

r1 Pos. 2(t)

Fig. 2.3. A robot scanning positions for its next move using the greedy method

Start

Initialize search area. Initialize robots positions and velocities. Move to NE (Northeast) search point.

Check and store fitness of first search point. Move to SE (Southeast), SW (Southwest), and NW (Northwest) search points. Check and store fitness at each point. Compare fitness of search points to determine position of best fitness. Move to search point of best fitness.

Have all robots converged on the target?

No

Yes End

Fig. 2.4. Flowchart of greedy ordinal search as applied to CRS

2 Swarm Intelligence for Collective Robotic Search

2.2.2

33

Particle Swarm Optimization

Particle Swarm Optimization (PSO) is a swarm intelligence based algorithm developed by James Kennedy and Russell Eberhart in 1995 [10, 16, 17, 23]. This algorithm is based on the behavior of a school of fish or flock of birds. In PSO, better solutions are evolved through the collective interactions of the individuals in the swarm. For this study, PSO particles are referred to as robots. The success of the group depends on the success of the individuals in the swarm which occurs due to the social interactions between individual robots in the swarm. Each robot randomly searches the environment by updating its position using its own memory and social information gathered from the other robots. The robots are randomly deployed into a defined search area as previously described in Fig. 2.1. Each robot is given an initial random velocity and ‘flown’ through the problem space. The robots use memory to keep track of the position that resulted in their highest fitness reading, known as the ‘Pbest ’ value. The best value of all the ‘Pbest ’ values is defined as the local best position, ‘Lbest ’ with respect to the target. The velocities and positions are updated every iteration until all of the robots have converged on the target location [28]. PSO only requires four values for memory requirements: ‘Pbest ’, ‘Lbest ’, and the previous iteration’s velocity and position values. The PSO velocity and position update equations are given by Eqs. (2.2) and (2.3). The parameters w, c1 , and c2 are adjusted to determine to what degree the current velocity, ‘Pbest ’, or ‘Lbest ’ values affects a robot’s next position, respectively [22]. ‘rand1’ and ‘rand2’ are random numbers from a uniform distribution. Several versions of PSO are detailed in [8]. Vnew = w × Vcur + c1 × rand1 × (Pbest − Pcur ) + c2 × rand2 × (Lbest − Pcur ) Pnew = Pcur + Vnew

Where, Vnew Vcur Pnew Pcur w c1 and c2 rand

(2.2) (2.3)

- New velocity calculated for each robot - Velocity of the robot from the current iteration - New position calculated for each robot - Position of the robot from the current iteration - Inertia weight - Cognitive and social acceleration constants respectively - Generates a uniform random value in the range [0 1]

A vector representation of the PSO algorithm in a 2-dimensional search space is given in Fig. 2.5. The robots are represented by stars and the target is represented by the labeled circle. The vectors Vpd and Vgd represent the effect of the ‘Pbest ’ and ‘Lbest ’ positions on the robots, respectively. To associate Eq. (2.2) with Fig. 2.5, the term Vpd in

34

L.L. Grant and G.K. Venayagamoorthy

Y T Target

PnewPnew

Vcur Vnew

Vgd Pcur

Vpd X Fig. 2.5. Vector representation of PSO

the figure represents ‘c1 × rand × (Pbest − Pcur )’ and the term Vgd in the figure represents ‘c2 × rand × (Lbest − Pcur )’. The robots use the values of ‘Pbest ’ and ‘Lbest ’ to find a new position closer to the target. The PSO algorithm can be summarized into the following steps: 1. Randomly initialize a swarm of robots in a defined n-dimensional search space with initial random velocities, Pbest , and Lbest values. 2. Evaluate each robot’s current position using the fitness function (the Euclidean distance from the robot to the target(s) based on sensory readings). Select the Lbest from all Pbests . 3. Compute the new velocities and positions using Eqs. (2.2) and (2.3). 4. Check if the robots’ positions are within the problem space and if the velocity exceeds the predefined limits. If not then set the velocity to the maximum velocity (Vmax ) and set the new position to its Pbests . 5. Calculate the new fitness for the robots’ new positions. Determine the new Pbest . Compare with the previous Pbest and update the value if necessary. 6. Calculate the new local best position Lbest among all the new Pbest positions. Compare with the previous best and update the Lbest . 7. Repeat procedure from step (3), until all robots converge on the target(s). A flowchart of the PSO algorithm applied to a CRS problem is illustrated in Fig. 2.6. 2.2.3

Fuzzy Logic

Fuzzy logic was pioneered by Lofti Zadeh in the 1960s [9, 29, 31]. This logic combines human knowledge with sensory measurements and mathematical formulas to try to model commonsense. A system incorporating fuzzy logic utilizes linguistic variables to represent real world parameters. Membership functions define the degree of the input and output variables in the fuzzy system. A rule set developed by heuristics defines

2 Swarm Intelligence for Collective Robotic Search

35

Start

Initialize search area. Initialize robots positions and velocities and set initial Pbest. Evaluate fitness function to determine Lbest. Compute new velocities and positions for all robots. Check new velocities and positions and reset if they are not within defined limits. Calculate new fitness value for each of the robots new positions. Update Pbest and Lbest values.

Have all robots converged on the target?

No

Yes End

Fig. 2.6. Flowchart of the PSO based strategy as applied to a CRS problem

the relationship between the input and output variable to tie the whole system together. During the fuzzification process, the inputs are transferred into the linguistic variables and then formed into fuzzy sets. An inference engine uses the defined rules to develop the fuzzy outputs from the inputs. Finally, in the defuzzification process, the outputs are mapped back to real world variables for use in controlling a real world application. Since the system’s performance depends strongly on the way the membership functions and rules are defined, it is essential to optimize these parameters [28]. A block diagram of a fuzzy control system is shown in Fig. 2.7. In this study, a swarm of robots guided by fuzzy logic controllers is used to locate targets in a CRS problem. Each robot has an identical fuzzy controller whose membership functions and rules are developed based on heuristics. The inputs to the fuzzy controller are the measured light intensities of the targets read by the robot’s sensor and the Euclidean distance of the robot’s current position to the position of the robot with the highest fitness or ‘Lbest ’ value in the swarm. The outputs of the fuzzy controller are displacements in the X-axis and Y-axis directions, which are added to the robot’s current position. Eqs. (2.4) and (2.5) are used to find each robot’s next position [28]. In Eqs. (2.4) and (2.5), Ldx = Lbestx − Px and Ldy = Lbesty − Py , where ‘Px ’ and ‘Py ’ are the X-Y coordinates of the robot’s current position and ‘Lbestx ’ and ‘Lbesty ’ are the X-Y coordinates of the ‘Lbest ’ of the swarm. The ‘Lbest ’ value is the position of the robot in

36

L.L. Grant and G.K. Venayagamoorthy

Real World Inputs

Real World Outputs Fuzzy Logic Controller

Fuzzification

Inference Engine

Defuzzification

Rule base

Fig. 2.7. Block diagram illustrating the stages involved in a fuzzy control system

the swarm with the highest intensity reading. If Ldx is negative then the X coordinate in Eq. (2.5) becomes (Xi − Δ Xi ) and if it is positive then it becomes (Xi + Δ Xi ). The same logic is applied to the Y coordinates as well. The inference engine used for this simulation is Mamdani’s product inference engine and the defuzzification is based on center average. Pi = (Xi ± Δ Xi ,Yi ± Δ Yi )

(2.4)

Δ Xi = f (Ii , Ldx ); Δ Yi = f (Ii , Ldy )

(2.5)

The membership functions of the three input variables to each robot’s fuzzy controller are described below. • The intensity reading of the sensor on each robot (Ii ) has zero (Z), small (S), medium (M) and large (L) triangular membership functions. • The difference of the X coordinates of the current position of the robot from the robot having the best position in the swarm given by ‘Ldx ’ has zero (Z), very small (VS), small (S), medium (M) and large (L) triangular membership functions. • The difference of the Y coordinates of the current position of the robot from the robot having the best position in the swarm given by ‘Ldy ’ has zero (Z), very small (VS), small (S), medium (M) and large (L) triangular membership functions. The membership functions of the two output variables from each fuzzy logic controller are described below. • The displacement amount Δ Xi to be added to the robot’s current X-axis position has very small (VS), small (S), medium (M) and large (L) triangular membership functions. • The displacement amount Δ Yi to be added to the robot’s current Y-axis position has very small (VS), small (S), medium (M) and large (L) triangular membership functions.

2 Swarm Intelligence for Collective Robotic Search

Fuzzy Inputs

A target is identified?

37

Yes Fine Rules

No Fuzzy Outputs

Coarse Rules

Fig. 2.8. Rule base switching logic Table 2.1. Coarse Rule Set

Ldx/Ldy Intensity(I)

Z

VS

S

M

L

Z S M L

M S S VS

M M S VS

L M M S

L L M S

L L M M

Table 2.2. Fine Rule Set

Ldx/Ldy Intensity(I)

Z

VS

S

M

L

Z S M L

VS VS VS VS

VS VS VS VS

S S VS VS

M S S S

M M M S

There are two different rule sets for the fuzzy controllers. The first set of rules is called the ‘coarse rule set’, which makes the robots move in larger steps if they are far from the target. This rule set helps to advance exploration of the search area. Once a target has been identified, the second rule set referred to as the ‘fine rule set’ is used to make the robots move toward the target in smaller steps. Fig. 2.8 shows the relationship between the robot’s position relative to the target, and the coarse and fine rule sets. Tables 2.1 and 2.2 show the coarse and fine rule sets, respectively. Both rule sets have a total of 40 rules. There are 20 rules each for X- and Y-axis. The rules are used in IF-THEN statements to determine the output or which size step the robot should take. An example is given in Eq. (2.6) taken from the coarse rule set [28]. A flowchart of the fuzzy algorithm applied to a CRS problem is illustrated in Fig. 2.9. IF Intensity is small(S) and Ldx is large(L) THEN Δ Xi is Large (L)

(2.6)

38

L.L. Grant and G.K. Venayagamoorthy

Start

Initialize search area. Initialize robots positions and velocities and set initial Lbest. Evaluate fitness function to determine Lbest. Check intensity/fitness of each robot’s current position. Use robot’s fitness and location relative to the Lbest to calculate X- and Y- axis fuzzy displacements. Move to next position based on fuzzy displacements. Evaluate fitness of new position to determine new Lbest. Have all robots converged on the target?

No

Yes End

Fig. 2.9. Flowchart of the fuzzy logic based strategy as applied to a CRS problem

2.3 Single Target Obstacle-Free Environment The PSO algorithm was simulated in a simple single-target, obstacle-free environment to determine the best combination of values to use for the PSO strategy parameters w, c1 , and c2 . Fig. 2.10 shows a search space containing a single target. The target location is fixed and 10 robots are randomly dispersed throughout the search environment. The performance of the swarm based on changes in PSO’s parameter values is shown in Table 2.3. The results are averaged over 100 trials. For the c1 and c2 variables, values of 0.5 and 2 were tested and values of 0.6 and 0.8 were tested for the w parameter. As can be seen from Table 2.3, the values producing the fastest swarm convergence and minimum iterations are w = 0.6, c1 = 0.5, and c2 = 2. The higher value of the desired social acceleration constant, c2 , as compared to the desired cognitive acceleration constant, c1 , indicates the PSO algorithm’s dependency on social interactions among the swarm and the importance of the Lbest parameter. The PSO parameters can be further optimized during the search process by using optimal PSO. For optimal PSO, there is an outer PSO algorithm and an inner PSO algorithm. The inner swarm provides the actual implementation of the target search problem while the outer swarm optimizes the PSO parameter values. Every time a cycle of the

2 Swarm Intelligence for Collective Robotic Search

Y

39

Mobile Robots/ PSO Particles

10

Target

X -10 -10

10

Fig. 2.10. Graphical representation of a single target case Table 2.3. Effect on PSO results in single target case with parameter changes

c1

c2

# of Iterations (w=0.6)

# of Iterations (w=0.8)

2 0.5 2 0.5

2 2 0.5 0.5

135.45 79.63 – 110.29

491.95 348.20 498.14 145.08

Table 2.4. Optimization of PSO parameters using optimal PSO strategy

w

c1

c2

# of Iterations

0.60 0.45 0.45 0.45

0.50 0.60 1.19 0.35

2.00 1.50 1.52 1.45

79.63 39.50 60.80 83.40

algorithm runs, a new set of PSO parameter values are calculated in the outer swarm and plugged into the inner swarm. Fig. 2.11 contains the flowchart for performing an optimal PSO algorithm [9]. Table 2.4 presents the parameter values tested in the optimal PSO trials. It is evident from the table that the combination of 0.45, 0.6, and 1.5 for w, c1 , and c2 respectively, provide the fastest search. The results in Tables 2.3 and 2.4 show that swarm communication has a large influence on the success of the search. The social acceleration constant, c2 , which controls the how much information is shared among the swarm is much higher than the cognitive acceleration constant, c1 which controls the robot’s use of it’s own gathered information for a faster search.

40

L.L. Grant and G.K. Venayagamoorthy

OUTER PSO Initialize PSO parameters for the outer swarm (wout, c1out, & c2out)

Initialize the values of win, c1in, & c2in to be used in the inner swarm

INNER SWARM FOR TARGET SEARCHING Uses the values win, c1in, & c2in as passed into it The target search application program that uses PSO Fitness: lowest number of iterations

Chooses the values of w in, c1in, & c2in that corresponds to the particle with the least number of iterations

Best values of win, c1in, & c2in (Lbest for outer PSO)

TARGET SEARCHING

Fig. 2.11. Flowchart for Optimal PSO

2.4 Multiple Target/Obstacle Environment There are two types of case studies performed for the multi-target/obstacle case. For the first case study, a single swarm performs a global search of the entire environment space. In the second case study, the environment is divided into 4 neighborhoods and the swarm is broken down into four local groups, one searching each neighborhood. With the introduction of obstacles into the search space it is necessary to provide the robots with a method for obstacle avoidance. The following sub-sections present a brief explanation of the obstacle avoidance technique chosen, followed by sections detailing both case studies and simulation results for the greedy, PSO, and fuzzy algorithms. 2.4.1

Obstacle Avoidance Method

Obstacle avoidance is a real-world issue in robot navigation implementation. A variety of methods have been proposed for obstacle avoidance [1, 11, 12, 13, 30]. These methods incorporate various CI and computational techniques, but many of the existing methods are computationally intensive and the necessary coordination scheme amongst the robots diminishes the robot’s ability to effectively explore the search space. The

2 Swarm Intelligence for Collective Robotic Search

41

For (i = 1:N )Do,

For ( j = 1 : m )Do,

dij =|| Pi - Xj || ; where ( j = 1,..., m ) If (dij £ rij )Then , Pi - Xj dij Set Pi = Pi + pij ´ Zij ´ sin (qij ) Set Zij =

End If End For End For

Fig. 2.12. Pseudo code for the repulsion algorithm

obstacle avoidance method considered in this study is a modification of the repulsion algorithm presented in [20]. This algorithm is chosen because of its computational simplicity as compared to other existing methods, and its compatibility with numerous robot coordination strategies. The repulsion strategy is used to repel the robots away from detected obstacles. The pseudocode for the repulsion algorithm is provided in Fig. 2.12 for a swarm of N robots and a search area containing m obstacles. Repulsion from an obstacle occurs if the robot’s next projected position lies within a prescribed boundary area around the obstacle, called the obstacle’s repulsion range - ri j [24]. The swarm of robot’s current positions are represented by Pi at a single iteration, where {i = 1, . . . , N}. The obstacles are represented by Xi for { j = 1, . . . , m}. The robot’s next position relative to the obstacle boundary condition is calculated in di j , if this variable is less than the value of the obstacle’s repulsion range (ri j ), then the robot’s next position will lie within an obstacle. To calculate the robot’s new adjusted position, the following are added to the robots current position: the repulsion strength (pi j ), a unit vector with the direction from the obstacle to the robots position (Zi j ), and a deflection angle (θi j ). For a real-world implementation, the robot would desirably have touch sensors which would activate the repulsion algorithm once a robot made contact with an obstacle. 2.4.2

Case Study 1: Single Swarm

The search environment used for simulation in the single swarm case study is provided in Fig. 2.13. The search area is restricted to a 200 by 300-unit space in the X-Y plane. Four stationary targets and three obstacles reside within the search area. In Fig. 2.13, the targets are represented by circles and each one is assumed to emit a different intensity level. The targets simulate radiation or other volatile sources, depending on the desired application. Obstacles in the environment, such as buildings, impassable mountain ranges and large bodies of water, are approximated by basic geometric shapes that the robots must navigate around in order to converge on the target of interest. The obstacles shown in Fig. 2.13 include: a lake approximated by a circle, a cluster of three small

42

L.L. Grant and G.K. Venayagamoorthy

Fig. 2.13. Graphical representation of multi-target/obstacle case utilizing single swarm. (♦=robots, o=targets)

square buildings approximated by a triangle, and a large irregularly shaped building approximated by a rectangle. The solid figures are the actual obstacles and the shapes enclosing them represent the obstacle approximation boundaries. Thirty robots, represented by stars, are randomly distributed in the search area. For simulation, any number of robots, obstacles, or targets is applicable. The PSO and greedy algorithms are applied to solve the CRS problem in this environment. The simulation results are provided in Table 2.5. The algorithms are analyzed based on the simulation run-time, fitness evaluations until swarm convergence, distance traveled by the robots, and percent convergence on all targets. The results are further broken down by the average value with standard deviation for one trial, and the maximum and minimum values observed out of 100 total trials. One trial involves the convergence of the entire swarm on all targets in the search environment. A single iteration is equal to each of the thirty robots completing one change in position. A fitness evaluation occurs at every instance when a robot checks the fitness of its current position relative to the targets. The distance traveled is the total distance traveled by the swarm in units. The convergence percentage is obtained by tracking whether or not each robot reaches every target by a distance of less than 0.01 units for a single trial within the set Table 2.5. PSO and Greedy Multi-Target/Obstacle Results (Averaged over 100 trials) Mean ± std PSO Greedy Time(sec) 15 ± 0.83 23.8 ± 1 Fitness Evaluations 15,666 ± 711 70,471 ± 2,188 Distance Traveled 19,848 ± 2,054 158,720 ± 6,988

Max PSO 17.91 18,660 31,588

Greedy 28.24 78,536 179,200

Min PSO 14.02 14,580 16,864

Greedy 22.33 65,836 146,200

2 Swarm Intelligence for Collective Robotic Search

Fig. 2.14. Trajectories of a greedy swarm converging on a target

Fig. 2.15. Trajectories of a PSO swarm converging on a target

43

44

L.L. Grant and G.K. Venayagamoorthy

limit of a maximum of 500 iterations per target. If the final fitness of a single robot is greater then 0.01 for any target, then the convergence for that trial is 0%. Both the PSO and greedy algorithms had 100% convergence on all targets over 100 trials. For time, fitness evaluations, and distance traveled, PSO performs better than greedy search. PSO converges around 1.6 times faster and performs around five times fewer fitness evaluations than the greedy search. The robots driven by PSO also covered eight times less distance than the greedy robots in order to locate the targets. Most of the extra distance gained by the greedy robots occurs during the mapping of the search points when the robot tries to locate the next position of best fitness. Also, the greedy algorithm is less compatible with the repulsion algorithm than PSO. The repulsion algorithm is unable to overcome the tendency of the greedy drive to keep the robots moving directly towards the target. The robots try repeatedly to move straight through the obstacles, attempting to reach the target on the other side. In some instances, this causes the robot to become trapped on one side of the obstacle, requiring more time, evaluations, and distance traveled while attempting to maneuver around the obstacle. This disturbance does not happen in PSO due to the swarm communication, which allows for more freedom in exploration of the search space instead of requiring a direct trajectory to the target. This phenomenon can be seen in Figs. 2.14 and 2.15 with robot paths traced for the greedy and PSO algorithms, respectively. The robot’s have the same initial positions for each algorithm indicated by the star symbols. In Fig. 2.14, the greedy robot paths are straighter than the PSO paths in Fig. 2.15 because the greedy robots are driven to take the most direct path to the target, and the PSO robot’s ability to communicate helps them avoid becoming trapped since they are not only drawn to the target but also to other robots with better fitness [3]. From the enlarged portions of the figures it can be seen that the greedy robot must re-position itself using the repulsion algorithm multiple times to get around the triangle shaped obstacle while the PSO robot is pulled toward the robots beside it and smoothly passes around the obstacle. 2.4.3

Case Study 2: Multiple Swarms

For cases where there is a large amount of terrain to explore or boundary restrictions such as large bodies of water exist and restrict explorations, it is necessary to confine the robot’s motion to a smaller area based on a divide and conquer scheme [27]. Multiple swarms are used to search smaller portions of the environment ensuring a fast and efficient parallel search. Fig. 2.16 presents a search area similar to the one in Fig. 2.13, but segregated into four neighborhoods. Two targets reside in swarm area 1 and a single target resides in swarm area 4. There are also obstacles present in swarm areas 1, 2, and 4. The robots are randomly dispersed throughout the search area and assigned to the swarm corresponding to the location of their initial search position. The fuzzy and greedy methods are applied to solve the CRS problem in this environment set-up. The results for both methods obtained over 100 trials are presented in Table 2.6. As seen in Table 2.6, the fuzzy method outperforms the greedy method with around 50% fewer iterations and a much shorter overall run-time. The greedy search also has a lower convergence percentage than the fuzzy method.

2 Swarm Intelligence for Collective Robotic Search

45

Fig. 2.16. Graphical representation of multi-target/obstacle case utilizing multiple swarms. (♦=robots, o=targets) Table 2.6. Multiple Swarm Results for Fuzzy and Greedy Searches

Fuzzy Greedy

Time

Iterations

Convergence

214.04 622.81

256.14 662.56

94% 87%

2.5 Conclusion This chapter has introduced two computational intelligence techniques and compared them with the greedy search approach for application to the navigation of a swarm of robots for a target search in a hazardous environment. All algorithms have been shown to be suitable for CRS applications based on reliability in performance when evaluating convergence. Both CI techniques, fuzzy and PSO, outperform the greedy search in terms of time, target convergence, fitness evaluations, and compatibility with an obstacle avoidance method. Greedy search requires no communication among the robots, in contrast to the other methods, but does not adapt well to the simple obstacle avoidance technique presented. Comparing the results for the greedy search in the single target search versus the multiple target search, convergence drops significantly from 100% to 87% for a more complex environmental setup. The PSO algorithm can be further optimized by the adjustment of its velocity equation parameters for a faster, more efficient search and this optimization can be carried out by PSO.

46

L.L. Grant and G.K. Venayagamoorthy

Future work involves an investigation of the effect of optimizing the membership functions and rules of the fuzzy controllers. Optimization of the membership functions, rule bases, and PSO parameter ‘w’, is expected to improve the performance of the CRS. Also, a hybrid combination of the PSO and greedy algorithms for implementing both a local and global search would be useful for this application. Recently, a collective robotic search platform has been developed for evaluating search algorithms in a laboratory setting and a successful implementation of PSO has been accomplished [19].

References 1. An, D., Wang, H.: VPH: a new laser radar based obstacle avoidance method for intelligent mobile robots. In: 5th World Congress on Intelligent Control and Automation, vol. 5, pp. 4681–4685 (2004) 2. Cassinis, R.: Landmines detection methods using swarms of simple robots. Intelligent Autonomous Systems 6 (2000) 3. Chuang-xin, G., Zhao, B.: A pooled-neighbor swarm intelligence approach to optimal reactive power dispatch. Zhejiang University SCIENCE A 7, 615–622 (2006) 4. Cook, B., Marthaler, D., Topaz, C., Bertozzi, A., Kemp, M.: Fractional bandwidth reacquisition algorithms for VSW-MCM. In: Multi-robot Systems from Swarms to Intelligent Autonoma, vol. 2, pp. 77–86. Kluwer Academic Publishers, Dordrecht (2003) 5. Cortes, J., Martinez, S., Bullo, F.: Robust rendezvous for mobile autonomous agents via proximity graphs in arbitrary dimensions. IEEE Transactions on Automatic Control 51, 1289– 1298 (2004) 6. Cui, X., Hardin, T., Ragade, R.K., Elmaghraby, A.S.: A swarm-based fuzzy logic control mobile sensor network for hazardous contaminants localization. In: IEEE International Conference on Mobile Ad-hoc and Sensor Systems, pp. 194–203 (2004) 7. Dadios, E.P., Maravillas Jr., O.A.: Cooperative mobile robots with obstacle and collision avoidance using fuzzy logic. In: IEEE International Symposium on Intelligent Control, pp. 75–80 (2002) 8. del Valle, Y., Venayagamoorthy, G.K., Mohagheghi, S., Hernandez, J.C., Harley, R.G.: Particle swarm optimization: basic concepts, variants and applications in power systems. IEEE Transactions on Evolutionary Computation (accepted, 2003) 9. Doctor, S., Venayagamoorthy, G.K., Gudise, V.G.: Optimal PSO for collective robotic search applications. In: IEEE Congress on Evolutionary Computation, vol. 2, pp. 1390–1395 (2004) 10. Eberhart, R., Shi, Y.: Particle swarm optimization: developments, applications and resources. In: IEEE Congress on Evolutionary Computation, vol. 1, pp. 81–86 (2001) 11. Er, M.J., Deng, C.: Obstacle avoidance of a mobile robot using hybrid learning approach. IEEE Transactions on Industrial Electronics 52, 898–905 (2005) 12. Fujimori, A., Nikiforuk, P.N., Gupta, M.M.: Adaptive navigation of mobile robots with obstacle avoidance. IEEE Transactions on Robotics and Automation 13, 596–601 (1997) 13. Hu, E., Yang, S.X., Chiu, D.K.Y., Smith, W.R.: Real-time tracking control with obstacle avoidance of multiple mobile robots. In: IEEE International Symposium on Intelligent Control, pp. 87–92 (2002) 14. Jadbabaie, A., Lin, J., Morse, A.S.: Coordination of groups of mobile autonomous agents using nearest neighbor rules. IEEE Transactions on Automatic Control 48, 988–1001 (2003) 15. Jennings, J.S., Whelan, G., Evans, W.F.: Cooperative search and rescue with a team of mobile robots. In: 8th International Conference on Advance Robotics, pp. 193–200 (1997) 16. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: IEEE International Conference on Neural Networks, vol. 4, pp. 1942–1948 (1995)

2 Swarm Intelligence for Collective Robotic Search

47

17. Kennedy, J., Eberhart, R.C., Shi, Y.: Swarm Intelligence. Morgan Kauffman, San Francisco (2001) 18. Lin, Z., Broucke, M., Francis, B.: Local control strategies for groups of mobile autonomous agents. IEEE Transactions on Automatic Control 49, 622–629 (2004) 19. Palangpour, P., Parrott, C., Grant, L.: Collective robotic search in software and hardware implementation. IEEE Industry Applications Magazine 14, 71–73 (2008) 20. Parsopoulos, K.E., Vrahatis, M.N.: On the computation of all global minimizers through particle swarm optimization. IEEE Transactions on Evolutionary Computation 8, 211–214 (2004) 21. Russell, R.A., Thiel, D., Deveza, R., Mackay-Sim, A.: A robotic system to locate hazardous chemical leaks. In: IEEE International Conference on Robotics and Automation 1 (1995) 22. Shi, Y., Eberhart, R.C.: Parameter selection in particle swarm optimization. Evolutionary Programming 7, 611–616 (1998) 23. Shi, Y., Eberhart, R.C.: Empirical study of particle swarm optimization. In: Congress on Evolutionary Computation, vol. 3, p. 1950 (1999) 24. Smith, L., Venayagamoorthy, G.K.: Optimized pso for collective robotic search in an environment with multiple targets and obstacles. In: The International Journal of Engineering Applications of Artificial Intelligence (in preparation, 2007) 25. Tang, Z., Ozguner, U.: Motion planning for multitarget surveillance with mobile sensor agents. IEEE Transactions on Robotics 21, 898–908 (2005) 26. Thayer, S., Digney, B., Dias, M.B., Stentz, A., Nabbe, B., Hebert, M.: Distributed robotic mapping of extreme environments. SPIE Mobile Robots XV and Telemanipulator and Telepresence Technologies VII, p. 4195 (2000) 27. Venayagamoorthy, G.K., Doctor, S.: Navigation of mobile sensors using PSO and embedded PSO in a fuzzy logic controller. In: 39th IEEE IAS Annual Meeting on Industry Applications, vol. 2, pp. 1200–1206 (2004) 28. Venayagamoorthy, G.K., Doctor, S., Smith, L.: Collective robotic search using hybrid techniques: fuzzy logic and Swarm intelligence inspired by nature. The International Journal of Engineering Applications of Artificial Intelligence (submitted, 2007) 29. Wang, L.X.: A course in fuzzy systems and control. Prentice Hall, Inc., Upper Saddle River (1996) 30. Ye, C., Yung, N.H.C., Wang, D.: A fuzzy controller with supervised learning assisted reinforcement learning algorithm for obstacle avoidance. IEEE Transactions on Systems, Man, and Cybernetics 33, 17–27 (2003) 31. Zadeh, L.A.: Fuzzy Logic. CSLI, 83–93 (1988)

3 A Glowworm Swarm Optimization Based Multi-robot System for Signal Source Localization K.N. Krishnanand1 and D. Ghose2 1

2

Mobile Robotics Laboratory (GCDSL), Department of Aerospace Engineering, Indian Institute of Science, Bangalore [email protected] Mobile Robotics Laboratory (GCDSL), Department of Aerospace Engineering, Indian Institute of Science, Bangalore [email protected]

Abstract. We address the problem of multiple signal source localization where robotic swarms are used to locate multiple signal sources like light, sound, heat, leaks in pressurized systems, hazardous plumes/aerosols resulting from nuclear or chemical spills, fire-origins in forest fires, hazardous chemical discharge in water bodies, oil spills, deep-sea hydrothermal vent plumes, etc. In particular, we present a multi-robot system that implements a modified version of the glowworm swarm optimization (GSO) algorithm, which is originally developed to solve multimodal function optimization problems, for this purpose. The GSO algorithm uses a leapfrogging behavior for the basic search capability and an adaptive decision range that enables the agents to partition into disjoint subgroups, simultaneously taxis towards, and rendezvous at, multiple source locations of interest. Transition of agent behaviors from simulation to real-robot-implementation needs modifications to certain algorithmic aspects mainly because of the point-agent model of the basic GSO algorithm and the physical dimensions and dynamics of a real robot. We briefly describe the basic GSO algorithm and the modifications incorporated into the algorithm in order to make it suitable for a robotic implementation. Realization of each sensing-decision-action cycle of the GSO algorithm requires the robots to perform subtasks such as identification and localization of neighbors, selection of a leader among current neighbors, updating of the associated luciferin and local-decision range, and making a step-movement towards the selected leader. Experiments in this regard validate each robot’s capability to perform the above basic algorithmic primitives. Real-robot-experiments are conducted in the context of light source localization in order to validate the GSO approach to localization of signal sources. These experiments constitute a first step toward implementation in multiple robots for detection of multiple sources.

3.1 Introduction Group intelligence found in biological swarms − cooperative foraging of ant colonies, evading of predators by fish schools, flocking of birds, group-hunting found in canids, herons, and several cetaceans − is a result of actions of each individual in a group based on simple interactions with immediate neighbors and perception of the local environment. Swarm robotics draws from these principles in order to devise decentralized algorithms for control and coordination of multiple mobile robots when the groupsize is very large. The topic represents one of the applications of the general area of D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 49–68. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

50

K.N. Krishnanand and D. Ghose

computational intelligence (CI) where order and structure emerges from an unstructured beginning [1]. Localization of multiple signal sources using mobile robot swarms has received some attention recently in the collective robotics community. In particular, the problem of multiple source localization involves the deployment of a group of mobile robots that use their sensory perception of signal-signatures at distances that are potentially far from each source, and interaction with their neighbors, as cues to guide their movements towards, and eventually co-locate at, the multiple signal-emitting sources. Examples of such sources include light, sound, heat, leaks in pressurized systems [2], hazardous plumes/aerosols resulting from nuclear/chemical spills [3], fire-origins in forest fires [4], hazardous chemical discharge in water bodies [5], oil spills [6], deep-sea hydrothermal vent plumes [13], etc. Most prior work related to signal source localization was devoted to using either single or multiple robot systems for seeking, detecting, and tracking of a single emission source location [2]−[12]. However, very little attention has been given to the equally important problem of localization of multiple sources [13], [14]. This problem has practical relevance to situations where it is imperative to simultaneously identify and neutralize all the radiating sources before they can cause a great loss to the environment and people in the vicinity. The above problem presents several challenges such as multiplicity of sources, need for simultaneous localization of all sources, time-varying nature of the sources, dynamically changing environment, and perceptional noise. In the above applications, the signal-source profile that spreads in the environment can be represented as a multimodal function. Therefore, the problem of multiple source localization can be modelled as optimization of multimodal functions. Multimodal function optimization has been addressed extensively in the recent literature. The traditional class of problems related to this topic focussed on developing algorithms to find either the global optimum or all the global optima of the given multimodal function, while avoiding local optima. However, the multiple signal source localization problem belongs to the class of optimization problems which has the objective of finding multiple optima having either equal or unequal function values. In this chapter, we present a multi-robot system that implements a modification to an algorithm called the glowworm swarm optimization (GSO) algorithm and demonstrate the GSO based robotic system’s suitability for source localization tasks. The basic GSO algorithm, which is originally developed for multimodal optimization problems, enables a homogeneous swarm of mobile agents to partition into subgroups, exhibit simultaneous taxis towards, and rendezvous at multiple optima (not necessarily equal) of a multimodal function. The underlying principles of the GSO algorithm and numerical experiments to show the algorithm’s efficacy in capturing multiple peaks of a wide range of multimodal functions are presented in [16]. We show the GSO algorithm’s applicability to the problems of hazard sensing in ubiquitous environments in [17] and pursuit of multiple mobile signal sources in [18]. However, in all the above work, the algorithm’s suitablity for multiple source localization tasks is tested using only non-embodied point simulations, where agents are considered as points and agentcollisions are ignored. In this chapter, we conduct real-robot-experiments, using a set of four wheeled robots, in order to assess the GSO algorithm’s suitability for source

3 A Glowworm Swarm Optimization Based Multi-robot System

51

localization tasks. There are mainly three issues that arise during a robotic implementation and are not taken into account in the algorithmic description of GSO: 1. The linear and angular movements of agents in the algorithm are instantaneous in nature. However, embodied robots spend a finite time in performing linear and angular movements based on their prescribed speeds and turn rate capabilities. 2. In the algorithm, agents are considered as points and agent collisions are ignored. These are tenable assumptions for numerical optimization problems. However, robots have a physical shape and foot print of finite size, and cannot have intersecting trajectories. Thus, robots must avoid collisions with other robots in addition to avoiding obstacles in the environment. 3. The GSO algorithm uses a leapfrogging behavior (described in Section 3.3), where agents move over one another, to perform local search. Since robots perform collision avoidance with other robots, a robotic implementation of the leapfrogging effect becomes a critical issue and is not straightforward. The above issues are very important and need a careful consideration as they call for changes in the agent movement models and alter the working of the basic algorithm when it is implemented on a swarm of real robots. For instance, since physical robots cannot move over one another, a direct implementation of the leapfrogging behavior is not possible. However, the obstacle avoidance feature alters the algorithm’s behavior, giving rise to a partial leapfrogging effect, which is described in Section 3.3. Therefore, the robot experiments that we conduct in this chapter serve as means to validate the transition of agent behaviors from simulations to a real world implementation of the GSO algorithm, examine the altered algorithm’s behavior, and study the robustness of the algorithm to implementational constraints. The chapter is organized as follows. Prior work on source localization is given in Section 3.2. An overview of the GSO algorithm and its special suitability for localizing multiple sources are described in Section 3.3. The GSO variant for robotic implementation is presented in Section 3.4. The multi-robot platform is described in Section 3.5. The results of real-robot-experiments are discussed in Section 3.6. Concluding remarks and directions for future work are given in Section 3.7.

3.2 Prior Work Approaches to robotic signal source localization include gradient-based [11], modelbased [12], biomimetic [8]− [10], and occupancy grid-based [13], [15] strategies. Multi-robot networks have been proposed to evaluate the performance of adding cooperation as an additional layer to the above strategies [10], [11]. Over the recent years, swarm robotics is increasingly being used for source detection applications [2], [3], [7], [14]. Fronczek and Prasad [2] propose bio-inspired sensor swarms to detect leaks in pressurized systems where sensory perception of agents is based on the biological response of cockroaches to changes in ambient air-flow patterns, communication architecture is modelled around a bee colony localizing an odor source, and the self-repair abilities emulates human decision making. Zarzhitsky et al. [3] design a swarm robotic network based on the principles of fluid physics for chemical plume tracing tasks. Hayes

52

K.N. Krishnanand and D. Ghose

et al. [7] describe a spiral-surge algorithm in which a collection of autonomous mobile robots use spiral plume finding, surge, and spiral casting behaviors to find the source of an odor plume. However, all these implementations are for single odor source detection and do not explicitly address the problems of simultaneous localization of multiple sources. Cui et al. [14] present a multiple source localization algorithm in which mobile agents use a grid map to represent the unknown environment, collect concentration values from all other agents through ad-hoc communication, and calculate a positive gradient direction using a biasing expansion swarm approach (BESA). According to this approach, each agent moves to one of the neighboring eight cells on which the net influence (called the biasing parameter) of other agents is maximum. Each agent’s influence is proportional to the concentration of the diffused source at its location and inversely proportional to the square of its distance to the cell. The cohesion property of this swarming approach maintains the connectivity of the ad-hoc network. Note that each agent uses global information to decide its movements. Even though the authors consider multiple sources, the algorithm does not explicitly take into account the interference problems that are caused due to multiplicity of sources. However, the GSO algorithm, that was originally designed for multimodal optimization problems [16], is specially suited for multiple source localization tasks, which is described in the next section.

3.3 The GSO Algorithm The glowworm swarm optimization (GSO) algorithm is an optimization technique that is originally developed for simultaneous capture of multiple optima of multimodal functions. The underlying principles of the GSO algorithm and numerical experiments to show the algorithms efficacy in capturing multiple peaks of a wide range of multimodal functions are presented in [16]. Other earlier work on the GSO algorithm can be found in [17] and [18]. As this chapter focusses on the implementation aspects of GSO algorithm for multiple source localization tasks using a multi-robot system, we limit ourselves to a brief description of the GSO algorithm. 3.3.1

GSO Algorithm Description

In the GSO algorithm, the agents are initially deployed randomly, according to a uniform distribution, in the objective function space. Each agent carries a luminescence quantity called luciferin that encodes the information about the function profile at its current location. Agents are thought of as glowworms that emit a light whose intensity is proportional to the associated luciferin. A glowworm considers other glowworms that are located within its local-decision domain and those with a higher luciferin value than that of its own as neighbors. The decision domain is adaptive by nature and is bounded above by a circular sensor range. Each glowworm selects a neighbor using a probabilistic mechanism and moves towards it. That is, each glowworm is attracted by the brighter glow of other neighboring glowworms. These individual agent movements, that are based only on local information, enable the swarm of glowworms to partition into disjoint subgroups, exhibit simultaneous taxis-behavior towards, and rendezvous at multiple optimums of a given multimodal function.

3 A Glowworm Swarm Optimization Based Multi-robot System

53

G LOWWORM S WARM O PTIMIZATION (GSO) A LGORITHM deploy agents randomly; ∀ i, set i (0) = 0 ∀ i, set rdi (0) = r0 { for each glowworm i do: i (t) ← (1 − ρ )i(t) + γ J(xi (t + 1)); for each glowworm i do: { Ni (t) = { j : di j (t) < rdi (t); i (t) <  j (t)}; for each glowworm j ∈ Ni (t) do: p j (t) =

 j (t) − i(t) ; ∑k∈Ni (t) k (t) − i (t)

j = select glowworm j; % (using p j (t))   x j (t) − xi(t) xi (t) ← xi (t) + s ; x j (t) − xi(t) rdi (t) ← min{rs , max{0, rdi (t) + β (nt − |Ni (t)|)}}; }

}

t ← t + 1;

The leapfrogging effect: According to the glowworm algorithm presented in this paper, a glowworm with the maximum luciferin at a particular iteration remains stationary during that iteration. Ideally, the above property leads to a dead-lock situation when all the glowworms are located such that the peak-location lies outside the convex-hull formed by the glowworm positions. Since the agent movements are restricted to the interior region of the convex-hull, all the glowworms converge to a glowworm that attains maximum luciferin value during its movements within the convex-hull. As a result, all the glowworms get co-located away from the peak-location. However, the discrete nature of the movement-update rule automatically takes care of this problem which could be described in the following way. During the movement phase, each glowworm moves a distance of finite step-size s towards a neighbor. Hence, when a glowworm i approaches closer to a neighbor j such that the inter-agent distance becomes less than s, i leapfrogs over the position of j and becomes a leader to j. In the next iteration, i remains

54

K.N. Krishnanand and D. Ghose

stationary and j overtakes the position of i thus regaining its leadership. This process of interchanging of roles between i and j repeats giving rise to a local-search behavior of the glowworm pair along a single ascent direction. A group of glowworms use the same principle to perform an improved local-search and eventually converge to the peak location. The leapfrogging behavior of the agents in the GSO algorithm are supported by simulations in Section 3.6. The various steps of the GSO algorithm are given in the inset box, where i (t) represents the luciferin level associated with glowworm i at time t, ρ is the luciferin decay constant (0 < ρ < 1), γ is the luciferin enhancement constant, J(xi (t)) represents the value of the signal strength at agent i’s location at time t, xi (t) ∈ ℜm is the location of glowworm i, at time t, in the m−dimensional real space ℜm , Ni (t) is the neighborhood of glowworm i at time t, di, j (t) represents the euclidian distance between glowworms i and j at time t, rdi (t) represents the variable local-decision range associated with glowworm i at time t, s (> 0) is the step-size, rs represents the radial range of the luciferin sensor, the glowworm i selects to move towards a glowworm j ∈ Ni (t) with probability p j (t), and nt represents the desired number of neighbors. 3.3.2

A Simulation Example to Illustrate Basic Working of GSO

Simulation results demonstrating the capability of the glowworm algorithm to capture multiple peaks of a number of complex multimodal functions have been reported in [17]−[19]. In this paper, we demonstrate the basic working of the algorithm using the multimodal function given below: Q

J(x, y) =

∑ ai exp(−bi ((x − xi)2 + (y − yi)2 ))

(3.1)

i=1

where, Q represents the number of peaks and (xi , yi ) represents the location of each peak. A workspace of (−5, 5) × (−5, 5) and a set of ten peaks (Q = 10) are considered for the purpose. Initially, we consider equal slopes for all the peaks by choosing bi = 3, i = 1, . . . Q. The values of ai , xi , and yi are generated according to the following equations: ai = 1 + 2 rand() xi = −5 + 10 rand() yi = −5 + 10 rand() where, rand() is a uniform random variable within the interval [0, 1]. The corresponding values of {ai , xi , yi , i = 1, . . . , 10} for the function profile are shown in Table 3.1. Figure 3.1(a) shows the multimodal function profile chosen for the purpose and Figure 3.1(b) shows the emergence of solution when n = 300 and rs = 5. We observe that when the slopes of the peaks are equal, the swarm splits into subgroups according to the Voronoi-partition of the peak locations, that is, agents deployed in the Voronoi-partition of a peak location remain in the same partition, during their movements, and eventually converge to the corresponding Voronoi-center that coincides with the peak location. However, a few agents located near the border region of

3 A Glowworm Swarm Optimization Based Multi-robot System

55

Table 3.1. Values of ai , xi , and yi used to generate the function profile J(x, y) 1 2 3 4 5 6 7 8 9 10 ai 2.6160 2.4270 1.2225 2.3660 1.3380 2.2851 2.3253 1.0020 1.7090 2.9846 xi 2.9329 -4.0835 3.4591 2.0655 -4.8141 4.3812 -1.1106 -1.7107 -1.7282 -3.2330 yi -0.5591 -2.1457 -2.8029 2.7104 4.7894 3.2815 -1.4957 -3.9765 0.2440 -3.1662

5

2.5 4 3

1.5

2

1

1 Y

7

J1

2

0.5

0 −1

5

−2

5 0

0

y

−3 −4

x −5

−5

(a)

−5 −5

−4

−3

−2

−1

0 X

1

2

3

4

5

(b)

Fig. 3.1. a) Multimodal function profile. b) Emergence of agent movements and their eventual co-location at the peaks.

a Voronoi-partition migrate into adjacent partitions and eventually get co-located at the respective peaks. Algorithmic aspects such as usage of a point-agent model with instantaneous linear/angular agent-movements, allowing of collisions between agents, and enabling the agents to move over one another are expected to create problems during implementation on real robots. These aspects that have an effect on the algorithm’s behavior will be discussed next.

3.4 Algorithmic Modifications for Robotic Implementation A robotic implementation of the GSO algorithm would require a collection of mobile robots where each robot has the following capabilities: 1. Sensing and broadcasting of profile-value (luciferin level) at its location. 2. Detection of number of neighbors and their localization with respect to its own position. 3. Receiving profile-values (luciferin levels) from all neighbors. 4. Selecting a neighbor using a probability distribution (based on the relative luciferin levels of its neighbors) and making a step-movement towards it. 5. Obstacle avoidance behavior in order to avoid collisions with obstacles and other robots in the environment. 6. Leapfrogging behavior for performing local search.

56

K.N. Krishnanand and D. Ghose

As we assume perfect sensing and broadcast in our initial experiments, the first three robot capabilities are rather straightforward to implement and do not need a discussion. However, there are a few issues related to resolution of the sensing elements and communication delays/interference which will be discussed later. Implementation details of obstacle avoidance and leapfrogging behaviors are given below: 3.4.1

Obstacle Avoidance Model

The footprint of the robot is considered to be a circle of radius rrobot . When a robot approaches closer than a safe distance dsa f e < s to another robot, it performs a simple collision-avoidance maneuver through a sequence of point-turn and straight-line movements. 3.4.2

Implementation Models for Leapfrogging Behavior

The leapfrogging effect of the GSO algorithm requires a glowworm to move over another glowworm whenever two glowworms are within a step distance s of each other. This is illustrated in Figure 3.2(a) where glowworm A moves over glowworm B to reach the position A(t + 1). However, this is not directly realizable with physical mobile robots. We describe three methods to achieve an explicit robotic implementation of the leapfrogging behavior. The first method (Figure 3.2(b)) involves interchange of agent-roles, through communication, where robot B moves to the desired position of

A(t+1) B(t)

B(t+1)

A(t)

A(t+k)

B(t+1) B(t) A(t)

(a)

A(t) (c)

(b)

A(t+k)

A(t+k) B(t)

B(t+k)

A(t)

B(t+k)

B(t)

A(t+1)

B(t) B(t+k) A(t)

(d)

(e)

Fig. 3.2. a) Leapfrogging behavior according to the GSO algorithm. b), c), and d) Three methods of robotic implementation of explicit leapfrogging behavior. e) Implicit leapfrogging behavior due to obstacle avoidance.

3 A Glowworm Swarm Optimization Based Multi-robot System

57

robot A and robot A replaces robot B’s position. The illustration in Figure 3.2(c) shows another method where the robot A takes a detour about robot B’s position and reaches its desired position. We propose a modification to this method where the detour is achieved using a blend of two circular paths as shown in Figure 3.2(d). Implicit leapfrogging using obstacle avoidance: We observe that the obstacle avoidance behavior automatically gives rise to an implicit leapfrogging effect, which is described using the illustration in Figure 3.2(e). Robot A alternatively performs collision avoidance with and seeks to move towards robot B until it crosses the equi-contour line on which robot B is situated. Thereafter, since robot A becomes a leader to robot B, robot performs similar movements with respect to robot A, thus leading to an implicit leapfrogging behavior of the glowworm-pair.

3.5 Kinbots: Multi-robot Platform for Real-Robot-Experiments We use a set of four wheeled robots, called Kinbots (See Figure 3.3), that were originally built for experiments related to robot formations [19]. Realization of each sensing-decision-action cycle of the GSO algorithm requires the robots to perform subtasks such as identification and localization of neighbors, selection of a leader among current neighbors, updating of the associated luciferin and local-decision range, and making a step-movement toward the selected leader. By making necessary modifications to the Kinbot hardware, the robots are endowed with the capabilities required to implement the various behavioral primitives of the GSO algorithm. A PIC16F877

Fig. 3.3. Multi-robot system built for source localization experiments

58

K.N. Krishnanand and D. Ghose

Left motor

Right motor

Photodiode output Light sensor

Driver stage

+5V

-

C

R

A/D +

PWM module

Bandpass filter

Step motor

Driver stage

Luciferin receiver

PIC 16F877 Micro controller RX (RS 232)

I/O PORT

Home sensor

Step + Direction

1 KHz signal

TX (RS 232)

33 KHz

(center freq. = 1KHz)

Array of 16 IR LEDs (Beacon)

Array of 8 IR leds (Luciferin transmitter)

Fig. 3.4. The hardware architecture of Kinbots

microcontroller is used as the robot’s processing unit. The hardware architecture of Kinbots is shown in Figure 3.4. The various hardware modules of each robot that are used to achieve the above tasks are shown in Figure 3.5. The tether to the robots (refer to Figure 3.3) provide only power to the robots from an external power supply and is not used for data communication purpose. However, the tethered power supply can be replaced by using an onboard battery. Real-robot-experiments are conducted for the purpose of

3 A Glowworm Swarm Optimization Based Multi-robot System

59

Light intensity pick-up sensor

M1

Sweep platform

M2

Luciferin receiver

M3

Luciferin transmitter (Array of 8 IR LEDs)

M4

Bandpass filter

M5

Photodiode (mounted on a rotary platform)

M6

Circular array of 16 LEDs placed radially outward (Serves as a beacon to other robots)

M7

Differential mobile robot base

M8

Fig. 3.5. Various hardware modules used by the Kinbots to implement the GSO behaviors

light source localization in order to validate the GSO approach to localization of signal sources. 3.5.1

Relative Localization of Neighbors

The various modules and sensors are marked as M1, M2, . . . , M8 in the Figure 3.5. The neighbor localization hardware module is achieved in the following way. A circular

60

K.N. Krishnanand and D. Ghose

array of sixteen infrared LEDs (M7 in Figure 3.5) placed radially outward serves as a beacon to obtain a near circular emission pattern around the robot. These IR LEDs are actuated by a 1 KHz square wave signal. A photodiode (M6), mounted on a rotary platform (M2), performs a sweep and records the intensity of the received infrared light as a function of the angle made by the direction of the source with the heading direction of the robot. Local peaks in the intensity pattern indicate the presence of neighbors and serve to compute their location in polar coordinates. In particular, the received intensity of infrared light follows a inverse square law with respect to distance which is used to compute range information to robot-neighbors and the angular position of the local peak is approximated as the relative angle to a neighbor. The photo-diode output is passed through a bandpass filter, centered around a frequency of 1 KHz, in order to make it sensitive only to the frequency of the actuation signal of the infrared-beacons and reject noise due to ambient light conditions and IR signals used to broadcast luciferin information. 3.5.2

Luciferin Glow/Reception

As we test the GSO algorithm in light source localization experiments, a light pick-up sensor (M1) is used to measure the light intensity at the robot’s location. The light sensor output is fed to an in-built analog-to-digital (A/D) convertor of the microcontroller. The output of the A/D module is converted into a luciferin glow that consists of an infrared emission modulated by an 8-bit serial binary signal that is proportional to the robot’s luciferin value at the current sensing-decision-action cycle. Eight IR emitters (M4) that are mounted symmetrically about the robot’s central axis are used to broadcast luciferin data in its neighborhood. An infrared receiver (M3) mounted on a sweep platform (M2) is used as a luciferin receptor. In order to avoid problems due to interference between data signals coming from different neighbors, the receiver sweeps and aligns along the line-of-sight with a neighbor before reading the luciferin data transmitted by it.

3.6 Real-Robot-Experiments A real-robot-implementation of the GSO algorithm would require the robots to have a similar set of capabilities that were used to implement the GSO algorithm in embodied simulations. The movements of the robots are achieved using a differential mobile platform (M8). The obstacle avoidance maneuver is achieved by the movement of the robot along an arc in embodied simulations. However, in real-robot implementation, when a robot approaches closer than a safe distance dsa f e < s to another robot, it performs a simple collision-avoidance maneuver through a discrete sequence of point-turn and straight-line movements. The resulting change in the obstacle avoidance behavior also leads to a change in the evolution of the leapfrogging behavior of the robots during real world experimentation. 3.6.1

Experiment 1: Basic GSO Behavior

According to the GSO algorithm, a glowworm moves toward a neighbor whose luciferin value is relatively higher than that of itself. It remains stationary either when it has

3 A Glowworm Swarm Optimization Based Multi-robot System

61

the highest luciferin value in its neighborhood or when it is isolated. The above basic behavior is implemented with two Kinbots in the following way. The first Kinbot (KA ), which is initially isolated, starts scanning its environment for neighbors using its sweepsensor module. Another Kinbot (KB ) is allowed to move from a far away point and stop once it enters the sensing range of KA . The robots emit constant luciferin values with A < B . Snapshots taken from a video of the above experiment at 2 sec intervals are shown in Figure 3.6. The circular arc imposed on the snapshots represents the sensor range of KA . Note that it shifts its relative position from one snapshot to the other as KA advances toward KB . At t = 0 sec, KA is isolated and hence remains stationary until KB enters its neighborhood at t = 6 sec. At this point of time, KA senses the presence of KB , performs a point-turn until its heading direction is aligned along the line-of-sight with KB , reads the luciferin value broadcast by KB , starts moving toward KB (since A < B ) at t=8 sec, traverses a distance of about 10 cm, and stops moving at t = 10 sec. Note that the sensing-decision-action cycle time is approximately 4 sec. This 4 sec-cycle is repeated twice from t=10 sec to t = 18 sec.

Sensing range

(a) t = 0 sec

(b) t = 2 sec

(c) t = 4 sec

(d) t = 6 sec

(e) t = 8 sec

(f) t = 10 sec

(g) t = 12 sec

(h) t = 14 sec

(i) t = 18 sec

Fig. 3.6. Snapshots taken from a video of the experiment used to demonstrate the basic GSO behavior

62

K.N. Krishnanand and D. Ghose

(a) t = 0 sec

(b) t = 5 sec

(c) t = 10 sec

(d) t = 15 sec

(e) t = 20 sec

(f) t = 25 sec

(g) t = 30 sec

(h) t = 35 sec

(i) t = 40 sec

(j) t = 45 sec

(k) t = 50 sec

(l) t = 55 sec

(m) t = 60 sec

(n) t = 65 sec

(o) t = 70 sec

Fig. 3.7. Snapshots taken from a video of the experiment used to demonstrate the leapfrogging behavior

3 A Glowworm Swarm Optimization Based Multi-robot System

3.6.2

63

Experiment 2: Leapfrogging Behavior

We conduct experiments with two Kinbots KA and KB to demonstrate the implicit leapfrogging behavior caused due to the incorporation of obstacle avoidance behavior into the basic GSO algorithm. Similar to the previous experiment, the robots emit constant luciferin values with A < B and KB is kept stationary. KB is placed such that it is within the sensing range of KA . However, the heading of KA is, not aligned along the line-of-sight with KB and instead, kept at a small acute angle away from it. Snapshots taken from a video of this experiment at 5 sec intervals are shown in Figure 3.7. Initially, KA performs a point turn such that it is oriented toward KB and then starts moving towards KB (Figure 3.7(b)). As KA reaches closer than about 10 cm to KB at t = 20 sec (Figure 3.7(e)), it toggles from the basic GSO behavior of moving toward a leader to an obstacle avoidance behavior at t = 25 sec (Figure 3.7(f)). The collision avoidance maneuver consists of the following sequence of movements: 1. A clock-wise point-turn of about 45o . 2. A straight-line movement of about 10 cm. 3. A counter clockwise point-turn of 45o . The above sequence is repeated for another two times until the inter-robot distance becomes more than 10 cm at t = 40 sec (Figure 3.7(i)). Now, KA switches to its basic GSO behavior and resumes to move toward KB . Thereafter, this process of toggling between basic GSO and obstacle avoidance behaviors is repeated that gives rise to an implicit leap-frogging behavior of KA around KB . The path traced by robot KA as it approaches, and leapfrogs over, robot KB is shown in Figure 3.8.

Fig. 3.8. Paths traced by the robot KA as it approaches, and leapfrogs over KB

64

K.N. Krishnanand and D. Ghose

Light source

(a) t = 0 sec

(b) t = 5 sec

(c) t = 10 sec

(d) t = 15 sec

(e) t = 20 sec

(f) t = 25 sec

(g) t = 30 sec

(h) t = 35 sec

(i) t = 40 sec

(j) t = 45 sec

(k) t = 50 sec

(l) t = 55 sec

(m) t = 60 sec

(n) t = 65 sec

(o) t = 70 sec

Fig. 3.9. Snapshots taken from a video of the light source localization experiment

3 A Glowworm Swarm Optimization Based Multi-robot System

(p) t = 75 sec

(q) t = 80 sec

(s) t = 90 sec

(t) t = 95 sec

(v) t = 105 sec

(w) t = 110 sec

65

(r) t = 85 sec

(u) t = 100 sec

(x) t = 115 sec

Fig. 3.9. (continued)

3.6.3

Experiment 3: Localization of a Single Light Source

In the following experiment, two Kinbots KA and KB implement the GSO algorithm to detect, taxis toward, and co-locate at a light source. The robots provide simple light intensity measurements at their current locations as inputs to the GSO algorithm running on their onboard processors. The robots use a photodiode based light sensor for this purpose. Snapshots from a video of this experiment are shown in Figure 3.9. The robots are initially deployed in such a way that KB is closer to the light source. Both the robots start scanning their respective neighborhoods and sense each other. As A (0) < B (0), KB remains stationary and KA moves toward KB until t = 10 sec (Figure 3.9(c)). However, between t = 10 sec and t = 30 sec (Figure 3.9(g)), KA remains stationary. This can be attributed to the fact that the luciferin value received by KA is corrupted. After t = 30 sec KA resumes to move toward KB until t = 35 sec (Figure 3.9(h)) when it switches to obstacle avoidance behavior. Note that KB starts moving toward KA at t= 45 sec (Figure 3.9(j)). However, at t = 45 sec, KA is still relatively far from the light source than KB . This can be attributed to the fact that when the robots are very close to each other, the difference in luciferin values interferes with the range of sensor noise. KB

66

K.N. Krishnanand and D. Ghose

Fig. 3.10. Paths traced by the robots KA and KB as they taxis toward, and co-locate at, the light source 7

KA Obstacle 6 aoidance GSO− 5 move Idle

4

Obstacle 3 aoidance

KB

GSO− 2 move Idle

1

0

0

20

40

60

80

100

120

t (sec)

Fig. 3.11. Toggling between basic GSO and obstacle avoidance behaviors of the two robots KA and KB as they interact with each other and search for the light source

3 A Glowworm Swarm Optimization Based Multi-robot System

67

stops moving at t = 65 sec (Figure 3.9(n)). However, at t= 110 sec (Figure 3.9(w)), KA is closer to the source and KA (110) > KB (110). Therefore, KB moves toward KA . This results in the robot pair moving further close to the light source. The paths traced by the two robots as the two robots execute the modified GSO algorithm is shown in Figure 3.10. The toggling between the basic GSO and obstacle avoidance behaviors of the two robots as they interact with each other, which eventually leads to their localization at the light source is shown in Figure 3.11. Note from the figure that the robots are idle during certain time intervals, which can be attributed to one of the following reasons: 1. A robot is isolated. 2. A robot is measuring the highest light intensity. 3. Both the robots are measuring almost the same light intensity (A robot is made to move toward another one only when |A − B| > 3). 4. Reception of luciferin data is corrupted. From Figure 3.11, it is also clear that the sensing-decision-action cycles of the robots are asynchronous with respect to each other. In the above experiment, two robots demonstrate localization of a single source. However, the experiment constitutes a first step toward implementation in multiple robots for detection of multiple sources.

3.7 Concluding Remarks We present a multi-robot system that employs glowworm swarm optimization algorithm to address the problem of simultaneous localization of multiple signal sources. We briefly describe the basic GSO algorithm and the modifications incorporated into the algorithm in order to make it suitable for a robotic implementation. We build a set of four robots that are endowed with the capabilities to implement the GSO algorithm and perform source localization tasks. We conduct real-robot-experiments where two robots localize a single light source in order to validate the GSO approach to signal source localization. However, by using more number of these robots, the GSO approach can be extended to multiple signal source localization.

References 1. Craenen BGW Eiben AE Computational Intelligence. Encyclopedia of Life Support Sciences, EOLSS Co. Ltd. 2. Fronczek, J.W., Prasad, N.R.: Bio-inspired sensor swarms to detect leaks in pressurized systems. In: Proceedings of IEEE International Conference on Systems, Man and Cybernetics (2005) 3. Zarzhitsky, D., Spears, D.F., Spears, W.M.: Swarms for chemical plume tracing. In: Proceedings of IEEE Swarm Intelligence Symposium (2005) 4. Casbeer, D.W., Beard, R.W., McLain, T.W., Li, S.-M., Mehra, R.K.: Forest fire monitoring with multiple small UAVs. In: Proceedings of American Control Conference (2005) 5. Farrell, J., Li, W., Pang, S., Arrieta, R.: Chemical plume tracing experimental results with a REMUS AUV. In: Proceedings of Oceans 2003 Marine Technology and Ocean Science Conference (2003)

68

K.N. Krishnanand and D. Ghose

6. Clark, J., Fierro, R.: Cooperative hybrid control of robotic sensors for perimeter detection and tracking. In: Proceedings of American Control Conference (2005) 7. Hayes, A.T., Martinoli, A., Goodman, R.M.: Robotica 21, 427–441 (2003) 8. Dhariwal, A., Sukhatme, G.S., Requicha, A.A.G.: Bacterium-inspired robots for environmental monitoring. In: Proceedings of IEEE International Conference on Robotics and Automation (2004) 9. Grasso, F.W., Consi, T.R., Mountain, D.C., Atema, J.: Robotics and Autonomous Systems 30, 115–131 (2000) 10. Lytridis, C., Virk, G.S., Kadar, E.E.: Co-operative smell-based navigation for mobile robots. In: Proceedings of the 7th International Conference CLAWAR 2004. Springer, Heidelberg (2004) 11. Sandini, G., Lucarini, G., Varoli, M.: Gradient driven self-organizing systems. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (1993) 12. Farrell, J.A., Pang, S., Li, W.: IEEE Transactions on System, Man, and Cybernetics 33, 850– 863 (2003) 13. Jakuba, M.V.: Stochastic mapping for chemical plume source localization with application to autonomous hydrothermal vent discovery. PhD Thesis, Massachusetts Institute of Technology and Woods Hole Oceanographic Institution (2007) 14. Cui, X., Hardin, C.T., Ragade, R.K., Elmaghraby, A.S.: A swarm approach for emission sources localization. In: Proceedings of the 16th International Conference on Tools with Artificial Intelligence (2004) 15. Lilienthal, A., Duckett, T.: Robotics and Autonomous Systems 48, 3–16 (2004) 16. Krishnanand, K.N., Ghose, D.: Multiagent and Grid Systems: Special Issue on Recent Progress in Distributed Intelligence 3, 209–222 (2006) 17. Krishnanand, K.N., Ghose, D.: Glowworm swarm optimization algorithm for hazard sensing in ubiquitous environments using heterogeneous agent swarms. In: Prasad, B. (ed.) Soft Computing Applications in Industry. Studies in Fuzziness and Soft Computing, vol. 226, pp. 165–187. Springer, Berlin (2008) 18. Krishnanand, K.N., Ghose, D.: Chasing multiple mobile signal sources: A glowworm swarm optimization approach. In: 3rd International Conference on Artificial Intelligence (2007) 19. Krishnanand, K.N., Ghose, D.: Robotics and Autonomous Systems 53, 194–213 (2005)

4 Evolutionary Robotics: From Simulation-Based Behavior Learning to Direct Teaching in Real Environments Seiji Yamada1 and Daisuke Katagami2 1 2

National Institute of Informatics [email protected] Tokyo Institute of Technology [email protected]

Abstract. In our work on evolutionary robotics, two different approaches have been developed: simulation-based behavior learning with genetic algorithm and direct teaching with a learning classifier system in real environments. As the first approach, action-based environment recognition for an autonomous mobile robot. The robot moves in various environments by executing behaviors designed by a human designer and obtains different action sequences. These action sequences are automatically classified by self-organizing maps and the structure of the environment is identified from them. We also developed a GA-based behavior learning method in which a robot can learn suitable behaviors to recognize, and conducted simulation experiments to verify the learning ability. However, all the experiments has been done through only computer simulation. Thus we attempted to develop a direct teaching framework in which a real robot learned from human teacher using LCS in real and physical environments. Direct teaching means a human teacher gives adequate actions to a mobile robot by a GUI at work, and this teaching can accelerate a robot to learn classifiers. This framework is important to realize evolutionary robotics which can learn sufficiently fast in real environments, and we confirmed that it is useful by experimenting with a small mobile robot. In this chapter, we describe these two innovative approaches in evolutionary robotics in detail and discuss them in various points of view.

4.1 Introduction We have developed evolutionary robotics with two different approaches: simulationbased behavior learning with genetic algorithm (GA) and direct teaching with a learning classifier system (LCS) in real environments. Regarding the first approach, this chapter describes action-based environment recognition for an autonomous mobile robot (Khepera), which can not perform localization because dead reckoning is very hard. The robot moves in various environments by executing embedded behaviors and obtains the different action sequences. These action sequences are automatically classified by self-organizing maps (SOM)[15] and the structure of the environment is identified from them. For this environment recognition, we developed a GA-based behavior learning method in which a robot can learn suitable behaviors to recognize, and conducted simulation experiments[33, 34]. However, all the experiments has been done through only simulation in the work. Thus we attempted to develop a direct teaching framework in which a real robot learned D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 69–96. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

70

S. Yamada and D. Katagami

from human teacher using LCS in real environments. Direct teaching involves a human teacher who gives adequate actions to a mobile robot through a GUI and this teaching can accelerate a robot to learn classifiers. This framework is important to realize evolutionary robotics which can learn sufficiently fast in real environments, and we confirmed that it is useful by experimenting with a small mobile robot. In this chapter, we describe these two innovative approaches in evolutionary robotics in detail and discuss them in various points of view. In the section 4.2, the simulationbased behavior learning with GA is described. Next, in the section 4.3, how we developed direct teaching with LCS in real environments is explained. The section 4.4 summaries and concludes our work.

4.2 Simulation-Based Behavior Learning Using Genetic Algorithm Some of the previous research on autonomous robots that can recognize their environments have tried to build precise geometric maps by using sensitive global sensors, e.g., vision sensor [6]. Since their main aim is to navigate a robot with accuracy, a precise map is necessary. However, a simple task like environment recognition may not require a precise map. Actually, many natural agents (like animals) can recognize their environments by using only low-sensitive local sensors [26]. In engineering, it would be useful to build a mobile robot that can recognize environments with inexpensive sensors. Thus, we tried to build a mobile robot that recognizes environments by using lowsensitive local sensors. Such a robot cannot determine its position by dead reckoning in the environment, and hence it cannot build a global map of the environment. Hence, we proposed a method whereby the robot can recognize its environments by performing a series of actions. We call this approach Action-based Environment Modeling (AEM) [33, 35]. In AEM, a mobile robot is behavior-based and acts using suitable given behaviors like wall-following. The action sequences executed in each environment are recorded and transformed into environment vectors. A robot identifies the current environment by referring to its store of environment vectors on the environments that it has already been trained in. Through research on AEM, we have recognized a significant problem: where do the suitable behaviors to AEM come from? An easy solution is that a human designs them and gives them to the robot. While this common approach works in simple environments, a human designer faces greater difficulties as the variety of environments increases because the search space becomes huge and it becomes difficult to get an intuitive understanding of the suitable behaviors. Hence, an automatic design method will inevitably be necessary [10], and for this we devised an evolutionary robotics approach [9, 18] in which the behaviors of a robot are designed using evolutionary computation. In this section, we applied evolutionary robotics to acquire suitable behaviors for AEM. We devised an evolutionary design method for such behaviors that uses GA and conducted an experimental evaluation [32]. In anticipation of a future implementation on a real mobile robot, we used a Khepera robot simulator in the experiments. The experimental results indicated that our evolutionary robotics approach is a promising way

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

71

to automatically acquire suitable behaviors for AEM, and that the acquired behaviors are more efficient than hand-coded ones. 4.2.1

Related Work

Several studies on robotics [20] and artificial life [24] have taken similar approaches to AEM. Nehmzow and Smithers studied recognizing corners in simple enclosures with a self-organizing network [24]. They used direction-duration pairs, which indicated the lengths of walls and shapes of past corners, as an input vector to a self-organizing network. After learning, the network became able to identify corners. However the recognized structure was very local. Mataric represented an environment with automatons in which nodes correspond to landmarks [20]. Although such a representation is more robust than a geometric one, a mobile robot must segment raw data into landmarks and identify them. Nakamura et al. utilized a sequence of sonar data in order to reduce the uncertainties in discriminating the local structure [23]. Although the sequence consisted of sensor data (not actions), their approach is similar to AEM. Kato et al. used modified random walks to discern the environment in a statistical way. Most of these studies chose wall-following behaviors as suitable [20, 24, 33]. The behaviors were described by human designers and they were independent of the tasks. Hence, despite their success, they all face a significant problem: it is very difficult to design behaviors. There are several studies on using genetic programming (GP) [17] for mobile robots to learn behaviors [13, 16, 25, 27]. Thus far, these studies have concentrated on very few simple behaviors such as obstacle avoidance. In contrast, our aim is to learn suitable behaviors for AEM and these behaviors are complicated ones consisting of several primitive behaviors. Our research is also relevant to automatic generation of pattern recognition procedure because the environment can essentially be described as a pattern. There are studies on evolutionary design of neural networks for pattern recognition [3]. However, they focus on the traditional pattern recognition task, and none deal with the cost of recognition, which is significant in our research. Reinforcement learning is also widely applied to design behaviors of an autonomous robot [28]. In reinforcement learning, behaviors are described as a policy consisting of states in an environment and actions which can be executed in each state, and rewards are assigned to some actions in some states. The purpose of reinforcement learning is to search for the optimal policy with which a robot can obtain the maximum expected reward by executing actions. All studies on reinforcement learning in robotics try develop to the optimal policy with which a robot can trace the shortest path in an environment. Since our objective of behavior learning is to search suitable behaviors for AEM, not the shortest path from a start point to a goal in an environment, the objective of reinforcement learning is significantly different from ours. Furthermore, it is intuitively difficult to design a reward function for reinforcement learning that is equivalent to the fitness function that we use in our evolutionary robotics task. This is because our fitness function evaluates the recognition ability of AEM using behaviors and does not directly evaluate actions in certain states as does the reward function of reinforcement learning.

72

S. Yamada and D. Katagami

Fig. 4.1. Overview of AEM

4.2.2

Task: Action-Based Environment Modeling

In AEM [33], a mobile robot is designed by taking a behavior-based approach [2]. Behaviors are mappings from states to actions. A state is described with conditions on sensed values, and an action is described with motor commands. A human designer describes states, actions, and behaviors so that action sequences can represent the environment’s structure. An AEM procedure consists of two stages: a training phase and a test phase (Fig. 4.1). In the training phase, classifiable training environments are given to a robot. The class is a category in which the environment should be included, and more than one environment may be included in the same class. The mobile robot acts in the training environments by using given behaviors, and obtains sequences of executed actions (called action sequences) for each of them. The action sequences (lists of symbols) are transformed into real-valued vectors (called environment vectors) using a chain coding-like method. The environment vectors are stored as instances, and a training phase finishes.

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

73

In the test phase, a robot is placed in a test environment (which is typically one of the training environments). The robot tries to identify the test environment as one of the training environments it has visited, and we call this task environment recognition. Note that although a test environment may be a training environment, the action sequence of the test environment is significantly different from that of the environment because of noise affecting sensing and actions. Thus, some form of generalization is necessary for environment recognition. We used using the 1-nearest neighbor method [7] for this purpose, i.e., the robot selects the most similar instance to the test environment and considers that the class of the instance is that of the test environment. Similarity is measured as the Euclidean distance between environment vectors. We implemented an unsupervised version of AEM in which SOM [15] classified the training environment without a teacher. Through experiments with a real mobile robot, we determined that AEM was promising for environment recognition and adaptive to noisy environments [33]. However, AEM faces a significant problem: Where do the suitable behaviors come from? Since the suitable behaviors depend on the environment structures that the robot should recognize, they have thus far been described by humans. However, the task is potentially very difficult because the search space for a suitable behavior is huge: the computational complexity is O(as ) where a and s are the number of actions and states. Thus, we propose an evolutionary robotics approach to automatically acquire suitable behaviors. 4.2.3

States, Actions and Environment Vectors

Using real mobile robots as individuals in GA is currently impractical because it is impossible to operate dozens of real robots for more than 100 generations. Thus, we decided to use a simulator for acquiring behaviors and then implement the learned behaviors on a real mobile robot. Note that in this research, on-line evolutionary learning for a real robot like [25] is inapplicable. In on-line evolutionary learning, the behaviors of each individual are executed in a few time steps or in a single time step as in a time-sharing system and a fitness function independently evaluates short sequences of actions. However, to evaluate AEM, a large number of actions need to be continuously executed by an individual. Thus, on-line evolutionary learning cannot be used to acquire suitable behaviors for AEM, and real mobile robots find it very hard to learn. 4.2.3.1 A Simple Mobile Robot: Khepera We used a miniature mobile robot, KheperaTM (Fig. 4.2) as a target real robot. It has a Motorola 68331 micro-processor and 256 Kbytes of RAM. As shown in Fig. 4.3, it has two DC motors as actuators and eight infrared proximity sensors which measure the distance to obstacles and light strength. Since the sensor data is imprecise and local, Khepera cannot know its position in a global map. The later experiments used a simulator build for Khepera.

74

S. Yamada and D. Katagami

55mm

32mm

Fig. 4.2. Khepera

Fig. 4.3. Actuators, sensors on Khepera

4.2.3.2 States and Actions We describe a state as a range of a sensed values. To reduce the search space of behaviors, we restrict the number of states and actions. Each sensor on Khepera returns a 10-bit (0∼1023) value for the distance and light strength, and the value is very noisy. Thus we transform the distance value into binary values of 0 or 1. The value ee0ff means an obstacle exists within 3 cm from a robot. The value ee1ff means that no object exists there. Only three sensors (0, 2 and 5 in Fig. 4.3) are used for reducing states. Four sensors (0, 2, 5 and 7 in Fig. 4.3) are used to measure light strength. We chose for this state the sensor measuring the strongest light and assign binary values which mean a light is eenearff or eefarff. A state in which all sensors measure almost the same value is also considered. As a result, the number of states for light is nine. The total number of states is 72 (= 23 × 9) We describe the following four actions. Through preliminary experiments [33, 35], we found that four actions were sufficient for a mobile robot to do simple behaviors like wall-following. • • • •

A1: Go 5 mm straight on. A2: Turn 30◦ left. A3: Turn 30◦ right. A4: Turn 180◦ left.

A mobile robot acts in an environment by executing the above actions, and consequently it generates an action sequence. Note that we utilize action sequences, not behavior sequences, because we consider that an action describes the environment structure and several behaviors may contain the same action. 4.2.3.3 Environment Vectors The generated action sequence is transformed into an environment vector. Let an action sequence and its environment vector be [a0 , a1 , a2 , · · · , an ](ai ∈ {A1, A2, A3, A4}, a0 = 0) and V =(v1 , v2 , · · · , vm )(m ≥ n), respectively. The vector values of V are determined by the following rules. They change the vector value when the direction of movement changes in a way similar to chain coding [1]. An example of an environment vector is shown in Fig. 4.4.

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

75

Fig. 4.4. Transformation into environment vectors

• • • •

If ai If ai If ai If ai

= A1 then vi = vi−1 . = A2 then vi = vi−1+1 . = A3 then vi = vi−1 − 1. = A4 then vi = −vi−1 .

As mentioned in Section 4.2.2, a robot is put in training environments in the training phase. The robot acts in these environments, and stores the environment vectors transformed from the action sequences. Next, in the test phase, the robot is put in a test environment. The robot identifies which training environment corresponds to the test environment by using the 1-nearest neighbor method and the Euclidean distance as the measure of similarity. 4.2.4

Applying GA to Acquire Behaviors

A behavior is a mapping from a state to an action. Since we have 72 states and 4 actions, the number of possible behaviors is 472 = 2.2 × 1043. We can hardly apply conventional mathematical programming to this search. Hence we use GA [11] to search for suitable behaviors to AEM in such a huge search space. Although other evolutionary computation methods like evolutionary system, genetic programming, may be also applicable to this search, we select GA because of its easy implementation and rich experimental knowledge to control it. 4.2.4.1 Ga Procedure and Coding We use a simple GA procedure and experimentally obtained parameters as follows: Step 1: Initialize population: Randomly generate the initial population I1 , · · · , IN . Step 2: Compute fitness: Compute the fitness f1 , · · · , fN for each individual I1 , · · · , IN . Step 3: If a terminal condition is satisfied, end the procedure. The terminal condition is defined in Section 4.2.5. Step 4: Selection: Using f1 , · · · , fN , select a child population C from the population. Step 5: Crossover: Select pairs randomly from C with probability Pcross . Generate two children by applying a crossover operator to each pair, and exchange the children with the pairs in C.

76

S. Yamada and D. Katagami

Step 6: Mutation: Mutate the individuals in C according to the mutation rate Pmut . Step 7: Go to Step 2. • • • • •

Population size: 50. Crossover operator: Uniform crossover. Selection method: Elite strategy and tournament selection (the tournament size = 2). Crossover rate Pcross: 0.8. Mutation rate Pmut : 0.05.

Since we deal with deterministic, not probabilistic, action selection, the behavior is a mapping from a single state to a single action. Thus we use the coding in which one of actions {A1, · · · , A4} is assigned to each state. The genotype is shown in Fig. 4.5. S1

S2

S3

S4

A4 A1 A3 A2

....

S 71 S 72

A4 A2

Fig. 4.5. A coded behavior

4.2.4.2 Defining a Fitness Function Since fitness is a very important for GA, we have to carefully define the fitness function for AEM. We consider three conditions on suitable behaviors for AEM: termination of actions, accuracy of recognition, and efficiency of recognition. Once defined, the fitness functions for each condition are then integrated. Termination of actions A mobile robot needs to stop its actions by itself. Otherwise it may act forever in an environment and no action sequence would be obtained. Thus, the termination of actions is the most important condition. We use homing as the means of terminating actions: the robot stops when it returns to the neighborhood of the start point. Homing makes the length of an obtained action sequence depend on the size of the environment. In contrast, terminating actions after a fixed number of actions does not have this advantage. Termination is evaluated with the following function g. g=

(Number of E-trials) + (Number of H-trials) 2 × (Total number of trials)

(4.1)

where E-trials and H-trials means trials in which a robot escaped from the neighborhood of the start point and trials in which it succeeded in homing. Hence, g evaluates escape as well as homing. Its range is [0, 1], and it returns 1 when the robot succeeds in homing in all of its environments.

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

77

Recognition accuracy Another important criterion is the accuracy of identifying test environments. Accuracy is evaluated with the following function h. Its range is [0, 1], and h = 0 when g = 1. h=

Number of successful test environments Total number of test environments

(4.2)

Recognition efficiency In AEM, a robot acts by operating physical actuators for recognizing an environment, and the actions have a significant cost. Hence, the actions should be as small as possible for efficiency. To enable this, we use the following evaluation function k. k = 1−

n S Σi=1 i n × Smax

(4.3)

where Si is the size of an action sequence obtained in the ith environment i, Smax is the given size of an action sequence, and k = 0 when h = 1. The function has a range of [0,1] and a higher value means more efficiency. The obstacle avoidance is implicitly evaluated by the function k because a collision increases the length of an action sequence.

Fig. 4.6. Whole procedure

78

S. Yamada and D. Katagami

We integrated these three fitness functions into one function f = g + h + k having the range of [0,3]. Since the function h (or k) takes 0 when the value of g (or h) is less than 1, the function f is phased: the actions are terminated when 1≤ f , the recognition is completely correct when 2 ≤ f , and the recognition efficiency is higher when 2 ≤ f < 3. There were an another way to optimize the fitness function f , in which we assigned weights to three terms like f = α g + β h+ γ k and optimize it by applying multi-objective optimization by GA[8]. However, in such a case, the interpretation of the dominant solutions will be very hard. Fig. 4.6 outlines the whole procedure of evolutionary design. It integrates a simple GA procedure with a Khepera simulation for evaluating an individualsf fitness. Note that the fitness is computed from the simulated behaviors. 4.2.5

Simulation Experiments

We implemented our behavior design method on a Khepera simulator [21]. The parameters were as follows: the whole environment was a square (100cm × 100cm), the neighborhood of the start point for homing was a circle with a 100 mm radius, and the action sequence was limited to 2000 actions. This 2000 actions limitation was set by preliminary experiments. In the simulator, the motor had ±10% random noise in velocity, and the robot’s body had ±5% random noise in rotation. Furthermore, each infrared proximity sensor had 10% random noise added to the distance and ±5% random noise added to the light strength. The added noise made the simulator similar to a real environment. If a robot couldn’t return home within a limited number of actions, the trial resulted in zero fitness. The terminal condition of GA was that the maximum fitness of the population exceeds 2 or the generation number reaches 100. The termination and accuracy conditions satisfied when the fitness exceeds 2. The robot was placed in each of the training environments once. The robot acted in the environments, and the environment vectors (transformed from the action sequences) were stored as instances. Next, the training environments were given to the robot as test environments, and the robot tried to identify them. The start points and directions were fixed (bottom, center, and left). As mention above, although the test environments were the same as the training environments, the action sequences were different because of the added random noise. Hence the simulated robot had to learn behaviors that would be robust to such noise. There were 10 different trials in the initial population, and we computed the averages and standard deviations of the fitness for the generation number at which GA stopped. We employed a PC with a Pentium Pro (200 MHz) and 128 Mbytes of RAM and programmed the system in the C programming language on Linux and FWM window manager. By using this computational resource, each learning needed 3 minutes ∼ 5 hours for convergence. 4.2.5.1 Exp-1: Different Contours First, we tested environments with twelve different contours and a part of the environments is shown as emp, L, L2, iL, s-emp in Fig. 4.7. The experimental results are shown

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

(a) emp

(b) L

(c) L2

(d) iL

(e) s-emp Fig. 4.7. A part of environments and trajectories in Exp-1

79

80

S. Yamada and D. Katagami

Table 4.1. Experimental results in Exp-1

Env. Training env. GN MaxF E1 emp, L 1.0 (0) 2.80 (0.106) E2 E1 + L2 2.6 (1.84) 2.43 (0.131) E3 E2 + iL 2.8 (1.69) 2.44 (0.025) E4 E3 + s-emp 2.8 (1.75) 2.44 (0.093) E5 10 env. 2.1 (0.738) 2.51 (0.024) E6 12 env. 5.2 (3.08) 2.48 (0.07)

Fig. 4.8. Fitness for 12 environments

in Table 4.1. Four subsets (E1, E2, E3 and E4 in Table 4.1) of the five environments were given to a robot. Additionally, 10 and 12 different contour environments were used (E5 and E6 in Table 4.1). The “GN” is the generation number in which GA stopped, and “MaxF” means the maximum fitness value at GN. The numbers in GN and MaxF stand for averages, and the numbers in parentheses are standard deviations. This format is common in all the experimental results. Fig. 4.7 also indicates the trajectories of the best individuals at GN in E4. Fig. 4.8 shows the maximum values and average values of fitness at every generation for E6 of Exp-1. The average and maximum fitness increased monotonously. In such simple environments, suitable behaviors for AEM were obtained within a few generations. The standard deviation was large in GN and small in MaxF, and this tendency was observed throughout all the experiments. Seeing from Fig. 4.7, different action sequences were obtained depending on the contours of the environments.

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

(a) emp

(b) 1-la

(c) 2-la

(d) 3-la

(e) 4-la Fig. 4.9. Environments and trajectories in Exp-2A

81

82

S. Yamada and D. Katagami

(a) emp

(b) 1-la

(c) L

(d) L1-la

(e) iT

(f) iT1-la

Fig. 4.10. Environments and trajectories in Exp-3

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

83

Table 4.2. Experimental results in Exp-2A

Env. Training env. GN MaxF E7 emp, 1-la 1.6 (0.966) 2.68 (0.196) E8 E7 + 2-la 4.8 (2.78) 2.59 (0.131) E9 E8 + 3-la 9.3 (5.19) 2.59 (0.111) E10 E9 + 4-la 10.0 (5.94) 2.62 (0.075)

4.2.5.2 Exp-2: Different Lights and Shape Next, we made five different environments, emp, 1-la, 2-la, 3-la, 4-la in Fig. 4.9, by varying the lighting in terms of number of lamps and their position. The light was strong enough that the robot could detect the light direction anywhere in the environment. The experimental results are listed in Table 4.2. Fig. 4.9 also indicates the trajectories of the best individual at GN in E10. The black circles in the figures indicate the lamp positions. Although the GN values exceeded those in Exp-1, suitable behaviors were obtained. Note that we cannot intuitively understand the behaviors in Fig. 4.9. This means that it is very hard for human to design such behaviors by hand-coding and that our automatic design method is effective. Next, we created six environments, emp, 1-la, L, L1-la, iT, iT1-la in Fig. 4.10 by adding different numbers of lights to an environment. Each environment had different classes and all of them should be distinguished. As a result, GN was 13.2 (7.67) and MaxF was 2.61 (0.091). Fig. 4.10 also shows the trajectories of the best individual at GN. The actions in the different environments differed significantly, and hence manual design seems to be difficult. 4.2.5.3 Exp-3: Classes Including More Than One Training Environment In the environments of Exp-1 and Exp-2, each had different classes. However, in this experiment, plural environments are included in a single class. Generalization is necessary to recognize such environments. We assigned three classes to six environments in Fig. 4.9: {emp, 1-la}, {L, L1-la}, {iT, iT1-la}, and made experiments Exp-3. These classes meant the three environments with different contours, in which the light was turned on or off. As a result, GN was 8.8 (4.24) and the maximum fitness was 2.61 (0.091). This showed that our approach was valid for induction from several instances of a class. 4.2.6

Discussion

4.2.6.1 Comparison with Hand-Coded Behaviors It is very difficult for a human to design the behaviors of the experiments described above. The search spaces were huge and there were few heuristics for efficiency. Note also that the obtained behaviors were different from manually designed ones for wall-following, random walking, etc.

84

S. Yamada and D. Katagami

Fig. 4.11. Comparison with wall-following

We compared the learned behaviors with wall-following. Fig. 4.11 plots the average lengths of action sequences for test environments in Exp-1 (Table 4.1). It shows that the automatically learned behaviors are more efficient than wall-following in all environment sets. Thus our automatic design approach designed more efficient behaviors than typical hand-coded behaviors. We repeated Exp-1 by adding 1% random noise to the initial position and orientation of the robot. Although the noise was very small, it complicated the experiments to such an extent that adequate were only acquired in E1, E2, E3. This difficulty has yes to be resolved[27]. 4.2.6.2 Difficulty of Behavior Acquisition As E1 in Table 4.1 indicates, suitable behaviors are already included in the initial population. This implies that although the search space is huge, a large number of suitable behaviors exist for E1. To verify this expectation, we investigated behaviors of individuals with maximum fitness in 10 different trials. The behaviors could be categorized into the five classes shown in Table 4.3. The numbers in parentheses indicate the number of categorized behaviors in the class, and the actions were described earlier. The intuitive explanations for the states are as follows: • • • •

State S0: No obstacle and light around a robot. State S9: Obstacle to the left and no light. State S18: Obstacle in front and no light. State S36: Obstacle to the right and no light.

As Table 4.3 shows, the 10 individual trials executed only the subsets of {S0, S9, S18, S36}. Thus, the actual combinations of states is 44 = 256. Also, BS-1 and BS-2, each had four suitable behaviors because BS-1 did not care about the action for S9 and there were four actions. BS-2 was similar. BS-3, BS-4, BS-5, each had a single suitable

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

85

Table 4.3. Executed behaviors

behaviors behaviors BS-1 (5) S0 → A1 BS-4 (1) S0 → A1 S18 → A2 S9 → A1 S36 → A2 S18 → A2 BS-2 (1) S0 → A1 S36 → A4 S9 → A3 BS-5 (1) S0 → A1 S18 → A3 S9 → A4 BS-3 (2) 0 → A1 S18 → A3 S9 → A3 S36 → A2 S18 → A2 S36 → A4

behavior. Consequently, the total number of suitable behaviors was 11. Since the population size was 50, the probability that any suitable behavior exists in the initial popula50 tion is 1 − ( 256−11 256 ) = 0.89. This supports the fact that suitable behaviors exist in the initial population. Furthermore, we found that the average number of suitable behavior classes was 3.4∼18.4. Hence we found that a robot actually experienced only a few of the 72 states.

4.3 Directly Teaching for Robots in Real Environments An autonomous robot must accomplish tasks in environments where human are present, but it’s difficult to prepare for some situations. The situations, for example, could be dynamic or involve unexpected interactions, and they would require autonomous behavior acquisition and adaptation to environments. Recently, reinforcement learning and evolutionary computation were used as a framework of learning and adaptation. Moreover, research that enables a robot get a controller autonomously has attracted attention. Needless designer bias should be eliminated from the interaction dynamics, with the robot’s embodiment and environment reflected in construction of a controller. To eliminate bias, an agent can learn by trial and error without using prior knowledge in the framework of reinforcement learning. However, the slowness of trial and error learning is a problem in a real environment. Interaction with humans in an environment could be helpful. Particularly for the robots that do not have a priori knowledge or use trial and error in the initial stage, human instruction is a very effective way of acquiring autonomous behavior. However, once it has reached a certain level, an autonomous robot does not have to follow instructions from humans all the time. At this stage, the robot should demonstrate its autonomy based on the instruction rules it stored through interactions with humans. Therefore, we need a technique of establishing a robot’s autonomy through interaction with humans. The approaches mentioned have a design stage in which the designer ignores possible inappropriate bias. Below, we discuss the design of interactions between humans

86

S. Yamada and D. Katagami

and robots, especially interactive design human teachers, as a design logic. Interactive design should account for the teacher’s load and the gap between of environmental recognition of the teacher and a learner. 4.3.1

Supervised Learning and Interactive Evolutionary Computation

Interactive evolutionary computation (IEC) is a popular way of exploiting human teachers. IEC directly incorporates human evaluations and searches in the evolutionary way. It has been used in various fields, e.g. design, music, and medicine [29]. IEC’s main problem is fatigue; the human teacher has to cooperate with a tireless computer, and such a process cannot be continued for many generations. Lund et. al. proposed an evolutionary robotics approach using IEC for constructing a controller for LEGO robots [19]. They reduced the construction time by merging an interactive genetic algorithm with reinforcement learning and employing neural networks as the robot controller. The IEC realized a programming-less program for children’s games and toys. Their experiment was surprising because only nine moving robots was all it took to generate neural networks for controlling a robot to do interesting movements after only five to seven generations. Generally, in interactive evolutionary learning, the more that is taught, the better the performance is. However, human labor is not unlimited. Evaluating many individuals (or rules) and inputting evaluations for every generation are extremely tiresome tasks in practice. Crandall et al. devised a theoretical approach to achieving interaction efficiency in designing human-robot coordination systems that considered cognitive fatigue [5]. By measuring hours worked, number of answers, and accuracy rate in a paired task, they showed that cognitive fatigue from a direct control system is higher than from a a shared-control system. We consider the following ways to apply their insights on interactive learning to a real robot environment. If a robot does not have an order from a teacher, then it tries to learn automatically. If a teacher intuitively teaches a robot through an input device (like a joystick) with arbitrary timing, the robot makes rules automatically. Fatigue might be reduced in this manner. Horiguchi et al. [12] used the concept of mutual leadership pattern interaction to design human-robot interaction and realized cooperative automation between a mobile robot and a human through power feedback. However, the result of learning had no effect on the behavior acquisition of the robot. 4.3.2

Teacher View and Learner View

There are big differences between a robot’s recognition and a human teacher’s one. Mishima and Asada et al. have proved that the efficiency of learning gets worse by passive teaching for the gap (cross perceptual aliasing) of the environmental recognition between a teacher and a learner [22]. Yet in terms of efficiency, passive teaching is considered to be a good method even though the teacher has to monitor the robot until it needs assistance.

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

87

We dealt with the gap between a teacher and a learner as a design problem[14]. We devised a faster learning system by using a classifier system that allows a teacher teach the appropriate action by directly teaching with a joystick controller. In the experiments, we could select two different view to teach a robot: teacher’s view (TV) and learner’s view (LV). By using TV, a teacher teaches a robot in the view point of a bird, and see a whole environment. In contrast, using LV, a teacher can see just the local scene which a robot actually sees in the environment. We were interested in gap between TV and LV, and their influence to interactive robot learning like interactive classifier system. This issue was experimentally investigated in Section 4.3.4. 4.3.3

Interactive Classifier System

In order to realize this evolutionary robotics by IEC, we developed a robot learning system, called an interactive classifier system (ICS). Fig.4.12 shows the teaching environment that we envision. Including the interactive function of IEC in a learning classifier system (LCS), the ICS is a learning model for a robot that can be instructed by a human or can autonomously learn. XCS [30] is a LCS algorithm that Wilson proposed. XCS adds a parameter called accuracy to ZCS [31]. Although the classifier was generalized by including “don’t care” symbols to the conditional part, the classifier system was not able to utilize it effectively. This problem originates in that LCS does not have a mechanism to advance generalization appropriately, and a phenomenon in which an over-generalized classifier degrades performance has been reported [4]. In XCS, both intensity and accuracy are used to check the validity of an over-generalized classifier. This accuracy is calculated according to the error of the reward received as a result of classification and a predicted value. It is reported that a rule can be appropriately generalized in this way [30]. Fig.4.13 shows an outline of the interactive classifier system.

Operator

Actual Environment

Monitor

Robot

Joystick Sensor Information

Teaching ICS

Interactive Classifire System

Fig. 4.12. Teaching environment

Action

88

S. Yamada and D. Katagami

Environment operator joystick

robot

monitor

sensor information

"left"

ICS SPC

DC 01

[P] p 43 32 14 27 18 24

#011:01 11##:00 #0##:11 001#:01 #0#1:11 1#01:10

e .01 .13 .05 .24 .02 .17

F 99 9 52 3 92 15

Match Set

Prediction Array

Action Set

[M] #011:01 #0##:11 001#:01 #0#1:11

RGC

43 14 27 18

.01 99 .05 52 .24 3 .02 92

nil 42.5 nil 16.6

#011:01 001#:01

43 .01 99 27 .24 3

max discount

GA RC

+

delay=1

Previous Action Set [A] -1

Fig. 4.13. Overview of interactive classifier system

ICS consists of a rule generation component (RGC), a sensor processing component (SPC), a display component (DC) and a reinforcement component (RC). All of them are developed on Linux and programmed by the C language and GTK+. Each module is explained as follows. RGC creates the rule by instruction. A teacher operates a robot by using an input equipment and looking at the robot’s information displayed on a GUI. SPC receives the operation history of a there, and the sensor information of the robot at that time. RGC creates a rule newly from it, and it adds to a rule list. The creation procedure of a rule was improved so that a rule could be created from instruction information (the action to which an operator operated the robot) from a human teacher on the basis of XCS [30]. 1. ICS receives robot’s sensor information X and instruction information from SPC. 2. Some classifiers that matched X are moved from a population [P] to a match set [M]. ICS turns regularly the prediction value of classifier which supports each act ai in [M] with a fitness value, and creates P(ai ). The value of P(ai ) is put on prediction array, and the act of classifier chosen by P(ai ) is chosen by action selection

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

89

methods. The action selection methods are performed by a deterministic selection or a roulette wheel selection. 3. If a j = at to compare act a j chosen by act selection methods and act at obtained by teaching, the action part of the rule which has a j in an action part in [M] will be rewritten to at . A change will not be made if a j = at . 4. The action set [A] which consists of classifiers in [M] which support selected action a j is created. When an action a j or at is sent to an actuator, and in case of at , reward rteach is given immediately. When there is no input of at , remuneration rimm is returned from an environment. RC is a reinforcement learning module in a classifier system. It learns by updating the parameter of classifiers chosen last time step. When there is no operation of a teacher, a robot can act autonomously with the rule created thus far. DC indicates the data processed by SPC. GTK+ is used for development of an interface. SPC performs processing of a robot’s various sensors and processing of teaching information. The information is sent to DC and RGC and the processed data is displayed and ICS creates classifiers from them. 4.3.4

Robot Experiments in the Real World

We conducted experiments in a real world to investigate difference in two teaching methods: TV and LV as mentioned in Section 4.3.2. Fig. 4.14 shows an experimental environment. As an experimental task, ICS reduced the number of steps from any start points to a light source which set up as a goal in a field surrounded with white plastic plates. We compared two teaching methods (TV and LV) with a traditional method in which a robot autonomously learns by simple EC. To investigate influence by gap between operator’s recognition and robot’s one, we prepared experiment-1 without an obstacle and experiment-2 with an obstacle. We considered 40 steps as a trial and began

Fig. 4.14. Experimental environment

90

S. Yamada and D. Katagami

teaching or autonomous exploration for five trials at random start points, and tested a trial at each of five start points as examination of learned rules. It performed this procedure six times, and consequently we had 30 trials for evaluation. 4.3.4.1 Experimental Settings The experiments were made with a standard miniature mobile robot Khepera (Fig.4.2). The mobile robot has cylindrical shape, a diameter of 6 cm and a height of 5 cm. It possesses two motors and on-board power supply. The motors can be independently controlled by a PID controller. It is equipped with eight infrared proximity sensors. The eight infrared sensors are distributed around the robot in a circular pattern. They emit infrared light, receive the reflected light and measure distances in a short range: 2-5 cm. The robot is also equipped with a Motorola 68331 micro-controller which can be connected to a computer via serial cable. Moreover, the system utilizes SONY analog controller DUALSHOCK as a joystick. Task goal was minimization of the number of steps which a robot needed to reach the goal from arbitrary initial locations in the region surrounded by white plastic board. We compared TV, LV and a learning method by using classifier system. An encoding of a classifier was as follows. A classifier was the 20-bits string “#000#100000###100#:01”. The robot’s condition was the left 16-bits string, it represented eight infrared proximity sensors (the left 8-bits) and light sensors (the right 8-bits) around the robot in a circular pattern. The bit was “1” if an input sensor value was larger than a threshold, else “0”. “#” is a “don’t care” symbol. The robot’s previous action was the next 2-bits string, represented as forward “11”, left-turn “01”, right-turn “10” and back “00”. The robot’s current action was a 2-bits string similarly. A fitness function of ICS was defined as follows. Reward F was computed by the sum of eight light sensors through a sigmoid function. 7

u = ∑ lighti i=0

PaymentRange , lightMAX × 8

F=

1 1 + exp(−u)

(4.4)

In the case of TV, the operator could recognize a small obstacles as the robot could perceive although he/she looked a whole environment. To implement TV, an operator taught a robot using a camera by which he/she could look a whole environment. Fig.4.15 shows information through GUI by a camera. In the case of LV, an operator could not look a whole environment although it could recognize directly a small obstacles and a recognition error which the robot perceived. To implement LV, ICS used GUI indicating sensor values. Since an operator hardly recognized an environment by reading only numerical values of sensor data, we developed GUI interface which can represent sensor values graphically. Fig.4.16 shows such sensor information through GUI interface. We considered 40 steps as a trial and began teaching or autonomous exploration for five trials from random start points, and tested a trial at each of five start points as examination of created rules by the teachings or the exploration. It performed this procedure seven times, and consequently we had 35 trials as examination. Table 4.4 shows experimental parameters.

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

91

Fig. 4.15. External Observation Settings: A teacher acquires environmental information by bird’seye view. The teaching is performed from the viewpoint of whole environment.

Fig. 4.16. Internal Observation Settings: A teacher acquired a robot’s sensor information and used it for teaching. The teaching was performed through interface which displayed robot’s sensor information.

92

S. Yamada and D. Katagami Table 4.4. Experimental Parameters

Parameters The number of problems in one experiment The number of experiments The maximum size of the population The probability to do crossover The probability of mutating one bit

Value 35 1 300 0.8 0.04

4.3.4.2 Experimental Results In the experiment-1, we examined the number of reach times to a light source from five start points. Although LV began to success the task at early stage, it didn’t success later. On the other hand, TV improved robot’s learning and achieved the task from all five start points before LV. TV improved a robot’s learning in the simple environment in which cognitive difference is little because an operator looked a whole environment. However, the difference is not so large. In the experiment-2, we introduced an obstacle to the environment for making a task more difficult. The obstacle was made of the transparent plastic board because so that a robot might perceive a direction of a light source. A robot must reach to the light source as avoiding the obstacle. We compared a teaching method by TV with LV in the same way as experiment-1. We examined the number of steps to a light source. Since ICS hardly used information of proximity sensors and acquired effective rules by using only information of light sensors in the experiment-1, the difference of cognition between an operator and a robot was little. For this reason, the difference of two teaching methods was not so large. ICS hardly learns from teaching of an operator in experiment-2 because there is the difference between the situation of the robot which an operator estimated by observation from the outside and the one of the robot in the real world. LV actually created effective rules because an operator performed teaching as he/she checked robot’s internal status. After we conducted experiments for TV and LV each twenty trials, we tested a trial at each of five start points as examination using each learned rules. Table 4.5 shows the experimental results. The robot could not reach to a light source at any start points of five in the examination of TV. On the other hand, the robot reached to a light source by six steps and nine steps respectively at two of five start points in LV. We can see that ICS can create effective rules independently of start points since LV improves the robot’s learning by teaching. Fig.4.17, 4.18 showed typical teaching behaviors of TV and LV respectively. In the case of TV, a teacher acquired locations of obstacles in advance, but could not the Table 4.5. Results of experiment-2

Trial 1 2 3 4 5 TV - - - - LV 6 9 - - -

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

93

Fig. 4.17. TV example of teaching

Fig. 4.18. LV example of teaching

changes of a robot’s sensor information because a teacher can observe a whole environment. On the other hand, a teacher observed the changes of internal situation in LV because a teacher could see robot’s information by using a sensor panel. 4.3.5

Teaching Efficiency

Table 4.6 shows ten rules which have best values of the system prediction in created ones after twenty trials with LV. ICS created effective and commonsense rules like as the robot moves forward when the light source faces in front of the robot and it

94

S. Yamada and D. Katagami

Table 4.6. Created rules by LV method

Condition Action Prediction Teaching 0#0#0#####0#10##1# 10 413.7 yes 0010###0##000#1### 01 364.0 yes 00###0#000#1#0#01# 11 292.0 yes ###000000101001011 11 256.0 yes

moves right when the light source faces right. In this table, “Prediction” is a fitness of a classifier and “Teaching” is whether it is taught by human or its offspring. Its number is the number of which a taught classifier alternate generations. We see that teaching was very effective because all the rules whose the system prediction is high created from teaching. Although an operator did not teach a robot to go back in the both experiments, the robot went back for avoiding an obstacle when the robot collided with a wall and reached to a light source. Because ICS created these rules by which a robot worked in cooperation with human.

4.4 Conclusion We reported evolutionary acquisition of suitable behaviors to Action-based Environment Modeling. GA was applied to search the behaviors, and the simulated mobile robots were used as individuals. States and actions were described for coding chromosomes, and we carefully defined a fitness function. We conducted various experiments using environments with different contour, lights and both of them, and found that our approach was promising to learn suitable behaviors for AEM. Furthermore some of learned behaviors were more efficient than hand-coded ones. Next, we proposed a novel interactive method from the viewpoint of observation and investigated its effects in a real world experiments. The direct teaching method to train a mobile robot in a real environment. It was found that internal observation increased the effects according to arise the difference between the robot’s recognition and operator’s one in a more complicated environment.

References 1. Arkin, E.M., Chew, L.P., Huttenlocher, D.P., Kedem, K., Mitchell, J.S.B.: An efficiently computable metric for comparing polygonal shapes. IEEE Transaction on Pattern Analysis Machine Intelligence 13(3), 209–216 (1991) 2. Brooks, R.A.: A robust layered control system for a mobile robot. IEEE Transaction on Robotics Automation 2(1), 14–23 (1986) 3. Cho, S.B., Shimohara, K.: Emergence of structure and function in evolutionary modular neural networks. In: Proceedings of the Fourth European Conference on Artificial Life, pp. 197–204 (1997) 4. Cliff, D., Ross, S.: Adding temporary memory to ZCS. Adaptive Behavior 3(2), 101–150 (1994)

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

95

5. Crandall, J.W., Goodrich, M.A.: Evolutionary robotics - a children’s game. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1290–1295 (2002) 6. Crowly, J.L.: Navigation of an intelligent mobile robot. IEEE Transaction on Robotics Automation 1(1), 34–41 (1985) 7. Dasarathy, B.V.: Nearest Neighbor (NN) Norms: NN Pattern Classification Techniques. IEEE Computer Society Press, Los Alamitos (1991) 8. Deb, K.: Multi-Objective Optimization Using Evolutionary Algorithms. John Wiley and Sons, Chichester (2001) 9. Floreano, D., Mondada, F.: Evolution of homing navigation in a real mobile robot. IEEE Trans. Systems, Man, Cybernetics. Part B: Cybernetics 26(3), 396–407 (1996) 10. Harvey, I., Husbands, P., Cliff, D.: Issues in evolutionary robotics. In: Proceedings of the Second International Conference on Simulation of Adaptive Behavior, pp. 364–373 (1992) 11. Holland, J.: Adaptation in Natural and Artificial Systems. University of Michigan Press (1975) 12. Horiguchi, Y., Sawaragi, T., Akashi, G.: Naturalistic human-robot collaboration based upon mixed-initiative interactions in teleoperating environment. In: IEEE International Conference on Systems, Man, and Cybernetics, pp. 876–881 (2000) 13. Ito, T., Iba, H., Kimura, M.: Robustness of robot programs generated by genetic programming. In: Genetic Programming 1996, Proceedings of the First Annual Conference, pp. 321– 326 (1996) 14. Katagami, D., Yamada, S.: Interactive classifier system for real robot learning. In: IEEE International Workshop on Robot and Human Interaction, pp. 258–263 (2000) 15. Kohonen, T.: The self-organizing map. Proceedings of the IEEE, 1464–1480 (1990) 16. Koza, J.R.: Evolution of subsumption using genetic programming. In: Proceedings of the First European Conference on Artificial Life, pp. 110–119 (1991) 17. Koza, J.R.: Genetic Programming. MIT Press, Cambridge (1992) 18. Lund, H.H.: Adaptiverobotics in entertainment. Applied Soft Computing 1(1), 3–20 (2001) 19. Lund, H.H., Miglino, O., Pagliarini, L., Billard, A., Ijspeert, A.: Evolutionary robotics - a children’s game. In: Characterizing Efficiency of Human Robot Interaction: A Case Study of Shared-Control Teleoperation, pp. 148–156 (1998) 20. Mataric, M.J.: Integration of representation into goal-driven behavior-based robot. IEEE Transaction on Robotics Automation 8(3), 14–23 (1992) 21. Michel, O.: Khepera simulator v.2 user manual. University of Nice-Sophia (1996) 22. Mishima, C., Asada, M.: Active learning from cross perceptual aliasing caused by direct teaching. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1420–1425 (1999) 23. Nakamura, T., Takamura, S., Asada, M.: Behavior-based map representation for a sonarbased mobile robot by statistical methods. In: Proceedings of the 1996 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 276–283 (1996) 24. Nehmzow, U., Smithers, T.: Map-building using self-organizing networks in really useful robots. In: Proceedings of the 1st International Conference on Simulation of Adaptive Behavior, pp. 152–159 (1991) 25. Nordin, P., Banzhaf, W.: A genetic programming system learn-ing obstacle avoiding behavior and controlling a miniature robot in realtime. Technical report, Department of Computer Science, University of Dortmund (1995) 26. Olton, D.S.: Spatial memory. Scientific American 236(6), 82–99 (1977) 27. Reynolds, C.W.: Evolution of obstacle avoidance behavior: using noise to promote robust solutions. In: Kinnear, J.K.E. (ed.) Advances in Genetic Programming, vol. 1, pp. 221–241. MIT Press, Cambridge (1994)

96

S. Yamada and D. Katagami

28. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998) 29. Takagi, H.: Interactive evolutinary computation: Fusion of the capabilities of ec optimization and human evaluation. Proceedings of the IEEE 89(9), 1275–1296 (2001) 30. Wilson, S.W.: Classifier fitness based on accuracy. Evolutionary Computation 3(2), 149–175 (1995) 31. Wilson, S.W.: ZCS: a zeroth order classifier system. Evolutionary Computation 2(1), 1–18 (1995) 32. Yamada, S.: Evolutionary design of behaviors for action-based environment modeling by a mobile robot. In: Proceedings of Genetic and Evolutionary Computation Conference, pp. 957–964 (2000) 33. Yamada, S.: Recognizing environments from action sequences using self-organizing maps. Appllied Soft Computing 4(1), 35–47 (2004) 34. Yamada, S.: Evolutionary behavior learning for action-based environment modeling by a mobile robot. Applied Soft Computing 5(2), 245–257 (2005) 35. Yamada, S., Murota, M.: Unsupervised learning to recognize environments from behavior sequences in a mobile robot. In: Proceedings of the 1998 IEEE International Conference on Robotics and Automation, pp. 1871–1876 (1998)

5 Ranked Pareto Particle Swarm Optimization for Mobile Robot Motion Planning D. Wang, N.M. Kwok, D.K. Liu, and Q.P. Ha ARC Centre of Excellence for Autonomous Systems, Faculty of Engineering, University of Technology, Sydney, Broadway, NSW 2007, Australia {Da-Long.Wang,nmkwok,dkliu,quangha}@eng.uts.edu.au

Abstract. The Force Field (F 2 ) method is a novel approach for multi-robot motion planning and coordination. The setting of parameters in the (F 2 ) method, noticeably, can affect its performance. In this research, we present the Ranked Pareto Particle Swarm Optimization (RPPSO) approach as an extension of the basic idea of Particle Swarm Optimization (PSO), which makes it capable of solving multiobjective optimization problems efficiently. In the RPPSO, particles are initiated randomly in the search space; these particles are then evaluated for their qualities with regard to all objectives. Those particles with highly-ranked qualities have preferences to enter the set of Global Best vectors, which stores many currently best solutions found by particles. Thus, particles in RPPSO will search towards many possible directions and the diversity among solutions is well preserved. Ideally, a set of optimal solutions will be found when the termination criterion is met. The effectiveness of the proposed RPPSO is verified in simulation studies. Satisfactory results are obtained for multiobjective optimization problems of multi-robot motion planning in challenging environments with obstacles.

5.1 Introduction An existing trend in robot applications is the utilization of a team of robots to accomplish complex tasks. Examples may include the deployment of autonomous unmanned vehicles in container terminals and material handling, the exploration of unknown terrain and cooperation in robot soccer, etc. For a team of robots operating in a common work space, motion planning and coordination for multiple robots has arisen as a very important issue to be addressed. Motion planning is one of the fundamental problems in robotics. It can be described as the construction of collision free trajectories that direct robots to their destinations. Researchers have developed a variety of algorithms on robot motion planning [1, 2]. Although many algorithms have been proven to be feasible and efficient for single robot motion planning and collision avoidance, they cannot be transferred directly to multi-robot systems. The existing methods can be broadly categorized into centralized and decentralized approaches. Centralized approaches consider all robots together as if they were forming a high degree-of-freedom system and capable of providing complete and optimal solutions. However, as the number of robots and obstacles in the working environment increases, such approaches may suffer from the exponentially increasing computational complexity. Decentralized approaches generate independent paths for D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 97–118. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

98

D. Wang et al.

individual robots and avoid collisions locally. Decentralized approaches are not affected by the number of robots but are usually incomplete, i.e., they may fail to find a solution even if it exists. The Potential Field method has been a popular approach for autonomous mobile robot and manipulator path planning in the past decades because of its mathematical simplicity and computational efficiency [3]. The basic concept of this method is to fill a working environment with an artificial potential field in which the robot is attracted by the goal and repulsed away by obstacles. Researchers have developed a variety of methods based on the potential field concept. For example, Connolly presented a method using Laplace’s Equation to avoid local minima [4]. Ge and Cui developed a potential field method which defines attractive and repulsive potentials by taking account of the relative position and velocity of a robot with respect to obstacles and targets [5]. A potential field model using generalized sigmoid functions is proposed in [6] to meet the requirement of accurate representation of complex objects in applying the artificial potential field in some practical applications. The F 2 (Force Field) method is a decentralized method for multi-robot path planning and collaboration [7]. Instead of generating potential fields or force fields based on the locations of obstacles, a virtual force field is constructed for each robot according to its status, including traveling speed, dimension, task priority, location and environmental factors, etc. The force field of a robot is different from those of other robots due to its different status and varies with the robot during its movement. A robot with larger volume, traveling with higher speed, or with higher task priority than other robots will have priority in collision avoidance. The interactions among the robots’ force fields and obstacles provide a feasible and efficient way for multi-robot motion planning and collaboration. The concept of F 2 is a further refinement of the Potential Field to some extent. Both concepts use the repulsive potential/force field to avoid collision with obstacles and the attractive potential/force to guide a robot to its target. But the differences between the F 2 and Potential Field are distinct. The Potential Field is generated based on environment information. That is to say, the potential value of a point in a potential field is determined by its location with respect to the goal and obstacles nearby. The potential field remains unchanged if environment does not change. In the F 2 method, the repulsive force field of a robot is generated around the robot body, instead of around obstacles as those of potential field based approaches. This force field is continuously changing during this robot’s movement. Collision avoidance is achieved by interaction of this protective field with the environment. This research builds upon the F 2 method proposed in our previous work [7], where the construction of force fields uses a number of parameters. The setting of these parameters can noticeably affect the method’s performance. Since the problem is highly coupled, non-linear and incomplete, closed-form solutions are not always available. To this end, an efficient and implement simple algorithm, such as the Particle Swarm Optimization (PSO) algorithm, is very attractive. A single objective optimization method was presented in our previous work [8], in which PSO is utilized to find suitable parameters for F 2 method. But the PSO-tuned F 2 method only deals with the optimization of a single objective. PSO has been proposed for multi-objective optimization in

5 Ranked Pareto Particle Swarm Optimization

99

tuning control parameters of the virtual robot tracking structure for robot formation [9]. In this research, we present a multiobjective optimization method - Ranked Pareto Particle Swarm Optimization (RPPSO) for motion planning of multiple mobile robots taking into account collision avoidance between robots and obstacles as well as minimum travel lengths. The RPPSO follows the basic idea of PSO but is capable of solving multiobjective optimization problems efficiently. At the beginning of RPPSO, particles are initiated randomly in the search space. These particles are then ranked by their qualities with regard to all objectives. Those particles with good qualities constitute the Global Best, which stores many current best solutions found by particles. In the same time, each particle maintains a memory of its previous best position, denoted by Particle Best. Particles then update their locations based on Global best and Particle best. Thus particles in RPPSO will search towards many possible directions. Ideally, a set of optimal solutions will be found when the termination criterion is met. The rest of the chapter is organized as follows. The proposed RPPSO method for multiobjective optimization is presented in Section 5.2. In Section 5.3, the F 2 method used for robot coordination is discussed. The effectiveness of RPPSO is evaluated through simulations on motion planning problems in Section 5.4. Finally, a conclusion is drawn in Section 5.5.

5.2 Ranked Pareto Particle Swarm Optimization In this section, we first introduce some key concepts in a Multiobjective Optimization Problem (MOP). Since the proposed optimization method builds upon the idea of Particle Swarm Optimization (PSO), we will then give a brief description of PSO. Our multiobjective optimization algorithm, the Ranked Pareto Particle Swarm Optimization (RPPSO) method, will be given in detail. 5.2.1

Key Concepts in MOPs

The concept of Pareto Dominance was proposed by Pareto and has been widely used in the research of MOPs. In general, an objective vector Fa in a minimization problem is said to dominate another objective vector Fb , denoted by Fa ≺ Fb , iff fa,i ≤ fb,i

∀ i ∈ {1, 2, ..., m}

(5.1)

fa, j < fb, j

∃

(5.2)

and j ∈ {1, 2, ..., m}

where fa,i and fa, j are components of Fa , fb,i and fb, j are components of Fb . For a multiobjective problem, a solution is said to be Pareto Optimal if it is not dominated by any other solution in the search space. The set of all Pareto Optimal solutions is called a Pareto Set. The aim of MOP is to derive a set of non-dominated solutions with objective values as close as possible to the objective values of the Pareto Set, denoted by Pareto Front.

100

D. Wang et al.

The so-called Multiobjective Evolutionary Algorithm (MEA) has been shown to be suitable for solving multiobjective problems [10, 11, 12, 13, 14]. At each iteration, fitness values (sometimes referred to as the objective values) are evaluated in order to determine better solutions for the next generation. As a result, this will ideally lead to a population of optimal solutions when some termination condition is satisfied. Following this, MEAs have to overcome two major problems [15]. The first problem is how to get close to the Pareto optimal front. The second is how to keep diversity among the solutions in the obtained set. These two problems have become common criteria for most current algorithmic performance comparisons. 5.2.2

Particle Swarm Optimization

Particle Swarm Optimization (PSO) was proposed by Kennedy and Eberhart in 1995, motivated by the social behaviour of organisms such as bird flocking [16]. In PSO, a problem space is covered with an initial population of random solutions in which they are guided to search for the optimum over a number of generations. The concept of PSO is that each particle randomly searches through the problem space by updating itself with its own memory and the social information gathered from other particles. Due to its simple mechanism and high performance of global optimization, PSO has been successfully applied for many optimization problems, including motion planning problems [17, 18, 19]. Suppose the dimension of the searching space is M and the number of particles is N. The position of the ith particle is represented as Xi = (xi1 , xi2 , ..., xiM ). Each particle maintains a memory of its previous best position denoted by Pbesti = (pi1 , pi2 , ..., piM ). The current best position found by all particles is denoted by Gbest = (g1 , g2 , ..., gM ). Vector V = (vi1 , vi2 , ..., viM ) is the velocity of the ith particle. Each particle updates its position according to the following equations: Vnew = ε × Vold + c1 × rand1() × (Pbest − X) + c2 × rand2() × (Gbest − X) Xnew = Xold + Vnew

(5.3) (5.4)

where ε is the inertia factor, c1 and c2 are two positive constants and rand1() and rand2() are two functions returning random numbers in the range [0, 1]. 5.2.3

Ranked Pareto Particle Swarm Optimization

This section introduces the Ranked Pareto Particle Swarm Optimization method (RPPSO). The flowchart of RPPSO is given in Fig. 5.1 (where equations invoked in calculating the variables are indicated within brackets). Symbols and abbreviations used are listed in Table 5.1. Calculation of the intermediate variables is based on the equations enclosed in brackets in the flowchart. 5.2.3.1 Fitness Assignment Consider a multiobjective minimization problem, min F(X), X ∈ Rn P∈φ

5 Ranked Pareto Particle Swarm Optimization

Start

Initialize particles

(5.7) (5.7)

Calculate Raw Fitness Value (RFV) of each function for particles

(5.8) (5.9) Modified Fiteness Value (MFV) indicates the particle qualities with regard to single objective (5.10) Global Modified Fitness Value (GMFV) indicates the particle qualities with regard to all objectives in this iteration (5.11) Particles are ranked according to their qualities

(5.12) Particle Rank Value (PRV) are rounded to integers Particle Copy Number (PCN)

Particles with good qualities form the Global Best

The best histories of particles form the Particle Best

(5.13) (5.14)

New particle locations

N Termination?

Y

End

Fig. 5.1. Flowchart of RPPSO

101

102

D. Wang et al.

Table 5.1. Nomenclature in RPPSO method

Symbols n m PN RFV RFV M MFV MFVV GMFV PRV PCN Gbest Pbest

Descriptions number of decision variable or parameters number of objectives to be minimized particle numbers raw fitness value raw fitness value matrix modified fitness value modified fitness value vector global modified fitness vector particle rank value particle copy number global best particle best

where X = {x1 , x2 , ....xn } is an n-dimensional vector having n decision variable or parameters and φ defines a feasible set of P. F = { f1 , f2, ....., fm } is a vector with m objective components to be minimized. At the beginning of iterations, some particles are initialized randomly in the search space. Assume particles are denoted by Xi . As stated above, Xi is an n-dimensional vector having n decision variables or parameters: Xi = {x1 , x2 , ......xn } ,

i = 1, 2, ......, PN

(5.5)

where n is the number of parameters, PN is the number of particles used in the optimization problem. For any objective, a raw fitness value (RFV) is obtained for each particle: RFV j,i = f j (Xi ),

j = 1, 2, ......, m

(5.6)

where i is the particle number. Thus we get a raw fitness value matrix (RFVM), which is a m × PN array of raw fitness values of this generation: ⎡ ⎤ RFV1,1 RFV1,2 · · · RFV1,PN ⎢ RFV2,1 RFV2,2 · · · RFV2,PN ⎥ ⎢ ⎥ RFV M = ⎢ (5.7) ⎥ .. .. .. .. ⎣ ⎦ . . . . RFVm,1 RFVm,2 · · · RFVm,PN The raw fitness value of each particle is compared with raw fitness values of all other particles. We define that for a minimization problem f j , particle i1 is said to dominate particle i2 at this problem, iff RFV j,i1 < RFV j,i2 ,

j = 1, 2, ......, m

(5.8)

For each particle, the modified fitness value (MFV) on an objective is the number of other particles which are dominated by this particle. MFV shows the quality of particles

5 Ranked Pareto Particle Swarm Optimization

103

regarding objective f j , i.e., particles with higher MFV are more desirable than those particles with lower MFV. For objective f j , a PN-dimensional vector of modified fitness value (MFVV) of this generation is obtained:  MFVV j = MFV j,1 , MFV j,2 , ......MFV j,PN , j = 1, 2, ...m (5.9) For every objective, the same processes is repeated and a MFVV is obtained. The sum of all MFVVs is called to be a global modified fitness vector (GMFV), which is a PN-dimensional vector: GMFV = ∑ MFVV j (5.10) j=1,...,m

The GMFV can be treated as a benchmark of particle quality, in which particle quality with regard to all objectives are taken into account. Particles with higher GMFVs have higher aggregate qualities than those with lower GMFVs in general. Please note that a particle with higher GMFV does not need to have better MFVs (or RFVs) on each objective than those with lower GMFVs. 5.2.3.2 Global Best Here, we set up rules to decide the Global Best, denoted by Gbest . In traditional PSO, Global Best stores the current best location found by all particles. Global Best in RPPSO does almost the same task, but Global Best in RPPSO is a PN-dimensional vector which stores PN best locations found by all particles. To do this, we first set a particle rank value, denoted by PRV, for each particle. The PRV of ith particle is defined by GMFVi PRVi = × PN, i = 1, 2, ..., PN (5.11) ∑ GMFV j j=1,...,PN

The PRVi is a decimal fraction which is proportional to the GMFVi of a particle. This value is then rounded to an integer, denoted by Particle Copy Number(PCN), which determines how many copies of this particle will enter the Global Best of this generation: PCNi = Round(PRVi),

i = 1, 2, ..., PN

(5.12)

To ensure that there are enough candidates for Global Best, this function rounds to the nearest integer towards infinity. The particles which hold higher PRVs have the priority to enter the Global Best vector. This process continues until the Global Best vector is filled up. Particles with lower PRVs, not contributing to solution improvements, will not enter the Global Best vector if it has been filled by copies of particles with higher PRVs and redundant particles will be abandoned. 5.2.3.3 Particle Best The best histories of particles are stored in a PN-dimensional vector, denoted by Pbest . To achieve this, the algorithm keeps the records of the best history of each particle. For a particle, its current location will replace its Pbest only when its current raw fitness values (RFVs) for all objectives are better than those of current Pbest .

104

D. Wang et al.

5.2.3.4 Update Each particle updates its position according to the following equations. vnew = ε × vold + c1 × rand1() × (Pbest − Xold ) + c2 × rand2() × (Gbest − Xold ) (5.13) Xnew = Xold + vnew

(5.14)

These equations look similar to those of conventional PSO. However Gbest in standard PSO usually only stores one best location for all particles. In RPPSO, Gbest is a PN-dimensional vector which stores best PN positions found by particles. As a consequence, particles in RPPSO will search towards many possible directions. In this way, the diversity among solutions is well preserved. 5.2.3.5 Case Study In this section, a simple case is studied to illustrate the process of RPPSO. This problem consists of two variables (x1 and x2 ) space and two objective functions ( f1 and f2 ). f1 (x1 , x2 ) = x21 − 2 × x2, f2 (x1 , x2 ) =

x1 , x2 ∈ [0, 5]

| x1 − 0.2 | + (2 × x2 + x1 )2 ,

x1 , x2 ∈ [0, 5]

(5.15) (5.16)

There are 5 particles in the search space. Particle 1 to 5 are denoted by star, square, diamond, circle and plus sign, respectively. Figure 5.2 (a) shows the particle locations in the 4th generation. (b) and (c) show the locations of Gbest and Pbest before the 4th generation. (e) and (f) show the locations of Gbest and Pbest after this generation. (d) shows the updated particle locations in the 5th generation. There are two points which should be noted in Fig. 5.2. • Gbest in RPPSO may contains some duplicate particles, e.g., square and star in (b), circle and diamond in (e). That means good particles have higher chances to enter the next generation than others. • From (c) and (f) we can see that some values of Pbest changed in this generation (square and star). This is because particle 1 (star) and 2 (square) have better fitness values than before and replaced corresponding Pbest . See (a), (c) and (f). In order to acquire the Pareto Front, we increase the number of particles to 2000. Fig. 5.3 shows the optimization results. The gray dots in Fig. 5.3 illustrate the particle locations and the black stars show the Pareto Optimal Set. If a third objective function f3 (x1 , x2 ) = x2 is added into this problem, the optimization results are given in Fig. 5.4 again. The gray dots show the particle locations and black stars show the Pareto Optimal Set. The number of particles in this simulation is 2000.

5 Ranked Pareto Particle Swarm Optimization

Fig. 5.2. Snapshots of the progress of RPPSO

105

106

D. Wang et al.

Fig. 5.3. Optimization results - 2 objectives

Fig. 5.4. Optimization results - 3 objectives

5.3 F 2 Method In the problem scenario considered in this work, robots are assumed to travel in a 2D environment and their locations can be precisely known. Each robot is aware of the updated information of all other robots, such as their locations, speeds, etc. For

5 Ranked Pareto Particle Swarm Optimization

107

simplicity of explanation, robots are represented as round discs, but can be easily extended to other complex geometric shapes. For more information regarding the F 2 method, refer to [7, 20, 21]. 5.3.1

Definition of a Force Field

A force field is defined as a virtual field of repulsive force in the vicinity of a robot when it travels in a working space. The magnitude and orientation of a force field are determined by and vary with the robot’s status. This virtual repulsive force increases with the decrease of the distance to the robot. Parameters in the F 2 method are listed in Table 5.2. Table 5.2. Parameters used in F 2 Method

Parameters Rr vr vmax Tp θr

We define:

Descriptions radius of a robot absolute value of a robot’s speed maximum absolute value of a robot’s speed a robot’s task priority robot’s orientation in global coordiantion

θ = ∠(θ0 , θr )

(5.17)

Er = vr /(vmax × C)

(5.18)

Dmax =

k × Er × Rr × Tp 1 − Er × cos θ

Dmin = ρ0 × Dmax

(5.19) (5.20)

For any point (x, y) in the 2-D space, assume θr is the robot’s orientation in global coordinate, θ0 is the orientation of this point in global coordinate, thus θ denotes the relative angle of this point to the robot’s orientation (Fig. 5.5). C is a positive number which assigns the environmental influence to the force field with C > 1. Er is a positive decimal fraction with 0 ≤ Er < 1. k is a positive multiplier which determines the coverage area of the force field. Dmax is the maximum active distance of a robot’s force field and Dmin is the distance at which this robot has maximum repulsive force. Dmax shows how far this robot can affect others in its vicinity. Dmin provides a safe distance for the robot to prevent other objects from moving into this area. ρ0 is a positive fractional number with 0 < ρ0 < 1 and heavily influences how close the robot can be separated from obstacles. Tp represents the priority of a task which is undertaken by the robot with Tp ≥ 1. The repulsive force generated by a robot is defined by:

108

D. Wang et al.

Fig. 5.5. Parameters in F 2 method

| Frepulsive robot

⎧ D > Dmax ⎨0 max −D |= P × DDmax −Dmin Dmax ≤ D < Dmin ⎩ Fmax D < Dmin

(5.21)

where D is the shortest distance from point (x, y) to the perimeter of the robot. P is a positive constant scalar which determines the magnitude of the repulsive force. When D changes from Dmin to Dmax , the magnitude of the repulsive force changes from P to 0 gradually. Fmax is the maximum repulsive force which will cause the maximum deceleration on the robot. P and Fmax should be selected based on the robot’s characteristics, with Fmax  P. Equation (5.21) shows that the magnitude of repulsive force varies with distance. To represent the force field in contours, we further define:

ρ = D/Dmax Then (5.21) can be rewritten in an alternative form as: ⎧ ρ >1 ⎨0 1−ρ | Frepulsive robot |= P × 1−ρ0 ρ0 ≤ ρ < 1 ⎩ ρ < ρ0 Fmax

(5.22)

(5.23)

When ρ changes from ρ0 to 1, the magnitude of the repulsive force changes from P to 0. 5.3.2

Attractive Force

When a robot is allocated a particular task, for example, traveling from start point (Xstart ,Ystart ) to goal point (Xgoal ,Ygoal ), a virtual attractive force which attracts the robot from the start point to the goal point is generated. | F f ield |= Q

(5.24)

5 Ranked Pareto Particle Swarm Optimization

109

where Q is a positive constant scalar which determines the magnitude of the attractive force. The attractive force directs to the goal point from the centre of the robot and can be assumed to be a constant. It drives a robot to its destination (Xgoal ,Ygoal ). 5.3.3

Reaction Force

When a robot is moving, its force field will cover some area in its vicinity to protect it from possible collisions. A robot’s force field will cover more area when it travels with a higher speed and shrink when it slow down. When a robot approaches an obstacle or other robots’ force fields, its force field is suppressed and the robot is repelled away by the virtual reaction forces from obstacles or other robots. A reaction force, Frep , can be calculated by: (5.25) | Frep |=| Frepulsive robot | Frepulsive robot is the virtual repulsive force generated by this robot at the interaction point (x, y) in the 2-D space. The reaction force Frep is a vector whose direction is defined to be the normal line of the interaction contour with respect to the interaction point. 5.3.4

V SF 2 Method

The Variable Speed Force Field method (V SF 2 ) is an approach using the concept of the F 2 method for multi-robot motion planning and collision avoidance [7]. Define a fixed reference frame O(X,Y ) and a moving reference frame o(x, y) attached to the robot’s body. Let x, ˙ y˙ be the longitudinal and lateral velocity of the robot in frame o(x, y), the absolute velocities , in the fixed reference frame O(X,Y ) are:        X˙ cos θr − sin θr x˙ x˙ = θ ) (5.26) = R ( r Y˙ y˙ sin θr cos θr y˙ Differentiation with respect the time gives     ax X¨ θ ) = R ( r ay Y¨

(5.27)

where ax , ay are the absolute accelerations expressed in the moving frame o(x, y). For a robot of mass mrobot and rotational inertia I about its centre of mass, equations of motion in the frame are: (5.28) mrobot ax = Fatt x + ∑ Frep x mrobot ay = Fatt y + ∑ Frep y

(5.29)

I ω˙ = Matt + ∑ Mrep

(5.30)

where Fatt x , Fatt y are the components of the virtual attractive force along the longitudinal direction and the lateral direction respectively. Frep x , Frep y are the components of the virtual repulsive forces along the longitudinal direction and the lateral direction

110

D. Wang et al.

Fig. 5.6. Parameters in V SF 2 method

respectively (see Fig. 5.6). ω is the angular velocity of this robot about its centre of mass in frame o(x, y). Matt and Mrep are the moments generated by the virtual attractive force and repulsive force respectively. The forces and moment are calculated as: Matt = r1 × Fatt

(5.31)

Mrep = r2 × Frep

(5.32)

where r1 is the vector from the centre of robot o to the attractive force (Fatt ) action point A. r2 is the vector from the centre of robot to the repulsive force (Frep ) action point B. Please note that we assume the attractive force acts on the robot’s front edge A, which is the interaction of x coordinate and a robot’s body (see Fig. 5.6). It should be noted that a robot needs to satisfy the limitations of translational speed (vr ), translational acceleration (v˙r ), angular speed (ω ) and angular acceleration (ω˙ ). These constraints are: −vmax ≤ vr ≤ vmax

(5.33)

−amax ≤ v˙r ≤ amax

(5.34)

−ωmax ≤ ω ≤ ωmax

(5.35)

−ω˙ max ≤ ω˙ ≤ ω˙ max

(5.36)

These parameters should be chosen on the basis of the robot’s dynamic and kinematic characteristics.

5 Ranked Pareto Particle Swarm Optimization

111

5.4 Using RPPSO for Multiobjective Optimization of F 2 Method Many parameters are used in F 2 method. The setting of these parameters would noticeably affect the method’s performance. In Fig. 5.7, Paths 1 to 5 are simulation results using randomly selected parameters. Please note that for Path 4, collision occurs when the robot travels close to the obstacle, thus the robot fails to achieve its goal in this case. This is a case of unsuccessful parameter selection. Path 6 (solid line) is the optimized result using single objective optimization approach in [8], which is much shorter than those using nonoptimal parameters.

Fig. 5.7. The influence of parameters in V SF 2 method

It can be found that robot runs close around the obstacle in Path 6. Although the path length is reduced significantly, operating too close to obstacles may be undesirable in real applications for safety consideration. Thus an approach which takes into account both efficiency (e.g. path length) and safety (e.g. sufficient safety distance) is needed. RPPSO provides a suitable solution for this multiobjective optimization problem. For all simulations in this section, we assume that all robots are identical and their characteristics are selected based on those of the Amigo robot [22], which has a radius of 0.18 m and a weight of 3.6 kg. The maximum speed is 0.75 m/s. Each robot has an assigned goal and each robot knows its start and goal positions. Robots are equipped with communication devices such that they know the status of other robots, including task priorities, velocities, positions, etc. Task priorities of all robots are set to Tp = 1, i.e., no robot has priority over other robots during collision avoidance. We start from the simple case used in [8], that is, one mobile robot and one static circular obstacle in the working space (Fig. 5.7). This robot is supposed to travel from locations (2, 3) to (8, 7). An obstacle is located at the centre of the work space.

112

D. Wang et al.

Parameters of the F 2 method included in the optimizations are: k, P, Q, C, ρ0 . Based on past experience, we set ranges for parameters as: k ∈ [2, 10], P ∈ [5, 20], Q ∈ [5, 20], C ∈ [1.5, 3], ρ0 ∈ [0.2, 1]. The definition of the ith particle is: xi = [k, P, Q,C, ρ0 ] ,

i = 1, 2, ..., PN

(5.37)

The optimization objectives are set to be: • to minimize the travel length • to keep away from obstacles and other robots as far as possible Two objective functions are chosen based on the above objectives: f1 (xi ) = FF(xi )

(5.38)

where FF(xi ) is the resultant path length obtained from the F 2 method using parameters xi . When a robot reaches its destination, f1 (xi ) will return the value of the real travel distance. If a robot fails to reach its destination using these parameters or collision occurs, f1 (xi )will return a very high value. Assume that for each time cycle, the shortest distance from this robot to obstacles in the working space is denoted by Dnearest . The safety measurement function is defined as: f2 (xi ) = ∑

1 Dnearest,xi

(5.39)

If a robot travels close to obstacles, f2 (xi ) will give a high value. On the contrary, when robot travel far from obstacle, f2 (xi ) will result in a lower value. 5.4.1

Case 1: Single Robot and One Obstacle

In this simulation, a robot is supposed to travel from locations (2, 3) to (8, 7), as shown in Fig. 5.8. The time cycle of simulation is set to 0.1 second. An obstacle is located at the centre of the work space. The optimization results are shown in Fig. 5.9, in which the optimal set is shown in black stars and other particles are shown in gray dots. Table 5.3 shows some optimization results using RPPSO. In Table 5.3, simulation results in [8] are given at the 1st row. Row 2 to 6 are simulation results using RPPSO, denoted by point A, B, C, D, E (particles in the Pareto Front) in Fig. 5.9. Table 5.3. Case 1: optimization results

Test 1 A B C D E

k

P

Q

C

ρ0

Path Danger length(m) index 4.2380 9.5660 9.2680 1.6230 0.2980 7.1060 43.5490 4.5068 10.9423 15.7974 1.1488 0.4605 7.5916 5.2414 4.5273 10.8845 16.3748 1.1476 0.4594 7.6023 5.2162 4.5280 10.8821 16.5471 1.1479 0.4587 7.6091 5.1895 4.5265 10.9539 16.7658 1.1475 0.4574 7.6177 5.1599 4.7715 10.8570 16.3219 1.1497 0.4588 7.7501 5.1409

5 Ranked Pareto Particle Swarm Optimization

113

Fig. 5.8. Case 1: resultant path

Fig. 5.9. Case 1: Pareto optimal set

The number of particles of the RPPSO process in this simulation is 1000. The termination criterion is set to be 100 iterations. In this case, RPPSO reaches a stable state after 60 generations. Figure 5.10 shows the parameter values of particle B (3rd row in Table 5.3) during the iterations.

114

D. Wang et al.

Fig. 5.10. Case 1: evaluation of optimized parameters

Fig. 5.11. Case 2: resultant paths

As shown in Table 5.3, the resultant parameters fall within the same order of magnitudes. The path lengths are close to, but slightly longer than, the case for the single objective optimization approach. The effectiveness of the proposed multiobjective optimization method is explicitly noticeable from significant reductions in the danger index (last column in Table 5.3). In Fig. 5.9 the path of point B is shown in solid line in

5 Ranked Pareto Particle Swarm Optimization

115

Fig. 5.8. Compared with previous result of [8], which is denoted by a dashed line in Fig. 5.8, the new path follows a larger contour around the circular obstacle. By using RPPSO, we can guarantee simultaneously efficiency and safety. 5.4.2

Case 2: Two Robots Navigating in a Corridor

In this simulation robot 1 is supposed to travel from locations S1 (1, 5.25) to G1 (9, 5) and robot 2 travels from S2 (9, 5) to G2 (1, 4.75). The distance between two walls (black patches) is 2 m. Robots start simultaneously from their start points. The time cycle is set to 0.01 second. Robot 1 and 2 use the same set of parameters in each RPPSO iteration. Fig. 5.12 shows the results of RPPSO after 50 generations using 50 particles. Particles of the optimal set (totally 6) are shown in black stars and other particles are shown in gray dots. Details of these optimal particles are given in Table 5.4. The parameters corresponding to the particle indicated by the circle (Fig. 5.12) are listed in the 3rd row

Fig. 5.12. Case 2: Pareto optimal set Table 5.4. Case 2: optimization results

Test 1 2 3 4 5 6

k

P

Q

C

ρ0

Path Danger length(m) index 2.4208 17.1789 19.3420 1.3274 0.4159 15.7011 1417.4227 2.3219 17.0566 19.7220 1.3278 0.4647 15.7066 1394.4961 2.3407 17.8411 19.6860 1.3381 0.3980 15.7589 1376.8007 2.3524 17.8243 19.9060 1.3340 0.3958 15.7755 1376.3172 2.2883 16.7237 19.4253 1.3278 0.3920 15.7789 1370.6500 2.3047 16.2929 20.0047 1.3304 0.4802 15.7951 1369.6322

116

D. Wang et al.

Fig. 5.13. Case 2: distances to obstacles

of Table 5.4. Its resultant paths are shown in Fig. 5.11. The paths of robot 1 and 2 are shown in dashed and solid lines, respectively. In Fig. 5.13, X coordinate is time and Y coordinate shows the nearest distances from a robot to obstacles (including walls and another robot). The distances to obstacles with regard to Robot 1 and 2 are shown in dashed line and solid line, respectively.

5.5 Conclusion In this chapter, we have proposed the Ranked Pareto Particle Swarm Optimization method (RPPSO) for planning collision free paths for multiple mobile robots. In RPPSO, particles are ranked by their qualities with regard to all optimization objectives. Those particles with higher aggregate qualities constitute the Global Best, which stores many current best solutions found by particles. So unlike conventional Particle Swarm Optimization (PSO) method, particles in RPPSO will search toward many directions while taking into account all optimization objectives. We have illustrated that RPPSO is capable of solving efficiently multiobjective optimization problems. To this end, the RPPSO is successfully utilized in the multiobjective parameter optimization problem for the Force Field (F 2 ) method, which is developed for collision avoidance between mobile robots and obstacles. Simulations and experiments are included to demonstrate satisfactory performance in motion planning of multiple robots taking into account not only collision avoidance but also minimum travel lengths.

5 Ranked Pareto Particle Swarm Optimization

117

References 1. Labtombe, J.C.: Robot motion planning. Kluwer Academic Publisher, Boston (1991) 2. Hwang, Y.K., Ahuja, N.: Gross motion planning - a survey. ACM Computing Surveys 24(3), 219–291 (1992) 3. Khatib, O.: Real-time obstacle avoidance for manipulators and mobile robots. International Journal of Robotics Research 1(5), 90–98 (1986) 4. Connolly, C.I.: Harmonic functions and collision probabilities. In: Proceedings of the IEEE International Conference on Robotics and Automation, pp. 3015–3019 (1994) 5. Ge, S.S., Cui, Y.J.: Dynamic motion planning for mobile robots using potential field method. Autonomous Robot 3(13), 207–222 (2002) 6. Ren, J., McIsaac, K.A., Patel, R.V., Peters, T.M.: A potential field approach using generalized sigmoid functions. IEEE Transactions on Systems, Man, and Cybernetics - Part B: Cybernetics 37(2), 477–484 (2007) 7. Wang, D., Liu, D.K., Dissanayake, G.: A variable speed force field method for multirobot collaboration. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2697–2702 (2006) 8. Wang, D., Kwok, N.M., Liu, D.K., Lau, H., Dissanayake, G.: PSO-tuned F2 method for multi-robot navigation. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and System, pp. 3765–3770 (2007) 9. Kwok, N.M., Ngo, V.T., Ha, Q.P.: PSO-based cooperative control of multiple robots in parameters-tuned formations. In: Proceedings of the IEEE Conference on Automation and Science Engineering (IEEE CASE 2007), pp. 332–337 (2007) 10. Schaffer, J.D.: Multiple objective optimization with vector evaluated genetic algorithms. In: Proceedings of the First International Conference on Genetic Algorithms: Genetic Algorithms and their Applications, pp. 93–100 (1985) 11. Fonseca, C.M., Fleming, P.J.: Genetic algorithms for multiobjective optimization: Formulation, discussion and generalization. In: Proceedings of the Fifth International Conference on Genetic Algorithms, pp. 416–423 (1993) 12. Horn, J., Nafpliotis, N., Goldberg, D.E.: A niched Pareto genetic algorithm for multiobjective optimization. In: Proceedings of the 1994 Congress on Evolutionary Computation, vol. 1, pp. 82–87 (1994) 13. Zitzler, E., Thiele, L.: Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach. IEEE Transactions on Evolutionary Computation 3(4), 257– 271 (1999) 14. Zitzler, E., Laumanns, M., Thiele, L.: SPEA2: Improving the strength Pareto evolutionary algorithm, Technical Report 103, Computer Engineering and Networks Laboratory (TIK), Swiss Federal Institute of Technology (ETH), Zurich, Switzerland (2001) 15. Zitzler, E., Deb, K., Thiele, L.: Comparison of multiobjective evolutionary algorithms: empirical results, evolutionary computation 8(1), 173–195 (2000) 16. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of the IEEE International Conference on Neural Networks, vol. 4, pp. 1942–1948 (1995) 17. Doctor, S., Venayagamoorthy, G.K.: Unmanned vehicle navigation using swarm intelligence. In: Proceedings of International Conference on Intelligent Sensing and Information Processing, pp. 249–253 (2004) 18. Chen, X., Li, Y.: Smooth path planning of a mobile robot using stochastic particle swarm optimization. In: Proceedings of the 2006 IEEE International Conference on Mechatronics and Automation, pp. 1722–1727 (2006)

118

D. Wang et al.

19. Kwok, N.M., Ha, Q.P., Fang, G.: Motion coordination for construction vehicles using swarm intelligence. International Journal of Advanced Robotic Systems 4(4), 469–476 (2007) 20. Wang, D.L., Liu, D.K., Wu, X., Tan, K.C.: A force field method for robot navigation. In: Proceedings of the Third International Conference on Computational Intelligence, Robotics and Autonomous Systems (2005) 21. Liu, D.K., Wang, D., Dissanayake, G.: A force field method based multi-robot collaboration. In: Proceedings of the IEEE International Conference on Robotics, Autonomous & Mechatronics, pp. 662–667 (2006) 22. Mobile Robot Bases Specifications, ActivMedia Robotics, http://www.activrobots.com/ROBOTS/specs.html

6 Path Planning Inspired on Emotional Intelligence R. Barber, V. Egido, M. Malfaz, and M.A. Salichs RoboticsLab. Carlos III University, Madrid, Spain [email protected], [email protected], {mmalfaz,salichs}@ing.uc3m.es

Abstract. This chapter is focused on how emotional intelligence is used in order to plan the most suitable path for a robot when fulfilling a certain task. Classic planners use several criteria such as the shortest path, or the minimum time, but those planners do not take into account some inner factors as human beings do in the decision making process. Those factors are for instance, previous experiences that led to failures or successes as well as inner factors that can determine the fact that sometimes people are more conservative or more brave when selecting a path to reach to some place. In this chapter, a general view of robot planning as well as the role of emotions in robots are exposed first. Moreover, an example of an emotion inspired planner applied to a topological navigation system is shown, as well as the results obtained with a real robot.

6.1 Introduction The path planning in robotics involves the knowledge and modeling of the environment as well as certain criteria for selecting the best path. In order to tackle this problem, some basic concepts related to the robot navigation are going to be described. Next, the role of emotions in the decision making process will be stated and later, the topological modeling of the environment will be introduced as the natural way of representing the environment. Finally, a planning algorithm inspired on emotions will be presented as an example. This algorithm is derived from another known algorithm, the Dijkstra algorithm. Making some modifications in the cost function of the Dijkstra algorithm the proposed “Guided Dijkstra” is obtained. This new algorithm tries to imitate one of the most common human behaviour of selecting longer but more familiar routes, minimizing the chance of getting lost. Differing from other similar research, as that one carried out by Roy [17], the chosen path here will depend on the previous navigation actions (past experiences) and on the successful or failed actions that the robot carried out (positive or negative reinforcement). This fact implies that the selected path will not be established a priori.

6.2 The Path Planning Problem As part of everyday life, people navigate from one place to another using their knowledge about the environment. This is a natural process people learn in their childhood D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 119–132. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com


120

R. Barber et al.

[11] and develop as they grow up. But how do people find their ways? Usually most of the computer models do not simulate the behaviour of human wayfinders. Researches in topological navigation in the last years tend to separate two problems that were before considered as one: path planning and wayfinding. The first one groups together problems related with obstacle avoidance until a defined point is reached. The second one is the task sequencing at higher level. This one can lead in a final event that does not necessarily have to be a specific geometric point. The path planning problem in a first approximation, can be divided in two tendencies. The first one, considered as the classic one, assumes the environment’s total geometric knowledge. The second one, based on the sensorial information, considers the lacks that are linked to the real information obtained of what surrounds such as: partial knowledge, sensor failure, obstacles that do not allow total vision, unexpected situation, etc. Within this field, all the classical avoidance techniques are found, as those based in potential fields or the use of maps based on reticular occupation. In relation to the wayfinding problem, planning movements from an initial place to a final one trying to imitate the human behaviour, researchers in the field of spatial knowledge have the same overall vision of the problem by layers. Timpf and Volta [21] establish 3 levels: The planning level, the instruction one and the guidance behaviour one. Based on the navigation model that focuses on the mental or cognitive map, these researches consider that it is through this mental map how the following task is accomplished: the task of obtaining verbal instructions to indicate paths, navigation and environment description. Some other research lines, such as those of Chatila [6], and Firby [9], are found. Both consider the planning in a high level of deliberative type. Chatila considers a more extended planning hierarchy, in which it is established that the planning concept can be divided into path and movement planning, perception planning, navigation planning, manipulation planning, communication planning, and task planning.

6.3 Emotions in Robotics One of the main objectives in robotics and artificial intelligence research is to imitate the human mind and behaviour. For this purpose the studies of psychologists on the working mind and the factors involved in the decision making are used. Several psychologists have realized the importance of the emotions in the intelligent thinking and behaviour. In fact, it has been proved that two highly cognitive actions are dependant not only on rules and laws, but on emotions: Decision making and perception [16]. Moreover, some authors affirm that emotions are generated through cognitive processes. Therefore emotions depend on ones interpretation, i.e. the same situation can produce different emotions on each agent, such as in a football match [15]. Moreover, emotions can be considered as part of a provision for ensuring and satisfaction of the system’s major goals [10]. Emotions play a very important role in human behaviour, communication and social interaction. Emotions also influence cognitive processes, particularly problem solving and decision making [7].

6 Path Planning Inspired on Emotional Intelligence

121

Based on similar assumptions Minsky conclude that, “the question is not whether the intelligent machines can have any emotions, but whether machines can be intelligent without any emotions” [13]. In this chapter some experiments where the planning process of a real robot is affected by an emotional factor will be shown. The selection of a certain path will be determined by the importance given to previous experiences. Depending on the weight of the negatives past experiences the robot will select or not an known path. This weight will be given by the inner state or personality of the robot.

6.4 Topological Representation of the Environment In the past years, some authors [14] [19] have studied the way to represent the environment and the way in which animals and humans move along it. These authors conclude that the natural method to represent the environment is by topological structures and topological relationships. Autonomous mobile robots need good environment models. Several maps representation have been considered in the literature. The map types used are the geometric and the topological ones. The first one uses exact metrical information and it allows us to work with accuracy in nearby robot environments. Its complexity makes us use complex planners, not being useful for big environments. The second one is a world-wide or an environment representation based on environment characteristic and relationships. This information obtained from this type of map is represented by graphs, allowing a big environment simple representation and facilitating fast planning. Topological maps are made up, in most cases, of nodes and edges. An example of this construction are the maps implemented by Kuipers and Byun [12], Wallner, Kaiser, Friedrich and Dillmann [23], Aycard, Laroche and Charpillet [1] and by Tiche and Hgli [20]. Kuipers and Byun built an environment topological map in which geometric information can be added. The topological map consists of nodes related to different places and links related to routes or paths between nearer nodes. Topological navigation has some advantages over the geometric navigation. Considering the geometrical navigation, if some information related to the robots odometry is lost, for example a small step on the floor, the robot can get lost easily. With topological information, the different sensorial perceptions and the different motor actions are the ones which tell the robot where it is. This obviously demands big accuracy to the sensorial systems. Besides this advantage, the topological maps present some other advantages versus the geometric maps. The topological maps make an easy representation of big environments, facilitating a fast planning, but complex sensorial and sensimotors skills are needed.

6.5 Path Planning Algorithm Inspired on Emotions In this section a path planning algorithm inspired on emotions is going to be introduced. First, the representation of the environment that has been used is going to be described. Moreover, the planning is going to be done based on this representation. Next, the planning algorithm is described in detail.

122

6.5.1

R. Barber et al.

Environment Representation: The Navigation Chart

The environment representation used in this paper is the Navigation Chart. The Navigation Chart is not formed by a succession of places of the environment but by a succession of elementary skills [2]. It is represented by a simple directed parameterized graph G(v,e) formed by nodes v and edges e. Nodes are events regarding sensorial perceptions (be in front of a door) and edges represent sensorimotor skills which lead the robot to the next event (move towards a door). A new situation is being described on the Navigation Chart when detecting an event. The robot finds itself in the specific situation or place in which a specific sensorial event is sensed. An example of this Navigation Chart is shown in Figure 6.1.

B finish A

Go straight

Turn on the right

Turn on the right

start

A on the left

Intersection Intersection

B on the right

Fig. 6.1. Example of Navigation Chart

6.5.1.1 Nodes The Navigation Chart nodes have the structure shown in Figure 6.2. Node identification: It is the digit which allows to identify each node. However, several nodes may have assigned the same sensorial event. This digit will be unique due to the fact that the same two events, for example the event left door, could be assigned to different doors. Sensorial event: Depending on the capacities carried out on the robot, a set of detectable events will be obtained. These events are generated, in the present work by

Id:

Node 0

Event:

left door

Sentinel:

(0,1)

Weight:

[-2...3]

Parameters:

d,α...

Fig. 6.2. Node structure in the Navigation Chart

6 Path Planning Inspired on Emotional Intelligence

123

robot skills. These skills, as it will be explained later, are implemented in the automatic level of a two level control architecture. Sentinel: It is a variable with range (0,1) which will show if the node has been previously visited or not. Weight: It is a variable, which allows obtaining an historic of the sensorial events that have already been tested and those that failed. Its value is of great importance to establish new plans considering past experiences. Parameters: The node can contain data and additional information, obtained from the sensorial skill data object that enables the event. 6.5.1.2 Edges Transitions between nodes are performed by robot actions or robot movements such as turn right, follow a corridor, move towards a door. The Navigation Chart transitions have the structure shown in Figure 6.3:

Skill:

Corridor follow

Planning parameters:

40s 10m

Activation parameters:

vmax, dmin...

Sentinel:

(0,1)

Fig. 6.3. Edge structure in the Navigation Chart

Sensorimotor skill: The transitions are identified by a name. The name indicates the sensorimotor skill that allows the robot to go from one event to another. Planning parameters: The planner considers them when trying to establish a plan. An example of this is the estimated distance between points or the time needed to reach the following event. Activation parameters: To activate motor actions, additional information could be required. This information is stored in those parameters. Sentinel: This parameter is different from the one explained in the nodes. It indicates when an error has occurred during the execution of the sensorimotor skill. 6.5.2

Topological Planning Algorithm: Guided Dijkstra

6.5.2.1 Simple Planning with Dijkstra Algorithm As the first approximation, the Dijkstra algorithm is used to carry out the path planning. This algorithm has been used by other authors, as [5], and it is considered as the most adequate algorithm to establish the path that minimizes a parameter in a graph. Therefore, this algorithm is used for searching for a sequence between nodes of the Navigation Chart which will optimized a certain parameter. Thinking on the way human beings decide which way to take, it was considered that one of the factors that

124

R. Barber et al.

Fig. 6.4. Graph G(v, e) and the sequence from v0 to v f

influences the decision making process is the time needed to get from one place to another. Therefore, the cost function that this algorithm tries to minimize is the time required from the source node to the final node. The Navigation Chart can be considered as a guided parameterized graph, G(v, e) made up of edges and nodes, both including information. To plan consists of establishing from an initial node v0 , the sequence of nodes v0 , ..., vi , ...v f and edges e1 , ..., ei+1 ...e f −1 that leads to the objective node v f , see Figure 6.4. The sequence will be translated in commands to perform, and events to detect until the sequence reaches the event v f of the final node. This algorithm is very efficient when establishing paths that minimizes times and distances. If the objective is to have an autonomous robot, the robot must take into account the time restrictions as well as its inner or emotional state, as humans do. Many times one do not choose the shortest path but the one that seems to be more secure. For this reason, a modification of the Dijkstra algorithm is going to be used. This new approach will allow the robot to consider its inner state when planning a certain path. 6.5.2.2 Path Planning with Guided Dijkstra Algorithm Based on previous works carried out by expert authors in algorithms [22] and observing the human being behaviour reflected in similar works dedicated to its study [18], the Dijkstra algorithm was modified to give the robot a behaviour that will take into account human traces. The psychologist R.Zajonc demonstrated in 1980 at Michigan University that human beings prefer stimulus that they have already experienced over new ones. Zajonc in [24] described that there exists a clear tendency towards what is familiar (more pleasant) over the unknown. In the case of a robot, the more the robot visits a certain place, the more secure the robot will be since the stored parameters will be updated. In order to take into account the previous experiences, the guided Dijkstra algorithm proposes the following modification of the cost function: guidedcost[e(i, j) ] = cost[e(i, j) ] − λ · r(v j , e(i, j) )S(v j )

(6.1)

6 Path Planning Inspired on Emotional Intelligence

125

Where: • cost[e(i, j) ], is the time needed to perform the motor action that takes the robot from event vi to another event v j . • r(v j , e(i, j) ) is the reinforcement parameter. If its value is greater than 0 works as a reward and as a penalization if its value is lower than 0. If the navigation between node i to node j was successful then the reinforcement value increases, on the contrary, if that navigation was a failure then the reinforcement decreases. • S(v j ) is the sentinel correspondent to node j. This parameter indicates if this node has been visited previously or not. • λ is the emotional factor, that gives more or less importance to previous experiences. If this factor is high, then the robot will be fearful and tends to select secure paths. The effect of the guided Dijkstra algorithm is the following: if we consider an initial situation in which no previous planning has been done, all the nodes have its sentinels set to 0 and it will choose the shortest path applying the Dijkstra Algorithm without modification. During navigation, the S(v j ) sentinel will take value 1 if is visited, and 0 if it is not visited or a problem while detecting the event has occurred. If the sentinel is 0 the planner will not consider the previous experiences and therefore, the reinforcements of the nodes in future planning. Once the event is detected, the sentinel will take value 1 again, considering previous navigations. The r value is set by: r(v j , e(i, j) ) = w(v j ) ∗ cost[e(i, j) ]

(6.2)

where w(v j ) is the weight vector. Its value begins at 0 and increases or decreases one by one according to the following criterion: if the node or event is detected at the time set by the Navigation Chart, during the navigation, the node is then considered as visited, its sentinel is set to 1 and w(v j ) is increased in one unit. On the other hand, if the event does not occur, w(v j ) is decreased in one unit and its sentinel is set to 0.

6.6 How the Guided Dijkstra Algorithm Works Next, an example showing how the previously described algorithm works is presented. The implementation has been done in a control robot architecture called AD, that runs on a B12 platform of i-Robot. 6.6.1

Planning in the AD Control Architecture

The control architecture used in this work is the AD architecture [3]. The architecture is made up of two levels: a deliberative level, which contains the reflexive processes and the automatic level, where the processes with sensor and motor capacities are found. Both levels present the same characteristic: they are made up of skills. Skills consist of the different capacities of reasoning or carrying out an action. These skills return data and events to the skills or sequencers which have activated them. In the higher level, see Figure 6.5, the skills related to the capacities of reasoning and decision-taking, can be found. The planner, described in this paper, and the navigator are examples of deliberative skills. The planner uses the Navigation Chart and it

R. Barber et al.

PRINCIPAL SEQUENCER

DELIBERATIVE LEVEL

Long Term Memory

Planner

Navigation Navigator Chart with modifications Sequencer

Events

To the Automatic Level

Short Term Memory

New plan needed event failure Data Flow Performance orders flow

New plan for the navigator Events Flow

Fig. 6.5. Deliberative level in the AD architecture

From the Deliberative Level

AUTOMATIC LEVEL

126

Corridor Travelling

Virtual Laser

Laser

Short term Memory

Cross Door

Virtual Sonar

Sonar

Data Flow Performance orders flow

Contour Following

Virtual Odometry

Odometry

Door Detection

Identify Location

Virtual Base

Base Events Flow

Fig. 6.6. Automatic Level in the AD architecture

6 Path Planning Inspired on Emotional Intelligence

127

establishes the plan as a subgraph of that representation. The navigation skill takes this subgraph and converts it in the sequence of events to be detected (nodes) and sensorimotor actions (edges). Once the process is sequenced, this skill is the one that modifies the Navigation Chart, storing the parameter changes as sentinels and nodes weights. In the automatic level there are skills which interact with sensors and actuators [4]. Examples of automatic skills used in the present work are: door detection, corridor traveling, door crossing, contour following and identifying location, see Figure 6.6. These skills will be associated with the nodes and edges of the Navigation Chart. The architecture considers the exchange of events and data through a Long Term Memory and a Short Term Memory. The Long Term Memory is exclusively used for the deliberative level skills, containing information which needs to be stored for long periods of time. The Long Term Memory contains, in this case, the environment representation as a Navigation Chart. On the other hand, the Short Term Memory stores temporary data and is used for both the deliberative and automatic level. The Short Term Memory contains, in this work, the generated plan for the planning skill. 6.6.2

Deliberative Planner Skill

The planner has been implemented as a deliberative skill in the AD control architecture. This skill has been considered one of the deliberative skills because it has to take decisions about the more suitable path according to some criteria. Its mission is to generate the sequence of nodes and edges which will be used by the Navigator skill to take the robot from an initial event v0 to a final event v f , considering the parameters from previous planning experiences. The basic structure of the Planner deliberative skill is shown in Figure 6.7. The value of the emotional factor λ is determined a priori as part of the personality of the robot. The skill is made up of an active object and a data object [4]. The active object receives as activation parameters the initial and final nodes (v0 ,v f ) which show the first and last planning events and as data input the Navigation Chart from the LongTerm Memory. Its output will be the events which will get to the sequencer from the

Fig. 6.7. The planning deliberative skill

128

R. Barber et al.

Deliberative Level and the plan extracted from the Navigation Chart that will be stored in the data object, that is, in the Short-Term Memory, for further use by the navigation deliberative skill. The navigator will execute the plan and will carry out the plan supervision, considering the possibility of changing weights and sentinels of the Navigation Chart that is found in the Long-Term Memory. 6.6.3

An Example of the Emotional Inspired Planning

To test the planner, an indoor environment formed by corridors and rooms has been considered. Before the planning experiments were carried out, the robot generated autonomously its own model of the environment. In this case, this model was the Navigation Chart. The process of autonomous generation of this Navigation Chart by the robot is explained in [8]. The environment layout during the planning experiments is shown in Figure 6.8 where doors which belong to rooms C8, C4, C12, C13 and the one which communicates C12 with C13 are opened. In this environment several plans are carried

Fig. 6.8. Work environment L.D.Det.

17

Corr.F.

. 24 L. D. Det .

R.D.Det.

c8

C. D o or

I.Ub. ht ig

C.F.

T. Le ft

R T. .

C.

21

Do or

oo r

or

oor

T. Rig ht .

Co rr. F

.

C. D oo r

C. D

Ub. Corr.F.

27

20 22

.

10

I.

I.Ub.

.F

14

15

26

C

.F.

r oo

.

13

C.F.

32 b. I. U C.F.

25

t igh

I.Ub.

T. R

F C.

ht

rr Co

.D C

r. oo 19

18

.

I.Ub.

R.D. Det.

c12

Rig

Corr.F.

L.D.Det.

Corr.F.

23

.D C

T.

7 C.F. I.Ub. 33

F C.

or Do 6

12

11

5

C. 8

or

C.F.

t Lef

ft Le T. 2

4

I.Ub.

t

End of Corridor

R.

r. oo i gh T. r

3

0

Corr.F.

16

Do C.

.D C

C. Pta

R.D.Det.

L.D.Det.

Corr.F.

9

C. Do

Corr.F.

1

C. D

R.D.Det.

c4

Corr.F. Corridor Following. T. Righ. Turn Right T.Left. Turn Left. I. Ub Identify Ubication C.F Contour Following C. Door. Cross Door R.D.Det. Right Door Detected. L.D.Det. Left Door Detected.

Fig. 6.9. Plan obtained from the Navigation Chart

30 29 . C I.Ub.

Corr.F .

c13

28 or Do

31 End of corridor

C.Door

6 Path Planning Inspired on Emotional Intelligence

129

End of corridor – Room C13 Planning

Corridor

1 0 -1

0

2

4

6

8

10

12

Room C13

-2 -3 -4

Room C12

Fig. 6.10. Path followed by the robot during the navigation, the shortest path

Fig. 6.11. Different navigation processes carried out successfully by the robot

out, showing how the planning criteria is modified as a consequence of the weight and sentinel changes during the navigation. At the beginning the robot is commanded to reach, from its current position (end of corridor), the event that identifies room C13. Since this navigation mission is the first one carried out in this environment, sentinels set to zero, the robot chooses the nodes and edges sequence that takes less travel time. Those sequences have been drawn in a thicker line in Figure 6.9, over part of the Navigation Chart. The representation of the path followed by the robot is obtained through its odometry, for this case, as it is shown in Figure 6.10. Afterwards several navigation processes that implied corridor traveling were carried out. This situation made that the nodes associated to events placed in the corridor were reinforced, compared with those that were unknown zones. The generated plans of these missions and the accomplished navigation processes are shown in Figure 6.11.

R. Barber et al.

10

17

R T. .

.

21 I.Ub. ht ig

C. D oor

22

Corr.F.

24 L.D. Det.

R.D.Det.

c8

27

C.F.

30 I.Ub.

28 r 29 . Doo C

31 End of corridor

T. L

C. Do or

Corr.F.

Co rr. F.

26

I.Ub.

20

C. Do or

or

Rig ht .

C. D o

C. D oor

14 I.Ub.

25

F. C.

. .F

I.Ub.

ht Rig

19

C

. rr. F Co

r

T.

15

32 b. I. U C.F.

oo .D C

r. oo .D C

C.F.

Corr.F.

23

13

R.D. Det .

c12

18

. .F

Corr.F.

L.D.Det.

ht Rig

7 C.F. I.Ub. 33

12 I.Ub.

C

r oo .D 6

T.

8

11

5

C

t ef

eft

C.F.

or Do

ht L T.

L R.

ig T. r 2

4

I.Ub.

Corr.F.

16

C.

r. oo .D C 3

0 End of Corridor

C. Pta

R.D.Det.

L.D.Det.

Corr.F.

9

C. Do or

Corr.F.

1

eft

L.D.Det.

R.D.Det.

T.

130

c4

Corr.F .

C.Door

c13

Corr.F. Corridor Following. T. Righ. Turn Right T.Left. Turn Left. I. Ub Identify Ubication C.F Contour Following C. Door. Cross Door R.D.Det. Right Door Detected. L.D.Det. Left Door Detected.

Fig. 6.12. Plan obtained when the robot is fearful and the corridor is a secure path

Fig. 6.13. Path followed by the robot when it is fearful and the corridor is a secure path

Next, a new planning to reach to room C13 again is done. In this occasion, the emotional factor λ is high, this is, the robot is fearful. Now, the robot chooses the path that includes the corridor in order to fulfil the previous task since the corridor has been reinforced and marked as a secure path. The plan is shown in Figure 6.12 and The path followed by the robot is shown in Figure 6.13. Next, a small value is given to the emotional factor and the robot, since is “brave”, selects again the first path, the shortest one, see Figure 6.9 and Figure 6.10. This is because the robot does not consider the previous navigation, so decides to select the path with the smallest time cost instead of the most secure one. Later, the corridor is blocked by us and the robot tries to navigate this area, see Figure6.14. Therefore, during this navigation the value of the reinforcement of the nodes of the corridor decreases. After this experiment, the robot with a high emotional factor, the robot is fearful, tries to reach again the room C13. As a consequence of the negative reinforcements of the nodes of the corridor, the robot selects again the shortest path, see Figure 6.9 and Figure 6.10, since the corridor is marked as an unsecured path.

6 Path Planning Inspired on Emotional Intelligence

131

Fig. 6.14. Path followed by the robot when the corridor is blocked

6.7 Conclusions The role of the emotional factor, as in human beings, is useful as well as necessary for the decision making process. In the case of path planning for mobile robots, this emotional factor is presented as an alternative to the classic navigators. The described planner uses information from previous navigation. Two criteria have been established for the plan generation: one that only minimizes travel time (Dijkstra) and a modification in the cost function that considers a human behaviour, minimizing the chance of getting lost by taking longer but known routes (Guided Dijkstra). As a consequence of this, when planning, the previous experiences make the robot choose a known zone instead of a shortest but unknown path. The robot navigation becomes then safer.

References 1. Aycard, O., Laroche, P., Charpillet, F.: Mobile robot localization in dynamic environments using places recognition. In: Proceedings of the 1998 IEEE International Conference on Robotics and Automation (1998) 2. Barber, R., Salichs, M.A.: A new human based architecture for intelligent autonomous robots. In: The Fourth IFAC Symposium on Intelligent Autonomous Vehicles, pp. 85–90 (2001) 3. Barber, R., Salichs, M.A.: Mobile robot navigation based on event maps. In: Field and Service Robotics, pp. 61–66 (2001) 4. Boada, M.J.L., Barber, R., Salichs, M.A.: Visual approach skill for a mobile robot using learning and fusion of simple skills. Robotics and Autonomous Systems 38, 157–170 (2002)

132

R. Barber et al.

5. Cara, A., Taylorb, G., Brunsdonc, C.: An analysis of the performance of a hierarchical wayfinding computational model using synthetic graphs. Computers, Environment and Urban Systems 25, 69–88 (2001) 6. Alami Chatila, R., Prayoux, R.: An architecture integrating task planning and reactive execution control. In: Workshop on Architectures for Intelligent Control Systems, IEEE Nice, France (1992) 7. Damasio, A.: Descartes’ Error - Emotion, reason and human brain. Picador, London (1994) 8. Egido, V., Barber, R., Boada, M.J.L., Salichs, M.A.: Self-generation by a mobile robot of topological maps of corridors. In: IEEE International Conference on Robotics and Automation, Washington, USA, pp. 2662–2667 (May 2002) 9. Firby, J.R.: Task networks for controlling continuous processes. In: Hammond, K. (ed.) In sensorial International Conference on AI Planning Systems, Menlo Park, California, pp. 49– 54. AAAI Press, Menlo Park (1994) 10. Frijda, N., Swagerman, J.: Can computers feel? theory and design of an emotional model. Cognition and Emotion 1(3), 235–357 (1987) 11. Inhelder, B., Piaget, J.: The child’s conception of space. Norton, New York (1967) 12. Kuipers, Y.T., y Byun, B.J.: A qualitative approach to robot exploration and map-learning. spatial reasoning and multi-sensor fusion. In: Prooceedings of the 1987 Workshop, Morgan Kaufmann, USA (1987) 13. Minsky, M.: The Society of Mind. Simon and Schuster (1986) 14. Nehmzow, U.: Animal and robot navigation. Robotics and Autonomous Systems 15, 71–81 (1995) 15. Ortony, A., Clore, G.L., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press, Cambridge (1988) 16. Picard, R.W.: Los ordenadores emocionales. Ed. Ariel S.A (1998) 17. Roy, N., Burgard, W., Fox, D., Thrun, S.: Coastal navigation – robot motion with uncertainty. In: Proceedings IEEE Conf. Robotics and Automation (ICRA) (1999) 18. Ruddle, R.A., Payne, S.J., Jones, D.M.: Navigating buildings in desk-top virtual environments: experimental investigations using extended navigational experience. J. Exp. Psychol., 143–159 (1997) 19. Thruiller, O., Meyer, J.A.: Biomimetic navigation models and strategies in animats. AI Communications 10, 79–82 (1997) 20. Ti´eche, F., H¨ugli, H.: From topological knowledge to geometrical map. In: Proceedings of the 3rd IFAC Symposium on Intelligent Autonomous Vehicles (IAV 1998) (1998) 21. Volta, F.S., Pollock, D.W., Timpf, S., Egenhofer, M.J.: A Conceptual Model of Wayfinding Using Multiple Levels of Abstraction. In: Theories and Methods of Spatio-Temporal Reasoning in Geographic Space, pp. 348–367 (1992) 22. Voudouris, C., Tsang, E.: Guided local search and its application to the travelling salesman problem. European Journal of Operational Research 113, 469–499 (1999) 23. Wallner, F., Kaiser, M., Freidrich, H., Dillman, R.: Integration of topological and geometrical planning in a learning mobile robot. In: IROS 1994. Proceedings of the IEEE/RSJ/GI International Conference on Intelligent Robots and Systems. Advanced Robotic Systems and the Real World (1994) 24. Zajonc, R.: Feeling and thinking: Preferences need no inferences. American Psychologist 35 (1980)

7 An Exploration of Online Parallel Learning in Heterogeneous Multi-robot Swarms Jim Pugh and Alcherio Martinoli ´ Distributed Intelligent systems and Algorithms Laboratory, Ecole Polytechnique F´ed´erale de Lausanne, 1015 Lausanne, Switzerland {jim.pugh,alcherio.martinoli}@epfl.ch

Abstract. Designing effective behavioral controllers for mobile robots can be difficult and tedious; this process can be circumvented by using unsupervised learning techniques which allow robots to evolve their own controllers online in an automated fashion. In multi-robot systems, robots learning in parallel can share information to dramatically increase the evolutionary rate. However, manufacturing variations in robotic sensors may result in perceptual differences between robots, which could impact the learning process. In this chapter, we explore how varying sensor offsets and scaling factors affects parallel swarm-robotic learning of obstacle avoidance behavior using both Genetic Algorithms and Particle Swarm Optimization. We also observe the diversity of robotic controllers throughout the learning process using two different metrics in an attempt to better understand the evolutionary process.

7.1 Introduction Designing even simple behaviors for robots that are efficient and robust can be very difficult for humans; it is often not hard to implement a rudimentary controller that accomplishes the task, but achieving near-optimal performance can be very challenging, especially for miniature robotic platforms with severe hardware and computational limitations. Unsupervised robotic learning allows for automated design of efficient, robust controllers, which saves much design time and effort. Unsupervised learning is also useful for allowing robots to adapt to situations where the task/environment is unknown beforehand or is constantly changing. Genetic Algorithms (GAs) are a very common method of accomplishing machine learning and optimization. Candidate solutions to a problem are modeled as members of a population, and breeding (selection and crossover) and mutation are applied to “parents” (high performing solutions) in the population to generate “children” (new candidate solutions). GA can be used to shape an Artificial Neural Network (ANN) controller by using the parameter set as the weights, and the evaluative function as a measure of the performance of a desired robot behavior. Particle Swarm Optimization (PSO) is a promising new optimization technique which models a set of potential problem solutions as a swarm of particles moving about in a virtual search space. The method was inspired by the movement of flocking birds and their interactions with their neighbors in the group. PSO achieves optimization using three primary principles: evaluation, where quantitative fitness can be determined D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 133–151. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

134

J. Pugh and A. Martinoli

for some particle location; comparison, where the best performer out of multiple particles can be selected; and imitation, where the qualities of better particles are mimicked by others. The algorithm can also be used to evolve ANN robotic controllers. In robotic learning, in order to evaluate a candidate controller solution, a robot must run the controller for some period of time (typically from several seconds to several minutes) and observe how well it performs over that duration according to a preestablished metric (fitness function). The computational and temporal resources needed for this evaluation are drastically higher than those needed for the processing of the learning algorithm itself in almost all cases (less than 1% for the case study presented here). Therefore, robotic learning can be considered to be a very expensive optimization problem, and speed-up efforts should be focused on decreasing the number/length of evaluations rather than the internal workings of the learning technique. Both GA and PSO use groups of interacting virtual agents in order to achieve their optimization. In distributed robotics, groups of robots interact to accomplish their goals. It is therefore possible to implement these algorithms in a parallel distributed fashion for learning in multi-robot systems. Each robot is responsible for several virtual agents, which it evaluates at each iteration. After each set of evaluations, the robots communicate to share the fitness information needed to progress to the next iteration of the algorithm. By running the algorithms in this fashion, we need no external supervisor to oversee the learning process, and the speed of learning is significantly improved, as many robots evaluating in parallel increase the rate of candidate solution evaluation and therefore decrease the total learning time. On real robots, sensors and actuators may have slightly different performances due to variations in manufacturing. As a result, multiple robots of the same model may actually perceive and interact with their shared environment differently, creating a physically heterogeneous swarm. While robotic controller evaluations are inherently stochastic due to partial, noisy environmental perception by individual robots, this heterogeneity may add a systematic bias between agents. This is a crucial difference; partial perception and noise can be overcome by information exchange between robots (i.e. shared evaluation measures of a controller between multiple robots will give a better estimate of its true performance), while sharing information between robots with different systematic biases may actually hinder the learning process, since biased results from other robots could cause a robot to incorrectly adapt its behavior. The result may be increased difficulty in evolving effective behavioral controllers. In order to model swarm heterogeneity, simulations of swarm-robotic learning can include fixed random variations in, for example, the sensitivity and offsets of on-board sensors. This should reflect a more realistic learning scenario, particularly for miniature robots where low-cost sensors often exhibit higher levels of manufacturer variation (often on the order of 10% or more). In this chapter, we explore the efficacy of parallel robotic learning using GA or PSO on swarms of heterogeneous robots with sensor variations. We observe the diversity of the GA and PSO populations in different instances to attempt to gain insight into the evolutionary process. Section 7.2 provides some background on GA, PSO, unsupervised robotic learning, and multi-robot learning. Section 7.3 details our robotic learning case study and gives results in the case of a homogenous robot swarm. Section 7.4 explores the impact of sensor offsets on the learning process, while section 7.5 studies

7 An Exploration of Online Parallel Learning

135

the impact of different sensor scaling factors. In section 7.6, we analyze the diversity of the GA and PSO populations throughout the learning process in order to determine the cause of differences between the algorithms with and without sensor variations. In section 7.7, we discuss the implications of our results, and section 7.8 concludes and provides outlook on future work.

7.2 Background Genetic algorithms were originally developed in the 1960s by John Holland. The algorithms are inspired by evolution, where the fittest members of a population tend to reproduce more often than the less fit members. Candidate solutions are modeled as a population of “chromosomes”. At each iteration of the algorithm, a new population is generated from the previous one. Selection of the parents of the new generation is implemented using one or more of several schemes, such as elitism (using only the top performing members of the population), Roulette Wheel sampling (stochastically choosing parents with weight proportional to performance), and rank selection (ranking chromosomes from best to worst and stochastically choosing parents with weight proportional to their rank). After parents have been chosen, crossover between the parents can occur with some probability (each chromosome is split into multiple pieces, and children use some parts from one parent and some parts from the other). This allows positive aspects from different chromosomes to be merged into a single chromosome. Last, mutation is applied, where each element of the chromosome may have its value randomly changed with some probability. This provides a random local search, which allows solutions to continue to improve beyond the genetic diversity that was available in the original population ([8], [17]). The original PSO method was developed by James Kennedy and Russel Eberhart [5], [10]. Every particle in the population begins with a randomized position (xi, j ) and randomized velocity (vi, j ) in the n-dimensional search space, where i represents the particle index and j represents the dimension in the search space. Candidate solutions are optimized by flying the particles through the virtual space, with attraction to positions in the space that yielded the best results. Each particle remembers the position at which it achieved its highest performance (x∗i, j ). Each particle is also a member of some neighborhood of particles, and remembers which particle achieved the best overall position in that neighborhood (given by the index i ). This neighborhood can either be a subset of the particles (local neighborhood), or all the particles (global neighborhood). For local neighborhoods, the standard method is to set neighbors in a pre-defined way (such as using particles with the closest array indices modulo the size of the population as neighbors, henceforth known as a “ring topology”) regardless of the particles’ positions in the search space. The equations executed by PSO at each step of the algorithm are vi, j = w · vi, j + pw · rand() · (x∗i, j − xi, j ) + nw · rand() · (x∗i, j − xi, j ) xi, j = xi, j + vi, j

(7.1) (7.2)

where w is the inertia coefficient which slows velocity over time, pw is the weight given to the attraction to the previous best location of the current particle and nw is the weight

136

J. Pugh and A. Martinoli

given to the attraction to the previous best location of the particle neighborhood. rand() is a uniformly-distributed random number in [0, 1]. PSO has been shown to perform as well as or better than GA in several instances. Eberhart and Kennedy found PSO performs on par with GA on the Schaffer f6 function [5, 10]. In work by Kennedy and Spears [11], a version of PSO outperforms GA in a factorial time-series experiment. Fourie showed that PSO appears to outperform GA in optimizing several standard size and shape design problems [7]. Unsupervised learning describes learning scenarios where there is no external entity which decides upon the training set inputs for the learning agent(s). Rather, inputs are generated dynamically as the agents interact with their environment. This is as opposed to supervised learning, where the inputs are generated/collected first and then used repeatedly. In supervised learning, the accuracy of the system at each iteration is usually decided by an external “teacher” evaluating the system output. The pre-defined inputs are split into two separate sets, one for training the system and the other for testing the performance. Supervised learning tends to be easier than unsupervised, as the data does not change between iterations of the algorithm and can be preselected to avoid using unusual or particularly noisy data points. However, supervised learning is not possible in situations where the input data to the system depends on the current state of the learning agent; this is the case for online robotic learning, since the robot’s movements affect what its sensors will perceive. Evolutionary algorithms have been used extensively for unsupervised learning of robotic behavior. A good survey of the work is given in [14] and more recently in [21]. More specifically, standard GA has been shown to be effective in evolving simple robotic controllers [6], and modified noise-resistant versions of both GA and PSO were shown to achieve very good performance on simulated unsupervised robotic learning, outperforming the standard versions of the algorithms [22]. Multi-robot learning has been used and explored in various ways; a survey of work (including learning on other multi-agent systems) can be found in [26]. Matari´c studied mechanisms to encourage individual agents in a group to act in ways to help the group performance [13]. Multi-robot learning using several methods in a wide variety of scenarios has been explored [2], [25]. Specialization in multi-agent systems using reinforcement learning was studied in [19]. Techniques for increasing individual learning speed via multi-robot learning were studied in [9] and [15]. A modified version of a genetic algorithm has been embedded onto a 2-robot system to allow for distributed parallel learning [20]. Pugh and Martinoli found that both GA and PSO could be used for effective distributed parallel multi-robot learning in [23] and applied to heterogeneous robot groups in [24].

7.3 Parallel Swarm-Robotic Learning Case Study: Obstacle Avoidance Following on the work presented in [23] and [24], we use the case study of obstacle avoidance for our swarm-robotic learning task.

7 An Exploration of Online Parallel Learning

Fig. 7.1. Flowcharts for noise-resistant GA (left) and PSO (right)

137

138

7.3.1

J. Pugh and A. Martinoli

Experimental Setup

For the learning techniques, we use the noise-resistant GA and PSO algorithms from [23]. GA uses elitism to select the best half of the population as the parent set, and then applies Roulette Wheel sampling to replenish the missing chromosomes. PSO uses a local neighborhood in a ring topology with one neighbor on each side. At every iteration, these algorithms reevaluate their previous best locations and parent sets for PSO and GA, respectively, combining the new fitness value with previous ones to get a more accurate measure of the actual fitness. Although this requires twice as many fitness evaluations at each iteration as their standard counterparts, this technique prevents noisy fitness evaluations from severely disrupting the learning process and gives much better results given the same number of evaluations of candidate solutions. Flowcharts of both algorithms are shown in Fig. 7.1. For both PSO and GA, initial population member elements are randomly generated in the range [−20, 20] but allowed to change to any value during evolution. Velocity in PSO is also randomly initialized in the range [−20, 20] but prevented from ever going outside this range. The parameters for the algorithms are given in Table 7.1. Table 7.1. GA and PSO Parameters for Unsupervised Learning GA

PSO

Population Size

20

Population Size

20

Mutation Probability

0.15

pw

2.0

Crossover Probability

0.2

nw

2.0

Mutation Range

[-20.0, 20.0]

w

0.6

We use Webots, a realistic simulator, for our robotic simulations [16], using 20 epuck1 robots [4] (this is as opposed to the Khepera robot [18] used in some previous experiments). The robot(s) operate in a 2.0 m x 2.0 m square arena with no additional obstacles (see Fig. 7.2 left). The robotic controller is a single-layer discrete-time artificial neural network of two neurons, one for each wheel speed, with sigmoidal output functions. The inputs are the eight infrared proximity sensors, as well as a recursive connection from the previous output of the neuron, lateral inhibitions and bias values (see Fig. 7.2 right), giving us 22 weights total. Sensors have a maximum range of 5.0 cm, and sensor output varies linearly from 0 at maximum range to 1 at minimum range (0.0 cm) with 3% noise. Slip noise of 10% is applied to the wheel speed. The time step for neural updates is 128 ms. We use the fitness function used originally in [6] and again in [23]. The fitness function is given by: √ (7.3) F = V · (1 − Δ v) · (1 − i) 0 ≤ V ≤ 1, 1

http://www.e-puck.org

0 ≤ Δ v ≤ 1,

0≤i≤1

(7.4)

7 An Exploration of Online Parallel Learning

139

Fig. 7.2. Left: Robot arena with e-puck robots. Right: Depiction of the artificial neural network used for the robot controller. Curved arrows are recurrent connections and lateral inhibitions.

where V is the average absolute wheel speed of both wheels, Δ v is the average of the difference between the wheel speeds, and i is the average activation value of the most active proximity sensor over the evaluation period. These factors reward robots that move quickly, turn as little as possible, and spend little time near obstacles, respectively. The terms are normalized to give a maximum fitness of 1. The evaluation period of the fitness tests for these experiments is 240 steps, or approximately 30 seconds. Between each fitness test, the position and bearing of the robots are randomly set by the simulator to ensure the randomness of the next evaluation. During evolution, each of the 20 robots is responsible for a single member of the GA/PSO population (i.e. they must evaluate that member at each iteration and communicate the resulting fitness measure). With 100 iterations of the learning algorithms, this results in a total simulated learning time of approximately 1 hour 40 minutes (100 iterations comprised of 2 evaluations each lasting 30 seconds). 7.3.2

Results

The average performance over 100 runs of the best evolved controllers in the population can be seen in Fig. 7.3 left. As previously found, PSO is able to achieve superior performance, as GA occasionally produces poorer solutions. The progress of evolution can be seen in Fig. 7.3 right. While GA improves faster in the first few iterations, it quickly changes to a gradually increasing plateau, while PSO continues improving throughout the entire process. Two primary types of obstacle avoidance controllers were observed in successful evolutionary runs of both GA and PSO. In the absence of obstacles, both types caused the robot to move ahead at full speed. The first controller consistently made the robot perform either a right turn or a left turn at the activation of any of its front sensors, thereby avoiding the obstacle in its path. The second controller used the recurrent neural connection to achieve a multi-state behavior; the robot would run forward until the

140

J. Pugh and A. Martinoli 1

Fitness

0.8 0.6 0.4 0.2 0

GA

PSO

0.7

Fitness

0.6

GA PSO

0.5 0.4 0.3 0.2 0

20

40 60 Iteration

80

100

Fig. 7.3. Left: Fitness of best evolved controllers in population after evolution averaged over 100 runs for GA and PSO. Error bars represent standard error. Right: Average fitness of population members throughout evolution averaged over 100 runs for GA and PSO.

front sensors detected an obstacle, at which point it would reverse its wheel speed and run backwards at full speed. When the rear sensors were activated, the robot would again resume forward movement, resulting in a back-and-forth behavior which somewhat resembles bouncing. Depending upon the environment, either of these two types of behavior might be preferable to the other.

7.4 Heterogeneity from Sensor Offsets One potential variation between sensors is different offset values (i.e. if some sensor A detects value x, some other sensor B would detect x + a, where a is the offset value). We can incorporate this variation in our simulation using sensor values of: vi = vi + ai

(7.5)

where vi is the original sensor value, vi is the offset sensor value used by the robot controller, and ai is a Gaussian offset value randomly generated at the start of the simulation with mean of 0 and standard deviation of s.

7 An Exploration of Online Parallel Learning

7.4.1

141

Experimental Setup

We repeat the experiments from section 7.3 using sensor offsets with standard deviation of 0, 0.05, 0.1, 0.2, and 0.5 (corresponding to 0%, 5%, 10%, 20%, and 50% of the maximum sensor value, respectively). In order to accurately evaluate the performance of the controller, the original sensor values are used for the proximity term of the fitness calculation. The same parameters are used for both GA and PSO. 7.4.2

Results

The average performance over 100 runs of the best evolved controllers in the population for different levels of sensor offset variation can be seen in Fig. 7.4 left. While the performance is not highly impacted for a standard deviation of up to 0.1, a significant decrease can be seen at 0.2, and a standard deviation of 0.5 yields quite poor performance. This suggests that neither GA nor PSO are affected by minor variations, but major variations could cause some sensors to be always/never perceived as active, which would cause very poor performance. The same trend can be seen in both GA and PSO, though it appears as though GA performs relatively better with large offsets. 0.8 GA PSO

Fitness

0.6

0.4

0.2

0

0

0.05

0.1

0.2

0.5

Standard Deviation of Sensor Offset

0.8

GA PSO

Fitness

0.6

0.4

0.2

0

0 0.05 0.1 0.2 0.5 Standard Deviation of Sensor Offset

Fig. 7.4. Fitness of best evolved controllers in population after evolution averaged over 100 runs for GA and PSO with different standard deviations (s) of sensor offset. Results shown for final evaluation with both heterogeneous (left) and homogenous (right) robots. Error bars represent standard error.

142

J. Pugh and A. Martinoli

0.5

Fitness

0.4 0.3 s = 0.0 s = 0.05 s = 0.1 s = 0.2 s = 0.5

0.2 0.1 0 0

20

40 60 Iterations

80

100

0.6

Fitness

0.5 0.4 0.3 s = 0.0 s = 0.05 s = 0.1 s = 0.2 s = 0.5

0.2 0.1 0 0

20

40 60 Iterations

80

100

Fig. 7.5. Average fitness of population members throughout evolution averaged over 100 runs for GA (left) and PSO (right) with different standard deviations (s) of sensor offset

While the previous results indicate how well evolved controllers perform on their heterogeneous robot group, we can evaluate the general performance of the controllers evolved on a heterogeneous swarm by testing them on a homogenous set of robots without sensor variations (see Fig. 7.4 right). We observe that while performance still decreases as sensor offsets increase, the effect is noticeably less. This indicates that the previously observed performance drop is due to a combination of suboptimal controllers from impaired learning and degraded performance from operating on a heterogeneous swarm. The relative performance of GA versus PSO is even higher for large sensor offsets, suggesting that learning is less impacted when using GA than when using PSO. The performance of GA throughout learning with different offset levels can be seen in Fig. 7.5 left. We notice that the trend of rapid initial improvement followed by a gradually increasing plateau is not affected, but rather performance seems to be scaled down as the standard deviation of the offset grows. A similar trend can be observed for the PSO results in Fig. 7.5 right.

7.5 Heterogeneity from Sensor Scaling Another variation that can occur between sensors is sensitivity. One sensor may be more sensitive to input, which causes its output to vary more quickly than others. We can model this in our simulation using the equation:

7 An Exploration of Online Parallel Learning

vi = mi vi

143

(7.6)

where vi is the original sensor value, vi is the offset sensor value used by the robot controller, and mi is a Gaussian scaling factor randomly generated at the start of the simulation with mean of 1 and standard deviation of s. 7.5.1

Experimental Setup

We repeat the experiments from section 7.3 using sensor scaling factors with standard deviation of 0, 0.05, 0.1, 0.2, and 0.5 (and mean of 1). In order to accurately evaluate the performance of the controller, the original sensor values are used for the proximity term of the fitness calculation. The same parameters are used for both GA and PSO. 7.5.2

Results

The average performance over 100 runs of the best evolved controllers in the population for different levels of sensor scaling variation can be seen in Fig. 7.6 left. The performance here is impacted less than for the case of sensor offsets; good performance is maintained even for a scaling factor standard deviation of 0.5, though we do see

0.8

GA PSO

Fitness

0.6 0.4 0.2 0 0 0.05 0.1 0.2 0.5 Standard Deviation of Sensor Scaling Factor 0.8

GA PSO

Fitness

0.6 0.4 0.2 0

0 0.05 0.1 0.2 0.5 Standard Deviation of Sensor Scaling Factor

Fig. 7.6. Fitness of best evolved controllers in population after evolution averaged over 100 runs for GA and PSO with different standard deviations (s) of sensor scaling factors. Results shown for final evaluation with both heterogeneous (left) and homogenous (right) robots. Error bars represent standard error.

144

J. Pugh and A. Martinoli

0.55 0.5

Fitness

0.45 0.4

s = 0.0 s = 0.05 s = 0.1 s = 0.2 s = 0.5

0.35 0.3 0.25 0.2 0

20

40 60 Iterations

80

100

0.6

Fitness

0.5 0.4

s = 0.0 s = 0.05 s = 0.1 s = 0.2 s = 0.5

0.3 0.2 0.1 0 0

20

40 60 Iterations

80

100

Fig. 7.7. Average fitness of population members throughout evolution averaged over 100 runs for GA (left) and PSO (right) with different standard deviations (s) of sensor scaling factors

an observable decrease here. The cause of this may be that, although the sensitivity is changed, sensors still react is the same way (i.e. higher for close proximity, lower for far proximity), and therefore the performance of the controller is not significantly impacted. We again see the same trend in both GA and PSO, and GA again performs relatively better with large scaling variations. Observing the performance on a homogenous set of robots (see Fig. 7.6 right), we see that the actual quality of the evolved controller has not been at all affected by sensor scaling variation in almost all cases. The one minor exception is for controllers evolved using GA with a scaling factor standard deviation of 0.5, where we see a very slight performance decrease. Following the theory suggested above, it could be that because sensors respond in a similar way in spite of scaling, the same obstacle avoidance approaches are selected by the learning algorithms, which results in unaffected performance when using a homogenous set of robots. The performance of GA throughout learning with different scaling levels can be seen in Fig. 7.7 left. As a result of the better performance, there is not such an obvious decrease as the standard deviation of the scaling grows, with the exception of a standard deviation of 0.5 where there is a marked performance drop. There is only very minor differences between the curves for PSO for different sensor scaling variation levels (see Fig. 7.7 right).

7 An Exploration of Online Parallel Learning

145

7.6 Diversity in Evolving Robot Swarms Although PSO was able to achieve superior performance to GA in evolving robotic controllers in [22] and [23], the reasons for this were never apparent. A major drawback to using stochastic multi-agent algorithms such as GA and PSO is that they lack transparency about what exactly is happening during the evolutionary process and why they achieve the results that they do. In an attempt to shed some light on this issue, we analyze the diversity of the GA and PSO populations through their evolution using two different metrics. 7.6.1

Euclidean Diversity

For our first diversity metric, we choose one of the most simple: Euclidean distance. We consider each candidate solution as a location in a virtual search space. The pairwise diversity between two members of a population is therefore given by:  (7.7) d(a, b) = ∑(ai − bi )2 i

where d(a, b) is the pairwise diversity between member a and member b, and xi is element i of member x. The diversity of the entire population is given by the average pairwise diversity of all pairs of members in the population, or:   1 (7.8) Deu = ∑ d(a, b) N(N − 1) ∑ a b=a where a and b are members of the population, and N is the total size of the population. 7.6.2

Entropic Diversity

For our second diversity metric, we make use of a diversity measure dubbed the “hierarchic social entropy”, first proposed in [3] and later used in [12]. For this metric, population members are divided into clusters, where clusters are defined as maximal sets of “connected” population members. Two population members are considered directly connected if the measure of some distance metric between them (Euclidean distance in this case) is less than some threshold value h, and two population members are considered connected if there exists a set of direct connections linking them. Each cluster i can then be given a “probability” pi which is calculated as follows: pi =

ni N

(7.9)

where ni is the number of members in cluster i and N is the total population size. Because clusters are maximal, an individual population member will appear in only a single cluster, and therefore:

∑ pi = 1 i

(7.10)

146

J. Pugh and A. Martinoli

We can therefore calculate the entropy of these probabilities as: H = − ∑ pi log pi

(7.11)

i

resulting in an entropy of zero for a single cluster and increasing entropy as the number of clusters increases. To calculate the hierarchic social entropy (or Entropic diversity), we integrate entropy over all values of the threshold distance h:  ∞

Den =

H(h)dh

(7.12)

0

where H(h) is the entropy of cluster probabilities as a function of the threshold distance h. While both Euclidean and Entropic diversities provide a measure of the distance between population members (high distance implies more clusters for the same value of h which yields higher entropy), only Entropic diversities directly reflects whether or not there exist compact groups of members in the virtual search space (which would suggest similar behaviors among these members). We measure diversity using both metrics throughout evolution for parallel learning using GA and PSO for homogenous robotic swarms and for heterogeneous robotic swarms with sensor offsets. We keep the same scenario and parameters as used previously. In order to make valid comparison, we normalize diversity measures to the diversity of the population at the first iteration (i.e. diversity always begins with value1). 7.6.3

Results

The progress of population diversity throughout evolution for GA and PSO on a homogenous swarm can be seen in Fig. 7.8. For both metrics, the diversity in GA drops quickly in the initial iterations and maintains a stable lower value for the rest of the evolution. In contrast, the diversity of PSO actually increases initially and maintains a higher value. This is evidence for the hypothesis given in [23] that by very quickly converging to a good candidate solution, GA sacrifices the ability to further explore for better possible solutions, which causes it to occasionally converge to a mediocre final solution. On the other hand, by maintaining high diversity throughout evolution, PSO is able to continuously discover new and better solutions and continue improving throughout the entire evolution. This also demonstrates why PSO is better able to maintain good performance with a smaller population size, while GA does very poorly with small populations since it requires enough genetic diversity to already include a good solution. Comparing Euclidean versus Entropic diversity, we see that while the trends are the same, the diversity increase in PSO is consistently less than for Entropic diversity (the difference between diversity measures, Deu − Den , can be seen in Fig. 7.9). This suggests that while PSO particles do indeed spread out in the virtual search space, they simultaneously form clusters, with some particles maintaining close proximity to some others. This follows as a result of the local neighborhood topology used by PSO, as neighboring particles should often assume similar values and therefore have a small Euclidean distance between them. For GA, the initial diversity decrease is greater for Entropic diversity than Euclidean, indicating that individuals are simultaneously converging and forming clusters. However, the difference in diversity plateaus and actually

7 An Exploration of Online Parallel Learning

147

Euclidean Diversity

1.5

1

GA PSO

0.5

0 0

20

40 60 Iterations

80

100

80

100

Entropic Diversity

1.5

1

GA PSO

0.5

0 0

20

40 60 Iterations

Fig. 7.8. Diversity of population members throughout evolution averaged over 100 runs for GA and PSO for a homogenous robot swarm for both Euclidean (left) and Entropic (right) diversity measures

Diversity Difference

0.4

0.3 GA PSO

0.2

0.1

0 0

20

40 60 Iterations

80

100

Fig. 7.9. Euclidean Diversity minus Entropic Diversity of population members throughout evolution averaged over 100 runs for GA and PSO for a homogenous robot swarm

148

J. Pugh and A. Martinoli

s = 0.0 s = 0.05 s = 0.1 s = 0.2 s = 0.5

Euclidean Diversity

1 0.9 0.8 0.7 0.6 0

20

40 60 Iterations

100

s = 0.0 s = 0.05 s = 0.1 s = 0.2 s = 0.5

1 Entropic Diversity

80

0.9 0.8 0.7

0

20

40 60 Iterations

80

100

Fig. 7.10. Diversity of population members throughout evolution averaged over 100 runs for GA with different standard deviations (s) of sensor offset for both Euclidean (left) and Entropic (right) diversity measures

decreases slightly in the latter portion of evolution, suggesting that all further diversity decreases are the result of convergence rather than clustering. We can also compare the progression of diversity for evolution of heterogeneous robotic swarms with sensor offsets using GA (see Fig. 7.10) and PSO (see Fig. 7.11). Both diversity metrics yield very similar results here. In the case of GA, sensor offset variation seems to have little impact on diversity, with much overlap between diversity measures for different standard deviations throughout the learning process. For PSO, there is no significant change in diversity for low sensor offset variations; for higher variations, the diversity grows at the same pace initially but achieves higher values towards the end of the evolution. This may be caused by less useful information sharing between robots due to major sensorial differences, promoting individual reliance. This might prevent convergence of particles, which would increase the diversity in the latter evolutionary stages.

7.7 Discussion For parallel swarm-robotic learning of obstacle avoidance behavior, offset variation seemed to have a much higher impact on performance than scaling variation. However, it is highly likely that this result is specific to our case study. As mentioned previously,

7 An Exploration of Online Parallel Learning

149

Euclidean Diversity

1.6 1.5 1.4 s = 0.0 s = 0.05 s = 0.1 s = 0.2 s = 0.5

1.3 1.2 1.1 1 0.9 0

Entropic Diversity

1.6 1.5 1.4

20

40 60 Iterations

80

100

40 60 Iterations

80

100

s = 0.0 s = 0.05 s = 0.1 s = 0.2 s = 0.5

1.3 1.2 1.1 1 0.9 0

20

Fig. 7.11. Diversity of population members throughout evolution averaged over 100 runs for PSO with different standard deviations (s) of sensor offset for both Euclidean (left) and Entropic (right) diversity measures

because of the linear manner in which proximity sensors are used (i.e. higher - there is an obstacle, lower - there is no obstacle), scaling does not have much effect, while offset variation could potentially confuse the sensor as to its current state. One can envision other scenarios where this is not the case: if a sensor has some intermediate optimum value (such as would be the case in wall-following or flocking behavior), both offset and scaling would disrupt its reading; comparing subsequent sensor readings would remove any offset bias but still be affected by scaling. Therefore, parallel learning in heterogeneous robotic swarms needs to be tested on many other case studies before any such generalizations can be made. Although not shown here, we also performed several parallel swarm-robotic learning tests using PSO without any velocity limit (w was kept at 0.6). While the performance of the resulting controllers was not significantly different, the diversity of the population exploded, growing exponentially until the end of the evolution (it was not uncommon to have a final Euclidean diversity on the order of 100,000). The plateau previously observed is therefore only caused by a hard velocity bound. We believe the reason for this explosive behavior is the same as the reason for the low impact of variations in sensor scaling; ANNs continue to function nearly the same if every weight is multiplied by a constant factor. The only change will be the sensitivity of the sigmoid output function, which will approach a step function as the neural weights increase.

150

J. Pugh and A. Martinoli

7.8 Conclusion and Outlook We have explored some of the effects of robot heterogeneity on parallel swarm-robotic learning. In the case of evolving obstacle avoidance, both Genetic Algorithms and Particle Swarm Optimization were able to withstand small variations in sensor offsets and large variations in sensor scaling factors, while showing poor performance with high offset variations. By observing population diversity throughout evolution, we discovered that PSO maintains much higher diversity, which could explain its superior ability to GA in avoiding local optima. Though not explored here, there could be ways in which heterogeneous robotic swarms could identify differences between robots by sharing and comparing the observed fitness values of common controllers. This information could be used to help avoid the fitness degradation previously observed and might even be useful for deliberately pursuing specialization within the swarm in scenarios not limited to evolving one single controller. We have shown that tracking population diversity can yield some insight into the progress of the evolving swarm. One could therefore imagine that these learning algorithms could be improved if they were to track their own diversity. A simple example would be allowing GA to scale the values of some of its population members if the diversity were too low. While calculating the diversity of the entire population requires a global manager, it might be feasible for individual robots to acquire a good estimation using local sensing of nearby teammates, which could allow this change to be implemented in a distributed fashion.

Acknowledgements This work has been mainly sponsored by a Swiss NSF grant (contract Nr. PP002116913).

References 1. Antonsson, E.K., Zhang, Y., Martinoli, A.: Evolving Engineering Design Trade-Offs. In: Proc. of the ASME Fifteenth Int. Conf. on Design Theory and Methodology, Chicago, IL, USA, paper No. DETC2003/DTM–48676 (September 2003) 2. Balch, T.: Behavioral diversity in learning robot teams. PhD Thesis, College of Computing, Georgia Institute of Technology (1998) 3. Balch, T.: Measuring robot group diversity. In: Balch, T., Parker, L.E. (eds.) Robot teams: From diversity to polymorphism, pp. 93–135 (2002) 4. Cianci, C., Raemy, X., Pugh, J., Martinoli, A.: Communication in a swarm of miniature robots: The e-puck as an educational tool for swarm robotics. In: S¸ahin, E., Spears, W.M., Winfield, A.F.T. (eds.) SAB 2006 Ws 2007. LNCS, vol. 4433, pp. 103–115. Springer, Heidelberg (2007) 5. Eberhart, R., Kennedy, J.: A new optimizer using particle swarm theory. In: Proc. of the Sixth Int. Symposium on Micro Machine and Human Science, MHS 1995, October 4–6, pp. 39–43 (1995)

7 An Exploration of Online Parallel Learning

151

6. Floreano, D., Mondada, F.: Evolution of Homing Navigation in a Real Mobile Robot. IEEE Transactions on Systems, Man and Cybernetics, Part B 26(3), 396–407 (1996) 7. Fourie, P.C., Groenwold, A.A.: The particle swarm optimization algorithm in size and shape optimization. Struct. Multidisc. Optim. 23, 259–267 (2002) 8. Goldberg, D.E.: Genetic Algorithms in Search, Optimization & Machine Learning. AddisonWesley, Reading (1989) 9. Kelly, I.D., Keating, D.A.: Faster learning of control parameters through sharing experiences of autonomous mobile robots. Int. Journal of System Science 29(7), 783–793 (1998) 10. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proc. of the IEEE Int. Conf. on Neural Networks, vol. 4, pp. 1942–1948 (November/December 1995) 11. Kennedy, J., Spears, W.M.: Matching algorithms to problems: an experimental test of the particle swarm and some genetic algorithms on the multimodal problem generator. In: Proc. of IEEE Int. Conf. on Evolutionary Computation, Anchorage, pp. 78–83 (May 1998) 12. Li, L., Martinoli, A., Abu-Mostafa, Y.: Learning and Measuring Specialization in Collaborative Swarm Systems. Adaptive Behavior 12(3-4), 199–212 (2004) 13. Matari´c, M.J.: Learning to Behave Socially. In: Proc. of the 3rd Int. Conf. on Simulation and Adaptive Behaviors - From animals to animats, vol. 3, pp. 453–462 (1994) 14. Matari´c, M.J., Cliff, D.: Challenges in evolving controllers for physical robots. Robot. and Autonomous Sys. 19(1), 67–83 (1996) 15. Matari´c, M.J.: Learning in behavior-based multi-robot systems: Policies, models, and other agents. Special Issue on Multi-disciplinary studies of multi-agent learning, Ron Sun (ed.) Cognitive Systems Research 2(1), 81–93 (2001) 16. Michel, O.: Webots: Professional Mobile Robot Simulation. Int. J. of Advanced Robotic Systems 1, 39–42 (2004) 17. Mitchell, M.: An Introduction to Genetic Algorithms. MIT Press, Cambridge (1996) 18. Mondada, F., Franzi, E., Ienne, P.: Mobile robot miniaturisation: A tool for investigation in control algorithms. In: Proc. of the Third Int. Symp. on Experimental Robotics, Kyoto, Japan, pp. 501–513 (October 1993) 19. Murciano, A., Mill´an, J.R., Zamora, J.: Specialization in multi-agent systems through learning. Behavioral Cybernetics 76, 375–382 (1997) 20. Nehmzow, U.: Learning in multi-robot scenarios through physically embedded genetic algorithms. In: Proc. of the 7th Int. Conf. on the Simulation of Adaptive Behavior: From animals to animats, pp. 391–392 (2002) 21. Nolfi, S., Floreano, D.: Evolutionary Robotics: The Biology, Intelligence, and Technology. MIT Press, Cambridge (2000) 22. Pugh, J., Zhang, Y., Martinoli, A.: Particle swarm optimization for unsupervised robotic learning. In: IEEE Swarm Intelligence Symposium, Pasadena, CA, pp. 92–99 (June 2005) 23. Pugh, J., Martinoli, A.: Multi-Robot Learning with Particle Swarm Optimization. In: Int. Conf. on Autonomous Agents and Multiagent Systems, Hakodate, Japan, May 8–12, pp. 441–448 (2006) 24. Pugh, J., Martinoli, A.: Parallel Learning in Heterogeneous Multi-Robot Swarms. In: IEEE Congress on Evolutionary Computation, Singapore, September 25-28, pp. 3839–3846 (2007) 25. Stone, P.: Layered Learning in Multi-Agent Systems. PhD Thesis, School of Computer Science, Carnegie Mellon University (1998) 26. Stone, P., Veloso, M.: Multiagent Systems: A Survey from a Machine Learning Perspective. Autonomous Robots 8(3), 345–383 (2000)

8 Robot Control in Dynamic Environments Using Memory-Based Learning Patrick M. McDowell1 , Brian S. Bourgeois2, and Frederick E. Petry2 1 2

Southeastern Louisiana University, Computer Science Department, Hammond, LA 70402 Naval Research Laboratory, Stennis Space Center, MS, USA 39529

Abstract. Learning systems used on robots typically require a-priori knowledge in the form of environmental models or trial-and-error approaches requiring a robot to physically execute multitudes of trial solutions. Neither approach is very suitable for dynamic unstructured environments in which a robot is sent to explore and gather information prior to human entry. This chapter presents a new approach, ‘memory-based learning’, in which a robot is provided with an initial baseline behavior whose performance is linked with a metric explicitly defined by a function whose arguments are sensory inputs and resulting robot actions. A neural network, using sensor inputs and action outputs, has been chosen as the basic controller building block for behaviors as it is very amendable to rapid in-situ generation and adaptation. The use of a neural network controller for a maneuvering task using electronic ‘ears’ for sensory input is demonstrated in this work, and genetic algorithms are shown to be an effective method for rapidly developing the network weights and transfer functions. The concept of memory based learning is introduced and a construct for representing coupled sense/action information is presented and demonstrated using both simple reactive and memory based controllers, whose performance is demonstrated through simulation studies and on land robots.

8.1 Introduction For a robot to work autonomously in a complex, changing environment it must be able to adapt dynamically. The work discussed in this paper develops an approach to this complex problem using a sensor input based metric for success and recently collected memories - a temporal series of sensor/action pairs. In essence this scheme monitors the progress of a robot while it carries out a particular task. If the robot is not achieving its goal based on the metric for success, then a learning process will be triggered that uses recently collected sensor/action pairs to train a new controller. The learning is achieved with a neural network controller utilizing a genetic algorithm to develop both the network weights and node transfer functions. This approach is applied to a formation maneuvering problem [3,28] where robots with ‘ears’ listen for a leader robot and attempt to follow. For this application the sensor input is the intensity of the sound in the left and right ear and the action is either turn left, go straight or turn right; the metric of success is to keep the intensity within a certain range and equal in both earscorresponding to the situation where the robot is successfully following the leader. Conventional robotic control follows basically a functional approach [14,18]. A key aspect is the use of a world model which can be updated by sensor inputs. To develop D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 153–170. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

154

P.M. McDowell, B.S. Bourgeois, and F.E. Petry

a set of actions to achieve the desired goals the functional approach utilizes a planning module. The planner uses the world model and current sensor inputs to produce the steps toward a goal that constitute the plan of actions. This approach is very brittle especially in its reliance on a pre-developed world model and is very limited in controlling multiple robots. Learning approaches such as those described in this chapter can avoid the drawbacks of a functional approach. In this chapter we first provide some background on Remote Operated Vehicles (ROVs) and Untethered Unmanned Underwater Vehicles (UUVs) by describing their potential applications and issues involved in their operation. The following section discuses the paradigm of recent memories to generate an adequate training set and collecting memories needed to train an initially ‘blank’ controller. Next we describe the neural network controller which is trained using genetic algorithms. In the following sections we explain the design and training of reactive and memory based controllers using recent memories. Results from simulation tests of these controllers are shown in the experimental results section.

8.2 ROV/UUV Background Remote Operated Vehicles (ROVs) are used extensively for salvage operations, ocean floor surveying and numerous inspection activities that support a wide range of underwater commercial activities. In deep water (greater than 1000 ft) an ROV is the platform of choice because of the depth and endurance limitations for human divers. The key disadvantage to an ROV is the requirement for the long tether. Untethered Unmanned Underwater Vehicles (UUVs) have demonstrated the ability to perform deep-water surveys faster and cheaper than towed vessels [9,10]. UUVs have the potential for supplementing and even replacing ROVs for many deep-water operations because of the cost and problems associated with the tether. One promising scenario for the near future is to use a ROV to control several UUVs in a local work area. Table 1 compares key factors of tethered vessels to single and multiple UUVs. A tethered vessel clearly has the advantage of power and endurance, but this advantage can be lost when multiple UUVs are deployed from the same host ship. The factors of deck gear size, launch and recovery time, maneuverability and coverage rate are distinct disadvantages of tethered vessels and directly impact their cost-effectiveness. Accurate positioning of deep-water vessels remains a difficult and expensive issue. At present this can be accomplished with ultra-short baseline systems on GPS equipped ships that remain directly over the deployed vessels [9]. A tether offers a distinct advantage in the areas of communications and intelligence since it provides high-bandwidth instantaneous communications and allows human intelligence to be in the loop for vehicle operation. The communications, intelligence and multi-vessel navigation capabilities required for deploying multiple UUVs are still in active development and of particular interest for our research approach. Figure 8.1 illustrates a UUV task force that could be used to conduct an oceanographic survey [5]. Data of interest in such an operation would include bathymetry,

8 Robot Control in Dynamic Environments Using Memory-Based Learning

155

Table 8.1. Comparison of UUVs and Tethered Vessels for Deep Water Work

Comparison Factors ROV or TOW Single UUV Multiple UUVs Power/endurance + + Deck gear size + + Launch/recovery time + + Maneuverability + + Coverage Rate + ++ Positioning Communications bandwidth + Intelligence + -

Fig. 8.1. A UUV task force concept for a notional oceanographic mission. The blue lines indicate range/bearing positioning links and the red links are range only positioning links. The black lines indicate communication links.

acoustic imagery, wave characterization, sediment type, water salinity and temperature, currents, bioluminescence, water clarity, etc. An important aspect to note is that a survey of this sort could cover large areas (many square miles) and need to be executed in a fairly short period (a few days). Due to the characteristics of the vessels and the sensors involved, high vessel speeds cannot typically be used. Consequently, a large number of vessels are required to accomplish the mission as is shown in the figure. As with most operations employing a large number of assets, it is not economically feasible or desirable for all vessels to have capabilities of the same type and grade. Bathymetry and acoustic imagery for example, require a vessel that can cruise smoothly at fairly low speeds. Sound velocity profiles (salinity and temperature) need a vessel that

156

P.M. McDowell, B.S. Bourgeois, and F.E. Petry

is good at performing vertical excursions. Obstacle sensors are not necessarily required for every vessel; a maximum safe depth can be set based on a priori knowledge of the operation area and a few vessels equipped with look-ahead sonar can be deployed ahead of the task force to ensure safe navigation for all vessels. Close inspection of an object of interest would require a vessel with a hover capability and a high-resolution imaging system. This will in general require an active coordination of a robot team [4]. A command link to the outside world is required for status monitoring and for task force redirection. Instead of equipping every vessel with this capability, this can be served by a vessel that provides a radio/acoustic communications interface to the task force’s Command Vessel (CV) as shown in Figure 8.1. Communications for status monitoring and asset redirection is also required at the task force level, so a communication capability is required between the CV and the rest of the task force. Asset redirection can typically be handled using low-bandwidth communications and the CV could be equipped with a low frequency long-range broadcast acoustic communications system for this task [6,20,26]. Military and commercial applications that would use multiple UUVs share the common problem of positioning and communications. While a commercial application typically has greater flexibility and could put a fixed infrastructure into place or use more ships, these approaches are not always economically desirable. Here a team concept can be used with lower cost UUVs that are co-coordinated to the higher capability UUV leader with an advanced positioning capability. This allows a group travel formation to and from the mission site where the individual UUVs will then carry out their mission specific goals [16,24].

8.3 Memory Based Learning Relevant related work for this topic includes Q-learning, anytime learning and valuedependent learning. Q-learning [17,25,27] is an algorithm that learns optimal control strategies through the exploration of its environment. The main strength of Q-learning is that it allows a robot or agent to operate in its environment without a priori knowledge. While it has been applied successfully to specific game playing tasks, it is best suited for well defined problems with finite state spaces and not for dynamic environments since it must exhaustively test alternatives. Anytime learning [13] allows a robot to learn without requiring it to physically execute many trial solutions. However, anytime learning’s functionality is derived through extensive simulation that requires significant processing power and a priori knowledge of the environment in which the robot will work. The research effort by Rucci et al. [21] developed ’value-dependent learning’ that uses a sensor based approach similar to this research, but requires significant physical repetition by the robot to converge on a solution. The approach described in this chapter is intended to control autonomous robots in unstructured dynamic environments, such as the undersea environment or in a desert. Environments such as these motivate this approach, which focuses on creating a method for the robot to explore its environment in a non-goal-oriented manner so it can learn what actions lead to success. The environmental exploration routines are intended to provide the robot with memories from which it can learn to achieve its goals, using the

8 Robot Control in Dynamic Environments Using Memory-Based Learning

157

reactive and sensor history driven (memory based) learning methods, without having to physically repeat each movement until it reaches a criterion for success. Learning from recent memories is used so that the robot or agent can avoid regimes of directed testing of trial solutions. The approach records a memory consisting of sensor data and associated action values and learns from this which actions helped to optimize goals. By learning from previous experiences the robot can create a “draft” controller that can be iteratively improved as the robot operates in its environment in fulfillment of its goals. Of primary importance in this strategy is for an agent to be able to identify where in the recent memory the robot’s goals were being realized. So the memory must contain sufficient usable examples to learn all the functions needed to realize its goals. If this information is not available, or cannot be generated from what is available, the robot will learn an incomplete strategy. Thus, for this method to be successful, some assumptions are necessary: 1. A goal must be specifiable in terms of sensor values. 2. The system must be able to observe its progress in order to detect when goals are not being met. 3. The data in the recent memory that the learning system is utilizing must contain sufficient information to form a complete strategy for the situation at hand. 4. The learning system must be able to generate a new controller before the environment changes enough to make it irrelevant. As discussed, the formation following goal can be specified in terms of sensor values in such a way that progress to the goal can be observed. Several approaches have been used to develop a baseline controller (non-adaptive) that would collect an adequate memory for training; the most successful was a random but purposeful algorithm that utilized slightly modified goal criteria to direct a robot during a memory collection process. This controller uses sensor feedback to continue actions that move it closer to its goal state, where the goal is for example an increasing sound intensity. As long as the feedback indicates that the metric is being met it continues the current action, but as soon as this is no longer valid, it randomly chooses another action. Since the selection of the new action is random, examples of correct actions are distributed relatively evenly. That is, there will be about an equal number of examples in which a right turn was the correct action as there will be for going straight or left. The structure of the process for a following robot is described next. Since we specifically have an underwater application context, the sensors that are being utilized are some sort of microphones. Acoustic sensing is typically the only feasible choice for such an environment, but the following discussion is not specific to this. The variables used in the subsequent discussions are ML(t) - Left Microphone (Sensor) value MR(t) - Right Microphone (Sensor) value A(t) - the current action of the robot: left, straight, right Now first, for the following robot, if both sensor values are increasing in intensity and close to each other, the robot will go to a neutral state, that is, it goes straight (A(t)= straight). We can express this case by the condition:

158

P.M. McDowell, B.S. Bourgeois, and F.E. Petry

CSraight = [(ML(t) > ML(t − 1)) ∧ (MR(t) > MR(t − 1)) ∧ (Little ∗ f ≥ Big)] (8.1) The first two parts of the condition represent the gains in intensity. The last term captures the concept of closeness of the sensor values. The values of Big and Little are the larger and smaller of the sensor values respectively and f is a user specifiable parameter (e.g. 1.05). CNochange = [(ML(t) > ML(t − 1)) ∨ (MR(t) > MR(t − 1))]

(8.2)

Lastly if neither of these cases apply then the approach will be to attempt a different action. The new action A(t) is a random choice from the set of possible actions {left, straight, right}.

Fig. 8.2. An example path generated by the random but purposeful controller. The sound source is at the center of the plot and the smiley faces with antenna denote the robot and its orientation as it follows the path.

Figure 8.2 shows a graphic example of the robot trajectory taken using the random but purposeful training controller. It is evident that this controller will generate a generous set of both good (moving towards the source) and bad (moving away from the source) examples for training a new controller. As noted earlier, a sufficient memory is required to generate a properly functioning controller. In the context of this application we could envision a situation in which the good examples predominately consist of left turns; as a result, the generated neural network controller may not know how to turn right to move towards the source. To contend with this situation, and due to the symmetry in this particular application, a mirroring technique was created that generated ‘extrapolated’ memories to complete the training set. Essentially this means that for every example where a left turn was executed to get closer to the source, a right turn

8 Robot Control in Dynamic Environments Using Memory-Based Learning

159

example was manufactured from this memory and vice versa. Also the positive straight example set takes advantage of the strategy, providing more variety in that example set. A high level description of the algorithm is illustrated below, where a positive example is one in which an action results in the situation in which sensor intensity increases the most: For each positive example Swap right and left sensor values if (action = left) then New action ← right; if (action = right) then New action ← left else New action ← straight In our approach the methods of collecting the memories and using that information to create controllers is addressed. Another equally important consideration is the method in which the memories and controllers developed from them are maintained. For instance the goal may remain constant but the environment may change rapidly, necessitating the use of a previously generated controller, or consider the case where the goal changes but the environment remains unchanged. The storage and management of these behaviors and memories is a robot architectural issue and is not addressed in this paper.

8.4 Genetic Algorithm Training of Neural Network Controllers The controllers used in this research are based on feed forward neural networks (FFNNs) with one hidden layer, since their structure readily lends itself to rapid generation on an autonomous robot. We utilize a genetic algorithm (GA) to train the neural network in which the GA is used to find the neural network’s transfer functions along with the weights. A GA is used to train the network for two main reasons: GAs provide an efficient and robust means of training neural networks[2], while avoiding many of the problems with becoming trapped in local minima as experienced with gradient techniques[23]. Since the best transfer function for a particular environment may not be known ahead of time, having it be GA selectable decreases the user’s need for a priori knowledge. The GA was selected over more traditional methods such as Back Propagation so that the need for explicit desired values for each training example could be avoided. In our current experiments involving formation maneuvering, this data could be acquired from a simulator, but for other applications the correct motions at each moment are often not known. In these instances a GA can use a fitness function that promotes overall system performance by rewarding controllers that best fulfill a programmed goal. There are a variety of issues that are involved in the effective and efficient execution of a genetic algorithm. The selection of suitable representation is a crucial component of GA design. Often the particular fitness function which carries out the evaluation of individuals in a population can be the most costly aspect of a GA’s execution. Also other

160

P.M. McDowell, B.S. Bourgeois, and F.E. Petry

parameters such as selection, crossover and mutation schemes must often be adjusted for efficient execution [8]. Although the inclusion of network transfer functions as part of the optimization algorithm has not been widely reported, there is at least one paper describing the use of a GA to optimize the number and placement of radial basis nodes in a modular neural network [15]. That research differs from our effort in that the GA worked with node placement and topology, while the weights were found with more conventional back propagation techniques. In our approach we use an Elite genetic algorithm to train the neural networks. For an Elite GA, the breeding portion of the population (best scoring individuals) is carried through to the next generation with the remainder of the population being derived using individuals from the breeding portion. A high level outline of the Elite GA algorithm used to find the weights and transfer functions of the feed forward neural network controller is described next. The population P = {pi } for the GA consists of n individuals (organisms) pi that represent feed forward neural networks. Each pi = (w,t) where w represents the weight and t the transfer functions of the FFNN respectively: w = (w1 , · · · wq )

− 1.0 ≤ w j ≤ 1.0

where q = [(number of inputs * number of hidden nodes)+(number of hidden nodes * number of outputs)] t = (t1 , · · ·tr ) r =number of hidden nodes where t j = 0 for discrete transfer function: t j = 1 for linear transfer function; t j = 2 for sigmoid transfer function. The fitness function of the GA is Fit(pi ) which measures the cumulative distance between the follower robot and a lead robot. The follower robot is controlled by the neural network described by individual pi , while the simulation system controls the lead robot. We use a notation topPercent(P) to represent the elite percent of the population P that is directly transferred to the next generation’s population. This percentage is a user adjustable parameter with a typical range of 10 to 33 percent. The Elite GA can be outlined as follows: Generate an initial population P0 by randomly initializing for each pi ∈ P0 the values of w and t. Run the simulation for M generations Evaluate the population P0 using the simulation in the fitness function Fit(pi) Order the evaluated individuals pi in P0 from best to worst P1 ← topPercent (P0) Use crossover and mutation on the topPercent portion of P1 to provide individuals for the remaining portion of the new generation’s population P1 P0 ← P1 Return the individual pi from P0 with the highest fitness value after M generations The benefits of using this method are its stable operation and straightforward implementation. This method was originally developed for generating feed-forward neural

8 Robot Control in Dynamic Environments Using Memory-Based Learning

161

networks for use in a formation maneuvering simulator [16]. In a typical simulation session, the user will use this algorithm to grow a controller. Usually the GA will build a working controller within 200 generations. For this research it was integrated into an online learning process that operates on our laboratory robots. In this capacity it trains neural networks based on training sets generated by reactive and memory based training processes. Unlike the brute force methods used in the formation maneuvering simulator, these methods take advantage of positive and negative example sets developed using the goal focused exploration described earlier. The next sections will discuss how the memory captured by the focused exploration is processed so that system can avoid excessive use of a priori knowledge and still use the GA to generate controllers.

8.5 Reactive Controller For a reactive controller, the actions taken at time t are based solely upon the current sensor readings. Consequently, the reactive learning algorithm uses a training set comprised strictly of individual sensor-action values that are marked as good if the action results in an increased sensor value and bad for a decreased value. The recent memory is examined for occurrences of these examples, where the sensor values at time t + 1 are examined to determine if the action at time t was a good or bad example. One known drawback of this type of controller is that it cannot react to trends that occur over time. A classic example for the formation maneuvering task is the forward-backward ambiguity: a following robot with a reactive controller cannot discern if it is heading directly away or towards the leader if the amplitude in both sensors remains the same; without an additional observation to detect this trend a robot could wander away from its leader in the wrong direction. Figure 8.3 and the steps below illustrate the process for creating the reactive controller using recent memories. A robot ‘trainee’ records a memory M consisting of sensor values and actions taken while seeking either a fixed or moving sound source. The robot could be guided by an operator or an algorithm such as the classic logic method, another neural network, or the random but purposeful controller. The sensor values and actions at a given time, denoted SA[t], is a 3-tuple (ML, MR, A) where ML / MR are the left / right microphone values respectively and A the action taken. A recent memory M will be defined as a sequence of sensor action values SA[0] · · ·SA[n − 1]. Now we can organize training sets of positive examples, P, and negative examples, N, from M by using the changes in the total sound energy E[t] where E[t] = ML[t] + MR[t]

(8.3)

The classification of the SA[t] values will be based on the total energy at times t and t + 1. So (8.4) P = P ∪ SA[t] i f E[t + 1] ≥ E[t] or N = N ∪ SA[t] i f E[t + 1] < E[t]

(8.5)

Using the set of positive examples as a new training set, the GA based technique can generate a new neural network. In order to increase the efficiency of the GA, the current best neural network can be used to seed the initial population.

162

P.M. McDowell, B.S. Bourgeois, and F.E. Petry

Fig. 8.3. The reactive learning process. Key to the process is the intensity filter. It ensures that the memory is composed of examples for which actions at time t had a direct influence on sensor values at t + 1. As a part of a robot’s architecture, this process would be called when an observer function notices goals are not being met.

We have described only the use of positive examples for which we present experimental results in section 7. However, it was also demonstrated in our laboratory simulations that negative examples could also be used by rewarding a neural network that is being trained if it takes a different action than the one associated with the example.

8.6 Memory Based Controller Figure 8.4 below shows the neural network configuration used for demonstrating learning with a memory based controller. The memory based controller behavior is achieved by using past sensor inputs in addition to the current input. The choice of the number of past inputs dictates the span of time over which the controller will be able to observe trends and also its ability to react quickly to sudden changes. The motivation for testing learning with this type of controller was to ensure proper handling of the forward-backward ambiguity mentioned earlier. Creating a training set for this type of controller proved to be significantly more challenging and was accomplished by splitting the recent memory into four categories with respect to the average sensor intensity: rising, dropping, stable and fluctuating. These

8 Robot Control in Dynamic Environments Using Memory-Based Learning

163

Fig. 8.4. A feed forward neural network with past inputs

Fig. 8.5. A time series of the goal parameter. The line is drawn in different line styles to illustrate the principal parts that it is made of. The bold part of the line fits in the category of rising parts, the dashed part in the dropping category, the dotted part in remaining stable category, and the thin part near the middle of the time series is the fluctuating category.

cases are illustrated in Figure 8.5, with the 4 categories referred to as the ‘principal parts’ of the memory. Since the controller uses 4 consecutive time inputs, the training set is composed of groups of 4 consecutive sets of sensor-action values obtained from the recent memory M. The classification of the memory into its principal parts will be based on the sound intensity value I ( the average of ML and MR). So now SA[t] becomes a 4-tuple

164

P.M. McDowell, B.S. Bourgeois, and F.E. Petry

including I : (I, ML, MR, A) and a recent memory M of length n is again a sequence: SA[0]..SA[n − 1]. Next we will describe the characteristics of the principal parts categories: RC: rising category where I(t) > I(t − 1). This is further subdivided into subsets in which the action is consistent. RC left - all actions are left turns RC straight - all actions are straight RC right - all actions are right turns Whether any given series of 4 consecutive SA values (referred to as a ‘run’) in the M belong to these categories is determined by applying the following rules time sequentially throughout the memory: SA[t] ∈ RC left if I(t) > I(t − 1) ∧ A(t) =left Membership in RC straight and RC right are defined similarly. SC: stable category where I(t) = I(t − 1) ± ε ; ε is a designer specified small value. This is further subdivided into subsets in which the action is consistent: SC left, SC straight and SC right as above except = is used instead of >, relative to I in the rules. DC: dropping category where I(t) < I(t − 1). Again we have subsets in which the action is consistent: DC left, DC straight and DC right and for which < is used instead of > for the rules. FC: fluctuating category. All series of 4 SA values in the memory that do not fall into the RC, SC, or DC categories are placed in the fluctuating category. The training sets are formed from the RC, SC and DC groups, with a slight variation of the DC components. For the RC and SC groups the action taken during each particular run is assumed to be correct, so the neural networks in training are rewarded when they take this same action. However, when the goal parameter is dropping the correct action is unknown. To determine the action to be used for a run that falls into the dropping category, an adjacent or overlapping rising run must be found. Once the overlapping example is found, its action is used as the correct action for the dropping run. It is used because it is the first correct action that occurs since the negative energy gain. If there is no adjacent or overlapping rising run, the dropping run is not used. Fluctuating runs are not used to create training examples at this time because there is no programmatic method of determining what a correct action is for these states.

8.7 Experimental Results Quantitative testing was conducted with a LabView [19] simulation that measured the distance and the heading differences between the leader and follower robot in a simple maneuvering task. In the test, illustrated in Figure 8.6, the lead robot traverses two laps around a preprogrammed course. The tests are started with the robots in formation, by maneuvering the leader-follower pair to the starting position and then engaging the

8 Robot Control in Dynamic Environments Using Memory-Based Learning

165

Fig. 8.6. Test course for the following robots. The lead robot (gray robot) is controlled by the simulator. The follower robot (clear robot) is controlled by the current controller being tested. Over two laps the inter-robot distance and course differences are recorded.

Fig. 8.7. The leader/follower distance over two laps. The follower robot is controlled by a reactive FFNN. As can be seen in the figure, the robot closed the distance to the leader and maintained it throughout the run except during the turns.

automatic navigation control for the leader robot [11,12,22]. The robots’ turning rates and speeds were set the same for all tests. The tests are started in the upper left hand corner of the track, when the leader/follower combination is in formation and tracking. Tests were also conducted using ActivMedia robots [1] equipped with microphones and speakers. The reactive feed forward neural nets (FFNNs) generated were able to control the robots in these tests although the sensor history network has not yet been tested with the robots. Figure 8.7 shows the average distance between the leader and a follower controlled by a typical reactive neural network trained as previously discussed. The distance between the two robots remains very stable except during the turns. The turns are the eight places in the curve where the distance fluctuates. Figure 8.8 next shows the heading

166

P.M. McDowell, B.S. Bourgeois, and F.E. Petry

Fig. 8.8. The heading difference between the leader and the follower robot controlled by a reactive FFNN during a two lap test in the simulator. Notice that each of the 8 peaks (the peak denotes a course change by the leader) occur at about the same time as the fluctuations occur in the curve in distance plot.

Fig. 8.9. The leader/follower distance for the two lap test where the follower is using the classic logic controller

differences measured during the same test. The heading difference grows large after the leader turns since it takes the follower a finite time to respond to the changing relative position of the leader and effect its own course change to follow. For comparison, Figures 8.9 and 8.10 show a similar test run using a classic logic controller based on Braitenberg’s robots [7,17]. The FFNN controller is seen to hold its distance almost as consistently as the classical controller. While the classic controller has a slight advantage on the robot-to-robot distance, the FFNN handles the turns a little

8 Robot Control in Dynamic Environments Using Memory-Based Learning

167

Fig. 8.10. This figure shows the course difference between the leader and follower where the follower is using the classic logic controller. Each peak corresponds to when the leader is making a turn.

Fig. 8.11. The leader/follower distance for the two lap test when the follower is using the principle parts controller

more smoothly, as can be seen by examining the oscillations in the robot-to-robot distance near the turns in Figures 8.7 and 8.9. Also, it was noted that the classic controller tends to oscillate more about the proper heading, depending upon the operator-specified ‘dead-zone’. This is observed in the thick lines in the course difference figure at the end of each turn, where the following robot oscillated about the desired course. One of the advantages of a learning system is the elimination of heuristically set parameters such as the dead zone size. Results for the principle parts (memory based) controller are shown in Figures 8.11 and 8.12. As can be seen in these graphs the simulated robot was able to follow

168

P.M. McDowell, B.S. Bourgeois, and F.E. Petry

Fig. 8.12. This figure shows the course difference between the leader and follower for the principle parts controller. Each peak corresponds to when the leader is making a turn. Notice that this controller takes more time to complete turns than the reactive controllers.

successfully, although it took much longer than the reactive controller to steady onto a heading. This result is anticipated since the reactive controller responds only to the last set of sensors values while the principle parts controller looks for a trend in the last n values and will typically be slower to respond. Tests were also conducted to demonstrate that the principle parts controller could properly deal with the forward/backward ambiguity problem, which is a known problem for the reactive controller. The outcomes of the tests were mixed. Holding the memorybased robot still (zero velocity), but letting it control its direction and moving the source directly at it quick enough for there to be a sensor gradient present in its input, allowed the robot to track the source. Slowing the source’s approach to where the gradient in the sensor history did not exist caused it to behave as the reactive system.

8.8 Conclusions Learning using recent memories was demonstrated successfully with both a reactive and memory based controller. Simulation testing with the formation following application showed promising results, and the reactive controller was also tested successfully on land robots. The use of the genetic algorithm to train a neural network controller based on selected recent memories gathered from the random but purposeful exploration was observed to be efficient and suitable for implementation in a real-time learning system. This work represents a new approach to in situ learning for an autonomous robot based on experiences (sensor/action pairs), without requiring exhaustive testing of all possible actions or the use of a simulator requiring a substantial amount of a priori knowledge about the operating environment. Future work may involve the development of a graph learning algorithm which can create a controller whose purpose is to achieve a specific goal. A directed graph can be built and updated from memories recorded as the robot explores and operates in the

8 Robot Control in Dynamic Environments Using Memory-Based Learning

169

environment. The graph’s structure can represent the robot’s experiences in a topological manner, preserving the temporal sequencing information. This could be used to train a neural network or by traversing the graph in real time, directly controlling the robot.

Acknowledgement We would like to thank the Office of Naval Research through the Naval Research Laboratory for sponsoring this research.

References 1. ActivMedia Robotics, Pioneer 2 Operations Manual, Version 1-1, Petersborough, NH (2002) 2. Angeline, P., Saunders, G., Pollack, J.: An Evolutionary Algorithm That Constructs Recurrent Neural Networks. IEEE Transactions on Neural Networks 5, 54–65 (1993) 3. Balch, T., Arkin, R.: Behavior-Based Formation Control for Multirobot Teams. IEEE Trans. on Robotics and Automation 14, 926–939 (1998) 4. Balch, T., Parker, L. (eds.): Robot Teams: From Diversity to Polymorphism. AK Peters. Wellesley (2002) 5. Bourgeois, B., Petry, F., Harris, M., Alleman, P.: A GIS Integration Approach for Dynamically Reconfigurable Surveys. Hydrographic Journal 91, 3–11 (1999) 6. Bradley, M.: Environmental Acoustics Handbook, 2nd edn. (1996) 7. Braitenberg, V.: Vehicles: Experiments in Synthetic Psychology. MIT Press, Boston (1986) 8. Buckles, B., Petry, F.: Genetic Algorithms: Introduction and Applications. IEEE Computer Society Press, Los Alamitos (1992) 9. Chance, T., Kleiner, A.: The Autonomous Underwater Vehicle (AUV): A Cost Effective Alternative to Deep-Towed Technology. Integrated Coastal Zone Management 8, 65–69 (2002) 10. Church, R., Warren, D.: New Technologies Rewrite History. Hydro International 6, 56–59 (2002) 11. Desai, J., Ostrowski, J., Kumar, V.: Modeling and Control of Formations of Nonholonomic Mobile Robots. IEEE Trans. on Robotics and Automation 17, 905–908 (2001) 12. Fierro, R., Das, A., Kumar, V., Ostrowski, J.: Hybrid Control of Formations of Robots In: Proc. IEEE Int. Conf. on Robotics and Automation, Seoul, Korea, pp. 157–162 (2001) 13. Grefenstette, J., Schultz, A.: An evolutionary approach to learning in robots. In: Proc. Learning Workshop on Robot Learning, New Brunswick, NJ, pp. 47–56 (1994) 14. Holland, J.: Designing Autonomous Mobile Robots. Elsevier, Amsterdam (2004) 15. Jiang, N., Zhao, Z., Ren, L.: Design of Structural Modular Neural Networks Genetic Algorithm. Advances in Engineering Software 34, 17–24 (2003) 16. McDowell, P., Chen, B., Bourgeois, B.: UUV Teams, Control From A Biological Perspective. In: Proceedings of the Oceans, MTS/IEEE Conference, Biloxi MS, pp. 331–337 (2002) 17. Mitchell, T.: Machine Learning. McGraw-Hill, New York (1997) 18. Nehmzow, U.: Mobile Robots: A Practical Introduction, 2nd edn. Springer, Heidelberg (2003) 19. NI LabVIEW Manual, National Instruments, Austin TX (2000) 20. Proakis, J., Sozer, E., Rice, J., Stojanovic, M.: Shallow Water Acoustic Networks. IEEE Communications Magazine 39, 114–119 (2001) 21. Rucci, M., Wray, J., Edelman, G.: Robust localization of auditory and visual targets in a robotic barn owl. Robots and Autonomous Systems 30, 181–193 (2000)

170

P.M. McDowell, B.S. Bourgeois, and F.E. Petry

22. Sgorbissa, A., Arkin, R.: Local Navigation Strategies for a Team of Robots. Robotica 21, 461–473 (2003) 23. Sofge, D., White, D.: Applied Learning: Optimal Control for Manufacturing. In: White, D., Sofge, D. (eds.) Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches, Van Nostrand Reinhold, pp. 272–273 (1992) 24. Stroupe, A., Balch, T.: Value-Based Action Selection for Observation with Robot Teams Using Probabilistic Techniques Journal of Robotics and Autonomous Systems 50, 85–97 (2005) 25. Touzet, C.: Distributed Lazy Q-Learning for Cooperative Mobile Robots. Int. Jour. Of Advanced Robotic Systems 1, 5–13 (2004) 26. Urick, R.: Principles of Underwater Sound, 3rd edn. McGraw-Hill, New York (1983) 27. Watkins, C., Dayan, P.: Q-Learning. Machine Learning 8, 279–292 (1992) 28. Yamaguchi, H., Arai, T.: Distributed and Autonomous Control Method for Generating Shape of Multiple Robot Group. In: Proceedings of IEEE Int. Conf. on Intelligent Robots and Systems, pp. 800–807 (1994)

9 A Behavior Based Control and Learning Approach to Real Robots Dongbing Gu1 , Jindong Liu2 , and Huosheng Hu3 1 2 3

Department of Computing and Electronic Systems, University of Essex, United Kingdom [email protected] Department of Computing and Electronic Systems, University of Essex, United Kingdom [email protected] Department of Computing and Electronic Systems, University of Essex, United Kingdom [email protected]

Abstract. Programming a real robot to do a given task in unstructured dynamic environments is very challenge. Incomplete information, large learning space, and uncertainty are major obstacles for control in real robots. When programming a real robot in unstructured dynamic environments, it is impossible to predict all the potential situation robots may encounter and specify all robot behaviors optimally in advance. Robots have to learn from, and adapt to their operating environment. In this chapter, we propose to use fuzzy logic to design robot behaviors and use a Markov decision process to model the coordination mechanism in the control and learning of real autonomous robotic systems. Based on the model, a Q-learning approach can be used to learn the behavior coordination. Two real robot applications are implemented by using such an approach, one is a Sony quadruped robot for soccer playing and another is a robotic fish for entertainment. Real robot testing results are provided to verify the proposed approach.

9.1 Introduction Programming autonomous robots raises many difficulties because they not only have multiple motors for movement, but also most of them contain heterogeneous sensors required to achieve goal-oriented tasks. The program requirements for these robotic systems are not limited to keeping them moving, but include the commitment to the perception of its environment, its autonomous operation in a changing environment, and achieving goal-oriented tasks. This chapter is concerned with programming with two autonomous robotic systems. One is an autonomous quadruped robot, the Sony quadruped robot, which is used as a member of a team to play soccer in the RoboCup challenge, and another a robotic fish, which is used for entertainment in aquarium. Autonomous robots are designed to achieve goal-oriented tasks, such as scoring a goal in RoboCup or randomly swimming in a tank. The major challenges when programming such a robot stem from: i) the dynamic environment in which the robot is located and it is unable to predict when events will occur. (ii) Uncertain sensory information that is inaccurate, noisy, or faulty. (iii) Imperfect actuators that cannot guarantee correct and perfect execution of actions D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 171–186. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

172

D. Gu, J. Liu, and H. Hu

due to mechanical, electrical, and servo problems. (iv) Limited time that constrains time intervals needed for sensor information processing, actuator control, and goal-oriented planning. For autonomous mobile robots, early programming approaches followed a sequence: sensing the environment, planning trajectories, and controlling motors to move. These approaches belong to a deliberative control approach that needs rigid design models in each part of the system [3]. Robots employing this type of control architecture are well known for their lack of real-time response and robustness. Reactive control or Behaviorbased robotics provides an alternative solution to the problems described above [3][2]. In behavior-based robotics, behavior design and behavior co-ordination are two essential topics. To design a behavior is to find a mapping function from environment states to motor actions associated to this behavior. Due to less computation in the direct mapping, behaviors possess a real-time response ability. For a specific domain, a set of behaviors defines robot skills in the domain and they constitute the fundamental basis for the control architecture. Each behavior is designed to achieve a specific subtask and their coordination forms the deliberative planning for goal-oriented tasks in the domain. When designing a behavior or behavior co-ordination mechanism, it is impossible to predict all the potential situation robots may encounter and specify all robot behaviors optimally in advance. Robots have to learn from, and adapt to their operating environments. Learning has been recognisee as one of the basic abilities for autonomous robots. To take advantage of the strengths of deliberative and reactive control architectures, and attempt to avoid their weakness, some research has been conducted into hybrid architectures [5][6]. Hybrid architectures possess the planning ability of a deliberative architecture to achieve goal-oriented tasks, but also apply the fast response ability of reactive architectures in order to deal with environment dynamics. In this chapter, we design a layered control architecture for autonomous robots, which includes a cognition layer for planning, a behavior layer for reactive control, and a locomotion layer for basic movement skills. In the cognition layer, a state-based planning method is applied for reasoning via a Markov Decision Process (MDP). The state-based planning can be combined with Q-learning to implement the learning in the cognitive layer. Q-learning is a trial-and-error learning approach that interacts with dynamic environments [1][11]. It explicitly considers the whole problem of a global goal, and its final purpose is to maximize a training signal over a period of time, rather than to maximize instant feedback in each trial. This matches our goal-oriented tasks in the cognitive layer. In the behaviors layer, reactive control is realized based on a group of behaviors. The design methodology of a behavior has a wide spectrum, including heuristic rules, potential fields, biological inspired approaches, fuzzy logic, and many learning algorithms. Using fuzzy logic to design behaviors can enable the behaviors to handle uncertain sensory readings and imperfect motor control and have been successful used for many real robots [10][7][4][9][8]. In fuzzy logic, fuzzy sets model the uncertainties and fuzzy rules work co-operatively to produce final control results. The produced control effect is smooth, reasonable, and robust, as each fuzzy rule makes its contribution to final outputs by its activated degree. Furthermore, using fuzzy logic to design behaviors can make use of prior experiences, which can be expressed as fuzzy rules or employed to

9 A Behavior Based Control and Learning Approach to Real Robots

173

select the parameters of fuzzy membership functions. Optimizing fuzzy behaviors or accurately defining fuzzy rules and their parameters is still a challenge. Learning or adaptation of fuzzy behaviors provides a feasible solution to the challenge, since the learning can modify the parameters of fuzzy logic to improve the behavior performance or autonomously select fuzzy rules to achieve the behavior automatic acquisition. In the following, we will introduce the layered control architecture in section 9.2 first. Then the using of fuzzy logic to design behaviors is described in section 9.3. The model and learning of the cognitive layer are presented in section 9.4. The testing results of both Sony quadruped robots and robotic fish are provided in section 9.5 and section 9.6, respectively. Finally a brief conclusion is summarized in section 9.7.

9.2 Layered Control Architecture A layered architecture is designed for autonomous robotic systems. Figure 9.1 presents the proposed layered control architecture, which consists of three layers: a Cognitive Layer, a Behavior Layer, and a Locomotion Layer. At the center of the architecture is the behavior layer; where most of the instant decisions are made. The locomotion layer is particularly designed for different kind of robotic systems. For example, it is called swim pattern Layer in our robotic fish and gait control Layer in our legged robots. In the behavior layer, states and actions are directly related to robot physical sensors and gaits or swim patterns in the locomotion layer. In the cognitive layer, states relating to high-level activities are extracted from the behavior layer, and actions correspond to behavior coordination.

Internal World States

MDP-based learning schema

Actions

Cognitive Layer Behaviour 1 Sensors Behaviour n Motions Behaviour Layer Motion 1

Motion n

Locomotion Layer Motors Fig. 9.1. The layered control architecture for autonomous robotic systems

174

D. Gu, J. Liu, and H. Hu

From control point of view, programming autonomous robots based on the layered architecture is a feasible and practical scheme. With this scheme, large sensory and action spaces can be separated into a number of small local regions both in the range and dimensions. Searching small local regions to find solutions is easier than operating on a large space. From learning point of view, a bottom-up learning structure can be applied to the control architecture. The locomotion layer establishes the optimal locomotion parameters first, and then provides robust locomotion to the behavior layer. The behavior layer then starts to optimize the parameters of each behavior. Finally, the cognitive layer takes the optimal behaviors as actions, and learns an optimal policy between states and actions. 9.2.1

Locomotion Layer

The Locomotion layer is at the bottom of the three-layer architecture. For our robotic fish, this layer consists of five swim patterns: Cruise-Straight, Ascent/Descent, Coast, Sharp-Left/Right-Turn and Cruise-in-Left/Right-Turn. A swim pattern is defined as a temporal sequence of the associative movement of tail joints. For our quadruped robots, the locomotion layer is to implement basic walking gaits. It provides a group of walking commands to the behavior layer. The walking commands includes Move Forward, Left Forward, Right Forward, Left Turn, Right Turn, and Stop. When the behavior layer finishes behavioral coordination, it produces a locomotion command to be executed. The robotic system inputs this locomotion command into the locomotion layer and implements it until a new command is received from the behavior layer. 9.2.2

Behavior Layer

The behavior layer is located in the middle of the control architecture. It is responsible for the reactive control of the robotic systems. It converts raw sensor data into sensor states using fuzzy membership functions first before inputting these states into each behavior. A fuzzy logic controller is designed for every behavior to process states. For our robotic fish, eight behaviors are created and listed as follows. • Keep-level: This is the most efficient for long distance swimming. • Avoid-obstacle: This behavior is used to move away from other fish, predators, and rocks, etc. A significant feature of this behavior is that a fish changes its swimming direction (pitch, roll, and yaw angles) to avoid obstacles, rather than slow down or come to a rest. This is because changing direction is much easier than braking. • Follow-wall: Fish swim along the surface of rocks to patrol their territory and find food. This behavior helps fish to explore an unknown environment in a safe way, because it decreases the risk of predator attack. • Wandering: Fish employ this behavior for relaxation. Our robotic fish apply this behavior to swim around in order to find potentially interesting objects. • Seek-goal: Food, mate, and home are the main goals for real fish. For the robotic fish, the charging station is one goal, and the tank wall is another when they implement a tank exploration task. This behavior enables robotic fish to find a way to a charging station.

9 A Behavior Based Control and Learning Approach to Real Robots

175

• Rescue: In some dangerous situations, when fish cannot get themselves out of trouble through other behaviors, a rescue behavior can be applied. In most cases, a fish will reduce its movement as much as possible and keep still. The spots and strips on their body confuse a predator, and provide camouflage. Likewise, our robotic fish implement this behavior if they face uncontrollable situations, such as a servo motor malfunction, or extremely high internal temperature, etc. They will stop performing all other behaviors, and empty their water chamber to float up slowly to the water surface. • Feed: This behavior allows fish to find food. For example, a shark chases a small fish and swallows it. For our robotic fish, the feed behavior enables them to adjust their movements to swim towards a charging station, in order to charge their battery. For Sony quadruped robots to play an attacker role, five behaviors are created and listed as follows. • Find-ball: The robot starts to scan the field through its head motion. Once a ball is found, the robot updates the ball’s position and its confident value. Simultaneously the robot rotates its body to increase its view and gradually moves towards the center of the pitch. • Chase-ball: The robot moves its head to track the ball. It moves towards the ball and tries to go behind the ball to face the goal in order to kick the ball. The sensory states of this behavior include ball’s co-ordination, goal’s position, and robot’s position. • Align with goal: The robot slightly adjusts its position to align the ball and its orientation with the goal in order to kick the ball into the goal. The sensory states of this behavior include ball’s co-ordination, goal’s position, and robot’s position. • Kick-ball: some specialized kick skills are applied. • Dribble-ball: The robot kicks the ball by using its feet and keeps its body moving. It’s very useful when the ball is close to edges or corners of the pitch. 9.2.3

Cognitive Layer

The top layer of the architecture is the cognitive layer. This layer first extracts the robot status from the sensor states in the behavior layer. It then performs reasoning and planning based on an internal world which defines the robot constraints and environment information. Finally, it makes decisions on how to coordinate behaviors to achieve a given task. In contrast to a traditional deliberative control architecture, the cognitive layer does not provide a trajectory for robots follow. The only way it affects and supervises the behavior layer, is through the global performance of emergent behavior. The aim of the cognitive layer is to organize actions to accomplish a task. An action can enable a behavior, or set a configuration parameter of a behavior. Actions in the cognitive layer are denoted ca and are related to all behaviors, States in the cognitive layer are extracted from sensor readings. They are the highlevel conditions and reflect significant physical events, which are sensed or recognized by a temporal and spatial combination of several sensors. For example, if the downfacing infrared sensor outputs a higher value and the pressure sensor is larger than a threshold value, they indicate that the fish is near the bottom of the tank. Events such as these constitute the state space. Formally, each event is denoted by a binary

176

D. Gu, J. Liu, and H. Hu

variant bv. Once an event occurs, the related bv is set to 1; otherwise it is cleared to 0. Grouping n events (bv1 , ..., bvn ) in an order, generates a state cs for the cognitive layer, i.e., cs ← bv1 , ..., bvn . The state space consists of 2n combinations of n events. A policy π is a mapping from states to actions. The optimal policy π ∗ maximizes the probability of reaching the goal. In section 9.4, we introduce the Q-learning, in which each policy generates a specify value and maximizing the value function leads to obtaining an optimal policy.

9.3 Behavior Design Using Fuzzy Logic Acquiring behaviors is a fundamental step in the control design of autonomous robots. Fuzzy logic is a widely used approach for the robot behaviors [10][7]. In fuzzy logic, linguistic rather than numerical variables are used (which mimics humans reasoning). Our experience is incorporated into the system in this way. Each behavior receives input signals, i.e. states, from a perception module. It then processes these using a fuzzy logic, and generates a response, i.e. an action. A typical fuzzy controller is constructed from four elements: a Fuzzifier, a Fuzzy Rulebase, a Fuzzy Logic Inference Engine, and a Defuzzifier. The Fuzzifier translates input signals into linguistic values, i.e. fuzzy input variables, using membership functions. The Fuzzy Logic Inference Engine evaluates the set of fuzzy rules defined in the Fuzzy Rulebase, identifies rules that apply to the current input variables, and calculates output linguistic values, i.e. fuzzy output variables. The Defuzzifier translates the output variable to a crisp control signal, i.e. a behavior response. In the following, a follow-wall behavior example is illustrated for behavior design in our robotic fish. This behavior keeps a robotic fish at a certain distance from a wall, based on the information gathered by its IR sensors. Figure 9.2 shows a block diagram of the follow-wall behavior. The left, right and bottom IR sensor data are the inputs of the perception module. One more input Wd , is added to define the relative position of the wall to the robotic fish. It is appointed by the cognitive layer, and defined in (9.1). For instance, if Wd = 1 the robotic fish should follow the wall on its left side. The perception module outputs an IR intensity ds , which is either dl or dr ; depending on Wd using (9.2). Via the fuzzy controller, the output ωd is tweaked using Wd to calculate ωd = Wd ωd ; which is one of the inputs of the Bev-coordination module. Finally, swim pattern S p is generated to be executed on the robotic fish. Another output of the perception module Wd

follow-wall behaviour

db

Fuzzy Logic Inference Engine

Defuzzifier

B_IR

Fuzzy Rulebase

ds Fuzzifier

R_IR

Perception

L_IR

ω 'd

ωd

Pt

Bev Coordination

Wd

Sp

Pt*

Fig. 9.2. A fuzzy logic representation of the follow-wall behavior

Robotic fish

9 A Behavior Based Control and Learning Approach to Real Robots

177

μ far

middle near

1 IR Intensity (ds,db) 0

25% 50% 75% 100% m11 m12 m13 m21 m22 m23

(a)

μ

μ RS

Z

NS

LS

1

-60 -30 n11

0 30 n12 n13

(b)

Z

PS

1

60

ω'd(degree)

-30 -15 n11

0 n12

15 n13

30

P t(mm)

(c)

Fig. 9.3. Membership functions of the input and output variable for follow-wall. mi j , nls : are the centers of the membership functions. μ : is the degree of truth. (a) the IR intensity ds ; (b) the yaw angle ωd relative to the current swimming direction; and (c) the position of the pitch weight Pt .

is db , which is the intensity of the bottom IR (which affects another fuzzy controller output–Pt ).  1 le f tside Wd = (9.1) −1 rightside  dl , if Wd = 1 ds = (9.2) dr , if Wd = −1 Figure 9.3 shows the membership functions of ds , db , ωd and Pt . The parameters (mi j , ni j ) are decided by the features of the IR sensors and the mechanical structure of the fish. For example, when a robotic fish follows a wall at an appointed distance (default 0.3 m), the obtained samples of the side IR intensity are approximately 25% to 75%. Therefore, the center of the middle term of ds is set to 50%, and the range is set from 25% to 75%. ωd is encoded in three terms, because the possible horizontal reactions of the follow-wall behavior are limited to the cruise straight, cruise in left turning, and cruise in right turning swim patterns. These three swim patterns have roughly a 0◦ , 30◦ and −30◦ turning angle. Therefore, the centers of the three terms are set as 0, 30 and −30. The membership of Pt is decided via the mechanical design of the pitch control module, where a pitch weight moves. For example, the robotic fish will dive slowly when the pitch weight moves to a −15 mm position; therefore the center of the negative small diving (NS) term is set to −15.

178

D. Gu, J. Liu, and H. Hu

Table 9.1. Rulebase for the follow-wall fuzzy controller. N: near, M: middle, F: far, RS: right small turn, LS: left small turn, Z: zero. PS: positive small (ascent), NS: negative small (descent), N/A: no input or output for the rule.

Rule Input Output No. ds db ωd Pt 1 N N/A RS N/A 2 M N/A Z N/A 3 F N/A LS N/A 4 N/A N N/A PS 5 N/A M N/A Z 6 N/A F N/A NS The rulebase for the fuzzy inference engine is listed in Table 9.1. It shows that ds and db independently affect ωd and Pt , because heading change is only related to the side IR sensors, and the diving control is only related to the bottom IR. These rules are decided based on our experience and the mechanical configuration of the IR sensors. For example, if Wd = 1 in (9.2), the first rule can be interpreted as: “IF the left IR intensity is near, THEN make the fish turn right a little, i.e. ωd = RS, regardless the bottom IR”. After all the rules are evaluated, a classical defuzzify method, the center of gravity (COG) method, is selected for defuzzification (see (9.3) and (9.4)). This method combines the outputs represented by the implied fuzzy set from all rules in order to generate the gravity centroid of the output distributions for the final output.

ωd =

∑6k=1 v1k qk ∑6k=1 v2k qk , P = t ∑6k=1 qk ∑6k=1 qk

(9.3)

qk = min{ μ1k1 , μ2k2 }

(9.4)

where v1k and v2k denote the estimated value of the kth rule output; which is related to the center of the membership functions of the fuzzy output variables ωd and Pt . qk is the conjunction degree of the IF part of the kth rule, which is calculated using a min(·) function (because the AND fuzzy logic operator is applied). μiki is the degree of the membership for the ith input contributing to the kth rule; ki is the term number of the input variable corresponding to the ith input in the kth rule. μiki is 1 if the corresponding input in Table 9.1 is “N/A”.

9.4 Q-Learning in the Cognitive Layer In the cognitive layer, the correct behavior should be selected to achieve specific tasks, i.e., the robot should reason or plan actions according to states. Reinforcement learning, specifically Q-learning, is used as a planner, which consists of three parts: Action Space - a set of possible actions; State Space - the discrete possible situations of a robot moving from an initial place to a goal; and World Model - a mapping from the state to

9 A Behavior Based Control and Learning Approach to Real Robots

179

action space. A Markov Decision Process (MDP) model can be used to formally model such a planner. Q-learning can be used to learn the mapping function in this model. The state transitions are described by the transition probabilities. P = Pr{cs(t+1) |cs(t) , a(t) }

(9.5)

where Pr{} denotes the transition probability. The expected value of the next reward, given the current state and action, cs(t) and a(t) , together with the next state, cs(t+1) , is expressed as. R = E{rt+1 |cs(t) , a(t) , cs(t+1) }

(9.6)

where E{} denotes the operation of calculating the expected value. A policy π is a mapping from states to actions. The optimal policy π ∗ maximizes the probability of reaching the goal. The value of a state s under a policy π , denoted V π (s), is the expected return when starting in s and following π thereafter. Function V π is referred to as the state-value function for policy π . So V π (cs) is defined as. ∞

V π (cs) = Eπ {Rt |cs} = Eπ { ∑ γ k rt+k+1 |cs}

(9.7)

k=0

where Eπ {} represents the expected value when the robot follows policy π . At the same time, the value of taking an action a in a state cs, is defined under policy π as Qπ (cs, a). ∞

Qπ (cs, a) = Eπ {Rt |cs, a} = Eπ { ∑ γ k rt+k+1 |cs, a}

(9.8)

k=0

The one-step Q-learning updated function is written as follows[12]:   Q(s, a) = Q(s, a) + α r + γ max Q(s , a ) − Q(s, a) a

(9.9)

where Q represents Qπ ; s denotes cs; α and γ are learning ratios; r is the observed reward when taking action a; and s is the succeeded state after taking action a. The ε -greedy method is used to generate a from s using the policy derived from Q. Assume that all Q values continue to be updated during learning, and that the learning rate α satisfies the following two conditions. ∞

∞

t=0

t=0

∑ αt = ∞ and ∑ αt2 < ∞

(9.10)

Then Q has been shown to converge with a probability 1 to optimal Q∗ [11]. Pseudo code for Q-learning is listed below. 1. Initialize: Q(s0 , a0 ) arbitrarily; 2. While t < TE do: 3. choose at from st using the policy derived from Q and ε -greedy; 4. Take action at , observe new   state st+1 and reward rt+1 ; 5. Q(st , at ) = Q(st , at ) + α rt + 1 + γ max Q(st+1 , a) − Q(st , at ) ; a 6. t=t+1; 7. endwhile

180

D. Gu, J. Liu, and H. Hu

9.5 Robotic Fish Experiments Figure 9.4 shows the profile of our robotic fish. They are approximately 52 cm long, and have 3 servo motors, and two DC motors. The servo motors are concatenated together in the tail to act as 3 joints; one DC motor is fixed in the head to change the Center of Gravity (COG) of the fish, while the other controls a micro-pump in order to adjust the weight of robot. The central controller for each robotic fish is based on a 400 MHz Gumstix computer [13]. The Gumstix samples data from sensors, processes data, makes decisions, and sends signals to several PIC micro-controllers. These PIC controllers include three servo control boards, a DC motor control board, and a sensor board. The head of the robotic fish is waterproofed and rigid to counteract water pressure. The tail consists of three joints, which are designed to seal three servo motors independently. Each fish has over ten embedded sensors: a gyroscope, a pressure sensor, two position sensors, two electrical current sensors, a voltage meter, four infrared sensors, and an inclinometer. These embedded sensors enable the fish to detect: the depth in the water, the yaw/roll/pitch angles of its body, and a range of obstacles in front. In addition, the turning angle and electrical current consumption of each servo motor can be obtained. Bluetooth and RS232 serial ports are used as a means of communication with an external PC, which is used to program the Gumstix and PIC controllers. A sensor log can be retrieved via Bluetooth. The robotic fish are fully autonomously controlled to swim underwater in 3D and no supervision is taken during their swimming. On the top of the fish body, a dorsal fin is fixed vertically to avoid the fish wobble to side. Two pectoral fins are attached on two sides of the head to help balance the fish body. Both dorsal fin and pectoral fins are inactive. The propulsion force for the robotic fish swimming comes from an associated undulated movement of three joints, which make the fish body to push the surrounding water and obtain reaction force from the water. The design of the joint movement is the key factor to affect the swimming performance, such as swimming efficiency and head undulation angle, etc. In the cognitive layer, states and actions are discrete variables, and the Q-learning (state based learning) can be implemented here conveniently.

Fig. 9.4. A robotic fish

9 A Behavior Based Control and Learning Approach to Real Robots

181

3

2.5

y−width (m)

2

1.5

1

0.5

0

0

0.5

1

1.5 x−length (m)

2

2.5

3

Fig. 9.5. The fish trajectory during learning. Please note that the dashed line is the desired path, α = 0.5, γ = 0.3, ε = 0.01. 60

Root Meas Square Error (m)

50

40

30

20

10

0

0

50

100

150 Episode

200

250

300

350

Fig. 9.6. The root mean square error between the learning trajectory and desired path, against the number of learning trips. The error is the sum of two parts: position error and heading error. Each trip had 50 steps.

To test the learning algorithm, a tank border exploration task was implemented using our robotic fish. The objective is to make a robotic fish autonomously explore a tank border. The fish should follow tank walls at an appointed distance, be able to avoid corners and other fish, and maintain itself in a desired depth level. Additionally it must keep the wall to its right hand side. The robotic fish is not supposed to know anything about its environment. It will learn to choose a correct action according to its state. The action space is structured according to the objective of the task. Three behaviors

182

D. Gu, J. Liu, and H. Hu

(Wander,Follow-wall, and Avoid-obstacle) are chosen and customized from the behavior layer for the level plane action subspace. There is no vertical action subspace in the cognitive layer for this task because the keep-level behavior is capable enough for the task. Three events are defined to create states (cs): • Is the wall on the right hand side of the robotic fish? (bv1 ): This is determined using the right, front, and left infrared sensors. bv1 = 1 if the recent history of these sensors satisfies some conditions. • Is the robotic fish at a reasonable distance from wall? (bv2 ): This event is similar to bv1 , but has more strict conditions. • Is the wall on the left hand side of the robotic fish? (bv3 ): Like bv1 , it recognizes the situation that the nearest wall is on the left hand side of the fish. In other words, the swimming direction is the reverse of the desired direction in this situation. In this task: r = 1 when the robotic fish in the follow-wall behavior is able to successfully follow the wall; r = −1 when the fish in the follow-wall behavior loses the wall it has been following; r = −3 when there is a bump between the wall and robotic fish; and r = 0 for all other situations. According to the Q-learning algorithm, policy π ∗ of the task is learnt in a 3D simulator, and then applied to the real robotic fish. Figure 9.5 presents fish swimming trajectories during learning. It shows that the robotic fish maintains a required distance from the wall most of the time. Figure 9.6 provides a history of the root mean square (RMS) error between the learning trajectory and desired path over 300 trips. The RMS error is the accumulated error for one learning episode. It decreases and converges to a low value as the number of learning trips increases. Figure 9.7 shows the trajectory of our robotic fish operating in a real tank. The optimized parameters in the simulation were implemented in the fish. The arrows in the figure show heading directions at specific points, while the circle points indicate the position of the robotic fish. The time step between two record points is 3 seconds. The robotic fish was put into the water at point S, with a heading direction B0 . During swimming, the fish encountered an obstacle at point B1 . It implemented a sharp-left-turning swim pattern to avoid it. It then found the wall again using the cruise-in-turning-right swim pattern. Finally, the robotic fish spent 105 seconds swimming around the interior perimeter of the tank and finally reached point E.

y−width (m)

1.5

S(0s) B0

1

0

B1

E(111s)

0.5

(60s) B2 0

1

2

3 x−length (m)

4

Fig. 9.7. A circular trajectory on a level plane

5

9 A Behavior Based Control and Learning Approach to Real Robots

183

9.6 Sony Quadruped Robot Experiments The Sony quadruped robot is equipped with a camera mounted on the head. The camera has approximately 60 horizontal and 40 vertical fields of view. The head has three DOFs, namely, pan, tilt, and roll. The robot has three gyros for measuring body postures, namely, roll, pitch, and yaw. There is an infrared range finder mounted in the head. Its locomotion is implemented by four legs, each of which has three DOFs. The robot has one embedded 64 bits R4000-series microprocessor with over 100 MIPS. To evaluate the performance, the environment provide payoffs to robots with a certain accuracy to improve the learning efficiency. A global monitor is set up in this research (see figure 9.8), which includes an overhead camera, a desktop computer, and a visual tracking software, to provide an external judgement for the interaction between the robot and its environment. The monitor feeds the position information of the robot and the ball to the robot. Then, the robot can autonomously test its control strategies and evaluate the results.

Fig. 9.8. A learning environment for a Sony quadruped robot

The goals are centrad on both ends of the field. Six unique colored beacons are placed around edges of the pitch, with one at each corner and one on each side of the halfway line. The monitor recognizes the robot and the ball according to their colors. Through the image processing, the monitor updates their positions continuously. The robot can ask for the information at any time during its learning process. The communication is achieved through the Internet where the monitor system acts as a server and the robot acts as a client. The server provides the global information to the client when the client has a request. Since the robot only evaluates its performance at the end of one run, there is no significant effect of the communication on the learning process. All five behaviors defined in section 9.2 are used. Four physical events are defined for a robot to play an attacker role, • Is the ball found? (bv1 ): It simply compares the ball confidence value with a threshold.

184

D. Gu, J. Liu, and H. Hu

• Is the ball near enough? (bv2 ): since the robot always tracks the ball, its tilt angle and pan angle show if the ball is just in the front of the robot. • Is the robot behind the ball? (bv3 ): it depends on the distance and the angle between the ball and the robot, and the angle between the robot and the goal. • Is the robot aligned with the goal? (bv4 ): it again depends on the information same as ps3, but with more accurate values. All experiments are conducted in five situations, each of which has different setting for initial ball position and initial robot position. Totally 30 trails are conducted. The initial Q table is randomly initialized. The Q-learning is used to find a final Q-table. Finally the robot is run again based on the final Q table to evaluate the performance. In the Q-learning algorithms, the ε -greedy algorithm used ε = 1% to explore the action space, i.e. 99% of behaviors were selected by maximizing Q values and only 1% of behaviors were selected randomly. The discount factor was selected as γ = 0.9. The larger the discount factor, the more the learning exploits the Q value predictions. The learning rate of the Q values was selected as α = 0.1. Figure 9.9 shows the increasing trend of average rewards received. Several decreasing points in the curve reflected the exploration mechanism. The number of policy changes was also decreased as shown in figure 9.10, indicating that the learning was going toward the steady results. 2.1 1.8

Rewards

1.5 1.2 0.9 0.6 0.3 0 1

3

5

7

9

11

13

15 17 Trials

19

21

23

25

27

29

21

23

25

27

29

Fig. 9.9. Average rewards 100

Policy changes

80 60 40 20 0 1

3

5

7

9

11

13

15 17 Trials

19

Fig. 9.10. Number of policy changes

9 A Behavior Based Control and Learning Approach to Real Robots

(a)

185

(b) Fig. 9.11. Two testing scenarios

The two successful scenarios after the robot learning are displayed in figure 9.11. The dark curves represent the robot’s trajectories and the light curves represent the ball’s trajectories. The two curves met together at some places. It means the robot moved near to the ball at the meeting places. Then, the ball was kicked and moved away from the robot. So the curves were separated after the meeting places. During the whole processes, the robot can move to the ball and manoeuvre the ball into the goal. It demonstrates that the robot acquired the state planning strategy after the learning.

9.7 Conclusion To design a control and learning system for autonomous quadruped robots, this chapter has examined a behavior-based control and learning approach. The methodologies make contributions for building successful autonomous robots with reactive behaviors, deliberative planning and learning abilities. A layered control architecture for autonomous quadruped robots consists of three layers and they are connected hierarchically. The bottom layer is called the locomotion layer that is directly related to the basic movements. Based on the locomotion skills, the functional behaviors are collected in the middle layer - behavior layer. The top layer is called the cognition layer in which the deliberative reasoning for a task is established. Fuzzy logic is well known for its ability to handle uncertainty and inaccuracy. Using Fuzzy logic to design robot behaviors has been explored in the applications. The parameterized fuzzy logic also provide an opportunity to learn robot behaviors. The planning problem is viewed as a behavior co-ordination problem and formulated as a MDP to accommodate the learning ability. The state space in the high-level layer

186

D. Gu, J. Liu, and H. Hu

is based on the definition of the feature states that are physical events of the interaction between robots and their environment. The state space in the low-level is based on the map that directly consists of sensor readings.

References 1. Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: A survey. Journal of AI Research 4, 237–285 (1996) 2. Arkin, R.C. (ed.): Behavior-based Robotics. MIT Press, Cambridge (1998) 3. Brooks, R.: A robust layered control system for a moible robot. IEEE J. Robotics and Automation 2(1), 14–23 (1986) 4. Saffiotti, A.: Fuzzy logic in autonomous robotics: Behaviour coordination. In: Proceedings of the 6th IEEE Int. Conf. on Fuzzy Systems, pp. 573–578 (1997) 5. Huber, M.: A hybrid architecture for adaptive robot control. Ph.D. dissertation, University of Massachusetts (2000) 6. Stone, P.: Layered Learning in Multi-Agent Systems. PhD Thesis, Carnegie Mellon University (1998) 7. Angelov, P., Filev, D.: An approach to online identification of takagi-sugeno fuzzy models. IEEE Transactions on Systems, Man and Cybernetics- Part B 34(1), 484–498 (2004) 8. Xu, W.L., Tso, S.K.: Sensor-based fuzzy reactive navigation of a mobile robot through local target switching. IEEE Transactions on Systems, Man, and Cybernetics- Part C: Applications and Reviews 29(3), 451–459 (1999) 9. Vadakkepat, P., Miin, O.C., Peng, X., Lee, T.H.: Fuzzy behavior-based control of mobile robots. IEEE Transactions on Fuzzy Systems 12(4), 559–565 (2004) 10. Rusu, P., Petriu, E.M., Whalen, T.E., Cornell, A., Spoelder, H.J.W.: Behavior-based neurofuzzy controller for mobile robot navigation. IEEE Transactions on Instrumentation and Measurement 52(4), 1335–1340 (2003) 11. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. Dietterich, T. (ed.). The MIT Press, Cambridge (1998) 12. Watkins, C.J.C.H.: Learning from delayed rewards. Ph.D. dissertation, Cambridge University (1989) 13. Gumstix Co., Gumstix-way Small Computing (2005), http://www.gumstix.com

10 Incremental Acquisition of Neural Structures through Evolution Yohannes Kassahun, Jan Hendrik Metzen, Mark Edgington, and Frank Kirchner Robotics Group, University of Bremen, Robert-Hooke-Str. 5, D-28359 Bremen, Germany [email protected], jan [email protected], [email protected], [email protected]

Abstract. In this chapter we present a novel method, called Evolutionary Acquisition of Neural Topologies (EANT), for evolving the structures and weights of neural networks. The method uses an efficient and compact genetic encoding of a neural network into a linear genome that enables a network’s outputs to be computed without the network being decoded. Furthermore, it uses a nature inspired meta-level evolutionary process where new structures are explored at a larger timescale, and existing structures are exploited at a smaller timescale. Because of this, the method is able to find minimal neural structures for solving a given learning task.

10.1 Introduction A meaningful combination of the principles of neural networks, reinforcement learning, and evolutionary computation is important in designing agents that learn and adapt to their environment through interaction [21, 35]. These principles can be combined to form an evolutionary reinforcement learning system in which each component plays a significant role. Neural networks are useful for evolving the control system of an agent [28]. They provide a straightforward mapping between sensors and actuators, which enables them to directly represent the policy (control) or the value function to be learned. Reinforcement learning is useful as a type of learning where the agent is not told directly what to do, but is fed with a signal (i. e. reward) that measures the quality of executing an action in a given state [38]. The purpose of the agent is to act optimally in its environment, such that its rewards are maximized. Learning through interaction, of which reinforcement learning serves as a good example, underlies nearly all the principles of intelligence [30]. Reinforcement learning plays, therefore, a very important part in an evolutionary reinforcement learning system. Like neural networks, evolutionary algorithms [15, 24, 32] are inspired from biology. Populations of organisms adapt to their particular environmental conditions through evolutionary selection (survival of the fittest), and due to variation among the organisms. From these principles of adaptation in nature, it is possible to derive a number of concepts and strategies for solving learning tasks, and to develop optimization strategies for artificially intelligent systems. D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 187–208. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

188

Y. Kassahun et al.

Fig. 10.1. Evolutionary reinforcement learning system. The agents (in which the neural networks are embedded) are evaluated in the environment, and their fitness values are returned to the evolutionary algorithm as rewards.

An evolutionary reinforcement learning system combining the principles of neural networks, reinforcement learning, and evolutionary algorithms is shown in Figure 10.1. The evolutionary algorithm works with genotypes representing neural networks to be evaluated in a given environment. Each neural network is evaluated and assigned a given fitness value (i. e. reward). The agents are improved by the evolutionary algorithm through the use of genetic operators. This process is repeated until a given number of generations have passed, or until an agent is found that solves the specified task. The method presented in this chapter uses an evolutionary reinforcement learning system, and is closely related to the works of Angeline et al.[1] and Stanley [35]. As in the work of Angeline et al., the method uses a type of parametric mutation that comes from evolution strategies and evolutionary programming, in which adaptive step sizes are used for optimizing the weights of the neural networks. Similar to the work of Stanley, the method starts with minimal structures, which are complexified along the evolution path. Despite sharing these two properties, the method has the following important features which distinguish it from previous works in the evolution of neural networks: 1. It introduces a genetic encoding called Common Genetic Encoding (CGE) that handles both direct and indirect encoding of neural networks with in the same theoretical framework. The encoding has important properties that makes it suitable for evolving neural networks: (1) It is complete in that it is able to represent all types of valid phenotype networks. (2) It is closed, i.e. every valid genotype represents a valid phenotype. Similarly, the encoding is closed under genetic operators such as structural mutation and crossover. These properties have been formally proven in [19].

10 Incremental Acquisition of Neural Structures through Evolution

189

2. For evolving the structures and weights of neural networks, a nature inspired metalevel evolutionary process is used, where exploration of structures is executed at a larger timescale, and exploitation of existing structures is done at a smaller timescale. The chapter is organized as follows. A summary of previous works in the area of evolution of neural networks is given in section 10.2. A more detailed discussion of the method presented in this work is given in section 10.3. Practical applications and evaluation of the method are presented in section 10.4. Finally, in section 10.5, a summary of the presented method and outlook are given.

10.2 Review of Related Works The evolution of neural networks can be divided into two major categories: the evolution of connection weights, and the evolution of both structure (i. e. network topology) and connection weights. In the first category, the structure of neural networks is fixed and is determined by the domain expert before the evolution begins. In the second category, both the structure and the connection weights are determined automatically by the evolutionary process. The work presented in this chapter falls into the second category. In this section, we review works in the second category that have been applied to reinforcement learning tasks. These works can be divided into two major groups, depending on whether their approach uses direct or indirect genetic encodings. For a detailed review of works in the evolution of neural networks, see [42]. 10.2.1

Direct Genetic Encoding Approaches

Approaches that use a direct encoding scheme specify explicitly every connection and node that will appear in the phenotype. Angeline et al. developed a system called GNARL (GeNeralized Acquisition of Recurrent Links) that uses only structural mutation for modifying the topology, and parametric mutation for modifying the weights [1]. The system is based on evolutionary programming, where the crossover operator is not used as a search operator. The system tries to maintain the behavior of the network, avoiding a radical jump from parent to offspring. This is done by initializing new links with zero weight, leaving the behavior of the modified network unchanged, and by adding hidden nodes to the network without any incident connections. The main problem this method faces is that genomes may end up having many extraneous disconnected structures that have no contribution to the solution. In contrast to GNARL, the Neuroevolution of Augmenting Topologies (NEAT) [35] evolves both the structure and weights of neural networks using both crossover and mutation operators. It starts with networks of minimal structure and increases their complexity along the evolution path. Every node and connection of the phenotype is encoded by the genotype. The algorithm keeps track of the historical origin of every gene that is introduced through structural mutation. The history is used by a specially designed crossover operator to match up genomes encoding different network topologies, and to create a new structure that combines the overlapping parts of the two parents as

190

Y. Kassahun et al.

well as their different parts. Structural discoveries of the evolutionary process are protected by niching (speciation). The speciation in NEAT is achieved by explicit fitness sharing, where organisms in the same species share the fitness of their niche. Unlike GNARL, NEAT does not use self-adaptation of mutation step-sizes. Each connection weight is perturbed with a fixed probability by adding a floating number chosen from a uniform distribution of positive and negative values. 10.2.2

Indirect Genetic Encoding Approaches

In indirect encoding, one specifies rules that are used in constructing the phenotype. Every connection and node in the network can be derived from the genome but is not specified explicitly. Kitano’s grammar based encoding of neural networks uses Lindenmayer systems (L-systems) [25] to describe the morphogenesis of linear and branching structures in plants [23]. L-systems are parallel string rewriting systems that rewrite a starting string into a new string by applying a set of production rules in parallel to all the symbols in the string. Sendhoff et al. extended Kitano’s grammar encoding with their recursive encoding of modular neural networks [33]. Their system provides a means of initialization of the network weights. Networks are trained using the standard back-propagation algorithm, and the encoding itself is variable and optimized on a larger timescale. Both Kitano and Sendhoff have used their systems to evolve feed-forward networks. In Kitano’s grammar based encoding, there is no direct way of representing the connection weights of neural networks in the genome. Gruau’s Cellular Encoding (CE) method is a language for local graph transformations that controls the division of cells which grow into an artificial neural network [13]. Through cell division, one cell, the parent cell, is replaced by two cells, the child cells. During division, a cell must specify how the two child cells will be linked. The genetic representations in CE are compact because genes can be reused multiple times during the development of the network. This saves space in the genome because not every connection and node need to be explicitly specified in the genome. Defining a crossover operator for CE is still difficult, and it is not easy to analyze how crossover affects the subfunctions in CE since they are not represented explicitly. Moreover, CE suffers from the same problem as that of Kitano’s grammar based encoding since there is no direct way of representing the connection weights of neural networks in the genome. Vaario et al. have developed a biologically inspired neural growth, which is based on diffusion field modeling combined with genetic factors for controlling the growth of the network [40]. The neural structures are grown in either a two or a three dimensional grid, resulting in a two or three dimensional tree-based neural structure, respectively. One weak point of this method is that it cannot generate networks with recurrent connections or networks with connections between neurons on different branches of the resulting tree structure. Nolfi and Parisi modelled biological development at the chemical level using a reaction-diffusion model [29]. Diffusion is modelled at the level of axon growth, and reaction is modelled as the interaction between axon and cell bodies. This method utilizes growth to create connectivity without explicitly describing each connection in the phenotype. Axons which do not reach other cells during development are pruned. The

10 Incremental Acquisition of Neural Structures through Evolution

191

complexity represented by the genome is limited since every neuron is directly specified in the genome. Other works in indirect encoding have simulated genetic regulatory networks (GRNs) in biology, where genes produce signals that either activate or inhibit other genes in the genome. The interaction of all genes forms a network that produces a phenotype. Representative works that use GRNs include the works of Dellaert and Beer [7], Jakobi [17], Bongard and Pfeifer [6], and Bentley and Kumar [5].

10.3 Evolutionary Acquisition of Neural Topologies (EANT) This section presents an evolutionary reinforcement learning system named Evolutionary Acquisition of Neural Topologies (EANT). It is a novel method that is suitable for learning and adaptation to the environment through interaction. The system evolves both the structures and weights of neural networks. With respect to the goal of selforganizing learning machines that start from minimal specifications and rise to great sophistication, EANT starts with neural networks of minimal structures, and increases their complexity along the evolution path. A defining characteristic of EANT is its ability to influence the ratio of exploration to exploitation that occurs during the evolutionary process. Although EANT can be used with any arbitrary genetic encoding that can represent neural networks, we have chosen to use the Common Genetic Encoding (CGE), an encoding with features that make it well suited for use in the evolution of neural networks. CGE will be presented in what follows, including a description of three genetic operators defined on CGE-genomes. Building on this, two methods for computing the output values of a CGE-genotype for a given input will be discussed. After this, a method will be presented which can be used to generate an initial population with a high amount of diversity, yet having low structural complexity. Finally, the exploitation/exploration scheme used in EANT will be discussed. 10.3.1

Common Genetic Encoding (CGE)

A flexible encoding method enables one to design an efficient evolutionary method that can evolve both the structures and weights of neural networks. A genome in CGE is designed by taking this fact into consideration. The genome in CGE is a linear genome consisting of genes (nodes) that can take different forms (alleles). The forms that can be taken by a gene can either be a neuron, or an input to the neural network, or a jumper connecting two neurons. The jumper genes are introduced by one of the genetic operators, the structural mutation (compare Subsection 10.3.2), along the evolution path. A jumper gene can either encode a forward or a recurrent connection. Jumper genes encoding a forward connection are not allowed to encode connections starting from a neuron with lower depth1 and ending at a neuron with higher depth. On the other hand, a jumper gene encoding a recurrent connection represents a connection between neurons having the same depth, or a connection starting from a neuron at a lower depth 1

The depth of a vertex is defined as the minimal topological distance (i.e. minimal number of connections to be traversed) from an output vertex of the network to the vertex itself.

192

Y. Kassahun et al.

(a)

(b)

Allele + ID N 0 N 1 N 3 I x I y I y N 2 JF 3 I x I y JR 0 Weight 0.6 0.8 0.9 0.1 0.4 0.5 0.2 0.3 0.7 0.8 0.2 Number of inputs 2 2 2 – – – 4 – – – – (c) Fig. 10.2. An example of encoding a neural network using a CGE. (a) The neural network to be encoded, with one forward and one recurrent jumper connection. (b) The neural network interpreted as a tree structure, where the jumper connections are considered as terminals. (c) The linear genome encoding the neural network shown in (a). In the linear genome, N stands for a neuron, I for an input to the neural network, JF for a forward jumper connection, and JR for a recurrent jumper connection. The numbers beside N represent the global identification numbers of the neurons, and x or y represent the labels of the inputs encoded by the input genes.

and ending at a neuron at a higher depth. Every node in a linear genome has a weight associated with it. The weight encodes the synaptic strength of the connection between the node coded by the gene and the neuron to which it is connected. Moreover, every node can save the results of its current computation. This is useful since the results of signals at recurrent links are available at the next time step. In addition to the synaptic weight, a neuron node has a unique global identification number and stores the number of input connections associated with it. A jumper node has also additionally a global identification number, which stores the neuron to which it is connected. An example of a linear genome encoding a neural network is shown in Figure 10.2. The linear genome has some interesting properties that makes it useful for evolution of the structure of neural networks [19]. Assume that integer values are assigned to the nodes of a linear genome encoding a neural network such that the integer values show the difference between the number of outputs of the nodes and the number of arguments of the nodes (inputs to the nodes). Note that every node in the linear genome has only one output. If a node is an input to the neural network, the integer assigned to it is 1 since an input to a neural network has only one output and no arguments (inputs) at all. An integer value of 1 is also assigned to the forward and recurrent jumper nodes since they are sources of signals from other neurons in the neural network encoded by the linear genome. A neuron node will take an integer value which is one minus the number of inputs to the neuron. In CGE, since there is no neuron without an input, the

10 Incremental Acquisition of Neural Structures through Evolution

193

Fig. 10.3. An example of the use of assigning integer values to the nodes of the linear genome. The linear genome encodes the neural network shown in Figure 10.2. The numbers in the square brackets below the linear genome show the integer values assigned to the nodes of the linear genome. Note that the sum of the integer values is one showing that the neural network encoded by the linear genome has only one output. The shaded nodes form a sub-network. Note also that the sum of the integer values assigned to a sub-network is always one.

maximum value of an integer assigned to a neuron node is zero. This is true for all neurons with only one input. One of the important properties of a linear genome is that the sum of the integer values assigned to the nodes in a linear genome encoding a neural network is the same as the number of outputs of the neural network. After assigning the integer values to the nodes of the linear genome, it is possible to detect a sub-genome (sub-network) of a linear genome. A sub-genome in CGE is defined as a collection of nodes starting from a neuron node and ending at a node where the sum of integer values assigned to the nodes between and including the start neuron node and the end node is one. An example is shown in figure 10.3. In [19], Kassahun et al. have discussed the representation capabilities of CGE. They proved that the linear genome is complete in that it can be used to represent any type of neural network. It is also a compact encoding of neural networks since the length of the linear genome is the same as the number of synaptic weights in the neural network. Moreover, the encoding scheme used is closed. An encoding scheme is said to be closed if all genotypes produced are mapped into a valid set of phenotype networks [3, 18]. 10.3.2

Genetic Operators

In this subsection, we present the three genetic operators defined on CGE genotypes. The first one, the parametric mutation operator perturbs only the synaptic weights, the network structure itself remains unchanged. In contrast, the structural mutation operator changes the network structure by adding genes to the genotype but does not change the network’s weights. The last genetic operator, the structural crossover takes two genomes as inputs (the parents) and produces one new genome (the offspring) which comprises the properties of its parents. Kassahun et al. [19] have proven that linear genomes are closed under parametric and structural mutation. A genome is said to be closed under a genetic operator if the newly generated structure always maps onto a valid phenotype network. Furthermore, CGE is also closed under the structural crossover operator. We continue with a detailed description of the three genetic operators. Parametric Mutation Parametric mutation is used for optimization of the weights of a given structure. It is accomplished by perturbing the synaptic weights of the networks according to the

194

Y. Kassahun et al.

uncorrelated mutation in evolution strategy or evolutionary programming [9, 31]. In addition to the associated weight wi , each node in a linear genome has an associated mutation step size or learning rate2 σi . The mutation mechanism is specified as follows [8]:  σi = σi eτ N(0,1)+τ Ni (0,1) , (10.1) wi = wi + σi Ni (0, 1) , (10.2)  √ √ where τ  = 1/ 2n and τ = 1/ 2 n, and N (0, 1) as well as Ni (0, 1) are random numbers drawn from a Gaussian distribution of zero mean and unity standard deviation. If one wants to employ a global learning rate σ , one can replace Equation 10.1 by

σ  = σ eτ N(0,1)

(10.3)

There are two main advantages of parametrically mutating the synaptic weights of neural networks in the same fashion as is done in evolution strategies or evolutionary programming: 1. Offspring created by mutation remain within a locus of similarity to their parents [1]. 2. Self-adaptation of mutation step sizes of learning rates is inherent in both evolution strategies and evolutionary programming [31, 9]. Structural Mutation The structural mutation used by CGE adds a forward or a recurrent jumper connection between neurons, or adds a whole new subgenome to the linear genome. If a subgenome is added, this genome consists of a neuron node followed by an arbitrary number of input or jumper genes. The initial weight of a newly added jumper connection or the initial weight of the first node of a newly added sub-network are set to zero so as not to disturb the behavior of the neural network. When applying the structural mutation, one neuron node from the genome is chosen randomly and will be mutated. This mutation can be either the insertion of a jumper gene or of a whole subnetwork after the neuron node. Which mutation is actually performed is determined randomly as well as the source nodes for jumper genes. If a forward jumper gene is added to the genome, it is taken care that this does not result in a closed cycle of forward jumper genes since this would yield in problems during the evaluation of the network later on (infinite loops). Figure 10.4 shows an example where the structural property of a neuron node N3 is changed through the structural mutation: A new forward jumper gene is added and the number of input of N3 is increased by one. The new forward jumper stores the id of its source node N2. In the corresponding phenotype, this results in an additional edge going from N2 to N3. 2

It is possible to enforce that all genes of a linear genome have the same learning rate (called global learning rate).

10 Incremental Acquisition of Neural Structures through Evolution

195

Fig. 10.4. An example of structural mutation. A forward jumper gene connecting the node N2 with the node N3 is added and the number of inputs of N3 is increased by one.

Structural Crossover The third genetic operator defined is the Structural Crossover operator. This operator exploits the fact that structures which originate from the same ancestor structure have some parts in common. By aligning the common parts of two randomly selected structures, it is possible to generate a third structure which contains the common and disjoint parts of the two mother structures. The resulting structure formed in this way maps to a valid phenotype network. This type of crossover was introduced and used by Stanley [35]. An example is given in Figure 10.5. The number of inputs to a neuron node which is common to the parent structures is updated using n(s1 × s2 ) = n(s1 ) + n(s2 ) − n(s1 ∩ s2 ),

(10.4)

where n(s1 × s2 ) is the number of inputs to the neuron node in the offspring, n(s1 ∩s2 ) is the number of inputs to the neuron node which are common to both structures, and n(s1 ) and n(s2 ) are the number of inputs to the neuron node in the parent structure s1 and s2 , respectively. Furthermore, if the parents have some genes in common whose weights differ, one of this weights is chosen randomly for the offspring. An other option would be to take the mean of these two values as new weight. 10.3.3

Evaluating a Linear Genome

In CGE, there are two methods of evaluating a linear genome. In the first method, one decodes the linear genome into a neural network and evaluate this neural network

196

Y. Kassahun et al.

Fig. 10.5. Performing crossover between two linear genomes. The genetic encoding is closed under this type of crossover operator since the resulting linear genome maps to a valid phenotype network. The weights of the nodes of the resulting linear genomes are inherited randomly from both parents.

directly. In the second method, it is not necessary to decode the linear genome into a neural network but one can use the linear genome directly to evaluate the neural network represented by it. The second method emphasizes the fact that it is not always necessary to create a separate phenotype structure from genotype by some sort of embryogeny [4]. In other words, it is not always necessary to decode the linear genome into a neural network for the purpose of evaluating the network encoded by it. For computing the output of the neural network without decoding the genome, we use a first in last out stack and the following rules: 1. Start from the right most node of the linear genome. 2. Traverse the linear genome from right to left in computing the output of the neural network:

10 Incremental Acquisition of Neural Structures through Evolution

197

Fig. 10.6. An example of evaluating a linear genome without decoding the neural network encoded by it. The linear genome encodes the neural network shown in Figure 10.2. For this example, the current values of the inputs to the neural network, x and y, are both set to 1. In the example, all neurons have a linear activation function of the form g(a) = a, where a is the weighted linear combination of the inputs to a neuron. The overlapped numbers above the linear genome show the status of the stack after computing the output of a node. The numbers in brackets are the weights associated with the nodes.

a) If the current node is an input node, push its current value and the weight associated with it onto the stack. b) If the current node is a neuron node with n inputs, pop n values with their associated weights from the stack and push the result of computation with its associated weight onto the stack. The output of a neuron node j is computed using   n

aj = g

∑ wi ai

,

(10.5)

i=1

where ai are the values popped from the stack and wi are their associated weights. g (.) is the transfer function of the neuron node. c) If the current node is a recurrent jumper node, get the last value (i. e. the output of the node in the last evaluation of the neural network) of the neuron node identified by the source unique id stored in this recurrent jumper. Then push the value obtained with the weight associated with the jumper node onto the stack. d) If the current node is a forward jumper node, first copy the subgenome starting from the neuron whose global identification number is the same as the global identification number of the forward jumper node. Then compute the response of the subgenome in the same way as that of the linear genome. Finally, push the result of computation with the weight associated with the forward jumper node onto the stack. 3. After traversing the genome from right to left completely, pop the resulting values from the stack. The number of the resulting values is the same as the number of outputs of the neural network encoded by the linear genome. Figure 10.6 shows an example of evaluating a linear genome encoding the neural network shown in Figure 10.2. As can be seen from the figure, one does not need to decode the neural network in order to evaluate it. The evaluation of a linear genome discussed above is equivalent to the evaluation of a decoded neural network represented by the genome: Let xˆi be a neuron with incoming

198

Y. Kassahun et al.

connections from the inputs xˆ1 , ..., xˆk and the neurons xˆk+1 , ..., xˆl , and the incoming recurrent connections from neurons xˆl+1 , ..., xˆm . For an arbitrarily chosen transfer function g, the output ai (t) of the neuron xˆi at time step t is computed using k

ai (t) = g( ∑ w j I j (t) + j=1

l

∑

m

w j a j (t) +

j=k+1

∑

w j a j (t − 1)),

(10.6)

j=l+1

where the values of I j (t) represent the inputs of the neural network and a j (t − 1) is the output of neuron j at time step t − 1. 10.3.4

Generating the Initial Population

Traditionally, there are two methods used for the generation of initial populations of genomes. These are the grow and full methods [4]. In EANT, we used a third approach which enforces that all genomes originate from the same ancestor genome. This “proto” linear genome contains only output neuron nodes and input nodes. It connects all inputs directly with all outputs3 without having neurons in a hidden layer. One requirement of the initial population of a neuroevolutionary method is that it is sufficiently diverse while its members (the individuals in the population) are still fairly simple. To achieve this, the following approach has been applied in EANT: For a population of size n, n copies of the “proto” individual are created. Each of these copies is then separately mutated by the structural and parametric mutation operator m times. The bigger m is, the more diverse the initial population gets while losing more and more the minimality of its members. All n individuals created with these procedure have still some parts in common, since the mutation operators do never erase a part of an ancestor. Note that this procedure can easily be adjusted to work together with techniques like speciation as discussed by Metzen [26]. 10.3.5

Exploitation and Exploration of Structures

As discussed in Subsection 10.3.4, EANT starts with networks of minimal structure whose initial complexity is controlled the by the choice of m, the number of initial mutations to be applied. The evolution itself is divided into two alternating processes: The exploitation process which is occurring at a smaller timescale and the exploration process occurring at larger timescale. In the exploitation process, the weights of existing structures are optimized. The exploitation process itself is subdivided into different generations. In each generation, first of all, the fitness of all individuals is determined. Based on this, a mating pool consisting of n < n individuals is created using well known selection methods like Stochastic Universal Sampling [2]. Based on this mating pool, the population of the next generation is created using the parametric mutation operator as only active genetic operator. Thereby, the underlying network structures remain unchanged while only the individuals whose weights perform good for the given structure survive. After a certain number 3

The number of outputs as well as number of inputs of the neural network needs to be specified for each task anew.

10 Incremental Acquisition of Neural Structures through Evolution

Trajectory in weight space

199

t N0 W=5.0

Ix W=5.0

Iy W=5.0

5.0

t+1

0

w(Iy)

w(Ix)

5.0

5.0

x

y

w(N0)

Fig. 10.7. The weight trajectory of the linear genome shown at the right side while being exploited. The quantities t and t + 1 are time units with respect to the larger timescale. The weights of the existing structures are optimized between two consecutive time units with respect to the larger timescale. The point clouds at t and t + 1 show populations of individuals from the same species.

of generations (usually between five and ten), the weights of the existing structures are fairly well optimized. An example for the change of the weights for a given structure during the exploitation process is shown in Figure 10.7. The exploration process occurs at a larger timescale and is used to introduce new structures. In each step of exploration, first the weights of all existing structures are optimized using the exploitation process. Then, selection is performed on this optimized genomes. Since the weights of all structures were optimized independently, the performance of these individuals in the exploration step should not depend too strong on the actual weights, rather the actual fitness should be a good estimate for the optimal fitness achievable with this structure. After the selection, the population for the next exploration step is created: New structures are formed from existing ones using the structural mutation and structural crossover operator. Since neither the structural mutation nor the structural crossover operator erase parts of a genome but only add new parts, there is a gradual increase in the complexity of the structures along the evolution. This allows EANT to search for a solution starting from a minimum structural complexity specified by the domain expert. The search stops when a structure with the necessary minimal structure that solves a given task is obtained. In EANT complexification is an emergent property that depends on the task to be solved.

10.4 Practical Applications In this section the performance of EANT when applied to practical problems is examined. In the first experiment, the problem of learning to move forward using a singlelegged robotic insect is considered. The second experiment tests the efficiency of EANT

200

Y. Kassahun et al.

on the standard benchmark problem of balancing two poles attached to a moving cart. Further experiments conducted with EANT include “Learning the Dynamic Model of a Robot Manipulator” [20] and a performance evaluation of EANT in the RoboCup Keepaway Benchmark, a challenging multi-agent benchmark with a continuous and noisy state space [26]. 10.4.1

Crawling Robotic Insect

In this experiment, the crawling robotic insect [22] is used. It has one arm having two joints where the joints are controlled by two servo motors. It has also a touch sensor which detects whether the tip of the arm is touching the ground or not. It is used for reinforcement learning task where the agents learns to move forward as fast as possible through interaction with the environment. The schematic diagram of the robot is shown in Figure 10.8.

Fig. 10.8. The crawling robotic insect. The robot has one arm with two joints and a touch sensor for detecting whether the tip of the arm is touching the ground or not.

The robot has bounded continuous and discrete state variables. The continuous state variables are the joint angles and the discrete state variable is the state of the touch sensor. The controller observes the joint angles and the state of the touch sensor. Depending on the state it perceives, the controller is expected to change the angles of the joints appropriately so that the robot can move forward as fast as possible. The first joint angle θ1 is bounded between 55 deg and 94 deg, and the second joint angle θ2 lies in the range [−34 deg, 135 deg]. For both of the joints, the angles are measured from the vertical as shown in Figure 10.8. The angle ranges are chosen so that they are equivalent to the angle ranges chosen by Kimura and Kobayashi. They measured the first joint angle from the horizontal and the second joint angle from the first link. The touch sensor φ takes the value 0 for non-touch state and 1 for touch state. Let the coordinates of the first and the second joints be (x0 , y0 ) and (x1 , y1 ), respectively and let the coordinate of the tip of the arm be (x2 , y2 ). The state of the robot at each time step t = 0, 1, . . . is given by st = (x0 , y0 , x2 , y2 , θ1 , θ2 , φ ). Since the coordinate (x1 , y1 ) can be calculated given a state s, it is not listed in the definition of the state of the robot. The state transition of the system is governed by equations (10.7) and (10.8). If the tip of the arm is not touching the ground (φ (t) = 0), then the state transition equation is given by

10 Incremental Acquisition of Neural Structures through Evolution

θ1 (t + 1) = θ2 (t + 1) = x0 (t + 1) = y0 (t + 1) = x2 (t + 1) = y2 (t + 1) =

θ1 (t) + δ1 θ2 (t) + δ2 x0 (t) , y0 (t) x0 (t + 1) + l1 sin θ1 (t + 1) + l2 sin θ2 (t + 1) y0 (t + 1) + l1 cos θ1 (t + 1) − l2 cos θ2 (t + 1)

201

(10.7)

and if the tip of the arm is touching the ground (φ (t) = 1), then the state transition equation takes the form

θ1 (t + 1) = θ2 (t + 1) = x2 (t + 1) = y2 (t + 1) = x0 (t + 1) = y0 (t + 1) =

θ1 (t) + δ1 θ2 (t) + δ2 x2 (t) . y2 (t) x2 (t + 1) − l2 sin θ2 (t + 1) − l1 sin θ1 (t + 1) l2 cos θ2 (t + 1) − l1 cos θ1 (t + 1)

(10.8)

The quantities δ1 and δ2 are the outputs of the neural controller to be designed, and l1 and l2 are the lengths of the first and the second link. The first link is between the first joint and the second joint while the second link is between the second joint and the tip of the arm. For the experiment, l1 = 34 cm and l2 = 20 cm are chosen. The first joint is located at right upper corner of the rectangular body of the robotic insect which has a height of 18 cm and width of 32 cm. A trial contains 50 time steps and at the beginning of a trial the robot is placed at the origin. The fitness function used to evaluate a neural controller is given by 1 N f = ∑ (x0 (t) − x0(t − 1)), (10.9) N t=1 where the difference x0 (t) − x0 (t − 1) is the velocity of the system at time t in the direction of the x−axis and f is the average velocity of the robot for a trial. The number of time steps used per trial is represented by N. Tsuchiya et al. applied their policy learning by genetic algorithm using importance sampling (GA-IS) for learning to move forward. They defined a three dimensional vector X = (x1 , x2 , x3 ) for representing the state space [39]. The dimensions of the state space is made up of the joint angles and the state of the touch sensor. The policy used in their experiment is a 7 dimensional feature vector F = [x1 , x2 , x3 , x4 (= 1 − x1) , x5 (= 1 − x2), x6 (= 1 − x3), 1.0]. A weight vector Λ = (λ1,i , λ2,i , λ3,i , λ4,i , λ5,i , λ6,i , λ7,i ) is used to select the action ai (t) from normal distribution with mean value μi = 1/(1 + exp(− ∑6k=1 λk,i xk )) and standard deviation σi = 1/(1 + exp(−λ7,i )) + 0.1. If the selected action is out of range then it is resampled. The number of the policy parameters is 14 and hence the search space for the genetic algorithm has 14 dimensions. In the experiment, the structure shown in Figure 10.9 (a) containing two output neurons connected to three input nodes (θ1 , θ2 , φ ) is used as the initial controller. The best controller shown in Figure 10.9 (b) is found after running EANT. Note that the best controller is more complex than the initial structure. Figure 10.9 (c) shows the waveforms of the joint angles and the touch sensor for the first 20 time steps as the robot moves forward and being controlled by the best controller.

202

Y. Kassahun et al. 2

θ

1

Angle in radian

1.5

1

φ

θ2

0.5

0

−0.5

−1 0

1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20

Forward steps taken by the robot

(a)

(b)

(c)

Fig. 10.9. Learning to move forward. (a) The initial structure. (b) The best controller found by EANT that enables the robot to move forward. (c) The waveforms of the joint angles and the touch sensor as the robot moves forward.

The method introduced by Tsuchya et al. (GA-IS) needed on the average 10000 interactions with the environment for learning to move forward. We have run EANT 50 times and obtained on the average 3520 interactions for learning the task. As compared to the GA-IS, EANT has reduced the number of interactions with the environment necessary to learn to move forward. The reduction in the number of interactions is due to the direct search for an optimal policy in the space of policies, starting minimally and increasing the complexity of the neural network that represents the policy. 10.4.2

Pole Balancing

The inverted pendulum or the pole balancing system has one or several poles hinged to a wheeled cart on a finite length track. The movement of the cart and the poles are

Center Fig. 10.10. The double pole balancing problem. The poles must be balanced simultaneously by applying a continuous force F to the cart. The parameters x, θ1 and θ2 are the offset of the cart from the center of the track and the angles from the vertical of the long and short pole, respectively.

10 Incremental Acquisition of Neural Structures through Evolution

203

constrained within a vertical plane. The objective is to balance the poles indefinitely by applying a force to the cart at regular time intervals such that the cart stays within the track boundaries. A trial to balance the poles fails if either the angle from vertical of any pole exceeds a certain threshold or the cart leaves the track. The problem has been a standard benchmark for the design of controllers for unstable systems over 30 years [34]. The first reason of using the problem as a standard benchmark is that it is a continuous real-world task that is easy to understand and visualize. Moreover, it can be performed manually by humans and implemented on a physical robot. The second reason is that it embodies many essential aspects of a whole class of learning tasks that involve temporal credit assignment [12]. The controller is expected to discover its own strategy based on the reinforcement signal it receives every time it fails to control the system. Figure 10.10 describes the double pole balancing problem for poles having unequal lengths. The equations of motion of a cart with N poles are given by [41] ˙ + ∑Ni=1 F˜i F − μc sgn (x) mc + ∑Ni=1 m˜ i   μi θ˙i 3 ¨ θi = − x¨ cos θi + g sin θi + 4li mi li  ˙  μi θi 3 2 F˜i = mi li θ˙i sin θi + mi cos θi + g sin θi 4 mi li   3 m˜ i = mi 1 − cos2 θi 4 x¨ =

(10.10)

for i = 1, . . . , N. In the equation, F is the force applied to the cart, x is the offset of the cart from the center of the track, and g is the acceleration due to gravity. The quantities mi , li , θi and μi stand for the mass, the half of the length, the angle from the vertical, and the coefficient of friction of the ith pole, respectively. The mass and coefficient of friction of the cart are denoted by mc and μc , respectively. The effective force from pole i on the cart is denoted by F˜i and its effective mass is given by m˜ i . For our benchmark double pole experiments, N = 2, mc = 1 kg, m1 = 0.1 kg, l1 = 0.5 m, l2 = 0.1l1 , m2 = 0.1m1 , μc = 5.10−4 and μ1 = μ2 = 2.10−6 . The length of the track is set to 4.8 m. The parameters are the most common choices for the double pole experiments. The dynamical system is solved using fourth-order Runge-Kutta integration with step size τ = 0.01 s. 10.4.2.1 Experimental Setup The setup used for the experiments is the same as the setup used by other researchers who tested their methods on the pole balancing problem [11, 16, 27, 36]. The controllers perceive continuous states and produce continuous control signals rather than jerky leftright or “bang-bang” signals. Two balancing configurations are used in the experiments: one having, and one lacking complete state information.

204

Y. Kassahun et al.

Double Pole Balancing with Velocities In this experiment the controller is provided with full state information (x, x, ˙ θ1 , θ˙1 , ◦ ˙ θ2 , θ2 ) and the initial state of the long pole is set to θ1 = 1 . The controller is expected to balance the poles for 105 time steps so that the angles of the poles from the vertical lie in the range [−36◦ , 36◦ ]. Each time step corresponds to 0.01s. Double Pole Balancing without Velocities In this setup, the controller observes only x, θ1 and θ2 . A fitness function introduced by Gruau et al. together with a termination criterion is used in this task [14]. The same fitness function is also used by other researchers [10, 35, 16]. The fitness function is the weighted sum of two separate fitness measurements 0.1 f1 + 0.9 f2 taken over 1000 time steps: f1 = t/1000, ⎧ ⎪ ⎨0 f2 =

⎪ ⎩

∑ti=t−100

(10.11) if t < 100

0.75 (|xi |+|x˙i |+|θ1i |+|θ˙ii |)

(10.12) otherwise,

where t is the number of time steps the pole is balanced starting from a fixed initial position. In the initial position, all states are set to zero except θ1 = 4.5 deg. The angle of the poles from the vertical must be in the range [−36 deg, 36 deg]. The fitness function defined favors controllers that can keep the poles near the equilibrium point and minimize the amount of oscillation. The first fitness measure f1 rewards successful balancing while the second measure f2 penalizes oscillations. The evolution of the neural controllers is stopped when a champion of a generation passes two tests. First, it has to balance the poles for 105 time steps starting from the 4.5 deg initialization. Second, it has to balance the poles for 1000 steps starting from at least 200 out of 625 different initial starting states. Each start state is chosen by giving each state variable (x, x, ˙ θ1 , θ˙1 , θ2 , θ˙2 ) each of the values 0.05, 0.25, 0.5, 0.75, 0.95, 0, 0, scaled to the range of input variable (54 = 625). The ranges of the input variables are ±2.16 m for x, ±1.35 m/s for x, ˙ ±3.6 deg for θ1 , and ±8.6 deg for θ˙1 . The number of successful balances is a measure of the generalization performance of the best solution. 10.4.2.2 Results Table 10.1 shows the average value of network evaluations needed by various methods in solving the double pole balancing task. For EANT, the experiments are done for 120 times for both test scenarios. From Table 10.1 one can see that EANT performs better than other algorithms that evolve both the structure and weights of a neural network (CE, NEAT). The results for EANT are better than the best algorithms which evolve both the network structure and weights of the neural networks. CMA-ES has outperformed EANT on both double pole balancing tasks. But for CMA-ES the topology (structure) of the

10 Incremental Acquisition of Neural Structures through Evolution

205

Table 10.1. The average network evaluations (trials) needed by various methods in solving the double pole balancing tasks. For CMA-ES, results for a neural network having 3 hidden nodes without a bias are shown.

Method CE [14] ESP [12] NEAT [37] CMA-ES [16] EANT

(a)

Double pole balancing Double pole balancing with velocity without velocity Evaluations Evaluations Generalization 34000 3800 3600 895 1580

840000 169466 33184 6061 15762

300 289 286 250 262

(b)

Fig. 10.11. The best controllers found by EANT for (a) double pole balancing with velocity, and (b) double pole balancing without velocity

neural network has to be chosen manually before optimizing the weights of the network. Figure 10.11 shows the best controllers found by EANT for both test scenarios. If one decodes the best structure of the last generations of sample evolutionary runs, one obtains the neural controllers shown in Figure 10.11. For double pole balancing with velocities, the controller has only one output node. Since the evolution starts with neural controllers of minimal structures, the algorithm was able to find this minimal structure for most of the experiments. For double pole balancing without velocities, the controller found has one output neuron and one hidden neuron. Both neurons have a self-recurrent connection. Once again, one can see that because of starting minimally, it is possible to obtain compact, efficient and clever solutions in the design of controllers. Our algorithm found both structures consistently for both test scenarios.

206

Y. Kassahun et al.

10.5 Summary A system that enables autonomous and situated agents to learn and adapt to the environment in which they live and operate has been developed. Self-organization is inherent in the system in that the system starts with networks of minimal structures and increases their complexity along the evolution path. This self-organization process is an emergent property of the system. CGE is a novel genetic encoding that encodes a neural network implicitly in the ordering of the elements of the linear genome representing the neural network. The linear genome enables one to compute the output values of the neural network it encodes, without needing a developmental process to transform the genotype into a phenotype. The presented genetic encoding is complete in that it can encode any type of neural network, and closed under structural mutation and a specially designed crossover operator. The crossover operator exploits the fact that neural structures that come from a common initial neural structure will have some parts in common. By aligning the common parts of two randomly selected structures, it is possible to generate a valid third structure which contains the common and disjoint parts of the two parent structures. In addition to this, a nature inspired meta-level evolutionary process is introduced that is suitable for exploring new structures incrementally and exploiting existing structures. Structural mutation and crossover operators are used to search for structures at a larger timescale, and parametric mutation and recombination operators are used to optimize the weights of the existing structures at a smaller timescale. Through introduction of neural structures by structural mutation, there is a gradual increase in the number and complexity of structures along the evolution path. Those structures that have a higher fitness evaluation survive and continue to exist in the population. This complexification process enables EANT to search for an optimal solution starting from a minimal structural complexity specified by the domain expert. Two experiments have been conducted: In both experiments, EANT has evolved policies that can deal with the complex dynamics of the system and its environment. A policy that solved the problem was found after a low number of fitness function evaluations, more quickly than than most other state-of-the-art neuroevolutionary approaches.

References 1. Angeline, P.J., Saunders, G.M., Pollack, J.B.: An evolutionary algorithm that constructs recurrent neural networks. IEEE Transactions on Neural Networks 5, 54–65 (1994) 2. Baker, J.E.: Adaptive selection methods for genetic algorithms. In: Proceedings of the 1st International Conference on Genetic Algorithms, Mahwah, NJ, USA, pp. 101–111. Lawrence Erlbaum Associates, Inc., Mahwah (1985) 3. Balakrishnan, K., Honavar, V.: Properties of genetic representations of neural architerctures. In: Proceedings of the World Congress on Neural Networks, pp. 117–146 (1995) 4. Banzhaf, W., Nordin, P., Keller, R.E., Francone, F.D.: Genetic Programming: An Introduction on the Automatic Evolution of Computer Programs and Its Applications. Morgan Kaufmann, San Francisco (1998)

10 Incremental Acquisition of Neural Structures through Evolution

207

5. Bentley, P., Kumar, S.: Three ways to grow designs: A comparison of embryogenies for an evolutionary design problem. In: Banzhaf, W., Daida, J., Eiben, A.E., Garzon, M.H., Honavar, V., Jakiela, M., Smith, R.E. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference, Orlando, Florida, USA, July 13-17, vol. 1, pp. 35–43. Morgan Kaufmann, San Francisco (1999) 6. Bongard, J.C., Pfeifer, R.: Repeated structure and dissociation of genotypic and phenotypic complexity in artificial ontogeny. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2001, pp. 829–836 (2001) 7. Dellaert, F., Beer, R.D.: A developmental model for the evolution of complete autonomous agents. In: Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior (1996) 8. Eiben, A.E., Smith, J.E.: Introduction to Evolutionary Computing. Springer, Heidelberg (2003) 9. Fogel, D.B.: Evolving Artificial Intelligence. PhD thesis, University of California, San Diego, CA, USA (1992) 10. Gomez, F.J., Miikkulainen, R.: 2-D pole balancing with recurrent evolutionary networks. In: Proceedings of the International Conference on Artificial Neural Networks, Skovde, Sweden (1998) 11. Gomez, F.J., Miikkulainen, R.: Solving non-markovian control tasks with neuroevolution. In: Proceedings of the International Joint Conference on Artificial Intelligence, Stockholm, Sweden (1999) 12. Gomez, F.J., Miikkulainen, R.: Robust non-linear control through neuroevolution. Technical Report AI-TR-03-303, Department of Computer Sciences, The University of Texas, Austin, USA (2002) 13. Gruau, F.: Neural Network Synthesis Using Cellular Encoding and the Genetic Algorithm. PhD thesis, Ecole Normale Superieure de Lyon, Laboratoire de l’Informatique du Parallelisme, France (January 1994) 14. Gruau, F., Whitley, D., Pyeatt, L.: A comparison between cellular encoding and direct encoding for genetic neural networks. In: Koza, J.R., Goldberg, D.E., Fogel, D.B., Riolo, R.L. (eds.) Genetic Programming: Proceedings of the First Annual Conference, Standford University, CA, USA, pp. 81–89. MIT Press, Cambridge (1996) 15. Holland, J.H.: Adaptation in Natural and Artificial Systems. MIT Press, Massachusetts (1975) 16. Igel, C.: Neuroevolution for reinforcement learning using evolution strategies. In: Sarker, R., Reynolds, R., Abbass, H., Tan, K.C., McKay, B., Essam, D., Gedeon, T. (eds.) Congress on Evolutionary Computation (CEC 2003), vol. 4, pp. 2588–2595. IEEE Press, Los Alamitos (2003) 17. Jakobi, N.: Harnessing morphogenesis. In: Proceedings of Information Processing in Cells and Tissues, pp. 29–41 (1995) 18. Jung, J., Reggia, J.: A descriptive encoding language for evolving modular neural networks. In: Deb, K., et al. (eds.) GECCO 2004. LNCS, vol. 3103, pp. 519–530. Springer, Heidelberg (2004) 19. Kassahun, Y., Edgington, M., Metzen, J.H., Sommer, G., Kirchner, F.: A common genetic encoding for both direct and indirect encodings of networks. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2007), pp. 1029–1036 (2007) 20. Kassahun, Y., Metzen, J.H., de Gea, J., Edgington, M., Kirchner, F.: A general framework for encoding and evolving neural networks. In: Hertzberg, J., Beetz, M., Englert, R. (eds.) KI 2007. LNCS, vol. 4667, pp. 205–219. Springer, Heidelberg (2007) 21. Kassahun, Y., Sommer, G.: Improving learning and adaptation capability of agents. In: Proceedings of 8th Conference on Intelligent Autonomous Systems (IAS-8), Amsterdam, pp. 472–481 (November 2004)

208

Y. Kassahun et al.

22. Kimura, H., Kobayashi, S.: Reinforcement learning for locomotion of a two-linked robot arm. In: Proceedings of the 6th European Workshop on Learning Robots, pp. 144–153 (1997) 23. Kitano, H.: Designing neural networks using genetic algoithms with graph generation system. Complex Systems 4, 461–476 (1990) 24. Koza, J.R.: Genetic programming: A paradigm for genetically breeding population of computer programs to solve problems. Technical Report STAN-CS-90-1314, Computer Science Department, Standford University, Stanford, CA, USA (1990) 25. Lindenmayer, A.: Mathematical models for cellular interactions in development, parts I and II. Journal of Theoretical Biology 18, 280–315 (1968) 26. Metzen, J.H., Edgington, M., Kassahun, Y., Kirchner, F.: Performance evaluation of EANT in the RoboCup Keepaway benchmark. In: Proceedings of the Sixth International Conference on Machine Learning and Applications (ICMLA 2007), Cincinnati, Ohio, USA, December 13–15 (accepted, 2007) 27. Moriarty, D., Miikkulainen, R.: Efficient reinforcement learning through symbiotic evolution. Machine Learning 22, 11–33 (1996) 28. Nolfi, S., Floreano, D.: Evolutionary Robotics. The Biology, Intelligence, and Technology of Self-Organizing Machines. MIT Press, Massachusetts (2000) 29. Nolfi, S., Parisi, D.: Growing neural networks. Technical Report PCIA-91-15, Institute of Psychology, Rome (1991) 30. Pfeifer, R., Scheier, C.: Understanding Intelligence. MIT Press, Massachusetts (1999) 31. Schwefel, H.P.: Numerische Optimierung von Computer-Modellen mittels der Evolutionsstrategie. Interdisciplinary Systems, vol. 26, Birkhaeuser, Basel (1977) 32. Schwefel, H.P.: Evolution and Optimum Seeking. John Wiley & Sons, New York (1995) 33. Sendhoff, B., Kreutz, M.: Variable encoding of modular neural networks for time series prediction. In: Congress on Evolutionary Computation (CEC 1999), pp. 259–266 (1999) 34. Shaffer, J., Cannon, R.: On the control of unstable mechanical systems. In: Proceedings of the Third Congress of the International Federation of Automatic Control (1966) 35. Stanley, K.O.: Efficient Evolution of Neural Networks through Complexification. PhD thesis, Artificial Intelligence Laboratory. The University of Texas at Austin., Austin, USA (August 2004) 36. Stanley, K.O., Miikkulainen, R.: Efficient reinforcement learning through evolving neural network topologies. In: Proceedings of the Genetic and Evolutionary Computation Conference, San Francisco, CA (2002) 37. Stanley, K.O., Miikkulainen, R.: Competitive coevolution through evolutionary complexification. Journal of Artificial Intelligence Research 21, 63–100 (2004) 38. Sutton, R., Barto, A.: Reinforcement Learning. An Introduction. MIT Press, Massachusetts (1998) 39. Tsuchiya, C., Kimura, H., Kobayashi, S.: Policy learning by GA using importance sampling. In: Proceedings of 8th Conference on Intelligent Autonomous Systems (IAS-8), Amsterdam, pp. 385–394 (November 2004) 40. Vaario, J., Onitsuka, A., Shimohara, K.: Formation of neural structures. In: Proceedings of the Fourth European Conference on Articial Life, ECAL 1997, pp. 214–223 (1997) 41. Wieland, A.: Evolving controls for unstable systems. In: Proceedings of the International Joint Conference on Neural Networks, pp. 667–673 (1991) 42. Yao, X.: Evolving artificial neural networks. Proceedings of the IEEE 87(9), 1423–1447 (1999)

11 Evolutionary Terrain-Based Navigation of Autonomous Mobile Robots Terrence P. Fries Department of Computer Science, Coastal Carolina University, Conway, South Carolina 29528 USA [email protected]

Abstract. Optimal motion planning is critical for the successful operation of an autonomous mobile robot. Evolutionary approaches are able to provide optimal paths to problems normally intractable for traditional search methods. However, most proposed methods lack the ability to operate in a dynamic environment in real-time, and few address the impact of varying terrain conditions on the optimal path. This evolutionary navigation approach employs a novel chromosome encoding scheme that provides both path and trajectory planning. The terrain conditions are modeled using fuzzy linguistic variables to allow for the imprecision and uncertainty of the terrain data. The method is extensible and robust, allowing the robot navigate in real-time and to adapt to dynamic conditions in the environment.

11.1 Introduction Efficient navigation, characterized by optimal motion planning, is crucial for the success of many applications involving autonomous mobile robots. An autonomous robot must have the capability of computing and following an optimal or near optimal path to a specified destination. It must also have the ability to dynamically make adjustments to the path in response to unexpected changes in the environment and adapt to uncertain environments. Motion planning consists of path planning and trajectory planning [18]. In many navigation approaches, these two functions are independent of each other, however, in certain approaches, they are intertwined. Path planning computes an optimal, collision-free path through an environment containing obstacles. The path is optimal with respect to criterion specific to the application. Trajectory planning generates the actions of the robot necessary to proceed along the computed path. This research presents an approach to motion planning in a dynamic environment using evolutionary computation. Fuzzy linguistic variables are used to represent uncertain terrain conditions. When evaluating robot navigation algorithms, it is necessary to consider whether they can operate the robot with a real-time response, can adapt to a dynamic environment, and address the issue associated with various types of terrain. To be of any practical use, a motion planning algorithm must adequately efficient to operate in real-time. Limited computational resources are available onboard common autonomous robots and few applications allow the luxury of extended periods of inactivity while elaborate D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 209–226. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

210

T.P. Fries

Fig. 11.1. Non-monotone path between rooms

computations are performed. It is common for evolutionary approaches to motion planning to reduce computational complexity by constraining operation to a static environment [1, 4, 9, 24, 28]. For many applications, this is not practical as their environment is not isolated. Frequently, mobile robots interact with people who move objects or themselves. The method presented in this research uses evolutionary computation techniques to provide motion planning in real-time that is adaptive to changing or uncertain environments and addresses the issues of varying terrain. Most evolutionary approaches to motion planning fail to provide real-time solutions [1, 4, 9, 24, 28]. The few that do offer real-time response place unacceptable restrictions on the operation of the robot, such as limiting solutions to x-monotone or y-monotone paths [34]. An x-monotone path is one in which the projection of the path on the x-axis is non-decreasing. Such a constraint is problematic because even a simple path between two rooms in a building is likely to be neither x-monotone nor y-monotone as shown in Figure 11.1. Another attempt to reduce computational complexity in evolutionary approaches is the use of fixed-length chromosomes [3, 34]. However, it has been shown that fixedlength chromosomes are too restrictive for robot path planning. It restricts the solution path by placing unacceptable constraints on the representation of the environment and on the path [3, 9]. The use of evolutionary algorithms for path planning has been studied by many researchers. Shibata and Fukuda [30] applied evolutionary computation to a point robot in a static environment. Davidor [5] proposed a motion planning strategy that attempts to minimize the accumulated deviation between the actual and desired path. However, this is not appropriate for a dynamic environment since it assumes that a desired path is already known. Nearchou [25] presented an evolutionary approach that compares

11 Evolutionary Terrain-Based Navigation of Autonomous Mobile Robots

211

favorably with other evolutionary techniques, but it requires the robot’s map to be converted to a graph. Conversion of a map in certain formats, such a Voronoi map, can be quite computationally intensive. Other navigation approaches apply evolutionary techniques to training of a neural network [27], potential fields [38], reactive navigation [7], and hierarchical structures [37]. None of these approaches account for dynamic conditions. Navigation attempts using standard search algorithms are NP complete. Approaches using genetic algorithms have been proven to be capable of solving normally intractable search problems [10, 12]. Another limitation of traditional search algorithms is that they can easily become trapped in a local minimum. On the other hand, GAs are inherently capable of escaping local minima. Due to the suitability of GAs for such search problems, many attempts have been made to apply them to robot path planning. Wu [39], defines robot motion as a series of Euclidean transformations that rotate or move the robot. The path is a sequence of Euclidean coordinates representing the positions of the robot as it moves. The coordinates are encoded as real numbers in the chromosome. The GAs uses the crossover and mutation operators to produce new offspring. The fitness of each chromosome depends on the length and the smoothness of path. Wu establishes convergence to a near optimal solution when the best element in a generation produces no more than a 0.5% fitness improvement over the best element in the prior generation. However, the encoding of real numbers requires that the chromosome be very long, thus, making it difficult to achieve rapid convergence necessary for real-time performance in an environment of any practical size. Using a similar representation for the chromosome with crossover and mutation operators, Li [21] simplifies the fitness function by not considering the path smoothness. The method improves on similar GA approaches by implementing learning algorithm to adjust the crossover and mutation probabilities. Although the algorithm performs well generating near optimal paths in simulations, it has not been tested in a real-time environment. Lu [22] has proposed a dual-layer GA to accommodate dynamic environments. The first layer of the GA handles static obstacles, while the second layer addresses dynamic obstacles. The path is represented as a series of waypoints in the robot’s path. As with Wu’s method, the fitness function incorporates both the length and the smoothness of the path. A simulation program coded in C++ has provided acceptable results, however, the method has not been tested in a real-world environment. Therefore, its ability to respond to real world dynamic features in a timely manner is unknown. Hu and Vie [13] propose a niche genetic algorithm (NGA). A niche is a heuristic that increases search efficiency by adjusting the fitness of each element in the population based on some criteria. The sharing niche used in this navigation method decreases the fitness of each population element (chromosome) by an amount related to the number of similar individuals in the population. The intent of the niche is to maintain diversity of the population in order to prevent stagnation toward a set of similar chromosome patterns. Although the approach converged quickly to a near optimal path, it was only tested on a 20 x 20 grid with 7 obstacles. The NGA method has not been tested on a larger grid so it is not possible to determine its performance for larger, higher resolution environments.

212

T.P. Fries

Tarokh [35] provides one of the few attempts to consider terrain conditions in evolutionary path planning. The fitness function attempts to optimize terrain roughness and path curvature. The environment is considered to be a grid of cells with each cell containing the height of rocks in it. A contour map is created with the height of rocks in each cell expressed as a fuzzy set with 5 possible options. A unique feature is that the method adjusts the probabilities of the genetic operators based on the diversity measure of path populations and the traversability measure of the paths. The method has been successfully tested on a 16 x 16 grid, but has not been tested on higher resolution grids. A further consideration among current motion handling approaches is the manner in which they handle terrain conditions. Most label each area on its map either clear or completely blocked by an obstacle [3, 28]. In many real world situations, an area may have terrain that is difficult to traverse, such as sandy areas which cause slippage, rocky areas that require minor course adjustments within them and/or loss of time, or sloped areas that may cause slippage or increased time to climb. Difficult terrain may be traversable at the cost of increased time, but may provide a more optimal path than easily traversable terrain. Iagnemma and Dubowsky [14] measure the roughness of the terrain as elevation changes. However, the relatively large elevation changes which are used lead to concern about wheel diameter and wheel-terrain contact force and angle. The resultant computation is quite complex, but suitable so the slow movement of the planetary rover under consideration. However, it is not suitable for the real-time response required in other real-world applications. Other researchers [2, 6, 11, 15, 17, 23, 26, 29, 32] have also considered issues of severe terrain such as rocky areas and large elevation changes. However, other terrain factors are equally influential on the navigability of a section of terrain with what may appear to be rather benign elevation changes. Terrain features such as a sandy or gravel surface or a mild slope may also have a significant impact on the traversability of the terrain. Mobile robots are confronted with these terrain issues even when not operating in extreme environments. The uncertainty inherent in terrain sensing devices and mapping methods must also be considered [15, 17, 19, 20, 36]. In these cases, the exact nature of the terrain in a particular area is uncertain and only an estimate of its true condition. The uncertainty of terrain conditions has been addressed by research using methods such as a Gaussian [14] and Kalman [16] filters. However, these methods are computationally intensive and, therefore, inappropriate for real-time operation. The evolutionary navigation approach presented in this chapter has many advantages over existing navigation approaches. First, it addresses the issue of terrain uncertainty by assigning to each section of terrain a fuzzy linguistic variable which indicates the difficulty of traversing that area. Most of the approaches to motion planning address only path planning and leave trajectory planning for a later step. The GA coding scheme used in this chapter combines path planning with trajectory planning, thus, eliminating the additional step of trajectory planning once an optimal path is found and reducing the computational time to allow a real-time response. In addition, unlike most navigation schemes, the

11 Evolutionary Terrain-Based Navigation of Autonomous Mobile Robots

213

evolutionary approach provides real-time motion planning in a dynamic environment without significant restrictions on the environment size or resolution. Section 2 presents the representation of the environment and GA basics. In Section 3, the new fuzzy genetic motion planning approach is presented. Section 4 provides a discussion of the implementation and test results of the new motion planning approach. Section 5 provides asummary of the approach and discusses future directions for research on this method.

11.2 Evolutionary Computation A genetic algorithm (GA) is a specific form of evolutionary computation commonly used for optimization [10, 12]. It represents a possible solution as a binary string, called a chromosome. Successful implementation is dependent upon the selection of a proper encoding of the possible solution paths as binary strings. The GA begins with an initial population of chromosomes, or possible solutions. The GA then creates new individuals using methods analogous to biological evolution. The fitness of each chromosome is calculated using a fitness function. The criteria for evaluation is domain specific information about the relative merit of the chromosome. For example, in the case of path planning, the fitness function may calculate the time required or distance traveled to move from the initial location to the goal. The fittest parents are chosen to reproduce to create offspring. The offspring are generated by subjecting the parent chromosomes to various genetic operators including crossover and mutation. The crossover operator combines parts of two different chromosomes to create two new ones. In single point crossover, the left part of a chromosome is combined with the right part of another, and then the remaining two parts of the originals are combined, thus, creating two new offspring. This type of crossover produces two offspring of the same size as the parents as shown in Figure 11.2. The two sections can also be combined to form offspring of differing sizes as shown in Figure 11.3. The crossover point is usually randomly selected, although it can be fixed for particular applications. Multiple point crossover divides the chromosome into multiple strings which are recombined with those of another chromosome. A fixed number of divisions can be specified for all multiple point crossovers, or the number of partitions can be randomly set for each pair of chromosomes being operated on. Additional crossover schemes which utilize heuristics also exist, but add too much computational complexity for this application. Not all chromosomes are subjected to crossover. The crossover rate specifies the percentage of parent chromosomes involved in the crossover. The mutation operator changes the value of one random position in a chromosome. The offspring produced is identical to the parent except at the mutation point. For chromosomes represented as bit strings, this can be considered as inverting a single bit. The mutation rate specifies the probability that a particular bit in a string is inverted. The most fit offspring replace the parents with the poorest fitness and the process continues until the population converges to a solution indicated by exceeding a fixed number of generations or until a chromosome attains a certain fitness value. Several components can significantly affect the performance of a genetic algorithm: encoding

214

T.P. Fries

0 1 0 0 1 1 0 1

0 1 1 0 0 0 1 0

0 1 0 0 1 0 1 0

0 1 1 0 0 1 0 1

Fig. 11.2. Crossover with same size offspring

0 1 0 0 1

1 0 1

0 1 0 0 1

0 1 1 0 0

0 1 1 0 0

0 1 0

1 0 1

0 1 0

Fig. 11.3. Crossover with different size offspring

of the chromosome, the initial population, the genetic operators and their control parameters, and the fitness function.

11.3 Motion Planning Algorithm 11.3.1

Assumptions

Several assumptions have been made to reduce the initial complexity of the navigation problem and allow concentration on the particular issues of motion planning and terrain considerations. First, the robot is assumed to be holonomic which means it is capable of turning within its own radius. This eliminates the complexity involved in the movement of non-holonomic robots [11, 18]. The robot is limited to the move-stop-turn course of action, where after moving, the robot will stop, turn, and then proceed to move again. This avoids issues related to planning an arc-shaped trajectory and to terrain-induced lateral forces on the wheels while turning in an arc while moving. To simplify the problem, localization is restricted to dead reckoning based on wheel odometry information and localization uncertainty is ignored. Localization problems will be addressed in future research. 11.3.2

Representation of the Environment as a Grid

The area in which the robot will navigate is divided into an environment grid and a path is defined as the traversal of a series of adjacent cells in the grid. It is common to represent a robot’s operational domain using an occupancy grid [8, 18]. The environment grid extends the occupancy grid by encoding the terrain characteristics for each cell

11 Evolutionary Terrain-Based Navigation of Autonomous Mobile Robots

215

Fig. 11.4. Diagonal traversal of cells with expansion of actual obstacles

within the grid itself. The distance d (a, b) between two adjacent cells a and b is defined as the Euclidean distance between the centers of the two cells. This representation of distance allows the map data to be stored in any efficient format, such as a quadtree [41]. Storage by such methods provides more compact representation of an environment by storing a large open area or large obstacle as a single grid location, rather than many uniformly sized small squares. It also allows the path to be represented by fewer grid transitions, thus, reducing the size of the GA encoding string, or chromosome, and the time required to determine a solution. Each cell in the grid is assigned a fuzzy value that indicates the difficulty in traversing the terrain in that cell. The use of fuzzy values allows cells with moderately hostile terrain, such as rocks or loose sand, to be considered in a possible solution path while being weighted by their difficulty of traversal. A cell which contains an obstacle is assigned a fuzzy value indicating it is impassable and any possible solution path containing it is unacceptable. For presentation purposes in this chapter, the grid will be restricted to 16 by 16 for simplicity, however, the algorithm has been successfully tested for much larger sized grids. Further discussion of this restriction and actual testing is found in the Test Results in Section 11.4. In the evolutionary navigation algorithm, the robot is considered to be a holonomic point, that is, it is able to turn within its own radius. Because the robot is holonomic, a path can change direction within a cell and does not require a large arc for turning. Since it is a point, when traversing between two diagonally adjacent cells, it is not necessary to consider the other cells sharing the common corner as shown in Figure 11.4. This is not as impractical as it may appear at first glance. All real obstacles are expanded by half the radius of the robot when marking which cells are obstructed, thus allowing the robot to be treated as a point. This permits navigation of the center of the robot along the side of an obstacle or diagonally between obstacles. In Figure 11.4, the actual obstacle is solid and the expansion is shaded. 11.3.3

Encoding the Chromosome

The first step is to choose a coding scheme which maps the path into a binary string or chromosome. Emphasis is placed on minimizing the length of the binary string.

216

T.P. Fries

Minimizing the length of the chromosome reduces the number of generations necessary to produce an acceptable solution because less permutations are possible. A variable length string composed of blocks which encode the direction of movement and the length of the movement was chosen. Consider the robot in the center cell as in Figure 11.5 (a) having just arrived from cell 4 and facing in the direction of the arrow. There are eight possible directions for movement. However, cell 4 can be eliminated from consideration for the next move since the robot came from that cell and returning to it would create a non-optimal path. Cells 1, 2, 6, and 7 can be eliminated because they could have been reached from cell 4 using a shorter distance than through the center cell in which the robot currently is positioned. Only three cells remain in consideration for possible movement. The three cells require only 2 bits to encode as in Figure 11.5 (b).

Fig. 11.5. Planning the next move: (a) possible movement to next cell; (b) allowable movement after applying constraints

The largest number of cells that can be traversed in a square grid is found by starting in a corner and moving as far as possible along a side or the diagonal. In an n by n grid, the maximum movement is an n-1 cells. Since the grid is constrained to 16 by 16 cells, the maximum number of cells that can be traversed in a single move is 15 which requires 4 bits to encode. As a result, each movement can be encoded in a 6-bit block (2 bits for direction and 4 bits for distance) as shown in Figure 11.6. For larger n x n grids, the block size would be 2 + log 2n. A chromosome composed of these 6-bit blocks contains not only the path, but also the necessary trajectory information for movement of the robot. Thus, this unique encoding provides both path planning and trajectory planning. 11.3.4

Initial Population

The motion planning approach begins by randomly generating an initial population of chromosomes. In an effort to direct the solution to the shortest path, another chromosome is added to the initial population. It represents a straight line from the start to

11 Evolutionary Terrain-Based Navigation of Autonomous Mobile Robots

0

1

direction

1

1

1

217

1

distance

Fig. 11.6. Block encoding of one movement of the robot

destination regardless of obstacles. If a straight line is not possible due to grid locations, the closest approximation to a straight line path is used. Through the testing of various combinations of variables, it was found that a population size, p = 40, was sufficient to seed the chromosome base while simultaneously minimizing computational complexity. 11.3.5

Genetic Operators and Parameters

The algorithm uses single point crossover with a crossover rate of 0.8. The crossover rate is the percentage of parent chromosomes involved in the crossover. The mutation rate, or probability that a particular bit in the string is inverted, was 0.02. These parameters were arrived at through experimentation. 11.3.6

Fuzzy Terrain Conditions in the Fitness Function

Selection of a fitness function is a critical aspect of this approach. Chromosomes are selected for reproduction through crossover and mutation based on the fitness function. The value provided by the fitness function is then used to retain the best members of the population for the next generation. Common approaches to using GAs for path planning set the fitness to an unacceptable value for any chromosome whose path traverses a grid cell with an obstacle in it. Otherwise, the fitness is based upon the distance traveled in the path. However, this does not account for varying terrain conditions. Recent attempts have been made to represent terrain surface texture as a fuzzy set corresponding to a traction coefficient [36]. This concept has been adapted for this research to represent adverse terrain conditions. However, the traction coefficient is unable to properly represent terrain slope or slippage potential in the cell. Rather than using a traction coefficient, the evolutionary navigation approach assigns to each cell a fuzzy set corresponding to the difficulty in traversing its terrain. The difficulty in traversing a particular terrain is imprecise because it may vary from one instance to another. In addition, it is problematical to compare different terrain conditions because of the varied nature of each. Further difficulty in a assigning a precise terrain difficulty exists because traversal of an cell in different directions may have significantly different difficulty levels. For example, traversing a sandy hill moving downhill, uphill, or across the side of the hill have dissimilar difficulty levels. Compared to downhill traversal, uphill traversal is more difficult due to the angle and more time consuming due

218

T.P. Fries Very Difficult

Difficult

Moderate

Easy

Very Easy

1

0 0

1

Fig. 11.7. Fuzzy representation of terrain conditions

to slippage. When moving across the face of a slope, slippage can result in deviation from the planned path, as well as loss of forward progress. Because of the imprecision of terrain conditions and the problems in directly comparing them, this research represents the terrain difficulty of each cell as a fuzzy set [42, 43]. The terrain condition for each cell is expressed using one of five linguistic variables: Very Difficult, Difficult, Moderate, Easy, and Very Easy. The linguistic terrain variables are repesented as fuzzy numbers shown in Figure 11.7. In testing the navigation algorithm, the terrain conditions were manually assigned to each cell. Disucssion of further research in automating the assignment of terrain conditions and assigning multiple terrain conditions for different directions of traversal are discussed in the Conclusions section of this chapter. The terrain conditions represent the difficulty in traversing the cell which can be affected by conditions such as slope, sand, rocks, etc. As a result, the fitness function must be expanded for this research. For any path not passing through an obstacle, the fitness function uses the Euclidean distance between the centers of the cells traversed weighted by the terrain conditions for each cell. 11.3.7

Operation in a Dynamic Environment

The evolutionary terrain-based navigation method allows the robot to function in a dynamic environment. If an obstacle is detected by the robot where it not expected, the planner simply recalculates a new optimal path from the current location of the robot to the destination, and the robot can continue its movement toward the destination. Further discussion of the algorithm’s performance in a dynamic environment is provided in the Test Results section of this chapter.

11.4 Test Results The evolutionary navigation approach was implemented using C++ and Saphira robot control software by SRI International. Tests were initially conducted in the Saphira

11 Evolutionary Terrain-Based Navigation of Autonomous Mobile Robots

-30

-10

10

-50

219

30 50

-90

90

500 mm Fig. 11.8. Sonar sensor suite on Pioneer 2-DX robot

simulator to determine which genetic operators offered sufficient performance and to establish appropriate ranges for the genetic parameters. Once these were determined, the navigation approach was tested on a Pioneer 2-DX mobile robot. The Pioneer 2-DX is a 3-wheeled, holonomic robot with a 25 cm radius. It is equipped with a suite of eight sonar sensors arranged as shown in Figure 11.8 and tactile bumpers in the front and rear. The environment grid was provided to the robot in a predefined map. The fuzzy terrain conditions were manually assigned to each cell. The Conclusions section of this chapter discusses more robust approaches to assigning terrain conditions. In the interest of readability of diagrams, test results are presented for a 16 by 16 environment grid. This allows the algorithm’s functionality and performance to be presented while maintaining readability of the images. Discussion of testing using much larger grids and quadtree representations of the environment will also be presented to demonstrate the extensibility of the navigation algorithm. Figure 11.9 shows the path generated by the evolutionary navigation method for a test environment with no cells labeled with terrain difficulty values. The S and D indicate the start and destination cells of the robot, respectively, and the solid-filled cells indicate obstacles. Manual examination confirms that the generated path is optimal. The optimal solution chromosome for the path required six 6-bit blocks, one for each straight segment in the path. A more complex path generated by the navigation algorithm for an different environment grid is illustrated in Figure 11.10. Due to the additional turns in this path, this solution required nine 6-bit blocks in the optimal solution chromosome. The evolutionary navigation approach produced the expected paths when the environment grid was labeled with fuzzy terrain conditions. The shaded cells on the grid in Figure 11.11 were labeled as having Moderate difficulty to traverse. This had no effect on the generation of an optimal path, as should be the case. The path generated was the same as in the first test without difficulty labeling. However, when the labeling of

220

T.P. Fries

S

V`

D Fig. 11.9. Path generation with no terrain problems

Fig. 11.10. Path in alternate environment with no terrain problems

the same area of difficulty was changed to Difficult, a different path was produced by the fitness function as shown in Figure 11.12. When the Moderate area was enlarged as in Figure 11.13, the fitness function again detected an optimal path which avoided the larger Moderate terrain area.

11 Evolutionary Terrain-Based Navigation of Autonomous Mobile Robots

221

Fig. 11.11. Path with Moderate area of difficulty

Fig. 11.12. Path with Difficult terrain area

The ability of the method to respond to dynamic changes in the environment was tested using the original environment grid with fuzzy terrain conditions from Figure 11.9. Figure 11.14 shows the planned robot path as solid circles and an unexpected obstacle blocking the path as X’s. When the unexpected obstacle, X, was encountered, a new near-optimal path was calculated from that point to the destination. The hollow circles in Figure 11.14 illustrate the dynamically recalculated path from the

222

T.P. Fries

Fig. 11.13. Path with large area of Moderate difficulty

Fig. 11.14. Navigation in a dynamic environment

current robot position to the destination. This allowed the robot to successfully traverse to the destination despite encountering the anomalous obstacle. Additional trials with randomly positioned obstacles yielded similar successful results. The flexibility of the grid representation of the environment was also investigated by representing the same environment as a quadtree. The quadtree representation and

11 Evolutionary Terrain-Based Navigation of Autonomous Mobile Robots

223

S

V`

D Fig. 11.15. Path using quadtree

? Fig. 11.16. Movement decision for quadtrees

resulting path are shown in Figure 11.15. This and additional tests confirmed that an optimal path is also generated when using a quadtree environment. The only adjustment to the method is inclusion of a heuristic to decide to which cell to move in the event several exist when moving from a large aggregate cell to one of several adjacent smaller ones as shown in Figure 11.16.

11.5 Conclusions This research presents a fuzzy genetic algorithm approach to motion planning for an autonomous mobile robot that performs in real-time without the limitations of monotone paths. Varying terrain conditions are represented as fuzzy values and are included in the path planning decision. The encoding of the chromosome provides full motion planning capabilities and the method is capable of operation in a dynamic environment. Further research directions include the ability to observe and learn terrain conditions during movement along the path.

224

T.P. Fries

The evolutionary navigation approach presented in this chapter has many advantages over existing navigation approaches. While most approaches address only path planning and leave trajectory planning for a later step, the GA coding scheme used in this chapter provides both path planning and trajectory planning, thus, eliminating the additional step of trajectory planning once an optimal path is found. This reduces the computational time required by the algorithm to provide a real-time response. Unlike most navigation schemes, the evolutionary approach is robust and extensible. It provides real-time motion planning in a dynamic environment without significant restrictions on the environment size or resolution. In addition, the evolutionary navigation algorithm addresses the issue of terrain traversability and uncertainty by assigning each cell of the environment grid a fuzzy linguistic variable which indicates the difficulty of traversing that area. It has been noted that terrain conditions for each cell are represented by a fuzzy set. However, it has already been observed that traversal of a cell in different directions may have a completely different level of difficulty. Further research will involve assigning multiple fuzzy sets to each cell grid based on the direction of traversal. Reseach is also underway to automate the assignment of terrain conditions using sensory data and localization discrepancies which indicate slippage or lack of forward progress. Perfect movement by the robot has been assumed without accounting for drift and slippage. Currently, the localization is accomplished through dead reckoning. Additional research will incorporate localization to ensure the robot is on the planned path and provide necessary adjustments to the motion plan. Localization may include the addition of inertial and visual data. This chapter has presented the algorithm using a very simplistic 16 x 16 grid for purposes of demonstrating its functionality and clarity of the images. The approach has been successfully implemented using much larger grids and with quadtree representations of the environment. It has also been assumed that the terrain conditions are known a priori. Since this is not realistic in many applications, further research directions include addition of the ability to observe and learn terrain conditions during movement along the path and to then adapt when more difficult terrain is discovered along the planned path.

References 1. Akbarzadeh, M.-R., Kumbla, K., Tunsel, E., Jamshidi, M.: Soft computing for autonomous robotic systems. Computers and Electrical Engineering 26, 5–32 (2000) 2. Bonnafous, D., Lacroix, S., Simeon, T.: Motion generation for a rover on rough tearrains. In: Proceedings of the 2001 IEEE international conference on intelligent robotics and systems. IEEE Press, New York (2001) 3. Cordon, O., Gomide, F., Herrera, F., Hoffmann, F., Magdalena, L.: Ten years of genetic fuzzy systems: current framework and new trends. Fuzzy Sets and Systems 141, 5–31 (2004) 4. Cuesta, F., Ollero, A.: Intelligent Mobile Robot Navigation. Springer, Heidelberg (2005) 5. Davidor, Y.: Genetic algorithms and robotics: a heuristic strategy for optimization. World Scientific, Singapore (1991) 6. Davidson, A., Kita, N.: 3D simulation and map-building using active vision for a robot moving on undulating terrain. In: Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. IEEE Press, New York (2001)

11 Evolutionary Terrain-Based Navigation of Autonomous Mobile Robots

225

7. deLope, J., Maravall, D.: Integration of reactive utilitarian navigation and topological modeling. In: Zhou, C., Maravall, D., Ruan, D. (eds.) Autonomous Robotics Systems: Soft Computing and Hard Computing Methodologies and Applications, pp. 103–140. Physica-Verlag, Heidelberg (2003) 8. Elfes, A.: Using occupancy grids for mobile robot perception and navigation. IEEE Computer 22, 46–57 (1989) 9. Fogel, D.B.: Evolutionary computation: toward a new philosophy of machine intelligence. IEEE Press, New York (2000) 10. Goldberg, D.E.: Genetic algorithms in search, optimization and machine learning. AddisonWesley/ IEEE Press, Boston/ New York (1989) 11. Hait, A., Simeon, T.: Motion planning on rough terrain for an articulated vehicle in presence of uncertainties. In: Proceedings of the 1996 IEEE/RSJ international conference on intelligent robotics and systems. IEEE Press, New York (1996) 12. Holland, J.H.: Adaption in Natural and Artificial Systems. University of Michigan Press, Ann Arbor (1975) 13. Hu, X., Vie, C.: Niche genetic algorithm for robot path planning. In: Proceedings of the third international conference on natural (2007) 14. Iagnemma, K., Dubowsky, S.: Mobile robots in rough terrain. Springer, New York (2004) 15. Iagnemma, K., Kang, S., Brooks, C., Dubowsky, S.: Multi-sensor terrain estimation for planetary rovers. In: Proceedings of the 8th international symposium on artificial intelligence, robotics, and automation in space. IEEE Press, New York (2003) 16. Jung, I.-K., Lacroix, S.: High resolution terrain mapping using low altitude aerial stereo imagery. In: Proceedings of the 9th IEEE international conference on computer vision. IEEE Press, New York (2003) 17. Kelly, A., Stentz, A.: Rough terrain autonomous mobility – part 2: an active vision predictive control approach. Journal of Autonomous Robots 5, 163–198 (1998) 18. Latombe, J.C.: Robot motion planning. Kluwer, Dordrecht (1991) 19. Laubach, S., Burdick, J.: An autonomous sensor-based planner for microrovers. In: Proceedings of the 1999 IEEE international conference on robotics and automation. IEEE Press, New York (1999) 20. Lee, T., Wu, C.: Fuzzy motion planning of mobile robots in unknown environments. Journal of Intelligent and Robotic Systems 37, 177–191 (2003) 21. Li, Q., Tong, X., Xie, S., Zhang, Y.: Optimum path planning for mobile robots based on a hybrid genetic algorithm. In: Proceedings of the sixth international conference on hybrid intelligent systems. IEEE Press, New York (2006) 22. Lu, J., Yang, D.: Path planning based on double-layer genetic algorithm. In: Proceedings of the third international conference on natural computation. IEEE Press, New York (2007) 23. Madjidi, H., Negahdaripour, S., Bandari, E.: Vision-based positioning and terrain mapping by global alignment for UAVs. In: Proceedings of the IEEE conference on advanced video and signal based surveillance. IEEE Press, New York (2003) 24. Nearchou, A.C.: Adaptive navigation of autonomous vehicles using evolutionary algorithms. Artificial Intelligence in Engineering 13, 159–173 (1999) 25. Nearchou, A.C.: Path planning of a mobile robot using genetic heuristics. Robotica 16, 575– 588 (1998) 26. Pai, D., Reissel, L.M.: Multiresolution rough terrain motion planning. IEEE Transactions on Robotics and Automation 14, 19–33 (1998) 27. Peters, J.F., Ahn, T.C., Borkowski, M., Degtyaryov, V., Ramana, S.: Line-crawling robot navigation: a neurocomputing approach. In: Zhou, C., Maravall, D., Ruan, D. (eds.) Autonomous Robotics Systems: Soft Computing and Hard Computing Methodologies and Applications, pp. 141–164. Physica-Verlag, Heidelberg (2003)

226

T.P. Fries

28. Pratihar, D.K., Deb, K., Ghosh, A.: A genetic-fuzzy approach for mobile robot navigation among moving obstacles. International Journal of Approximate Reasoning 20, 145–172 (1999) 29. Seraji, H., Howard, A.: Behavior based robot navigation on challenging terrain: a fuzzy logic approach. IEEE Transactions on Robotics and Automation 18, 308–321 (2002) 30. Shibata, T., Fukuda, T.: Intelligent motion planning by genetic algorithm with fuzzy critic. In: Proceedings of the 8th IEEE symposium on intelligent control. IEEE Press, New York (1993) 31. Shiller, A., Chen, J.: Optimal motion planning of autonomous vehicles in 3-dimensional terrains. In: Proceedings of the 1990 IEEE international conference on robotics and automation. IEEE Press, New York (1990) 32. Spero, D., Jarvis, R.: Path planning for a mobile robot in a rough terrain environment. In: Proceedings of the 3rd international workshop on robot motion and control. IEEE Press, New York (2002) 33. Stafylopatis, A., Bleka, K.: Autonomous vehicle navigation using evolutionary reinforcement learning. European Journal of Operational Research 108, 306–318 (1998) 34. Sugihara, K., Smith, J.: Genetic algorithms for adaptive motion planning of an autonomous mobile robot. In: Proceedings of the 1997 IEEE international symposium on computational intelligence in robotics and automation. IEEE Press, New York (1997) 35. Tarokh, M.: Hybrid intelligent path planning for articulated rovers in rough terrain. Fuzzy Sets and Systems 159, 1430–1440 (2008) 36. Tunstel, E., Howard, A., Huntsberger, T., Trebio-Ollennu, A., Dolan, J.M.: Applied soft computing strategies for autonomous field robotics. In: Zhou, C., Maravall, D., Ruan, D. (eds.) Autonomous Robotics Systems: Soft Computing and Hard Computing Methodologies and Applications. Physica-Verlag, Heidelberg (2003) 37. Urdiales, C., Bandera, A., Perez, E., Poncela, A., Sandoval, F.: Hierarchical planning in a mobile robot for map learning and navigation. In: Zhou, C., Mara-vall, D., Ruan, D. (eds.) Autonomous Robotics Systems: Soft Computing and Hard Computing Methodologies and Applications. Physica-Verlag, Heidelberg (2003) 38. Vadakkepat, P., Lee, T.H., Xin, L.: Evolutionary artificial potential field - applications to mobile robot planning. In: Zhou, C., Maravall, D., Ruan, D. (eds.) Autonomous Robotics Systems: Soft Computing and Hard Computing Meth-odologies and Applications. PhysicaVerlag, Heidelberg (2003) 39. Wu, J., Qin, D.-X., Yu, H.-P.: Nonholonomic motion planning of mobile robot with ameliorated genetic algorithm. In: Proceedings of the 2006 international conference on intelligent information hiding and multimedia signal processing. IEEE Press, New York (2006) 40. Xu, W., Liang, B., Li, C., Qiang, W., Xu, Y., Lee, K.: Non-holonomic path planning of space robot based on genetic algorithm. In: Proceedings of the 2006 IEEE international conference on robotics and biometrics. IEEE Press, New York (2006) 41. Yahja, Stentz, A., Singh, S., Brumitt, B.: Franed-quadtree path planning for mobile robots operating in sparse environments. In: Proceedings of the 1998 IEEE international conference on robotics and automation, IEEE Press, New York (1998) 42. Zadeh, L.A.: Fuzzy sets. Information and Control 8, 338–353 (1965) 43. Zadeh, L.A.: A theory of approximate reasoning. Machine Intelligence 9, 149–194 (1979) 44. Zhang, B.-T., Kim, S.-H.: An evolutionary method for active learning of mobile robot path planning. In: Proceedings of the 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation. IEEE Press, New York (1997)

12 A Novel Intelligent Control Approach for Precise Tracking of Autonomous Robots Sukumar Kamalasadan1 and Adel A. Ghandakly2 1

2

Department of Engineering and Computer Technology, The University of West Florida, Pensacola, FL-32514, USA [email protected] Department of Electrical and Computer Engineering, California State University, Chico, CA-95929, USA [email protected]

Abstract. This chapter presents an architecture for precise tracking of autonomous robots based on a novel intelligent adaptive control approach. The scheme evolved from a supervisory loop concept([9]-[12],[18]) consisting of two control algorithms that work in an autonomous manner for the precision control of such nonlinear dynamic systems. The first algorithm is a neurocontroller that monitors the system’s closed loop dynamics while functional changes occur, based on supervised motor learning. The second one is an adaptive controller which controls the system dynamics when parametric changes occur based on a Model Reference Adpative Control (MRAC) strategy. First, the chapter discusses theoretical formulation and design considerations. The theoretical formulations are also analyzed for closed loop stability based on a Lyapunov function candidate. Then, results are presented for simulation and real-time implementation of the position tracking conducted on a single link flexible manipulator nonlinear model and a mobile wheeled robot, respectively. Finally, the chapter concludes by illustrating how this concept can be expanded in the presence of multiple-modes and emergent behaviors.

12.1 Introduction Controlling nonlinear, parametric uncertain and functionally unknown systems, such as robotic manipulator and mobile robot for precise tracking, are extremely difficult due to drastic changes in such system’s operation. Since most of the physical systems are nonlinear in nature, there is definitely a need to understand and control these systems. To this end, computationally intensive control algorithms that can be developed based on nonlinear control theory are of particular interest in modern day control. The design procedure of such a controller consists of two parts: (i) mathematical formulation and stability analysis (ii) development of intelligent algorithms and implementation. The first part deals with performing theoretical and stability analysis based on nonlinear control theory and the second deals with the controller design, algorithmic development and implementation. During the past twenty years there has been considerable progress in geometric nonlinear control theory, and several nonlinear feedback designs have been developed. In many such efforts, neural network based controllers are shown to be really both useful and efficient. Some noteworthy contributions involve nonlinear adaptive D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 227–254. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

228

S. Kamalasadan and A.A. Ghandakly

controller using neural networks such as the multiple model approach proposed in Chen et. al. [1]. In that approach, the control law is established using a linear robust adaptive controller and neural network based nonlinear adaptive controller, and switching between them by a suitably defined switching principle. The neural network provides improved response as a nonlinear controller when the linear controller by itself does not provide satisfactory performance. In another method, a direct adaptive output feedback control design for highly uncertain nonlinear systems which does not rely on state estimation is suggested in Calise et. al. [2]. While Lo et. al. [3], proposes multilayer perceptrons with long and short-term memories (LASTMs) for adaptive processing, Arciniegas et. al. [4] studies the use of Radial Basis Function (RBF) neural networks as key components of adaptive controllers aimed at controlling flexible robotic arms. Main problems with soft computing techniques are implementation and issues related to tuning, controlling while switching and computational complexities. To attend some of these issues, agent based approaches have been discussed in several works (Lange et. al.,Aridor et. al., Peine et. al., Maturana et. al., [5]-[8]). These techniques not only allow for modular design but will also make these schemes suitable for practical implementation. This chapter presents a novel neural network based intelligent adaptive controller for functional and parametric uncertain systems. In this scheme an Intelligent Supervisory Loop (ISL) is incorporated into a conventional Model Reference Adaptive Controller (MRAC) framework by utilizing an online growing dynamic Radial Basis Function Neural Network (RBFNN) structure in parallel with MRAC (Kamalasadan et. al. [9][12]). The idea is to control the manipulator or mobile robot (here after referred as system/plant) by a direct MRAC with a suitable single reference model, and at the same time respond to the plant nonlinearities, unmodeled dynamics and functional uncertainties by on-line tuning of an RBFNN controller. This parallel RBFNN controller is designed to precisely track the system output to a desired command signal trajectory, regardless of system uncertainities and/or unmodeled dynamics. Also the updating details of online RBFNN width, centers and weights are derived in order to ensure error reduction and for improved tracking accuracy. Moreover, the RBFNN structure avoids the ‘dimensionality problem’ inherent in such architecture and is dynamic and growing in nature. The chapter details this new intelligent controller viz., (Neural Network Parallel Adaptive Controller (NNPAC)), starting with the concept of Intelligent Supervisory Loops (ISL), the method of incorporating neuro-controller as an ISL that can augment MRAC, and the difference between this new approach and Feedback Error Learning (FEL) scheme(Kawato et. al. [13]). Then, a brief review of direct MRAC is presented. Subsequently, theoretical derivation, modification of the adaptive controller based on the proposed concept and a brief assessment of the stability analysis are discussed. Further, the design details of RBNN structure are explained. The ability of the control scheme is validated by conducting simulation studies based on a nonlinear dynamic system model for the position control of a single link flexible manipulator undergoing several mode swings due to tip load variations. Finally, implementation of a real-time test bed for controlling a mobile robot based on this architecture is explained and results are discussed.

12 A Novel Intelligent Control Approach for Precise Tracking

229

12.2 Formulation of NNPAC Scheme The supervisory loop concept evolved from the concept that except for a central controller (in this case the direct MRAC), other controllers, identifiers or switching devices (basically all other contributors) can just monitors the ‘health’ of the system continuously. When there are untoward incidences such as plant changes, other parallel controllers ‘act’, augmenting the central one with the goal of overall system error minimization. The advantage of such a concept is that the ‘actors’ perform in a ‘need based’ manner (controller computations are only performed when the error function is greater than a threshold value), and thus do not contribute to controller complexity. Also, as these plant abnormalities are not considered to be present at all times, this approach works perfectly well in the case of normal conditions as it does not disturb the closed loop system performance. Another advantage is that since the loops are autonomous in nature, controllers can work either local or remote to the plant location, which can be beneficial in the context of decentralized control. Fig. 12.1 shows the ISL concept. The method of incorporating Intelligent Supervisory Loops (ISL) is as follows. The parallel controller originates from the fact that a neural network can be used to learn from the feedback error of plant controller closed loop. The plant, which is assumed to undergo vast dynamic changes, is controlled by an adaptive controller that tracks the reference model output. During operation, when the plant encounters un-modeled dynamics, nonlinearity, functional uncertainty or a steady state error between the reference model output and the reference trajectory, the RBFNN controller develops a control signal such that the error is reduced to zero, asymptotically. This is infact carried out continuously such that the output error is zero, and precision tracking is achieved. Thus, in this parallel structure, controllers complement each other’s contribution. It is worth noting that in this case, as opposed to multimodal adaptive controllers (unlike

Fig. 12.1. NNPAC Concept

230

S. Kamalasadan and A.A. Ghandakly

moving the reference structure), the additional controller augments the adaptive controller dynamics. As the MRAC scheme provides a reference model and adaptive control, further tuning in the presence of functional uncertainty is performed using an online RBFNN. Thus, there is no need for any offline design or a-priori knowledge of the system dynamics. However, due to the parallel structure, initially the system needs to be kept under a stable state by the direct adaptive controller when the RBFNN controller learns about the plant dynamics. As opposed to the multilayer perceptrons, the use of RBFNN ensures that the change in system dynamics acts linearly to the modification in weights of the network. Moreover, the addition of the centers creates a growing network, which increases the nodes, depending on the plant dynamics. The corresponding weights in the network will act as a memory module for each operating point such that when the plant revisits previous operating points, these weights are activated. Thus, this online supervisory motor learning approach complements the adaptive controller. 12.2.1

NNPAC Design Based on Feedback Error Learning

Feedback Error Learning scheme (FEL) is a computational theory of supervised motor learning. This was originally proposed by Kawato et. al. [13], which was inspired by the way the central nervous system (CNS) of the brain controls and learns the voluntary movement. The cerebellum is known to be involved in the control of quick limb movements. It has been hypothesized that the major role of cerebellum is both the temporal and spatial coordination of movements. Also, it has been shown that cerebellum provides a model for the body and the physical environment, so that accurate control movement is made possible despite time delays in sensory feedback. Kawato extended this hypothesis by saying that cerebellum acquires an internal model of inverse dynamics of the controlled object. In the inverse model, the input/output pairs act as output/input pairs of the controlled object respectively, and the feedback error is used as the training signal to acquire the inverse dynamic model. Our hypothesis is that cerebellum allocates separate models based on long and short term memory for temporal and spatial allocations, and the internal model of inverse dynamics generated by an online trained neural network work as a temporal model. An adaptive controller based on spatial reference model allocations acts as feedback controller. From control theory standpoint, the feedback error learning is essentially an adaptive two-degree freedom control with inverse model on the forward path. The direct MRAC with adaptation mechanism provides additional adaptation in the presence of parametric changes in the system. In the presence of only parametric changes, this spatial reference model and adaptive control of cerebellum effectively track the system reference. Thus, the neural network controller acts as a system supervisor as well. Fig. 12.2 shows the NNPAC approach and the FEL for comparison. The approach is different from FEL, which uses a PID controller in the feed forward path, while the NNPAC scheme uses an MRAC design. Further, a fully tuned online RBFNN structure is employed as a feedback error learning design. The advantage of this design is adaptation in the conventional controller and ability to work on multiple modes of operation. The reference model design also enables for a fully online neurocontroller as opposed to offline learning schemes.

12 A Novel Intelligent Control Approach for Precise Tracking

231

Fig. 12.2. Comparison of NNPAC concept and FEL a) FEL approach b) NNPAC approach

12.2.2

Direct Model Reference Adaptive Control

The adaptive control sub framework of the NNPAC is evolved from direct model reference adaptive control concept. A detail discussion of adaptive controllers and specifically the direct MRAC is in Ioannou et. al. [14]. Fig. 12.3 outlines a basic MRAC structure. Briefly, the design of MRAC can be explained as follows. Consider an unknown linear SISO plant described by the following differential equations (12.1)

x˙ = Ax + BU, x(0) = x0 y = hT x ℜ1 ,

(12.2) ℜ1 ,

y is the plant output which is ∈ x is the state Where, U is the input which is ∈ vector which is ∈ ℜn , A,B and h are appropriate dimensions. The objective is that the plant output should track the reference model that represents the desired trajectory by the following differential equations: x˙m = Am xm + Bm r

(12.3)

232

S. Kamalasadan and A.A. Ghandakly

Fig. 12.3. Direct model reference adaptive control framework

ym = h T xm

(12.4)

The control objective begins by defining the tracking error, ε as

ε = y − ym

(12.5)

Then the controller is designed to force this error asymptotically to zero for arbitrary initial conditions, and an arbitrary piece-wise continuous and uniformly bounded reference signal r. The control structure for such scheme is Uad = Θ T ω

(12.6)   T T is the control parameter vector and ω = r y p ω1T ω2T is where, Θ = k θ0 θ1T θ2T the regressor vector. The regression vectors are updated online based on the following equation, ω˙1 = Λ ω1 + LUad (12.7)

ω˙2 = Λ ω2 + Ly p

(12.8)

where, Λ is a stable matrix of order (n-1)x(n-1) such that the | sI-Λ | = Zm (s) and the vector L is defined as LT = [0...0, 1]. Further, the control signal is calculated at each sample time based on following adaptation laws k˙ = −sgn(K p)er

(12.9)

θ˙ = −sgn(K p)ey p

(12.10)

12 A Novel Intelligent Control Approach for Precise Tracking

233

T θ˙1 = −sgn(K p)eω1T

(12.11)

T θ˙2 = −sgn(K p)eω2T

(12.12)

where, K p is a positive constant value. From (12.9)-(12.12) it can be seen that the control law is adjusted using the adaptive adjustment mechanism in response to plant parametric changes. In this sense any parametric uncertainty is controlled effectively by this scheme. This control process develops a piece-wise linear model of the system and successfully tracks the desired trajectory represented by the reference model. However, plant assumptions, such as sign of the high frequency gain should be unknown and relative degree requirements, affects this linear adaptive controller. Also, when there is a drastic change in the plant characteristics, which can be described as functional changes or other nonlinear disturbances, the framework of MRAC alone will most probably fail to control the plant, efficiently. This is mainly because the plant model developed for appropriate tracking will no longer be valid. If there is a learning controller which can learn these functional changes online, then it can augment the adaptive controller significantly well. The main purpose of the intelligent module is to produce an additional control signal, and in turn, linearize the system by canceling the nonlinearities, functional changes and un-modeled dynamics. The intelligent module in this case is a growing dynamic online RBFNN Controller. 12.2.3

Generic RBFNN Architecture and Design

RBFNN is one of the most popular types of neural network because of its accuracy and simplicity. Like the feed forward neural network structure, it has three layers, the input layer, the node (hidden) layer and the output layer. Fig. 12.4 shows a block diagram of three-layered RBFNN. The input layer uses the incoming data and distributes it to the hidden layer without performing any mathematical calculation. The hidden layer is made of elements called nodes. Each of these nodes consists of two parts, a center and a nodal radius. Center is basically a vector that defines a single, unique point in the network state space. Each center has an area of influence around it, termed as nodal radius. Each node calculates the nodal distance and basis function. Calculating the basis function determines the value of the output from each node, which can be classified as individual nodal output. Finally, the total nodal output is calculated by a weighting factor, which symbolizes the ability of the RBFNN to learn the function, which it represents. The network learns by adjusting the weights having known the input/output pairs running through it. In the offline learning weights are adjusted applying the input/output pairs continuously for a period of time. However, in the online training, the pair is applied at each time instant and the network is learned during the course of time. RBFNN are generally universal approximators of a function. Mathematically it can be represented as N

y(X) = ∑ ωi φ i=1

(x − ci )2 r2

(12.13)

Where, y(X) is represented as a sum of N radial basis functions, each associated with a different center ci , and weighted by an appropriate coefficient ωi . If a Guassian function

234

S. Kamalasadan and A.A. Ghandakly

Fig. 12.4. Basic RBFNN architecture

is used, then φ (r) = exp(−d 2j /r2 ). The estimation of the weights can be performed by offline or online learning and for this type of networks, the interpolation is linear in weights ωi .

12.3 Theoretical Analysis This section discusses theoretical assessment of NNPAC technique and how stable adaptive laws are derived from the previously described generic MRAC and RBFNN structure. The objective is to develop a hybrid controller with two control laws working in parallel and at the same time supervising the system that shows functional and parametric changes. 12.3.1

Development of the RBFNN Controller

The feed forward neural network controller does not mimic the adaptive controller, but acquires a fully nonlinear inverse model by trying to eliminate feedback error. In this architecture the novelty of FEL, in which feedback error is used to teach for learning the inverse model of the controlled object, is retained. The objective of this controller is to minimize the error e between the command signal r and the controlled object output y. The plant input U to the plant is composed of the output Unn of the feed forward controller and Uad of the adaptive controller. The feed forward controller is characterized by the following dynamics: Unn = ϕ (W, yd , y˙d , y¨d )

(12.14)

Where, ϕ is a vector function, W is weight matrix, and yd and its derivatives are the desired output and its derivatives respectively. If the controlled object F is known and

12 A Novel Intelligent Control Approach for Precise Tracking

235

F −1 exists and is stable, then choosing ϕ (W ) makes the tracking perfect. If Unn = F −1 r, Uad = Θ T ω and y = F(Uad + Unn) , then during perfect estimation of parametric and functional uncertainities, y=r. In this sense, the neural network controller plays the role of learning inverse dynamics of the function F, and tuning of the neuro-controller can be regarded as the modeling of the inverse dynamics F −1 . However, the MRAC learns the parametric changes in the control object and its adaptive laws performs parametric estimation. The reference model models the overall closed loop of the system dynamics. The RBFNN approximates the system non-linearity and develops a control value such that the error between the desired trajectory and the plant output is asymptotically forced to zero. This is accomplished based on the appropriate derivations of the centers, weights and radii of the RBFNN structure. 12.3.2

Mathematical Formulation

Let the nonlinear system to be controlled with input u and output yi can be represented as x˙ = f (x, θ ) + ∑m i=1 z j (x, θ )U y(t) = x1

(12.15)

Where, Θ = [θ1 , θ2 , ..., θ p ]T is the unknown parameter vector, U is the control input, x is the state vector, f(.) and z(.) are smooth nonlinear functions. Assumption 1 (Relative degree): The nonlinear system model represnted has a relative degree r around equillibrium xe but not necessarily a well defined one. It should have a robust relative degree γ in uε (xe ). Assumption 2 (Linear parametric dependence): The function f (x, Θ ) and z(x, Θ ) are unknown functions, but may be parametrized linearly in uknown parameters/functions p p θi fi (x) and z(x, Θ )= ∑ i=1 θi zi (x). Combining changing varisuch that f (x, Θ )= ∑i=1 ables, the above mentioned system can be x˙ = F(x, Θ ,U)

(12.16)

Assumption 3 (Relative degree of the output): The output is having full relative degree n for all (x,U) Based on assumption 3, and using Lie derivative transformation, Eq. (12.15) can be expressed as follows, ˙ ξˆi =ξˆi+1 . . ˆ p ˆ ˙ ξn =F(ξ ,U). ∑i=1 (Θi − Θˆ i ) ˆ y = ξ1

(12.17)

Where, F(ξ ,U) = Lnf F, the Lie derivative and ξ is the linear transformation of the states at the equillibrium point. The control objective is to force the system output y to track

236

S. Kamalasadan and A.A. Ghandakly

a given bounded reference trajectory, by using the only measurable command signal ξi in presence of parametric (Θ ) and functional (F) uncertainity. Assumption 4 (Reference Signal): The reference trajectory ym (t) and its r first derivatives are bounded. Remark 1: Based on the objective, system states ξ should track the states xm of an asymptotically stable linear reference model where Am and Bm are in controllable canonical form and r is a bounded reference input. 12.3.3

Controller Design

The above mentioned system dynamics needs a control law U such that F(ξ ,U) can be best approximated by a dynamic online RBFNN which act as a neuro-controller. Let ˆ ξ ,U), the RBFNN controller approximates the system such thata psuedo control ν = F( −1 ˆ ˆ is the best approximation. Assuming F is invertible, U = F (ξ , ν ) making ˙ ξˆi =ξˆi+1 . . ˙ˆ p −1 ˆ ξn = F ([νnn + Δ (ξ , ν )]. ∑i=1 (Θi − Θˆ i )) ˆ y = ξ1

(12.18)

Where, Δ represents the difference between the true and approximate dynamics. The pseudo control can be represented as ν = νnn + νad . Further, the overall system dynamics can be represented as ˙ ξˆ = Aξˆ + Bνad + W Φ + MΘ˙ˆ + χ (x,U) ˆ Θˆ ) , χ (x,U) = [0, ..., χn ]T , A and B are Where, M = ∂ ξn∂(x, Θˆ and W = [w1 , w2 , ..., wn ]T are the RBFNN weight vector.

(12.19)

in appropriate canonical form

The corresponding feedback linearized system will then be

μ˙ = Ac μ + Bc (ym − ξ1 )

(12.20)

Where, μ represents the linear system states and Ac , Bc and Dc are corresponding coefficient matrices. The overall error is defined as e = ξˆ − xm and the output error ε = y− ym . The tracking error dynamics can be written as e˙ = (A − BDcC)e + BCc μ + B(νnn − Δ ) 12.3.4

(12.21)

Stability Analysis

The overall stability of the system can be assessed based on the following assumption. Assumption 5 (Zero dynamics): The zero dynamics of the linearized system are locally stable with q locally Lipschitz in ξˆ . Remark 2: These zero dynamics are the subsystem of those of the true system. Thus, this assumption does not need the plant to be minimum phase which is required for most

12 A Novel Intelligent Control Approach for Precise Tracking

237

adaptive systems. For any RBFNN structure it is guaranteed that due to the universal approximation property there exists a set of bounded weights W and basis function Φ such that the nonlinear inversion error Δ can be approximated to a compact set Ψ by a linear in weighted parameter set. Let the adaptive laws be W˙ˆ = −F −1 φˆ T BPe Φ˙ˆ = −Wˆ BPeR−1 Θ˙ˆ = −S−1Π T BPe

(12.22)

Theorem 12.1. Consider the system of Eq. (12.15) satisfying assumption 1 and assumption 5 with parameterized vector as in assumption 2 and are unknown. Suppose the linear system Eq. (12.19) is formed and assume that the functions χ , Π and M are locally Lipischitz. Furthermore, assume all the signals in the resulting closed loop adaptive system remain bounded. Then, for the given trajectory ym , it follows that the controller U will track the system for an adaptive law as in Eq. (12.26). Proof: Consider a Lyapunov function as follows 1 1 1 V (e, Θ˜ ,W, Φ ) = e˜T Pe˜ + eT Pe + tr(W˜ T F W˜ ) + Φ˜ T RΦ˜ + Θ˜ T SΘ˜ 2 2 2

(12.23)

where P, Q, R and S are postive definite numbers. Then the derivative of (12.23) using (12.26) can be represented as V˙ = −e˜T Q1 e˜ − eT Q2 e − ε BPe + Π T Pe

(12.24)

Thus V˙ can be shown to be negative such that the upper bound will be V˙ = −|e|λmin Q|e| − |ε (x)||BPλmax||e| + |Π T Bλmax ||e|

(12.25)

P Qδ From (12.25) we can conclude that |e| ≥ λλmax min Thus V˙ is negative and guarantees input/output stability. This completes the proof •

12.3.5

Modifications of Adaptive Laws Based on NNPAC Theory

Based on the previous discussions the adaptive laws for RBFNN controller can be rewritten as follows W˙ˆ = −F −1 φˆ T BPe (12.26) Φ˙ˆ = −Wˆ BPeR−1 Similarly the adaptive law for the MRAC controller is,

Θ˙ˆ = −S−1Π T BPe 12.3.6

(12.27)

Online Growing RBFNN Design and Its Properties

As discussed before, the RBFNN structure is used along with the MRAC to learn the inverse dynamics of the plant. Let us now discuss some unique features of this intelligent algorithm.

238

S. Kamalasadan and A.A. Ghandakly

Fig. 12.5. Center movement and node growth in RBFNN structure

a) Selection of the node and node growth Inputs to the RBFNN controller are the plant output, input and the error. This input vector is represented as I which consists of derivatives of the desired states and the command signal. Initially the network starts with three nodes. The centers for these nodes will be initialized with a random number. Further, each time when a new input is passed on by the input layer, the distance and the outputs are calculated as   N 2  O j = ∑ exp − Ik − c( j,k)  /r2j

(12.28)

i=1

Where, j=1: number of nodes and k=1: number of input vector elements. It could be seen from Eq. (12.28) that if the new input vector falls close to any one of the existing centers, it will result in the output value extremely close to unity. Then, the value of the centers is adjusted to a new position such that both inputs are within their range. This is as shown in Fig. 12.5a. On the other hand, if none of the input vector values fall near the center, then the norm (•) will have a non-zero value and the output value falls below one. In this case the node number will be increased with additional center and radius located at a new input point as in Fig. 12.5b. Thus the center adjustment can be summarized as adding new centers if d j > threshold and max(e) > emin . b) Distance measurement The distance measurement is Euclidean in nature based on the norm. First, the distance between each input vector element is calculated with all the centers, and then, the norm is measured in the Euclidean space to get the distance with respect to each node for all the developed nodes. This can be represented as  N  i (12.29) d j = ∑ [Ik − c j,k ]2 k=1

Where, Ni is number of input vector elements, di is the distance.

12 A Novel Intelligent Control Approach for Precise Tracking

239

c) Calculation of the radius The radius is calculated from each center using r = max(d j ) ∗ μ

(12.30)

Where, μ is a distance factor less than unity. d) Selection of basis function Basis function is developed using the most commonly used Gaussian function expressed as

φ j = exp[−(

dj 2 ) ] rj

(12.31)

This function has been chosen due to the fact that it gives most weight to points that are close to the nodal center and exponentially less weight to the points, which move, farther from the node centers. Finally, the total network output O is calculated based on the summation of individual network ω output. e) Weight selection and online adjustment In order to sufficiently approximate any non-linear function, weights must be chosen correctly. To get the correct weights, RBFNN is ‘trained’ to learn the function that is being approximated. In order to train the weights properly, data regarding the inputs and the outputs need to be collected. This is accomplished as follows. The weights are initially adjusted to zeros. At each instant when a new value of the input/output vector is formulated, the weights are updated. The updated law is based on a gradient descent technique. If the error between the desired value and the output is represented as a set, then the weights are adjusted as w∗j = w j + σ ∗ ∇W y(x) ∗ d j

(12.32)

w∗j

Where, is the new value of the weights, w j the old value, σ is an update rate which is pre-defined number and, ∇W y(x) is the gradient of output with respect to the weight matrix. The input vector is fed to the network each time it appears while the system moves online. Thus initially, the RBFNN output will not be accurate. Eventually, as RBFNN starts learning the system inverse dynamics, the network output will become more and more accurate. f) Features incorporated to solve the curse of dimensionality One of the main drawbacks of RBFNN is its large network size in an offline setting or an alarmingly growing structure in an online environment mainly due to the node growth when new input arrives. This is termed as ‘Curse of Dimensionality’ Buhmann et. al. [15]. It is a known fact that as the system moves in its operating domain there is no need to use all the nodes at a given point of time as only a few of these nodes may be needed at a particular instant. In the NNPAC approach an algorithm based on Fabri et. al. [16] is developed, which is a modified version of ‘Clustering Algorithms’.

240

S. Kamalasadan and A.A. Ghandakly

Fig. 12.6. Development of active and passive nodes for dynamic operation

The process includes updating the relevant centers and subsequently the radii using an adaptive learning rate. Fig. 12.6 shows the effect of center movements and thus, the creation of active nodes as and when required. By this approach the number of nodes required at a given instant is kept at a minimum value as opposed to the static network nodal region. Steps below illustrate the procedure which will ensure that nodes are not increased immediately when new inputs close to the existing centers arrives. 1. Initially centers are chosen randomly as described before. Let these initial centers be represented as ci where (i=1, 2,...,m). Further based on a learning rate, the centers are modified every instant of time as ci (k) = ci (k − 1) + σ ∗ [I(k) − ci (k − 1)]

(12.33)

2. Then for each of the centers, Mean Square Error (MSE), between the actual and desired output for a certain sample is calculated . If this MSE falls above a threshold value (typically 0.01), the update rate (pre-defined as less than but close to unity) is reduced by a factor. The MSE calculation is based on Σ (y(k − i) − yd (k − i))2 ). 3. This process is continued for all the centers initializing the update learning rate to the original value. g) Issues related to node and error growth During the process of center adaptation, eventually the network nodes grow even though some nodes are passive and some are active. Thus, there is a need to prune passive nodes from the system after a certain period of time in order to control the node growth. To control the node growth, each nodal output is calculated first. Subsequently the ratio (α ) of the value of this output to the maximum (normalization) is evaluated. This can be denoted as Out put α= (12.34) Out putmax For a reasonable observation length, if this value falls below a threshold, then corresponding radii is pruned from further calculation.

12 A Novel Intelligent Control Approach for Precise Tracking

241

Even though the RBFNN structure is not dimensionally complex, one important aspect to be considered is the network approximation error. Since initial RBFNN network gives a random output, network approximation error will be high which can cause the plant to be unstable. In order to address the issue of reducing such approximation error in the network-bounded region, a method to increase the rate of convergence towards the minimum value(Chen et. al. [17]) is developed. The procedure applies a dead-zone function to the weight adaptation described as follows. 0 if e ≤ d0 D(e) = e − d0 if e ≥ d0 e + d0 if e ≤ −d0

(12.35)

This function is used in the weight update rule which will then be w∗j = w j + σ ∗ ∇W y(x) ∗ D(e) ∗ d j

(12.36)

Incorporation of a dead zone function to update the weights addresses two issues. First, at the initial stages, since the number of nodes and neurons is fixed, total error can be quite large. As it can be derived from Eq. (12.35), the dead zone function maintains the error within some threshold value. Secondly, if unattended, the error growth can finally reach a point that the system states will become unbounded and unstable. Once the system is linearized, then an adaptive controller will be able to control the plant even when there is still a change in the plant dynamics. In this essence the adaptive controller makes sure that system dynamics are stable and follow the reference model output. The task of preparing the plant output to track the trajectory and develop the control output to address the un-model dynamics, nonlinearity and functional uncertainty are accomplished by the neural control. 12.3.7

NNPAC Features and Algorithmic Development

The features of the developed algorithm as opposed to other intelligent adaptive structures reported to-date are as follows: 1. Ability to control systems that show mode swings and precisely tracks the plant output to a desired signal. 2. Ability to perform in presence of un-modeled dynamics, functional uncertainties, nonlinearities and parametric uncertainties. 3. Novel RBFNN architecture which is online, growing and dynamic in nature. 4. A new parallel control structure, which learns and adapts from the plant output based on need based intelligence using ISL concept. 5. Ability to enhance the architecture as remote controllers and autonomous agents and mitigates computational complexities. The functional flow chart of the NNPAC scheme is as shown in Fig. 12.7. The psuedo codes and algorithms are developed based on this figure. Thus, if there is any un-modeled dynamics during the course of operation of the system or the system mode swings, the online growing dynamic RBFNN controller cancels the nonlinearities and creates a stable linear system. On the other hand, at the

242

S. Kamalasadan and A.A. Ghandakly

Fig. 12.7. Block diagram of overall NNPAC with system under consideration

initial stage while the RBFNN controller is learning, the adaptive control stabilizes the system dynamics.

12.4 Application to Position Tracking of Robots Let us now look at the effectiveness of this new intelligent controller for tracking the desired command signal of the position trajectory in case of variations in tip load torque. In order to check the controller, a single link flexible manipulator model has been developed based on practical data. The intelligent controller is applied to track the desired value of the manipulator angular position generated as a command signal. To this end, the dynamic model as discussed in Kamalasadan et. al. [18] is utilized to track the command signal applied for eight seconds. To generate functional uncertainty, the tip load is varied arbitrarily at different time instant. The variation in the tip load contributes to the mode swings. The robotic arm has the position zero, x and y-axis denoted as x0 and y0 . At each time instant, depending on the angular movement and based on the required trajectory, there will be a deflection which causes the arm to settle to a new axes denoted as x1 and y1 . The corresponding angular deflection can be represented by the angle β .

12 A Novel Intelligent Control Approach for Precise Tracking

243

Table 12.1. Case 1:Tip Load Variation

Time Range(sec) 0-1 1-3 3-5 5-7 7-8 Load Torque(Nm) 0 20 10 17 0 Table 12.2. Case 1:RBFNN Controller Parametric Range

RBFNN Parameters Center Radius Distance Weight Max.Value -2.8,446.8,485.0 463.6 0.9 102.8 Min.Value -722.7,-703.1,-253.7 105.2 0 -27.3

Fig. 12.8. Case 1: Manipulator position output comparison

The movement of the arm basically depends on the mass and weight of the link. Further the arm tip will have the effect on the tip load which is denoted as the ‘Load Torque’. Physically the arm tip may not track the trajectory (denoted by the angle α ) due to these deflections. In order to create this effect in the robotic manipulator dynamic model, the load torque is changed online while the intelligent controller changes the input voltage to the manipulator each instant depending on the output error.

244

S. Kamalasadan and A.A. Ghandakly

12.4.1

Simulation Results and Discussions

Out of several cases conducted to track the desired command signal in case of arbitrary variation in the tip load torque at different time instant, three most important ones are presented next. In each of these cases tables associated with them, show proposed variation in the load torque. Further, the output angular position for desired time span is plotted for the proposed scheme and conventional MRAC (without intelligent supervisory loop) along with the desired trajectory. Corresponding, control contributions and RBFNN growth is also recorded. The reference model representation in all the cases has the following specific form, Wm (s) =

ωn2 s2 + 2ζ ωn + ωn2

Table 12.3. Case 2:Tip Load Variation

Time Range(sec) 0-3 3-3.5 3.5-6 6-6.8 6.8-8.0 Load Torque(Nm) 0 15 0 20 0

Fig. 12.9. Case 1:Control Contribution

(12.37)

12 A Novel Intelligent Control Approach for Precise Tracking

245

Where ζ is set to 0.7 and ωn varying for each case. In the first case the value of ωn for the single reference model is set as five. 12.4.1.1 Case 1 The tip load of this manipulator is changed at different time instants as in Table 12.1. Fig. 12.8 shows the position reference trajectory, the output of the robotic arm controlled by NNPAC and the output of robotic arm controlled by single reference model adaptive Controller (MRAC). It can be seen that the traditional single reference model adaptive controller failed to respond and subsequently became unstable at around seven seconds. However, with the NNPAC approach the output tracks the reference very well. Also, there has been significant contribution on the control value of RBFNN (Fig. 12.9) which emphasizes the importance and uniqueness of the proposed methodology. It is Table 12.4. Case 2:RBFNN Controller Parametric Range

RBFNN Parameters Center Radius Distance Weight Max.Value 7.5,1.3,4.3 474.8 1 4.2 Min.Value -7.2,-5.6,-3.3 67.1 0 -134.5

Fig. 12.10. Case 2: Manipulator position output comparison

246

S. Kamalasadan and A.A. Ghandakly

also worth noting that the node number increases (confirming network growth), starting at four nodes and ending at seven nodes. The details of the maximum and minimum values of the online RBFNN parameters such as centers, radius, distance and weight at the end of the eight seconds are as shown in Table 12.2. 12.4.1.2 Case 2 In Case 2, the tip load of this manipulator is changed at different time instant as in Table 12.3. Moreover, additional to the load variation as shown in Table 12.3, in order to create the un-modeled dynamics, a random signal has been introduced to the angular position throughout the simulation time of eight seconds. In this case, the single reference model adaptive controller failed to perform and showed instability when used with a second order reference model structure. Then, the proposed scheme is compared with a fuzzified moving multiple reference model Table 12.5. Case 3:Tip Load Variation

Time Range(sec) 0-1 1-3 3-5 5-7 7-8 Load Torque(Nm) 0 20 10 17 5

Fig. 12.11. Case 2:Control Contribution

12 A Novel Intelligent Control Approach for Precise Tracking

247

structure and an MRAC viz., FMRMAC. Fig. 12.10 shows the position reference trajectory plot and the robotic manipulator output with the proposed scheme and that of FMRMAC. It can be seen that the FMRMAC failed to perform and the proposed intelligent controller performed well in this condition. Fig. 12.11 shows the controllers (adaptive controller and RBFNN controller) contribution. It was also observed that the node changes occur dynamically from 4 to 15 nodes. Table 12.4 shows the RBFNN parametric values for this case. Thus, the main advantage of this novel intelligent controller is its ability to control systems with model swings and unmodeled dynamics. Further, it’s shown that the output of the plant tracked perfectly the desired trajectory. This is effective when there is a need for precision tracking, as the reference model(s) output will have a steady state error with the desired pattern always. The natural frequency ωn for this case has been kept as seven. 12.4.1.3 Case 3 It is important to understand and test the plant offline modeling constraints that are required to be taken care of before developing the reference model structure and applying the proposed controller.

Fig. 12.12. Case 3:Manipulator position output comparison

248

S. Kamalasadan and A.A. Ghandakly

In Case 3 the tip load of this manipulator is changed at different time instant as in Table 12.5. This case demonstrates the importance of developing suitable offline reference model and effect of of proposed scheme with a different reference model. Fig. 12.12 shows the position trajectory plot and the output of the robotic arm controlled by NNPAC with the adaptive controller having a reference model of natural frequency ωn equal to three. As it can be seen, the parallel control becomes unstable when a different reference model is chosen. Thus, it is apparent that the issue of appropriate reference model selection is critical and important for the successful performance of the proposed intelligent adaptive controller.

12.5 Real-Time Implementations and Experimental Results As a part of real-time implementation, the NNPAC concept has been applied to trajectory tracking of autonomous robot. The robot is controlled using an on-board microcontroller and a remote computer. Fig. 12.13 shows the overall implementation block flow diagram. The set up consists of a two wheeled robot equipped with two separate servo motors. The motors are controlled by a PWM signal from a remote microcontroller. The on-board robotic microcontroller board relays the control signal from the remote board using a communication link based on transceivers. The remote microcontroller interacts with a computer which runs the control architecture and delivers a digital signal converted using a DSP channel. Fig. 12.14 shows the overall implemenation architecture. 12.5.1

Design Details and Algorithmic Flow Chart

R environment. The design First, the proposed algorithm is implemented in Matlab structure is embedded in a Digital Signal Processor (DSP) board (Texas Instrument

Fig. 12.13. Overall implementation flow diagram

12 A Novel Intelligent Control Approach for Precise Tracking

249

Fig. 12.14. Overall Implementation Architecture

Fig. 12.15. Algorithmic Flowchart

TMS 320C6711) and the real-time digital output is then interfaced with a Motorola microcontroller (68HCS12). This remote microcontroller will allow us to send a PWM signal to a local microcontroller (68HCS12) board located on the robot via a transceiver. The algorithmic flowchart and design details are shown in Fig. 12.15. The Matlab routine will generate a control signal which is then digitized using DSP. This signal is been send every 10 msec to the remote microcontroller.The remote controller transmits the signal via a TLP 434A wireless transmitter to the local microcontroller. The digital signal which is a PWM output will be received using a data receive algorithm coded in the local microcontroller. Based on sensors located in the robots the speed will be encoded and sent back to the remote controller using a similar transmit flowchart and the digital signal is relayed to the PC through the DSP by the remote microcontroller. The overall execution time is 50 msec. The transmit and receive routines are embedded in the microcontrollers and uses an error checking and correction

250

S. Kamalasadan and A.A. Ghandakly

Fig. 12.16. Tracking error in PWM signal

12 A Novel Intelligent Control Approach for Precise Tracking

251

Fig. 12.17. Position trajectory plot with proposed controller and PID controller without disturbance

252

S. Kamalasadan and A.A. Ghandakly

Fig. 12.18. Position trajectory plot with (adaptive only) and with (supervisory loop) with disturbance

12 A Novel Intelligent Control Approach for Precise Tracking

253

subroutine. The A/D converters and serial communication interface is used for transmitting and receving data. 12.5.2

Experimental Results

Several tests were conducted to realize real-time implemenation of the proposed algorithm. In this section two important test results are discussed. First, the ability of the proposed architecture for a straight and turn experiment is discussed. For comparison a PID controller is utilized. Second, the ability of supervisory loop is assessed in the presence of external disturbance. For comparison the results with MRAC alone and with MRAC+RBFNN will be illustrated. Fig. 12.16 shows the error signal from the output which is proportional to the input PWM signal. The first part of the figure uses the proposed controller. As it can be seen the error deviation is minimal. The second part uses the PID controller and the error change is obvious. Please note that this is a real-time indication of the errors with wider pulse width indicating larger error. Fig. 12.17 shows the trajectory and the actual position during a 30 second time periond for the first case mentioned earlier. The robot was used to track a command signal generated from the PC as shown in Fig. 12.17. Fig. 12.17 also shows the culumative error in speed with the proposed controller and with a PID counterpart. The supervisory controller (as indicated as adaptive+RBFNN) reacted well and has the lowest error as indicated in the figure. Fig. 12.18 shows the case with disturbance. The comparison is with the MRAC alone and with MRAC+RBFNN. As it can be seen from the figures, the proposed controller reacted well in presence of external disturbance and normal tracking as well.

12.6 Conclusions In this chapter, a new approach to intelligent adaptive control based on a dynamic growing RBFNN in parallel with an adaptive controller has been presented. The NNPAC controller works better when compared to an adaptive controller in the presence of functional nonlinearity with the help of the novel growing dynamic RBFNN structure. The controller is best suitable for functional and parametric uncertain systems. The neural network algorithm is unique as it grows and adapts due to the change in nonlinear function. The adaptive controller contributes when there is a parametric uncertainty. Some of the features of this control approach are: ability to provide need based intelligence, reduced computational burden, ability to track online uncertainties, stability in the presence of functional and parametric changes and modularity.Several study cases for the position tracking of the robotic manipulator indicates that the NNPAC scheme tracks the position value when there is a change in the tip load without any predetermined calculations, even when those changes are of different patterns. Real-time implementations indicate the ability of the proposed controller for tracking a mobile robot in presence of parametric changes and disturbances. It is worth noting that this supervisory concept can be expanded to incorporate several reference model in the presence of multi-modal operation of the plant. This also shows the ability of the control concept to adapt based on system changes.

254

S. Kamalasadan and A.A. Ghandakly

Acknowledgement The first author would like to acknowledge National Science Foundation (NSF) USA, for providing the financial support through the CAREER grant ECS 0748238 for performing the theoretical work in this research direction.

References 1. Chen, L., Narendra, K.S.: Nonlinear Adaptive Control using Neural Networks and Multiple Models. Automatica 37, 1245–1255 (2001) 2. Calise, A.J., et al.: Adaptive output feedback control of nonlinear systems using neural networks. Automatica 37, 1201–1211 (2001) 3. Lo, J.T., Bassu, D.: Adaptive Multilayer Perceptrons With Long and Short-Term Memories. IEEE Transactions on Neural Networks 13(1), 22–33 (2002) 4. Arciniegas, J.I., et al.: Neural networks based adaptive control of flexible robotic arms. Neuro computing 17, 141–157 (1997) 5. Lange, D.B., Oshima, M.: Programming and Deploying Java Mobile Agents with Aglet. Addison-Wesley, Reading (1998) 6. Aridor, Y., Lange, D.B.: Agent Design Patterns: Elements of Agent Application Design. In: Proceedings of the Second International Conference on Autonomous Agents (Agents 1998), pp. 108–115. ACM Press, New York (1998) 7. Peine, H., Kaiserslautern, U.: Assessing the Performance Advantage of Mobile Agents in Distributed Searching. In: The fourth Plenary Workshops for CaberNet Members, Pisa, Italy, pp. 9–12 (2001) 8. Maturana, F., Staron, R., Discenzo, F., Scheidt, D., Pekala, M., Bracy, J., Zink, M.: An Interoperable Agent-based Control System for Survivable Shipboard Automation. In: ASNE Reconfiguration and Survivability Symposium 2005 (RS 2005), Sea Turtle Inn, Atlantic Beach, Florida (2005) 9. Kamalasadan, S.: A Novel Multi-Agent Controller for Dynamic Systems based on Supervisory Loop Approach. Special Issue on Soft Computing on Artificial Intelligence, Web and Data Mining, Machine Learning, Journal of Engineering Letters (March 2007) 10. Kamalasadan, S., Ghandakly, A.A.: A Neural Network Parallel Adaptive Controller for dynamic Systems Control. IEEE Transactions of Instrumentation and Measurements 6(5), 1786–1796 (2007) 11. Kamalasadan, S.: A Novel Intelligent Agent Supervisory Loop based Control Algorithm for Dynamic Systems. In: Proceedings of 2nd IICAI conference, pp. 178–197 (December 2005) 12. Kamalasadan, S., Ghandakly, A.A., Al-Olimat, K.S.: A Neural Network based Intelligent Model Reference Adaptive Controller. In: Proceedings of 2004 IEEE International Conference on Computational Intelligence for Measurements Systems and Applications, pp. 174– 179 (July 2004) 13. Kawato, M., Gomi, H.: A compuational model of the four regions of the cerebellum based on feedback-error-learning. Biological Cybernatics 68, 95–103 (1992) 14. Ioannou, P.A., Sun, J.: Robust Adaptive Control. Prentice-Hall, Upper Saddle River (1996) 15. Buhmann, M.D., Ablowitz, M.J.: Radial Basis Functions: Theory and Implementations. Cambridge University, Cambridge (2003) 16. Fabri, S.G., Kadirkamanathan, V.: Functional Adaptive Control: An Intelligent Systems Approach, chs. 1, 7-9. Springer, Heidelberg (2001) 17. Chen, F.C., Khalil, H.K.: Adaptive Control of a class of nonlinear discrete time systems. IEEE Trans. on Automatic Control 40(5), 791–801 (1995) 18. Kamalasadan, S.: A new generation of adaptive control: An intelligent supervisory loop approach, Ph.D. Dissertation (August 2004)

13 Punctuated Anytime Learning to Evolve Robot Control for Area Coverage Gary B. Parker Connecticut College, New London, CT, USA [email protected]

Abstract. Evolutionary computation is a powerful learning method that has successfully been used to evolve control programs for autonomous robots. Nevertheless, its practical use has been limited by a significant problem. Most evolutionary computation methods operate on a population of possible solutions, which all must be tested to attain their fitness. Testing the entire population of solutions on the actual robot is impractical for all but the simplest problems. Conducting the tests on a simulation of the robot significantly saves time and wear on the robot, but requires an accurate model of the robot, which is in itself is a difficult task, and does not accommodate for changes in the performance of the robot, which are commonplace once it is in operation. In this chapter, we present a method for linking the simulation to the actual robot to allow the learning system to learn even when the simulation is not completely accurate and adapt the control program for changes in the robots capabilities. The methods viability is demonstrated by its application to learning the control program for an actual hexapod robot’s performing area coverage.

13.1 Introduction Pre-programming the on-board controller for an autonomous robot can yield the desired results, but makes development laborious and does not provide a means for real-time adjustments necessitated by miscalculation or degradation of the system. Development time can be reduced and the adaptability required for autonomy provided by learning control through some form of evolutionary computation (EC). However, it is typically too computationally intense to be carried out on board the robot. This is especially true when the robot is small with an on-board controller that is purposely simple to accommodate weight and expense restrictions. An off-line learning system that is capable of making adjustments to the on-board controller needs to be employed so that the robot can adapt to changes in real time. Most forms of evolutionary computation require that a population of possible solutions be tested over several iterations. This necessity for multiple trials can be excessively time consuming and detrimental to the wear of the robot if conducted on the robot. A simulation of the robot can be used for training, but does not guarantee that the resultant controller will be correct for the actual robot. This issue is a significant topic of discussion and research in the area of evolutionary robotics. The previously established techniques for evolving control programs for robots can be divided into three general approaches. The first is to do all of the training off line on a simulation and transfer the D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 255–277. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

256

G.B. Parker

results to the actual robot when it is complete [1, 2, 3]. Significant attention must be paid to the simulation to ensure that it accurately models the robot in order to assure the usefulness of the results. The time and effort put into the model can sometimes exceed the work that would be required to program the controller by hand. In the second method, all of the training is done on the actual robot [4, 5, 6]. If the task can be completed and the fitness accurately judged in minimal time, this is a viable option. It precludes the need for any simulation. The problem is that training using evolutionary computation takes num-generations × num-individuals × time-to-complete-task time to complete the training. Assuming the num-generations and num-individuals are set by the nature and complexity of the problem, the time-to-complete-task is the most probable factor to influence the time for training. In most cases the time-to-complete-task is significantly longer when the tests are done on the robot, which precludes this method except for very simple tasks. The third method is to start the training off line and transfer training to the robot for the remaining generations [7, 8]. This does not require the same level of simulation accuracy, although a model is still required. In addition, since the EC learning curve typically starts out steep dealing with a starting population that is filled with randomly generated solutions, this method passes through the initial phase with faster training on the simulation preserving time for more accurate training on the robot as the system tweaks the solution. Nevertheless, several generations are still required on the robot, which can take significant time for all but the simplest tasks. The problem with all of these techniques is that either time has to be to put into the simulation or into the training on the actual robot. In addition, none of them are designed to be able to refine the solution while the robot is in operation unless it is only doing the task being learned. A help would be to use EC to evolve the characteristics of an adaptive learning system for the robot. An example is the work done by Floreano and Urzelai [9] in the evolution of plastic control networks. They evolve the learning characteristics of the neural network as opposed to the control weights. This allows the system to use neural network learning methods to complete the controller and makes it adaptive to changes in the environment. The extra means of learning lessens the amount of learning required by EC, which reduces the number of generations required for training. This could contribute to whatever method is used for evolving the robots controller, yet does resolve the issues addressed earlier. A possible solution when there are several robots working concurrently to perform the same task is call embodied evolution, which uses each robot as an individual in the population that is evolving [10]. Recombination of genes takes place between robots in close proximity to each other. Since all of the solutions (one per robot) are being tested concurrently, the num-individuals factor is effectively equivalent to one. This method reduces the total time for training, but requiring a robot for each chromosome of the EC population causes another limitation. Since the typical problem being solved by EC requires population sizes greater than 50 individuals, this method can only be employed where several robots are doing the same task or when the task is simple enough that lower population sizes will suffice. Judging from the previous research completed in evolutionary robotics, some form of simulation is required to solve real-world problems. What is needed is a method for maintaining the accuracy of the simulator, a way of coupling the simulator to the actual

13 Punctuated Anytime Learning to Evolve Robot Control for Area Coverage

257

robot [11, 12, 13]. The research reported in this chapter describes punctuated anytime learning, a method for making the simulation/robot link. This method is distinct from the previously established techniques in that it uses periodic tests on the actual robot to improve the accuracy of training done in simulation. The application used to test this learning system is the learning of a control program for a robot that is performing area coverage.

13.2 Area Coverage Path planning is the formation of the set of moves that the robot will take to transport itself from a starting point/configuration to a goal point/configuration. One specific category of path finding is referred to as area coverage, which is concerned with ensuring that the robots path brings it over a specified percentage of points in an area. Some applications are mine sweeping, search and rescue, haul inspection, painting, and vacuuming. 13.2.1

Previous Research

Previous research in area coverage has concentrated primarily on covering a specified area while contending with obstacles within that area. Zelinsky et al. [14] divided the area into cells that were marked with the distance to the goal and used an extension of path planning to form a cell to cell path through the area. The tests were done on a wheeled robot, Yamabico, which could execute precise movements for dead reckoning. Ultrasonic sensor sightings of landmarks were used to re-adjust its position when required. Choset and Pignon [15] divided the area into obstacle free sub-areas and found an exhaustive path through the adjacency graph representing these sub-areas. Within each sub-area, back-and-forth boustrophedic motions (Figure 13.1) were used to assure that the obstacle free area was properly covered. The tests were done in simulation and on a Nomadic 200 wheeled robot. The results were good in simulation, but tests on the actual robot using only dead reckoning resulted in tracks that were not perfectly parallel. In further work, Acar et al. [16] developed demining methods to improve the robot performance and [17] used Morse functions to locate the sub-area boundaries in a method referred to as Morse decompositions. Ollis and Stentz [18] used a Demeter harvester robot, which used boustrophedic motions to achieve full coverage while doing automated harvesting. Hofner and Schmidt [19] used templates appropriate for the type of robot to determine the best path within obstacle free sub-areas. This gave the robot the option of doing other motions rather than boustrophedic if it lacked the capability to make the required turns. Icking et al. [20] proposed a method for determining the best path to take as the robot visits a set of cells within the search area. The problem assumed that the robot has a limited distance vision sensor. Even though the robot did not know the environment, they show that their strategy produced tours that cover the area yet are of minimal length. Jager and Nebel [21] present a dynamic and decentralized method for a team of cooperating robots to partition an area for complete coverage. All of these works assume that the robot has precise control and/or some means of navigational that allows it to make corrections in its movements. However, precise

258

G.B. Parker

Fig. 13.1. Back-and-forth boustrophedic motions and search area for coverage

control and navigational sensors are not always available. Legged robots, especially inexpensive ones with minimal sensors and precision of movement, cannot be positioned perfectly with exact headings. What is often taken for granted in these papers, a capability to perform perfect search patterns such as back-and-forth boustrophedic motions within a sub-area, is not easily done by legged robots. Although calibrated wheeled robots can be commanded to turn in a circular arc for a specific time to a specific heading or rotate a specific number of degrees, simple legged robots can only turn for a specified number of gait cycles with one of a variety of turns. With these robots the exact time and rate of turn cannot be specified. To further complicate matters, a straight gait, which is intended to yield a straight track, often results in a small drift to one side or the other due to performance differences from one side to the other of the robot. The best straight-line path may actually be attained by calling for a minimal turn. Efficiency is also an issue in search patterns since differing turn rates can have vastly different effects. The most efficient path to cover the area given depends greatly on the capabilities of the robot. Although applicable to most all robots, this is especially true for legged robots. If the robot can efficiently rotate or turn sharply, its best strategy may be to do a ladder pattern (boustrophedic with square turns). If tight turns cost in efficiency, it may be better to make large sweeps with some coverage overlap or to buttonhook the ladder turns, keeping the rungs about the same as a standard boustrophedic pattern. Both efficiency and a path that properly covers the area despite the imprecision of the robot are important for legged robots performing area coverage. A means of learning the optimal cycle of turns and straights could compensate for the lack of calibration in robot turning systems and greatly improve the efficiency of area coverage within cells (sub-areas). 13.2.2

Research Environment

For the area coverage problem used in this research, the robot was to fully search, starting from a specific point, an area of specific width. The area was not bound on one side since the robot was judged by the area covered in a set amount of time. The purpose was to find the most efficient cycles of behavior required to cover as much area as possible in the time allowed. The area width was purposely small to force the controller to produce a cycle of movements that could be repeated continually to cover any area length.

13 Punctuated Anytime Learning to Evolve Robot Control for Area Coverage

259

The search area was 180cm in width. The search was for simulated mines that would be fully contained in the area. In order to detect a mine, the robot had to have the entire width of its body (excluding the legs) at its mid point, within the same 60x60 cm square as the mine. For test purposes, 60x60 blocks with mines were placed to completely fill the area. The robot’s task was to find as many mines as possible while ensuring that no mines had been missed. There were no physical constraints requiring the robot to stay within the mine area, so its movement was not restrained in any way by the environment. The simulated search area, which was 180x∞ cm, is shown in Figure 13.1. The mine blocks are shown as squares within the area. The actual test area was a 180x210 cm area of the floor where the 60x60 cm mine blocks were marked by tape. Tests in this area were done by observing the squares covered as the robot completed the first 50 gait cycles of its search path. Through these observations it was determined which mines were detected.

13.3 The ServoBot The robot used for this research was the ServoBot (Figure 13.2), a robot developed by David Braun to facilitate legged robot control experimentation. The robot’s six legs each have two degrees of freedom (up/down, forward/backward). The twelve hobby servos, two per leg, provide thrust and vertical movement. A control program made up of sequences of primitives and control instructions is transmitted from a workstation to the controller through connecting lines or wireless. The controller (BASIC Stamp II) stores and executes the sequence of primitive instructions, which designate the pulses to be sent to the individual servos to control the movement of the legs. 13.3.1

Gait Cycles

A gait for the ServoBot is made up of a control sequence, which is a list of activations that the on-board controller will continually repeat. Each activation is a twelve bit number that contains a signal to control the instantaneous movement of the twelve servo

Fig. 13.2. Photograph of a ServoBot with an BASIC Stamp II controller

260

G.B. Parker

actuators. The activation can be thought of as six pairs of actuations or control signals. Each pair is for a single leg with the first bit of the pair being that leg’s vertical activation and the second being that leg’s horizontal activation. The legs are numbered 0 to 5 with 0,2,4 being on the right from front to back and 1,3,5 being the left legs from front to back. A repeatable sequence of these activations can be evolved by a cyclic genetic algorithm (CGA) to produce an optimal gait for a specific ServoBot [22]. The gait that the CGA learned for the test robot was a tripod gait, where legs 0, 3, & 4 alternate with legs 1, 2, & 5 in providing the thrust for forward movement. While one set of legs is providing thrust, the other set is repositioning for its next thrust. In the case of the robot used, the entire cycle lasted for 58 activations with each set of legs providing 29 activations of thrust. 13.3.2

Production of Turns in Gait Cycles

Gait cycles of differing degrees of turn were provided through the use of affecters. These affecters interrupted activations to the thrust actuators for either the left or right side of the robot. The normal gait consisted of a sequence of 29 pulses of thrust to move the leg from the full front to full back position. Interrupting these activations by sending anything less than 29 would result in less thrust movement of the legs on that side of the robot. Affecters from 0 to 15 were possible. A one bit indicator specified if the affecter was right or left. A four bit number indicated the strength of the effect. The affecter four bit number was multiplied by 2 to determine the number of thrust pulses on that side. This yielded a possible span from 0 to 30; the 30 was reduced to 29 to maintain the optimal gait cycle. For example, a right side affecter of 7 would allow only 14 (2 × 7) thrusts on the right side while keeping 29 on the left. The result would be that the left side would move further than the right resulting in a right turn. An affecter of 0 would result in no thrust on that side producing a maximum turn; 15 should result in a straight track, since it was normal for a tripod gait.

Fig. 13.3. The model of the robot is a list of the results of the 32 possible gait cycles

13 Punctuated Anytime Learning to Evolve Robot Control for Area Coverage

261

Developing a set of gait cycles (Figure 13.3, each of which was made up of 58 activations, was accomplished by assigned each an affecter, which resulted in a turn throughout that cycle. A single gait cycle was defined as one step, being complete when the legs returned to their starting positions (in this case, after 58 activations). Each gait cycle started with all the legs on the ground, with legs 0, 3, & 4 full forward and legs 1, 2, & 5 full back. As the gait cycle started, legs 0, 3, & 4 would provide the trust as legs 1, 2, & 5 would start to lift and move forward to reposition for their thrust after 29 activations. Once the trust provided by legs 1, 2, & 5 was complete (simultaneously legs 0, 3, & 4 had repositioned), the single gait cycle was complete. 13.3.3

Simulated Robot

In order to run the genetic algorithm on a simulation and test the punctuated anytime learning method, a model of the robot needed to be developed. The robot was simulated by maintaining its current state, which was made up of its xy position in the area and its orientation (heading). The robot’s resultant movement after it performed the 32 possible gait cycles was stored in a list (Figure 13.3). Each element of the list could be identified by its gait cycle number (five bits). The high order bit of the gait cycle describes whether the turn will be left (1) or right (0) and the remaining four bits indicate the affecter (corresponding to the level of turn). The list (F T Δ H) of three numbers after that indicates the result of applying that gait cycle for one cycle. The model was formed by running each gait cycle on the actual robot for four cycles, measuring (Figure 13.4) the change in position/orientation, and dividing the result by four to determine the movement after a single cycle. F represents the distance in centimeters that the robot moves forward in a single cycle. The F axis is defined as the heading of the robot before movement. T represents the distance traveled left or right. The T axis is defined as a perpendicular to the F axis. Left movement results in a negative T, right in a positive T. Δ H represents the change (in degrees) in heading from the start heading (F axis) to the heading after execution of the gait cycle. Left is negative, right is positive. The values in the model represent the average turn rate of each gait cycle. For the sharpest turns, those where the affecters are less than 3, only three cycles were used to make the measurements since four cycles resulted in turns of greater than 90◦ .

Fig. 13.4. Gait cycle turn values: F, T, and Δ H

262

G.B. Parker

13.4 Cyclic Genetic Algorithms The cyclic genetic algorithm (CGA) [23], which was developed to produce control programs for the cyclic behavior of robots, is a variation of the classic genetic algorithm (GA) [24]. One of the major differences is in what is represented in the chromosome. In both the GA and CGA the chromosome is made up of genes. In the GA each gene or a combination of them usually represents some characteristic of the individual (Figure 13.5). The CGA incorporates time into the chromosome structure by making each gene a task that is to be accomplished in a set amount of time. In evolving a better chromosome, the CGA is developing a single loop program. Each gene represents a task that can be as simple as a primitive command (machine instruction or robot actuation) or as complicated as a subprogram or another cycle. Although CGAs can have either fixed or variable length chromosomes, we only used fixed length for this application. The fixed length chromosome is designed such that it can produce variable length programs. This is done through the use of two parts to the task. One is a primitive instruction or activation and the other is a repetition, which designates the number of times to repeat the activation. The number of times that each task is repeated can vary the total number of tasks in the iteration. The CGA uses genetic operators that are similar to those used with any other GA. One general characteristic, however, is that crossover in the iterative section is a twopoint crossover. This is done because the iterative section is considered a cycle, so two-point crossover is like removing a segment from one cycle and replacing it with a corresponding segment from another cycle to form a new offspring. If one point crossover were used, the second point of crossover would always be between the last and first gene in the cycle.

Fig. 13.5. GA and CGA chromosomes

13 Punctuated Anytime Learning to Evolve Robot Control for Area Coverage

13.4.1

263

CGAs Applied to Area Coverage

For the area coverage problem the genes of the CGA represented a set of gait cycles with a corresponding number of repetitions [25]. The trained chromosome contained the cycle of these gait cycles that was continually repeated by the robot’s controller to produce a path that was to efficiently cover the designated area. The program for the robot controller can have up to nine changes of gait cycles, each repeated up to 63 times, to produce a search program. Since there are 32 possible, it takes five bits to designate the gait cycle. The repetition number for each gait cycle could be anywhere from 0 to 63 so six bits were used for representation. The CGA chromosome directly resembled the required input to the controller. Each chromosome was made up of nine genes and each gene was made a gait cycle number and a repetition. The scheme representation of the chromosome is shown in Figure 13.6. The first number in each pair represents the gait cycle while the second represents the number of times to repeat that gait cycle.

Fig. 13.6. Area coverage chromosome

Fig. 13.7. Fitness calculation. This track would result in a fitness of 4.

The CGA used stochastic (roulette wheel) selection based on the fitnesses of individuals in the population. Fitness was calculated by counting the number of mines detected (mine blocks covered by robot’s path) after it had completed a specified number of gait cycles (Figure 13.7). Counting, which was not done until the search path was completed, began by rows from the bottom of the area. If a mine block was missed, no mines in subsequent rows were counted, although the mines from the partial row were counted. However, this did not prevent search patterns where mines were missed during a first pass, but covered in a later pass. For the fitnesses

264

G.B. Parker

calculated during training, mines visited more than once did not increase fitness, although they did count for row completions. This was to discourage paths that were wasting time re-searching covered area. Once a fitness was calculated for each individual in the population, pairs were stochastically selected for reproduction (crossover and mutation). Gene-by-gene evaluation, a genetic operator peculiar to CGAs, was used to clean up the chromosome by randomly picking one or two individuals from the population on each set of trials and examining each gene one at a time. Genes were evaluated moveby-move by comparing the previous move’s fitness to the present. Fitnesses for this operator were not computed in the same way as for CGA selection. The total area covered (assuming a 20 cm sensor coverage width) within the search area, not the number of mines detected, was calculated for each single gait cycle. This differing fitness calculation was necessary because it gave an immediate fitness for each gait cycle. Since search patterns would sometimes take several gait cycles between each mine detection (even if the search was being done in the most efficient manner), the usefulness of a single cycle would be difficult to predict using the normal CGA (mine blocks covered) fitness calculation. Judging from this area covered fitness, genes that performed poorly in their current position were eliminated and genes that were good in the execution of their early repetitions and subsequently dropped in the later repetitions were modified by reducing their repetitions. Genes that had zero repetitions were moved out so that only active genes were at the start of the cyclic section. Momentarily elevated mutation was initiated whenever it was determined that the CGA had converged prematurely. This determination was made by checking the improvements in the best fitness of the population. It is assumed the CGA is stuck (or at least taking too long) when there is no improvement in over 50 generations of training. The assumption is that when the learning stalls it is due to premature convergence and many of the population’s individuals are nearly identical. At this point, momentary elevated mutation is used to introduce several small changes while retaining important sub-solutions within the population. Although a high rate of mutation can be very disruptive to each individual, with enough identical individuals the good building blocks should survive at least somewhere within the population. Continued evolution can use these to rebuild the original individual, but with the added chance of generating a superior individual due to the introduced perturbations. 13.4.2

Tests

The test area (Figure 13.1) was simulated by an xy grid where point (0,0) was the lower left corner. The lower right corner of the area was the point (180,0). The lower boundary was at y = 0, the left boundary was at x = 0, the right boundary was at x = 180, and there was no upper boundary. Mines were considered to be in 60×60 square blocks. The first row had centers at (30,30), (90,30), and (150,30). The second row started at (90,30), etc. The robot’s start position was placed at (45,30). This location assured acquisition of the first mine and put it in a good starting place to acquire the first row of mines. Motion was determined by applying each gait cycle from the chromosome one at a time. Using the current xy position and heading of the robot, a new position was calculated by applying the forward (F) and left/right (T) movements stored for that gait cycle as described in

13 Punctuated Anytime Learning to Evolve Robot Control for Area Coverage

265

Section 13.3.3. The new heading was an addition of the current heading and the gait cycle heading change (Δ H). The path was not restricted from going outside of the area and the calculations remained the same if it did. This allowed, if appropriate, for the robot to do its turns out of the area so that it could attempt straight tracks within the area. An initial population of 64 individuals, made up of chromosomes described in Section 13.4.1, was randomly generated. Each individual, representing a cycle of gait cycles that would form a path, was tested to determine its fitness after 100 of these gait cycles were executed; 100 gait cycles were enough to ensure that some turning was required to cover optimal mines. It was not, however, enough to force the formation of a cycle that could provide continuing full coverage. Due to this limitation, the required number of gait cycles was randomly (with a 50% probability) increased to 200. This allowed for faster fitness computations while using 100 gait cycles, yet put selection pressure on the population to evolve individuals capable of performing well at 200 gait cycles. The CGA was run for 5000 generations with the best solution (individual chromosome) saved whenever there was an increase in fitness (more mines covered in the allotted time). Five initially random populations were each trained using the CGA. All five populations resulted in a search pattern that was similar to the back-and-forth boustrophedic pattern. One solution looked as though it was almost optimal for 200 gait cycles. Tests were done on the actual robot to see how well it performed in comparison to the model. The robot’s pattern of movement was similar to that of the simulation although its performance was not as good due to inaccuracies in the model.

13.5 Punctuated Anytime Learning Punctuated anytime learning addresses two main issues: anytime learning without immediate feedback and dynamic simulator accuracy in evolutionary robotics. 13.5.1

Anytime Learning

Dean and Bobby [26] defined anytime algorithms to be algorithms that could be interrupted at any time and return a solution. This made them particularly applicable to robot control since an anytime algorithm is always ready with a current best answer as it strives to find the optimal answer. Extending this thought, Grefenstette and Ramsey [27] used the term anytime learning to refer to a method of learning that has the characteristics of an anytime algorithm, except that it never stops; it continually gives a meaningful answer any time the solution is improved. As part of a robot control system, anytime learning integrates the execution (control) module and the learning module. A simplified version of the anytime learning system is shown in Figure 13.8. There are two main modules. The Control Module contains the control program, which directs what actions to take in response to the input sensors. The Learning Module’s function is to use an internal simulation of the robot and its environment to learn a better control program. Feedback from the Control Module can be used to update the internal representation in the Learning Module.

266

G.B. Parker

Fig. 13.8. The anytime learning system for robot control

The anytime learning system was used by Grefenstette and Ramsey to solve a simulated problem of cat-and-mouse. The system was to be operating in a mobile agent (the tracker or cat) that was to track a second mobile agent (the target or mouse) that was pre-programmed for avoidance. The task of the tracker’s learning module was to compensate for the changing speed characteristics that it could sense in the target. Anytime learning allowed the learning component to continually compute a best control solution while the robot operated in the environment using the latest best solution. This system worked well for a robot that could carry both the operations and learning components on board. An extension of anytime learning was needed for robots that are simple enough that their learning component cannot be carried on board. This is a reasonable configuration for teams of expendable robots. They need to have enough computing power to provide autonomous control, but do not want to carry the significantly more equipment on board to support learning (especially learning with a GA). With an extension to anytime learning, the learning component can be off-line while the operations component, which is on the robot, receives periodic downloads of the best solution. Grefenstette’s system could adapt quickly to changes in the robot or the environment by having the robot’s sensors continually update the status of the environment and the robot’s capabilities in the learning component’s simulation. This is where a difficulty develops when this methodology is applied to remote learning, where the learning system can only depend on global observation, instead of feedback from precise sensors, to determine the robot’s capabilities. The modifications to anytime learning were needed to compensate for the lack of quick feedback, precise sensors, and high computational power in robots. 13.5.2

An Extension to Anytime Learning

Punctuated anytime learning is a solution for the needed modification to anytime learning and the model verses robot issue discussed in the Introduction. Training with EC takes place off-line on a simple model while periodic checks on the actual robot help to improve the learning system’s output. This type of anytime learning is referred to as punctuated because the learning, although it is continuous, has break points where it is working on newly updated information about the actual robot.

13 Punctuated Anytime Learning to Evolve Robot Control for Area Coverage

267

Fig. 13.9. Punctuated anytime learning system where the learning module cannot be on the robot

A punctuated anytime learning system is shown in Figure 13.9. In this system, since the Learning Module is not part of the robot, it no longer has direct input to the Control Module. Updated control programs are sent via some form of external communication. In addition, feedback is no longer sent via a direct link. It can be transmitted, such as the control program, if there is an evaluation unit on the robot. Alternatively, it may be provided to the Learning Module from an outside observer (such as an external overhead camera [28]). The second option becomes more likely as the complexity of the robot goes down since it would carry a less sophisticated evaluation unit and communications transmitter. There are two versions of punctuated anytime learning: the co-evolution of model parameters and fitness biasing. To help discuss the two types of punctuated anytime learning, let’s consider the three main parts of a learning system in evolutionary robotics. There is the actual robot, the simulation (the model), and the genetic algorithm (or some other form of EC). The simulation is designed before training to be as accurate as possible given the time constraints in development and the simplicity desired to speed fitness computation. The GA is set up for training and initiated with a random starting population. The program in the controller of the actual robot is determined by the other two components of the system. The physical capabilities of the actual robot cannot be altered by the system, only the program in its controller (which is the purpose of the system). The periodic checks on the actual robot during punctuated anytime learning can be used to alter either the simulation or the GA in an attempt to improve the result of training. 13.5.3

Fitness Biasing

For fitness biasing the results of periodic tests done on the actual robot are used to bias the fitnesses calculated by the GA, which uses a model of the robot. Figure 13.10 shows how punctuated anytime learning with fitness biasing allows the system to modify the GA based on the performance of the robot. The model is not affected; its parameters were set before anytime learning began. The periodic checks on the actual robot alter the processing within the GA to improve the result of training. This involves the biasing of evolved solution fitnesses by comparing their performance on the model to their performance on the actual robot.

268

G.B. Parker

Fig. 13.10. Fitness biasing in the learning system

As in a standard ER learning system, the probability for selection is normally determined by computing each individual’s fitness on the model. This fitness is computed for each individual in the population. With fitness biasing, after each n generations all solutions of the GA population are tested on the actual robot. The results are used to bias the fitnesses found on the model to make them equal to the actual robot fitnesses. These biases are used as the GA continues training. In this way, solutions that are good on the actual robot but poor on the model have boosted fitnesses during training, which results in their production of more offspring. This solution requires population-size actual tests every n generations. The bias for each solution of the population is stored with its corresponding solution. It is used in subsequent generations of the GA to alter the fitness of the solution computed on the model of the robot. This is done by multiplying the fitness computed on the model by the bias. In the first generation after the biases are set, this corrected fitness is equal to the actual fitness for each chromosome. As offspring are produced, they inherit the averaged bias of their parents. Examples of the use of punctuated anytime learning using fitness biasing are discussed in previous work on gait generation [29] and the learning of leg cycles [30]. 13.5.4

The Co-evolution of Model Parameters

For the co-evolution of model parameters [31] periodic tests of evolved solutions on the actual robot are used to co-evolve the accuracy of the robot’s model with the GA produced solution. Figure 13.11 shows how the co-evolution of model parameters allows the system to modify the model based on the performance of the robot. The periodic checks on the actual robot alter the model to improve the result of training. After each n generations the best and two other solutions, which are selected from the current population or are generated from the best solution, are tested on the actual robot. The results are used to judge the accuracy of a population of model parameters. This population can start out either as randomly generated individuals or as a combination of perturbations (to varying degrees) of the original model parameters. The most accurate model is used for the controller evolution; the population of model parameters will continue to evolve until interrupted by updated actual test information. This solution requires three tests on the actual robot every n generations. A pseudocode representation of the co-evolution of control and model parameters is shown in Figure 13.12.

13 Punctuated Anytime Learning to Evolve Robot Control for Area Coverage

269

Fig. 13.11. Co-evolving model parameters

Fig. 13.12. Simplified pseudo code of the co-evolution of model parameters. In this case both the Control Evolution (CE) and the Model Parameter Evolution (MPE) are running on their own processor and continue while the robot is in operation. After each n generations of evolution, the CE performs 3 tests on the robot. It then interrupts the MPE to send the results and get the latest best model from the MPE. If the latest best model is better than the current simulation being used, this simulation is replaced. After this, the MPE continues evolution and the CE continues for n more generations of evolution before the next test on the robot.

The form of evolutionary computation used in this work to co-evolve the model parameters is a basic genetic algorithm. A parameter population of individuals is randomly generated before training begins. Each individual is made up of a set number of genes. Each gene represents a corresponding field in the robot’s model. These genes evolve to produce models that correspond in performance to the actual robot. The fitnesses of the individuals in the parameter population are computed using tests on the robot. Three representative solutions are selected from the solution population; the best (using the current best robot model) and two other solutions are used. The two additional solutions can be generated from the best solution or selected randomly from the solution population or through some other criteria, such as the current worst

270

G.B. Parker

Fig. 13.13. Standard algorithm used in the co-evolution of model parameters to compute a model’s fitness. When used for the area coverage problem, S2 and S3 are generated as variations of S1.

solution. Each of these three solutions is run on the actual robot to gain a true fitness. This fitness should be general enough so that it can be acquired by external observation (such as an overhead camera). These three true fitnesses are used to evolve a more accurate model. Each chromosome of the model parameter population is used to compute its fitness using the algorithm in Figure 13.13. In the case shown the second solution is randomly selected and the third is selected by the worst-solution criteria. S1 is the best solution, S2 is a solution that is randomly selected, and S3 is the worst solution from the solution population. M1 is the fitness of the S1 solution when run on the model, M2 is the fitness when run using S2, and M3 is the S3 fitness. The R1, R2, and R3 are the robot’s fitnesses using the corresponding solutions. The value for Fit-Comp is computed using a formula that compares the accuracy of the fitnesses when run on the model as opposed to the robot. This computation is problem specific because it depends on the parameters being evolved; more details will be given for this in Section 13.6. After a parameter fitness is computed for each individual in the parameter population, selection is done stochastically. The couple selected produce a single offspring through crossover and mutation. This is done enough times to produce an entirely new parameter population. The co-evolution of model parameters is designed to be used in either of two ways. One is to have both the solution evolution and the model parameter evolution be operating concurrently on separate processors. The second is to schedule alternating evolution of both GAs on the same processor. They should both yield similar results although the former would be considerably faster. In the work reported in this chapter, the alternating method was used since the experiments were done on a single processor.

13.6 Learning Area Coverage Using the Co-evolution of Model Parameters Section 13.4 discussed a method for learning the cycle of turns and straights required for a hexapod robot to solve the area coverage problem. Although successful in simulation, the learning system needed an anytime component to make it adaptable enough

13 Punctuated Anytime Learning to Evolve Robot Control for Area Coverage

271

to be used on the actual robot. This section discusses the creation of a viable learning system by making a adjustment to the CGA and adding the punctuated anytime learning technique of co-evolving model parameters. The solution population is made up of chromosomes as described in Section 13.4. The parameter population is made up of chromosomes that each look like Figure 13.3, with the F, T, and Δ Hvalues being evolved. 13.6.1

Partial Recombination Used in the Evolution of Model Parameters

The basic GA was found to be inappropriate for evolving the model parameters. Since the control program used up to nine gait cycles, 23 of the 32 gait cycles would have no bearing on the fitness. During model parameter evolution they would be altered as a side effect while the nine or less being evolved tended toward an optimal. This could result in the loss of vital building blocks required to evolve the gait cycles destined for future use. In addition, since some means of keeping related building blocks close to each other should be advantageous (Holland, 1975), related gait cycles needed to be together during evolution. Although several gait cycles may be related by their probable use together in the execution sequence, predicting these relations in advance is difficult. Both of these problems are addressed by using partial recombination. In a standard GA, the chromosome is fixed and related building blocks can be separated by significant distances. One means of lessening this gap is to group related building blocks together, but this requires that knowledge about which are related. Another solution, messy GAs [32] can be used to learn this ordering. In the model parameter learning problem, the related building blocks cannot be predetermined, but they become better known during runtime. There is no need to learn the best building block positions; they can be assigned during learning. They do change, however, so re-assignments must be possible. Partial recombination solves this problem by extracting the needed gait cycles for application of the genetic operators. These gait cycles (each of which could be considered a building block) were used to build a new chromosome by placing them in the order they were needed in the CGA generated turn cycle solution, which was probably the best guess for related order. The genetic operators were applied to this partial list of all the gait cycles for the designated number of generations. Upon completion of training, they were re-inserted into the main model parameter population. This allowed for training on the appropriate gait cycles (building blocks) arranged in the proper order without disturbing the rest of the building blocks. Co-evolution starts when the best solution from the solution population is sent to the model parameter GA by the controller generating CGA. This will be the S1 from Figure 13.13. Two more solutions (S2 and S3) are generated using this best solution. One is a perturbation of up to ±1 on each of the non-zero repetitions in the turn cycle. The other uses a .25 probability starting from the first gene to find a gene that it sets the repetitions to 50. The partial recombination GA chromosome is built by extracting the needed gait cycles from each individual of the model parameter population. The three control solutions (the best found by the CGA, plus two that were generated) are each run on the actual robot and on the 64 partial robot models. The fitness of each is judged by comparing (finding absolute difference) its performance to the actual robot’s performance. Two figures are compared for each gait cycle solution – the number of

272

G.B. Parker

blocks covered and the number covered only once. This being done on the three control solutions results in six total differences. Each of these is squared and the 6 values are added together to get the fitness (Fit Comp from Figure 13.13). This fitness is used to perform the standard GA operators of selection, crossover, and mutation. After 40 generations, each partial chromosome is re-inserted into its main chromosome with the best chromosome designated as the current model for further CGA training. 13.6.2

Punctuated Anytime Learning in Simulation

In order to test the punctuated anytime learning system in simulation, the gait cycle list of the model needed to be different from that of the simulated robot, since in simulation, this gait cycle list defined the movements of the robot. A “correct” gait cycle list that was different from the one created through the capability measurements was generated to represent the simulated robot. It made all of the turn strengths less than expected by shifting each turn rate to be off by one. Gait cycle n of the new list would be equivalent to gait cycle n + 1 of the old list. The max turns (strength 0 for both left and right) were thrown out and the turns with strength 14 were equal to strength 13 turns. This new gait cycle list was used to simulate an actual robot that did not match with the model. This simulated robot did not turn as sharply as the training model indicated so all the path planning solutions would be slightly off. This could accurately represent an actual situation where the robot lost some of its turn capability in all turns. The co-evolution of model parameters was tested using 5 distinct parameter populations. Each parameter population of 64 gait cycle lists was generated by perturbing the original gait cycle list (Figure 13.3) used for CGA training. For the solution populations, the final (5000 generation) solution populations for area coverage generated by the CGA were used for continued training. The punctuated anytime learning system tested all of the individuals in the solution populations using the original gait cycle list (the best known model at the time). This test was done for 50 gait cycles (steps) as opposed to 100 or 200 since this is what would be done on the actual robot. The best solution plus two perturbations of it, as described in Section 6.2, were used (tested on the simulated robot) to find a fitness for each individual (gait cycle list (model)) in the parameter population. A genetic algorithm with partial recombination was run for 40 generations to evolve the parameter population. At the completion of this training, the best model from the parameter population was used to replace the original model in the simulated robot. The CGA was run again for 40 generations on the solution population using this new model to evolve a new best path planning solution. This process was continually repeated for a total of 1000 CGA generations. The best path planning solution (judged by its performance on the best know model at each 40 generation mark) replaced the simulated robot’s operational solution if its performance was better on the robot. The co-evolution of model parameters was applied to these resultant populations (5 pairs, each made up of a unique solution population and a unique parameter population). A graph of the results is shown in Figure 13.14, which shows the number of blocks covered after 50 gait cycles are executed. The simulated robot, which was the “corrected” model, was used to test the best solution (plus two perturbations) for fitness calculations used to co-evolve the model parameters. Although the same simulated robot was

13 Punctuated Anytime Learning to Evolve Robot Control for Area Coverage

273

Fig. 13.14. Simulated punctuated anytime learning using the co-evolution of model parameters. Each iteration is equal to 3 trials on the robot, 40 generations of model training using the parameter population, and 40 generations of path planning training using the solution population.

used for all five tests, the starting solution populations differ since they originated from the previous test. In addition, the model parameter populations differ since they were randomly generated. Each line represents the solution results of one of the population pairs tested as training was conducted using punctuated anytime learning with the CGA learning the turn cycle control sequence and the GA with partial recombination learning the model parameters. As the best model was updated using the evolution of the parameter population, it was stored for use by the CGA to learn the best solution. As the best solution (control program for the robot) was updated it was stored in the operational side of the control system where it remained in effect until a new control solution evolved that was found to be superior. The graph lines trace these best control solutions as the robot’s performance continually improved. As can be seen, four of the five populations reached a fitness of 10 by the sixth iteration of the learning system. Each iteration consisted of three tests on the robot, 40 generations of model evolution, and 40 generations of control evolution. One of the populations was stuck at a fitness (blocks covered) of 9, but after 16 iterations finally reached 10, which was judged to be optimal when using 50 gait cycles. The heavy solid line shows the average performance of the best control solutions. The heavy dashed line indicates what the average fitness of the five populations without punctuated anytime learning – there is no improvement over time. Since the solution populations used in this test originated from the solution populations generated in the previous test (no anytime learning test from Section 13.4), they had already evolved for 5000 generations on the static model; continued training yielded no improvement. This heavy dashed line can be compared to the heavy solid line, which shows the

274

G.B. Parker

average when punctuated anytime learning is employed. Now since the best model is continually being updated, the CGA has the latest best model (a more accurate training environment) for evolution, which continually yields improved results. 13.6.3

Punctuated Anytime Learning on the Actual Robot

The co-evolution of model parameters on the robot was done in essentially the same way except that the tests were performed on the actual robot instead of the simulated robot, which was simulated by a correct gait cycle list. These tests, which will be automated in future research, were performed by an observer who ran the robot using each of the three path plans in the test environment. The blocks covered were recorded for each run and entered by hand for continued training. Training was stopped after the robot covered 10 blocks, which was considered to be optimal for 50 gait cycles. Figure 13.15 shows the results of using punctuated anytime learning on the actual robot. A few things are represented differently in this graph. Although the heavy lines still represent the same information (dashed is the no anytime learning average best, solid is the anytime learning average best), the population lines now show the evolved path planning solution at each test. These each end when the number of blocks covered is 10 because that is when training stopped. The best overall path plan is still stored in the robot’s operational side, but not shown on the graph. It is, however, used with the other populations’ best overall plans to calculate the heavy solid line. Pops 1, 2, & 4 (Pops 1 & 4 are co-located) would be the same if the best was always shown since the path planning solution improved at every test. Pops 3 & 5 however, went through some poor solutions before they found the optimal. For example, the Pop3 solution was good on the first two block rows but then had insufficient right turn to stay in the third row. After the robot tests and 40 more generations of training, the

Fig. 13.15. Actual learning using the co-evolution of model parameters

13 Punctuated Anytime Learning to Evolve Robot Control for Area Coverage

275

CGA increased the rate of turn. This may have taken care of the problem, but at the same time it cut some repetitions from the second straight path (probably to increase efficiency), which resulted in a turn from the second row before all the blocks were covered. The CGA corrected this error after 40 more generations gaining a fitness of 9. Making several small adjustments over the next six iterations of training it tried a more drastic change at the nine iterations mark. This again dropped the fitness, but upon rebuilding, it gained a nice path that covered 10 blocks.

13.7 Conclusion Punctuated anytime learning is a method of learning that allows an off-line system equipped with some form of evolutionary computation to evolve control programs for controllers on board autonomous robots. This system requires only minimal computational power on the robots while providing them with the learning needed to allow them to be adaptive to changes in capability and the environment. The learning systems can be equipped with enough computational power to handle evolutionary computations in a reasonable period of time because they will not have the weight or cost restrictions of the expendable exploratory robots. Two methods of punctuated anytime learning were discussed. Fitness biasing uses tests on the robot to bias fitnesses of solutions evolved on the learning system’s model and the co-evolution of model parameters uses tests on the robot to concurrently evolve the model’s parameters while the CGA evolves the control solution. Area coverage path planning provided an interesting problem for an evolutionary computation learning system, such as a CGA, to solve. The setup was such that only a cycle of actions (turns and straights) could produce a usable solution. The learned cycle of actions provided the basis for a robot controller to repeat its pattern of movement as required to cover an area of any specified length. The addition of punctuated anytime learning, although it improved the practical usefulness of the system, was mainly intended to check the ability of the co-evolution of model parameters to provide real-time corrections on an actual robot being trained using evolutionary computation. Tests showed the success of the system. In simulation, the average increase in blocks covered after training with the co-evolution of model parameters was nearly two times its starting average. Tests on the actual robot also confirmed the success of this type of punctuated anytime learning that only needs global observation to improve the outcome. All five tests showed that this technique could result in near optimal paths (covering 10 or more blocks in 50 gait cycles) with no more than 39 trials on the robot. Four of the five tests attained 10 blocks within twelve trials on the robot. This system moves the work in learning away from the robot allowing it to go about its operations while the path planning takes place off line. These tests demonstrate that a punctuated anytime learning system based on the coevolution of a path plan using a cyclic genetic algorithm and the parameters of a simulation model using a genetic algorithm with partial recombination is an effective means of learning adaptive control strategies for hexapod robots performing area coverage. The CGA was successful in generating strategies for five out of five random start populations within 5000 generations. The punctuated anytime learning based on the co-evolution of

276

G.B. Parker

model parameters was shown to successfully adapt all five populations to the specifics of the robot. Tests in simulation and on an actual robot confirmed the method’s viability.

References 1. Jakobi, N.: Half-baked, ad-hoc and noisy: minimal simulations for evolutionary robotics. In: Advances in artificial life: Proceedings of the 4th european conference on artificial life (1997) 2. Schultz, A., Grefenstette, J., Adams, W.: RoboShepherd: learning a complex behavior. In: Proceedings of the robots and learning workshop (RoboLearn) (1996) 3. Lee, W.-P., Hallam, J., Lund, H.: Applying genetic programming to evolve behavior primitives and arbitrators for mobile robots. In: Proceedings of IEEE 4th international conference on evolutionary computation (1997) 4. Harvey, I.: Artificial evolution and real robots. In: Proceedings of international symposium on artificial life and robotics (AROB), pp. 138–141 (1996) 5. Mondada, F., Floreano, D.: Evolution of neural control structures: some experiments on mobile robots. Robotics and autonomous systems 16 (1995) 6. Mahdavi, S., Bentley: An evolutionary approach to damage recovery of robot motion with muscles. In: Seventh European conference on artificial life (2003) 7. Lund, H., Miglino, O.: From simulated to real robots. In: Proceedings of IEEE 3rd international conference on evolutionary computation (1996) 8. Miglino, O., Lund, H., Nolfi, S.: Evolving mobile robots in simulated and real environments. Technical report, Institute of Psychology, CNR, Rome (1995) 9. Floreano, D., Urzelai, J.: Evolution of plastic control networks. Autonomous robots 11(3), 311–317 (2001) 10. Watson, R., Ficici, S., Pollack, J.: Embodied evolution: distributing an evolutionary algorithm in a population of robots. Robotics and autonomous systems 39(1), 1–18 (2002) 11. Husbands, P., Harvey, I., Cliff, D., Miller, G.: Artificial evolution: a new path for artificial intelligence? Brain and cognition 34, 130–159 (1997) 12. Mataric, M., Cliff, D.: Challenges in evolving controllers for physical robots. Evolutional robotics, special issue of robotics and autonomous systems 19(1), 67–83 (1996) 13. Nolfi, S., Florano, D., Miglino, O., Mondada, F.: How to evolve autonomous robots: different approaches in evolutionary robotics. In: Proceedings of the international workshop on the synthesis and simulation of living systems (1994) 14. Zelinsky, A., Jarvis, R., Byrne, J., Yuta, S.: Planning paths of complete coverage of an unstructured environment by a mobile robot. In: Proceedings of international conference on advanced robotics, pp. 533–538 (1993) 15. Choset, H., Pignon, P.: Coverage path planning: the boustrophedon cellular decomposition. In: Proceedings of the international conference on field and service robotics (1997) 16. Acar, E., Zhang, Y., Choset, H., Schervish, M., Costa, A., Melamud, R., Lean, D., Graveline, A.: Path planning for robotic demining and development of a test platform. In: Proceedings of the 3rd international conference on field and service robots (FSR), pp. 161–168 (2001) 17. Acar, E., Choset, H., Rizzi, A., Atkar, P., Hull, D.: Morse decompositions for coverage tasks. The international journal of robotics research 21(4) (2002) 18. Ollis, M., Stentz, A.: Vision-based perception for an autonomous harvester. In: Proceedings of the IEEE/RSJ international conference on intelligent robotic systems (1997) 19. Hofner, C., Schmidt, G.: Path planning and guidance techniques for autonomous mobile cleaning robot. Robotics and autonomous systems 14 (1995)

13 Punctuated Anytime Learning to Evolve Robot Control for Area Coverage

277

20. Icking, C., Kamphans, T., Klein, R., Langetepe, E.: Exploring simple grid polygons. In: Proceedings of the 11th international computing and combinatorics conference (2005) 21. Jager, M., Nebel, B.: Dynamic decentralization area partitioning for cooperating cleaning robots. In: IEEE international conference on robotics and automation (ICRA 2002) (2002) 22. Parker, G.: Evolving gaits for hexapod robots using cyclic genetic algorithms. International journal of general systems 34(3) (June 2005) 23. Parker, G., Rawlins, G.: Cyclic genetic algorithms for the locomotion of hexapod robots. In: Proceedings of the world automation congress (WAC 1996), Robotic and Manufacturing Systems, vol. 3, pp. 617–622 (1996) 24. Holland, J.: Adaptation in natural and artificial systems. The University of Michigan Press, Ann Arbor (1975) 25. Parker, G.: Evolving Cyclic control for a hexapod robot performing area coverage. In: Proceedings of 2001 IEEE international symposium on computational intelligence in robotics and automation (CIRA 2001), pp. 561–566 (2001) 26. Dean, T., Boddy, M.: An analysis of time-dependent planning. In: Proceedings of the seventh national conference on AI (AAAI 1988) (1988) 27. Grefenstette, J., Ramsey, C.: An approach to anytime learning. In: Proceedings of the ninth international conference on machine learning (1992) 28. Lund, H., Cuenca, V., Hallam J.: A simple real-time mobile robot tracking system. Technical paper 41, Dept of AI, University of Edinburgh (1996) 29. Parker, G., Mills, J.: Adaptive hexapod gait control using anytime learning with fitness biasing. In: Proceedings of the genetic and evolutionary computation conference (GECCO 1999), pp. 519–524 (1999) 30. Parker, G.: Fitness biasing to produce adaptive gaits for hexapod robots. In: Proceedings of the 2004 IEEE/RSJ international conference on intelligent robots and systems (IROS 2004), pp. 2843–2848 (2004) 31. Parker, G.: Co-Evolving model parameters for anytime learning in evolutionary robotics. Robotics and autonomous systems 33(1) (October 31, 2000) 32. Goldberg, D., Deb, K., Korb, B.: Don’t worry, be messy. In: Proceedings of the fourth international conference in genetic algorithms and their applications (1991)

14 Local Methods for Supporting Grounded Communication in Robot Teams Nathan Wiebe and John Anderson Autonomous Agents Laboratory, Department of Computer Science, University of Manitoba, Winnipeg, Manitoba, Canada R3T2N2 [email protected]

Abstract. For a mobile robot to be able to communicate usefully with others in a group, the references it makes to points in space must be grounded in concepts that are shared among the group. In the past it has been common to hand-construct a complete set of such groundings, either by individual enumeration or by enforcement of a common coordinate system and origin among all team members. Such assumptions remove the ability to add new robots with no knowledge of the environment in an ad hoc manner, and also require knowledge which may not be available. In an urban search and rescue (USAR) setting, for example, robots may be released into rubble from a collapsed building with no shared starting point for an origin, under conditions where GPS reception is disrupted. Preconstructed groundings are also anthropocentric in that they are a best guess by humans as to what is useful from their perspective, and may be nothing like what robotic agents would come up with on their own. This chapter describes the an approach that allows a group of robotic agents to develop consistent shared groundings for useful locations in an environment over time, using only local communication and interaction. This approach is thus suitable for domains in which broadcast communication may be sporadic, such as USAR, or jammed, such as military applications. The evaluation of this approach, which compares several different grounding techniques, shows that a consistent set of shared groundings can be developed effectively by a team of robots over time using only local interactions, and that these improve the effectiveness of communication in a multi-robot setting.

14.1 Introduction: Grounded Communication in Multi-robot Systems In order to realize the benefits of deploying a team of robots, individuals are often required to communicate to coordinate their activities. While the benefits of communication to everyday human cooperative activities are easily observed, communication has also been empirically shown to be of great benefit when solving problems in multi-agent systems. Matari´c [16] showed that communication was necessary to deal with credit assignment problems in multi-agent learning, for example, while Balch and Arkin [3] showed the advantages of employing communication in team-based foraging tasks. Even in domains such as robotic rescue, where interference and disruption of infrastructure may limit communication to short ranges or line-of-sight, communication can still be effective in improving the group navigation and mapping of a robot team. Communication, however, can only be of use to the degree that the symbols used for communication are associated with (grounded to) concepts in the environment D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 279–301. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

280

N. Wiebe and J. Anderson

Fig. 14.1. 16 robots in an 8m x 8m domain

consistently among communicating agents. The problem of creating and maintaining these groundings is known as the symbol grounding problem [14]. In most multi-agent domains, groundings across a population of agents are hand-constructed. While this is certainly possible to do for small environments, it becomes unwieldy very quickly as the complexity of the environment increases, and simply will not scale to environments of any reasonable size [15]. Such hand-constructed groundings are also biased to the perspective of the humans creating them, as opposed to what might be arrived at by the agents in the domain themselves [15]. The problem of scale in providing groundings becomes much more significant when agents are mobile. In mobile settings, robotic agents must typically communicate spatial information, and thus have potentially a very large number of grounded physical locations that they might wish to communicate to others. Typically a shared coordinate system and the ability to localize well within it are provided by system designers, so that all references to locations become absolute. Robots entering a collapsed building from different starting points, for example, might share a common GPS coordinate system and know one another’s starting location to exchange partial maps in a setting such as that shown in Fig. 14.1. When such a shared coordinate system is unavailable, being able to reference spatial locations in communication between members of a group in a sensible way becomes a very difficult problem. If GPS were unavailable, for example, which could happen due to interference (e.g. in a disaster setting), locale (e.g. underground) or nonexistence (e.g., in a military situation where signals may be jammed, or in space exploration applications), each robotic agent would have to begin with its own coordinate system. Common references would then have to be built up across a population over time, so that an agent could refer to a location by name and have others understand it in terms of their own internal coordinate systems. The difficulty inherent in requiring multiple agents to learn groundings is that in isolation, different agents will naturally ground locations using different symbols, which is obviously not suitable for the purposes of shared reference. Moreover, when agents begin at different coordinates and orientations, there is no way to immediately relate shared groundings. Consider Fig. 14.2, which shows

14 Local Methods for Supporting Grounded Communication in Robot Teams

281

Fig. 14.2. Three robots in an office environment in our implementation. Left, objective view; Center/Right, internal maps of two robots, each with a different internal coordinate origin and a different orientation from the objective (90◦ and 210◦ ).

an objective domain in our implementation, and the internal world maps of two robots. Each has a different starting point (cartesian plane origin) and begins at a different orientation, and the same world is represented very differently in each. What is needed is a mechanism to develop shared groundings and reconcile these into a consistent set across the population, in spite of individual differences between agents. The bounded resources at each agent’s disposal preclude simply sharing each and every grounding among all agents. Communication to agree on consistent symbol grounding in any complex domain would be combinatorially more significant than the amount of communication necessary for problem-solving itself, and in domains where broadcast communication is sporadic, would be physically impossible. Broadcasting all groundings would also require some means of dealing with the differences in perspective between agents illustrated in Fig. 14.2 as well. However, building up common-sense shared references to locations (and other entities) over time is a much more human-like approach to the problem, and one that serves as a more generally intelligent solution. A team of robots using such an approach would be able to function anywhere, and would be able to be developed independently of one another and form teams in an ad hoc manner. The ability to form teams without a great

282

N. Wiebe and J. Anderson

deal of prior knowledge assumed between individuals is an important element in developing robot teams broadly adaptive to many applications. Beyond these issues, another motivation for allowing the development of shared groundings by a population of agents is avoiding developer bias. The hand-constructed groundings that have been used in most systems thus far reflect anthropocentric (humancentered) categorizations, which can limit the resulting system in ways that would not occur if those categorizations were made by the system itself [15]. In designing a behaviour-based system [2], for example, designers of place linguistic labels on the robot’s internal behaviours, such as follow-wall. These have meaning to the system designer and affect the way the designer thinks about the system. However, the agent performing the behaviours carries none of the label semantics used by the designer: internally the complex interactions between a collection of such behaviours may be operating in a very different manner than might be assumed given their labels. This leads to misunderstandings and errors on the part of developers who use these labels to understand the system. A rift forms between what the designer intended and what the system actually does [15]. Similar issues exist with symbol groundings. Choosing the entities worth grounding may be obvious when one is designing a small system. As systems become larger and more complex, however, not only does selecting all groundings become impossible, but the designer-system rift comes into play. The groundings chosen by the designer are based on his or her own knowledge of only the isolated elements of that system, and of how the agents are expected to perform. Were agents to evolve their own set of groundings, those might be very different from the groundings envisioned by the human designers. They would arguably be more effective as well, since unneeded groundings that designers might supply would not develop, and groundings would be tailored to the activity at hand [15]. This chapter describes the an approach by which an ad hoc team of robots may explore an environment and develop a consistent set of shared spatial groundings for the purpose of useful communication. A number of strategies for creating groundings are presented, and the development of a consistent set of groundings across a team is achieved using the local interactions that occur in the course of performing useful work, without sharing a coordinate system. A location grounding, for our purposes, is a symbol that is associated with an agent’s internal spatial coordinates, allowing the agent to make reference to a location in the environment (and ultimately be able to go there or refer to it in reasoning or communication). A shared grounding is one where the same symbol refers to the same absolute location in the world, even though the agents sharing the grounding may have completely different representations of that location internally. The approach presented here begins with no prior knowledge of the environment or shared points of reference, requires no explicit location labelling phase, requires only local encounters between agents rather than broadcast communication, does not flood the environment with groundings, and is demonstrated to be functional with larger populations (up to 16 robots) than previous work (e.g. [15], which uses only 2 robots). This approach is then evaluated in a domain where communication about locations in the environment can improve performance, and is shown to provide significant benefits, indicating that agents using this approach

14 Local Methods for Supporting Grounded Communication in Robot Teams

283

can reap the benefits of grounded communication without predefined groundings. The evaluation compares the alternatives employed under varied environmental conditions and team sizes. There are a number of ways in which references can be made to entities in the environment in a grounded fashion [9]. An iconic reference is an association based on observable features (e.g. seeing a fire and associating that sensor data with that concept). An indexical reference associates two icons (e.g.smoke may be an index for fire), while the most powerful is a symbolic reference, which is an arbitrary relationship between icons, indices, and/or other symbols. While associating sensory phenomena to internal representations in an iconic manner is a relatively straightforward form of learning, the other two types require much more sophistication. The approach described here demonstrates the use of both indexical and symbolic references. The remainder of this chapter overviews related work in this area, describes our approach and its implementation, and then describes the results of an empirical evaluation.

14.2 Related Work Much practical reasoning employs symbols as placeholders for objects in the physical world. The groundings between perceivable objects and the symbols used to refer to them are commonly referred to as anchors, and the problem of developing and maintaining anchors for individual mobile agents through perception (a subset of the broader symbol grounding problem) is known as the the anchoring problem. Coradeschi and Saffiotti presented a preliminary formal logic solution to the anchoring problem in 2000 [6], defining the major components in predicate logic. They later extended this work to use symbols in actions and plans [7]. While this work provides a formal framework for maintaining anchors to symbols via perception in single agent systems, it does not deal with the multitude of practical issues that arrive when the shared groundings necessary for communication in multi-agent systems are considered, since the primary problems in group are not only deciding whether something is worth grounding, but reconciling the multitude of inconsistencies among the groundings of various agents. Comparatively little work in grounding has been performed in multi-agent systems, because of the problems of developing and maintaining consistency of groundings among agents. Steels has done work in evolutionary linguistics, examining how individual agents in a group may be able to generate discrimination trees to distinguish one object from other in the environment, without the aid of a teacher [17]. Vogt [18] later improved on this work. These works are more limited than the work we present here, in that they operate with only a very limited number of objects in the environment, and the domain itself exists only to perform groundings as opposed to having groundings acquired over the course of activity. However, these do show that it is possible to allow agents to develop shared symbol associations in coherent ways. Billard and Dautenhahn [5] studied the benefits of social skills to learning in heterogeneous multi-agent systems. In their environment, robotic agents were provided with a following behaviour, allowing them to keep a similar sensory context to another robot by maintaining physical proximity, and learned to associate words with color descriptions of flooring after receiving demonstrations by following a teacher

284

N. Wiebe and J. Anderson

broadcasting the correct color name. While agents could eventually become teachers, the language used never changed: groundings were never invented, merely spread from an individual teacher through the population. In the approach described in this chapter, all agents are both teachers and learners all of the time, since each is developing useful groundings in the course of its own work, as well as trying to spread those groundings to others in order to be able to communicate about those spatial locations. This is thus suited to unknown environments where an existing competent teacher cannot be assumed. Jung and Zelinsky [15] describe the implementation of a heterogeneous cooperative robotic cleaning task which benefits from the use of a symbolic references for communication. Two robots were required to clean a laboratory floor: Flo sweeps litter into piles, while Joh vacuums the piles to remove them. Their system adds three layers of behaviours for coordination between agents, each of which improves performance as it is added. First, Joh is given the ability to track Flo visually (useful here since Flo will likely be near a pile of litter). Next, Flo is given the ability to signal the location of a new litter pile to Joh, in coordinates relative to itself. Joh and Flo use the same wheel encoders, providing a pre-defined shared grounding for the communicated coordinates. Finally, communication using predefined symbolic references is employed. Each reference is communicated in terms of the angle and distance from a reference line connecting two points in the robots’ environment, thus allowing a new point to be referred to using two points that are already known. This requires that robots go through an explicit location labelling phase in order to develop a common set of reference points so that communication can be understood. While this work shows that a human-centered coordinate system is unnecessary for navigation and communicating about locations, and that symbolic references are very useful for coordinated activity that takes place across a spatial area, the approach employed is not practical for problems of any significant size. Labelling locations is done by arbitrarily labelling enormous numbers of points (in excess of 500 for a 3.4 x. 5.1 m room) during a phase in which both robots devote their full effort to collectively labelling locations (i.e. there is never a situation where one robot grounds a location independent of the other and then shares that grounding later). The number of points grounded is impractical for any everyday activity, but more importantly in a multi-robot setting, is impractical to make consistent across any significant population. This is unrealistic compared to most situations, where individuals are likely to take note of points that are interesting to them during the course of useful work, and share these with others when it is convenient, as is done in the scheme we present in the remainder of this chapter.

14.3 Developing Grounded Spatial References Any scheme for developing grounded spatial references among a team of agents requires three distinct policies: deciding when a particular location is worth grounding by an agent (i.e. allocating some label and maintaining an association for the agent’s own activities); deciding when such a grounding is worth sharing with others; and deciding how to reconcile groundings when they conflict with those shared by others. We present these in the separate subsections that follow, with specific implementation details

14 Local Methods for Supporting Grounded Communication in Robot Teams

285

provided in Section 14.4. In our implementation, labels are random integers and individual coordinate systems are cartesian planes, but these techniques place no restrictions on the representation used for labels or coordinate systems, and no shared representation for coordinates is assumed. 14.3.1

Grounding Locations

Whether any given location in an environment is worth maintaining a correspondence with a label is largely dependent on the task a robot is performing. If one is cleaning a room, one will consider different locations to be of significance than if one is painting it. During these and many other tasks involving mobility, however, a robot is navigating the room, and so we attempted to develop several approaches that will suit this task and thus indirectly assist in others. The general operation of each scheme is described briefly below, and implementation details of each are found in Section 14.4.1. The relative utility of these schemes in practice is examined in Section 14.5. The first of the strategies we employ, label-at-meeting, creates a shared grounding between a pair of agents at the current location when they encounter one another in the environment. This is a basic strategy that takes advantage of chance encounters that will inevitably occur in multi-robot domains. The dynamics of this strategy result in more groundings in areas where robots tend to spend most of their time, since encounters are more common in these areas. When a robot R1 encounters another robot R2 , it stops and invites R2 to ground a symbol. If R2 is amenable (since it may be busy with other things), it requests a label for R1 ’s current location. R1 names the location and the new label is subsequently used by both. A delay is built into the implementation (Section 14.4) to stop this from occurring too frequently so that agents are not taken away from their primary task in this environment. While label-at-meeting is domain independent, it is also limiting in that an agent is restricted to creating a grounding only when another agent is present. The second strategy we developed, label-spatial-entropy, is more general in that it tries to predict, in a domain-independent way, which locations in the environment are likely to be useful. To make this prediction, we note the previous use of information theory in categorizing space for path planning. Baltes and Anderson [4] define the entropy of a spatial area (referred to as spatial entropy in this work), a metric that uses information theory to measure how mixed a given portion of the environment is in terms of free and occupied space. Spatial entropy is highest where there is the largest mixture of open and blocked areas (i.e. where the environment contains the most information). Here we employ this measure to choose useful locations for grounding: when an entropy threshold is exceeded, a grounding is generated (Section 14.4). To contrast these general techniques and examine the effect of providing more specific information to agents ahead of time, we also define the label-environment-feature technique, where specific phenomena useful to the task at hand are explicitly described to the agent and grounded when these are recognized. This is an anthropocentric attempt to capture useful domain-specific properties, such as doorways and hallways. It also serves as an upper bound for the quality of groundings that can be made in a given domain.

286

N. Wiebe and J. Anderson

14.3.2

Sharing Groundings

The label-at-meeting strategy is the only one of those above that contains some means of sharing a grounding among agents, and this is limited in that only the two agreeing to make the grounding possess it. To share groundings among agents generally, some means beyond mere communication must be provided. Since each agent may employ its own internal coordinate system, communicating these coordinates would be meaningless to others. Our approach employs the physical demonstration of groundings, to allow other agents to ground a location by perceiving it themselves. Demonstrations are initiated when agents encounter one another. If a robot R1 perceives another robot R2 within a specific distance, and a known grounding is also within a given distance, R1 will offer to demonstrate it to R2 . If the second agent accepts the offer, it will signal R1 and begin to follow that agent as it moves to the demonstrated location. When R1 arrives at the location of the grounding, it will then send a message to R2 indicating the label it employs to designate the location. If this location is novel to the R2 , that agent will also ground the location to that label, and will signal R1 that is has done so. This offer-and-acknowledge approach allows either agent to avoid participating in the demonstration if other tasks take priority, and in turn support balancing the importance of the overall task of the team with the benefits that another shared grounding might bring. Further details of our implementation of this strategy may be found in Section 14.4.2. 14.3.3

Consistency among a Group of Agents

An agent may learn many groundings by demonstration, but since each agent is also learning from others and making groundings itself, it will encounter many inconsistencies as well. The most important part of creating a set of groundings among a team of agents is mediating these conflicts appropriately as they are discovered, in a decentralized manner in which the computation is entirely local. Some mechanism is needed so that when groundings are shared, any conflict in labels or locations will be more likely to be resolved toward greater consistency among team members as a whole, rather than away from it. In our approach, this is done by maintaining a reference count for each grounding within each agent. This reference count indicates, to the best of that agent’s knowledge, the number of agents in total that employ this same grounding. This is a heuristic value, because it is maintained based only on local encounters rather than global communication. Many of the agents with which it has been shared will have since gone on to demonstrate the grounding to others, without the originator of the grounding being able to know this. Others which which the grounding has been shared may have since had a conflicting grounding replace this information. When performing a demonstration as described above, the demonstrating agent (R1 ) will send its reference count to the encountered agent (R2 ) in addition to communicating the symbol used, and will update this count by one if R2 accepts its demonstrated grounding. In the case of a conflict during a demonstration, reference counts of the symbols involved allow a local determination of which is likely to be more widespread, and thus a means of encouraging less widely used groundings to be discouraged and those more widely used to be propagated.

14 Local Methods for Supporting Grounded Communication in Robot Teams

287

In practice, there are three types of conflicts to be resolved. This section overviews these at a high level, and leaves the implementation details to Section 14.4.2. The most obvious (Case 1) is that R2 may know the location grounding already, but under a different symbol, either because it has learned it from a third party or developed it itself. In this case, the agent with the lowest reference count gives up its grounding and adopts that with the higher reference count, and each updates the count accordingly. As this reference count becomes higher, it will eventually propagate to other agents and replace competing labels for the same groundings across the population. In the case of equal reference counts, the grounding of R2 will be used in both agents: this leaves the two agents consistent in terms of this grounding after an encounter, and serves to add some change to the population and eventually cause one grounding to dominate the other. The second type of conflict (Case 2) occurs when the R2 recognizes the symbol communicated, but already uses that symbol for a different location grounding. Here, consistency demands that one of these groundings must ultimately be abandoned, and the reference count is used to preserve the more prevalent of the two. A grounding is lost to one agent, but that grounding is less valuable to the group of agents as a whole than the one that is preserved. If the lost grounding is something of use, it will be rediscovered or relearned from others. It may encountered again with the same symbol, but eventually one of the two competing uses will prevail across the population due to reference counts. The alternative to this–keeping the grounding and choosing yet another new label–is not followed because while it would preserve the grounding in one agent, it would cause even more groundings with redundant labels to ultimately spread across the population, further reducing consistency. In the final form of conflict (Case 3), both the location and the label are known to the R2 . This may in fact be the same grounding, but recorded slightly differently due to errors in perception or odometry. The implementation described in the next section details the exact methods by which agents judge two locations to be the same. If these locations are not the same, then the viewing agent has both a different symbol grounded to the current location, and the symbol used by the demonstrator in use for another grounding. This amounts to both the two prior cases, and these are resolved as described above. While conflicts involving different agents independently choosing the same label for different groundings can be somewhat mitigated by using a broad a range of labels to limit the likelihood of duplication, they cannot be guaranteed to be avoided through such means. There will always be some likelihood of duplication, and beyond this, it is possible that errors in demonstrations (e.g. through incorrect localization of the demonstrator or viewer) would lead to the spread of the same symbol with multiple locations, and any scheme to promote consistency must deal with these. The approach presented here is also general in that it allows agents to operate more independently, by not forcing any particular scheme for choosing labels (or even a consistent scheme between agents). In our evaluation (Section 14.5), labels are generated randomly in individuals, allowing such duplication. To the degree that duplication can be lessened, of course, the results shown can be improved.

288

N. Wiebe and J. Anderson

14.4 Implementation Since this approach is designed to operate as a group of robots performs some other duty, it could be implemented in any environment. To keep this general, we chose a basic multi-robot exploration domain, and implemented this approach in a behaviourbased robotic control system [2] for Pioneer DX2 robots, using Java 1.4, under Player/Stage [12]. Simulation was chosen to evaluate our implementation because it allows the support of a larger team than would be possible with the physical robots available in our lab. As others have previously noted (e.g. [13, 11]), simulation also lets us run trials much faster and with a much greater level of control and repeatability than the physical world. In addition, Player/Stage has been validated as accurately simulating the behaviour of ActivMedia’s Pioneer robots, and code developed under Player/Stage will run under Pioneer robots directly. The mapping performed by our software employs only the sonar that is standard with the Pioneer DX2, in order to ensure that the approach works with low-resolution mapping and would thus be applicable to much simpler robots as well. A laser rangefinder was added to the robot, but was only used to exploit the ability of this sensor to identify particular markers in the environment. Each robotic agent is given a unique numeric marker detectable by any agent’s laser range finder hardware, and this thus serves to uniquely identify each robot. This is not necessary for the approach itself, but is required by this implementation because the simulation software does not provide any direct agent-to-agent messaging options. In our simulated domain, agents map the world individually as they explore it, and can plot paths to specific locations based on their map. The task environments were 11x11m and 8x8m arenas, containing walls, doors, and obstacles (e.g. Fig. 14.1). Grounded communication in this domain is employed in the form of goal communication. A goal is placed randomly in the environment for the agents to find. As each agent explores the environment, it creates its own groundings using the methodologies described in Section 14.3, and shares these as other agents are encountered during the course of exploration. When an agent finds the goal, its location is broadcast to all agents in a grounded manner (this is the only broadcast communication used, and is used as a means of evaluating the efficacy of groundings in this domain). The subset of agents with the shared grounding(s) necessary for the goal location to have meaning are then able to plan a path and navigate to the goal. As greater consistency develops among a team of agents over time, all agents in the group should be able to arrive at the goal location more and more quickly. Two different techniques are used to describe the goal location. The first is simply broadcasting the label for the grounding the discovering agent has nearest to the goal (an indexical reference). This location will not necessarily be at the goal itself, but it is the closest location to it that can be understood by others directly. A second technique implemented was to specify the goal location using two grounded locations and a more complex symbolic reference. The symbolic reference is specified as (p1 , p2 , a, d). The indicated position is found by drawing a reference line from p1 to p2 and then, starting at p2 , traveling d times the distance from p1 to p2 , at angle a × 360o to the reference line. This is similar to the symbolic reference used in [15]. Understanding these two grounded locations allows the basis of a shared coordinate

14 Local Methods for Supporting Grounded Communication in Robot Teams

289

system between communicator and receiver, and allows an agent to specify a goal location precisely even if it has no grounding near the actual goal. This flexibility comes at the cost of requiring two shared groundings instead of one. Robot control is behaviour-based, and robots are given basic navigation and exploration behaviours (e.g. wandering, obstacle avoidance) and communication behaviours. Higher level behaviours support mapping and path planning. Mapping uses an occupancy grid approach [10], with each robot maintaining a cartesian coordinate system based on its occupancy grid, with the 0, 0 origin marking its own random starting location, and a 0 heading marking its own random orientation. This results in each agent having its own private coordinate system, offset and rotated from the real world (i.e. the Stage coordinate system). An agent has no knowledge of the relationship between its own coordinate system and the coordinate system of others: it is thus not able to communicate meaningful spatial information by using raw coordinates and headings, and must use labels for commonly grounded locations. Agents also make no assumptions about the size of the environment, thus requiring internal mapping to dynamically resize its world map as new areas are discovered. In our implementation, we begin with 20 grid cells (of 10cm x 10cm each, the atomic unit size we employ for mapping) in each direction from the robot’s origin, and increase this by 5% along the appropriate axis as new areas of the environment are discovered, and cells are marked as blocked or free with a likelihood based on repeated sonar reflections or lack thereof. Path planning follows a quad-tree-based approach, with path improvements based on the technique of [8]. This simplification is based on the realization that some nodes along a path planned through a quad-tree decomposition are unnecessary. For example, if a planned path goes from node a to node b to node c, it may be possible to remove the middle node, node b. This is possible when there is an obstacle-free straight line path from node a to node c, then node b is unnecessary and may be eliminated. This simplification results in shorter paths. 14.4.1

Location Grounding Behaviours

Behaviours are implemented to encompass the three grounding generation strategies described in the previous section. Only one of these grounding strategies will be active at a time. All grounding strategies label new groundings with a randomly generated integer in the range 0 to 10000. When an agent generates a new label, if a randomly chosen number is already in use to ground a location within the agent’s knowledge base, a new random label is selected. The label-at-meeting strategy involves an interaction between two robots, and a series of communications as a result of that interaction. These are implemented in a behaviour whose operation is summarized in Figure 14.3. When a robot R1 senses another robot R2 within 4m, R1 stops in place, and sends a message (LABEL MEETING START, an invitation to begin a grounding creation) to R2 . If R2 is receptive, it begins the process of creating an acceptable label as described below. If R2 is not receptive, it responds negatively to R1 and both agents continue to explore the environment, performing useful work (in our implementation, searching for a goal). R1 will not try to initiate another label-at-meeting conversation for 20 seconds after receiving a negative response.

290

N. Wiebe and J. Anderson

Fig. 14.3. Label-at-meeting agent interaction

If R2 is receptive, it indicates this receptiveness by responding with a proposed label l p . This proposed label will ground the location halfway between them, and is sent to R1 (as an argument of a LABEL MEETING LOC NAME message). R1 then evaluates the proposed location label, and if it is not already used in R1 , then l p is accepted. R1 signals this to R2 with an acknowledgement, and both agents then know the location

14 Local Methods for Supporting Grounded Communication in Robot Teams

291

halfway between them as l p . l p starts with a reference count of two, since there are two agents which know the location as l p . In the event that R1 already has a grounded location by the label of l p , the role of choosing a grounding label is reversed, and R1 sends a message to R2 with a different proposed label for the location. R2 processes this message in the same manner that R1 did. This process goes back and forth until a novel label in both R1 and R2 is found, or the conversation times out after 15 seconds. Recall that the label-at-meeting strategy is the only one where the location of a grounding is decided upon through the interaction of two agents. The other strategies require some judgement as to whether it is useful to ground the location a single agent currently occupies. The behaviour implementing the label-spatial-entropy strategy performs this judgement by considering the fifteen occupancy grid cells that surround the agent’s current location every ten seconds, calculating the spatial entropy of these cells as the robot moves. A grounding is made if the spatial entropy threshold exceeds the standard value of 0.75 (a value arrived at by preliminary experimentation in the domains employed). If the agent is near the edge of the environment, the box of adjacent occupancy grid cells would extend outside the environment, and it is moved to be entirely within the environment. Implementing the label-environment-feature strategy in a behaviour requires recognizing locations that are useful in a domain-specific manner, which can be arbitrarily complex. Because the focus of this work is not on robotic perception, in our implementation we employed feature labelling by placing perceivable feature markers that could be easily recognized. Each marker was given a unique identifier in the same way that agents are uniquely identified – these thus allow the unique identification of environmental features when they are discovered. An agent using this strategy creates a grounding when it perceives a marker within 2m. While the use of uniquely perceivable identifiers could easily allow these to serve as artificial aids to localization compared to more generic phenomena such as doorways and hallways perceived by sonar, at no time was this done in our implementation. In all labelling strategies, we avoid flooding the environment with groundings by removing any groundings with a reference count of 1 within 1.5m whenever a new grounding is produced. Redundant groundings serve no purpose and since they have not been shared their removal has no impact on the rest of the population. 14.4.2

Sharing Groundings

When sharing groundings, a fundamental question that arises is whether a location being demonstrated or referred to by one agent is identical to that used by another. Due to errors in both perception and localization, determining the spatial equality of two locations is more complex than simply comparing numeric coordinates for equality, even if disparate coordinate systems could be reconciled. For the purpose of implementing the approach described in Section 14.3, a tolerance factor ε is employed. If two locations are physically within ε range (50 cm in this work, a value arrived at through preliminary experimentation in the domains employed), they are considered the same. Figure 14.4 illustrates the detailed interaction between two agents attempting to develop a shared grounding, in a behaviour known as the location exchanger. Note that this behaviour is the same irrespective of the method used to originally create the

292

N. Wiebe and J. Anderson

grounding being shared. When an agent R1 detects another agent R2 nearby (by default 5m), and has grounded a nearby location (within 2.5m of its current position), R1 may attempt to share a grounding with R2 . The interaction is initiated by R1 sending a DEMO LOC message to R2 . If R2 is available it turns to face R1 and replies with a WILL FOLLOW message, indicating that it will follow R1 to the location to be demonstrated. R2 may not be available because it is already creating a shared grounding with another agent or has just completed doing so (our implementation requires an agent to wait 30 seconds between sharing groundings). If available, R2 begins to track R1 . Once R1 has arrived at the location to be demonstrated (democoordinates in the objective world, but recorded as (R1location in R1 ’s own coordinate system), it stops and sends a THIS IS LOC message to R2 . This message contains R1 ’s label (R1name ) and reference count for this location. In addition to the perceivable location, and the data communicated by R1 , R2 also may have a grounding R2name using the same label (which may be at another location, R2labelled location in its own coordinate system). To determine R2 ’s response to this demonstration, R2 needs to determine whether or not it has a grounding fitting R2labelled location (i.e. knows a different grounding within ε tolerance), and whether or not it has a grounding fitting R2name (i.e. already uses the label R1name elsewhere). The cases that result from these two decisions have already been explained in Section 14.3.3. Here, we explain what occurs in each case from an implementation perspective. If R2 knows neither democoordinates nor demoname , there is no conflict. R2 adopts demoname for democoordinates , and replies to R1 with a LOC LABELLED message. Both R1 and R2 now know R1 ’s current location as demoname . Each agent also increments the reference count to reflect that one additional agent knows this grounding. There are also three possible types of conflicts, as described in Section 14.3.3. In case 1, R2 knows democoordinates , but not demoname , and must determine whether to use its own label for this location, or R1 ’s. If the label that R2 is currently using has a higher reference count of the two, this is the more consistent label and R2 tells R1 its label and reference count for R1 ’s current location via a LOC OVERRIDE message. If R1 ’s label has the higher reference count, R2 adopts the supplied label for the supplied coordinates, and acknowledges this via a LOC LABELLED message, allowing both to update reference counts. In case 2, R2 knows demoname , but not democoordinates and R2 must determine whether the location currently grounded by demoname is more useful than this new proposed grounding. If R2 ’s label has a higher reference count than R1 ’s, R2 ’s location is more valuable. However, since the location is an arbitrary one in the environment, and would require travel to demonstrate, the best option is for R1 to simply forget about demoname . To effect this, R2 sends a LOC FORGET message to R1 . If R1 ’s label has a higher reference count, R2 forgets its current information about demoname and grounds demoname to democoordinates . R2 then sends a LOC LABELLED message to R1 , allowing both to update reference counts. In case 3, R2 knows both demoname and democoordinates . In this case R2 must first determine if it employs both of these in the same grounding. If so, R1 and R2 already share a grounding for this location, and the reference counts must be made consistent.

14 Local Methods for Supporting Grounded Communication in Robot Teams

293

Fig. 14.4. Agent interaction during development of a shared grounding

R2 corrects the reference count in R1 to the higher of the two using a LOC OVERRIDE message (LOC LABELLED is used only to mutually increment the reference counts). If R1 and R2 do not share the same grounding for democoordinates , R2 must determine which location should be kept based on the reference counts. If the reference count provided by R1 is higher than that for R2 ’s grounding of the label demoname , R2 adopts R1 ’s label and reference count. R2 then sends a LOC LABELLED message to R1 , allowing

294

N. Wiebe and J. Anderson

both to update reference counts. If R2 has a higher higher reference count than R1 , R1 should forget its grounding for demoname , so R2 sends a LOC FORGET message to cause this to occur. The behaviours described in this section summarize the core of our implementation. These encompass a number of lower-level behaviours (e.g. a behaviour to move to a nearby grounding to demonstrate it, follow another robot, etc.). Additional behaviours are also required that are not described fully here for reasons of brevity and the fact that these behaviours are reasonably obvious. For example, a goal seeking behaviour is implemented that attempts to perceive if the robot has reached a given goal, and listens to goal locations broadcast in terms of grounding labels by other robots. Further details on all of these behaviours are available in [19].

14.5 Evaluation The evaluation of this approach was done in simulation using Player/Stage [12], using six Linux-based PCs organized in three pairs. The first computer in each pair was used to run the Stage simulator, while the second was used to run all the of the simulated robots. The evaluation performed involved examining the efficacy of the grounding schemes described in Section 14.3, in the domain described in Section 14.4. Agents are place in the domain with random locations, and headings, and begin with no groundings. Agents then construct, demonstrate, and resolve conflicts between groundings, using the methods described previously, as they explore the environment. Whenever the goal is found by an agent, the location is broadcast using an indexical or symbolic reference as described in Section 14.4. This allows the goal more quickly by any others that share the groundings necessary to understand the reference. If an agent has not found the goal after 10 minutes of searching, it gives up. After all agents have found the goal or given up, one iteration of a trial is complete. During this iteration, agents collectively created a set of groundings which should be useful to improve their performance in this environment in future. These groundings should become more extensive and more consistent as agents gather more experience in this environment. In order to examine this effect, a full experiment consists of a series of 200 of these iterations, where agents maintain their groundings between iterations and are placed in new random locations (with a new, randomly located goal) for each iteration. Experimental trials were conducted in many different configurations. The three grounding creation strategies were explored independently, as were the two goal communication mechanisms. The size of team was varied (2, 4, 8, and 16 robots), as was the size of the domain (8x8m and 11x11m). We also attempted to vary the configuration of the space, since domains of a given size can have very different complexity, by defining four different layouts: open space for a baseline, a general partitioned domain meant to be typical of an office (Fig. 14.1), a split domain (Fig. 14.5) where barriers more strongly restrict travel from one end to the other, and a hallway domain with areas partitioned off either side. The distance at which a demonstration would be performed and the ε constant were 5m and 50cm respectively (additional experiments also examined varying these [19]).

14 Local Methods for Supporting Grounded Communication in Robot Teams

295

Fig. 14.5. The split (left) and hallway (right) 8x8m environments

These experiments all tracked group performance over trials as shared groundings were developed, and also examined both the final density of groundings produced and the percentage of consistency in groundings among agents that was ultimately achieved. The latter was computed by gathering all individual groundings after an experiment into a full set, removing duplicate labels for the same locations (where two robot widths 80cm - was used as threshold for equality) and keeping only the most commonly used as the set of globally ground locations. The percentage in each agent that was consistent with this global grounding set was tallied, and these were averaged across all agents to produce an average consistency. This metric is a pessimistic one, in that by throwing multiple values away, we are discounting the partial consistency that does exist in the removed values (e.g. 49% of agents may have a consistent label for a location that is still useful to this subset, but this is removed in favour of the 51% that have a different label). This measure is also biased against larger populations, because unique individual groundings affect the measure more in larger populations. For example, if there is only a single globally ground point, and one agent that possesses that grounding, this amounts to a 50% global consistency in a 2-robot team, but only a 6.25% consistency in a 16robot team. In all cases, results are presented by comparing the team performance against a control group consisting of the same configuration but without the ability to develop and communicate with shared groundings, and expressed as a percentage of improvement over this baseline. Here we present a representative subset of the results obtained, focusing only the larger environment size, and the reader is referred to [19] for further results of interest. Over all domain types, in the larger domain size, the average improvement using a goal communicated as a single grounding (an indexical reference) is shown in Fig. 14.6. This showed good improvement as team sizes increased up to 8 robots, followed by similar performance when doubling the population to 16 robots. In an 8x8m domain, the performance dropped after increasing the team size from 8 to 16 robots. In both domain sizes this is showing the point at which overcrowding and the resulting interference

296

N. Wiebe and J. Anderson

Fig. 14.6. Average percentage improvement by grounding strategy in 11x11m environments

between agents (e.g. collisions, more time spent avoiding obstacles) tended to limit the functionality of a larger team. Nonetheless, the techniques described here produced observable performance gain. The label-at-meeting strategy had the largest performance improvement. This is attributed to already having a shared grounding between two agents when the grounding is made, leading to an immediate increase in the ability to spread that grounding quickly. This is especially significant for 2-agent teams, since a shared grounding is already globally consistent. Looking at consistency (Fig. 14.7) in these environments illustrates this. In this task, meetings were naturally distributed across the environment, leading label-at-meeting to produce wide coverage in groundings locations that led to a greater likelihood that a goal could be usefully described to most agents. In situations where meetings were infrequent (e.g. a small team over a large domain), this would be less useful, but in such a situation there is also a much stronger likelihood of agents working independently in different areas, and arguably less of a need for grounded spatial communication than in closer confines. Label-spatial-entropy was generally the second most successful approach. This was despite the fact that it produced significantly more groundings than the other two strategies, especially with large numbers of agents (Fig. 14.8). This increase is still not large, and the number of points grounded by 16 robots is far fewer than the approach of [15] makes for 2 robots. For 2 and 4 agent populations, agents are able to exchange enough locations to maintain a reasonable level of consistency. With larger population sizes, the number of groundings exceeds the number that can be reasonably shared, leading to lower consistency. This does not rule out spatial entropy as an valuable alternative: a domain could be adjusted to use a higher entropy threshold and still use this reasonably domain-independent technique. It may seem surprising that grounding random encounters performed better than a technique that was specifically designed to ensure useful areas for navigation were grounded. However, not only is there greater sharing built into the label-at-meeting strategy, the points being grounded by label-spatial-entropy are not being exploited in this domain as strongly as they might be. Despite these groundings being interesting

14 Local Methods for Supporting Grounded Communication in Robot Teams

297

Fig. 14.7. Average global grounding consistency by grounding strategy (indexical reference) averaged over all 11x11m environments

Fig. 14.8. Grounding density by grounding strategy (indexical reference) averaged over all 11x11m environments

from the standpoint of navigation, the grounded communication employed references only the goal location. In that respect, these points are no better than other random locations. We anticipate that if an environment were to be set up where the grounded communication was more navigationally-oriented, spatial entropy would be more successful. An example of this would be an environment where information about waypoints to a goal were communicated instead of the goal itself. Labelling environment features produced the lowest overall improvement, mainly because there were too few locations to be grounded (as indicated by grounding density). Having few locations to ground seems like a factor that should encourage high consistency. However, it also limits the opportunities for an agent to demonstrate an

298

N. Wiebe and J. Anderson

Fig. 14.9. Average improvement by grounding strategy (symbolic reference) in the split 11x11m environment

Fig. 14.10. Average improvement by grounding strategy (indexical reference) in the split 11x11m environment only

already-grounded location to others, since these locations are only demonstrated when agents encounter one another with a grounding nearby. Because of the large numbers of trials demanded by the breakdown of all the controlled variables being evaluated, and the time taken to run trials (One trial of 200 iterations with 16 agents in one environment, for example, took six days using 2 computers), we examined some factors only in subsets of these environments. In particular, the effect of using symbolic references involving two groundings rather than using a single grounding as an indexical reference, was examined only in the split environment configuration. This was selected because it showed a high average improvement in the baseline experiments, and it also had high absolute times, and so we should expect a more sophisticated technique to be just as well-received. Results with goals communicated to team members symbolically using two grounded locations as described in Section 14.4 in an 11x11m split configuration are illustrated in Fig. 14.9. Because this affects only how goals are communicated, density and consistency information should

14 Local Methods for Supporting Grounded Communication in Robot Teams

299

not differ the results presented earlier. To provide a baseline for comparison, the results shown in Fig. 14.6, disaggregated to show only the split domain, are shown in Fig. 14.10. From these Figures, it can be seen that the label-at-meeting grounding strategy using an indexical reference to communicate a goal performs as well or better than using a symbolic reference. The dynamics of the strategy provides good distribution of shared groundings over the environment, with high consistency, allowing the goal’s location to be specified accurately enough with the single reference. This is coupled with the fact that a symbolic reference requires twice as many shared groundings for the reference to be useful, leading to approximately equal performance between the two references. The label-spatial-entropy grounding strategy shows little difference in performance across all agent populations, using either form of goal reference. The increased flexibility of a symbolic reference seems to be mitigated by the need to share twice the number of groundings in order to make use of it. Future work could explore this by selecting a domain in which a simple reference is less likely to be useful. The label-environment-feature grounding strategy fared better than all others for an 11x11m split environment when using symbolic references. This illustrates the advantage of using symbolic references over indexical references when sufficient groundings are available to support them. Since agents using the label-environment-feature grounding strategy are limited to grounding certain locations, a single grounded reference (used when indexical references are communicated) has limited ability to specify the goal’s location. This is in contrast to a more complex symbolic reference, which gives label-environment-feature the ability to specify an arbitrary point in the environment, and hence the goal’s location, more accurately.

14.6 Conclusions In this chapter, we have presented an approach to developing consistent shared groundings over time that uses local interaction between agents to bring consistency to a population. We have described a number of alternatives for creating groundings, detailed the implementation of these techniques, and presented an evaluation comparing them. In comparing techniques for grounding, label-at-meeting was the most successful, because the implicit sharing when agents create a new grounded location leads to a high global location consistency, which improves the effectiveness of communication about the goal. This approach allows agents to use their own independent spatial representations and coordinate systems internally, and develop the means to function together as a team without assuming a shared coordinate system or external facilities such as GPS. This in turn leads to more general agents. It also allows groundings to be developed that are naturally useful from the point of view of the agents themselves, rather than the viewpoint of the agent developers. Such non-anthropocentric mechanisms are an important part of developing truly autonomous agents that can function across a broad range of environments. In this approach, the symbols used are initially undefined, as opposed to known by an initial teacher, and assumes no fixed reference points or common coordinate system, as opposed to the approach presented in [5]. Unlike the approach presented in [15],

300

N. Wiebe and J. Anderson

this approach is carried out during other work as opposed to during a labelling phase, grounds a practical number of points, and is shown to work with greater than 2 agents. We are currently exploring a number of avenues of future research. The most obvious of these is evaluating these techniques in physical environments. The major potential discrepancy is in the error and noise associated with the real world as opposed to that supported by Player/Stage. While there is the possibility of localization and perception problems in Player/Stage, these significantly underestimate those of most complex realworld domains. Greater error in the real world will lead to mistaken groundings which the techniques described here, or adaptations of them, will have to be powerful enough to overcome. For example, a robot could misperceive a feature and ground an incorrect location, or demonstrate a grounding based on incorrect assumptions about its localization. When this occurs, the process of learning and sharing groundings must be used to correct these problems. Our laboratory is currently performing an evaluation of these techniques in the real world, using pioneer robots in an office environment, under conditions of sensor noise and imperfect localization using a particle filter. Here groundings can also serve a dual purpose, for communication as described in this work, but also as aids to localization itself. As part of this work, we are also adapting these techniques to heterogeneous agents. Heterogeneity adds great complexity and potential error to this work, since one agent may try to describe a grounding to another that cannot actually perceive the same phenomena. In this case, agents must attempt to ground new locations they cannot perceive in terms of existing groundings that are perceivable [1]. There are also numerous additional areas for potential future research. For example, the specific techniques we developed for choosing groundings were meant to be examples of fully domain-independent, task-specific (navigation), and domain-specific (office navigation), in order to examine agent performance against the amount of knowledge supplied by human developers. A broad range of other techniques (or combinations of techniques) are possible. The application of machine learning or evolutionary approaches to this would also be very interesting, so that agents could learn appropriate grounding techniques for the domain at hand. As a final note, it is noted in [19] that a bug in recognizing the second category of conflict detailed in Section 14.3.3 may have negatively influenced the results obtained. These experiments were entirely re-done with this bug fixed, and resulted in only a very small (1%) difference.

References 1. Abdellateef, A.K.E.: Grounded communication for heterogeneous multi-robot systems. Master’s thesis, Department of Computer Science, University of Manitoba, Winnipeg, Canada (2008) 2. Arkin, R.: Behavior-Based Robotics. MIT Press, Cambridge (1998) 3. Balch, T., Arkin, R.: Communication in reactive multiagent robotic systems. Autonomous Robots 1(1), 27–52 (1994) 4. Baltes, J., Anderson, J.: Flexible binary space partitioning for robotic rescue. In: Proceedings of IROS 2003, Las Vegas, pp. 3144–3149 (2003) 5. Billard, A., Dautenhahn, K.: Experiments in learning by imitation – grounding and use of communication in robotic agents. Adaptive Behavior 7(3/4), 415–438 (1999)

14 Local Methods for Supporting Grounded Communication in Robot Teams

301

6. Coradeschi, S., Saffiotti, A.: Anchoring symbols to sensor data: preliminary report. In: Proceedings of AAAI 2000, Austin, pp. 129–135 (2000) 7. Coradeschi, S., Saffiotti, A.: An introduction to the anchoring problem. Robotics and Autonomous Systems 43(2-3), 85–96 (2003) 8. Davis, I.: Warp speed: Path planning for star trek: Armada. In: Proceedings of the AAAI Spring Symposium on AI and Interactive Entertainment (2000) 9. Deacon, T.: The Symbolic Species: the co-evolution of language and the human brain. Penguin Books, Toronto (1997) 10. Elfes, A.: Occupancy grids: a probabilistic framework for robot perception and navigation. PhD thesis, Department of Electrical and Computer Engineering, Carnegie-Mellon Univesity (1989) 11. Etzioni, O., Segal, R.: Softbots as testbeds for machine learning. In: Proceedings of the Machine Learning Workshop at AI/GI/VI 1992 (May 1992) 12. Gerkey, B., Vaughan, R., Støy, K., Howard, A., Sukhtame, G., Matari´c, M.: Most Valuable Player: A Robot Device Server for Distributed Control. In: Proceedings of IROS 2001, Wailea, Hawaii, pp. 1226–1231 (2001) 13. Hanks, S., Pollack, M.E., Cohen, P.R.: Benchmarks, test beds, controlled experimentation, and the design of agent architectures. AI Magazine 14(4), 17–42 (1993) 14. Harnad, S.: The symbol grounding problem. Physica D 42, 335–346 (1990) 15. Jung, D., Zelinsky, A.: Grounded symbolic communication between heterogeneous cooperating robots. Autonomous Robots 8(3), 269–292 (2000) 16. Matari´c, M.: Using communication to reduce locality in distributed multiagent learning. Journal of Theoretical and Experimental AI 10(3), 357–369 (1998) 17. Steels, L.: Constructing and sharing perceptual distinctions. In: van Someren, M., Widmer, G. (eds.) Proceedings of the European Conference on Machine Learning, pp. 4–13. Springer, Heidelberg (1997) 18. Vogt, P.: Symbol grounding in communicative mobile robots. In: Proceedings of the AAAI Fall Symposium on Anchoring Symbols to Sensor Data (2001) 19. Wiebe, N.: Developing grounded communication in multi-agent systems. Master’s thesis, Department of Computer Science, University of Manitoba, Winnipeg, Canada (January 2006)

15 Vision SLAM Maps: Towards Richer Content Daniel C. Asmar1 , Samer M. Abdallah2, and John S. Zelek3 1 2 3

American University of Beirut, Riad El Solh, Beirut, Lebanon [email protected] University of Waterloo, 200 University Ave. West, Waterloo, Ontario, Canada [email protected] University of Waterloo, 200 University Ave. West, Waterloo, Ontario, Canada [email protected]

Abstract. Simultaneous Localization and Mapping (SLAM) is a recursive probabilistic inferencing process for concurrently building a map of a robot’s surroundings and localizing that robot within this map. The ultimate goal of SLAM is to operate anywhere, allowing a robot to navigate autonomously and producing a meaningful purposeful map. Research in SLAM to date has focused on improving the localization part of SLAM, while lagging in the ability to produce useful maps. Indeed, all feature-based SLAM maps are built from either low level features such as points or lines or from artificial beacons; such maps have little use other than to perform SLAM. There are benefits in building maps from real natural objects that are indigenous of the environment for operations such as surveying of remote areas or as a guide for human navigation in dangerous settings. To investigate the potential of SLAM to produce such maps, an InertialVisual SLAM system is designed and used here which relies on inertial measurements to predict ego-motion and a digital camera to collect images of natural landmarks about the scene. Experiments conducted on a mobile vehicle show encouraging results and highlight the potential for Vision SLAM to generate meaningful maps which agree with ground truth. The Computer Vision system is capable of recognizing the environment type, of detecting trees within this environment, and recognizing different trees based on clusters of distinctive visual features.

15.1 Introduction Dimensionality explosion, consistency, and representation are the three major topics of interest in the community of Simultaneous Localization and Mapping (SLAM). Since the incubation of SLAM in the mid eighties to date the majority of work in this field has focused on the first two of these issues (i.e. dimensionality explosion and consistency), where most of the time SLAM is implemented in structured indoor settings using simple image features as landmarks. Although such features work well for solving the localization part of SLAM, the resulting maps can not serve any other purpose. It would be extremely beneficial if SLAM systems produce maps with landmarks which are meaningful (i.e. trees, rocks, and buildings) rather than insignificant saliencies (i.e. edges, corners). With real objects, landmarks can be recognized based on their appearance and the problem of data association which has been traditionally dealt with via statistical methods is avoided. Furthermore, the rich semantic make-up of the maps D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 303–329. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

304

D.C. Asmar, S.M. Abdallah, and J.S. Zelek

promotes their usage for applications such as surveying, or human navigation in dangerous settings during search and rescue operations. Finally, the issue of dimensionality explosion, which is common in feature-based SLAM can be avoided because of the lower number of landmarks involved in object-based SLAM. On the downside, there is a risk of not having enough objects in the environment to perform SLAM; an issue which will be investigated in the experimental section of this chapter. Instead of sonar or laser, a camera can be used as the primary exteroceptive sensor onboard a mobile platform. Cameras are cheap, consume very little power, and carry a high bandwidth of information, which if properly managed can be used to detect, recognize, and localize real objects. All-object based Vision SLAM systems developed so far use artificial landmarks rather than natural landmarks. Panzieri, Pascucci, and Ulivi [19] present a vision-based SLAM implementation in an indoor environment using trinocular vision. Artificial light sources located in the ceiling above the robot are used as landmarks. Kim and Sukkarieh [15] demonstrate a Vision SLAM system on an unmanned aerial vehicle using an inertial sensor for dead-reckoning prediction and a monocular camera for landmark detection. Landmarks are artificial beacons with known shape and size, whose 3D position is determined based on their projected sizes in the images that are collected during navigation. Fitzgibbons [8] performs both RangeBearing (R-B) SLAM and Bearing-only (B-only) SLAM in an outdoor environment using artificial beacons as landmarks. The landmarks are bright white parking dots and are detected by thresholding the images, and selecting regions that are brightest in the images. Fitzgibbons [8] further implements a B-only SLAM using colored poles as landmarks and the system uses a panoramic camera to detect the relative bearing of landmarks to the vehicle. Bryson and Sukkarieh [5] implement their B-only SLAM on an Unmanned Aerial Vehicle (UAV) over unstructured, natural environments. Delayed initialization is used to fix the 3D position of landmarks from several bearing readings obtained from different robot poses. Landmarks are salient white patches similar to those used in [15]. Examples include large white squares [15] and colored poles [8]. Although artificial objects are meaningful, using them as landmarks suffers from several disadvantages. Firstly, it is often desired to map environments which are inaccessible by man either because they are dangerous such as in an earthquake aftermath or they are inaccessible such as the surface of the planet Mars. Secondly, even if a site were accessible the associated cost of installing the artificial landmarks is usually high. Given these limitations, the advantage of using natural objects as landmarks is apparent and it surprising to see that no Vision SLAM system to date has attempted to do so. This reluctance is predominantly due to the difficulties and high processing times associates with Computer-Vision-based object detection and recognition algorithms. This chapter investigates the potential of Vision SLAM systems to produce maps which are semantically-rich. The success, failures, difficulties, challenges, and processing times are discussed. Vision SLAM is performed on a video stream collected during the journey of a mobile land vehicle. Ego-motion estimation is conducted via an Inertial Navigation System which is detailed in Section 15.2. Section 15.3 explains the Computer Vision system which is used to first recognize the environment class, and then to detect, recognize, and localize landmarks. Environment recognition is done to constrain

15 Vision SLAM Maps: Towards Richer Content

305

the search space to a type of landmark that is available in the recognized setting. The INS and Computer Vision systems are integrated into an EKF-SLAM system which is detailed in Section 15.4. Section 15.5 presents results and a relevant discussion. Section 15.6 concludes.

15.2 Inertial Navigation System Inertial Navigation Systems (INS) are used to estimate ego-motion by transforming acceleration and rotation readings from an Inertial Measurement Unit (IMU) into displacement and orientation. IMU’s are less sensitive to slippage than wheel encoders and can provide more accurate motion estimation. Traditional IMU’s are of the ‘gimballed’ type, which are expensive and only used by the military. With the recent interest in IMU’s for the automotive industry, low-cost MEMS-based strapdown units have been developed which are not as accurate as the conventional ones but are adequate for robot navigation. All IMU’s drift with time unless the navigation system is augmented by an additional source of information such as a Geographical Positioning System (GPS), or is used within a SLAM environment (Section 15.4). In an attempt to reduce INS drift, we impose non-holonomic constraints, which restrain the vehicle to move in only one direction at a time. In other words, as the platform moves in the forward direction it does not slip laterally and does not jump off the ground. These constraints are enforced via an Extended Kalman Filter (EKF) as proposed by Dissanayake et al. [7], where each observation represents the vehicle velocities in the lateral and vertical directions and are therefore set to zero at each iteration. Mathematically, this implies: Vby − νy = 0

(15.1a)

Vbz − νz = 0,

(15.1b)

where Vby and Vbz are the velocity components of the vehicle in the body coordinate frame, νy and νz are the respective noise values associated with these velocities. The velocity vector Vn expressed in the navigation frame is transformed to the body frame through Vb = [Cbn ]T Vn ,

(15.2)

where Cbn is the direction cosine matrix from the navigation frame to the body frame (See Fig. 15.7). Expanding (15.2) yields the measurement model h(x(k),w(k)): ⎤ ⎡ Vnx (sφ sθ cψ − cφ sψ )+   ⎢ V (cφ cψ + sφ sθ sψ ) + V sφ cθ ⎥   nz νy Vby ⎥ ⎢ ny =⎢ , (15.3) ⎥+ Vbz νz ⎦ ⎣ Vnx (cφ sθ cψ + sφ sψ )+ Vny (−sφ cψ + cφ sθ sψ ) + Vnzcφ cθ where s and c are the trigonometric Sine and Cosine, and φ , θ , and ψ are the Euler angles.

306

D.C. Asmar, S.M. Abdallah, and J.S. Zelek

Equation (15.3) represents the velocity of the vehicle in the body coordinate frame. It is desired to set the two rows in (15.3) to zero at each time step in order to satisfy the non-holonomic constraints. The navigation system is designed in the framework of an EKF, where predictions are obtained from the INS state transition equations, which includes three entries for position (Pn ), three for linear velocities (Vn ) and three for the Euler angles (φ , θ , ψ ). ⎡ ⎤ ⎡ ⎤ Vn P˙n n ⎢V˙n ⎥ ⎢ ⎥ Cb Ab − G ⎢ ⎥ ⎢ ⎥ ⎢ ⎢ ⎥ ⎥, ˙ w + (w sin φ − w cos φ ) tan θ (15.4) x˙ = ⎢ φ ⎥ = ⎢ bx by bz ⎥ ⎣ θ˙ ⎦ ⎣ ⎦ wby cos φ − wbz sin φ wby sin φ +wbz cos φ ψ˙ cos θ The equations for prediction, observation, and update of the INS EKF are as follows Prediction xˆ (k|k − 1) = f (ˆx(k − 1|k − 1), u(k − 1), (k − 1))

(15.5a)

zˆ (k|k − 1) = h(ˆx(k|k − 1))

(15.5b)

P(k|k − 1) = Js (k − 1)P(k − 1|k − 1)Js (k − 1)T + Ju (k − 1)QJu (k − 1)T

(15.5c)

ˆ ...) is the state where xˆ is the predicted state (x, y, z,Vx ,Vy ,Vz , φ , θ , ψ ) of the vehicle, f (x, transition function shown in (15.4), u(k) is the process (ax , ay , az , wx , wy , wz ). zˆ (k|k − 1) is the predicted observation at time tk−1 obtained from the observation model h calculated in (15.3). Js and Ju are the Jacobians of the state transition function f (ˆx) with respect to the state and process respectively. The calculation of these Jacobians is performed numerically as shown by Bailey [9]. Q is the noise associated with the process (i.e., expected noise in IMU gyros and accelerometers). Observation The observation model here is used to impose non-holonomic constraints through the mechanics of an EKF. This objective is achieved by setting the observations (i.e., Vby and Vbz ) to zero at each time step, where the innovation μ is expressed as

μ = 0 − zˆ (k|k − 1)

(15.6)

At this stage an innovation covariance can also be calculated as S(k|k − 1) = Jx (k − 1)P(k|k − 1)Jx(k − 1)T + Jw (k − 1)R(k − 1)Jw(k − 1)T

(15.7)

where Jx and Jw are the Jacobians of the observation model with respect to the state and observation (vby ,vbz ) respectively. R is the noise in the assumed non-holonomic model, or in other words, the precision of the non-holonomic assumptions.

15 Vision SLAM Maps: Towards Richer Content

307

Update The update is based on the EKF formulation as follows: xˆ (k|k) = xˆ (k|k − 1) + W(k)(μ )

(15.8a)

P(k|k) = P(k|k − 1) − W(k)S(k)WT (k)

(15.8b)

where W(k) is the gain matrix and is given by W(k) = P(k|k − 1)Jx(k − 1)T S−1 (k)

(15.9)

where P(k|k − 1) is the predicted covariance obtained through (15.5c) and S is the innovation covariance calculated in (15.7). Fig. 15.1 shows the effect that the non-holonomic constraints have on the velocity in the lateral vehicle direction Vby . Ideally, one would want the velocity to remain zero but in reality the best the EKF can do is to have this value fluctuate about zero. In the left image, the system includes no constraints, notice how Vby increases very quickly. In the right image, non-holonomic constraints are imposed, which is evidenced by the fluctuating velocity about zero. Notice that the highest value on the y axis for the unconstrained system is 18 m/s, while it is 0.45m/s for the constrained system.

Fig. 15.1. Effect of enforcing non-holonomic constraints on the lateral velocity. In the left image no constraints are imposed whereas in the right image, the non-holonomic constraints are imposed via the EKF.

15.3 Computer Vision System The Computer Vision system explained here has two main functions: (1) to recognize the environment within which the robot is located and (2) to detect, recognize, and localize natural object (e.g., trees) around the robot. The details of the environment recognition system are presented in our previous work [3]. Environment recognition offers

308

D.C. Asmar, S.M. Abdallah, and J.S. Zelek

valuable information which can be used to confine the landmark search space to features that are characteristic of the robot’s surroundings. Alternatively, the environment information could be used in a less restrictive fashion to produce a prior probability on the detection of each landmark type and additional factors such as landmark color, texture, and shape would also be used in conjunction with the environment information to probabilistically infer the appropriate landmark for each environment. This second approach would allow SLAM to use landmarks that are not very characteristic of a particular environment but are nevertheless abundant there, such as trees in an urban setting. 15.3.1

Visual Environment Recognition

There are two methods to achieve visual environment recognition. The first method is bottom-up and involves segmenting the query image, grouping areas that share commonalities, matching these areas to model objects, and finally inferring the environment from the recognized objects. This type of environment determination is too time consuming to be implemented in real time. The second, quicker approach, is top-down, where the spectral rather than the intensity values of the image are processed to infer the environmental setting. Windowed Fourier Transforms, Gabor filters and Wavelet Decomposition filters are the most common frequency decomposition methods that are used to extract image spatial structure. Tieu and Viola [20] applied several low-level filters to discover salient features in images. Oliva and Torralba [18] used a Windowed Fourier Transform to acquire a low dimensional representation of the scene, which they termed the Spatial Envelope. Torralba [21] performed Gabor filtering on images to

Fig. 15.2. System Architecture

15 Vision SLAM Maps: Towards Richer Content

309

extract salient features and Torralba et al. [22] applied a Steerable Pyramid (SP) filter to their images to extract feature vectors for the sake of place and object recognition. Our approach to environment recognition [2] is shown in Fig. 15.2 and consists of a machine learning algorithm built from an Artificial Neural Network (ANN). Images of different settings are collected and used as a training set for the ANN. Each image is filtered with a Steerable Pyramid (4 scales, 6 orientations) which produces a 384 dimensional feature vector. These feature vectors are first reduced in dimension from 384 to 40 (via PCA) before using them to train the ANN for environment classification. Each time the robot queries an environment, it collects one hundred images of that setting, classifies them via the above ANN, and concludes the environment class based on a majority vote. Experiment done on a test set revealed 86% success rate for park environment and 77% correct classification for images of a downtown urban setting. In this work, experiments are performed in only one type of environment (inside a park), where environment recognition is only useful at the outset of the experiment. Nevertheless, one could easily see how such a system could be useful if the mobile vehicle navigates between two different settings (e.g., a park to an urban setting), where different objects would have to be used as landmarks (e.g. lampposts instead of trees). In this work, trees were used as landmarks for SLAM. Trees possess a quasi-structured shape and orientation and are abundant in a park environment. Unfortunately, natural objects such as trees exhibit variability in their appearance and it is intractable to represent all these objects by one generic model. Alternative object recognition methods include structure-based methods as was adopted in this work, or flexible appearancebased approaches where objects are recognized if a number of these parts are present in the image. An alternative strategy to explore in the future is a method for object category detection and recognition that we have recently developed that is a combination of a structure-based as well as appearance based strategies [23]. In the following section we will describe our structured-based system. 15.3.2

Tree Detection

Tree detection is based on a structure-based system developed by the authors in [4]. The first routine uses a Canny edge detector [6] tuned to find dominant edges in the vertical direction. Edges are then tracked into lines based on an eight point neighborhood. After all potential lines are tracked, a number of them are pruned according to two heuristics: minimal length and average texture response. Firstly, all lines which are smaller than a minimal length are pruned. Experiments run on a database of tree images indicate that lines less than 30 pixels long can be safely pruned without loosing any significant information. 30 pixels corresponds to a tree 1 meter high located 17 meters away from the camera. Secondly, in order to avoid confusing birch texture for tree profile, images are filtered with a Gabor filter using the component sensitive to the vertical orientation at the finest scale. Then, the average texture value for each of the tracked lines above is calculated and those below a threshold h are discarded. The idea is that tree profiles exhibit significantly larger average texture values in the vertical direction than those of lines generated by birch texture, which can therefore be pruned. All the lines that are obtained after line tracking and pruning are checked for continuity and symmetry based on several metrics which characterize the ‘degree’ of mutual

310

D.C. Asmar, S.M. Abdallah, and J.S. Zelek

Hand label 180

GS 180

Hand label 40

GS 40

Fig. 15.3. Results of the Ground-Sky detection system (right), compared to ground truth (left). While the segmentation of Image 180 appears close to ground truth, the segmentation of Image 40 is flawed.

Fig. 15.4. Variables for Continuity(left) and Symmetry (right)

continuity and symmetry between two lines (Fig. 15.4). Then, keeping in mind that tree trunks are anchored to ground, the Ground-Sky (G-S) separation line in each image is used as a basis to further prune non-tree lines. Lines representing tree trunks should be anchored close to the G-S line or be paired to symmetric or continuous lines that are close to G-S. Lines which are anchored above the G-S line and further than a given number of pixels away are pruned. Lines that are anchored below the G-S line but cross G-S are kept and represent tree trunks that are close to the camera. Calculating G-S is no trivial matter and represents an open research problem. One simple but unreliable method is to use the longest detected horizontal lines as G-S. One could easily see how this method would fail in a picture exhibiting a long stretch of trees of equal height at the horizon; where the line joining the tips of these trees would be taken as G-S. Figure 15.3 shows an example of an acceptable G-S segmentation and another which is not. Notice how the G-S line segmentation for Image 40 is correct to the right of the trees but is false elsewhere. Nevertheless, given that this ‘longest line’ method is the best we can do at this point, it will be adopted here. Once all bad lines are pruned, the remaining lines are grouped into trees via an entropy minimization scheme. Entropy reflects the amount of disorder in an image. For example, two separate lines (labeled as line1 and line2) possess a higher entropy

15 Vision SLAM Maps: Towards Richer Content

311

than those grouped into a single tree trunk (labeled as tree trunk1). This is done on an iterative fashion by testing the effect of a specific grouping on the net total entropy of the image. The combination of lines which results in the lowest entropy of each image is adopted. 15.3.3

Tree Recognition

Tree trunks are recognized and matched to previously viewed trees based on the features which they exhibit. These features must be very distinctive in order to avoid false positives, and must be recognizable from different camera viewpoints and varying lighting conditions. Mikolajczyk and Schmid [17] compare the performance of many descriptors and find that SIFT features are the most stable and distinctive of them all. For this reason, SIFT features are used in SLAM as the basis for the tree recognition system. Once a tree is detected, the SIFT features located within its boundaries are matched to those in a database, and the corresponding match indicated the identity of the queried tree. In a real-time SLAM, the SIFT database should be compiled online and updated each time a new tree is observed by augmenting the SIFT database with the SIFT’s belonging to that tree. For the sake of simplicity, the SIFT database is not compiled online in SLAM; rather, it is prepared off-line by pre-processing a number of images containing trees seen from several viewpoints. A more realistic implementation should incorporate new unseen landmarks and add their features to the SIFT database on the fly. Figure 15.5 shows the images that are used to extract the SIFT descriptors for Tree ‘six’. Images are taken from nine different viewpoints (i.e., orientation and scale) to capture sufficient SIFT features to represent the tree from several viewpoints. Tests on the same database used for the detection system result in 85.54% correct recognition rate. 15.3.4

Tree Initialization

The initial intention of this work was to use a Bearing-only (B-only) SLAM system. Unfortunately, EKF-based SLAM requires a good estimate of the landmark during

Img. 258

Img. 343

Fig. 15.5. Tree ‘six’ shown from nine different viewpoints, with overlaid SIFT features on every view. The SIFT features provide distinctive marks by which trees are identified.

312

D.C. Asmar, S.M. Abdallah, and J.S. Zelek

Tree ‘One’

Depth ‘One’

Tree ‘Six’

Depth ‘Six’

Fig. 15.6. Trees initialized using depth from stereo. Trees ‘One’ and ‘Six’ initialized when they are within the range of stereo camera.

initialization in order to avoid large non-linearities which would make the SLAM filter inconsistent. In B-only SLAM, no range information is available to help localize a landmark when it is first observed. Therefore, a landmark is initialized via a triangulation method which requires it to be observed from two different viewpoints which must be separated by a minimum baseline. This condition is difficult to achieve in our system since the mobile vehicle navigates most of the time in a straight line and would have to wait for a long period of time before the minimum baseline requirement is satisfied. For this reason, it is found advantageous to initialize each new landmark with a more accurate method. Since the mobile unit (see Fig. 15.7) is equipped with a stereo camera, we used stereo information to locate each tree when it is first observed. In stereopsis, the parallax between the two lenses of the stereo pair is used to calculate the depth of a landmark. One important issue is that depth quality decreases with range of the camera. The system here uses a range cut-off point, beyond which potential trees are not initialized. This point is chosen at 7.5 meters, where an accuracy of no larger than 18.72cm is possible (according to manufacturer - Point Grey Research Inc. - specifications). These trees are then introduced into the SLAM map by adding half the diameter of each tree to its corresponding range reading. Figure 15.6 shows two examples where the stereo system initializes trees into the system. Notice that in the second image (Tree ‘Six’) there is a tree in the background which is not initialized because of the stereo range limitation.

15.4 Vision SLAM System This Vision SLAM system is built in the framework of EKF, where predictions consist of the INS corrected predictions (i.e., via the non-holonomic constraints) and observations consist of bearing measurements of each tree to the camera. VisSLAM’s process model is based on the equations developed for the INS in Section 15.2, where a prediction of the vehicle motion is made with (15.8). The second source of information used by the EKF are the observations. A camera is mounted on board the navigating vehicle to detect the bearing of landmarks around the vehicle. The observation model is as follows zi (k) = Hi (pn (k), Ψ n (k), mni (k), k) + v(k),

(15.10)

15 Vision SLAM Maps: Towards Richer Content

313

Fig. 15.7. Coordinate frames used for INS dead-reckoning. Subscripts n, b and c stand for the navigation, body and camera frames respectively. XL and YL are the coordinates of the landmarks.

where Hi is the observation model, which is a function of the position pn (k) and bearing Ψ n (k) of the vehicle in the navigation frame, as well as the position of the landmarks mni (k) in the navigation frame; v(k) is the zero-mean observation noise error with covariance R. The observation model Hi is expressed as

c 

 Hi (x(k|k ˆ − 1) = ϕi = arctan yxc , (15.11) xc and yc , are the 2D coordinates of landmarks in the camera reference frame and are determined from  c x c pmc = c = Ccb Cbn [mni − pn − Cnb pbcb ], (15.12) y where pbcb is the position of the camera lens with respect to the body coordinate frame center (i.e., inertial sensor location); mi and pn are the respective position of a landmark ‘i’ and the vehicle in the navigation frame. The expression between the square brackets in (15.12) represents the distance between the landmark and camera ΔLC expressed in the navigation frame. Cbn is the matrix expressing ΔLC in the body frame (Fig. 15.7). Cbn is in effect the inverse of the direction cosine Cnb , and can be simply calculated from the transpose of Cnb since only rotations are involved in this transformation; pn is the position of the vehicle in the navigation frame and mni is the position of a landmark ‘i’, once initialized, in the navigation frame. Equations (15.11) and (15.12) present an expression for the expected bearing (azimuth) of a landmark as seen from the camera. The difference between the expected and actual bearing measurements yields a measure called Innovation, which is used in the update stage of the EKF to improve the state and covariance estimates. Cbc

314

D.C. Asmar, S.M. Abdallah, and J.S. Zelek

transforms ΔLC from the body frame to the camera frame. During vehicle motion, the camera pans back and forth in increments of π /4 between +π /2 and −π /2 in the body reference frame (black frame in Fig. 15.7), taking an image at each increment. This is done to increase the range view of the navigating vehicle. Since the camera rotates throughout the vehicle motion, the position of the camera lens depends on the camera shot. For simplicity, the center of PTU Yaw axis is taken as the camera origin rather than the actual camera center. This amount to an error of 6 cm at most in the position of the camera origin. Let α represent the camera angular rotation with respect to the Z axis of the body frame. The camera center offset from the body frame, seen in Fig. 15.7, is 5 cm in the x direction, 0 cm in the y direction and -20.6 cm in the z direction. Therefore, Cbc is best expressed in homogenous coordinates as follows: ⎡ ⎤⎡ ⎤ cα −sα 0 0 1 0 0 +0.05 ⎢sα cα 0 0⎥ ⎢0 1 0 0 ⎥ ⎥⎢ ⎥ Cbc = ⎢ (15.13) ⎣ 0 0 1 0⎦ ⎣0 0 1 −0.206⎦ 0 0 01 000 1 where Ccb and Cbn are the inverse of Cbc and Cnb respectively. The actual bearing measurements (ϕa ) as seen from the camera are calculated based on their image coordinates as follows 

 0 , (15.14) zi (k) = ϕ = arctan u−u fu where uo , v0 are the coordinates of the image center in pixels, u, v are the coordinates of the landmarks in pixels and fu and fv are the focal lengths in u pixels and v pixels respectively (Fig. 15.8 ). The focal length for the camera is 4mm or 514.92 pixels at a resolution of 640 x 480 pixels.

Fig. 15.8. Calculating bearing from an image. f is the focal length of the camera, u and v are the coordinates of the image of the base of the tree. u0 and v0 are the coordinates of the center of the image, and ϕL is the azimuth angle to the landmark.

15 Vision SLAM Maps: Towards Richer Content

315

The EKF recursive process consists of a prediction (INS), an observation (tree bearing from camera), and an update (EKF equations). The mathematical details of these three steps are presented next. Prediction P(k|k − 1) = Jx P(k − 1|k − 1)JTx + Jw QJTw ,

(15.15)

where Jx and Jw are the Jacobians of the state transition function with respect to the state vector and noise input respectively. Update xˆ (k|k) = xˆ (k|k − 1) + W(k)ν (k)

(15.16)

where

ν (k) = zi (k) − Hi (x(k|k ˆ − 1)) W(k) = P(k|k − 1)J[HTx (k)]S−1 (k) S(k) = J[Hx (k)]P(k|k − 1)J[Hx (k)]T + R

(15.17) (15.18) (15.19)

where J[Hx (k)] is the Jacobian of the observation model with respect to the state vector. The state covariance is finally updated as follows P(k|k) = P(k|k − 1) − W(k)S(k)WT (k).

(15.20)

15.5 Experiments and Results The Vision SLAM system described above is now tested on a data set of time-stamped GPS, IMU and images collected from a manually driven mobile vehicle at walking speeds (i.e., less than 1 m/sec) (see Figure 15.7) during its journey inside a park [1]. The aim of these tests is to investigate the potential of using natural objects as landmarks for robot navigation in general and for SLAM in particular. In the first test, trees are detected via the aforementioned tree detection algorithm but once they are detected, the are identified by the user who also initializes them into the SLAM state vector and covariance matrix. The aim here is to investigate the efficiency of the landmark detection part of the system, while assuming that recognition and initialization are solved. In the second test, the system automatically detects and recognizes trees, although landmarks are still initialized by the user using the known ground truth coordinates for each landmark. In the third and final scenario, full SLAM is investigated using the above Computer Vision system to detect, recognize, and initialize landmarks. Fig. 15.9 shows the map obtained in the semi-automatic test and Fig. 15.10(a) shows vehicle covariance for that run. In the worse case scenario, the variance in the x direction reaches 20m2 which corresponds to a standard deviation of approximately 4.6 meters in the x position (at t = 330 seconds). This peak corresponds to the final bend in Fig. 15.9 near Tree 16 (T16). A closer observation of this region of the test site reveals

316

D.C. Asmar, S.M. Abdallah, and J.S. Zelek

Fig. 15.9. SLAM map developed during Experiment ‘one’ (auto tree detection, manual recognition and initialization). ‘T’ refers to a Tree, ‘L’ refers to a Lampost, ‘P’ refers to a Post, ‘S’ refers to a Sign, and ‘FH’ refers to a Fire Hydrant. ‘Start’ is the starting point of the vehicle, ‘end’ and ‘loop closure point’ are the true and estimated (via SLAM) location of the vehicle at the point of loop closure. The red and green arrows are overlaid on the map to simply help follow the trajectory of the vehicle from ‘Start’ to ‘End’.

that the ground is inclined beyond the normal ground inclination of the rest of the test site. Therefore, the error is probably due to an incorrect estimate of the vehicle pitch, which in turn results in a over-estimation of the gravity component. This error would appear to the INS system as if the vehicle were accelerating backwards, where in fact it is undergoing forward acceleration. The highest covariance in the y direction is approximately 8 m2 (std = 2.8m) and occurs at 200 seconds, which corresponds to the region right after the U-turn at the bottom left of the image in Fig. 15.9 where the estimated vehicle trajectory slightly looses control (green line goes off track). This large variance is due to the deficiency of the non-holonomic slip model at regions where high rotation rates occur and some lateral movement does in fact exist. SLAM immediately recovers and re-stabilizes after this. Figure 15.10(b) shows the variance of the vehicle heading direction during the experiment. This value is small throughout the journey of the vehicle, indicating the continued large confidence in the heading direction. Indeed, observation of the vehicle motion in Fig. 15.9 shows good turning predictions at all bends. Note that even at the ‘T16’

15 Vision SLAM Maps: Towards Richer Content

a

b

317

c

Fig. 15.10. (a) Covariance for vehicle position for Experiment ‘one’ (auto tree detection, manual recognition and initialization). Px and Py are the covariances in the x and y directions respectively. (b) Covariance for vehicle heading for Experiment ‘one’ (auto tree detection, manual recognition and initialization). Pyaw is the covariances in yaw. (c) Covariance for vehicle position for Experiment ‘one’ (auto tree detection, manual recognition and initialization). Px and Py are the covariances in the x and y directions respectively.

tree the vehicle correctly rotates (look at the orientation of the tail of the vehicle) but moves backwards rather than forwards after the rotation. The covariance of the positions of landmarks are, as expected, very low since the landmarks are manually initialized. It is interesting to note that the covariances for all three landmarks decrease during the run, which is indicative of increasing confidence in the position of each landmark. Figure 15.10(c) shows the covariance variation for the three landmarks during SLAM. Notice that these values remain relatively low during the entire journey. The objective of the second experiment is to investigate the effectiveness of the tree recognition system developed in Section 15.3.3. The SLAM map is not as close to ground truth as in the first test, the vehicle trajectory is still well estimated. The exception is around tree ‘T13’ (See Fig. 15.11 for reference) where the vehicle follows a false trajectory and then rectifies itself. This error is due to the low number of tree trunks that are recognized by the automatic recognition system in this part of the map. Perhaps this suggests that the system cannot solely rely on a single landmark and other landmarks or features might have to be used if there are insufficient numbers of the primary landmark. A low recognition rate implies a low number of EKF updates and insufficient bounding of the INS errors. A similar error occurs at loop closure, where the system reverses track instead of heading to its correct destination. In Figure 15.12(a), the covariance in vehicle heading is still low (average of 0.002 rad 2 ); albeit, not as low as the first experiment (average of 0.0012 rad 2 ). The covariance in heading is not affected due to the fact that the vehicle remains on a straight path for the majority of the journey and only has to negotiate six curbs. The highest orientation covariance occurs at the bottom left of the map, when the vehicle performs a U-turn. Figure 15.12(b) shows the covariance variation for three landmarks during SLAM. As expected, the covariance in landmark position reduces due to increasing confidence in their position after several sightings. The difference between this experiment and the previous one is that in this case the landmarks covariances reduce at a slower rate.

318

D.C. Asmar, S.M. Abdallah, and J.S. Zelek

Fig. 15.11. SLAM map developed during Experiment ‘two’ (auto tree detection and recognition, manual initialization). ‘T’ refers to a Tree, ‘L’ refers to a Lampost, ‘P’ refers to a Post, and ‘FH’ refers to a Fire Hydrant. The red and green arrows are overlaid on the map to simply help follow the trajectory of the vehicle from ‘Start’ to ‘End’.

a

b

c

Fig. 15.12. (a) Covariance for three landmarks during Experiment ‘two’ (auto tree detection and recognition, manual initialization). As an example P7x is the covariances of tree 7 in the x direction. (b) Covariance for vehicle position for Experiment ‘two’ (auto tree detection and recognition, manual initialization). Px and Py are the covariances in the x and y directions respectively. (c) Innovations taken during Experiment ‘two’ (auto tree detection and recognition, manual initialization). Px and Py are the covariances in the x and y directions respectively.

15 Vision SLAM Maps: Towards Richer Content

319

Fig. 15.13. SLAM map developed during Experiment ‘three’ (auto tree detection, recognition, and initialization). ‘T’ refers to a Tree, ‘L’ refers to a Lampost, ‘P’ refers to a Post, and ‘FH’ refers to a Fire Hydrant. The red and green arrows are overlaid on the map to simply help follow the trajectory of the vehicle from ‘Start’ to ‘End’.

Notice that the final covariances here are not as low as those in Fig. 15.10(c). This result is expected since all landmarks, including these three (‘T7’, ‘T10’, and ‘T27’) are recognized fewer times during the journey, resulting in less reduction of covariance. Looking at Figure 15.12(c), it is apparent that a much smaller number of innovations occur than in the previous experiment (270 verses 700), reflecting the fact that many less landmarks are recognized than in the manual recognition case. Less landmarks signifies less SLAM updates and a less stable system. In this last test, SLAM automatically detects, recognizes, and initializes trees. Landmark initialization is performed using depth information extracted from a stereo camera. Since the stereo system only initializes trees that enter into the range of the camera, a smaller number of landmarks are initiated into the SLAM state during the journey of the mobile platform. The SLAM map shown in Fig. 15.13 shows a degeneration in the results of SLAM from the previous two tests. Landmarks are no longer initialized at their correct position, which results in eventually destabilizing the EKF. In the outbound journey (red and white) SLAM faithfully tracks the correct vehicle trajectory while initializing landmarks into SLAM. Many of the landmarks (e.g., ‘L2’, ‘T9’, ’T24’) are initialized at their correct positions. Those landmarks falsely initialized (e.g., ‘T3’, ‘T4’,

320

D.C. Asmar, S.M. Abdallah, and J.S. Zelek

a

b

c

Fig. 15.14. (a) Covariance for vehicle position for Experiment ‘three’ (auto tree detection, recognition and initialization). Px and Py are the covariances in the x and y directions respectively. (b) Covariance for vehicle position for Experiment ‘three’ (auto tree detection, recognition, and initialization). Px and Py are the covariances in the x and y directions respectively. (c) Innovations taken during Experiment ‘three’ (auto tree detection, recognition, and initialization). Px and Py are the covariances in the x and y directions respectively.

‘L1’) appear to slowly converge towards their correct positions. On the inbound journey, SLAM is de-stabilized near ‘T9’ and the INS dead-reckoning is lacking. The reason for this failure is primarily caused by not initializing ‘P1’, which in the previous tests contributed to the stability of SLAM around the bend after tree ‘T9’. The system does not recover after that and requires a second loop around the park to correct the SLAM state. This weakness in landmark initialization suggests that either an alternate method is necessary or the current method should be aided by a second initialization technique. Interestingly enough, the vehicle covariance in position (Fig. 15.14(a)) is lower than the one in the second test. This result makes sense since in the previous tests landmark position estimates initially are independent of that of the vehicle; whereas in this test landmarks are initialized at positions that are related to those of the vehicle. Therefore, smaller innovations are expected here, which in turn result in lower covariances. In Figure 15.14(a), the covariance in vehicle position is approximately one order of magnitude lower than the previous test. The rest is expected for the sames reasons of the vehicle heading. Figure 15.14(b) shows the covariance variation for the same three landmarks as in the previous two experiments (‘T7’, ‘T10’, and ‘T27’). The first observation is that ‘T7’ covariance does not change. This is simply due to fact that ‘T7’ is never initialized throughout the vehicle journey. The position covariance of ‘T10’ reduces much later than in the previous tests because it is not initialized until a later time in the inbound journey. In Figure 15.14(c), innovations are plotted at all EKF updates. The first observation is that a much smaller number of innovations than before (175 versus 260 and 700), caused by fewer landmarks initialized and subsequently observed. Second, notice that many spikes exist, which is indicative of the much longer time the system uses it dead-reckoning estimation.

15 Vision SLAM Maps: Towards Richer Content

321

In conclusion, it is unfortunate to report at this time that maps of natural objects generated via a full Vision SLAM system are lacking. Although the results are promising, much work remains to be done. The primary shortcoming of the system is in the visual sensor in terms of detection, recognition, and initialization. The need is therefore evident for more capable and robust Computer Vision systems capable of overcoming the above shortcomings. Research effort should also be put in improving the INS system because a more accurate ego-motion estimate could help mediate the detrimental effect of a mismatched or undetected tree. 15.5.1

Run-Time

All of the SLAM modules, except for the SIFT feature extraction, are written in Matlab and implemented off-line on a laptop featuring a 1.8 GHz Pentium M processor, and 1GB of RAM. All processing times are shown in Table 15.1. SIFT features are extracted using the code of Lowe [16] which is written in C. The time for extracting SIFTs is added to the tree recognition time. The size of all the images is 640x480 pixels. Each INS prediction requires 0.25 seconds. This time includes the IMU prediction as well as the application of non-holonomic constraints. The average time spent for tree detection is approximately 20.22 seconds for a typical image containing two trees. This time includes importing an image, edge detection, line tracking, line pruning, continuity and symmetry calculations and line grouping. Furthermore, this time also includes filtering the image with a Gabor filter (2 scales and 2 orientations). Each tree recognition step requires approximately 7 seconds. This time includes SIFT extraction, and matching 3 of them to the those saved in the SIFT database. Tree initialization requires approximately 4.78 s and is in effect the time for the stereo camera API to unwrap a stereo image, rectify the resulting right and left images, extract the disparity map, and calculate depth. A SLAM update involves executing the EKF equations which is done in a fraction of a second. The total time for a SLAM iteration, including prediction, observation, and update is 32.25 seconds. Although these numbers are unacceptable for a real-time implementation, all of our results were computed off-line. The computer vision components are the most time-consuming and can be sped up if they are re-written in a non-interpretive language such as C and compiled in an optimal fashion on an embedded platform, possibly using a DSP (Digital Signal Processor) platform. Table 15.1. Run-time for SLAM processes

Operation time (s) INS prediction .25 Tree detection 20.22 Tree recognition 7.00 Tree initialization 4.78 SLAM update ∼0 Total SLAM iteration 32.25

322

D.C. Asmar, S.M. Abdallah, and J.S. Zelek

15.6 Conclusions This chapter presented a Vision-Inertial SLAM system that uses real natural objects as landmarks. Using real objects verses artificial ones affects the type of maps that are obtained. Feature-based maps are uninformative because they are composed of low level features; object-based maps serve as navigation maps for humans or for the sake of surveying a site. Environment recognition determines the robot context and dictates the type of landmark to use for SLAM. Trees are detected, recognized and localized via a Computer Vision system. Trees are recognized by matching their distinctive features to those of model trees in feature space. SLAM is tested on real data collected onboard a mobile platform while it is navigating in a park setting. Three scenarios for these tests are proposed from semi automatic to fully automatic. While the first test results in an loop closure error of 2.4 meters in position and 2 degrees in orientation, the second test achieves an error of 9 meters in position and 3 degrees in bearing. Furthermore, in both of these tests, the covariance in vehicle position is low (approximately 4m2 ) with the exception of some spikes which are evident in the second test due to erroneous data associations. The vehicle covariance in heading is low for both tests ranging around 0.002 rad 2 . For the last test, where SLAM is completely autonomous, the performance is considerably degraded. Loop closure error in position is not acceptable (30 meters in position), although the error in heading is still low. The covariance in vehicle position during the journey is high, indicating the need for a better initialization technique. The poor results in loop closure for the third test can be blamed on the computer vision component. Note that the tree recognition rate is only 85% and that this error rate will propagate through the tour of the park. Also the stereo camera range limits reduces the number of trees initialized as landmarks, reducing the robustness of the approach. This possibly suggests that we should not rely on a single landmark in any environment. Another possible route of improvement is the sampling of our image capture. During vehicle motion, the camera pans back and forth in increments of π /4 between +π /2 and −π /2 in the body reference frame. The rate of movement of the camera is 53 π radians per second. Given these shortcomings which can be addressed, we suggest that the use of real natural landmarks is the direction of future research in using computer vision with SLAM. The bonus is that the use of real natural landmarks opens up the opportunity of creating a meaningful purposeful map.

Acknowledgments The authors would like to acknowledge the Canadian NSERC (National Sciences and Engineering Research Council) and the University Research Board at the American University of Beirut for their partial support.

Appendix This section deals with issues related to the SLAM experiments including the equipment used, the experimental setting and the collected data sets.

15 Vision SLAM Maps: Towards Richer Content

323

Equipment The experimental equipment consists of a hand held Global Positioning System (GPS), an Inertial Measurement Unit (IMU) and a stereo camera. These three units are mounted on a mobile platform (Figure 15.15), which is manually guided around a track. The GPS unit is a carrier phase hand held unit made by Trimble, model named “GeoExplorer3” [13]. Accurate and precise absolute position is assured by tracking between 4 to 12 satellites at one time. Carrier phase processing is possible for sub-meter precision. The GeoExplorer3 is a stand alone item, where data is collected on site and later downloaded to the computer. The collected data is differentially corrected during post-processing. The IMU is a three-axis strap-down inertial sensor made by Cloud Cap Technologies, model named “Crista” [11]. It comprises three orthogonal Micro Electro Mechanical System (MEMS) accelerometers and three gyroscopic rate sensors. The gyros and accelerometers feature ±300◦/sec and ±10g angular velocity and linear acceleration ranges respectively. The gyros operate below 0.05% Full Scale(FS) bias error and below 1% scale error. The accelerometers operate below 0.025%FS bias error and below 1% scale error. This unit is calibrated over −40◦C to 80◦C temperature range. The output rate and over-sample averaging rates of output data can be controlled by the user. Over-samples are the number of A/D measurements made and averaged for each signal for each IMU update. The suggested IMU rates are 10Hz sampling rate and 200Hz over-sampling rate. The camera is a stereo camera made by Pt. Grey, model named “Bumblebee” [14]. This unit features two 1/3 progressive scan CCD’s. Accurate pre-calibration is done for lens distortions and camera misalignments. The frame rate is controllable, peaking at a maximum frame rate of 30 f pm. Connection to the camera is insured via IEEE1394 firewire. The focal length of the cameras is 4mm, which corresponds to 70◦ Field of

a

b

Fig. 15.15. Experimental equipment. (a) Cart used to collect data, equipped with PTU, stereo camera, hand held GPS (not shown) and strap-down IMU. (b) hand held GPS and strap-down IMU.

324

D.C. Asmar, S.M. Abdallah, and J.S. Zelek

Fig. 15.16. Aerial image of the test site where IMU and GPS data, as well as stereo images were collected for Vision SLAM navigation runs. The location is the open area near St Jerome’s building at the University of Waterloo campus. The overlaid circles are the landmarks (tree trunks) that will be used for Vision SLAM.

View (FOV). Images are aligned within 0.05 pixel RMS error. The unit is equipped with a calibration retention system that prevents it from losing calibration when the device is subject to mechanical shock and vibration. The camera is mounted on a Pan Tilt Unit (PTU). The PTU is the “PTU-D46-17” model manufactured by Directed Perception [12]. It features a 78◦ tilt angle range and a 318◦ pan angle range. Its maximum speed is 300◦/s with on-the-fly speed and position changes. A laptop is mounted on the vehicle to control the PTU unit and collect IMU data and images of the surroundings during navigation. The laptop features a Pentium M, 1.5 GHz processor with 1MB of onboard RAM. The IMU, stereo camera, and PTU unit are powered by a 12V battery pack [10].

15 Vision SLAM Maps: Towards Richer Content

Tree 1

Tree 2

Tree 3

Tree 4

Tree 5

Tree 6

Tree 7

Tree 8

Tree 9

Tree 10

Tree 11

Tree 12

Fig. 15.17. Trees in the test site

325

326

D.C. Asmar, S.M. Abdallah, and J.S. Zelek

Table 15.2. IMU data file format. ‘N’ signifies that this column is not used. ‘Clks’ stands for the number of clock cycles that have passed since the IMU was initiated. The clock is a 10 MHz and recycles each 429 seconds. ‘S’ is an 8 bit sequence. ‘h’, ‘m’, and ‘s’ are the PC clock in hours, minutes and seconds respectively. ‘Gx’,‘Gy’, and ‘Gz’ are the gyroscope readings in ◦ /s and ‘Ax’, ‘Ay’, and ‘Az’ are the accelerometer readings in m/s2 .

N Clks S N h m s Gx Gy Gz Ax Ay Az N N N - Clks # - h m s ◦ /s ◦ /s ◦ /s m/s2 m/s2 m/s2 - - - - - - - - - - - - - - -

Environmental Setting and Ground Truth An outdoor area is sought that is relatively flat and smooth (no inclinations higher than 10◦ ) with few bumps, populated with large trees exhibiting visible trunks that could be used as landmarks. Keeping in mind these constraints, the open area outside St Jerome’s building on the University of Waterloo campus is chosen (Figure 15.16). This area spans a width of 70 meters by a length of 93 meters. A total of 25 conifer trees occupy the area, through which run several intertwining paved paths. The trees exhibit tree trunks that are large and salient. Determining the effectiveness of any SLAM system requires first establishing ground truth for the coordinates of the tracked landmarks and the pose of the vehicle as it navigates. These issues are tackled in the following section. Ground truth for the experimental setting is acquired from a geographically referenced aerial image of the site [24]. This consists of an image similar to a photographic image, where each pixel is reference to a latitude and longitude. Next, a Computer Aided Drawing (CAD) of the site is obtained and overlaid on top of the geographically referenced image. The CAD image comprises the features of the site, such as paths, trees, and buildings. Vector data images (i.e., CAD) are often out of date and lacking. For the sake of this work, the 2D location of trees trunks and their diameter is of interest for developing ground truth. This information is not available and had to be collected. The standard method to locate features in Geographic Information Systems (GIS) is to locate oneself next to the object with a hand held Global Positioning System (GPS) and fixate the object via the GPS. This procedure is not possible to locate trees since GPS suffers from outages and multipath errors under dense foliage such as conifers. Due to this limitation, it is necessary to fixate alternative features (lampposts) that are distant from foliage using GPS and then locate the position of trees by triangulation. The images of each of the trees that is used as a landmark are shown in Figures 15.17 and 15.18. Ground truth for the pose of the vehicle during navigation is established via a differential GPS. The vehicle follows a path that is distant from trees for the majority of the time, where GPS reception is adequate for positioning. Data Sets The information collected from the IMU, camera, and gyroscope consist of three data files and a sequence of images. The first data file, entitled ‘IMU *.txt’, includes information pertinent to the IMU sensor and its format is presented in Table 15.2. The time stamp at which an IMU reading is taken can be calculated in one of two methods. In the first method, the time stamp is calculated as the simple addition of the

15 Vision SLAM Maps: Towards Richer Content

Tree 13

Tree 14

Tree 15

Tree 16

Tree 17

Tree 18

Tree 19

Tree 20

Tree 22

327

Fig. 15.18. Trees in the test site Table 15.3. Image data file format including the time stamp of each image in hours, minutes and seconds. ‘N’ signifies not used.

Iter h m s N N N # hms - - - - - - - - -

hour, minute, and second entries of the PC clock. This time stamp represents the time at which the IMU reading is processed by the PC and includes some delay in it. The second, more precise method is to use the PC clock only to obtain an initial time stamp t0 and then calculate subsequent time stamps by adding the number of PC clocks to t0

328

D.C. Asmar, S.M. Abdallah, and J.S. Zelek

Table 15.4. GPS data file format. ‘N’ signifies that this column is not used. ‘Long’ stands for longitude, ‘Lat’ stands for latitude. ‘h’, ‘m’, and ‘s’ are the GPS clock in hours, minutes and seconds respectively.

Long Lat hh mm ss m m hh mm ss - - - 1 , (15.21) 107 where tk is the time stamp at IMU reading k, Clks is the number of clock cycles since t0 and the number 107 refers to the clock frequency. Although this second time stamp is more representative of the time an IMU reading is taken, better synchronization with the camera images are obtained using the first method. The second data file ‘IMG *.txt’, shown in Table 15.3 relates to the images captured during navigation and each entry is time stamped based on the PC clock. The time stamps in the IMU and IMG data files are synchronized since they are both based on the PC clock. The final data file is the one related to GPS readings. The first two columns include longitude and latitude readings. The remaining column represent the GPS time stamp, which is acquired from the atomic clock onboard the GPS satellite and is extremely precise. The GPS time stamps are synchronized to those of the IMU and IMG files by observing the time differential between the PC clock and that of the GPS unit. In addition to the three text files, stereo images are saved in a folder to be later uncoupled into their corresponding right and left images. tk = t0 + Clks ∗

References 1. Abdallah, S.M., Asmar, D.C., Zelek, J.S.: Towards benchmarks for vision slam algorithms. In: IEEE Proceedings of the Conference on Robotics and Automation, Orlando, USA (2006) 2. Asmar, D.C., Zelek, J.S., Abdallah, S.M.: SmartSLAM: localization and mapping across multi-environments. In: IEEE Proceedings of the Conference on Systems, Man & Cybernetics (Robotics and Intelligent Sensing VI), The Hague, Netherlands (2004) 3. Asmar, D.C., Zelek, J.S., Abdallah, S.M.: Seeing the trees before the forest. In: Computer and Robot Vision, Victoria, BC, Canada (2005) 4. Asmar, D.C., Zelek, J.S., Abdallah, S.M.: Tree trunks as landmarks for outdoor vision SLAM. In: IEEE Proceedings of Computer Vision and Pattern Recognition (Perceptual Organization Workshop), New York, NY, U.S.A (2006) 5. Bryson, M., Sukkarieh, S.: Bearing-only SLAM for an airborne vehicle. Tech. rep., ARC Centre for Excellence in Autonomous Systems, Australian Centre for Field Robotics, University of Sydney (2005) 6. Canny, J.F.: A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 8, 679–698 (1986) 7. Dissanayake, G., Sukkarieh, S., Nebot, E., Durrant-Whyte, H.: The aiding of a low-cost strapdown inertial measurement unit using vehicle model constraints for land vehicle applications. IEEE Transactions on Robotics and Automation 17 (2001)

15 Vision SLAM Maps: Towards Richer Content

329

8. Fitzgibbons, T.: Visual-based simultaneous localisation and mapping. Ph.D. thesis, University of Sydney, Sydney, Australia (2004) 9. http://www.acfr.usyd.edu.au/homepages/academic/tbailey/ tbailey/index.html 10. http://www.canadiantire.ca/assortments/ 11. http://www.cloudcaptech.com/ 12. http://www.dperception.com/ 13. http://www.geoplane.com/trimble/geoexplorer.html 14. http://www.ptgrey.com/ 15. Kim, J.H., Sukkarieh, S.: Airborne simultaneous localization and map building. In: IEEE Proceedings on the Conference on Robotics and Automation, Taipei, Taiwan (2003) 16. Lowe, D.G.: Distinctive image features from scale invariant keypoints. International Journal of Computer Vision 60(2), 91–110 (2004) 17. Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 257–263 (2003), http://lear.inrialpes.fr/pubs/2003/MS03 18. Oliva, A., Torralba, A.: Modeling the shape of the scene: A holistic representation fo the spatial envelope. International Journal of Computer Vision 42(3), 145–175 (2001) 19. Panzieri, S., Pascucci, F., Ulivi, G.: Vision based navigation using Kalman approach for SLAM. Tech. rep., Universit´a degli Studi La Sapienza (2001) 20. Tieu, K., Viola, P.: Boosting image retrieval. In: IEEE Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 228–235 (2000), citeseer.nj.nec.com/tieu00boosting.html 21. Torralba, A.: Contextual priming for object detection. Int. J. Comput. Vision 22. Torralba, A., Murphy, K.P., Freeman, W.T., Rubin, M.A.: Context-based vision system for place and object recognition. In: Proceedings of the International Conference on Computer Vision (2003), http://www.ai.mit.edu/∼murphyk/Papers/iccv03.pdf 23. Fazl, E.E., Zelek, J.S.: IEEE Fourth Canadian Conference on Computer and Robot Vision. Local Graph Matching for Object Category Recognition, pp. 73–80 (2007) 24. Waterloo, U.: Geography department (2005), http://www.fes.uwaterloo.ca/geography/

16 Learning of Hierarchical Perception-Action Cycles Florian Hoppe and Gerald Sommer Cognitive Systems Group, Christian-Albrechts-University of Kiel, Germany {fh,gs}@ks.informatik.uni-kiel.de

Abstract. To ease the development of artificial cognitive systems, a new software architecture is proposed in this chapter. It makes extensive use of machine learning techniques that handle data of different degrees of abstraction. The architecture is a hierarchical structure of multiple modules that realize nested feedback cycles. Data that represents the perceptions of the system is sent from the lower to the higher levels. In the other direction action commands are fed back. These perception-action cycles run at different speed and with different types of data. This allows that the lowest levels can control the actuators in a reactive manner based on frequently acquired continuous sensor data. On the other hand the higher levels can handle long-term goals represented with abstract data types. Applying incremental learning techniques makes it possible to realize a visual guided robotic system that can adapt its abilities to changes of the environment. This chapter presents the overview of a robotic system that is programmed according to the above outlined principles. Its task is to learn to solve a shape-sorter puzzle which contains blocks of different shapes and colors and a plate with appropriately shaped holes. The system comprises a 6 DOF robot arm and two video cameras. Two learning tasks of the system are explained in detail. One is solved by a module from the lower level that has to drive the robot arm guided by visual feedback. It demonstrates that variations of the camera position can robustly be compensated by the system. The other module that is presented works on the highest level and is responsible to generate a correct sequence of basic actions to solve the puzzle. It uses a new machine learning technique and a special coding scheme that allows to learn an equivalent of a finite state automaton. This automaton represents the strategy to solve the puzzle game. The performance of both modules is demonstrated with experiments.

16.1 Introduction As Erdmann [9] concisely stated, the ultimate goal in engineering robot systems is: “We want specific things to happen quickly and cheaply.” As the Czech word ’robot’ implies we human want that artificial systems act as workers for us. Robots should take over all these jobs which are physically too demanding, too exhausting or just too tedious for us. In each application area robots have to perform specific tasks in an efficient manner. So, the expenses of resources for robots should be well balanced with the gained benefit in product quality or in comfort of our life. A very important component of this economical constraint is the needed effort to develop robot systems because – simply stated – the demanded tasks can not easily be solved. The plain reason for that is, the world we are situated in is very complex. This property of the world is grounded in the huge variations of its state: everything is constantly changing at different time scales. Hence acting in the world depends on an adequate perception of its state. D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 331–353. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

332

F. Hoppe and G. Sommer

To realize the demanded tasks robots require three non-trivial components: appropriate effectors, instructive sensors and powerful working strategies. Of course, effectors have to have the right size, dexterity and strength to handle things or to move a robot. Sensors must acquire a system internal picture of the world that serves as a valid base to realize the desired activity. Last but not least, the interplay between action and perception has to be well organized so that to every possible situation an appropriate response is computed. Latter is at the moment the most demanding research topic in the field of robot development. The attainable precision and power of effectors and sensors is far beyond those of humans or animals. But still, even smallest insects demonstrate competences and adaptivity in natural environments that are impressive when compared with common artificial systems. 16.1.1

Challenges and State of the Art Solutions

One hard problem that limits robots to realize complex and goal directed actions is the purposive interpretation of sensory data. Software modules have to bridge the big gap between the available low-level signals and the needed instructive information to control the effectors. Especially the interpretation of image data has not been accomplished with generic methods. The existing ones that are sufficient fast and robust are highly customized (the methods and/or the environment). Characteristic for the data processing in robotic systems is the need of both abstraction and concretization. On one hand, noisy low-level signals have to be stripped of all the components that are irrelevant to a task. The signals have to be transformed so that invariants in the data can be compared to high-level goals of a system. On the other hand, these abstract goals have to be translated into concrete motor commands that precisely drive the effectors. This chain of data processing has to be reflected in the system’s software architecture. But there exist quite different ideas to solve this. Classical AI approaches delineate three functional modules: a sensing, planning and acting component [20]. The first module generates an internal model of the world which is matched by the planning module against a given goal state. In case of a disagreement a symbol-based reasoning mechanism has to devise a plan which is finally executed by the acting component. Characteristic for such systems is their strong emphasis on symbol processing which implies extensive modelling of the world and an unidirectional linear data stream from the sensors through the planning component to the effectors. Tests outside of customized environments showed that this approach is not useful in practice. The world is too complex to be manually modelled. Open-loop plan execution turned out to be inadequate to realize systems capable of fast reactions. So, it became obvious that abstract reasoning is not as important for acting in the world as assumed. Consequently, Brooks proposed with his famous subsumption architecture [4] the afterwards so-called behavioral design approach. He and his numerous successors argue that complex behaviors of robots can be produced by a loosely coupled network of simple asynchronous processors that have full access to both sensors and effectors. The full access is necessary to realize fast feedback cycles that involve perceptions and actions. They follow a strict bottom-up design approach which emphasizes

16 Learning of Hierarchical Perception-Action Cycles

333

low-level reactivity instead of high-level planning. The goal was to realize systems that are adaptive to their environment because a new insight was that intelligent behavior of an acting entity is a product of the entity and its environment. The argumentation is that quite simple mechanisms in robots can make specific things happen as long as these robots fit to their environment. A simple example for this problem solving strategy is the combination of more or less random moves with basic obstacle avoidance methods that drive a mobile robot like “Roomba” (see ’www.irobot.com’) through rooms to vacuum-clean them. This design paradigm can be seen as the antithesis to symbol-based AI. Brooks itself stresses this rivalry with articles entitled “Elephants don’t play chess” [5] and “Intelligence without reason” [3]. But his key concepts of Situatedness, Embodiment, Intelligence and Emergence [3] (see [21, 22] for reformulations) lead to a serious problem: the system designer is quite limited to introduce high-level goals into the system. The architecture is not meant to contain a centralized module that plans and pursues longterm goals. So, the designer can not easily integrate knowledge that would be gained by a normal engineering processes where goals are decomposed into subgoals. Instead one is supposed to define only basic behaviors from which the intended overall behavior has to emerge. But it remains unclear what kind of generic design principles can be applied to develop behavioral robots that do what they are intended to do (for detailed criticism see [6, 21, 25]). The synthesis of these two research directions has become present in visual robotic systems. In 1991 three research groups [2, 8, 12] independently developed quite similar architectures with three layers that combines purely reactive with long-term planning modules (for an overview see [13]). An intermediate layer is used to keep the balance between these two extremes. This architecture is still relevant as it was applied e.g. in the robot that won the first time the DARPA Grand Challenge [24]. 16.1.2

Machine Learning Techniques in Robotics

Another substantial shift in methodology is the replacement of intensive modelling by the appliance of learning strategies. Two reasons are important: explicit modelling of the world is very time-demanding and the results tend to be too inflexible. The advances in machine learning techniques (MLT) made is possible to teach different aspects of robotic tasks so that the system generalizes over the given training samples. Furthermore, MLTs can help to realize modules (especially those directly connected to sensors) that are adaptive to changes in the environment. So, learning strategies became very important to ease the development and increase the robustness of visual robotic systems. The typical task is to learn a transfer function that maps directly from sensor data to motor commands which steer the effectors. Such transfer functions define basic behaviors like e.g. wall-following or obstacle avoidance since they are used to realize close-loop controllers. These are meant to perform fast perception-action cycles that are necessary for highly reactive systems. These transfer functions are classically hand-crafted by means of explicit models. But they can also be trained with different learning strategies. E.g. in [21] and [15] an arm robot performs certain movements and the resulting changes of sensory data (i.e. image position of a tracked object) are measured. By this means samples can be

334

F. Hoppe and G. Sommer

collected how robot commands correspond to sensory data. Given a set of such samples a supervised learning technique is used to establish the needed transfer function. Although in robotics the concept of transfer functions is mainly associated with lowlevel controller, it is quite general. These functions can be used at various levels within a software architecture working on differently abstract data. E.g. instead of pure image data, classified object features could be the input of a transfer function and its output could be the invocation of one basic behavior. This type of task is typically solved with manually designed finite state automatons [13]. These have the disadvantage that they are fixed. In [7] a framework is proposed that learns such transfer functions with first order predicate logic. An alternative is described in [10] where the dynamic field formalization of Amari [1] is chosen to model pattern formations in neuronal tissue. In both cases the correct sequence of basic behaviors had to be learned by observing a human instructor. Furthermore transfer functions offer a good possibility to introduce high-level goals into low-level controller. The realized basic behavior can be conditioned by decisions made at different levels within the software architecture. To do so, internal state variables can be added to the sensory data that serve as input for the transfer function. These state variables can reflect the past or the desired future of the system in order to alternate the actions of the robot. Learning on Different Time Scales One can also ask at which point of time should learning happen in a visual robotic system. In [23] three major time scales are suggested: the ontogenetic, the refinement and the situated time scale. In the first phase, offline learning methods should be used in order to bootstrap basically those competences without the system would not be able to do anything. This phase is also helpful if the learning problem is too complex to be handled during the operation of the system (e.g. because the training sample acquisition is too time demanding). On the refinement scale, the basic competences can be improved during their actual operation by means of online learning principles. Typical examples are rather slowly changing calibration problems such as changes of the lighting over the day. The fastest learning mechanisms are needed for the situated time scale. Ultimately, the system should be capable of learning from single examples like the demonstration of a perfect swing with a golf club. 16.1.3

Structure of the Chapter

The remaining of the text is organized as follows. In the next section the general ideas of the proposed software architecture and a concrete robot system are presented. The implementation of two submodules of the realized system are detailed in Sec. 16.3 and validated with experiments described in the consecutive section. The chapter is concluded with a summary of it.

16.2 Software Architecture for Artificial Cognitive Systems The following subsection proposes a software architecture suited for artificial cognitive systems i.e. robotic systems that exhibit reactive behavior in and adaptivity to their

16 Learning of Hierarchical Perception-Action Cycles

335

environment. In Sec. 16.2.2 such a concrete system is presented that served as a testbed for our approach. 16.2.1

Overview of the Software Architecture

The proposed software architecture is structured similar to the three-layer architecture outlined in the last section. The system contains three software modules: the perceptionaction (PA), the grounding-management (GM) and the symbol processing module (SP). As the names should indicate, the modules encapsulate functionalities that are working on data of different degrees of abstraction. The PA module realizes the competences at the lowest level where tasks such as image processing, object recognition and basic robot behaviors are solved. The goal of the GM modules is to allow data exchanges between the PA and the SP module. This means that – in the direction from PA to SP – continuous signals are classified in order to ground symbols. The other way round, the GM module resolves symbolic IDs that refer to entities in the world to their non-symbolic representation. The SP module is responsible to fulfil the long-term goals of the system and offers an interface to the user. Besides the type of processed data, the three modules can also be characterized w.r.t. their scope of time and knowledge about the world. As already hinted, the SP module works on the global scale: it establishes and maintains plans to solve the overall task. To do so, it builds the most complete and most abstract model of the world as it is perceived by the system. On the other extreme, the PA module is concerned with shortterm goals that are realized with a small but detailed portion of all sensory data. At the intermediate level, the GM module connects these two perspectives onto the world by means of a so-called short-term memory. Design Principles Characteristic to the proposed architecture is the strong emphasis on learning. The goal was to create a system that can learn at all levels of its architecture. The competences should be acquired, refined or extended at all three time scales. To realize this a recently introduced machine learning technique [15, 16, 17] is applied in various sub-modules distributed over the whole system. This so-called Hierarchical Network of Locally Arranged Models (HLAM) is a supervised function approximation technique i.e. it builds a model of a function that is only implicitly given as a set of input-output samples. The HLAM approach was especially designed to support incremental learning tasks that can handle data of different degrees of abstraction. In contrast to common MLTs (e.g. Multi-layer Perceptrons, Support Vector Machines), it offers means to exploit task specific knowledge for a given learning problem. Another important feature of the system is the use of layered feedback cycles. To give a simple example: whenever the SP module commands a basic behavior (i.e. a special type of movement) it will finally receive a high-level feedback about the outcome of the action. This feedback is generated by similar processes started at other levels of the architecture. The difference of these processes lays in the type of exchanged data and the speed of the feedback cycles. The fastest are running at the lowest level in the PA module. There the robot will be commanded to drive some millimeters and the feedback will be extracted from a new acquired camera image of the changed scene.

336

F. Hoppe and G. Sommer

16.2.2

Concrete Visual Robotic System

To study the abilities of the outlined software architecture, a shape-sorter puzzle was chosen as a concrete task. An exemplary setup of the system is shown in Fig. 16.1. Besides the industrial arm-robot and two cameras, the shape-sorter box and the puzzle blocks are visible. The goal is that the robot learns to insert the blocks into the correct holes of the box. How this demonstrator looks like in detail is described in the next subsections by clarifying its physical constraints and the dynamics of the robot.

Fig. 16.1. General setup of the implemented visual robotic system. Two static cameras are observing the robot and the shape-sorter puzzle.

16.2.2.1 Static Properties The following enumeration specifies the setup of the implemented visual robotic system and the physical constraints of the shape-sorter puzzle: • The central part of the system is a St¨aubli RX90 industrial arm robot with six DOFs. It is equipped with a gripper with two parallel bars as fingers. • Two color video cameras (Sony DFW X710, focal length 6.0 mm) are used to observe the scene. The so-called end-effector camera is attached to the end-effector and can monitor everything underneath the gripper. The so-called static camera is mounted on a tripod placed beside the robot. The two cameras are setup so that everything needed is visible in their images. No other arrangements are made. The exact spatial relations between the coordinate systems of the cameras and of the robot are unknown. • The working area (i.e. the ground where puzzle blocks can be placed) is planar with a size of approximately 25 cm × 35 cm. It is virtually parallel to the x-y-plane of the robot coordinate system.

16 Learning of Hierarchical Perception-Action Cycles

337

• The puzzle game comprises a number of blocks and a plate with five matching holes. The blocks are of uniform color (red, green, yellow, blue, or silver) and have either the form of a cylinder, a bridge, a triangle, a cube, or a crescent. In the following both blocks and holes are referred to as objects. The plate with holes is originally the cover of the shape-sorter box, but it is placed directly onto the working area for the experiments. • The end-effector is marked with an uniquely colored round patch which is always visible in the static camera. This patch is tracked by the object recognition method in order to estimate an image position of the robot. 16.2.3

Dynamic of the Demonstrator

In the following it is explained how the robot movements look like when the puzzle is played by the system. Four basic behaviors are defined: objects have to be approached, the gripper has to be aligned to them, blocks have to be grabbed or inserted. With an approach movement the end-effector is driven in a plane parallel to the working area about 15 cm elevated above it. Such a movement stops whenever the target object becomes visible in the end-effector camera i.e. the gripper is coarsely positioned near the target object. The goal of the align movement is then to drive the gripper precisely over a target object and to align the fingers to the shape of it. For the grab and insert movement the gripper is first vertically lowered towards the working area, the fingers are closed or opened, respectively and then the gripper is lifted up again.

16.3 Example Submodules To point out the new strategies to develop a robot system, the implementation of two functionalities of the shape sorter are highlighted in this section (see [17] for a complete description). One is concerned with the visually guided control of the robot, while the other one explains how long-term goals are realized within the system. The first functionality - an adaptive visual servoing scheme - is realized with different submodules of the PA module. On the other hand the long term goals are handled in the SP module by a special finite state automaton that is trained on the basis of examples of successful puzzle game plays. Both implementations rely on the same idea that so-called transfer functions are trained to compute the correct action upon a given perception. The transfer functions are realized with the HLAM approach and differ w.r.t. the processed type of data and frequency of their invocation. 16.3.1

Internals of the Perception-Action Module

In order to explain the adaptive visual servoing scheme, first some more details of the PA module have to be given. Fig. 16.2 gives an overview of the internal structure of the PA module. It contains four submodules which are called pilot, navigator, observer and hardware access. The hardware access submodule is a straightforward interface to the robot and the two cameras attached to the system. It provides images of different resolutions (1024 ×

338

F. Hoppe and G. Sommer

Fig. 16.2. Internal structure of the perception-action module of the robot system. The arrows inside of the PA module indicate which submodules send and receive status information. The arrows outside of the boxes symbolize that the system realizes perception-action cycles that are working with different scopes of knowledge running with different frequencies.

768 pixels for the end-effector and 800 × 600 pixels for the static camera), the position of the robot (x-y-z position and yaw-pitch-roll angles) and status information about the gripper (either closed, totally opened or holding something). On the other hand, it receives target positions for the robot (6D) and commands for the gripper (open or close). The image processing routines of the observer submodule solve object recognition and categorization problems. The observer provides different information about recognized objects (i.e. blocks or holes). From the static camera image the x-y position and the color class of the object is extracted. From the image of the end-effector camera a 3D pose (x-y coordinate and an angle) and three labels are estimated. These labels encode the object’s color, shape and type. As color and shape class labels the observer outputs unique integers. Color labels can be red, green, yellow, blue, silver, and black (for the holes), whereas shape labels are given for cylinders, bridges, triangles, cubes, and crescents. The type label defines ten different classes for all objects of these five shapes: A unique type label is given to each hole and to all blocks with different colors but equal shape. This type label allows a simple discrimination between holes and blocks and the different shape classes. The pilot differs from the navigator submodule w.r.t. to its scope of knowledge about a currently performed behavior. The general idea is that the pilot has a local view onto the task, while the navigator has more extensive responsibilities. This is for example reflected in the fact that the navigator defines the target position of the end-effector while the pilot computes the single steps to reach this position. Significant for this division is that the perception-action cycles realized by the pilot are performed with a higher frequency than those of the navigator. The navigator encloses the alternations between an image processing step of the observer and a robot movement of the pilot. With this principle the pilot realizes for the navigator subtasks of the different behaviors.

16 Learning of Hierarchical Perception-Action Cycles

16.3.2

339

Adaptive Visual Servoing Scheme

The pilot realizes a visual servoing scheme1 that is able to drive the robot to a target position that is given in image coordinates. The basic idea is to implement an iterative process that gradually steers the robot towards the target position under continually visual control. This iterative process described in the next subsection is a very generic scheme. An example how it can be customized for specific movement is given in the last paragraph of this section. 16.3.2.1 Generic Adaptive Visual Servoing In Fig. 16.3 the realized visual servoing scheme is depicted. The figure shows in details those parts of the PA module that are important for the pilot submodule. It makes obvious that the base of the pilot is defined with different transfer functions. In every cycle t of the visual servoing loop, these functions map the target image position I ∗ , the most up-to-date image information It and robot state St to a robot command Δ St . This command defines how the robot has to change its position in order to approach the target position. It is executed by the robot controller that is a part of the used RX90 system.

Fig. 16.3. The control paths of the visual servoing scheme of with the pilot submodule

So, the key to realize a needed movements are the transfer functions. The functional structure stems from generic control theory. These transfer functions are specifically trained HLAMs. For the pilot submodule different HLAMs are realized (e.g. one to let the robot approach an object). These HLAMs use different image information (e.g. only positions from the static camera) and control different parameters of the robot (e.g. only the roll angle). But they are applied in the same generic algorithm that implements the visual servoing scheme and an online refinement of the transfer function. Its pseudo code is given in Alg. 1. Depending on the needed movement the algorithm is invoked by the navigator submodule with an image target position and a certain transfer function.2 It realizes the 1 2

The term visual servoing stems from [14]. It refers to control schemes that use image information as error signals to control a robot. For an overview see [19]. To be most general, these transfer functions are defined with I ∗ , I t and St as input arguments. But the actual implemented transfer functions use only some components of these information.

340

F. Hoppe and G. Sommer

Algorithm 1: Pseudo code of the online learning visual servoing control scheme that steeres the robot to the target image position I ∗ . With a specific control function f a specific movement can be realized. Function ServoRobotToTargetPosition Input : I ∗ , f begin t←1 It ←GetCurrentImageInformation() while (It − I ∗  > ε ) ∧ (t < T ) do St ←GetCurrentRobotAndGripperStatus() Δ St ← f t (I ∗ , It , St ) Δ S˜t ←EnsureValidity(Δ St ) DriveRobotTo(St + Δ S˜t ) It+1 ←GetCurrentImageInformation() f t+1 ←UpdateTransferFunction( f t , It , It+1 , St , Δ S˜t ) t ← t +1 if It − I ∗  < ε then return success else return failure end

above described visual servoing loop that essentially alternates status acquisition with robot movements. This loop runs as long as the sensed image position differs from the target position by a certain amount or too many attempts were made to reach it. For both criteria the individual thresholds (ε and T) have to be chosen appropriate to the application. To secure the system the robot command Δ St computed by the transfer function is processed by the procedure EnsureValidity. This procedure outputs the finally executed command Δ S˜t . It is useful to check that all robot parameters are within their valid range or to limit the magnitude of the movement. Besides driving the robot to a target position the algorithm enables the system to learn from its actions. In each cycle the information how the world looked like before and after the robot moved can be utilized to refine the transfer function. This is reasonable because of the following principles. The transfer function essentially maps a difference in image space to a difference in the robot coordinate system. The difference in image space is the displacement between a sensed image position It and the given target position I ∗ . While the difference in the robot coordinate system (the executed ˜ is representing the movement of the robot that is needed to realize the command Δ S) displacement in image space. On the other hand, every movement of the robot produces a change in the image position. It delivers a sample how a difference in the robot coordinate system is related to a difference in image space. In short, a movement reveals the correspondence between an executed robot command and the resulting change in image space. Exactly this relation has to be established by the transfer function. Hence, each movement can be employed to refine the transfer function. Such a correction is

16 Learning of Hierarchical Perception-Action Cycles

341

always appropriate when changes of the environment are possible or the function was only coarsely established in the first place. 16.3.2.2 Special Transfer Function for Approaching In the following the described adaptive visual servoing scheme is exemplified with the so-called approaching movement of the realized robot system. It is a movement of the end-effector parallel to the working field and is supposed to position the gripper in the vicinity of a target object (e.g. a puzzle block). To do so, the navigator submodule determines the target position r∗ of the robot given in coordinates of the static camera image and invokes the visual servoing scheme with a specialized transfer function fApproach (·). This function takes as input a 4-dimensional vector v and computes the next robot step Δ St . The first two components of v contain the target image position r∗ and the last two contain the difference rt − r∗ between the most up-to-date image position rt of the robot and its target image position r∗ . This definition is motivated further down below in details. The output of the function Δ St is the 6-dimensional vector that specifies a translation of the robot in its coordinate system. It defines how the robot changes its position to reach or to get at least closer to the target position. For the approaching movement only the x and y component of Δ St are non-zero since this realizes the needed movement parallel to the working area (cf. Sec. 16.2.2.1) without any rotation of the end-effector. The applied safety procedure EnsureValidity checks that the norm of Δ S does not exceed a certain threshold. By this means the step size of a robot movement in one cycle is limited to be smaller than e.g. 2 cm. This results in a movement that is homogeneously interrupted for possible corrections to reach the target position.   The function fApproach (r∗ , rt − r∗ )T is realized with an HLAM that is trained during a bootstrapping and the above described online refinement phase (i.e. whenever the visual servoing scheme is used for an approaching movement). The network is bootstrapped with training samples that are acquired with a small number of pre-specified movements. To do so, the end-effector is driven to some training positions on the approaching plane where its position r in the static camera image and the state S are saved. This results in a set PApproach = {(ri , Si )} of correspondences of robot positions in the image and the robot coordinate system. With two different samples i and j of P one training sample (x, y) for the HLAM is created where x = (ri , ri − r j )T , y = S j − Si .

(16.1)

According to this equation a training set for the HLAM is produced with all pairwise combinations of samples from P. Given this set the HLAM can be bootstrapped with one of the learning algorithm presented in [17]. During the refinement phase, with each cycle of the control loop, one training sample is generated according to Eq. (16.1). With this sample the network is updated by means of the HLAM online learning algorithm. The HLAM basically learns the correspondence between the difference of two image positions and the difference of the same positions in robot state space. Mainly due to perspective effects these correspondences vary at different positions of the robot. Two robot steps of same size will result in different position changes in the camera image

342

F. Hoppe and G. Sommer

depending on the distance of the end-effector to the camera. Similarly, lens distortions will change the appearance of robot movements between different local regions of the camera image. Because of this dependencies to local regions of the image the HLAM approach is specially well suited to solve this function approximation problem. It builds a solution out of a set of linear models that are only valid at local regions of the image (for more details see [17]). Another benefit of the proposed visual servoing scheme is its easy extensibility to movements with more degrees of freedom. E.g. movements in 3D can simply be realized by adding a second static camera to the system and driving the robot to more positions during the bootstrapping phase. The additional camera is needed to resolve ambiguities between 2D image and 3D robot positions. The extra training positions have to reflect that also the z-component of the robot state can vary. Everything else of the servoing algorithm and training method can remain unchanged. When comparing this learning based servoing approach with the literature, it becomes obvious that the HLAM approximates the pseudo inverse of the so-called image Jacobian [11, 18, 19]. This matrix expresses the differential relationship between the camera and the robot coordinate frame. It is normally derived from a certain camera model (e.g. pin hole camera with radial lens distortion). Such assumptions imply that the external and internal camera parameters have to be calibrated whenever something in the experimental setup changes. With the above proposed learning scheme the bootstrapping phase replaces the first of such calibrations. Any other needed recalibration can be omitted as the servoing algorithm ensures that the system adapts itself to environmental changes. The quality of this visual servoing scheme is validated in Sec. 16.4.1. 16.3.3

Learning the Long Term Goals of the Puzzle Game

This subsection explains how long term goals are learned and achieved by the realized system. Long term goals are needed to solve the shape sorter puzzle. It is the task of the SP module to define and pursue long term goals. Via the GM module the SP module can invoke the four basic behaviors (i.e. approach, align, grab and insert) realized by the PA module. The SP module has to ensure that the right basic behaviors are performed in the correct sequence and w.r.t. appropriate reference objects (i.e. puzzle blocks or holes). How this can be learned is described in the following. The key to an efficient learning scheme for the puzzle solver is a high-level representation of the correct game play. When many variations of the real-world situation (e.g. different positions of the blocks) can be expressed with a single code (i.e. symbol) less training samples are needed to learn the similarities between game situations. The set of the four basic behaviors and the different object type, color and shape labels represent already such an abstract view onto the game. With these symbols the game play can be coded as a sequence of – in the following called – command samples. Such a command sample defines which behavior was performed w.r.t. what kind of reference object. For example a red cylinder would be correctly grabbed and inserted into the black round hole by the command sequence: approach red round; align red round; grab red round; approach black round; align black round; insert black round. Such a sequence exemplifies how one specific block is prop-

erly handled. A number of sequences is needed to reveal that there is a common pattern

16 Learning of Hierarchical Perception-Action Cycles

343

within these behavior invocations and object attributes. This pattern is specific to the game and can be learned with such a high-level representation. 16.3.3.1 HLAMs to Learn the Correct Sequence of Basic Behaviors A number of HLAMs are designed to play the shape sorter game. More specifically, the HLAMs are used to generate a sequence of commands sent to the GM module so that a certain block is grabbed and inserted into the right hole. The whole puzzle can be solved by repeating such a sequence for all blocks that are in reach of the robot. Which block is handled at which point of time is finally irrelevant but must be specified when such a grabbing-inserting sequence should be performed. To do so, the user has the possibility to enter the color and shape label the next to be handled object should have. By this means he or she can supervise the system. Still, such supervising information have not necessarily be produced by a human user. The system itself can decide which block should be grabbed and inserted next. How this works is described in further below. After defining what information is available (i.e. training sequences of command samples and the supervising information) the strategy to learn the puzzle game can be explained. The basic idea is to learn with the HLAM approach a transfer function that maps a game situation to a command that is appropriate to be executed. A game situation summarizes the history and the present of the system, while the computed command represents the future of the system. For the shape sorter puzzle, a game situation is defined with the last executed command, the type, color and shape attributes of the last reference object and the supervising information. Given such a game situation the transfer function has to output which basic behavior should be executed and which color and shape attribute the next reference object should have. By repeatedly applying this transfer function the system will play the puzzle. Each execution of a command will change the game situation so that the correct sequence of actions will be produced. For the shape sorter puzzle it is sufficient that the history of actions that is encoded in the game situation description includes only the last action. This will be shown in experiments described in Sec. 16.4.2. For more complicated tasks the description of the game situation could be extended to former commands and more information about the present state of the system. The transfer function is realized with two different types of HLAMs. In a preprocessing step, one type of network is necessary to transform the available information about a game situation into a representation that serves as input for the second HLAM type. The latter computes the needed next command. HLAM for Transforming the Information about a Game Situation For each basic behavior one HLAM of the type needed for the preprocessing step has to be trained. Such an HLAM determines whether the basic behavior the network represents is applicable to a certain object or not. E.g. the HLAM trained for the grabbing behavior would indicate that a red and round object can be grasped, while that is not the case for a black and round object (because this represents a hole). These HLAMs essentially learn the relationship between the visual information about objects and the possibility to perform actions with them. To do so, the input of such an HLAM contains

344

F. Hoppe and G. Sommer

the type, the color and the shape label of an object. The output is a binary flag that denotes the applicability of the basic behavior. Training samples for this type of HLAM are generated with training sequences of command samples. These sequences are given as a set of samples where each sample contains a label for the performed basic behavior and the visual attributes of reference object. This set of samples has to be ordered w.r.t. the type, color and shape label of the reference objects. This results in subsets of samples that have the same object description but different action class labels. For each of these subsets one training sample can be generated for each preprocessing HLAM. As input of such a training sample the objects description unique in the subset has to be taken. The output is either true or false depending on whether the subset does or does not contain a sample with an action class label that corresponds to the basic behavior specific HLAMs. For further processing the outputs of the different HLAMs for a certain object are combined in a vector. This vector will have the same values for all types of blocks but different ones for holes. Just because, in contrast to holes, blocks can be the reference object for the approaching, aligning and grabbing but not for the inserting behavior. Such a vector is an object description that allows to group objects w.r.t. their utility in the shape sorter puzzle. HLAM for Learning the Shape Sorter Transfer Function One HLAM is trained to realize the needed transfer function to let the robot play the puzzle. The network’s input comprises the supervising information (i.e. color and shape label of the desired reference object) and the last game situation. Latter is the label of the last performed basic behavior and the output of the preprocessing HLAM computed for the visual description of the last reference object. The output of the HLAM is the basic behavior next to be performed and the color and shape specification of the reference object. Again, the training sequences of command sample are the base to produce training samples for this HLAM. In contrast to the training of the preprocessing HLAM, the order of the command samples is important. Two successive command samples are used to generate one training sample for the transfer function HLAM. The first sample delivers the needed information about the last game situation. Whereas the second sample defines the target output since it represents what has to be done after the system has reached a certain situation. For all samples that belong to one grab-insert sequence the same supervising information is taken. It is the color and shape label of the block that was grabbed during the sequence. Furthermore, an extra training sample is generated for each grab-insert sequence. Its input is the game situation about the finally performed inserting behavior and the supervising information that describes the inserted block. The target output that defines the next command is a code that represents a stop action. It explicitly marks the end of a grab-insert sequence. The realized shape sorter puzzle player explained in the next subsection uses this stop action to divide a course of game where a number of blocks are inserted. This division allows that a user can let the system handle single blocks he or she wants to get inserted. Without this stop action the number of commands for one sequence must have been hard-wired by the system designer. Furthermore the number of

16 Learning of Hierarchical Perception-Action Cycles

345

commands would have to be constant for all sequences, whereas the explicitly marked end of a sequence enables the system to learn different types of command sequences. This is not needed in the actually solved shape sorted puzzle but could be helpful. E.g. a special type of block could require another behavior that rotates the block before it is grabbed properly. Without changing the code for the system, this additional behavior can be inserted into the command sequence and it will be learned by the transfer function HLAM. 16.3.3.2 Automatic Shape Sorter Puzzle Player By a repeated invocation of the trained transfer function, the system can grab and insert a block into the matching hole. To do so, a loop is programmed that has a similar structure as the visual servoing algorithm applied in the pilot submodule: at the beginning of each cycle, the situation the system is in is determined, given this information the next command is computed with the transfer function, the command is executed, and then the loop can start again. In the following the loop is explained in more details. The information about the situation in which the system is has to comprise the last performed action, the description of the last reference object and the supervising information. The first two parts are simply memorized from the last cycle of the loop. The supervising information is determined once before the loop starts as it remains constantly valid throughout the full grab-insert sequence. Three options are implemented how the color and the shape label of the designated reference object is determined. Either the human user selects both of them from a list of available colors and shapes. Or he or she defines only one of these two categories so that the system decides about the missing one. Or finally, the system fully automatically selects both color and shape features. The first option is implemented in a straightforward manner with a simple console where the user types in the wanted color and shape specifications. If he or she leaves one feature category unspecified, the system completes the supervising information in accordance to the blocks distributed on the working area. To do so, the observer submodule generates a list of all visible blocks, this list is compared with the shape or color label specified by the user, and one block with the correct feature is selected randomly to supply the missing label. By this means it is possible that the user can only type in red and all blocks that are red but have an arbitrary shape are sorted into the matching hole. The system works in a similar manner when it should play the puzzle fully automatically. For one grab-insert sequence it chooses randomly a pair of shape and color labels from one block of the list of available objects. The command next to be executed is computed by the HLAMs explained in the last subsection. First, the HLAMs for preprocessing are used to transform the information about the game situation (i.e. the last used reference object) into the binary vector that describes which behavior can be performed for this type of object. This vector, the memorized last action and the supervising information serves as input for the transfer function HLAM. The HLAM’s output is the next command which is then executed. To do so, the computed shape and color label of the reference object is matched with the list of objects recognized by the observer. This is done in order to determine the system internal identifier that specifies the reference object when the navigator submodule performs the commanded basic behavior.

346

F. Hoppe and G. Sommer

Such a sequence of assessing the game situation, computing the next command, and executing it is repeated in a loop as long as the output of the transfer function HLAM is not the stop action. After this loop came to an end (i.e. after a block was grabbed and inserted properly), the supervising information can be changed and the loop can be started again. The result will be that all blocks are cleared off the working area and the puzzle is solved.

16.4 Experiments 16.4.1

Validation of the Adaptive Visual Servoing Scheme

The quality of the visual servoing scheme presented in Sec. 16.3.2 is assessed by an experiment where the main question of interest is to see how successful the online refinement of the transfer function works. To do so an experiment was performed where the end-effector is steered by the pilot submodule to different test positions and the resulting trajectory is judged w.r.t. the number of needed servoing steps and its straightness. The HLAM specified in Sec. 16.3.2.2 is bootstrapped with eight training samples. To do so, the end-effector is driven over a 3 × 3 grid in a plane parallel to the working area. The grid’s step size is 5 cm. At each vertex of the grid the position of the end-effector is recorded in the coordinate system of the robot and the static camera image. With this data, eight training samples for the HLAM are generated which correspond to the eight possible movements from the center position of the grid to the border positions. These movements are depicted in Panel (a) of Fig. 16.4. These few training sample are used to bootstrap an HLAM with the learning algorithm proposed in [17]. To prepare the validation of the trained HLAM the robot is driven manually to four test positions. These serve as target positions of the test trajectory. For each corner the position of the end-effector is recorded in the coordinate system of the robot and the static camera image. The former allow a comparison with the positions actually reached by the robot during testing, while the latter are needed to specify where the pilot should drive to. Given the image coordinates of the four test positions, the robot can be steered by the visual servoing scheme along the test trajectory. It is a loop that connects every corner. The thresholds ε and T of the servoing algorithm are selected so that the robot is driven to the target position with the highest possible accuracy. To do so, ε is set to zero which means that no misplacement of the end-effector from its target position is permitted. The maximal number of servoing steps T is left unrestricted. With these settings the robot is driven five times along the test trajectory. In all trials all positions were reached. The trials differed only w.r.t. the number of needed visual servoing steps per target position and the straightness of the trajectory. In Panel (b) of Fig, 16.4 it is visible that in the first trial the robot is not steered along the ideal way on straight lines to the target positions. After the fifth trial the realized trajectory plotted in Panel (c) is improved. The effect that the online refinement increases the quality of the HLAM can also be demonstrated with the number of needed servoing steps. The mean number drops from 13.4 to 5.6. It shows that the HLAM learns to steer the robot faster along the test trajectory.

16 Learning of Hierarchical Perception-Action Cycles

(a) Movements for bootstraping

(b) Trial 1 with bootstrapping setup

(c) Trial 5 with bootstrapping setup

(d) Trial 1 with changed setup

(e) Trial 2 with changed setup

(f) Trial 5 with changed setup

347

Fig. 16.4. Effects of refining the approaching HLAM of the pilot submodule. Panel (a) shows the static view with the eight training samples for bootstrapping the HLAM. In Panels (b) and (c) the trajectory is plotted how the robot drives to four test positions (red crosses) in the bootstrapping setup. The servoing results after the static camera is shifted and tilted are shown in Panels (d)-(f).

348

F. Hoppe and G. Sommer

To test the robustness of the proposed visual servoing algorithm, the static camera is displaced from its original position and five more trials are performed with the bootstrapped HLAM. The static camera is shifted sideways by approximately 40 cm, lifted by 10 cm and tilted by 10 degree. As preparation for the actual trials the robot is steered manually to the four test positions. These are the same physical positions as in the first five trials. But their new image coordinates are needed for the pilot. The trajectory of the first trial is plotted in Panel (d) of Fig. 16.4. It shows that the robot clearly deviates from the ideal path. Still, all target positions are reached with a speed that is comparable to the speed with the original camera position. In Panel (e) and (f) the results of the second and the final trial are plotted. It demonstrates that the performance of the HLAM improves rapidly. With the fifth trial the same quality is realized as for the camera setting that is used to produce the bootstrapping training set. This proves that the HLAM approach is suited for fast online learning. Since the test positions are also recorded in the coordinate system of the robot, the precision of the system can be quantified. The mean displacement over all ten trials is 2.1 mm. The maximal and minimal error is 2.2 mm and 1 mm, respectively. These errors are totally depending on the object recognition methods since the visual servoing loop does not stop before the target position is accurately reached (due to setting ε to zero). A closer look at the data reveals that the misplacement of the robot is higher at target positions that are more distant from the camera. This matches well the conception that precision is mainly a matter of camera resolution. 16.4.2

Validation of the Shape Sorter Puzzle Player

The HLAMs proposed to learn the game play of the shape sorter puzzle (cf. Sec. 16.3.3.2) are tested w.r.t. their ability to handle new types of objects properly. The question is whether the networks can generalize from examples where blocks with some colors and shapes were inserted to blocks that can have arbitrary shapes and colors. The subsequent question is how this competence is achieved. 16.4.2.1 Validation of the HLAMs For the experiment a set of training and test data is generated manually. This data is a sequence of commands and supervising information that shows how the puzzle game is properly played with the four basic behaviors realized by the PA module. The command sequences are generated for all blocks available in the physical world. With five different colors and four different shapes, 140 ordered command samples with corresponding supervising information are synthesized. Only a small portion (i.e. 21) of them are used as training samples. These are listed in Fig. 16.5. The training samples demonstrate how the game is played with three different blocks. They show that for all blocks a fixed sequence of seven commands (i.e. approach; align; grab; approach; align; insert; stop;) has to be issued. Furthermore the general pattern for the color and shape attributes of the reference object becomes obvious. E.g. for the first six commands the shape label is equal to the corresponding supervising information. Or the attributes of the reference object for the stop command are always set to the label none. This set of samples is selected because

16 Learning of Hierarchical Perception-Action Cycles 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21:

Approach Align to Grab Approach Align to Insert Stop Approach Align to Grab Approach Align to Insert Stop Approach Align to Grab Approach Align to Insert Stop

red red red black black black none blue blue blue black black black none blue blue blue black black black none

bridge bridge bridge bridge bridge bridge none bridge bridge bridge bridge bridge bridge none half-cylinder half-cylinder half-cylinder half-cylinder half-cylinder half-cylinder none

(supervisor: (supervisor: (supervisor: (supervisor: (supervisor: (supervisor: (supervisor: (supervisor: (supervisor: (supervisor: (supervisor: (supervisor: (supervisor: (supervisor: (supervisor: (supervisor: (supervisor: (supervisor: (supervisor: (supervisor: (supervisor:

red red red red red red red blue blue blue blue blue blue blue blue blue blue blue blue blue blue

349

bridge) bridge) bridge) bridge) bridge) bridge) bridge) bridge) bridge) bridge) bridge) bridge) bridge) bridge) half-cylinder) half-cylinder) half-cylinder) half-cylinder) half-cylinder) half-cylinder) half-cylinder)

Fig. 16.5. Training set for the puzzle player. For the HLAMs the command, color and shape labels are internally represented as integers.

it contains enough information to allow a generalization to arbitrarily blocks. It demonstrates for the learning algorithm that different color and shape labels are possible and that these are specifically related to the supervising information. With this training set, five HLAMs for the preprocessing step and the HLAM for the transfer function are trained. The five HLAMs for the preprocessing step correspond to the five possible commands: approach, align to, grab, insert, and stop. The latter is added to the four basic behaviors to explicitly mark the end of a grabinsert sequence. The corresponding HLAM is trained by the same means as those for the basic behaviors. It learns to separate any real world object – block or hole – from a pseudo object which color, shape and type label is none because this labeling occurs only together with the stop command. The remaining 119 command samples are used to test the performance of the trained networks. They show a 100 % correct reproduction of command sequences to handle blocks with any combination of color and shape. This guarantees that the automatic puzzle player will invoke the right basic behaviors for whatever block the human user selects the supervising information. 16.4.2.2 Analysis of the HLAMs To understand how the automatic shape sorter works its HLAMs are examined. This analysis relies on features of the HLAM approach which to explain is beyond the scope of this chapter. So, the following contains only a rather superficial explanation. For more details see [17]. The HLAM that is trained for the shaper sorter transfer function contains seven socalled local models. These are called local because they are only valid in a local region of the input space of the approximated function. The input space of the HLAM for the transfer function escentially reflects the information about the last game situation (cf. page 344). This space has six dimensions: one dimension corresponds to the last

350

F. Hoppe and G. Sommer

Fig. 16.6. Finite state automaton learned by the transfer function HLAM. Each circle corresponds to a local model. The vertical axis reflects three classes of objects. The edge annotation specifies the color and shape attributes of the reference object where “s” means that the supervising information is taken. The local model LM6 is the start and end state.

performed action, while the remaining are the output of the five preprocessing HLAMs. The seven local models are plotted in the 2D diagram shown in Fig. 16.6. One dimension corresponds to the invoked behavior. The other one represents a projection of the 5D space of the preprocessing HLAMs onto a 1D subspace. Later can be done because the five preprocessing HLAMs learn to separate holes, blocks and the pseudo object from each other.3 In Fig. 16.6 the resulting dimension is labelled with “Color, Shape” as these two labels are essentially the inputs to the preprocessing HLAMs. According to this, the block class is labelled with “any but black, any”, the holes with “black, any” because the only difference between the two classes is that holes are black. The pseudo object is labelled with “none, none”. The seven local models plotted in Fig. 16.6 are augmented by arrows that represent their output. A simple example is the local model labeled LM7 that is selected to compute the output of the HLAM whenever the last performed behavior was an insertion and the last reference object was a hole. This local model has a fix output defining that the stop action has to be performed next and that the shape and color label of the new reference object are both none. Hence, an arrow with the annotation “none, none” is drawn from LM7 to LM6 since the position of LM6 in the input space of the transfer function represents the command “stop none, none”. So, the whole diagram can be interpreted as a finite state automaton that is learned by the HLAM. The drawn local models represent states, while the arrows define the state transitions. This automaton is realized by the loop described in Fig. 16.3.3.2 that repeatedly applies the trained HLAM to compute the next command. The automaton 3

The proof for this is omitted due to lack of space.

16 Learning of Hierarchical Perception-Action Cycles

351

will traverse through its possible states until the stop command has to be executed and the loop ends. As the local model LM6 corresponds to this stop command it is marked as the finite state of the automaton. With this explanations the working principle and the correctness of the automatic shape sorter puzzle player becomes obvious. A full grab-insert sequence is performed along the path from LM6 through LM4, LM5, LM3, LM2, LM1, LM7, and finally back to LM6 again. With the first three commands a block is approached, the gripper is aligned to it and it is grasped. The attributes of the reference object for these commands are defined by the supervising information until the second approach command is computed by LM3. At this stage of the puzzle play the color label is selected to be black, while the shape label remains to be defined by the supervising information. By transversing from LM3 over LM2 and LM1 to LM7 the puzzle player invokes the approaching, aligning and inserting behavior for a hole with the same shape attribute as the grabbed block. With reaching LM6 again, the stop action is computed and the grab-insert sequence is finally completed.

16.5 Summary In this chapter concepts for a new software architecture for artificial cognitive systems were presented. Such systems should exhibit robust behaviors in changing environments. It is argued that this requires abilities to learn at all levels of the software architecture and at different time scales. The suggested software architecture comprises three different modules that handle data of different degree of abstraction and realize nested perception-action cycles. At the lowest level continuous data is processed in fast running cycles that allow reactive behavior of the system. At the other extreme a discrete encoding scheme is used to pursue long term goals. The important contribution is the emphasis to apply machine learning techniques at all levels of the architecture. This general scheme is exemplified with a concrete robot task of a shape sorter puzzle player. The implementations of two submodules - one from the lowest, one from the highest system level - were described. Both showed that the utilization of the HLAM approach which is a supervised function approximation technique eases the development of the proposed robot system. It grounds the proposed adaptive visual servoing scheme which can robustly cope with changes of the camera setup and is easy to extend to different kinds of movements. Equally extendable, the HLAM approach can be used to learn long term goals given in a discrete coding scheme. As shown with an experiment, the possibility to analyze this type of network makes it possible to proof the correctness of the gained solution. The basic idea for this success of the machine learning methodology is the concept of trainable transfer function. As in common approaches, such transfer functions map the perceptions of a system to its next actions. But instead of the explicit modeling of these functions they are learned by machine learning techniques. To do so, examples how the system should act under certain conditions have to be collected and used as a training set. That this strategy is feasible is shown with our system.

352

F. Hoppe and G. Sommer

Acknowledgments The work presented here was supported by the the European Union, grant COSPAL (IST-2003-004176). However, this paper does not necessarily represent the opinion of the European Community, and the European Community is not responsible for any use which may be made of its contents.

References 1. Amari, S.A.: Dynamics of pattern formation in lateral-inhibition type neural fields. Biological Cybernetics 27, 77–87 (1977) 2. Bonasso, R.P.: Integrating reaction plans and layered competences through synchronous control. In: Proc. of IJCAI, pp. 1225–1233 (1991) 3. Brooks, R.A.: Intelligence without reason. In: Proc. of IJCAI, pp. 569–595 (1991) 4. Brooks, R.A.: A robust layered control system for a mobile robot. Technical Report AIM864, AI Lab, MIT (September 1985) 5. Brooks, R.A.: Elephants don’t play chess. In: Maes, P. (ed.) Reasoning about Actions and Plans, pp. 3–16. Elsevier, Amsterdam (1990) 6. Chatila, R.: Deliberation and reactivity in autonomous mobile robots. Robotics and Autonomous Systems 16(2-4), 197–211 (1995) 7. Chella, A., Dindo, H., Infantino, I.: A cognitive framework for imitation learning. Robotics and Autonomous Systems, Special Issue 54(5), 403–408 (2006) 8. Connell, J.H.: SSS: A hybrid architecture applied to robot navigation. In: Proc. of IEEE Int. Conf. on Robotics and Automation, Nice, France (May 1992) 9. Erdmann, M.: Understanding action and sensing by designing action-based sensors. I. J. Robotic Res. 14(5), 483–509 (1995) 10. Erlhagen, W., Mukovskiy, A., Bicho, E., Panin, G., Kiss, C., Knoll, A., van Schie, H., Bekkering, H.: Goal-directed imitation for robots: A bio-inspired approach to action understanding and skill learning. Robotics and Autonomous Systems, Special Issue 54(5), 353–360 (2006) 11. Espiau, B., Chaumette, F., Rives, P.: A new approach to visual servoing in robotics. IEEE Trans. Robotics and Automation 8(3), 313–326 (1994) 12. Gat, E.: Reliable Goal-directed Reactive Control for Real-World Autonomous Mobile Robots. PhD thesis, Virginia Polytechnic and State University, Blacksburg, USA (1991) 13. Gat, E.: On three-layer architectures. In: Bonnasso, R.P., Kortenkamp, D., Murphy, R. (eds.) Artificial Intelligence and Mobile Robots. MIT Press, Cambridge (1997) 14. Hill, J., Park, W.T.: Real time control of a robot with a mobile camera. In: Proc. 9th ISIR, pp. 233–246 (March 1979) 15. Hoppe, F., Sommer, G.: Local linear models for system control. In: Proc. of Int. Conf. on Neural Information Processing (ICONIP), pp. 171–176 (2005) 16. Hoppe, F., Sommer, G.: Ensemble learning for hierarchies of locally arranged models. In: Proc. of IEEE World Congress on Computational Intelligence, pp. 10612–10619 (2006) 17. Hoppe, F.: Local Learning for Visual Robotic Systems. PhD thesis, Christian-AlbrechtsUniversit¨at zu Kiel, Institut f¨ur Informatik (2007) 18. Hutchinson, S.A., Hager, G.D., Corke, P.I.: A tutorial on visual servo control. IEEE Trans. Robotics and Automation 12(5), 651–670 (1996) 19. Kragic, D., Christensen, H.: Survey on visual servoing for manipulation. Technical Report CVAP259 (January 2002) 20. Nilsson, N.J.: Principles of Artificial Intelligence. Tioga (1980)

16 Learning of Hierarchical Perception-Action Cycles

353

21. Pauli, J. (ed.): Learning-Based Robot Vision. LNCS, vol. 2048, p. 171. Springer, Heidelberg (2001) 22. Sommer, G.: Algebraic aspects of designing behavior based systems. In: Sommer, G., Koenderink, J.J. (eds.) AFPAC 1997. LNCS, vol. 1315, pp. 1–28. Springer, Heidelberg (1997) 23. Steil, J.J., R¨othling, F., Haschke, R., Ritter, H.: Situated robot learning for multi-modal instruction and imitation of grasping. Robotics and Autonomous Systems 47(2-3), 129–141 (2004) 24. Thrun, S., et al.: Stanley: The robot that won the DARPA grand challenge: Research articles. J. Robot. Syst. 23(9), 661–692 (2006) 25. Tsotsos, J.K.: On behaviorist intelligence and the scaling problem. Artifical Intelligence 75(2), 135–160 (1995)

17 Advancing Artificial Intelligence through Minimalist Humanoid Robotics Jacky Baltes and John Anderson Autonomous Agents Laboratory, Department of Computer Science, University of Manitoba, Winnipeg, Manitoba, Canada R3T2N2 [email protected]

Abstract. While the robots that most quickly come to mind to the general public are those with the most elaborate features and movements, those that are most useful in advancing the state of the art in artificial intelligence (AI) are very different. Minimalist robots are inexpensive and therefore more broadly available for research and educational purposes, but also force the researcher to rely on good, adaptable solutions to hard AI problems rather than relying on expensive specialized hardware that will only work under strict conditions. This chapter describes our work in minimalist humanoid robots, focussing mainly on Tao-Pie-Pie, a robot that competed successfully in numerous RoboCup and FIRA competitions. The chapter describes our motivations in designing minimalist robots and our rationale for working with humanoid robots, and describes the development of Tao-Pie-Pie, including contrasting this robot with other work and developing its walking gait and balancing reflexes. We then describe some issues in evaluating humanoid robots, and describe ongoing work.

17.1 Introduction The robotics examples that best capture the imagination of the general public are generally those that are the most elaborate physically - those that can accomplish the most elaborate movements and appear the most human, even if the movements are preprogrammed and non-adaptable. In contrast to this, we believe that the best examples of robots from the standpoint of advancing artificial intelligence are much more minimal in nature: in our work, we always attempt to design a robot with the most minimal features necessary to accomplish the task at hand. There are a number of important reasons for this. One is affordability, and the practical benefits that working with affordable research platforms brings to a laboratory. More broadly affordable robots means being able to support research more broadly, such as being able to explore issues in teamwork as opposed to being limited to individuals. It also means being able to equip more students and better support education and research in general. A second reason is practicality: even in cases where robots have a great expensive additions, such as gyroscopes, accelerometers, and other forms of sensing, these are in many cases not used in robotics competitions, either because particular devices are not crucial for the task at hand, or because of the complexities of integrating numerous senses and control [7]. In cases such as this, the additional features of the robot serve no purpose. D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 355–376. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

356

J. Baltes and J. Anderson

A much larger but less obvious advantage, however, is the fact that a minimalist approach serves to force a grounding on researchers, requiring them to provide more generally intelligent and adaptable approaches to any given problem. This is because the difficult parts of any problem cannot be overcome simply by using specialized hardware. For example, a robot whose sole goal is to play soccer on an even surface can use specialized rubber feet and mechanics to improve its performance. While such a robot might be an excellent soccer player, it would not be able to walk at all over uneven terrain or a gravel road. On the other hand, a researcher that does not have access to this specialized hardware must instead provide better solutions to basic problems such as localization and balancing, and these solutions are applicable to both the soccer problem and a broad array of problems that a solution requiring specialized hardware would not be. More general solutions require additional effort on the part of the researcher, but serve to greatly advance the state of artificial intelligence (AI), since we rely more on AI than specialized hardware. In the same fashion, inexpensive equipment generally does not guarantee standards of precision. Even using similar mechanics and control, a system that uses cheap motors would still need to adapt to situations such as the right leg motor being stronger than the left leg motor. However, because of its inherent need to adapt to variations, such a system is much more robust when deployed in a real world environment. Similarly, a robot whose walking gait is not precise due to hardware limitations forces the designer to rely on better balancing and better localization techniques, for example. All of these are the kinds of solutions that we see humans adapting to problems: not optimal, but sufficiently well in a wide range of situations. The human form itself is an example of this: a humanoid robot cannot traverse flat ground faster or with less effort than a wheeled robot, for example. In fact, any individual activity in a specific setting can be performed faster or better using a less general form than a humanoid. Even in the biological world, other species can outperform humans in any area (e.g., horses can run faster, dolphins swim better, and monkeys climb better than humans), but humans are among the top five performers in all categories. The form itself has evolved not to do any one task well, but to perform adequately and adaptably to a huge range of tasks. In addition to having similar benefits to minimalist robotics generally, a focus on the humanoid form brings about important side benefits. Robots of any kind will have to function in a world that is designed for humans, and while the humanoid form is not optimal, it does not require specialized environmental adaptations that wheeled robots require. In addition to being able to interact with the wide variety of equipment designed for humans, humans can also relate more easily to a robot in a human form: we anthropomorphise devices that look more like us, and humans already have a set of expectations that fit the humanoid form, allowing these to be exported more easily to a machine. We believe that a focus on the humanoid form combined with that of minimizing specialized hardware will ultimately be a significant advantage in improving research in artificial intelligence. . In recent years we have been working with humanoid robots of varying complexity, attempting to focus on the minimum necessary hardware to support various forms of intelligent interaction with the world.

17 Advancing Artificial Intelligence through Minimalist Humanoid Robotics

357

This chapter focuses in detail on our experiences designing and working with one minimalist humanoid robot: Tao-Pie-Pie, which competed in a number of RoboCup and FIRA HuroCup competitions, and became well-known for both its small size and the degree of functionality it displayed despite its minimalist nature. The first part of the chapter discusses the methodology used to create stable walking, turning, and kicking motions for Tao-Pie-Pie. The second part of the chapter describes the addition of balancing reflexes to allow Tao-Pie-Pie to traverse uneven terrain. The last part describes how the lessons learned on Tao-Pie-Pie were used in the implementation of Blitz, a much more complex humanoid robot designed to participate in the new HuroCup challenge. In addition, this chapter also discusses issues in evaluating humanoid robots in a competitive setting.

17.2 Tao-Pie-Pie This section describes the design and implementation of a stable walking gait, a turning motion, and a kicking motion for Tao-Pie-Pie, a minimalist humanoid robot developed in our laboratory. We begin by comparing the design goals for Tao-Pie-Pie with those of other well-known humanoids. Recent advances in material science, control engineering, robotics, and Artificial Intelligence has enabled researchers to build fully autonomous humanoid robots that can walk, dance, climb stairs, and other functions. For the first time, these robots are not limited to academia and research laboratories, but have been developed as commercial products. Recently, several companies have developed commercial humanoid robotic platforms, for example Honda, Fujitsu, Mitsubishi, and Sony. These designs have many degrees of freedom and are very complex mechanical and electronic systems. Correspondingly, they are very expensive. Despite the existence of these mechanically-advanced robots, many research questions about humanoid robots remain unanswered. Apart from the general problems of localization, computer vision, path planning, motion planning, and task planning, these also include some problems that are specific to the control of humanoid robots. For example, what is the minimum set of actuators needed for stable walking? What sensor information is necessary to walk over uneven terrain? How can we minimize the energy required in walking? Despite visually impressive demonstrations, these advanced models have not yet answered these or many other open questions. Dealing with these many questions is part of the reason why Kitano and Asada presented humanoid robots as a millennium challenge for robotics [4]. Since this time, humanoid robotics competitions have become well known and well covered in the popular media. The two major competitions in particular are also of strong academic interest: 1. The RoboCup Initiative included a humanoid challenge for the first time in 2002. Humanoid robots competed in three events: walking, penalty kicks, and a free demonstration. 2. The HuroCup competition is part of the FIRA International robotic competition. The competition has become more elaborate each year, and currently consists of a octathlon (robot dash, penalty kick, lift and carry, weightlifting, basketball,

358

J. Baltes and J. Anderson

climbing wall, and obstacle run) and the robot with the best summary score over all events is declared the winner. Tao-Pie-Pie competed in RoboCup events from 2002-2005, and in FIRA HuroCup events from 2002-2005. While Tao-Pie-Pie was then retired, mainly due to its aging framework, it could still compete at a favourable techical level in RoboCup events today. The broad array of events at current FIRA HuroCup competition is beyond the capabilities of Tao-Pie-Pie (in part because some of the events require arms), Tao-PiePie led us to develop later robots that do currently compete in these events, and are equally minimalist for their expanded tasks. One of these later robots, Blitz, will be introduced later in the chapter, and we will also examine issues in evaluating robots, using the RoboCup and FIRA HuroCup events as examples. We begin our overview of Tao-Pie-Pie with a brief summary of related work, both in terms of other similar robotics projects and background on walking gaits. We then describe the design of the robot, the development of its walking gait, and its balancing reflexes.

17.3 Related Work Many research groups worldwide have also developed small humanoid robots for scientific exploration and competition. This section gives an overview over other small humanoid robot designs. This section attempts to give an overview over the different types of designs rather than trying to be exhaustive. 17.3.1

Viki

Viki [6] was developed by Hendrik Lund’s team from the University of Southern Denmark. Like Tao-Pie-Pie, Vicki is a minimalist design approach. It embodies a bottom up approach focusing on the interaction between physical properties and control. Viki uses only five motors. Two motors are used to turn the legs sideways, one motor moves the hip, and one motor moves the upper body. Another motor is used to control the arms of the robot. Viki only uses four motors for the walking motion compared to Tao-Pie-Pie’s six RC servos. However, Viki’s mechanical design is significantly more complex including a gear box and timing belts. It is difficult to compare the kinematic abilities of the two robots. Viki has the ability to turn either leg sideways, but cannot kick a ball straight forward. Tao-Pie-Pie can not turn its hips sideways, but it can kick with the right or left foot. It is also interesting to note that Tao-Pie-Pie and Viki (both based on minimalist design philosophies) were by far the smallest robots in the competitions they entered. Tao-Pie-Pie is 28cm tall, whereas Viki is about 25cm tall. Viki does not include any sensors for balancing and walking. Rather it seems to have been developed primarily for the RoboCup Junior competition, since it has a set of infrared sensors that can detect an infrared emitting ball over long distances.

17 Advancing Artificial Intelligence through Minimalist Humanoid Robotics

17.3.2

359

Pino and Morph

Pino and Morph are a humanoid robots developed by the Kitano Symbiotic Systems group [9]. Pino is approximately 50cm tall. It uses high torque servo motors to control 26 DOF (6 DOF in each leg, 5 DOF in each arm, 2 DOF in the trunk, and 2 DOF in the head). Pino includes a vision sensor, posture sensors, joint angle sensors (using the potentiometers of the servos), and force sensors in the feet. Morph is a next generation humanoid robot. It is similar to Pino’s design, but smaller. 17.3.3

Robo Erectus

Robo Erectus is a series of small humanoid robots developed by Changjiu Zhou at Singapore Polytechnic, Singapore [10]. It is a approximately 40cm tall and has 12 DOFs. It uses standard RC servos as actuators. Although its walking gait is not as stable as that of other robots, it can move at an amazing speed of approximately 10cm/sec. Robo Erectus was the fastest of the small sized robots in the RoboCup competition achieving a second place finish in the robot walk even against taller robots. 17.3.4

KAIST Robot

Jung-Hoon Kim from KAIST, Daejon, Korea developed a 50cm tall robot which uses DC motors and timing belts. The robot performed superbly during the 2002 HuroCup competition and won first prize. [3]. 17.3.5

Johnny Walker

Johnny Walker is a 40cm tall humanoid robot developed by Thomas Br¨aunl from the University of Western Australia [1]. It also uses the Eyebot controller. It has four DOFs in each leg. Feedback is provided by a vision cameras as well as two accelerometers, two gyroscopes, and force sensors in the feet. 17.3.6

Development of Walking Gaits

There is a rich literature on the design of walking gaits for specific humanoid robots (see for example [8, 5]). However, much of this work was done in simulation only, and those using real robots succeed only by constraining the range of motion of the robot (e.g., through the use of oversize feet, or by restricting the motion in the sagittal plane). The representation of the walking gait is also important. A smooth control curve is needed, since otherwise the motion of the robot will be too jerky. Jerking introduces unwanted forces into the motion of the robot, is not energy efficient, and introduces even more noise into the output of gyroscopes and accelerometers. Ijspeert describes the walking gaits with a set of differential equations [2]. The differential equations describe a set of attractors using Gaussian kernel functions. A walking gait is represented as a set of weights on the kernel functions, and thus an attractor landscape.

360

J. Baltes and J. Anderson

17.4 Design Requirements Tao-Pie-Pie was intended as a research vehicle to investigate methods for deriving control methods for stable walking patterns for humanoid robots. Stable walking, especially over uneven terrain is a difficult problem. One problem is that current actuator technology (RC Servos, DC motors) generate less torque in comparison to their weight than human muscle. Another problem is that feedback from gyroscopes and actuators is very noisy. The necessary smoothing of the input signals makes it hard to use it in actively controlling the walking motion. Another research direction was the investigation of computer vision based methods for balancing and walking. Humans use vision when balancing. This can be demonstrated convincingly by having a person balance on one leg and then ask them to close their eyes, which makes balancing much harder for people. The idea is to use the optical flow field of the camera to control the robot’s walking motion. This requires fundamental scene interpretation since to extract a motion vector (in general, a six dimensional vector representing translation in the X, Y, and Z plane as well as pan, tilt and swing angles) from an image sequence requires knowledge of the geometry of the scene. This problem can be simplified by making assumptions about the world and limiting the amount of Furthermore, Tao-Pie-Pie was intended to compete at international humanoid robotic competitions such as RoboCup and HuroCup. Among other, this meant that Tao-Pie-Pie had to be able to balance, walk, run an obstacle course, and kick a ball. Cost was an important design criteria in Tao-Pie-Pie’s development. Previous experience has shown us that the use of commonly available cheap components does not only help to keep the cost of a project down and the Head of Department happy, but it also has lead to the development of novel, versatile, and robust approaches to problems in robotics. Another design goal was to reduce the number of degrees of freedom (DOF) of the robot. This reduces the cost of the humanoid robot as well as increases its robustness. Each DOF adds extra complexity in the mechanical design and the design of the control electronics. Furthermore, reducing the number of DOFs in the robot allows us to exploit the dimensions of the humanoid walking problem. The minimum set of DOFs that allow a humanoid robot to walk is of interest since it leads to energy efficient designs.

17.5 Mechanical Design of Tao-Pie-Pie Tao-Pie-Pie is the third generation of humanoid robots developed in our lab. The first two humanoid robots, RX-78 and Zaku, were both based on commercially available toy model figures (See Fig. 17.1). Both model provided important stepping stones and insights into the design of a small humanoid robot. A number of lessons were learned from RX-78 and Zaku and included in the mechanical design of Tao-Pie-Pie. First, the plastic model kits were not strong enough to withstand the pressures and forces during walking and thus would often break. Therefore, Tao-Pie-Pie was constructed from aluminum.

17 Advancing Artificial Intelligence through Minimalist Humanoid Robotics

361

Fig. 17.1. Front views of RX-78 (left) and Zaku (right)

Fig. 17.2. Front and side view of Tao-Pie-Pie

Secondly, Zaku proved the need for a joint in the left-right (so-called sagittal) plane to shift the center of gravity from right to left. Although Zaku was able to shuffle in a straight line, the absence of a joint in the sagittal plane meant that it could not lift its feet of the ground and thus had a very short stride length. The only way for Zaku to shift the center of gravity from left to right was by bending its knees. Tao-Pie-Pie therefore had an extra DOF in each ankle which allows it to shift in the sagittal plane. Thirdly, the importance of feedback in the control of humanoid walking was demonstrated by RX-78 which used a modified mouse as make-shift pitch and roll sensor.

362

J. Baltes and J. Anderson

Tao-Pie-Pie includes two gyroscopes to measure the angular velocity in the left-right and forward-backward plane. Figure 17.2 shows the mechanical construction of Tao-Pie-Pie. The actuators and sensors consist of widely available RC servos and RC gyroscopes for remote controlled cars and helicopters. The Eyebot controller ([1]) was chosen as embedded processor, since it is relatively inexpensive, yet powerful enough to provide vision information. A small CMOS camera provides visual feedback for the robot. The details of the mechanical construction of Tao-Pie-Pie are shown in Fig. 17.3. The mechanical design was done in conjunction with Nadir Ould Kheddal’s robotics group at Temasek Politechnic, Singapore. Tao-Pie-Pie is constructed out of 0.5mm aluminum. The RC servos are used as structural components in the design.

Fig. 17.3. CAD Drawing of Tao-Pie-Pie

17.6 Development of Walking Gaits One of the fundamental problems in humanoid robots is the development of stable walking patterns. A walking pattern is dynamically stable if the COP is within the supporting area. A statically stable walking pattern also has the COM within the supporting area. We used a divide and conquer approach and divided the walking gait into six phases: three for the right and three for the left leg. The phases were selected in such a way that the robot is statically stable at the end of each phase. 17.6.1

Inverse Kinematics

Inverse Kinematics allow us to compute the correct joint angle to position a robotic link at a target position. After determining the desired location for the COM, we compute

17 Advancing Artificial Intelligence through Minimalist Humanoid Robotics

363

Fig. 17.4. Inverse Kinematics model of the robot. P, represent center of mass. L2 - hip joint link, L1 - knee joint link.

joint angles for all RC servos in the leg. By controlling the joints angles, we can control the stability during the motions. To keep the humanoid balanced, we must keep the center of mass within the supporting region during the termination of all phases of the walking motion. To solve the inverse kinematics model of the humanoid robot we use a simplified two link model of the robot. Solving the equations for θ1 first and then for θ2 , we can find a solution to the inverse kinematics problem. The solution for θ1 and θ2 are shown below. cos(x2 + y2 − L21 − L22 ) 2L1 L2 −(L2 sin (θ2 )x + (L1 + L2 cos(θ2 )y) θ1 = (L2 sin (θ2 )y + (L1 + L2 cos (θ2 )x)

θ2 =

17.6.2

(17.1)

Bezier Curves

The walking gait is based on a cyclical pattern generator. The walking patterns described in the next subsections are implemented using a set of control points. There are many possible methods for interpolating points between the user defined control points. During our work on RX-78 and Zaku, we used linear interpolation, similar to Br¨aunl. However, using linear interpolation the resulting curve is in general not differentiable around the control points. This leads to jerky motion of the robot, which introduces destabilizing forces into the motion. A smooth motion is much preferred since jerky motions are usually not energy efficient and more importantly aggravate noise in the sensors. Instead, we used Bezier curves to interpolate setting between the control points. The advantage of Bezier curves are that they are simple cubic equations which can be computed quickly. Bezier curves are also versatile and generate smooth and natural gaits.

364

J. Baltes and J. Anderson

Another possibility is the use of cubic splines. Cubic splines seem especially suited to this problem since they guarantee that the resulting curve minimizes the second derivative (and therefore the acceleration in the robot). This greatly reduces unintended forces. 17.6.3

Pattern for Straight Walk

The walking pattern for a straight walk is shown in Fig. 17.5. The pattern is based on the control curves shown in the previous section. The walking pattern consists of six phases. The walking pattern repeats itself after the sixth phase. The bottom row of images shows the approximate position of the COM. Tao-Pie-Pie starts in phase 1 — “Two Leg Stand” — where the right leg is in front and the left leg is behind. Both legs are on the ground and the COM is between the two legs. From phase 1, Tao-Pie-Pie moves to phase 2 — “One Leg Stand” —. In this phase, the ankle servo generates a torque which moves the COM to the inside edge of the right leg. This also results in the back (left) leg to lift off the ground. During the transition from phase 2 to phase 3 — “Ready for Landing” — is in static balance. Tao-Pie-Pie moves the free left leg forward and positions it so that it is ready for land. The COM moves to the front of the supporting leg. This stabilizes the transition to phase 4. During the transition from phase 3 to phase 4 — “Two Leg Stand Inverse” — the robot is in dynamic balance. The supporting leg extends its knee joint to shift the COM over the front edge of the supporting leg. The ankle servo of the supporting leg generates a torque to move the COM over the right side. The left leg will touch the ground in front of the right leg.

Fig. 17.5. Walking Pattern of Tao-Pie-Pie

17 Advancing Artificial Intelligence through Minimalist Humanoid Robotics

365

Fig. 17.6. Turning Pattern of Tao-Pie-Pie

Phases 5 to 6 are the mirror images of phases 2 to 3. After phase 6, the motion continues with a transition to phase 1. 17.6.4

Turning Patterns

Tao-Pie-Pie possesses two different patterns for changing the direction of its walk from a straight line walk: (a) varying the stride length, and (b) lower body twist. By changing the speed at which Tao-Pie-Pie moves through phases 2 and 5 respectively, it can vary the stride length of the left and right part of the walking pattern which turns the robot into this direction. However, the turning rate is slow. Using this method it takes about 20 steps to turn by 45 degrees. Instead Tao-Pie-Pie can turn on the spot by twisting its lower body, which is shown in Fig. 17.6. The turn occurs in phase “turn.” In this phase, the front and back legs will swap position. During the turn, the COM and the COP are in the center between the two feet. 17.6.5

Kicking Pattern

The RoboCup competition also required our robots to kick a ball. Therefore, we developed a kicking pattern for Tao-Pie-Pie. The kicking pattern shown in Fig. 17.7 is similar to the walking pattern.

366

J. Baltes and J. Anderson

Fig. 17.7. Kicking Pattern of Tao-Pie-Pie

The difference is that in phase 2, the rear leg is moved back as far as possible. This increases the range of motion of the kick, which results in more energy for the kick. To keep the robot balanced, Tao-Pie-Pie leans the upper body forward by moving both hip joints. This is necessary to keep the COM over the supporting area of the front leg. Tao-Pie-Pie then snaps the back leg forward as quickly as possible. At the same time, it straightens out the upper body, which readies it for landing on the kicking foot.

17.7 Balancing Reflexes The only feedback about the motion of Tao-Pie-Pie is provided by two gyroscopes that provide information about the angular velocity in the left-right (referred to as the Yplane in the remainder of this chapter) and forward-backward plane (referred to as the X-plane in the remainder of this chapter) respectively.

17 Advancing Artificial Intelligence through Minimalist Humanoid Robotics

367

Fig. 17.8. Gyroscope Readings in the X and Y Plane over 10 Steps. Linear Approximation of the Safe Zone.

368

J. Baltes and J. Anderson

The raw sensor data of the gyroscopes is very noisy. We therefore compute a running average over five samples to smooth out the noise. Fig. 17.8 shows the gyroscope readings for the X and Y plane over approximately twenty steps. Since Tao-Pie-Pie did not fall over during this extended walking trial, these gyroscope readings were used to determine a “safe zone” for the velocity feedback of the gyroscopes. We then created a linear approximation of the “safe zone envelope” and generated minimum and maximum thresholds for the gyroscope readings. The approximation is shown using red and blue lines in Fig. 17.8. 17.7.1

Sensor Feedback in Detecting a Fall

Initially, we ran a series of experiments to verify the accuracy of the approximated “safe zone” by making Tao-Pie-Pie beep whenever the measured angular velocity was above or below the threshold in the X and Y plane respectively. The goal was to show that TaoPie-Pie would beep just before falling over. These experiments proved very successful. Tao-Pie-Pie detected a fall with 95% accuracy with few (< 5%) false positives. 17.7.2

Balancing Reflexes

After verifying that the gyroscope data can be used to predict a fall for Tao-Pie-Pie, the next step was to develop a method for modifying the motion parameters to avoid a fall. There are three inputs to the active balancing algorithm: 1. X-plane gyroscope reading; 2. Y-plane gyroscope reading; and 3. the current phase of the walk. Initially, the most common cause for Tao-Pie-Pie falling over was a fall to the right in phase 2 (see Fig. 17.5) or to the left in phase 5. This is due to the fact that because of the limited number of DOFs, Tao-Pie-Pie uses the ankle servo to move the COM over the right or left foot. Since the torso of Tao-Pie-Pie is fixed, Tao-Pie-Pie is precariously balanced at this point and the robot sometimes moves to far, resulting in a fall to the right or left respectively. The first balancing reflex algorithm is active when the Y-plane gyroscope reading is larger/smaller than the maximum/minimum velocity threshold in phase 2/5 respectively. In this case, the robot tends to fall towards the right/left. There are two ways in which the rotational velocity in the Y-plane can be controlled: 1. the set point for the right or left ankle servo can be changed to induce a torque in the opposite direction to the fall; 2. the robot can extend the knee and hip joint, resulting in a slowed down rotation. This effect is similar to the effect of slowing down the rotation of a chair while seated in it by extending one’s arms.

17 Advancing Artificial Intelligence through Minimalist Humanoid Robotics

369

Fig. 17.9. Servo Settings for a Straight Walk

We focus on modifying the angular velocity through the first method, since during a straight walk, the left-right velocity is mainly generated through the ankle servos. The second method is disadvantageous in that it also modifies the forward-backward balance of the robot. The set points for the servos are based on linear interpolations between a set of control points. If the angular velocity is too large, then the balancing reflex modifies the set point of the servo by moving it 10% closer to the start point of the pattern. Similarly, if the angular velocity is not large enough, then the set point is slightly extended. The same approach is used when controlling falls in the sagittal plane. In this case, however, there is no single servo that is responsible for the angular velocity. Instead, both set points for the knee and hip joint are modified by 90% to prevent a fall. The feedback from the gyroscopes is also used to detect abnormal behavior. For example, if the robot’s foot is caught on the carpet, instead of moving the leg forward, the robot will fall onto the leg too early. If this abnormal feedback is detected the robot attempts to stabilize itself by constraining all movement within the phases, in essence putting both feet on the ground as quickly as possible and straightening up its upper body. The constrainment will continue until both gyroscopes show appropriate angular velocities.

370

J. Baltes and J. Anderson

Fig. 17.10. Comparison of original walking gait (left column) and walking gait with balancing reflexes (right column) in the forward-backward (top row) and left-right (bottom row) plane

17.8 Evaluation of Balancing Reflexes We evaluated the balancing reflexes algorithm by subjectively looking at the static walking pattern. The standard walking pattern of Tao-Pie-Pie is quite stable even without balancing reflexes. The robot did not fall during any of these experiments. However, the walking gait with balancing reflexes was more balanced resulting in a straight line walk. Without balancing reflexes, Tao-Pie-Pie would veer to the right significantly. The walking speed of the robot remains unchanged. We also evaluated the balancing reflexes by subjectively by comparing the gyroscope feedback with and without balancing reflexes. The results of this comparison are shown in Fig. 17.10. As can be seen from the plots, the balancing reflexes does constrain the walking gait so that the gyroscope feedback is more in the desired envelope. Most of the time, the walking gait remains in the desired velocity envelope. Work is currently underway on developing a dynamic (shuffle-like) walk with correspondingly dynamic turns. When feedback correction was applied to walk, the walk covered more distance than without. Subjectively as well, the gyroscope feedback was not only better maintained within the desired envelope, but also formed a much more regular path, as shown in Fig. 17.11.

17 Advancing Artificial Intelligence through Minimalist Humanoid Robotics

371

Fig. 17.11. Corrected and uncorrected dynamic walking gait in the forward-backward (top) and left-right (bottom) plane

372

J. Baltes and J. Anderson

17.9 Moving Forward: The HuroCup Competition The balancing reflex algorithm used in Tao-Pie-Pie is simple, but works surprisingly well in practice. Tao-Pie-Pie has shown itself to be a powerful and flexible platform for research into humanoid robotics, and has placed numerous times in both RoboCup and FIRA competitions. As humanoid robots become more able, however, these competitions have become more challenging as well, moving from simple walking gaits and basic balancing, tasks which are the most basic for the human form, to the many skills that humans possess once walking and balancing are mastered. These skills are currently best embodied in the FIRA HuroCup event, the most difficult competition for humanoid robots in the world (Fig. 17.12). The initial HuroCup competition took place in 2002 in Seoul, Korea and HuroCup has been an integral part of the FIRA Robot World Cup ever since. Even though exciting to watch for spectators, the HuroCup event is first and foremost a research event. It focuses on the three main research areas for humanoid robotics: (a) active balancing - humanoid robots must be able to traverse a vast variety of even and uneven surfaces quickly and efficiently under varying loads, (b) complex motion planning - humanoid robots must be able to manipulate various objects without loosing their balance. For example, a humanoid robot must be able to pick up a box from under a chair without falling down or colliding with the chair, and (c) human robot interaction - the hope is that a humanoid shaped helper robot is more acceptable to elder or disabled people. However, this implies that people want to communicate with such a robot in the same or similar way that they communicate with other humans. The organizers of the HuroCup competition are aware of the fact that since there are no practical real world applications for small humanoid robots yet, competitions such as HuroCup act as a benchmark problem for research. It is thus very important that the benchmark problems are not biased but indicate true performance improvements in the real world. Even benchmarks for comparably much simpler domains, such as performance measurements for processore are fraught with pitfalls. For example, until the emergence of RISC processors, Million Instructions Per Second (MIPS) were considered a good processor benchmark. MIPS was a useful processor benchmark when the tested processors have a similar instruction set. Only the advent of RISC processors which had extremely high MIPS rating but in practice performed not much better than CISC processors with slower MIPS ratings lead to new benchmark problems. Therefore, the HuroCup competition is using the following guidelines • Full perception - action cycle: Even though it is sometimes desirable to isolate performance on very specialized tasks, there is a danger that competitors over-engineer their solutions to a particular problem without a resulting improvement on general purpose tasks. Instead, the competitions should include sensing, perception, reasoning, and execution cycle for all events. • Unmodified environments: Simply providing uniform lighting would make many of the vision challenges much simpler. However, in practical tasks, we always must assume unstructured lighting and therefore should not base our research on unrealistic assumptions.

17 Advancing Artificial Intelligence through Minimalist Humanoid Robotics

373

Fig. 17.12. Collection of images from HuroCup 2007. Top left shows the participants. Top right shows the robot dash event. Bottom right image shows the obstacle run with wall, gates, and step obstacles. The bottom right image shows our robot Blitz during the marathon run.

• Humanoid research: the competitions should focus on issues specific to humanoid robots. • Evolution: as technology and research results improve the performance of robots, the competitions must be adapted (e.g., increasing the number of obstacles or moving from a coloured ball to a textured ball). • Variability: the variability of the robot must be tested by requiring the robot to perform several tasks with a single robot. This reduces the possibility for teams to over engineer the robot by adding large feet so that it is easier to kick a ball, but much harder to walk. Based on these considerations, the HuroCup competition was developed. In the HuroCup, a robot has to perform well in eight different events to win the overall crown. The list of HuroCup events for 2008 is: 1. Robot Dash: The robot dash is a sprint event where a robot must walk 1.2m forward and then backwards over an even surface. Teams must implement at least two different gaits (forward walk and backward walk). This event is an evolution of the original HuroCup events. The current world record for this event is 27 secs.

374

J. Baltes and J. Anderson

2. Penalty Kicks: A robot must approach a randomly positioned ball and try and score on a goal defended by a robot playing goal keeper. This is also an evolution of the original three HuroCup events. 3. Obstacle Run: A robot must cross a 1.2m wide zone without touching any of the randomly placed obstacles. In 2002, the obstacles consisted of circular obstacles. Continued improvements of teams led to increasingly more difficult setups. In 2007, obstacles consisted of walls, steps (i.e., a 2.5cm high obstacle that a robot may step over), and gates (i.e., an obstacle that has enough room for the robot to crawl under). Even though the obstacle run was one of the original three challenges, the change of obstacles has greatly increased the difficulty of the event in recent years. 4. Lift and Carry: Lift and carry was introduced in 2004. The robot must carry weights across a random uneven surface. The robot that can carry the largest load is declared the winner. 5. Weight lifting: Weight lifting is similar to the Lift and Carry event, but the robot walks on an even surface only. Furthermore, in 2008, the robot must detect and grasp the weight bar autonomously. This event was introduced in 2007. 6. Basketball: A robot must throw a table tennis ball into a basket ball hoop. The robot will score three points if the ball was thrown outside of the 40cm circle, otherwise 2 points. The robot with the most points after five throws wins. 7. Marathon: An endurance race for humanoid robots. The robots must cover 42.195m (1/1000th of the distance for a human marathon run) with a single battery charge. The track is marked with colored tape. 8. Climbing Wall: In 2008, the HuroCup competition will include for the first time a climbing wall event. The robot that can climb the highest on the climbing wall is declared the winner. A robot receives points for its placing in the individual events (10 points for 1st place, 8 points for 2nd place, and so on) and the HuroCup winner is the robot with the most points over all eight events. We believe the current organization of the HuroCup challenge is ideal for advancing the state of AI through minimalist robotics. The HuroCup champion must perform well in all events to win, which encourages the entrant’s intelligent systems to be adaptable to a broad range of tasks, rather than being designed specifically for one. The nature of these challenges do mean that they are well out of the range of abilities of a robot such as Tao-Pie-Pie, however - the limited number of degrees of freedom precludes some of the necessary deformations, and the robot has no arms with which to participate in the throwing events. The balancing reflexes demonstrated by Tao-Pie-Pie, and the focus on strong AI as opposed to specialized hardware, however, allow the basic principles embodied in TaoPie-Pie to be applied to new robots that are adaptable to tasks such as these, while being minimalist in their design. We currently embody these principles in our latest robot, Blitz, which will be described in the next section. Tao-Pie-Pie was retired in 2005, and currently is on display in the Heinz Nixdorf Museum in Paderborn, Germany.

17 Advancing Artificial Intelligence through Minimalist Humanoid Robotics

375

17.10 Discussion: From Tao-Pie-Pie to Blitz Tao-Pie-Pie had two main limitations, which made it unsuitable for the HuroCup competition: (a) the 68332 did not possess sufficient processing power for vision and motion control to deal with the much richer world of the 2007 FIRA HuroCup competition, and (b) Tao-Pie-Pie only had a very limited set of motions, since it was limited to six DOFs. Rather than trying to extend the aging Tao-Pie-Pie platform, we decided to built a brand new platform. This platform still emphasizes the minimalist approach introduced in the beginning of this chapter. In 2007, we developed Blitz as an intended entry into the FIRA HuroCup and similar events. Instead of being custom built as Tao-Pie-Pie and many of our humanoids are, Blitz is based on the affordable Bioloid robotics kit developed by Robotis of Korea. Because of its very low cost (about $1,000 US), its robustness, and its flexibility, the Bioloid kit has become very popular with both researchers and hobbyists. Blitz has 18 DOFs, which means that it is able to perform many of the functions necessary to compete in HuroCup. It is able to walk, crawl, stand up, kick, pick up a ball, and lift a weight. We are currently adding a 3 axis accelerometer to provide Blitz with more sophisticated feedback for active balancing, walking over uneven surfaces, and better recovering from falls. Blitz also uses a Nokia 6600 mobile phone as its brain. The Nokia phone provides a fully programmable computing platform with a CMOS camera, bluetooth wireless communications, and a 160MHz arm processor. Similar development kits cost about $2,000CAD, but since mobile phones are produced in vast quantities and new models are released every 6 month, this and similar phones can be bought used for less than $100. We received the phones used on Blitz and Buster (Blitz’s twin) as a donation from Prof. Kopatchek, TU Vienna, Austria. This much larger set of motions means it is impossible to pre-program all motions. This introduces complex motion planning as a new research issue. In the simplest form, this means that several motions need to be combined. For example, a walking motion may need to be executed with the arms straight or while carrying a weight bar. Clearly, it is impractical to have to design different walking motions for all possible positions of the hands. However, the position of the hands, especially while carrying a weight influences the walk. To solve this problem, we have developed a new representation for motions (based on the Fast Fourier transform), which allows us to overlay motions as sequences of cosine motions. Secondly, we are currently working on a high speed, but sufficiently accurate simulation of the robot dynamics, which allows the robot to not just combine previous motions, but to generate new motions from scratch. Consistent with our approach, the minimum platform was constructed to enable to do research on complex motion planning. As thus, Blitz is a clear evolution of work that began with Tao-Pie-Pie, but with a new focus. Instead of a small set of walking and kicking motions, Blitz is used to explore issues when trying to manage an almost infinite number of possible motions.

376

J. Baltes and J. Anderson

While Blitz is a much more sophisticated robot than Tao-Pie-Pie, it is true to our original design goals of minimalism. Adding an accelerometer, for example, is a minimal way to be able to perform diagnosis after a fall, in terms of knowing which direction the fall occurred in order to ensure the ability to recover from the fall. While the demonstrations provided by sophisticated commercial robots may be more visually interesting, we believe that the HuroCup approach of creating significant challenges, coupled with minimalist solutions to those problems, will go much further in answering the basic robotics questions laid out in the introduction to this chapter.

Acknowledgements Most recently, this work has been funded through personal discovery grants by the National Science and Engineering Research Council of Canada (NSERC) to the authors. The authors would also like to acknowledge the contribution of former and current students who implement and flesh out the ideas described in this chapter. We would like to especially thank Patrick Lam (MSc., University of Auckland, New Zealand, 2002), Sara McGrath (MSc., University of Manitoba, Canada, 2007), and Jonathan Bagot (MSc., University of Manitoba, Canada, ongoing).

References 1. Braunl, T.: Embedded Robotics. Springer, New York (2003) 2. Ijspeert, A.J., Nakanishi, J., Schaal, S.: Movement imitation with nonlinear dynamical systems in humanoid robots. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA 2002), Washington, DC, pp. 1398–1403 (May 2002) 3. Kim, J.-H., Park, I.-W., Oh, J.-H.: Design of a humanoid biped robot lower body. In: Proceedings of the 3rd International Workshop on Human-friendly Welfare Robotic Systems, Daejon, Korea (January 2002) 4. Kitano, H., Asada, M.: The robocup humanoid challenge as the millennium challenge for advanced robotics. International Journal of the Robotics Society of Japan 13(5), 723–726 (2000) 5. Kun, A.L., Miller, W.T.: Control of variable speed gaits for a biped robot. IEEE Robotics and Automation Magazine 6(3), 19–29 (1999) 6. Lund, H.H., Pagliarini, L., Paramonov, L., Jrgensen, M.W.: Viki humanoid: Towards an integrated approach. In: Proceedings of 2nd International Symposium on Adaptive Motion of Animals and Machines, Kyoto, Japan (March 2003) 7. McGrath, S.: Active reflex-based balancing for small humanoid robots. Master’s thesis, Department of Computer Science, University of Manitoba, Winnipeg, Canada (June 2007) 8. Pratt, J., Pratt, G.: Intuitive control of a planar bipedal walking robot. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA 1998), Leuven, Belgium (1998) 9. Yamasaki, F., Matsui, T., Miyashita, T., Kitano, H.: PINO the humanoid: A basic architecture. In: Stone, P., Balch, T., Kraetzschmar, G.K. (eds.) RoboCup 2000. LNCS, vol. 2019, pp. 269–279. Springer, Heidelberg (2001) 10. Zhou, C.: Linguistic and numeral heterogenous data integration with reinforcement learning for humanoid robots. In: Proceedings of the 1st IEEE Conference on Humanoid Robotics (Humanoids 2000), Cambridge, MA (September 2000)

18 Design, Modeling and Control of an Ankle Rehabilitation Robot Yun-Ho Tsoi1 , Shane Q. Xie2 , and Andrew E. Graham3 1

2

3

Medical and Rehabilitation Robotics Research Group, Department of Mechanical Engineering, University of Auckland, New Zealand [email protected] Medical and Rehabilitation Robotics Research Group, Department of Mechanical Engineering, University of Auckland, New Zealand [email protected] Medical and Rehabilitation Robotics Research Group, Department of Mechanical Engineering, University of Auckland, New Zealand [email protected]

Abstract. A robotic device has been designed to carry out the range of motion and muscle strengthening exercises required for ankle rehabilitation. This chapter presents the design, modeling and control of this robotic device. Analysis on the ankle anatomy and required rehabilitation procedures resulted in the use of a parallel robot. Based on singularity and workspace analysis, suitable robot kinematic parameters were selected for the redundantly actuated parallel robot. Modeling of the manipulator and the human ankle was carried out to facilitate controller design. The manipulator was modeled through application of Lagrange’s equations while ankle kinematics and dynamics was assumed to be based on the biaxial joint ankle model. A computed torque impedance controller has been developed to regulate the relationship between the applied moments at the ankle and the ankle rotary motion. To ensure accurate estimation of state variables, a kinematic self calibration routine has also been developed for the parallel robot using redundant sensing. To allow for further development of the controller, the recursive least squares algorithm has been applied to estimate the ankle stiffness and damping parameters. This information will be used in future work for stability analysis and controller parameter adaptation. Finally, results on the simulation of the system were presented to show the performance of the developed controller.

18.1 Rehabilitation Robots Rehabilitation robots can be considered as robotic devices that are designed to assist the disabled and the elderly in carrying out everyday activities. In recent years, there has been growing research interests in the use of robots for physical therapy. In fact, some robotic systems have already been successfully developed and are currently used at hospitals and clinics. Among the most notable robots used in physical therapy are the MIT-MANUS upper limb rehabilitation device and the LOKOMAT gait training system [12, 17, 18]. Both these devices are used to facilitate motor training of stroke survivors, based on the notion that physical therapy can contribute to neuro-recovery and hence improved motor functions at the affected limbs [17]. Researchers have also developed devices which assist in the rehabilitation of joint injuries such as ankle sprains [11, 28]. D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 377–399. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

378

Y.-H. Tsoi, S.Q. Xie, and A.E. Graham

Rehabilitation robots have been developed to address the fact that physical therapy is a labor intensive task for the physiotherapist [18]. This is mainly due to the repetitive nature of rehabilitation exercises and the significant forces required in assisting the patients. Robotic devices on the other hand are ideal for such tasks. However, due to the close interaction with humans and large variability in environmental conditions such as stiffness and damping parameters of the limbs and joints, intelligent force or interaction control strategies are necessary for a safe and effective implementation of the robotic device. Another advantage of a robotic device is that it can collect large amounts of data accurately to facilitate analysis on the patient’s progress, thus allowing the clinician to tailor an optimal rehabilitation program to enable faster recovery [17]. In fact, the expert knowledge of the clinician can be embedded into the robotic system itself and appropriate changes in the exercise parameters could be done adaptively. Such adaptive behavior could be based on measurements or estimation of certain quantities such as completion time for each exercise cycle and accuracy of the motion as compared with a reference path [17]. Computational intelligence methods such as fuzzy systems and adaptive algorithms are therefore highly relevant for the realization of an intelligent rehabilitation robot. This chapter will focus on the mechanism design, modeling and control of a robot for rehabilitation of the human ankle. Sect. 18.2 will elaborate on the human ankle, its rehabilitation procedure and existing ankle rehabilitation devices. Sect. 18.3 will discuss the design of the parallel manipulator used in this research. Sect. 18.4 then covers the dynamic modeling of the robot and the modeling of human ankle as an operating environment. Sect. 18.5 details the control strategy implemented and proposed in this research, while also exploring self-calibration and environmental estimation methods. Simulation results are presented in Sect. 18.6 and finally, Sect. 18.7 provides conclusions for the current research and highlights the required future work.

18.2 Ankle Rehabilitation The human ankle plays an important role in maintaining the balance of an individual during gait. It is capable of rotational motion about all three anatomical planes as shown in Fig. 18.1. It could also be subjected to impact forces with magnitudes of up to several times the body weight. According to [16], passive resistive moments at the ankle can go up to about 50 N m while moment applied around the ankle through active muscle contraction can be in the range of 100 N m. Ankle sprain is a common injury that requires physical therapy. This injury occurs when ligaments at the ankle are over-stretched or torn, typically due to excessive motion in the inversion and plantarflexion directions [20]. Careful treatment must be applied to sprained ankles to minimize recurrent injuries. The procedures involved in the treatment of sprained ankles can be found in [23]. It normally begins with the immobilization of the affected ankle to reduce swelling and to stop effusions. Once active effusion has ceased, non-weight bearing range of motion exercises should be carried out to stimulate the healing of ligaments and reduce muscular atrophy. As the patient develops full weight bearing capability, strength training exercises are commenced to

18 Design, Modeling and Control of an Ankle Rehabilitation Robot

379

Fig. 18.1. Motion available at the human ankle

allow strengthening of the muscles around the ankle to prevent future injuries. As the patient improves further, proprioceptive training is then started to enhance the patient’s sense of joint position and joint motion, which is critical in maintaining balance during gait. Several autonomous ankle rehabilitation devices have already been developed by various research groups. One such devices is the Rutgers Ankle presented in [11] where a pneumatically actuated six degree of freedom Stewart platform has been used. The device was aimed at allowing patients to carry out rehabilitation through a virtual reality environment, using the Stewart platform as a haptic interface. Apart from sprained ankle rehabilitation, the Rutgers Ankle was also used in clinical trials for motor training for stroke survivors [8]. An ankle rehabilitation platform with four degrees of freedom has been developed in [27, 28]. Due to the reconfigurable nature of the platform, this device can be used for range of motion, strengthening and proprioceptive exercises required in the ankle rehabilitation program. Different exercise requirements could be met by varying the impedance parameters using impedance control. Although existing ankle rehabilitation devices are already capable of carrying out basic rehabilitation exercises, elements of adaptability and intelligence are among other aspects which are still less developed compared with more mature rehabilitation robots such as the MIT-MANUS. These existing devices also appear to neglect modeling of the ankle in mechanism and controller design. The current research therefore aims to develop an ankle rehabilitation robot that can adapt to the patient’s characteristics in terms of ankle kinematics and dynamics, by integration of online parameter estimation and intelligent control.

18.3 Robot Design Properties of the ankle have to be identified for the development of a practical design. A review of publications in the area of biomechanics has provided specifications on two important design criteria, namely the range of allowable motion and the required ankle moment. The range of motion and the passive resistive moments observed at the ankle are presented in table 1 below. The mechanism developed by the Medical and Rehabilitation Robotics Group at the University of Auckland for this research is that of a three degree of freedom rotational

380

Y.-H. Tsoi, S.Q. Xie, and A.E. Graham

Table 18.1. Range of motion and maximum passive moment at the ankle

Type of Motion

Max. Allowable Motion Max. Passive Moment

Dorsiflexion Plantarflexion Inversion Eversion Abduction/External rotation Adduction/Internal rotation

20.3◦ to 29.8◦ a 37.6◦ to 45.8◦ a 14.5◦ to 22.0◦ a 10.0◦ to 17.0◦ a 15.4◦ to 25.9◦ a 22.0◦ to 36.0◦ a

−33.1 ± 16.5 N m b 40.1 ± 9.2 N m b 34.1 ± 14.5 N m b −48.1 ± 12.2 N m b 41 N m c −41 N m c

a: Range of motion of the ankle as determined by [24]. b: The quasi-static moment required to cause failure in the ankle joint as determined by [22]. c: The moment capacity of the Rutgers Ankle about the vertical axis [11].

parallel robot. Parallel mechanisms in general are capable of large loads and high accuracy compared with their serial counterparts. However, one drawback of such mechanisms is the limited workspace. Fortunately in this application, the range of motion of the human ankle is also rather limited, making parallel mechanisms a suitable design choice. An aspect that sets the design of the robot in this research apart from most existing devices is the fact that the human ankle in this case forms part of the robot kinematic constraint. This allows the ankle to stay stationary during the rehabilitation process, while the foot is moved around for the required exercises. This is thought to create an arrangement which bears closer resemblance to the case where the exercise is assisted by a therapist. The utilization of human joint as part of the robot kinematic constraint can also be found in [4], where the robotic device is applied in robotic gait training. To simplify the design process, the ankle is approximated as a ball and socket joint fixed at a predefined location. Electrically driven linear actuators have been selected to drive the device and produce orientation changes. The individual link vector and link length of the parallel mechanism can be represented by Eq. (18.1) and Eq. (18.2) respectively. (18.1) Li = r + RPi − Bi li = Li  ⇒ li =

 (r + RPi − Bi )T (r + RPi − Bi )

(18.2)

The position vectors used are as shown in Fig. 18.2. Li is the ith link vector, r is the position vector from the origin of the base coordinate frame to the spherical joint representing the ankle, R is the 3 × 3 rotational transformation matrix, Pi is the position vector from the ankle joint to the attachment point of link i at the end effector platform, expressed in the local coordinate frame of the end effector, Bi is the position vector from the origin of the base coordinate frame to the connection position of link i at the base plate and li is the scalar length of the vector Li .

18 Design, Modeling and Control of an Ankle Rehabilitation Robot

381

Fig. 18.2. Position vectors used in robot kinematics

By representing the task space orientation in terms of the XYZ Euler angles, the R matrix can be expressed as a function of three angles, namely θ , ϕ and φ . The rates  T of change of link lengths, l˙ = l˙1 · · · l˙i is related to the rates of change of the Euler T  angles, Θ˙ = θ˙ ϕ˙ φ˙ via the inverse Jacobian matrix, J−1 . The ith row of the inverse Jacobian, Jinv,i is given by Eq. (18.5).  dli d  T (18.3) r r + Bi T Bi + Pi T Pi + 2 (r − Bi )T RPi − 2rT Bi 2 li = dt dt   ∂R ˙ ∂R ∂R ˙ = 2 (r − Bi )T θ+ ϕ˙ + φ Pi (18.4) ∂θ ∂ϕ ∂φ

dli 1 = Jinv,iΘ˙ = (r − Bi )T ∂∂ Rθ Pi ∂∂ ϕR Pi ∂∂ Rφ Pi Θ˙ (18.5) dt li The kinematics of the parallel mechanism were first considered in the design process to allow assessment of the achievable workspace based on a particular linear actuator stroke length. This is done by first identifying the corresponding link lengths Eq. (18.2) for a particular task space orientation using inverse kinematics, and then testing these lengths against the range of allowable actuator stroke lengths. In terms of force analysis, the transpose of the inverse Jacobian obtained from Eq. (18.5) was used to evaluate the required forces at each actuator in order to produce the desired resultant moment at the ankle, since τ = J−T F. Based on the workspace and force analysis, the design parameters were selected in a trial and error fashion so that the workspace and moment requirements were satisfied as closely as possible. Regions close to the singularity points in the mechanism were also identified numerically and this had led to the addition of a redundant linear actuator to reduce singularities in the workspace of the robot. The regions close to singularity of the three linked manipulator are shown in Fig. 18.3a. A point is considered to fall within this region if the determinant of the inverse Jacobian has a magnitude lower than a value of 10−6 (this corresponds to a condition number of about 1095 and larger condition numbers generally lead to less accurate numerical solutions). It can be seen that the singular regions form almost a surface across the workspace. This will lead to high actuator forces or even loss of controllability if the end effector orientation trajectory approaches these regions. Fig. 18.3b on the other hand shows the points at which the condition number of the

382

Y.-H. Tsoi, S.Q. Xie, and A.E. Graham

Fig. 18.3. The regions close to singularity in the (a) non-redundant case and (b) redundant case

Fig. 18.4. CAD model of the ankle rehabilitation robot

Jacobian is greater than eight for the redundantly actuated design. Condition number has been used in the analysis for the redundantly actuated design due to the non-square inverse Jacobian matrix. The small condition number was chosen, somewhat arbitrarily, to better illustrate the regions approaching singularity in the redundantly actuated option, which in actual fact would be almost non-existent if comparable condition number as the non-redundant case is used as the threshold. Hence, it is clear that the addition of a redundant actuator can significantly improve the manipulability of the robot. A computer aided design (CAD) model of the resultant parallel robot prototype is shown in Fig. 18.4, while the workspace of the parallel robot within the range of ankle motion is shown in Fig. 18.5.

18 Design, Modeling and Control of an Ankle Rehabilitation Robot

383

Fig. 18.5. Desired workspace and achievable workspace of the robot. The black region represents the desired region reachable by the big toe while the overlapping gray region represents the actual achievable region on this rehabilitation robot.

DC motor driven linear actuators with a stroke length of 0.1 m and maximum force of 2230 N were chosen for this application. In terms of sensor selection, linear potentiometers are included in the actuators to provide feedback of the actuator lengths. Additionally, a two axis inclinometer was chosen to provide orientation feedback in terms of roll and pitch angles. The redundant sensors can be used for self calibration of the parallel robot and for estimation of instantaneous center of ankle rotation during rehabilitation exercises. Due to the relatively large size of industrial force/torque sensors and the space constraints on the end effector, it was proposed that in line force sensors be used to measure the actuator axial forces. These measured forces can then be utilized for the estimation of the external moment applied based on the inverse Jacobian and robot dynamics.

18.4 Modeling Modeling of the system to be controlled is vital to enhance the understanding of the platform and ankle characteristics, which is critical in the development of an effective controller. In this particular application, both the rehabilitation robot and the operating environment need to be modeled due to their coupled nature. At the current stage of this research, dynamic models of the parallel mechanism and the linear actuator dynamics have been obtained while preliminary modeling of the human ankle has also been carried out using simplified approaches. The dynamics of parallel robots are generally more complicated compared to serial robots. However, the final form of the dynamic equations for a parallel robot is still equivalent to that of serial robots. These equations could be represented in task space in the form of Eq. (18.6). MΘ¨ + C(Θ˙ , Θ ) + G(Θ ) = J−T F + τe

(18.6)

M here is the mass or inertia matrix, C is the centripetal and Coriolis terms, G is the gravitational terms while F and τe are respectively the actuator force vector and

384

Y.-H. Tsoi, S.Q. Xie, and A.E. Graham

the vector of external moments applied on the end effector. Note that friction has been ignored at this stage. The Lagrange formulation has been chosen for the development of the robot dynamic model due to its systematic approach which is highly suitable for complex mechanical systems. The dynamics of the end effector platform and individual links were modeled separately and subsequently combined using the relationships between the task space and joint space coordinates. Lagrange’s equation for the platform in task space is given below in Eq. (18.7). L p represents the Lagrangian of the platform.  T The vector xi yi zi is equivalent to the ith link vector Li as given in Eq. (18.1), while fxi , fyi , fzi denotes the magnitude of the Cartesian components of the reaction force. N in this case refers to the total number of linear actuators.    N  ∂ Lp d ∂ Lp ∂ xi ∂ yi ∂ zi = ∑ fxi + fyi + fzi − (18.7) dt ∂ Θ˙ ∂Θ ∂Θ ∂Θ ∂Θ i=1 Similarly, the Lagrange’s equation for individual links in joint space is given in Eq. (18.8), where Li represents the Lagrangian of the ith link and Fi is the force applied along that linear actuator. The vector ρi here represents the joint space parameters  T αi βi li , which are made up of the two angular displacements of the universal joint at the base plate (αi , βi ) and the link length li . Note that the directions of the reaction forces are reversed here compared to the task space equations.   T d ∂ Li ∂ Li ∂ xi ∂ yi ∂ zi  = − fxi − fyi − fzi + 0 0 Fi (18.8) − dt ∂ ρ˙i ∂ ρi ∂ ρi ∂ ρi ∂ ρi

T ∂ xi It is useful to note that ∂Θ = ∂∂((αθi,,ϕβi,,lφi)) ∂∂ ρxi , where: i

⎡ ∂ αi

∂ (αi , βi , li ) ⎢ = Jρ i = ⎣ ∂ (θ , ϕ , φ )

∂θ ∂ βi ∂θ ∂ li ∂θ

∂ αi ∂ϕ ∂ βi ∂ϕ ∂ li ∂ϕ

∂ αi ∂φ ∂ βi ∂φ ∂ li ∂φ

⎤

⎡

⎥ ∂ xi ⎢ =⎣ ⎦, ∂ ρi

∂ xi ∂ αi ∂ xi ∂ βi ∂ xi ∂ li

⎤

⎡ ∂x ⎤ i

∂ xi ⎢ ∂∂ xθi ⎥ ⎥ = ⎣ ∂ϕ ⎦ ⎦ and ∂Θ ∂ xi ∂φ

With this established, Eq. (18.7) can be expressed as Eq. (18.9). Equation (18.10) can then be obtained by rearranging Eq. (18.8) and substituting it in Eq. (18.9). By rewriting Eq. (18.10) to give Eq. (18.11), the dynamic equations of the complete system can then be represented as Eq. (18.12), which can subsequently be manipulated to give the general robot dynamic equations as in Eq. (18.6).  

  N  ∂ Lp ∂ Lp ∂ (αi ,βi ,li ) T ∂ yi ∂ xi ∂ zi d fxi ∂ ρ + fyi ∂ ρ + fzi ∂ ρ (18.9) dt ∂ Θ˙ − ∂Θ = ∑ ∂ (θ ,ϕ ,φ ) i=1

i

i

i

⎡ ⎤⎞⎤ ⎡ ⎛ 0  

T   N ∂ L ∂ L ∂ ( α , β ,l ) p p d i i i ⎣ ⎝− d ∂ Li + ∂ Li + ⎣ 0 ⎦⎠⎦ − = ∑ ˙ dt ∂ Θ dt ∂ ρ˙i ∂Θ ∂ ρi ∂ (θ ,ϕ ,φ ) i=1 Fi  

T     N  N ∂ li ∂ Lp ∂ Lp ∂ (αi ,βi ,li ) ∂ Li d d ∂ Li Fi =∑ dt ∂ Θ˙ − ∂Θ + ∑ dt ∂ ρ˙i − ∂ ρi ∂ (θ ,ϕ ,φ ) ∂Θ i=1 i=1  

    N  ∂ Lp ∂ (αi ,βi ,li ) T d ∂ Li ∂ Li d ∂ Lp = J−T F dt ∂ Θ˙ − ∂Θ + ∑ dt ∂ ρ˙ − ∂ ρ ∂ (θ ,ϕ ,φ ) i=1

i

i

(18.10)

(18.11) (18.12)

18 Design, Modeling and Control of an Ankle Rehabilitation Robot

385

Fig. 18.6. The upper ankle and subtalar joints (reproduced from [9])

Dynamic equations of linear actuators which relate the input voltage to the output actuator force and motion were also obtained by modeling the actuator as a geared permanent magnet DC motor driving a ball screw. Equation (18.13) below gives the actuator dynamics under Laplace transform. V (s) is the input voltage, F(s) is the output force, v(s) is the linear velocity of the actuator, L is the motor inductance, Ract is the winding resistance, Kt is the motor torque constant, Ke is the motor electrical constant for back emf, J is the rotational inertia, and b is the damping coefficient, with m and s subscripts denoting motor and ball screw respectively. N is the gear ratio between the motor and the ball screw, Ka is the actuator gain relating the motor angular velocity to the linear velocity (i.e. ωm = Ka v) and η is the efficiency of the ball screw. The resulting model has been used in simulation and controller implementation.    F(s) Js bs V (s) − Ke Ka v(s) − Ka Jm + 2 s + bm + 2 v(s) = (18.13) Kt Ls + Ract N N Ka η Although the design of robot has assumed the ankle to be equivalent to a spherical joint, the actual human ankle kinematics does differ quite significantly from this assumption since the rotational motion about various axes are not independent. As a result, a more realistic kinematic model of the ankle was also used in the simulation of the overall system. This structure models the ankle with two oblique revolute axes. One of these axes represents the upper ankle joint which is primarily responsible for the plantarflexion/dorsiflexion motion. On the other hand, the second axis is used to approximate the subtalar joint which allows coupled rotations on all three anatomical planes, but with a more significant influence on inversion/eversion [9]. Fig. 18.6 shows the orientation and location of these axes at the human ankle foot complex. As the parameters which define the upper ankle and subtalar joints vary considerably among individuals, this model could be used as the basis for parameter estimation to enable better understanding of the kinematical constraints imposed by the environment. The characteristic stiffness of the ankle is also important for the modeling of operating environment. A lot of work in the literature has been done in determining the passive and active ankle stiffness in the plantarflexion/dorsiflexion direction. For instance, [5, 16, 22] have all studied the flexibility of the human ankle in the plantarflexion/dorsiflexion

386

Y.-H. Tsoi, S.Q. Xie, and A.E. Graham

direction by finding the ankle passive moments at a corresponding angular displacement. Results have shown that the stiffness at the ankle is very much non-linear and the stiffness tends to increase as the foot orientation moves further away from the neutral position. Unfortunately, not much information can be found on the ankle stiffness about the other two axes while damping values are also unavailable. Reference [5] did however measure the moments about inversion/eversion and abduction/adduction directions. The results obtained suggest that all three axes of rotations at the ankle have similar passive stiffness trends . This has led to modeling of stiffness using a nonlinear function in the simulation. Damping has been neglected at this stage as the velocity involved in the rehabilitation exercises is expected to be reasonably slow.

18.5 Control 18.5.1

Impedance Control

As significant forces and moments are required to carry out the passive range of motion and resistive rehabilitation exercises, it is evident that force/interaction control of the ankle rehabilitation robot is critical. This is to ensure safe and correct operation of the robot. Impedance control was proposed by [13] as a generalized approach for force and motion control and has been adopted widely in existing rehabilitation robots [10, 17, 26, 28]. Many more advanced control strategies were also built upon the impedance control principle. In essence, an impedance controller aims to regulate the impedance of the manipulator so that a desired force and motion relationship is observed when the robot manipulator interacts with an external environment [15]. The mechanical impedance, Z is given in Eq. (18.14) as the transfer function between force, F and velocity, X˙ . M here is the inertia parameter, B is the damping parameter while K is the stiffness parameter. X is normally taken to be the difference between the reference manipulator position (Xd ) and the actual manipulator position (Xm ), i.e. X = Xd − Xm . Z(s) =

K F(s) = Ms + B + ˙ s X(s)

(18.14)

One distinct advantage of impedance control compared to hybrid force position control strategies is that no switching in the controller is required between free motion and constrained motion as the impedance relationship will allow position tracking when no contact is made. Hybrid force position controllers also require accurate knowledge of the kinematical constraints of the environment to allow determination of the force controlled and position controlled directions in the task space. Since environmental characteristics are largely unknown in rehabilitation applications, an impedance controller was implemented for the ankle rehabilitation robot. To address nonlinearities in the parallel mechanism, a nonlinear compensator has been used to cancel out the centripetal/Coriolis and gravitational terms in the robot dynamic equations (i.e. C and G in Eq. (18.6)). Such an approach is normally termed computed torque control. Computed torque controllers have been widely used for trajectory tracking in robots. Without consideration of the external forces/moments, the control law of such a controller is given in Eq. (18.15).

18 Design, Modeling and Control of an Ankle Rehabilitation Robot

    F = JT M Θ¨ d + Kd (Θ˙ d − Θ˙ ) + Kp(Θd − Θ ) + C(Θ˙ , Θ ) + G(Θ )

387

(18.15)

Θd here is the desired/reference task space coordinates, Kd is the derivative gain and Kp is the proportional gain of the controller. All other variables have the same meaning as those in Eq. (18.6). In the absence of external torques and with suitable choice of Kd and Kp , the error will decay to zero, thus allowing trajectory tracking. The control law used in impedance control of the ankle rehabilitation robot is essentially a rearrangement of Eq. (18.15) and is given in Eq. (18.16) below, where B and K are the desired manipulator damping and stiffness matrices respectively.   J−T F = M Θ¨ d + B(Θ˙ d − Θ˙ ) + K(Θd − Θ ) + C(Θ˙ , Θ ) + G(Θ ) (18.16) The use of this control law will produce the manipulator behavior as shown in Eq. (18.17) when external torques applied to the manipulator are included in the manipulator dynamic equations. This represents an impedance relationship between the external torques and the angular errors, with inertial, damping and stiffness parameters of M, B and K respectively. Under this control law, the parameters B and K can be selected as desired while the apparent manipulator inertia will be dependent on the manipulator inertia matrix at the corresponding end effector configuration. M(Θ¨ d − Θ¨ ) + B(Θ˙ d − Θ˙ ) + K(Θd − Θ ) = −τe

(18.17)

It should be noted that since the robot manipulator used is redundantly actuated, computation of the matrix inverse for J−T in the control law Eq. (18.16) cannot be done as a matrix inversion operation due to the matrix J−T being a 3 × 4 non-square matrix. To resolve this issue, several approaches were investigated. The first can be found in many papers, which requires the use of pseudo inverse of J−T . This way of redundancy resolution produces actuator forces with minimal Euclidean norm [6]. Alternatively, the extra degree of freedom provided by the redundant actuator could be used to satisfy some constraints or optimize some objectives [2]. The implementation for this ankle rehabilitation robot has used the extra degree of freedom in actuation to control the vertical reaction forces at the ankle. This is done by ensuring that the sum of the vertical components of the actuator forces results in the desired vertical force. The decision to control the vertical ankle reaction force was made so that different levels of ”weight” can be applied at the ankle depending on the requirements of the rehabilitation exercise. Using this method, the dynamic equations of the robot Eq. (18.6) is augmented with an additional criterion and its new form is shown in Eq. (18.18) below.     τ MΘ¨ + C(Θ˙ , Θ ) + G(Θ )  (18.18) = J F+ e 0 fvert   J−T  Where J = lz1 lz2 lz3 lz4 l1

l2

l3

l4

Furthermore, fvert is the desired vertical ankle reaction force, lzi is the vertical component of the ith link vector in base coordinates and li is the length of the ith link vector. The inclusion of this additional control criterion therefore results in a control force vector as given in Eq. (18.19).

388

Y.-H. Tsoi, S.Q. Xie, and A.E. Graham

F = J−1



M(Θ¨ d ) + B(Θ˙ d − Θ˙ ) + K(Θd − Θ ) + C(Θ˙ , Θ ) + G(Θ ) fvert

 (18.19)

Recalling that the actuator dynamics are as given in Eq. (18.13) and assuming that the inductance of the motor is negligible to simplify computation, Eq. (18.20) has been used to calculate the control voltage for the ith actuator.      1 Ract Js bs Kt Ke ˙ Vi = Fi + Jm + 2 Ka l¨i + bm + 2 + Ka li (18.20) Kt Ka η N N Ract Inspection of the control law used to obtain the actuator force vector reveals that measurements of the rates of change of task space coordinates (i.e. the Euler angles) of the robot are required for the computation process. However, as mentioned in the design discussion, the position sensors available are in the form of four linear potentiometers which measure the link lengths and a two axis inclinometer that measures the roll and pitch of the end effector. This means that the yaw angle of the end effector will have to be estimated based on these sensor measurements. It has also been pointed out that the ankle kinematics cannot be realistically modeled by a simple spherical or gimbal joint. Using the available sensors, it is possible to construct an algorithm for a state estimator that computes the yaw angle and the position vector of the instantaneous center of ankle rotation in the base coordinate frame. The relationship between the orientation of the platform, the link lengths and the ankle joint center were given earlier in Eq. (18.1). Due to the use of redundant actuation, four such equations can be formed and with the use of redundant sensing to measure the end effector roll and pitch angles, the rotational matrix R becomes a function of φ alone. Together with the three x, y and z coordinates of the instantaneous ankle center, four nonlinear equations with four unknowns can be formed. By rewriting this set of equations in the form of gi = 0, i = 1, 2, 3, 4 the NewtonRaphson method can be used to solve for the unknown variables. The expression for gi is given in Eq. (18.21). (18.21) gi (x, y, z, φ ) = li 2 − r(x, y, z) + R(φ )Pi − Bi 2 = 0 T T  T T   Let G = g1 g2 g3 g4 and w = x y z φ = r φ , the solution of G = 0, if it exists, can be found using the Newton-Raphson method with the parameter update rule given in Eq. (18.22):   −1 ∂ G  wn+1 = wn − G|wn (18.22) ∂ w wn   Where ∂∂ G w w is the Jacobian matrix of the vector G with respect to vector w at wn n   and n = 0, 1, . . . is the iteration number. The individual elements of ∂∂ G can be found w using the relationships in Eq. (18.24).

T ∂G = ∂∂gw1 ∂∂gw2 ∂∂gw3 ∂∂gw4 ∂w ⎡ ∂ g  ⎤    i −2 (rn + R(φn )Pi − Bi )  ∂ r w ∂ gi   = ⎣ ∂ g  n ⎦ = −2 (r − B )T ∂ R  P i n i i ∂ wn wn ∂ φ φ ∂φ  n wn

wn

(18.23) (18.24)

18 Design, Modeling and Control of an Ankle Rehabilitation Robot

389

Fig. 18.7. Block diagram of the control framework

One drawback for the use of such a numerical solution approach is the sensitivity of the solution with respect to constant parameters such as Pi and Bi in the nonlinear equations. It was observed that inconsistencies in these parameters with the measured link lengths could potentially lead to the Jacobian matrix for G becoming singular or imaginary. As a result, it is vital that the correct values of the constant parameters be found through kinematic calibration before any attempt is made on solving these nonlinear equations. The details of the kinematic calibration procedure developed in this research are presented in the next section. It should also be noted that due to the time varying nature of the ankle center of rotation (represented by vector r), the actual kinematic inverse Jacobian matrix will be different from the existing formulation. However, as the analytical relationship between r and the task space angles has yet to be established, the kinematic inverse Jacobian is computed using the same formulation as in Eq. (18.5), but with the estimated r. Further work is necessary to identify the correct formulation either through numerical methods or closer investigation of the ankle kinematics. The control scheme used in the ankle rehabilitation robot can be summarized as the block diagram in Fig. 18.7. Note that the angular velocities required in the control law are actually computed from the measured linear velocities. Low pass filtering is applied to reduce high frequency noise due to numerical differentiations of link lengths. 18.5.2

Kinematic Self Calibration

Accuracy of the kinematic parameters is critical in ensuring the existence and reliability of the numerical solution for the yaw angle and ankle centre of rotation. Kinematic calibration of the robot must therefore be done to identify the correct parameters for the robot. Since there are six position sensors and only three degrees of freedom of motion, self calibration of the parallel robot is possible. To simplify the calibration process, an artificial structure will be used to replace the lower limb. This structure can be seen in Fig. 18.4 and is essentially a vertical link with a spherical joint towards its end which is used to represent the centre of rotation of the ankle.

390

Y.-H. Tsoi, S.Q. Xie, and A.E. Graham

With the setup described above, the parameters to be calibrated include: • the z location of the centre of rotation of the spherical joint, expressed in the fixed base coordinate frame, which origin coincides with the point obtained by projecting the ankle spherical joint on the base plate, • the x and y coordinates of the universal joint centers at which linear actuators are attached to the base plate, again expressed in the base coordinate frame (note that it is assumed that all z coordinates are identically zero to reduce the number of parameters), • the x, y and z coordinates of the attachments of the linear actuators at the end effector platform, expressed in the moving end effector local coordinate frame (which is centered at the ankle spherical joint and rotates with the end effector platform), and • the offsets between the length measured by the linear potentiometers and the actual link length. This gives a total of 25 independent parameters to be calibrated. The calibration algorithm is based on the use of forward kinematics. The end effector orientation is estimated using the link length measurements and this estimate is then compared to the measured pitch and roll angles to obtain an error. This error is subsequently used to obtain the next parameter estimate. The rationale behind the calibration algorithm is presented below. Equation (18.25) shows that the actual Euler angles, Θ are dependent on the measured link lengths, l and the actual kinematic parameters, w while Eq. (18.26) shows the same relationship, but in terms of estimated values denoted with an additional ˆsymbol.

Θ = f (l, w) ˆ Θˆ = f (l, w)

(18.25) (18.26)

Linearization of the forward kinematics relationship using first order approximation of the Taylor Series gives Eq. (18.27).  ∂ f (l, w)  Θ − Θˆ = Δ wˆ (18.27) ∂ w l,wˆ Due to the use of forward kinematics for a parallel mechanism, analytical expressions of the partial derivatives of the end effector orientation with respect to individual kinematic parameters cannot be easily found. The numerical Jacobian of the Euler angles vector with respect to the calibration parameter vector was therefore required. Each element of the Jacobian was calculated by observing the change in forward kinematics results due to a small perturbation in the calibration parameter corresponding to that particular matrix element. The magnitude of the perturbation used was 10−6 m, as recommended by [3]. As only two Euler angles are measured, each observation of robot configuration will only contribute two rows to the Jacobian matrix, while there are 25 parameters. This means that more observations will have to be recorded and the least squares method should be used on the augmented Jacobian matrix to identify the change in parameters required to compensate for the calculated errors. With this in mind, the final recursive

18 Design, Modeling and Control of an Ankle Rehabilitation Robot

391

algorithm of the self calibration is presented below in Eq. (18.28) Where γ is the adaptation or learning rate, A∗ denotes the pseudo inverse of matrix A, subscripts k = 1 to N denotes the observation number and subscript n denotes the iteration number. Also, JFk  is the first two rows of the kth Jacobian ∂ f∂(l,w) and Θk is the first two rows of the w  ˆn lk ,w

kth Euler angle vector Θk , since the inclinometers can only measure the roll and pitch angles, from which θ and ϕ can be obtained. This algorithm is run repeatedly with the same set of observations until the change in Euclidean norm of the angular error falls below a certain threshold. ⎡  ⎤ ⎡ ⎤ Θ1 − Θˆ 1 JF1 ⎢ ⎥ ⎢ . ⎥ .. ˆ n+1 = w ˆ + γ A∗ ⎣ w (18.28) ⎦ , A = ⎣ .. ⎦ .   ˆ JFN Θ −Θ N

N

As this entire calibration process lacks information regarding the yaw angles, the solution of the link attachment positions on the end effector found will almost certainly be rotated about the vertical axis. A correction step must therefore be performed at the very end of the calibration process to rotate these positions back so that at the yaw angle reads zero at the neutral position. This therefore necessitates the observation of the link lengths at the neutral position. With these lengths, the estimated yaw angle can be found and reset back to zero by rotating the end effector link attachment points in the reverse direction. Initial trials have shown that the adaptation/learning rate, γ is critical in determining the convergence and efficiency of the algorithm. It was noted that smaller γ tends to give better convergence properties while also slowing down the calibration process and vice versa. A γ value of 0.05 was used in the current self-calibration algorithm. The current algorithm has been tested using simulations in MATLAB whereby two kinematic models of the parallel robot have been used. One uses the designed kinematic parameters as the computer model while the other uses kinematic parameters that are slightly distorted by some random variations to simulate the actual manipulator. Sensor measurements were generated using the “real” manipulator while the parameters in the computer model are calibrated so that it approaches the actual values. The implemented algorithm was used with 36 observations of link lengths and end effector orientations. The criterion for termination of the algorithm is the reduction of the rate of change of error norm to a value below 10−8 . It was observed that starting with an error norm of 0.16 rad, the calibration required 291 iterations to give a final angular error norm of 8.7 × 10−8 rad. The largest discrepancy between the calibrated and actual parameter was determined to be 1.8 × 10−6 m. Further work will be required to adaptively identify a suitable γ depending on the size of the error and other properties such as the condition number of the augmented Jacobian matrix. This will improve the efficiency of the calibration algorithm. Online implementation of the algorithm could also be developed to reduce waiting time for the calibration process. Another important task will be to devise a method whereby the calibration could be done with the patient’s lower limb in place. This would cause the observations to differ from the model where the ankle is treated as a spherical joint. The double jointed ankle kinematic model could perhaps be adopted to provide more

392

Y.-H. Tsoi, S.Q. Xie, and A.E. Graham

parameters for identification while providing more accurate relationship between the link lengths and the end effector orientation. This will allow both the robot parameters and environmental kinematical constraints to be identified simultaneously. 18.5.3

Environment Parameter Estimation

One major issue associated with force control is the stability of the manipulator when it comes into contact with the external environment [1, 7, 14]. Although it is mostly possible to design a stable fixed gain force/interaction controller for a particular set of environmental parameters, a variation in these parameters could still lead to deterioration of performance or instability of the system [25]. In applications when not much is known about the environment, it is ideal if the force/interaction controller can adapt its parameters to suit the environment. In such a controller, knowledge of the environment will have to be estimated through system identification or adaptation methods. Reference [10] has devised an interaction controller for upper limb rehabilitation which estimates the arm impedance of the patient and obtain optimal gains for a PI controller by feeding the estimated parameters through an artificial neural network that has been trained with optimal gains for particular impedance parameters. The ultimate aim of this research is to develop a similar adaptive controller for ankle rehabilitation, where manipulator impedance parameters are adaptively computed based on the ankle dynamic characteristics, so that more anatomically correct motion can be achieved with less force applied in the non-compliant directions, all while maintaining stability. As a starting point, an online estimation algorithm for the ankle dynamic parameters has been developed using the recursive least squares (RLS) method. In this estimation algorithm, the environment has been estimated to behave as a linear spring damper system as given in Eq. (18.29).

τe = Be (Θ˙ 0 − Θ˙ ) + Ke (Θ0 − Θ ) (18.29) Where τe has the same meaning as in Eq. (18.6) and Θ˙ 0 is the task space coordinates at the neutral position (normally = 0). The current implementation has assumed that the environmental damping and stiffness matrices are diagonal. This had led to the following formulation for the RLS method as shown in Eq. (18.32), where wn are the dynamic parameters to be identified, e is the error between the estimated external torque vector and actual external torque vector, λ the exponential forgetting factor (λ ≤ 1. When λ = 1, all previous values will be considered), α a constant scalar which is normally very large in value (108 to 1015), I the 6 × 6 identity matrix and n is the iteration number. en+1 = τe,n+1 − xTn+1 wn

 −1 T 1 Pn − Pn xn+1 λ + xTn+1 Pn xn+1 xn+1 Pn Pn+1 = λ wn+1 = wn + Pn+1 xTn+1 en+1 ⎤ 0 0 0 −θ˙n −θn 0 xn = ⎣ 0 0 −ϕ˙ n −ϕn 0 0 ⎦ ˙ 0 0 0 0 −φn −φn ⎡

(18.30) (18.31) (18.32)

18 Design, Modeling and Control of an Ankle Rehabilitation Robot

393

Fig. 18.8. Time history of the estimated damping and stiffness values for (a) X Euler angle axis, (b) Y Euler angle axis and (c) Z Euler angle axis obtained through the use of RLS. Solid gray lines represent damping in N m s/rad while dashed black lines represent stiffness in N m/rad.

T  wn = bθ ,n kθ ,n bϕ ,n kϕ ,n bφ ,n kφ ,n P0 = α I Without the use of torque sensors, the external torque acting on the ankle rehabilitation robot was computed by rearranging Eq. (18.6). The measured forces along each individual linear actuator, the nonlinear term as determined by the dynamic model and the link lengths measurements can then be used to estimate the actual external torque as in Eq. (18.33).   τe = −J−T F + M J¨l − JJ˙ −1 J˙l + C(Θ˙ , Θ ) + G(Θ ) (18.33) Note that: Θ¨ = J¨l − JJ˙ −1 J˙l, Θ˙ = J˙l The resulting algorithm was tested through a simulation using Simulink and SimMechanics, where linear damping and stiffness was applied to a gimbal joint which represents the ankle. Using a sinusoidal reference trajectory, the simulation was carried out with a time step of 0.5 ms for one second. The results show that the estimated parameters begin to converge to the true values in about one second. The results of this simulation are shown in Fig. 18.8. A possible extension of the current algorithm is to allow it to recognize the variation in environmental parameters across the task space and hence allow a creation of a nonlinear environmental model. The first reaction was to modify the RLS algorithm above so that it has a smaller forgetting factor to restrict estimation to the most current measurements only. This however does not appear to capture the actual stiffness variation when a nonlinear stiffness profile as discussed in the modeling section is applied to the gimbal ankle joint. This is possibly due to the fact that the forgetting factor has its decaying effect on the time rather than the space/positions at which the measurements were obtained. This research is therefore still investigating suitable methods that can

394

Y.-H. Tsoi, S.Q. Xie, and A.E. Graham

achieve this. Reference [19] has proposed the division of the task space into discrete local segments, with the use of RLS estimation within each segment. Other approaches could involve the use of computational intelligence methods such as artificial neural networks or fuzzy systems.

18.6 Controller Simulation A SimMechanics model has been created in Simulink to represent the parallel robot dynamics. Actuator dynamics have also been implemented in Simulink. The simulation results presented here are based on the use of the biaxial jointed ankle kinematic model, Table 18.2. Kinematic and dynamic parameters used in the simulation

Parameter End effector attachment for link 1, P1 End effector attachment for link 2, P2 End effector attachment for link 3, P3 End effector attachment for link 4, P4 Base plate attachment for link 1, B1 Base plate attachment for link 2, B2 Base plate attachment for link 3, B3 Base plate attachment for link 4, B4 Position vector for ankle joint location, r End effector platform mass, M p End effector platform rotational inertia, I p Position vector of end effector platform center of mass from ankle center,dcg,p Mass for non translating part of actuator, mL1 Mass for translating part of actuator, mL2

Value [0.069 -0.04 0.04]T m [0.078 0.045 0.04]T m [-0.078 0.045 0.04]T m [-0.069 -0.04 0.04]T m [0.141 -0.141 0]T m [0.127 0.127 0]T m [-0.127 0.127 0]T m [-0.141 -0.141 0]T m [0 0 0.36]T ,m 3 kg I px = 0.005kg m2 , I py = 0.005kg m2 , I pz = 0.01kg m2 [0 0 0.09]T m

0.8 kg 0.5 kg IL1x = 0.003kg m2 , Rotational Inertia for non translating part of IL1y = 0.003kg m2 , actuator,IL1 IL1z = 2 × 10−4 kg m2 IL2x = 0.002kg m2 , Rotational Inertia for translating part of actuator, IL2y = 0.002kg m2 , IL2 IL2z = 6 × 10−5kg m2 Distance from center of mass of non translating part of actuator to center of actuator universal 0.1 m joint,d1 Distance from center of mass of translating 0.1 m part of actuator to center of actuator spherical joint,d2

18 Design, Modeling and Control of an Ankle Rehabilitation Robot

395

Fig. 18.9. Time history of angular displacements and angular errors in terms of XYZ Euler angles. X Euler angle values are shown in solid lines, Y Euler angle values are shown in black dashed line and Z Euler angle values are shown in gray dash-dot lines.

Fig. 18.10. Simulation results showing the time history of control voltages, motor currents and actuator forces. Values for actuator 1 are given by gray solid lines, values for actuator 2 are given by black dashed lines, values for actuator 3 are given by black solid lines while values for actuator 4 are given by gray dash-dot lines.

396

Y.-H. Tsoi, S.Q. Xie, and A.E. Graham

Fig. 18.11. Simulation results for the reaction forces as measured at the upper ankle joint. Solid line represents force component along the x Cartesian coordinates, dashed line represents force component along the y Cartesian coordinates and dot-dash line represents force component along z Cartesian coordinates.

while the resistive moments at the ankle were taken to be nonlinear and have the form of inverse sine functions within the workspace. This is implemented using lookup tables in SimMechanics. The kinematic and dynamic parameters used for the simulation are given in Table 18.2 below. To simulate the ankle kinematics, the upper ankle joint was taken to be parallel to the x-axis of the base coordinate frame and coincident with the ankle centre while the subtalar axis was estimated by a vector that is parallel to the vector resulting from the rotation of the y unit vector by −51◦ about the x-axis followed by a rotation of −18◦ about the z-axis (these values were reported as the average axes tilt angles by [21]). The centre of this joint was placed 0.06 m below the centre of the upper ankle joint. As stated above, the resistive moment about the two revolute joints takes the form of the inverse sine functions and they are given in Eq. (18.34) and Eq. (18.35) below, where θ1 is the angular displacement about the upper ankle joint and θ2 is that about the subtalar joint. This simulation was run using the reference trajectories as given in Eq. (18.36) to Eq. (18.38). The variable t is the simulation time. Upper ankle joint: τ1 = −45 arcsin(θ1 /50◦)

(18.34)

Subtalar joint: τ2 = −40 arcsin(θ2 /35◦)

(18.35)

θd = 40π sin(0.9t)/180 ϕd = 20π sin(0.9t)/180 φd = −30π sin(0.9t)/180

(18.36) (18.37) (18.38)

Simulation results are given in Figs. 18.9 to 18.11. The desired manipulator damping was set to 30 N m s/rad while the stiffness was selected as 200 N m/rad so as to produce reasonable actuator forces. Fig. 18.9 gives the angular displacements and angular errors.

18 Design, Modeling and Control of an Ankle Rehabilitation Robot

397

It was observed that impedance control has the ability to trade off positional accuracy for reduced actuator forces, thus also maintaining lower forces and moments at the ankle joint. Fig. 18.10 presents the control voltages, motor currents and the actuator forces. The desired vertical ankle reaction force in this case was set to 0 N but the measured values from the simulation as presented in Fig. 18.11 shows average forces of about 40 N. This can mainly be explained by the mass of the foot and platform which has yet to be included in the redundancy resolution scheme used in Eq. (18.19). Other sources of errors could be caused by imperfect estimation of the angular velocities and accelerations since these quantities were obtained via numerical differentiation and were low pass filtered (with a bandwidth of 5 Hz) to reduce the noise in the signal.

18.7 Conclusions A four link redundant parallel robot has been designed for ankle rehabilitation. The complicated and uncertain kinematics and dynamics of the human ankle and parallel robot present challenging issues in controller design. It is therefore vital that modeling of the robot and environmental characteristics be carried out to facilitate schemes such as computed torque control and environmental parameter estimation. Simulation results have shown that the use of impedance control can reduce the risk of excessive torques being applied at the ankle due to the uncertain kinematical constraints. Observations during controller development also highlighted the importance of kinematic self-calibration and environmental parameter estimation. As it stands, basic schemes were put in place to carry out these tasks. However, it is clear that further extension of existing algorithms will be required to allow more accurate estimation in terms of kinematic and dynamic parameters, which will be required as inputs for suitable impedance adaptation laws. Other future work will also include the development of more advanced impedance control schemes that provide better performance without compromising system stability. More study is also needed to investigate the requirements of different rehabilitation exercises and to identify the influence of these requirements on the controller parameters.

Acknowledgment This work is funded by the Faculty Research Development Fund of the University of Auckland, New Zealand. The work of the first author is supported by the Top Achiever Doctoral Scholarship granted by the Tertiary Education Commission in New Zealand.

References 1. An, C.H., Hollerbach, J.M.: Dynamic stability issues in force control of manipulators. In: IEEE International Conference on Robotics and Automation, pp. 890–896 (1987) 2. Belda, K., Bohm, J., Valasek, M.: State-space generalized predictive control for redundant parallel robots. Mechanics Based Design of Structures and Machines 31(3), 413–432 (2003)

398

Y.-H. Tsoi, S.Q. Xie, and A.E. Graham

3. Besnard, S., Khalil, W.: Identifiable parameters for parallel robots kinematic calibration. In: IEEE International Conference on Robotics & Automation, Seoul, Korea, pp. 2859–2866 (2001) 4. Bharadwaj, K., Sugar, T., Koeneman, J., Koeneman, E.: Design of a robotic gait trainer using spring over muscle actuators for ankle stroke rehabilitation. Journal of Biomechanical Engineering 127, 1009–1013 (2005) 5. Chen, J., Siegler, S., Schneck, C.D.: The three-dimensional kinematics and flexibility characteristics of the human ankle and subtalar joint- Part II: Flexibility characteristics. Journal of Biomechanical Engineering 110, 374–385 (1988) 6. Cheng, H., Yin, Y.K., Li, Z.: Dynamics and control of redundantly actuated parallel manipulators. IEEE/ASME Transactions on Mechatronics 8(4), 483–491 (2003) 7. Colgate, J., Hogan, N.: Robust control of dynamically interacting systems. International Journal of Control 48(1), 65–88 (1988) 8. Deutsch, J., Latonio, J., Burdea, G., Boian, R.: Post-stroke rehabilitation with the rutgers ankle system: A case study. Presence 10(4), 416–430 (2001) 9. Dul, J., Johnson, G.: A kinematic model of the human ankle. Journal of Biomedical Engineering 7, 137–143 (1985) 10. Erol, D., Mallapragada, V., Sarkar, N.: Adaptable force control in robotic rehabilitation. In: IEEE International Workshop on Robots and Human Interactive Communication, pp. 649– 654 (2005) 11. Girone, M., Burdea, G., Bouzit, M., Popescu, V., Deutsch, J.: A stewart platform-based system for ankle telerehabilitation. Autonomous Robots 10, 203–212 (2001) 12. Hesse, S., Schmidt, H., Werner, C., Bardeleben, A.: Upper and lower extremity robotic devices for rehabilitation and for studying motor control. Current Opinion in Neurology 16, 705–710 (2003) 13. Hogan, N.: Impedance control: An approach to manipulation: Part I - Theory. Journal of Dynamic Systems, Measurement, and Control 107(1), 1–7 (1985) 14. Hogan, N.: On the stability of manipulators performing contact tasks. IEEE Journal of Robotics and Automation 4(6), 677–686 (1988) 15. Hogan, N., Buerger, S.: Impedance and interaction control. In: Kurfess, T. (ed.) Robotics and Automation Handbook. CRC Press, New York (2005) 16. Kearney, R., Weiss, P., Morier, R.: System identification of human ankle dynamics: intersubject variability and intrasubject reliability. Clinical Biomechanics 5, 205–217 (1990) 17. Krebs, H., Palazzolo, J., Dipietro, L., Ferraro, M., Krol, J., Rannekleiv, K., Volpe, B., Hogan, N.: Rehabilitation robotics: Performance-based progressive robot-assisted therapy. Autonomous Robots 15, 7–20 (2003a) 18. Krebs, H., Volpe, B., Aisen, M., Hening, W., Adamovich, A., Poizner, H., Subrahmanyan, K., Hogan, N.: Robotic applications in neuromotor rehabilitation. Robotica 21, 3–11 (2003b) 19. Love, L.J., Book, W.J.: Force reflecting teleoperation with adaptive impedance control. IEEE Transactions on Systems, Man, and Cybernetics 34(1), 159–165 (2004) 20. Mattacola, C., Dwyer, M.: Rehabilitation of the ankle after acute sprain or chronic instability. Journal of Athletic Training 37(4), 413–429 (2002) 21. Nester, C.J.: Rearfoot complex: a review of its interdependent components, axis orientation and functional model. The Foot 7, 86–96 (1997) 22. Parenteau, C.S., Viano, D.C., Petit, P.Y.: Biomechanical properties of human cadaveric anklesubtalar joints in quasi-static loading. Journal of Biomechanical Engineering 120, 105–111 (1998) 23. Safran, M.R., Zachazewski, J.E., Benedetti, R.S., Bartolozzi, A.R.I., Mandelbaum, R.: Lateral ankle sprains: a comprehensive review. Part 2: treatment and rehabilitation with an emphasis on the athlete. Medicine & Science in Sports & Exercise 31(7), 438–447 (1999)

18 Design, Modeling and Control of an Ankle Rehabilitation Robot

399

24. Siegler, S., Chen, J., Schneck, C.D.: The three-dimensional kinematics and flexibility characteristics of the human ankle and subtalar joints-Part I: Kinematics. Journal of Biomechanical Engineering 110, 364–373 (1988) 25. Tarokh, M., Bailey, S.: Force tracking with unknown environment parameters using adaptive fuzzy controllers. In: IEEE International Conference on Robotics and Automation, Minneapolis, Minnesota, pp. 270–275 (1996) 26. Wheeler, J.W., Krebs, H.I., Hogan, N.: An ankle robot for a modular gait rehabilitation system. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Sendai, Japan, pp. 1681–1684 (2004) 27. Yoon, J., Ryu, J.: A novel reconfigurable ankle/foot rehabilitation robot. In: IEEE International Conference on Robotics and Automation, Barcelona, Spain, pp. 2290–2295 (2005) 28. Yoon, J., Ryu, J., Lim, K.B.: Reconfigurable ankle rehabilitation robot for various exercises. Journal of Robotic Systems 33, S15–S33 (2006)

19 Markovito: A Flexible and General Service Robot H´ector Hugo Avil´es-Arriaga, L. Enrique Sucar, Eduardo F. Morales, Blanca A. Vargas, Josu´e S´anchez, and Elva Corona National Institute of Astrophysics, Optics and Electronics, Computer Science Department, Luis Enrique Erro 1, 72840 Tonantzintla, M´exico {haviles,esucar,emorales,blanca,elvacx}@inaoep.mx http://ccc.inaoep.mx

Abstract. The development of service robots has recently received considerable attention. Their deployment, however, normally involves a substantial programming effort to develop a particular application. With the incorporation of service robots to daily activities, it is expected that they will require to perform different tasks. Fortunately, many of such applications share common modules such as navigation, localization and human interaction, among others. In this chapter, a general framework to easily develop different applications for service robots is presented. In particular, we have developed a set of general purpose modules for common tasks that can be easily integrated into a distributed, layered architecture, and coordinated by a decision–theoretic planner to perform different tasks. The coordinator is based on a Markov decision process (MDP) whose reward is set according to the task’s goal, the states are represented by a set of variables affected by the general modules, and the actions correspond to the execution of the different modules. In order to create a new application the user only needs to define a new MDP whose solution provides an optimal policy that coordinates the different behaviors for performing the task. The effectiveness of our approach is experimentally demonstrated in four different service robot tasks with very promising results. Additionally, several aspects include some novel ideas; in particular in navigation, localization and gesture recognition.

19.1 Introduction Service robots are mobile robots that help people in different activities, such as helping elderly people in their home, serving as hosts and guides in museums or shopping malls, aiding in hospitals, etc. This idea of service robots to assist humans in every-day life has been around for many years. Although there is much work in developing different abilities for this kind of robots, little attention has been paid to the integration of these behaviors into a complete functional system. Such robots need different capabilities to perform their tasks, such as navigation, mapping, localization and obstacle avoidance to move around in an uncertain and changing environment. They also need clear, simple and natural interactive interfaces to communicate with humans. In general, service robots are developed for executing a particular application, such as guides in a museum, however, it is desirable that a service robot could be capable of performing different tasks; or at least it should be easy to program it for other, similar, applications. Developing applications for service robots requires a considerable effort, however, most 

Currently at IIMAS, UNAM, Mexico.

D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 401–423. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

402

H.H. Avil´es-Arriaga et al.

of them share a core of basic capabilities. It is then natural to develop different modules that can perform such common capabilities and combine them into a general architecture to produce different applications relatively easily. In this chapter we present a service mobile robot called Markovito. Markovito has the following main characteristics: • Generality: it is based on different general–purpose modules for performing common tasks such as planning, navigation, localization, human tracking, object localization, voice interaction, etc. These modules are designed so they can be easily adapted and combined for different tasks. • Flexibility: it is coordinated by a decision–theoretic controller that serves as an orchestra director of different general–purpose modules. New service robot applications can be easily constructed by defining a goal for the task, represented as a Markov decision process (MDP), and solving the MDP that coordinates different modules to perform this task. Using this framework, Markovito has successfully performed different applications in a home environment. In particular, it has solved the tasks of: (i) following a human under user commands, (ii) navigating to several places in the environment designated semantically, (iii) finding one of a set of different objects in a house, and (iv) delivering messages and/or objects between different people. The first three tasks are part of the RoboCup@Home challenge [1]. Experiments have shown that Markovito is reliable in the execution of these tasks, while keeping a simple and intuitive robot software architecture able to incorporate new abilities. This chapter is organized as follows. Section 19.2 describes the software architecture used in Markovito. Section 19.3 describes different general software modules that can be used for different service robot’s applications, namely map building and localization, navigation and planning, speech synthesis and recognition, and visual perception. In Section 19.4, the coordinator based on MDPs is presented. Section 19.5 describes the main implementation issues. Section 19.6 describes the different applications where Markovito has been tested. Finally, conclusions and future research directions are given in Section 19.7.

19.2 Software Architecture Our software architecture is based on a Behavior-based architecture [2]. A behavior is an independent software module that solves a particular problem, such as navigation or face detection. In this chapter, behaviors are also referred to as modules. Behaviors exist at three different levels: Functional level: The lowest level behaviors interact with the robot’s sensors and actuators, relaying commands to the motors or retrieving information from the sensors. Execution level: Middle level modules perform the main functions, such as navigation, localization, speech recognition, etc. These interact with the lowest level through a shared memory mechanism. Each middle level module computes some aspect of the state of the environment. The outputs of these modules are typically reported to the highest level modules.

19 Markovito: A Flexible and General Service Robot

403

Decision level

MDPs Action policy Task Coordinator

State variables Communication via TCP/IP Sockets

Execution level Map building Planning Navigation Localization

Mobility

Speech synthesis and recognition

Face detection Object recognition Gesture recognition

Human−robot interaction Communication via TCP/IP Sockets

Functional level

Robot Server and software libraries

Physical devices

Fig. 19.1. Software architecture

Decision level: The highest level coordinates the middle level modules based on a global planner. The planner consists of a Markov decision process that controls each module based on the goal (defined as a utility function) to perform a task. The software architecture is depicted in Figure 19.1. This architecture can be implemented in a distributed platform, such that each level and each module within a level could be on a different processor. A transparent communication mechanism via TCP/IP sockets permits different configurations without need to modify the modules. Thus, some processing could be done on board the robot (lower level modules) and other off board (high level modules). Also a module can be changed without affecting the rest of the system. The intermediate or execution level compromises the general–purpose modules that perform the main tasks for Markovito. Each module can receive different commands from the coordinator, which correspond to the actions of the MDP. Also, each module affects certain state variables, which are part of the state vector, used by the MDP to decide the best action according to its policy. The following section describe these modules, while Section 19.4 describes the coordinator.

19.3 Modules We have implemented different general-purpose modules that are common to several service robot’s applications, which are described in the following sections.

404

H.H. Avil´es-Arriaga et al.

19.3.1

Map Building and Localization

A mobile robot requires a model or map of its environment to perform tasks. Our map building module combines information from a laser scanner, the odometer and three rings of sonar sensors (8 in each one) [3] to construct an occupancy map based on particle filters. In this approach each particle represents a trajectory followed by the robot and a map associated with that path. The main idea is to represent the required posterior density function by a set of random samples with associated weights, and to compute estimates using those samples and weights. Increasing the number of samples increases the precision and approaches to the optimal Bayesian estimate [4]. To increase efficiency a Rao-Blackwellised particle filter approach [5] is normally used, that essentially marginalizes some variables. The algorithm we have implemented is based on PMAP [6] which we extended: (i) to work on line, (ii) to combine information from the laser and sonar sensors, and (iii) to have some parallel processing to increase efficiency. The ability for mobile robots to locate themselves in an environment is not only a fundamental problem in robotics but also a pre-requisite to many tasks such as navigation. There are two types of localization problems: local and global. Local localization techniques aim to compensate for odometric errors during navigation and require information about the initial position of the robot. Global localization aims to locate the robot’s position without prior knowledge of its current location. These problems are particularly hard in dynamic environments where new obstacles can corrupt the robot’s sensor measurements [7]. In order to locate itself either during navigation or globally, this module uses natural landmarks such as discontinuities, corners and walls as described in [8]. The landmark extraction algorithm proceeds in three stages as follows: i) first, from a discretized laser reading of 180◦, a set of discontinuities –identified by abrupt variations of at least 20cm between the measured distance of two consecutive rays are obtained (see Fig. 19.2), and these discontinuities are used to divide the whole laser range into segments; ii) secondly, corners are identified –following the approach proposed in [9]– by dividing these segments into sub-segments; iii) and finally, the third step takes the remaining sub-segments and performs a fast local Hough transform to detect walls of at least 1m length. The result of this process is a tree-like structure of natural landmarks as depicted in Figure 19.3. One advantage of this approach is that the combination of these three types of landmarks gives the robot the capacity to identify enough information from any position

Fig. 19.2. An example of a discontinuity extracted from laser scanner data. D represents the difference between the measured distance of the two consecutive laser rays.

19 Markovito: A Flexible and General Service Robot

405

Fig. 19.3. Tree of natural landmarks after the extraction of discontinuities, corners and walls

in a wide variety of environments. Another advantage is that these landmarks can be easily extracted from laser scanner data with high accuracy. Moreover, by identifying from the simplest to the more complex features simplifies the segmentation process. Finally, each landmark is associated with a set of distinctive attributes such as its type, distance to the robot, its angle with respect to the robot’s orientation, and in case of discontinuities, depth or the difference in continuous readings that originated the landmark. Given a set of landmarks, a triangulation process is performed between all the visible landmarks to estimate the robot’s position. The information from all the visible landmarks is combined considering the angle between landmarks, the distance between the robot and its farthest landmark, and if there are landmarks at both sides of the robot or only one, to give more accurate estimates (see [10] for more details). For the global localization problem, a ray tracing approach is used to simulate laser readings on the map. Each cell is associated with all its visible landmarks and their values using a field of view of 360◦. This process is performed off-line once a map is constructed. To match a cell with the current readings of the robot –e.g., Figure 19.4a–, an initial stage filters out a large number of candidate positions by taking into account only the number and type of landmarks visible for a given map’s cell. This filter counts the number of landmarks obtained from the laser data that matches the distance, depth, and orientation of the previously stored discontinuities associated with each cell. An example of the result of this process can be seen in Figure 19.4b. In this example dense clusters of points can be interpreted as the most probable positions of the robot. A modified discrete relaxation algorithm is used in a second stage to determine the similarity of each cell with the observations of the robot, considering the distances between the discontinuities in this stage. Our global localization algorithm is able to locate the robot

406

H.H. Avil´es-Arriaga et al.

Fig. 19.4. Global localization a) a simulated environment showing the robot –depicted in green– and some discontinuities, b) results of the initial stage, and c) the estimated position of the robot after the second stage

even with new obstacles as can be seen in Figure 19.4, where five new obstacles are added to the environment [8]. 19.3.2

Navigation

There are different strategies that can be applied for navigation. For instance, we have implemented a navigation module that uses a dynamic programming algorithm to assign costs to each cell of the map; in this approach an exponential cost function is used to value every cell near an obstacle below a distance threshold –i.e., the shorter the distance, the higher the cost assigned to the cell [11]. Following this criterion, the least expensive path is obtained by selecting cells with a lower cost. In order to avoid new obstacles, the robot is sensing its environment while moving. In case a new obstacle is placed in front of the robot the module finds an alternative path, as shown in Figure 19.5 [10]. In this chapter, a novel navigation strategy using machine learning techniques is presented. To illustrate more clearly the motivation behind this work, imaging going to a new place, e.g., a conference site. You would normally ask for directions of places of interest, like the registration desk or a toilet, and you will get general directions, like

19 Markovito: A Flexible and General Service Robot

407

Fig. 19.5. Re-planning with obstructed paths: 1) map of the environment with the original position of the robot and its desired goal, 2) initial least expensive path following our dynamic programming algorithm, 3) obstructed path with new obstacles depicted in blue, and 4) an alternative path to the goal position

“at the end of the aisle to your right” or possibly “in room 203”. You will have then to navigate without collisions in an unknown and dynamic environment to a particular destination point through well known natural marks like walls and doors and expected dynamic conditions, like walking people. Imagine you want your robot to learn how to perform a similar skill. You place your robot in an unknown environment and you want it to navigate to a particular point, like a charging station, but the robot is only given the general direction of its destination point. The robot has to learn how to perform simple skills, like obstacle avoidance and orientation towards a goal, and use them to safely go to a particular goal in a dynamic and unknown environment. This is the approach followed in this chapter for the navigation module. The application of machine learning techniques with expressive representation languages to domains like robotics have received little attention due to the huge amount of low level and noisy data produced by the sensors. The use of relational representations in this area is novel, and their advantages have just recently being addressed (e.g., [12]). In this chapter, first-order logic relations are learned to build the navigation module. This module consist of a set of reactive rules that sense the environment continuously and apply actions whose continuous execution eventually satisfy a goal condition. These rules are known as TOPs (Teleo-Reactive Programs) [13], an effective framework to achieve goals when unexpected events occur. The objective of our learning process is to provide a mobile robot with abilities to move through indoor environments and to accomplish goals. In order to learn TOPs, we combine three machine learning techniques: (i) behavioural cloning, (ii) inductive logic programming (ILP), and (iii) a simple grammar learning algorithm. Behavioural

408

H.H. Avil´es-Arriaga et al.

cloning is a technique to learn skills from examples [14]. The key idea of this method is to show the robot what to do instead of how to do a task. For instance, in order to learn how to navigate without collisions, the robot is presented with human traces steering the robot avoiding obstacles, simplifying the programming effort. We introduced a two phase learning process to learn TOPs for mobile robots in indoor environments: (i) learning of basic TOPs, and (ii) learning of complex TOPs. In the first phase, a system called TOPSY uses human-guided traces and the natural landmark identification process, described in the previous section, to reduce the information from the sensors into a small set of ground predicates (landmarks) suitable for an ILP system called ALEPH [15]. A small set of background knowledge is given to the system from which other predicates are learned and used in the induction of simple TOPs. TOPSY was able to learn the following TOPs from human traces: • avoid(State,Action): to wander around without collisions. Action can take the following values: go forward, turn left, and turn right. • orient(State,Action): Given a target point, the robot has to turn until it is oriented towards the goal, only if it is located in a safe turn zone. Action can take the following values: turn right, turn left and nothing. In order to learn how to combine TOPs to perform more complex tasks (e.g., the goto TOP), previously learned TOPs that apply to states in traces are identified, returning high-level traces of applicable TOPs. The goal is to learn a grammar with TOPs able to reproduce a set of traces. With this purpose, FOSeq (First Order learning from Sequences), an algorithm to learn grammars from sequences based on association rule learning is introduced and applied to induce the goto TOP. This algorithm is in its initial stage, however, the learned TOPs were used for navigation tasks on different environments with dynamic obstacles. Figure 19.6 shows some examples of the Goto TOP under different scenarios. Although the trajectories are not optimal in terms of the distance traveled, the approach is quite robust even in partially unknown and changing environments. See [16] for more details of this work. 19.3.3

Planning

The previous TOPs are used with a probabilistic roadmap module that returns collision free paths. A probabilistic roadmap (PRM) [17, 18] is built using a random generation of points in the configuration space. These points are joined if there is a free path between them and the information is stored in a graph G. Given an initial configuration s and a goal configuration g, the idea for path planning is to connect s and g to nodes s and g in G. Given the random nature of the algorithm, some paths in G may be too long and irregular, so a smoothing algorithm, trying to find shortcuts, is applied. This process is illustrated in Figure 19.7. A combination of the roadmap points with the TOPs produces a robust navigation module. 19.3.4

Speech Synthesis and Recognition

One of our main initial goals was to provide Markovito with the ability to speak and understand Spanish language. Among the advantages of this approach we emphasize the

19 Markovito: A Flexible and General Service Robot

409

possibility to test our robot with a greater number of people with different backgrounds. Markovito is able to synthesize many pre-defined phrases in Spanish. We identified that the robot needs to talk with its user in order to: i) notify when it is ready to initiate an interaction with a user (e.g., receive an instruction), ii) request some aid or information in order to effectively complete the task at hand (e.g., to ask for the name of the current user), and iii) inform when the task has been completed. In this way, many of these phrases are reused for the different tasks performed by Markovito. For some tasks, several sentences with the same meaning were defined and one of them was randomly selected during interaction to produce a more realistic conversation with the user. Speech recognition is based on the Dimex project [19]. This work is aimed to develop general acoustic-phonetic models and pronouncing dictionaries for Mexican Spanish language inside the Sphinx2 framework for continuous and multi-speaker recognition. The speech recognition system works with male and female speakers. Unfortunately, during the initial development cycle, this approach obtained poor results due to several factors: i) people had to talk very close to the Peoplebot robot’s original twinmicrophones in order to be heard by the robot and ii) the recognition step was highly affected by other environmental sounds (e.g., other conversations) and the robot itself (e.g., robot’s sonars). To cope with these problems it was decided to incorporate a directional microphone SHURE model SM81 to the robot. With this microphone a person can talk to the robot at a distance that ranges from 30 cm to 1 m. In order to eliminate some noise frequencies from the sound signal before the recognition process, we filter this signal with a noise removal filter (adapted from the Audacity’s open source code [20]). In order to enhance the signal, this was amplified using an adaptive approach. These simple modifications resulted in a significant improvement in the usability and accuracy of the speech recognition system. 19.3.5

Visual Perception

Service robots must integrate visual abilities such as people detection, recognition of their activities and object recognition in order to interact effectively with people and its environment. In this section, we discuss the visual capabilities implemented in

(a) Round obstacle

(b) Several obstacles

(c) Goto in a corridor

Fig. 19.6. Goto in different scenarios

410

H.H. Avil´es-Arriaga et al.

Fig. 19.7. Probabilistic road maps (PRMs). The top figure shows a simplified diagram of the algorithm with the two main phases: learning and consulting. The bottom figure depicts the 3 stages in the construction of the path from the origin to the goal.

our service robot. These features include face detection, object detection and gesture recognition. 19.3.5.1 Face Detection Face detection is based on the AdaBoost algorithm proposed by [21]. The main idea of this algorithm is the linear combination of classifiers for face features such as eyes, mouth, and nose. Under this scheme, instead of using a single classifier for the whole face, they propose the use of a combination or cascade of simple classifiers. The detection performance of AdaBoost is competitive in comparison to similar approaches [22] while decreasing the required processing time.

19 Markovito: A Flexible and General Service Robot

411

19.3.5.2 Object Recognition Object recognition is carried out using the SIFT or Scale-Invariant Feature Transform algorithm [23]. This algorithm aims to detect and describe distinctive features of objects. As its name says, features are invariant to scale, translation, and rotation. This is due to the extraction process that searches stable features in the image at different scales. The algorithm proceeds as follows. First, the original image is convolved with various Gaussian filters (by varying σ ) at different scales to obtain a set of blurred images grouped by scale. Then, the difference (difference of Gaussians, Do f G) from adjacent blurred images of the same scale is computed. From this new set of DofG images a set of features are extracted. These features can be stored for a future match with features extracted from incoming images to recognize an object or a scene. In Markovito, a data base of objects is created by showing to the robot a particular object, whose distinctive features are obtained and stored associated to the name of the object, which is given by the user (using speech). Later, the robot can be asked to look for any of the objects that were previously shown. 19.3.5.3 Gesture Recognition Gesture recognition is a key element for natural human–robot communication. In our work, we propose to approach this problem by integrating posture and motion information to characterize a set of 9 gestures (Figure 19.8). This scheme allows us to increase recognition rates significantly, even when considering gestures with similar motions. To recognize gestures we propose an alternative model for hidden Markov models (HMMs), that we call dynamic naive Bayesian classifiers (DNBCs) [24]. A DNBC is composed by: i) a set C = {Ct |t = 1, . . . , T }, where each Ct is a random variable that can take one of N possible classes at each time t, and ii) a set A = {At |t = 1, . . . , T },

Fig. 19.8. Gestures considered by our system: a) come, b) attention, c) stop, d) go-right, e) go-left, f) turn-left, g) turn-right, h) waving-hand and i) pointing; j) shows the initial and final position for each gesture

412

H.H. Avil´es-Arriaga et al.

where each At = {At1 , . . . , AtM }, is a set of M instantiated random variables generated by the process at time t. The main difference between the DNBCs and HMMs probability functions is that the attributes are assumed independent given the class or state variable. This factorization: i) enables us to consider techniques to explore statistical relationships among attributes, ii) reduces the number of parameters, so it could also reduce the number of training samples required, and iii) augments the clarity of the graphical representation of the model. Figure 19.9 shows a DNBC unrolled three times with three attributes. Training equations for DNBCs can be derived in a similar manner as it is for HMMs [25]. For recognition, maximum-likelihood (ML) criterion can be used to select the model that maximizes P(A|·). In comparison to HMMs, DNBCs requires less iterations of the EM algorithm for training, while keeping competitive classification rates. To capture gestures we implemented a monocular visual system using the OpenCV libraries for Linux [26]. The system starts with a person standing in a rest position, at a distance of about 3 m. in front of the video camera. Face detection is carried out by using the Adaboost algorithm proposed in [27]. The position of the right-hand and torso regions are estimated using anthropometric measures based on face dimensions. To segment the hand by skin color different lighting conditions of various users, we developed an adaptive scheme by combining a general, Pg(rgb|·), probability distribution constructed off-line, with a personal one, Pp(rgb|·), created on-line by sampling randomly the face and torso of the user. Those functions are combined by means of the next rule: P(rgb|·) = Pg(rgb|·) × Pp(rgb|·). In this way, one pixel is classified as skin (s) if P(rgb|s) > P(rgb|¬s). This strategy allows the visual system to track the hand accurately in several environmental conditions. Once the initial position of the hand is estimated, the CAMSHIFT algorithm [28] is used to track the hand motion over the rest of the image sequence. Our visual system is able to process up to 30 f.p.s. (using an IBM PC Intel Pentium 1.6Ghz, 512Mb RAM), although we sampled attributes at a lower rate. The image resolution is 640 × 480 pixels. See [24] for more details of this work (see [29] for a video of a robot controlled with gestures). Next the top–level coordinator, that directs the different software modules, is described.

C1

C2

A 11 A 21 A 31 A 21 A 22 A 23

C3 A 31 A 32 A 33

Fig. 19.9. Graphical representation of a dynamic naive Bayesian classifier

19 Markovito: A Flexible and General Service Robot

413

19.4 Coordinator 19.4.1

Markov Decision Processes

Markov decision processes (MDPs) have become the semantic model of choice for decision theoretic planning (DTP) in the artificial intelligence community [30]. They are simple for domain experts to specify, or can be learned from data. They have many well studied properties including approximate solution and learning techniques. An MDP is a tuple {S , A , Pr, R}, where S is a finite set of states and A is a finite set of actions. Actions induce stochastic state transitions, with Pr(s, a, s ) denoting the probability with which state s is reached when action a is executed at state s. R(s, a) is a real-valued reward function, associating with each state s and action a. Solving an MDP is finding a mapping from states to actions. Solutions are evaluated based on an optimality criterion such as the expected total reward. An optimal solution is one that achieves the maximum over the optimality measure, while an approximate solution comes to within some bound of the maximum. A solution to an MDP is a policy that maximizes its expected value. For the discounted infinite–horizon case with any given discount factor γ ∈ [0, 1), there is a policy V ∗ that is optimal regardless of the starting state that satisfies the Bellman equation [31]: V ∗ (s) = maxa {R(s, a) + γΣs∈S Pr(a, s, s )V ∗ (s )}

(19.1)

Two popular methods for solving this equation and finding an optimal policy for an MDP are: (a) value iteration and (b) policy iteration [30]. The space and time complexity of MDPs increases with the number of states. This problem can be reduced by using factored representations [32], in which the state is decomposed in a set of variables or factors, and the transition functions is represented using a factored representation (dynamic Bayesian nets). In this chapter, we use a factored representation to specify the MDPs and SPUDD [33] to solve them. SPUDD uses the value iteration algorithm to compute an optimal infinite-horizon policy of action for each state, with expected total discounted reward as the optimality criterion. It uses a representation of MDPs as decision diagrams, and is able to take advantage of structure in the underlying process to make computation more efficient and scalable towards larger environments. The modularity of our system facilitates the representation as a factored MDP, and typically results in a sparsely connected Markov network. Such sparseness leads to very efficient calculations when using a structured solution approach as in SPUDD. 19.4.2

Task Coordination Based on MDPs

To apply MDPs for coordinating a task for Markovito, we have to define the model and then solve it using SPUDD. The model compromises the following elements: • The global state of the system described by a vector S. This is the set of high level variables that are relevant for the task. The coordinator synthesizes this state vector by collecting information from each module. • The set of high level actions, A, required to complete the task. These actions are implemented by calling the different modules with the corresponding parameters.

414

H.H. Avil´es-Arriaga et al.

• The goal of the task defined as a reward function, R, that specifies high positive values for the desired states and negative values for the states that should be avoided. • The transition function Pr(s, a, s ). This model is currently specified manually by the programmer according to the task. In general, it is relatively easy to specify this model, in particular the state variables, actions and rewards. It is more complicated to specify the transitions function; however, once it has been specified for given a task, it is in general similar in other tasks in the same or similar environments. We use an iterative approach to define the model. An initial model is defined and solved with SPUDD. Then we use a simulator to verify the obtained policy, and if there are inconsistent or strange actions, the model is modified and the process is repeated. For the tasks developed so far, it took less than one week to develop each model. Based on the final model, the optimal policy is obtained, and this is used to coordinate all the software modules to perform the task. The model is specified and solved off-line, and just the final policy is used on-line. A simple program, that we call manager, implements the optimal policy (stored in a table), by reading the state vector and selecting the optimal action, which is directed to the corresponding module. We are assuming that Markovito knows with certainty its state, but in reality there is uncertainty, so we might consider instead the belief state. We are currently compressing the probabilistic belief states reported by the modules by choosing the value with maximum likelihood. In the general case, the planners would take advantage of the information contained in the belief state, by using partially observable MDPs (POMDPs). However, it is P-SPACE hard to find policies for POMDPs, calling for approximate techniques for robotics applications [34]. Hierarchical methods are another way to combat the complexity of POMDPs [35, 36, 37]. Markovito considers the state vector to be fully observable, which is a good approximation when the uncertainty in not too high. That is, when the probability of the most probable state is significantly higher than other states (the entropy is low). As future work we propose to extend our approach to POMDPs. The combination of a coordinator based on MDPs with a 3-level architecture can reduce the development costs of different service robots applications. By simply rearranging the modules, changing the goal (reward), and solving a new MDP, different applications can be developed. Different software modules of common tasks can be incorporated in a transparent way. In the following section, we describe Markovito’s hardware and implementation issues.

19.5 Implementation Issues In this section, we describe the hardware and software platform used for Markovito and the general structure followed for its implementation. We believe this kind of explanation is important because in the literature it is common to present the architectural design of the system only, ignoring almost completely the implementation details, which are very useful to developers.

19 Markovito: A Flexible and General Service Robot

415

Fig. 19.10. Markovito

19.5.1

Hardware and Software Platforms

Markovito is based on a PeopleBot robot platform (ActivMedia). It has two rings of sonars, a Laser SICK LMS200, one video camera Canon VCC5, two infrared sensors, a directional microphone SHURE SM81, a gripper, the robot’s internal computer, a Laptop, bumpers and a frame grabber WinTV USB 2 (see Figure 19.10). Table 19.1 summarizes the software libraries and source code considered to develop each module. As it is shown in Figure 19.1 each module was independently implemented. This gave us the facility to: i) test each module separately (even on different computers), ii) in some cases to use simulation (e.g., navigation, localization, map construction and object grasping behaviors), and iii) speed up the development and testing processes. 19.5.2

General Structure and Operation

In our design, each module has assigned a set of state variables and actions that it can modify and execute, respectively. The system starts by running each module first, and lastly the MDP module. Once this has been done, the MDP immediately tries to send the corresponding variables and their values to each module. This operation serves as an initialization step for the variables stored in the modules. When the MDP has received in response an acknowledgment from the modules, then it knows that everything Table 19.1. Modules and libraries used in Markovito

Module Source code/Libraries NAVIGATION Player/Stage server, pmap utility, sw-prolog VISION Player/Stage server, OpenCV, SIFT algorithm DEVICES Player/Stage server SPEECH Sphinx2, Audacity, Listener, Festival Player/Stage server COORDINATION SPUDD (MDP)

416

H.H. Avil´es-Arriaga et al.

is working fine and it is time to send the first action – from the policy obtained off-line with SPUDD.When a module receives an action, it proceeds to execute it. In all cases our implementation design considers that the action is performed on its own thread, enabling the module to receive a new command if it is necessary. In this case, the module can: i) abort the current action and start the new one, ii) queue the second one or, iii) ignore it, if it is desired. This could be useful when considering parallel actions – the same action executed at the same time for two or more modules. The information exchange between modules is carried out only through the MDP. The communication between each module and the MDP is done via client-server tcp/ip sockets. The MDP considers 2 threads per module, one to write to and another to read from each module. The later one selects and sends actions once the value of a new variable has been received from a given module. This enable the MDP to send and receive actions and variables asynchronously. Currently Markovito has two computers, the internal one and a laptop mounted on top the robot. The communication between these two machines is done through an 100Mb/s Ethernet cable that connects the laptop and the robot; in this way we: i) avoid possible problems with wireless networks that are common in practice, and ii) decrease the delay in the communication that is usually slower when using wireless networks. This is important in some contexts and to enhance autonomy for the robot. Moreover, by simply changing the line-command parameters of each module we can run the system on the laptop computer or in the robot’s computer. The complete system was developed using C/C++ language mainly (except for the TOPs rules that were programmed in Prolog). A video that shows the system running can be found at [38].

19.6 Applications This section presents the different tasks Markovito is able to perform. These tasks are based on those proposed for the RoboCup@Home 2007 competition [1]. As mentioned in Section 19.4, the different modules are coordinated with an MDP. 19.6.1

Navigate through Three Places and Return to the Starting Position

In this task, Markovito has to navigate safely through three places, that are given by a user at the beginning of the interaction, and then return to its starting position. As soon as the robot reaches a destination point it has to announce it. All the interaction is made in natural language. Six variables were defined for this MDP: localize, indicates when the robot’s position is known; get goals, it is true when the robot has obtained the three places to visit; arrived at destination point, is set when the robot arrives to each place; has trajectory, indicates when it has a plan for navigation; end position, indicates when it returns to the starting position. The MDP has six actions to coordinate the modules which are described in Table 19.2. Figure 19.11 shows a sequence of Markovito navigating through the environment reaching different destination points.

19 Markovito: A Flexible and General Service Robot

417

Table 19.2. The six actions used to coordinate the modules for the navigation task

Action

Module

Description

Localize

Navigation Global localization

Wait for a goals

Speech

Generate trajectory

Navigation Give a path to a destination place

Go next goal

Navigation Navigate to reach the next place

Announce arrival at ith. goal Speech

Obtain the places to visit

Announce that the robot has reached a new place

Announce finish

Speech

Announce that the robot has returned to the initial position

Fig. 19.11. Markovito navigating through the environment reaching different destination points at the Mexican RoboCup@Home 2007 competition

19.6.2

Lost and Found

In this task, Markovito is shown an object which it later has to search for in the environment. The user places several objects, one by one, in front of the robot’s camera. Afterward, the user tells the robot which object to search for and a destination area. Markovito has to navigate to the indicated area and search for the object. It tells using speech synthesis when the object is recognized. Markovito’s first action is to find its global position in the environment. Once this has been done, the robot notifies that it is ready to accept a user instruction. We considered three different commands: i) learn an object, ii) search an object and iii) forget an object. If the robot receives a command to search or forget an object before knowing it, a notification is given to the user. We included the forget instruction just in case of any mistake when positioning the object in front of the robot’s camera. Once at least one object has been presented to the robot, the user can instruct the robot to search for the

418

H.H. Avil´es-Arriaga et al.

Fig. 19.12. Markovito learning to recognize an object (left) and later searching for it (right)

object. This instruction can be complemented with the name of a place in the environment (e.g., kitchen, table, etc.) where the robot can go to search the object. When the robot has reached the given place, it starts to pan and tilt its camera to capture images of the environment. These images are processed using the SIFT algorithm to detect the object. This MDP has 11 variables: localize, has order, forget object, confirm forgetting, learn object, confirm learning, object in data base, found, reached destiny, is near, and confirm object. The actions consider in this MDP are: global localization, wait for order, forget object, learn object, confirm learn object, confirm forget object, go to search area, look for object, get close to object, and confirm found object. All the interactions in this task are in natural language. This task uses the localization, navigation, planning, speech and perception modules. Figure 19.12 shows Markovito learning an object, in this case a Teddy bear (left) and then searching for it (right). 19.6.3

Follow a Person under User Request

For this task, Markovito has to follow a human through an unknown track in a homelike environment. After reaching the end position, the robot has to return to its starting position. The task consisted of two stages. In the first stage, the human stands in front of the robot at a distance of one meter for about one minute for calibration. At this state, the torso detection module learns a color model of the persons cloths, which is later used for tracking. In a second stage, the human starts walking towards the end position, passing through a number of places. This task used the navigation module based on TOPs to go dynamically to changing places given by the torso tracking module. This MDP has 13 variables: localize, person, calibrate order, follow order, stop order, calibrate, follow, searching, announce following, announce searching, announce finish, has trajectory, and get destination. All the interaction is also in natural language. Table 19.3 shows the actions that are used to coordinate the different modules. 19.6.4

Deliver Messages and Objects between People

Markovito is able to act as a messenger robot to deliver spoken messages, objects, or both under a user’s request. Markovito first obtains its global position and orientation

19 Markovito: A Flexible and General Service Robot

419

Table 19.3. The actions used in the task to follow a person and the different modules used in this task

Action

Module

Description

Localize

Navigation Global localization

Wait order to calibrate

Speech

Wait for a calibration order

Calibrate

Vision

Get information about the person

Wait order to follow

Speech

Follow

Navigation Follow a person

Announce searching person Speech

Wait for a follow order Announce that the robot lost the person

Search the person

Vision

Announce found person

Speech

Search for a person Announce that the robot found the person

Wait order to stop

Speech

Wait for a stop order

Generate trajectory

Navigation Give a path to destination place

Return

Navigation Navigate to the starting position

Announce arrive

Speech

Announce arriving to the starting position

(x, y, θ ) relative to its environment –it is assumed that a map was previously built as described in Section 19.3.1 and uses natural landmarks for its global localization. Once it has found its global position, Markovito waits for the spoken salute of a user and when it arrives, the interaction starts. At this stage the robot asks the name of the sender and receiver, and requests the message or object, or both. Then, the robot proceeds to navigate to the receiver’s position – assumed known in advance by the robot. At this place Markovito tries to detect the face of a user. If a face is detected, Markovito delivers the message or the object and resets the state variables to their default values waiting for a new request; if not, it continues searching for the receiver until certain time limit is reached. An MDP with 12 binary variables was used for this task: localized, greetings, sender, receiver, message, object, trajectory, reached destiny, found receiver, object/message delivered, message, and object. Two multi-valued variables were also used for the battery level and for the type of delivery. Thirteen actions were defined for this task: localize globally, generate path to destination place, execute trajectory, as part of the navigation module. Wait for greeting, request type of delivery, request sender’s name, request receiver’s name, record message, deliver message, confirm delivery, as part of

420

H.H. Avil´es-Arriaga et al.

Fig. 19.13. Markovito delivering a beer

the speech module. Grasp object, release object as part of the delivery module. Detecting a person is part of the vision module. Figure 19.13 shows Markovito delivering a beer to a person watching T.V.

19.7 Conclusions and Future Work There is an increasing interest for the development of service robots. Having a service robot to perform even a simple task, normally involves a substantial programming effort as different capabilities, such as navigation, localization and human interaction, need to be developed. Fortunately many of these capabilities are common to different tasks so they can be reused for new applications. This chapter described a general framework for creating service robots applications based on the reuse of general purpose modules. We have implemented different modules, some of which include novel ideas, like natural landmarks based on discontinuities for localization, machine learning techniques for inducing TOPs for navigation, and dynamic na¨ıve Bayesian classifiers for gesture recognition. The architecture uses an MDP framework to coordinate in a simple and effective way these modules. It is common in robotics to create new algorithms for a particular module or to describe a single application involving several modules. Although there are previous works on robot coordination based on MDPs, these consider an specific task. Contrary to previous approaches, this chapter shows how different service robot applications can be easily constructed using a set of modules and defining a new MDP to obtain an optimal policy for each task. We have illustrated the capabilities of this framework with different applications. There are several research directions to follow. In particular, we plan to incorporate a module that can be used to teach the robot by example, to incorporate a face recognition module, and to combine gesture recognition with the natural language module to enhance our human-robot interface. We are also improving the coordinator to deal with conflicting situations. Finally, we plan to use this framework for developing other applications, such as a tour guide in a museum or university, enhance our messenger robot with new capabilities, and develop an assistant in a social event.

19 Markovito: A Flexible and General Service Robot

421

Acknowledgments The work reported in this chapter was partially supported by CONACyT Grant No. 47968. We appreciate the constributions of several students in the group, in particular, Apolinar Ram´ırez, M´onica Rodr´ıguez and Orlando G´omez. The authors are grateful to the anonymous reviewers for their constructive comments on a previous draft of this chapter.

References 1. RoboCup, The robocup@home webpage (2007) (last retrieved February 9, 2008), http://www.ai.rug.nl/robocupathome/ 2. Arkin, R.: Behavior-based Robotics. MIT Press, Cambridge (1998) 3. J´aquez, V.: Construcci´on de mapas y localizaci´on simult´anea con robots m´oviles. Master’s thesis, Instituto Tecnol´ogico y de Estudios Superiores de Monterrey - Cuernavaca (2005) 4. Maskell, S., Gordon, N.: A tutorial on particle filters for on-line nonlinaer/non-gaussian bayesian tracking. In: IEEE Colloquim on Tracking (2002) (Last retrieved February 9, 2008), http://citeseer.ist.psu.edu/504843.html 5. Murphy, K., Russell, S.: Rao-blackwellized particle filtering for dynamic bayesian networks. In: Doucet, A., de Freitas, N., Gordon, N. (eds.) Sequential Monte Carlo Methods in Practice, pp. 499–517. Springer, Heidelberg (2001) 6. Howard, A.: Simple mapping utilities (pmap) (last retrieved February 9, 2008), http://robotics.usc.edu/∼ahoward/pmap/index.html 7. Burgard, W., Cremers, A., Fox, D., Hahnel, D., Lakemeyer, G., Schulz, D., Steiner, W., Thrun, S.: The interactive museum tour-guide robot. In: Proceedings of the Fifteenth National Conference on Artificial Intelligence (AAAI 1998), Madison, Wisconsin, pp. 11–18 (July 1998) 8. Hern´andez, S., Morales, E.: Global localization of mobile robots for indoor environments using natural landmarks. In: Proceedings IEEE Conference on Robotics, Automation and Mechatronics (RAM 2006), pp. 1–6 (2006) 9. Xavier, J., Pacheco, M., Castro, D., Ruano, A.: Fast line, arc/circle and leg detection from laser scan data in a plater driver. In: Proceedings of the International Conference on Robotics and Automation (ICRA 2004), pp. 3930–3935 (2004) 10. Hern´andez, S.: Navegaci´on de un robot m´ovil en ambientes interiores usando marcas naturales del ambiente. Master’s thesis, Instituto Tecnol´ogico y de Estudios Superiores de Monterrey - Cuernavaca (2005) 11. Romero, L., Morales, E., Sucar, L.: Exploration and navigation for mobile robots with perceptual limitations. International Journal of Advanced Robotic Systems 3(3), 249–258 (2006) 12. Cocora, A., Kersting, K., Plagemann, C., Burgard, W., De Raedt, L.: Learning relational navigation policies. In: Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Beijing, China, pp. 2792–2797 (2006) 13. Benson, S., Nilsson, N.: Reacting, planning, and learning in an autonomous agent. Machine Intelligence 14, 29–62 (1995) 14. Michie, D., Sammut, C.: Behavioral clones and cognitive skill models. Machine Intelligence 14, 395–404 (1995)

422

H.H. Avil´es-Arriaga et al.

15. Srinivasan, A.: The aleph 4 manual: version 4 and above (2005) (last retrieved Februay 9, 2008), http://web.comlab.ox.ac.uk/oucl/research/areas/machlearn/ Aleph/aleph.html 16. Vargas, B., Morales, E., Hern´andez, S.: Learning navigation teleo-operators with behavioural cloning. In: Memoria T´ecnica del Octavo Encuentro de Investigaci´on (Instituto Nacional ´ de Astrof´ısica, Optica y Electr´onica (INAOE), Tonantzintla Puebla, M´exico), pp. 209–212 (2007) 17. Barraquand, J., Latombe, J.C.: Robot motion planning: A distributed representation approach. Journal of Robotics Research 10(6), 628–649 (1991) 18. Latombe, J.C.: Robot motion planning. Kluwer Academics Publishers, Dordrecht (1991) 19. Pineda, A., Villase˜nor, L., Cu´etara, J., Castellanos, H., L´opez, I.: DIMEx100: A new phonetic and speech corpus for mexican spanish. In: Lemaˆıtre, C., Reyes, C.A., Gonz´alez, J.A. (eds.) IBERAMIA 2004. LNCS, vol. 3315, pp. 974–983. Springer, Heidelberg (2004) 20. Audacity, The free, cross-platform sound editor 21. Voila, P., Jones, M.: Robust real-time object detection. International Journal of Computer Vision, 137–154 (2001) 22. Rowley, H., Baluja, S., Kanade, T.: Human face detection in visual scenes. Tech. Rep. CMUCS-95-158R, School of Computer Science, Carnegie Mellon University (1995) 23. Lowe, D.: Distint image features from scale-invariant keypoints. International Journal of Computer Vision 60(2), 91–110 (2004) 24. Avil´es-Arriaga, H.H., Sucar, L., Mendoza, C.: Visual recognition of similar gestures. In: Proc. IEEE Inter. Conf. on Pattern Recognition (ICPR), China, pp. 1100–1103 (August 2006) 25. Rabiner, L.: Tutorial on hidden markov models and selected applications in speech recognition. In: Readings in Speech Recognition, pp. 267–296. Morgan Kaufmann Publishers, San Francisco (1990) 26. Community, O.: Open source computer vision library (Last retrieved March 3, 2008), http://www.intel.com/technology/computing/opencv/index.htm 27. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2001), pp. 511–518 (2001) 28. Bradski, G.: Real time face and object tracking as a component of a perceptual user interface. In: Society, I.C. (ed.) Proceedings of the 4th IEEE Workshop on Applications of Computer Vision (WACV 1998), pp. 214–219 (1998) 29. Avil´es-Arriaga, H.H., Sucar, L.: Video: Teleoperation of a mobile robot using gestures (2002) (last retrieved February 9, 2008), http://www.youtube.com/watch?v=opAUo0zJHGY 30. Puterman, M.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, New York (1994) 31. Bellman, R.: Dynamic Programming. Princeton University Press, Princeton (1957) 32. Kaelbling, L., Littman, M., Cassandra, A.: Planning and acting in partially observable stochastic domains. Artificial Intelligence 101(1-2), 99–134 (1998) 33. Hoey, J., St-Aubin, R., Hu, A., Boutilier, C.: Spudd: Stochastic planning using decision diagrams. In: Proc. of the 15th Conf. on Uncertainty in AI, UAI 1999, pp. 279–288 (1999) 34. Montemerlo, M., Pineau, J., Roy, N., Thrun, S., Verma, V.: Experiences with a mobile robotic guide for the elderly. In: Proceedings of the AAAI National Conference on Artificial Intelligence (AAAI 2002), Edmonton, Canada, pp. 587–592 (2002) 35. Simmons, R., Koenig, S.: Probabilistic robot navigation in partially observable environments. In: Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence (IJCAI 1995), Montreal, Canada, pp. 1080–1087 (1995)

19 Markovito: A Flexible and General Service Robot

423

36. Theocharous, G., Rohanimanesh, K., Mahadevan, S.: Learning hierarchical partially observable Markov decision process models for robot navigation. In: Proc. of the IEEE International Conference on Robotics & Automation (ICRA 2001), Seoul, Korea, pp. 511–516 (May 2001) 37. Pineau, J., Gordon, G., Thrun, S.: Policy-contingent abstraction for robust robot control. In: Proceedings of Uncertainty in Artificial Intelligence (UAI), Acapulco, Mexico, pp. 477–484 (August 2003) 38. Avil´es-Arriaga, H.H., Corona, E., Sucar, L., Morales, E., Hern´andez, S., J´aquez, V.: Video: A service robot named markovito. In: AAAI 2007 AI Video Competition (http://aivideo.org/). Organizers: Thrun, S., Aha, D.: (2007) (Last retrieved February 9, 2008), http://videolectures.net/aaai07 aviles srm/

20 Task Based Hybrid Closure Grasping Optimization for Autonomous Robot Hand Tetsuyou Watanabe1 , ZhongWei Jiang2 , and Tsuneo Yoshikawa3 1

2 3

Graduate School of Natural Science & Technology, Kanazawa University, Kanazawa, 920-1192, Japan [email protected] Graduate School of Science and Engineering, Yamaguchi University, Ube, 755-8611, Japan [email protected] College of Information Science and Engineering, Ritsumeikan University, Kusatsu, 525-8577, Japan [email protected]

Abstract. Multi-fingered/robot hand is usually equipped with robots in order to perform some tasks, such as, lifting up an object on a table and putting it down on another table; putting an object into a hole; assembling some objects. To perform such tasks, the multi-fingered hand with functions of grasping and manipulating is generally requested for practical use. In order to design the desirable grasping which can perform such a given task, the concept of hybrid closure considering both the contact positions and finger configuration is introduced in this chapter. Based on this concept, the fundamental theory and algorithm on how to make the configuration of fingers and how to determine the contact positions as the fingers are grasping and manipulating an object are described in detail. This chapter consists of 9 sections. In Section 20.1, we first make a brief introduction of the researches on grasp analysis and performance of robot fingers. Further we describe some basic concepts and properties for grasping, especially the importance of the hybrid closure with considering both the contact positions and finger configuration. In Section 20.2, the target system handled in this chapter is shown and the kinematics of the system and the frictional constraint required for maintaining contact is described in detail. Section 20.3 shows the important properties of a hybrid closure grasp. In Section 20.4 the dynamic relationship of the hand system is derived. Section 20.5 describes the concept of required external force set which is important for designing a grasps to perform a given task. Based on the concepts of hybrid closure and required external force set, we define required acceleration and equilibrium-force sets. In Section 20.6, we formulate an optimization problem to design the desirable grasp to perform a given task, using the defined required acceleration and equilibrium-force sets. The algorithm for solving the optimization problem is explained in detail in Section 20.7, and the some numerical examples are demonstrated in Section 20.8. Section 20.9 is conclusion of this chapter.

20.1 Grasping by a Robot Hand When grasping an object, the fingers have to contact with the object. Therefore, to analyze grasping performance, a contact model has to be introduced. Here the fundamental contact model is described first in following. Contact model: Assuming that all contacts, including line and surface contacts, can be approximated by finite point contacts, the following classification can be made [1]: D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 425–451. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

426

T. Watanabe, Z. Jiang, and T. Yoshikawa

Point contact without friction: The finger applies only a normal force to an object at a contact point Point contact with friction: The finger applies any force satisfying frictional condition to an object at a contact point Soft-finger contact: The finger applies a translational force and a moment around the contact normal to an object at a contact point, which satisfy frictional condition. Grasping can be defined as “To restrain an object and to let it be stationary state.” Therefore, it is important to investigate a way to balance external forces and moments. Some important properties of grasping related with this problem have been analyzed. Properties of grasping: Commonly used properties of grasping can be classified as [1] Equilibrium (grasp), Force closure[2], and Form closure[3]. When some external force and moment w e is applied to a grasped object, the grasped object is in equilibrium if the contact force, f , satisfies the frictional condition at each contact point and the following equation holds G f + we = o where G denotes the grasp matrix which represents the relationship between the contact force, and the resultant force and moment applied to the object by the robot hand. Note that we describe more detail about this relationship later. A grasp is said to be of the force closure type if a grasped object can be in equilibrium for any external force and moment w e in any arbitrary direction. A grasp is said to be of the form closure type, if the motion of a grasped object is completely constrained, irrespective of the magnitude of the contact force, even when any arbitrary external force and moment w e is applied to the object. If a grasp is of the force closure type with frictionless contacts and any external force and moment with arbitrary magnitude can be balanced in the grasp, then the grasp is also of form closure type. Therefore, form closure is a stronger type of closure than force closure. For achieving form closure, a 2 dimensional grasp needs at least 4 contact points and a 3 dimensional grasp needs at least 7 contact points. Whether the above properties can be achieved depends on contact positions, number of contact points, direction of contact normal, and frictional conditions at the contact points. Now, if paying attention to the contact positions, there are an infinite number of the contact positions where we can achieve one of the above properties. Then, which contact positions should be chosen among such contact positions is a very important problem in design of grasp. Many researchers have investigated optimal grasp points. Omata [4, 5] investigated the regions of contact points on a convex polyhedral object for balancing gravitational force. Trincle et al. [6] considered where contact points should be located, in the case of lifting up an object on a table by 2 fingers without friction. Nguyen [2] developed a method for investigating the regions of contact positions for achieving force closure. Chen et al. [7] presented a graphical method for investigating optimal contact positions when a 2-D/3-D object is grasped by 3/4 fingers, using some grasp qualities: robustness, manipulability, and task compatibility. Chen et al. [8] investigated optimal contact positions for 2-finger grasps, based on the concept of antipodal points and using a grasping energy function. Ponce et al. [9] showed that force closure by 4 fingers can be classified into concurrent grasp, pencil grasp and regulus grasp. They also developed a

20 Task Based Hybrid Closure Grasping Optimization for Autonomous Robot Hand

(a)

(b)

427

(c)

Fig. 20.1. Comparison of the directions in which accelerations are generable with the directions in which forces/moments are generable; (a) precision/fingertip grasp, (b) intermediate grasp between precision/fingertip and power grasp, and (c) power grasp. Note that in power grasp, any external force in any arbitrary direction can be balanced without changing joint torques.

method for investigating the regions of contact positions for achieving the concurrent grasp. Ding at al. [10] investigated the eligible set of fixturing surfaces on a polyhedral workpiece for achieving form closure at first and then investigated the optimal fixturing positions on the surfaces contained in the eligible set, which minimize the locating error of the workpiece (Note that the form closure of Ding at al. implies that arbitrary forces and moments exerted on a grasped object can be balanced by contact forces applied by fingers). Markenscoff et al. [11] investigated the grasp points which minimize the magnitudes of the contact forces needed to balance an external force, when a polygonal object is grasped by 3 or 4 fingers. Mirtich et al. [12] investigated the grasp points which minimize the magnitudes of the contact forces needed to balance external forces and moments, taking into account the difference between the units of force and moment. Mantriota and his colleagues [13] ([14]) investigated the grasp points which minimize the magnitude of the contact forces (the coefficient of friction) needed to balance external forces and moments, using generalized disturbance force and torque. Wang et al.[15] investigated optimal fixturing points, using the concept of point-set domain. In this research, they presented some efficient algorithms for planning, and evaluated the results based on some performance criteria. Watanabe et al.[16] investigated the contact positions which minimize the magnitudes of the contact forces needed to balance external forces and moments, using the concept of required external force set. How about the case considering not only contact positions but also finger configuration? Are there any difference between the case considering only contact positions, and the case considering both contact positions and finger configuration? If considering finger configuration, we can characterize whether contact force is applied to the object actively or passively. The contact force applied actively is associated with applying acceleration. The contact force applied passively means the force acts as a reaction to external force due to the mechanism of geometrical constraints. There is no description about “active” and “passive” in the definition of force closure. Namely, in force closure grasps, “active” and “passive” do not need to be distinguished. Therefore, in the synthesis of force closure grasps, there are no difference between the case considering only contact positions, and the case considering both contact positions and finger configuration. However, in actual system, the “active” force works to move the object while the “passive” force does not. Consider the case shown in Fig.20.1, for example. Fig.20.1 (a) shows the case where not only an acceleration but also a force/moment in every direction is generable. Fig.20.1 (b) shows the case where in some directions,

428

T. Watanabe, Z. Jiang, and T. Yoshikawa

both an acceleration and a force/moment might be generated, but in the other special directions only a balancing force/moment is generable. The arrow in Fig.20.1 (b) shows the direction where only a balancing force/moment is generable. Fig.20.1 (c) shows the case where a balancing force/moment in every direction is generable, but an acceleration is not generable in any directions. Therefore, “active” and “passive” forces should be distinguished for solving the practical grasping problem. The “active” and “passive” force closure grasps are defined as follows [17]: Active force closure any arbitrary acceleration and angular acceleration of a grasped object can be generated by joint actuating torques Passive force closure if external force/moment in any direction is exerted on a grasped object, we can resist the external force/moment and restrain the motion of the object without changing joint torques It is evident that in most grasping systems (see Fig.20.1 (b)), both the active and the passive force closure properties have to be taken into account. We call such grasps hybrid active/passive closure. Accordingly, to consider not only contact positions but also finger configuration in synthesis of grasping means to consider the hybrid closure grasp. Previously, the active and the passive force closures were studied separately. For active force closure, Li et al.[18], Cole et al.[19], and Yokokohji et al.[20] presented control algorithms for the case of point contact, rolling contact, and soft-finger contact, respectively, for the manipulation of an object grasped by the fingertips. Cole et al.[21] and Zheng et al.[22] extended the algorithm to the case of a sliding contact. Harada et al. analyzed the active force closure[23] and presented a control algorithm[24] for manipulating multiple objects. As for researches focusing on active force closure in general grasping systems, Trinkle et al. [25] discussed a grasp planning for manipulating an enveloped object with sliding contacts in 2-dimensional space. Bicchi et al. [26] analyzed manipulability of general grasping systems. Harada et al.[27] proposed the concept of Envelope Family and presented a sufficient condition for the manipulation of an object in that system. Park et al.[28] derived contact forces and accelerations consistent with dynamics and friction law for a given torque-wrench pair. On the other hand, the passive force closure (power grasp) has been researched with respect to the analysis of robustness, indeterminate forces, the formulation of contact force distribution, and the optimization of pre-loaded joint torques, for example [29, 30, 31, 32, 33, 34, 35, 36, 37]. There are many cases where a grasping system has both active and passive forceclosure properties simultaneously. However, analyses of this type of grasps are going on. Here, we focus on how to design such a type of grasp, and analyze it from the viewpoint of optimizing finger configuration and contact positions.

20.2 Target System and Kinematics 20.2.1

Target System

The target system described in this chapter is shown in Fig.20.2. An arbitrary shaped rigid object is supposed to be grasped by N fingers of a robot hand. Several assumptions for analysis of the grasping performance are made as follows:

20 Task Based Hybrid Closure Grasping Optimization for Autonomous Robot Hand

429

ΣR

ΣR

pC21 ΣC21

pO ΣO

pO

Object 2nd finger

1st finger

pCij

o

Base

ΣO Fig. 20.2. Target System (N = 2)

pCij ΣCij

Fig. 20.3. Relation between the pCi j , p o and R o

1) Each finger makes a frictional point contact with the object, and sliding doesn’t occur at any contact points. 2) There exists at most one contact point on each link of the fingers. 3) The number of contact points does not change and the remove of contact points does not occur. 4) The finger configuration is uniquely determined by its contact and base positions (finger has no redundancy). 20.2.2

Nomenclature

col : A column vector or matrix formed by the following elements. For example, col[xi ] means (x1 x2 · · · xn )T . diag : A block diagonal matrix of the following elements. Mi : Number of joints of the ith finger (i = 1, 2, · · · , N). Li : Number of contact points on the ith finger. N M ). M : Number of total joints (= Σi=1 i N L). L : Number of total contact points (= Σi=1 i D : 3/6 in 2/3 dimensional space. d : 2/3 in 2/3 dimensional space. ΣR : Reference coordinate frame. ΣO : Object coordinate frame fixed at the object. ΣCi j : Coordinate frame fixed at the contact point Ci j ( j=1, · · · , Li , i = 1, · · · , N). qi j : The jth joint angle of the ith finger ( j = 1, · · · , Mi , i = 1, · · · , N). q i : = (qi1 qi2 · · · qiMi )T (∈ R Mi ). p o : Position of the origin of ΣO (∈ R d ). R o : Rotation matrix[41] representing posture of ΣO (∈ R d×d ). ω o : Angular velocity of ΣO (∈ R d ). r˙ : = ( p˙ To ω To )T (∈ R D ). (qq : = col (qqi ) (∈ R M ).) pCi j : Position of the origin of ΣCi j (∈ R d ). op d Ci j : pCi j with respect to Σ O (∈ R ). Ld pC : = col (ppCi j ) (∈ R ).

430

T. Watanabe, Z. Jiang, and T. Yoshikawa

w : Resultant force and moment applied to the object (∈ R D ). f i j : Contact force vector (∈ R d ). f i : = ( f i1 f i2 · · · f iLi )T (∈ R Li d ) f : = col ( f i ) (∈ R Ld ) τi j : The jth joint torque of the ith finger ( j = 1, · · · , Mi , i = 1, · · · , N). τ i : = (τi1 τi2 · · · τiMi )T (∈ R Mi ). τ : = col (τ i ) (∈ R M ). τ ci : The joint torques equivalent to f i (∈ R Mi ). τ c : = col (τ ci ) (∈ R M ). 20.2.3

Kinematic Constraints

The relation between the pCi j , p o and R o shown in Fig.20.3 are given by pCi j = p o + Ro o pCi j .

(20.1)

By differentiating this equation with respect to time, we get Ro o pCi j ) = G Tij r˙ p˙ Ci j = p˙ o + ω o × (R

(20.2)

where G i j denotes  Gi j =

   I I = Ro o pCi j )×] [(R [(ppCi j − p o )×]

in 3 dimensional space, where I represents an identify matrix, [aa ×] represents a skew symmetric matrix equivalent to the cross product operation ([ a × ]bb = a × b ). In 2 dimensional space, ⎛ ⎞ 1 0 ⎠ 0 1 Gi j = ⎝ −(pCi j y − poy ) (pCi j x − pox ) where pCi j x and pCi j y represent the x and y components of pCi j respectively, and pox and poy represent the x and y components of p o respectively. The relation between the pCi j and q i are given by pCi j = pCi j (qq i ). By differentiating this equation with respect to time, we get p˙ Ci j = J i j q˙ i . where J i j = ∂ pCi j /∂ q Ti ∈ R d×Mi denotes the Jacobian matrix.

(20.3)

20 Task Based Hybrid Closure Grasping Optimization for Autonomous Robot Hand

431

By using the following vectors and matrices  T q˙ = q˙ T1 q˙ T2 · · · q˙ TN ∈ R M , ⎞⎛ ⎞ ⎞⎞ ⎛⎛ ⎛ J 21 J 11 J N1 ⎟⎜ ⎟ ⎟⎟ ⎜⎜ ⎜ J = diag ⎝ ⎝ ... ⎠ ⎝ ... ⎠ · · · ⎝ ... ⎠ ⎠ ∈ R Ld×M , J 1L1 J 2L2 J NLN  D×Ld , G = G 11 G 12 · · · G NLN ∈ R from Eq. (20.2) and Eq. (20.3), we get G T r˙ = J q˙ = p˙ C .

(20.4)

Differentiating both sides of this equation with respect to time, we get ˙ T r˙ = J q¨ + J˙ q˙ = p¨ C . G T r¨ + G 20.2.4

(20.5)

Statics

From Eq. (20.4) , the relation of virtual displacements of r , q and pC is G T Δ r = J Δ q = Δ pC . Then, from the principle of virtual work, we get (Δ q )T τ c = (Δ pC )T f = (Δ q )T J T f ,

(Δ pC )T f = (Δ r )T G f = (Δ r )T w .

Hence,

τc = JT f , 20.2.5

G f = w.

(20.6)

Frictional Constraint

We assume friction obeys Coulomb’s law. Then, the frictional constraint at the jth contact point on the ith finger is represented by F f i j = { f i j ||tt fi j | ≤ μi j n fi j , n fi j ≥ 0}

(20.7)

where μi j denotes the frictional coefficient at the contact point, n fi j denotes the normal force component of the contact force f i j , t fi j = (t fi j ,1 t fi j ,2 )T in 3 dimensional space and (t fi j ,1 ) in 2 dimensional space, and t fi j ,1 and t fi j ,2 denote the tangential force components of the contact force f i j .

20.3 Mechanics of Hybrid Closure First, we consider object velocity in a hybrid closure grasp. Using  GT ∈ R Ld×(M+D) , A = J −G

432

T. Watanabe, Z. Jiang, and T. Yoshikawa

y

y x

x

(a)

(b)

Fig. 20.4. Examples of hybrid (active/passive) closure

from Eq. (20.4), we get

  q˙ = o. A r˙

By solving Eq. (20.8), we obtain     q˙ Λq ˙ = Λ ζ˙ = ζ r˙ Λr

(20.8)

(20.9)

where Λ ∈ R (M+D)×a (Λ q ∈ R M×a and Λ r ∈ R D×a ) denotes an orthogonal matrix whose columns form bases of the null space of A , ζ˙ ∈ R a is an arbitrary vector expressing the magnitude of each column of Λ . Differentiating both sides of Eq. (20.10) with respect to time, we get   q¨ = Λ ζ¨ + Λ˙ ζ˙ . r¨ If considering quasi-static motion where velocities are almost zero, This equation becomes     q˙ Λq ¨ ¨ =Λζ = ζ. (20.10) r˙ Λr

Λ corresponds to the available directions of the object and finger motions within the constraint applied by the fingers, and ζ¨ corresponds to the object and finger motions within the constraint applied by the fingers. In a hybrid closure grasp, object velocity in any direction can not always be generated. Consider the case shown in Fig.20.4. Frictional constrains are assumed to be satisfied. Let the length of each link of the fingers be 1. Note that in Fig.20.4 (b), the positions of the origins of ΣR and ΣO are located at the contact point between the environment (left finger) and the object. Then, J and G are given by ⎛ ⎞ ⎛ ⎞ 10 1 0 10 ⎜0 0⎟ ⎟ ⎝ ⎠ for Fig.20.4 (a), J =⎜ ⎝0 1⎠ G = 0 1 0 1 0 −1 0 1 00 ⎞ ⎛ ⎛ ⎞ 0 0 1010 ⎟ ⎜ 0 0 ⎟ ⎝ √ ⎠ for Fig.20.4 (b). J =⎜ ⎝ − 3/2 0 ⎠ G = 0 1 0 1 0002 −1.5 −1

20 Task Based Hybrid Closure Grasping Optimization for Autonomous Robot Hand

Then, A is given by

⎛

433

⎞ 0 1 ⎟ ⎟ for Fig.20.4 (a), 0 ⎠ −1 ⎞ 0 0 −1 0 0 ⎜ 0 0 0 −1 0 ⎟ ⎟ √ A=⎜ ⎝ − 3/2 0 −1 0 0 ⎠ for Fig.20.4 (b). −1.5 −1 0 −1 −2

1 ⎜0 A=⎜ ⎝0 0 ⎛

0 0 1 0

−1 0 −1 0

0 −1 0 −1

Then, Λ is given by

 T Λ = −0.5774 0.5774 −0.5774 0 0 for Fig.20.4 (a),  T Λ = 0 −0.8944 0 0 0.4472 for Fig.20.4 (b).

Therefore, object velocity can be generated in only x direction in Fig.20.4 (a) while object can be rotated but can not be moved in translational directions in Fig.20.4 (b). Next, we consider resultant force and moment applied to the object in a hybrid closure grasp. From Eq. (20.6), the following relation is obtained: f c = (JJ T )+ τ c + Γ ξ ,

(20.11)

denotes the pseudo-inverse matrix of J and Γ is an orthogonal where matrix whose columns form bases of the null space of J T , ξ ∈ R p denotes an arbitrary vector. Note that Γ ξ expresses an internal force which makes no influence on joint torque. By substituting Eq. (20.11) into Eq. (20.6), we get (JJ T )+

JT

w = G (JJ T )+ τ c + GΓ ξ .

∈ R Ld×p

(20.12)

The second term GΓ ξ expresses an internal force which can be generated without changing joint torques. Namely, this force can be passively generated. If GΓ can span the space of w c , any external force can be balanced passively. Such a grasp is called power grasp. In the system shown in Fig. 20.4, GΓ is ⎛ ⎞ ⎛ ⎞ 10 00 GΓ = ⎝ 1 0 ⎠ for Fig.20.4 (a) and ⎝ 0 1 ⎠ for Fig.20.4 (b). 00 01 Therefore, external force in y direction and external moment can be balanced passively in Fig.20.4 (a) while external force in any direction can be balanced passively in Fig.20.4 (b).

20.4 Dynamics The equation of motion of every finger can be expressed by M ri q¨ i + hri + gri + τ ci = τ i

434

T. Watanabe, Z. Jiang, and T. Yoshikawa

where M ri , h ri and g ri represent the generalized inertia tensor, centrifugal and Coriolis forces, and gravitational force of the ith finger, respectively. Note that ever finger contacting with the object applies contact force to the object, and then the effect of the contact force has to be included in the equation of motion. Aggregating the equations for all fingers and from Eq. (20.6), we get M r q¨ + hr + gr + J T f = τ

(20.13)

M ri ), h r =col(hhri ), and g r =col(ggri ). where M r =diag(M Combining Newton’s and Euler’s equation of motion, the equation of motion of the object can be written by M B r¨ + hB = w , where MB =



mb I O O HB



 , hB =

O H Bω o) ω o × (H

 (20.14)

where mb is the mass of the object, and H B is the inertia tensor of the object. If considering the case when external force, w e , is exerted on the object, the equation of motion of the object becomes M B r¨ + h B − w e = w . From Eq. (20.6), we get M B r¨ + hB − w e = G f .

(20.15)

If aggregating the static and dynamical relation between the object and the fingers, from equations (20.5), (20.13) and (20.15), we get ⎛ ⎞ ⎞⎛ ⎞ ⎛ ⎞ ⎛ h r + gr Mr O JT τ q¨ ⎝ O M B −G G ⎠ ⎝ r¨ ⎠ = ⎝ o ⎠ − ⎝ h B − w e ⎠ . (20.16) T f o G O J q˙ − G T r˙ J −G

20.5 Concept of Required External Force Set For optimization of grasping performance, the concept of required external force set, and several related sets are defined. Required External Force Set: We call an external force (moment) exerted on the object, required to be balanced for completing a given task, a “required external force”. We call a set, composed of all possible required external forces, a “required external force set”. Note that the required external force set must be given in such a way that it is guaranteed that the given task can be completed if any force contained in the required external force set can be balanced.

20 Task Based Hybrid Closure Grasping Optimization for Autonomous Robot Hand

435

Since the set is constructed based on a task, the magnitudes of external forces and moments, which must be balanced during grasping the object, are preliminarily given by the set. Therefore, in this case there is no need to consider how to evaluate the difference between the magnitudes of contact forces required for balancing external force and the ones required for balancing external moment. If any external force contained in the set can be balanced, it is guaranteed that the given task can be completed, irrespective of both how to control the robot hand and how to manipulate the object. It means that in the grasping optimization there is no necessary to consider how to control the robot hand and to manipulate the object doing the task. When the origin is in the interior of the set, a force-closure grasp is required (note that this is one of the necessary and sufficient conditions for forming force closure). Further, when the set is constructed by only one external force, an equilibrium grasp is just required. Therefore, by using the required external force set, we can investigate any desirable grasp including forceclosure and equilibrium grasps. In addition, since only the forces contained in a given required external force set, not the whole generable resultant force, need to be evaluated, the optimal grasping procedure can be computed with smaller computational load. While there are above several merits in the concept of a required external force set, the set can not deal with the difference “whether the force can be generated in the direction or not” and “whether we can apply the acceleration in the direction or not”. In order to take the difference into consideration, we newly define required acceleration set and equilibrium-force set, which are also given by task and have the same merits as a required external force set. Required Acceleration Set: We call a set, composed of all possible desired object accelerations, required acceleration set AR ⊂ R D . Required Equilibrium-Force Set: We call a set, composed of the resultant forces and moments needed to balance all possible external forces and moments exerted on the object and to maintain the current position and posture of the object, required equilibriumforce set WR ⊂ R D . All forces and moments contained in the set WR must be able to be applied on the object. Note that there could exist directions in which not only required acceleration but also required equilibrium force are required to be generated. If we can calculate the corresponding required acceleration to the required equilibrium force, the required equilibrium force can be replaced by the calculated corresponding required acceleration. However, it is not easy because the direction of acceleration is not always same as the direction of force which induces the acceleration. Therefore, in such a case, we require to generate both the required acceleration and the required equilibrium force simultaneously in that direction. Note that we call a set, composed of all accelerations which the object can generate, generable acceleration set A ⊂ R D , and that we call a set, composed of all resultant forces and moments which we can apply on the object, generable force set W ⊂ R D . When required acceleration set AR and required equilibrium-force set WR are given, using r¨ d and w d which are an element of AR and an element of WR respectively, we define the combinational required set as follows: T  (20.17) Wc = {xx = r¨ Td w Td |¨r d ∈ AR , w d ∈ WR }.

436

T. Watanabe, Z. Jiang, and T. Yoshikawa

20.6 Formulation of Grasping Optimization Problem Using Required External Force Set Now let’s consider to optimize a general grasp including a hybrid closure grasp. In order to deal with such a general grasp, we have to distinguish the acceleration applying to the object and the resultant force and moment. So an optimization problem is considered here for the case when a grasped object, with resisting external forces and moments, is manipulated in a given workspace. Also, for the distinction of acceleration and force, both the grasping points and configurations of fingers have to be taken into account in the optimization problem. How to optimize both the grasping points and configuration of fingers is discussed in detail in following. At every position and posture of the object in the given workspace, in general, there exist an infinite number of configurations of fingers and grasping points where AR ⊂ A and WR ⊂ W are satisfied. If the acceleration and force, which are contained in the required acceleration and equilibrium-force sets, can be generated by smaller joint torques, we can get the following merits: 1) we can reduce the cost of actuators embedded in the joints; 2) we can reduce the perturbation of the resultant force and moment due to the perturbation of the joint torques (note that such a perturbation could be caused by the difference between the external force and its balancing force applied by the joint torques); 3) we can grasp the object more stably. Let C be the combination of configuration of fingers and contact positions. Then, we consider the following problem. Problem : Find C to minimize the magnitudes of the joint torques required for generating both any acceleration and any equilibrium force which are, respectively, contained in the given required acceleration set and equili-brium-force set, at every position and posture of the object in the given workspace. Next, we formulate this optimization problem. Based on a task, the workspaces of the robot hand and the object, and the target motion of the object are assumed to be given. Note that the target motion corresponds to required accelerations and possible object configurations. Let r B be the position and posture of the object in the workspace. Let SB be a set of all possible candidates of r B . Since there could exist several object configurations in the target motion, the finger configuration to be optimized is not always one. This means that a certain optimum finger configuration in a certain object configuration is not always optimum in another object configuration. In order to avoid the situation, we assume that Cr is a given C in a certain object configuration. We consider the robot motion from the Cr without changing the contact positions with respect to the object. In this case, any possible induced C due to the motion is unique with respect to every r B ∈ SB . This assumption is reasonable because from the assumption at Section 20.2.1, it is evident that the contact points do not change with respect to the object in the motion and the finger has no redundancy. Here, we set SCt as the set of C induced due to the motion of robot from a certain configuration Cr . SCt is assumed to have C ’s for all r B ’s ∈ SB , and C ∈ SCt for every r B ∈ SB is assumed to be unique. Then, SCt is the target to be optimized. Let SSCt be a set of all possible candidates of SCt .

20 Task Based Hybrid Closure Grasping Optimization for Autonomous Robot Hand

437

Since the required acceleration and force could be changed with respect to the system configuration due to robot motion, the combinational required set Eq. (20.17) should be given for every r B ∈ SB . Then, let WcB be the combinational required set Eq. (20.17) given at r B (∈ SB ). Let |τi j |(= τiTj τi j ) be the norm of the jth joint torque of the ith finger. Let φ be the largest norm among |τi j |’s ( j = 1, 2, · · · , Mi , i = 1, 2, · · · , N):

φ = max |τi j |. i, j

(20.18)

Then, φ has the following relation with every joint torque.

φ ≥ |τi j | ( j = 1, 2, · · · , Mi , i = 1, 2, · · · , N).

(20.19)

Consider to generate both r¨ d and wd ∈ WcB in the case when the object is at a certain r B and the finger and the contact positions are at a certain C . We consider the conditions for the joint torques to be satisfied in this case. From equations (20.5), (20.7) and (20.16), the condition is that there exists τ which satisfies the following constraints: A 1 τ + A2 f = G T r¨ d + a , G f = M B r¨ d + w d + hB , f i j ∈ F f i j ( j = 1, 2, · · · , Li ) (i = 1, 2, · · · , N)

(20.20) (20.21) (20.22)

where T −1 ˙ T r˙ . J M −1 g r + hr ) − J˙ q˙ + G A 1 = J M −1 r , A 2 = −J r J , a = J M r (g

we ). Note that wd (=−w To minimize φ with respect to τ and f satisfying equations (20.19)∼(20.22), the minimum magnitude of the joint torques, which is required for generation of the r¨ d and the wd , is expressed by τ,f

min φ. satisfies equations (20.19)∼(20.22)

(20.23)

Now consider the minimum magnitude of joint torques obtained from Eq. (20.23) in a certain SCt , with respect to every r¨ d and w d (∈ WcB ) at every r B (∈ SB ). The maximum magnitude among the magnitudes will be the minimum required magnitude of joint torques in the SCt . The magnitude, φ , is given by max

x ∈WcB ,rr B ∈SB τ , f

min φ. satisfies equations (20.19)∼(20.22)

(20.24)

Further, consider the required magnitude of joint torque set at every SCt ∈ SSCt . The smallest required magnitude of joint torques, by minimizing the magnitude φ of Eq. (20.24) in the SSCt , is then given as min

max

SCt ∈SSCt x ∈WcB ,rr B ∈SB τ , f

min φ. satisfies equations (20.19)∼(20.22)

(20.25)

438

T. Watanabe, Z. Jiang, and T. Yoshikawa

Then, the optimization problem can be formulated as follows: Optimization Problem: Find the ρ and the SCt∗ which gives the ρ such that,

ρ = min

max

SCt ∈SSCt x ∈WcB ,rr B ∈SB τ , f

min φ satisfies equations (20.19)∼(20.22) . (20.26)

Note that obtaining the SCt∗ means obtaining every C ∗ which is optimal at every r B ∈ SB . This optimization method can deal with the problem of a general grasp including hybrid closure grasp, by characterizing the acceleration applying to the object and the resultant force and moment.

20.7 Algorithm In this section, we describe an algorithm to solve the optimization problem in Eq. (20.26). This optimization problem has to deal with a huge number of configurations (of fingers and object), and further, the relationship between each configurations is nonlinear and complex. The changes of the angles of fingers are continuous, while the change of the number of contact points is discontinuous (Note that here we means change due to SCt change). Also, the object shape is expressed by continuous and discontinuous factions. Therefore, It is difficult to solve the problem theoretically. We represent the all possible configurations of fingers as a finite number of configurations, the shape of object (the all possible contact positions) as a finite number of points, and the all possible positions and postures of object as a finite number of positions and postures. Then, we just need to select the optimal configuration of fingers and contact positions among the candidate configurations and positions, based on the consideration of the motion at the discrete object states. In order to solve the problem in Eq. (20.26), we use a branch-and-bound method [38, 39, 40]. The criterion value φ in the problem, (see Eq. (20.26)), is discontinuous and can change discontinuously due to the discontinuous changes of the contact positions, the number of contact points, and so on. Therefore, if using a method such as greed method which can provide local optimum, it is possible that reasonable solution can not be obtained. Hence, in order to obtain reasonable solution, we use branch-and-bound method which can deal with discrete domain problem and guarantees to obtain global optimum. Utilizing branch-and-bound method, we find an optimal solution among a finite number of feasible solutions. The feasible solutions are usually the solutions of the subproblems into which we divide the original problem. This method finds the optimal solution by enumerating the solutions of feasible subproblems. In the process, we cut the subproblems which we don’t have to solve, by using the solution of a subproblem of the subproblem. It makes the computational time reduced. In order to use the branch-and-bound method, we make the following assumptions. 1) The number of the all candidates of SCt is finite nc . Let SCt k (k = 1, 2, · · · , nc ) be SCt contained in SSCt .

20 Task Based Hybrid Closure Grasping Optimization for Autonomous Robot Hand

439

2) The number of the all candidates of r B is finite nb . Let r BI (I = 1, 2, · · · , nb ) be r B contained in SB . Let CkI be the C in the case where the SCt is SCt k and the object is at r BI (k : k of the SCt k , I : I of the r BI ). CkI is uniquely determined by SCt k and r BI . 3) We only have to consider quasi-static motion in which q˙ ∼ = o and r˙ ∼ = o at every r BI . 4) The required acceleration set and equilibrium-force set given at every r BI (ARI and WRI ), respectively, can be expressed by convex polyhedra composed of la and le vertices. Note that la denotes the number of vertices of ARI and le denotes the number of vertices of WRI . 5) When the object is at a certain r BI and the C is a certain CkI , the term representing gravitational force of the fingers (ggr in Eq. (20.13)) can be expressed by a convex polyhedron composed of le vertices, whose element corresponds one to one to element contained in the required equilibrium-force set WRI (Note that here, we suppose the case where the direction of gravitational force can change with respect to the object coordinate frame fixed at the object, resulting from the manipulation of the robot arm equipped with the robot hand). For more detail, see appendix in Section 20.10. Now, we define some subproblems of the problem in Eq. (20.26) to solve it at first, and then we describe the procedure of the algorithm. 20.7.1

Subproblems

At first, we consider the minimum required magnitude of joint torques in the case when the SCt is a certain SCt k ∈ SSCt . In this case, we can obtain the following subproblem of the problem (20.26): Subproblem 1 max

x ∈WcB ,rr B ∈SB τ , f

min φ satisfies equations (20.19)∼(20.22) .

(20.27)

If we can solve this Subproblem 1 for all SCt k ’s (k = 1, 2, · · · , nc ), we can get the optimal solution of the problem in Eq. (20.26). T

where the object is at the r BI (∈ SB ), r¨ dI j is the jth vertex Let xI jξ be r¨ TdI j w Td Iξ of ARI and w dI ξ is the ξ th vertex of WRI . We consider the minimum required magnitude of joint torques in the case when in the Subproblem 1, the object is at a certain r BI (∈ SB ) and the x required to be generated is a certain x I jξ . In this case, we can obtain the following subproblem of the Subproblem 1: Subproblem 2 τ, f

min φ satisfies equations (20.19)∼(20.22) .

(20.28)

Note that C in Subproblem2 is the CkI (k is the k of the SCt k and I is the I of the r BI ). Now, we consider the case when the optimal solution of Subproblem2 for every r BI and every x I jξ where SCt = SCt k can be obtained. Let ρSct ,I be the solution in k jξ the every case. Then, the largest ρSct ,I among ρSct ,I ’s (k is fixed, I = 1, 2, · · · , nb , k jξ

k jξ

440

T. Watanabe, Z. Jiang, and T. Yoshikawa

j = 1, 2, · · · , la , ξ = 1, 2, · · · , le ) is the solution of the Subproblem 1 for SCt k . The proof is described in appendix 20.11. Here is how to get the solution of the Subproblem 1 for a certain SCt k : (1) Compute the solution of every Subproblem 2 which is a subproblem of the Subproblem 1 (Compute the ρSct ,I ’s (k is fixed, I = 1, 2, · · · , nb , j = 1, 2, · · · , la , ξ = k jξ 1, 2, · · · , le )). (2) If obtaining the solution for every Subproblem 2, the largest ρSct ,I among the soluk jξ

tions ρSct ,I ’s (k is fixed, I = 1, 2, · · · , nb , j = 1, 2, · · · , la , ξ = 1, 2, · · · , le ) is the solution k jξ of the Subproblem 1. Otherwise, there is no solution. Note that the Subproblem 2 can be solved by a simplex method, by approximating the frictional constraint by a polyhedral convex cone[42]. 20.7.2

Procedure of the Algorithm

In this subsection, we describe the procedure of the algorithm to solve the optimization problem (20.26). Fig.20.5 shows the flow chart of the algorithm. We define the following nomenclatures:

ρˆ The (best known) feasible solution at each iteration. ρˆ Sct k The (best known) feasible solution of Subproblem1 for SCt k at each iteration. LIST The list of feasible SCt k ’s . Nn The number of feasible SCt k ’s at the initial. ρˆ The lower bound value of ρˆ . ρˆ Sct k The lower bound value of ρˆ Sct k . Note that the feasible SCt k is the SCt k which can satisfy the following conditions at every corresponding CkI (k is the k of the SCt k , I = 1, · · · , nb ): (i) there is no interference between finger and object; (ii) there is no interference between fingers; (iii) every joint angle of fingers is not beyond its workspace. Here is the procedure of the algorithm. Step 1. Put all feasible SCt k ’s into the LIST . Set ρˆ = ρˆ and ρSct = ρSct (k = k k 1, 2, · · · , NN ). Step 2. First, we search the first eligible value of ρˆ , solving Subproblem1 at a certain SCt k . Select a certain SCt k from the LIST and solve the Subproblem1 for the SCt k using the method described in Section 20.7.1 (see Fig.20.6 (a)). Step 3. If we can get the solution of the Subproblem1 in Step2, substitute the solution into ρSct (= ρˆ Sct ). Let ρˆ = ρSct . Let r BIˆ be the r BI which gives the solution, and let k k k ˆ j,ˆ and ξˆ are the I, j, and ξ of the largest x ˆ be the x I which gives the solution (I, jξ

I jˆξˆ

ρSct k ,I ). Otherwise, we eliminate the SCt k from the LIST and go back to Step2. Note jξ

20 Task Based Hybrid Closure Grasping Optimization for Autonomous Robot Hand Step 1 Set LIST, ρˆ = ρˆ , ρˆ Sctk = ρˆ Sctk

The initial values Step 2

Select S ct kfrom the LIST Eliminate S ct k

Find ρ Sctk ,I jξ for all I, j, ξ by solving Subproblem 2

Search the first eligible ρ

N

Can we get for all I, j, ξ ? Y Step 3

Compute ρ Sctk = max ρ Sctk , I jξ , ρˆ Sctk = ρ Sctk I , j ,ξ

Search I, j, ξ which will most possibly give ρ Sct k at every Sctk .

ρˆ = ρ Sctk , Iˆ, ˆj , ξˆ = arg max ρ Sctk , I jξ I , j ,ξ

Step 4

k=1

Set

N Search and eliminate non optimal Sct k.

Set k=k+1

Is S ct k contained in the LIST ? Y Find ρ Sctk , Iˆ ˆjξˆ (Solve Subproblem 2)

Eliminate S ct k

N

Can we get ? Y Compute ρˆ Sctk = max{ ρˆ Sctk , ρ Sct

k

, Iˆ ˆjξˆ

}

N ρˆ ≥ ρˆ Sctk ? Y

N

k = NN ? Y Step 5 Select Sct k which will most possibly be optimal

Compute kˆ = arg min{ρˆ Sctk , Iˆ ˆ ˆ } jξ

k

Step 6

Find ρ Sctkˆ , I jξ for all I, j, ξ by solving Subproblem 2 Can we get for all I, j, ξ ? Y

N

ρ Sctkˆ , I jξ , ρˆ Sctkˆ = ρ Sctkˆ Compute ρ Sctkˆ = max I , j ,ξ

Derive the best known ρ and Sctk at this moment

Iˆ, ˆj , ξˆ = arg max ρ Sctk , I jξ I , j ,ξ

ρˆ ≥ ρ Sctkˆ ?

N

Y

IF ρˆ > ρ Sct ˆ , k

ρˆ = ρ Sctkˆ

Step 7 Judge whether the best known ρ and Sct k are optimal

N

| ρˆ − ρ Sctk |< ε ,

∀

S ct k ∈ LIST ? Y

Finish

Fig. 20.5. Flow chart of the algorithm

Eliminate S ct k

441

442

T. Watanabe, Z. Jiang, and T. Yoshikawa Select Sctk Sctk

I, j, ξ

Select Sctk which will most possibly be optimal

If Sctk is non-optimal, eliminate it

Sctk I, j, ξ

I, j,^ ξ which give ρ Sct ^

^

^

I, j,^ ξ

k

^

(a) Step 2 and 3, and Step 6

(b) Step 4 and 5

Fig. 20.6. The diagram of Step 2∼6

ˆ j,ˆ and ξˆ for the search, supposing that the corresponding ρ that we select I, Sct

ˆ k ,I jˆξˆ

at

every SCt k will be most possibly the largest one among ρSct ,I ’s for all I, j, and ξ . k jξ Step 4. Solve the Subproblem 2 for the r BIˆ and the x Iˆ ˆ , at every SCt k contained in jˆξ

ˆ jˆ, and ξˆ . k is the k of each SCt contained in the the LIST (I, j, and ξ are fixed to I, k LIST ) (see Fig.20.6(b)). Note that in the every Subproblem 2, the C is the CkIˆ whose Iˆ is the Iˆ of the r BIˆ and whose k is the k of the each SCt k . In other words, we fix r BI and x I jξ , and search eliminable SCt k ’s. If we cannot get the solution at a certain SCt k (CkIˆ), eliminate the SCt k from the LIST . If we can get the solution ρSct ,Iˆ at a certain k jˆξˆ

SCt k (CkIˆ), compute ρˆ Sct = max{ρˆ Sct , ρSct k

k

ˆ k ,I jˆξˆ

}. Here, we compute the (best known)

lower bound of the solution of Subproblem1 for the SCt k at this moment. If ρˆ ρS jˆξ

cannot get the solution or ρˆ Π − Φm ) or (−Π < θe ≤ −Π + Φm )), d = 0, otherwise

(21.9)

The value of Φm can be set to a different value for individual robots in a collaborative task. It essentially decides at what point the linear component of the velocity kicks in when the robot moves from its present position to a destination. As the robot starts turning towards the desired direction, the angle error is decreasing and hence the velocity component, Ka p θe , is reduced (Eq. 21.8). When the robot is in line with the desired direction, α , the angle error, θe , is zero, thereby bringing the velocity component, Ka p θe , to zero. Choosing the optimal values for Kd p , Kdd and Ka p is a tedious process.

21.5 Advanced Robot Positioning Algorithm In the proposed algorithm, the position function can be decomposed into three sectionspre-processing inputs, ’retargeting’ system and velocity profiling, as shown in Figure 21.5.

Fig. 21.5. Robot Positioning Process

21.5.1

Pre-processing Inputs

The vision layer generates data pertaining to the position, orientation, an-gular and linear velocities of the robots. At the pre-processing section, the robot-to-target distance, the robot-to-target angle and the alignment error are calculated. Also, all the angles are normalized. Figure 21.6 shows the various angles and linear distances which are pre-processed. λ is the robot-to-target angle, δ is the final robot angle at the target position, β is the alignment error, dRT is the distance error (distance between robot and target positions) The robot-to-target angle, alignment error and the distance error are calculated using equations 21.10:    −1 dy λ = tan (21.10) , β = λ − δ , dRT = dx2 + dy2 dx

21 Triangular Targeting Algorithm for Accurate Motion Control

459

Fig. 21.6. Pre-processing inputs

21.5.2

Re-targeting System

An important issue with targeting systems is that the robot must approach the target inline with the final angle. If the target is a ball, as is the case in robot soccer, it would be hit in an undesired direction if not approached in line with the final angle, which would usually be towards the opponent’s goal or another collaborating robot of the team to which the ball has to be passed. Figure 21.7 shows a simple positioning methodology in which the robot first aligns with the target angle, moves to the target, and then aligns with the final angle (turnmove-turn approach). This approach may suffice in some applications, such as moving loads on a production floor, in which the robot stops on reaching the target position. In fact in some applications such as search and rescue, foraging and scavenging etc. such an approach is highly desirable. However, in many general form of conveyance the target is continually updated. For example, in a robot soccer game, the ball, which is the target, is always moving and its position relative to the opponent’s goal is continually changing. In such dynamic scenarios, the turn-move-turn approach will be very inefficient; the robot may end up just turning most of the time, trying to align with the target, before it can enter the move phase. To approach the target in line with the final angle, the proposed re-targeting system creates a new virtual target to control robot positioning when approaching the real target. Various algorithms can be used to calculate the most appropriate virtual target so as to give the robot more stable and accurate control [19]. The presented re-targeting method has two distinct parts- one is based on a ’triangular targeting algorithm’ (TTA) and the other is a ’proximity positioning algorithm’ (PPA). The new target position calculated by the re-targeting system is used in the profiling stage to derive the wheel velocities. The most appropriate re-targeting algorithm is selected based on the distance between the target and the robot. A circular area, with a very small diameter, around the target defines the transition boundaries to switch between TTA and PPA as shown in

460

G.S. Gupta, C. Messom, and S. Demidenko

Fig. 21.7. Turn-move-turn approach to reach a target

Fig. 21.8. Re-targeting System Selection

Figure 21.8. As long as the robot is outside the transition boundary, TTA remains in force to position the robot. When the robot is very close to the target it is often struggling to maintain stability - it not only has to reach the target precisely (i.e. zero distance error) with a linear motion, it also has to align with the final direction (i.e. zero alignment error) with an angular motion. In the presented system, as soon as the robot enters the circle, PPA takes over. The algorithms are explained in detail in later sections. The use of two different algorithms to control the robot’s targeting system increases the accuracy of positioning and alignment, and increases the stability of the robot when it reaches the target. Hysteresis at the transition boundary is defined to prevent oscillations and smooth switching between the TTA and PPA methods.

21.6 Triangular Targeting Algorithm (TTA) The Triangular Targeting Algorithm (TTA) allows the robot to traverse around objects and come in behind a set target position which is aligned with the final angle. This is especially good for pushing objects in a desired direction in a collaborative task. This also gives the robot angular stability because the angular velocity of the robot is reduced to near zero as it approaches the target.

21 Triangular Targeting Algorithm for Accurate Motion Control

461

Fig. 21.9. Triangular Targeting Algorithm: Robot’s path towards the target

Fig. 21.10. Various angles and distances for TTA

The robot’s path, as it moves towards a target, is shown in Figure 21.9. The robot is commanded to go to a ’virtual target’. The robot position, target position and the position of the virtual target form the three vertices of a triangle, hence the name Triangular Targeting Algorithm (TTA). In the presented algorithm, the height of the triangle is calculated based on the alignment error β (Figure 21.6 and 21.10). As the robot starts to move towards the virtual target position (indicated by the star at the top of the triangle in Figure 21.9), the alignment error reduces. This in turn reduces the height of the triangle. The end result is that the robot approaches the target from behind facing the desired final angle. In Figure 21.10 R, V and T denote the position of the robot, virtual target and the real target respectively. The various angles and distances used in the TTA are as follows: dRV is the distance of the robot to the virtual target, dRT is the distance between the robot and the target, h is the triangle height (VQ), μ is the robot-to-virtual target angle, η is the inner triangle angle, θr is the robot angle, θe is the angle error, λ is the

462

G.S. Gupta, C. Messom, and S. Demidenko

Fig. 21.11. Relationship between the maximum triangle height and the distance error

robot-to-target angle, δ is the final robot angle at the target position, β is the alignment error. The alignment error is calculated using Eq. 21.10. The triangle height is an exponential function of the alignment error and is given by equation 21.11:     β  − triCur  (21.11) hmax h = d 1−e where d is the triangle protrusion direction and is dependent on the value of β , the alignment error; d is -1 if β ≥ 0, otherwise it is 1. The value of β determines how far the robot has to arc around to align with the target and the final angle; it is zero when the robot is perfectly aligned with the final direction, at which stage the triangle height, h, is zero too. hmax is the maximum triangle height and is related to the distance, dRT , by equation 21.12: hmax =

triMax dRT triO f f

(21.12)

The maximum triangle height is clamped at triMax. From the Eq. 21.11 and 21.12, it can be deduced that the triangle height is proportionately small when the robot is closer to the target. This prevents the robot from traversing an expensive arc even if the alignment error is large. The value of hmax needs to be clamped; otherwise the arc traversed by the robot will keep getting bigger as the robot goes farther away from the target. The relationship between the maximum triangle height and the robot-to-target distance is illustrated in Figure 21.11. For a given alignment error, β , the curvature, triCur, determines the rate at which the robot will converge behind the target, aligned with the final angle. The smaller the value of triCur, the steeper will be the curve that relates triangle height, h, to the alignment error, β . This is shown in Figure 21.12. If the curve is too steep, the robot will oscillate along the final angle vector as it approaches the target. If the curve is too shallow, the robot will slowly align with the final angle vector and may not reach the final angle when it reaches the target. This is depicted in Figure 21.13. Figure 21.14 shows the triangle heights for different robot positions in a workspace of dimension 150 cm x 150 cm. The target is at location (0, 0) and the final angle is 0o .

21 Triangular Targeting Algorithm for Accurate Motion Control

Fig. 21.12. Exponential profiling of triangle height

Fig. 21.13. Effects of curvature on robot movement

Fig. 21.14. Triangle heights for different robot positions

463

464

G.S. Gupta, C. Messom, and S. Demidenko

21.7 Proximity Positioning Algorithm (PPA) As explained in section 21.5.2, when the robot is very near the target, within a predefined circular zone, the positioning algorithm switches over to PPA (Proximity Positioning Algorithm). The purpose is to increase the absolute positioning accuracy. It also helps to overcome the noise in the vision system. The algorithm works by locking the robot’s angle to the final angle when the robot is in close proximity of the target. Figure 21.15 shows how the algorithm works. Line AB is orthogonal to the final direction vector, F. For reference, X-axis is defined along the direction vector F and so the Y-axis is along the line AB. When the robot enters the inner circle, it turns to align with the final direction vector and continues to move (with pure linear motion) till it reaches the line AB. This reduces the final angle error and the X-axis error of the robot’s position with respect to the target. However, there is a possibility of the robot reaching the target with a positional error along the Y-axis. In practice, this error is very small since the diameter of the circle (i.e. length of line AB) is kept very small, generally much less than half the robot width. The algorithm is very accurate when used in conjunction with Triangular Targeting Algorithm (TTA) and greatly improves the stability of the robot when it reaches the target. The hysteresis helps to overcome the oscillatory effects of the sensor noise at the transition boundary.

Fig. 21.15. PPA algorithms at work

21.8 Velocity Profiling With reference to Figure 21.10, the robot has to move the distance dRV at an angle μ . These are given by the Eq. 21.13 and 21.14. The wheel velocities are calculated to move the robot to the virtual target and are a function of the distance error (de ) and angle error (θe ).   dRT 2 2 (21.13) de = dRV = h + 2   h −1 μ = λ + tan (21.14) dRT /2

21 Triangular Targeting Algorithm for Accurate Motion Control

θe = θr − μ

465

(21.15)

Velocity profiling ensures rapid and accurate response to robot positioning. This system has three parts - angular, linear and cross-effects profiling. 21.8.1

Angular Velocity Profiling

Angular profiling controls the turning performance of the robot. It is desirable that the robot’s angular velocity be high when there is a large turning angle and low when the turning angle is small so that the desired angle is not overshot. The general profiling function used here is an exponential curve as shown in Figure 21.16.

Fig. 21.16. Exponential profiling of angular velocity

The angular velocity, VA, is given by equations 21.16: VA = 0||θe |