Adaptive and Natural Computing Algorithms: Proceedings of the 7th International Conference in Coimbra, Portugal, March 21-23 2005

SpringerWien NewYork

Bernardete Ribeiro Rudolf EAlbrecht Andrej Dobnikar David W. Pearson Nigel C. Steele (eds.) Adaptive and Natural Computing Algorithms Proceedings of the International Conference in Coimbra, Portugal, 2005

SpringerWienNewYork

Dr. Bernardete Ribeiro Departamento de Engenharia Informatica Universidade de Coimbra, Coimbra, Portugal

Dr. Rudolf F. Albrecht Institut fur Informatik Universitat Innsbruck, Innsbruck, Austria

Dr. Andrej Dobnikar Fakulteta za Racunalnistvo in Informatiko Univerza v Ljubljani, Ljubljana, Slovenia

Dr. David W. Pearson Equipe Universitaire de Recherche en Informatique de Saint-Etienne (Groupe de Recherche de Roanne) Institut Universitaire de Technologie de Roanne Universite Jean Monnet, Saint-Etienne, France

Dr. Nigel C. Steele Division of Mathematics School of Mathematical and Information Sciences Coventry University, Coventry, U.K.

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically those of translation, reprinting, re-use of illustrations, broadcasting, reproduction by photocopying machines or similar means, and storage in data banks. Product Liability: The publisher can give no guarantee for the information contained in this book. This also refers to that on drug dosage and application thereof. In each individual case the respective user must check the accuracy of the information given by consulting other pharmaceutical literature. The use of registered names, trademarks, etc., in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. © 2005 Springer-Verlag/Wien Printed in Austria SpringerWienNewYork is part of Springer Science + Business Media, springeronline.com Camera-ready copies provided by authors and editors Printed by Novographic Druck G.m.b.H., 1230 Wien, Austria Printed on acid-free and chlorine-free bleached paper SPIN 11393269

With 433 Figures

ISBN 3-211-24934-6 SpringerWienNewYork

Preface

The ICANNGA series of Conferences has been organised since 1993 and has a long history of promoting the principles and understanding of computational intelligence paradigms within the scientific community and is a reference for established workers in this area. Starting in Innsbruck, in Austria (1993), then to Ales in Prance (1995), Norwich in England (1997), Portoroz in Slovenia (1999), Prague in the Czech Republic (2001) and finally Roanne, in France (2003), the ICANNGA series has established itself for experienced workers in the field. The series has also been of value to young researchers wishing both to extend their knowledge and experience and also to meet internationally renowned experts. The 2005 Conference, the seventh in the ICANNGA series, will take place at the University of Coimbra in Portugal, drawing on the experience of previous events, and following the same general model, combining technical sessions, including plenary lectures by renowned scientists, with tutorials. In a rapidly advancing world, where technology and engineering change dramatically, new challenges in modern informatics compel us to broaden the Conference themes in order to take into account new developments. Nevertheless we have kept the acronym ICANNGA, which now stands for International Conference on Adaptive and Natural ComputiNG Algorithms It is interesting to note that the themes of this conference have their origins in attempts to obtain understanding of natural processes. Indeed, Nature itself seems capable of providing a source of inspiration both for new research areas and new methodologies. Thus the focus of ICANNGA is on both the theoretical aspects and also on the practical applications of computational paradigms inspired by adaptive and natural processes, with special emphasis on adaptive and machine learning systems, neural networks, evolutionary algorithms, fuzzy systems, support vector machines, biological computation and quantum computing. Amongst the papers accepted for presentation, there are contributions reporting theoretical developments as well as practical applications both for neural networks, genetic algorithms and evolutionary computation. Unsupervised and clustering techniques as well as self-organising maps are also the subjects of a substantial number of papers. Also, the relevance of computational neuroscience with a strong biological trend is evidenced by the number of papers submitted. It is pleasing to observe the number of contributions reporting successful applications of the technology to the solution of industrial problems. Predominantly, there are applications of neural networks to process engineering and to robotics and control which may reflect the maturity reached by the methodology. Contributions also abound in the field of evolutionary computation particularly in combinatorial and optimisation problems. They range from evolutionary computation theory to real-world applications, from global numerical optimisation to combinatorial optimisation, from evolutionary learning to multi-agent systems. The topics covered by the accepted papers reflect the breadth and depth of these fields. It is worth noting that they are inherently interdisciplinary. They interact closely with computer science, engineering, biology, mathematics, economics, and many papers at this conference illustrate such interdisciplinarity quite clearly. Contributed papers in the field of natural computing, swarm optimisation and bioinformatics and computational biology are no less compelling. The awareness in the academic communities that hybrid approaches of intelligent systems will be necessary if the real challenges in computational intelligence are to be tackled is a fundamental stimulus to their investigation and has been acknowledged for the many papers with cross fertilising ideas submitted. Many papers are dedicated to machine learning and heuristics and others to soft computing applications. Some papers are devoted to the theory of computation in which traditional information is replaced by its quantum physical counterpart showing that it is an interesting challenge to study quantum computation. In addition, kernel based algorithms, able to solve tasks other than classification, represent a revolution in pattern recognition bridging existing gaps. Some papers either theoretical or application oriented are dedicated to pattern recognition, intelligent signal processing and computer vision. We would like to express our gratitude to everyone who contributed in any way to the completion of this volume. In particular, we thank the members of the Programme Committee for reviewing the submissions and making the final decisions on the acceptance of papers, and the additional reviewers Mika Hirvensalo, Jorge Tavares, Pino Buzzanca, Jose Neto, Armando Vieira and Miguel Rocha for their work. We thank also

VI Jorge Tavares and Pedro Martins (University of Coimbra) for their difficult and time consuming task of checking paper source files for correctness and preparing the WT^K.volume initial pages, Silvia Schilgerius (Springer-Ver lag) and finally to all authors for their valuable research work and for their submissions. B. Ribeiro R. Albrecht A. Dobnikar D. W. Pearson N. C. Steele

University of Coimbra, Portugal University of Innsbruck, Austria University of Ljubljana, Slovenia Jean Monnet University of Saint-Etienne, France Coventry University, England

Contents

Advisory and Programme Committees

XIV

Part I - Neural Networks Neural Networks Theory I ADFUNN: An Adaptive Function Neural Network D. Palmer-Brown, M. Kang Certain Comments on Data Preparation for Neural Networks Based Modelling B. Beliczynski A Simple Method for Selection of Inputs and Structure of Feedforward Neural Networks H. Saxen, F. Pettersson The Concept and Properties of Sigma-if Neural Network M. Huh, H. Kwasnicka Beta Wavelet Networks for Function Approximation W. Bellil, C. Ben Amar, A. M. Alimi

1 5 9 13 18

Neural Networks Theory II Speeding up Backpropagation with Multiplicative Batch Update Step P. Cruz Generating Sequential Triangle Strips by Using Hopfield Nets J. Sima The Linear Approximation Method to the Modified Hopfield Neural Network Parameters Analysis .. 5. I. Bank, S. M. Perovich, A. Lompar The Analytical Analysis of Hopfield Neuron Parameters by the Application of Special Trans Function Theory S. M. Perovich, S. I. Bank, N. Konjevic Time-Oriented Hierarchical Method for Computation of Minor Components M. Jankovic, H. Ogawa

22 25 29 33 38

Learning Theory Evolution versus Learning in Temporal Neural Networks H. Soula, G. Beslon, J. Favrel Minimization of Empirical Error over Perceptron Networks V. Kurkovd Interval Basis Neural Networks A. Horzyk Learning from Randomly-Distributed Inaccurate Measurements J. Eidson, B. Hamilton, V. Kanevsky Combining Topological and Cardinal Directional Relation Information in QSR H. Sun An Evidence Theoretic Ensemble Design Technique H. Altingay

42 46 50 54 62 66

Computational Neuroscience and Neurodynamics Cortical Modulation of Synaptic Efficacies through Norepinephrine .. O. Hoshino Associative Memories with Small World Connectivity N. Davey, L. Calcraft, B. Christianson, R. Adams

70 74

VIII A Memory-Based Reinforcement Learning Model Utilizing Macro-Actions M. Murata, S. Ozawa A Biologically Motivated Classifier that Preserves Implicit Relationship Information in Layered Networks C. C. Peck, J. Kozloski, G. A. Cecchi, A. R. Rao Large Scale Hetero-Associative Networks with Very High Classification Ability and Attractor Discrimination Consisting of Cumulative-Learned 3-Layer Neural Networks Y. Yatsuzuka, Y. Ho

78

82

86

Neural Networks in Process Engineering Crack Width Prediction of RC Structures by Artificial Neural Networks 92 C. Avila , Y. Tsuji, Y. Shiraishi A N e u r a l N e t w o r k S y s t e m for Modelling of C o a g u l a n t Dosage Used in D r i n k i n g W a t e r T r e a t m e n t . . . 9 6 B. Lamrini, A. Benhammou, A. Karama, M. V. Le Lann ANN Modeling Applied to NOx Reduction with Octane. ANN Future in Personal Vehicles 100 M. Ronnholm, K. Awe, K. Erdnen, F. Klingstedt, T. Salmi, H. Saxen A Method for Detecting Cause-effects in Data from Complex Processes 104 M. Helle, H. Saxen Predictive Data Mining on Rubber Compound Database 108 M. Trebar, U. Lotric

Neural Networks in Robotics and Control Applying Neural Network to Inverse Kinematic Problem for 6R Robot Manipulator with Offset Wrist 112 Z. Bingul, H. M. Ertunc, C. Oysu Local Cluster Neural Network Chip for Control 116 L. Zhang, J. Sitte, U. Rueckert A Switching Controller for Nonlinear Systems via Fuzzy Models 120 M. Boumehraz, K. Benmahammed Competitive Decentralized Autonomous Neural Net Controllers 124 T. Ohba, M. Ishida Improved Hierarchical Fuzzy Control Scheme 128 T. M. Jelleli, A. M. Alimi On-line Inference of Finite Automata in Noisy Environments 132 /. Gabrijel, A. Dobnikar

Clustering and Unsupervised Learning Improved Clustering by Rotation of Cluster Centres D. W. Pearson, M. Batton-Hubert Hierarchical Growing Neural Gas K. A. J. Doherty, R. G. Adams, N. Davey A Clustering Algorithm Using Cellular Learning Automata Based Evolutionary Algorithm R. Rastegar, M. Rahmati, M. R. Meybodi E s t i m a t i n g t h e N u m b e r of C l u s t e r s from D i s t r i b u t i o n a l R e s u l t s of P a r t i t i o n i n g a G i v e n D a t a S e t . . . U. Moller AUDyC Neural Network Using a New Gaussian Densities Merge Mechanism H. A. Boubacar, S. Lecoeuche, S. Maouche

136 140 144 151 155

Self-Organising Maps The Growing Hierarchical Self-Organizing Feature Maps and Genetic Algorithms for Large Scale Power System Security M. Boudour, A. Hellal 3D Self-organizing Convex Neural Network Architectures F. Boudjemai, P. B. Enberg, J. G. Postaire

-^y 164

IX Novel Learning A l g o r i t h m Aiming a t G e n e r a t i n g a Unique Units Distribution in S t a n d a r d S O M . . . . 1 6 8

K. Marzouki, T. Yamakawa SOM-Based Estimation of Meteorological Profiles T. Tambouratzis An Efficient Heuristic for the Traveling Salesman Problem Based on a Growing SOM-like Algorithm C. Garcia, J. Moreno

173 177

Part II — Evolutionary Computation Evolutionary Computation I Evolutionary Design and Evaluation of Modeling System for Forecasting Urban Airborne Maximum Pollutant Concentrations H. Niska, T. Hiltunen, A. Karppinen, M. Kolehmainen Evolving Evolvability: Evolving Both Representations and Operators G. W. Braught A Multi-Objective Evolutionary Algorithm for Solving Traveling Salesman Problems: Application to the Design of Polymer Extruders A. Gaspar-Cunha The Pareto-Box Problem for the Modelling of Evolutionary Multiobjective Optimization Algorithms M. Koppen, R. Vicente-Garcia, B. Nickolay

181 185 189 194

Evolutionary Computation II Implementation and Experimental Validation of the Population Learning Algorithm Applied to Solving QAP Instances J. Jedrzejowicz, P. Jedrzejowicz Estimating the Distribution in an EDA 5. Shakya, J. McCall, D. F. Brown Modelling Chlorine Decay in Water Networks with Genetic Programming P. Jonkergouw, E. Keedwell, S.-T. Khu Evolving Blackjack Strategies Using Cultural Learning D. Curran, C. O'Riordan Two-Criterion Optimization in State Assignment for Synchronous Finite State Machines Using NSGA-II N. Gupta, V. K. Agrawal

198 202 206 210 214

Genetic Algorithms Theory Offspring Selection: A New Self-Adaptive Selection Scheme for Genetic Algorithms 218 M. Affenzeller, S. Wagner Using Genetic Algorithms with Real-coded Binary Representation for Solving Non-stationary Problems 222 J. Kubalik Dynamics in Proportionate Selection 226 A. Agrawal, I. Mitchell, P. Passmore, I. Litovski Generating Grammatical Plant Models with Genetic Algorithms 230 L. E. Da Costa, J.-A. Landry

Complex Systems and Genetic Programming Golomb Rulers: Experiments with Marks Representation J. Tavares, F. B. Pereira, E. Costa Evolving Segments Length in Golomb Rulers J. Tavares, T. Leitao, F. B. Pereira, E. Costa Resource-Limited Genetic Programming: Replacing Tree Depth Limits S. Silva, P. J. N. Silva, E. Costa

235 239 . 243

X

Treating Some Constraints as Hard Speeds up the ESG Local Search Algorithm Y. Kilani, A. Mohdzin

247

Swarm Optimization Applications of PSO Algorithm and OIF Elman Neural Network to Assessment and Forecasting for Atmospheric Quality L. M. Wang, X. H. Shi, M. Li, G. J. Chen, H. W. Ge, H. P. Lee, Y. C. Liang A Hybrid Particle Swarm Optimization Model for the Traveling Salesman Problem T. R. Machado, H. S. Lopes Perceptive Particle Swarm Optimisation B. Kaewkamnerdpong, P. J. Bentley Wasp Swarm Optimization of Logistic Systems P. Pinto, T. A. Runkler, J. M. Sousa A Parallel Vector-Based Particle Swarm Optimizer /. L Schoeman, A. P. Engelbrecht

251 255 259 264 268

Molecular and Quantum Computing Numerical Simulations of a Possible Hypercomputational Quantum Algorithm A. Sicard, J. Ospina, M. Velez Simulated Fault Injection in Quantum Circuits with the Bubble Bit Technique M. Udrescu, L. Prodan, M. Vldduiiu Redundant Quantum Arithmetic A. Pereira, R. Rodrigues A Special Class of Additive Cyclic Codes for DNA Computing T. Abualrub, A. Ghrayeb, X. Zeng

272 276 280 284

Part III — Adaptive and Natural Computing Biolnformatics and Computational Biology I Evolutionary Algorithms for Static and Dynamic Optimization of Fed-batch Fermentation Processes. M. Rocha, J. Neves, A. C. A. Veloso, E. C. Ferreira, I. Rocha Benchmark Testing of Simulated Annealing, Adaptive Random Search and Genetic Algorithms for the Global Optimization of Bioprocesses R. Oliveira, R. Salcedo Dynamic Modelling and Optimisation of a Mammalian Cells Process Using Hybrid Grey-box Systems A. Teixeira, A. Cunha, J. Clemente, P. M. Alves, M. J. T. Carrondo, R. Oliveira Adaptive DO-based Control of Substrate Feeding in High Cell Density Cultures Operated under Oxygen Transfer Limitation R. Oliveira, A. Cunha, J. Clemente, M. J. T. Carrondo Evolutionary Design of Neural Networks for Classification and Regression M. Rocha, P. Cortez, J. Neves

288 292 296 300 304

Biolnformatics and Computational Biology II Pelican - Protein-structure Alignment Using Cellular Automaton Models D. K. Gangadhar An Efficient Algorithm for De Novo Peptide Sequencing S. Brunetti, D. Dutta, S. Liberatori, E. Mori, D. Varrazzo Emergent Behavior of Interacting Groups of Communicative Agents A. Bisler Integrating Binding Site Predictions Using Meta Classification Methods Y. Sun, M. Robinson, R. Adams, A. G. Rust, P. Kaye, N Davey

308 312 316 321

XI Reverse Engineering Gene Networks with Artificial Neural Networks A. Krishna, A. Narayanan, E. C. Keedwell

325

Adaptive Technology in AI A Meta-Level Architecture for Adaptive Applications F. J. Barth, E. S. Gomi Adaptive Finite State Automata and Genetic Algorithms: Merging Individual Adaptation and Population Evolution H. Piston, P. 5. Martins, A. A. de Castro Jr. Modeling a Tool for the Generation of Programming Environments for Adaptive Formalisms A. R. Camolesi Adaptive Decision Tables: A Case Study of Their Application to Decision-Taking Problems T. C. Pedrazzi, A. E. Tchemra, R. L. A. Rocha Robotic Mapping and Navigation in Unknown Environments Using Adaptive Automata M. A. de A. de Sousa, A. R. Hirakawa An Adaptive Framework for the Design of Software Specification Languages J. J. Neto, P. S. M. Silva

329 333 337 341 345 349

Natural Computing Swarm Intelligence Clustering Algorithm Based on Attractor Q. Li, Z. Shi, Z. Shi Ant-based Distributed Optimization for Supply Chain Management C. A. Silva, J. M. Sousa, T. Runkler, J. M. G. Sa da Costa Comparison of Nature Inspired and Deterministic Scheduling Heuristics Considering Optimal Schedules U. Honig, W. Schiffmann An External Memory Supported ACO for the Frequency Assignment Problem A. Acan, A. Gunay

353 357 361 365

Part IV - Soft Computing Applications Machine Learning and Heuristics I N e u r a l N e t w o r k s for E x t r a c t i o n o f F u z z y L o g i c R u l e s w i t h A p p l i c a t i o n t o E E G D a t a . . . . . . . . . . . . . . 3 6 9

M. Holena A New Neuro-Based Method for Short Term Load Forecasting of Iran National Power System 373 R. Barzamini, M. B. Menhaj, S. Kamalvand, M. A. Fasihi Approximating the Algebraic Solution of Systems of Interval Linear Equations with Use of Neural Networks 377 N. H. Viet, M. Kleiber Comparing Diversity and Training Accuracy in Classifier Selection for Plurality Voting Based Fusion 381 H. Altingay

Machine Learning and Heuristics II Visualization of Meta-Reasoning in Multi-Agent Systems D. Rehof, J. Tozicka, P. Slavik Intelligent Agent Inspired Genetic Algorithm C. G. Wu, Y. C. Liang, H. P. Lee, C Lu Combining Lazy Learning, Racing and Subsampling for Effective Feature Selection G. Bontempi, M. Birattari, P. E. Meyer Personalized News Access D. G. Kaklamanos, K. G. Margaritis A More Accurate Text Classifier for Positive and Unlabeled Data R.-M. Xin, W.-L. Zuo

385 389 393 397 401

XII

Machine Learning and Heuristics III Efficiency Aspects of Neural Network Architecture Evolution Using Direct and Indirect Encoding . .. H. Kawasnicka, M. Paradowski Genetic Algorithm Optimization of an Artificial Neural Network for Financial Applications S. Hayward A Method to Improve Generalization of Neural Networks: Application to the Problem of Bankruptcy Prediction A. Vieira, J. C. Neves, B. Ribeiro An Adaptive Neural System for Financial Time Series Tracking A. C. H. Dantas, J. M. Seixas Probabilistic Artificial Neural Networks for Malignant Melanoma Prognosis R. Joshi, C. Reeves, C. Johnston

405 409 417 421 425

Feature Selection and Kernel Methods Boosting Kernel Discriminant Analysis with Adaptive Kernel Selection S. Kita , S. Maekawa, S. Ozawa, S. Abe Product Kernel Regularization Networks K. Petra, S. Terezie Statistical Correlations and Machine Learning for Steganalysis Q. Liu, A. H. Sung, B. M. Ribeiro The Use of Multi-Criteria in Feature Selection to Enhance Text Categorization S. Doan, S. Horiguchi Text Classification from Partially Labeled Distributed Data C. Silva, B. Ribeiro

429 433 437 441 445

Computer Security Recovering the Cyclic-Code of Generated Polynomial by Using Evolutionary Computation K. Li, Y. Li, H. Mo Intrusion Detection System Based on a Cooperative Topology Preserving Method E. Corchado, A. Herrero, B. Baruque, J. M. Saiz Model Selection for Kernel Based Intrusion Detection Systems S. Mukkamala, A. H. Sung, B. M. Ribeiro A Comparison of Three Genetic Algorithms for Locking-cache Contents Selection in Real-Time Systems E. Tamura, J. V. Busquets-Mataix, J. J. Serrano Martin, A. M. Campoy A Binary Digital Watermarking Scheme Based on the Orthogonal Vector and ICA-SCS Denoising... D. Han, W. Li

449 454 458 462 466

Part V — Computer Vision and Pattern Recognition Computer Vision and Image Processing Simulating Binocular Eye Movements Based on 3-D Short-Term Memory Image in Reading S. Morita An Algorithm for Face Pose Adjustment Based on Gray-scale Static Image W. Cao, S. Wang Learning Image Filtering from a Gold Sample Based on Genetic Optimization of Morphological Processing S. Rahnamayan, H. R. Tizhoosh, M. M. A. Salama

470 474 478

Signal Processing and Pattern Recognition Associative Memories and Diagnostic Classification of EMG Signals C. Shirota, M. Y. Barretto, C. Itiki

482

XIII Discretization of Series of Communication Signals in Noisy Environment by Reinforcement Learning. K. Shibata The Research of Speaker-Independent Continuous Mandarin Chinese Digits Speech-Recognition Based on the Dynamic Search Method of High-Dimension Space Vertex Cover W. Cao, X. Pan, S. Wang A Connectionist Model of Finding Partial Groups in Music Recordings with Application to Music Transcription M. Marolt Adaptive ICA Algorithm Based on Asymmetric Generalized Gaussian Density Model F. Wang, H. Li

486 490 494 498

Part VI - Hybrid Methods and Tools Hybrid Intelligent Systems Toward an On-Line Handwriting Recognition System Based on Visual Coding and Genetic Algorithm M. Kherallah, F. Bouri, M. A. Alimi Multi-objective Genetic Algorithm Applied to the Structure Selection of RBFNN Temperature Estimators C. A. Teixeira, W. C. A. Pereira, A. E. Ruano, M. Graga Ruano Assessing the Reliability of Complex Networks through Hybrid Intelligent Systems D. E. Torres, C. M. Rocco Autonomous Behavior of Computational Agents R. Vaculin, R. Nevada Neural Network Generating Hidden Markov Chain J. Koutnik, M. Snorek

502 506 510 514 518

High Performance and Parallel Computing Tools Datamining in Grid Environment M. Ciglaric, M. Pancur, B. Ster, A. Dobnikar Parallel Placement Procedure Based on Distributed Genetic Algorithms M. Yoshikawa, T. Fujino, H. Terai Massive Parallelization of the Compact Genetic Algorithm F. G. Lobo, C. F. Lima, H. Mdrtires Parallel Implementations of Feed-forward Neural Network Using MPI and C # on .NET Platform . .. U. Lotric, A. Dobnikar HeuristicLab: A Generic and Extensible Optimization Environment 5. Wagner, M. Affenzeller The Satellite List: A Reversible Doubly-Linked List C. Osterman, C. Rego, D. Gamboa

522 526 530 534 538 542

XIV ICANNGA International Conference on Adaptive and Natural Computing Algorithms Coimbra, Portugal, March 21-23, 2005

International Advisory Committee R. Albrecht, University of Innsbruck, Austria A. Dobnikar, University of Ljubljana, Slovenia V. Kurkova, Czech Academy of Science, Czech Republic D. Pearson, University Jean Monnet, Saint-Etienne, France B. Ribeiro, University of Coimbra, Portugal N. Steele, Coventry University, UK Programme Committee B. Ribeiro (Chair), University of Coimbra, Portugal R. Albrecht, University of Innsbruck, Austria G. Andrejkova, P.J. Safarik University, Kosice, Slovakia B. Beliczynski, Warsaw University of Technology, Poland H. Bischof, Institute for Computer Graphics and Vision, Graz, Austria L. Borges de Almeida, Technical University of Lisbon, Portugal W. Brauer, Technische Universitaet Miinchen, Germany P. Carvalho, University of Coimbra, Portugal K. Cios, University of Colorado, USA E. Costa, University of Coimbra, Portugal A. Dobnikar, University of Ljubljana, Slovenia M. Dorigo, Universite Libre de Bruxelles, Belgium A. Dour ado, University of Coimbra, Portugal G. Dray, Ecole des Mines d'Ales, France E. Gaura, Coventry University, UK C. Giraud-Carrier, ELCA Informatique SA, Lausanne, Switzerland J. Henriques, University of Coimbra, Portugal K. Hlavackova-Schindler, Inst. of Computer Aided Automation, Vienna, Austria O. Hoshino, Oita University, Japan P. C. Kainen, Georgetown University, USA H. Karatza, Aristotle University of Thessaloniki, Greece M. Karny, Inst. of Information Theory and Automation, Czech Republic V. Kurkova, Institute of Computer Science, Czech Republic P. Larranaga, University of the Basque Country, Spain A. Leonardis, University of Ljubljana, Slovenia T. Martinez, Brigham Young University, USA F. Masulli, University of Pisa, Italy L. Mokhnache, University of Batna, Algeria R. Neruda, Institute of Computer Science, Czech Republic N. Oliveira, University of Coimbra, Portugal N. Pavesic, University of Ljubljana, Slovenia D. W. Pearson, University Jean Monnet, Saint-Etienne, France E. Pelikan, Institute of Computer Science, Czech Republic A. Pinkus, Technion Israel Institute of Technology, Israel J. Principe, University of Florida, USA

XV

V. Rayward-Smith, University of East Anglia, UK C. Reeves, Coventry University, UK S. Ribaric, University of Zagreb, Croatia A. Rosa, Technical University of Lisbon, Portugal A. Ruano, University of Algarve, Portugal M. Sanguineti, University of Cenoa, Italy H. Saxen, Abo Akademy, Finland C. Silva, Polytechnic Institute of Leiria, Portugal F. Silva Leite, University of Coimbra, Portugal J. Sima, Institute of Computer Science, Czech Republic K. Smith, Monash University, Australia N. Steele, Coventry University, UK A. Sung, New Mexico Tech, USA T. Tambouratzis, University of Pireaus, Greece K. Warwick, University of Reading, UK P. Zufiria, University Politecnica de Madrid, Spain

ADFUNN: An Adaptive Function Neural Network Dominic Palmer-Brown, Miao Kang Computational Intelligence Research Group, School of Computing, Leeds Metropolitan University, UK E-mail: {d.palmer-brown, m.kang}@leedsmet.ac.uk

Abstract An adaptive function neural network (ADFUNN) is introduced. It is based on a linear piecewise artificial neuron activation function that is modified by a novel gradient descent supervised learning algorithm. This Af process is carried out in parallel with the traditional Aw process. Linearly inseparable problems can be learned with ADFUNN, rapidly and without hidden neurons. The Iris dataset classification problem is learned as an example. An additional benefit of ADFUNN is that the learned functions can support intelligent data analysis.

1 Motivation True to its name the artificial neuron derives from a joint biological-computational perspective. Taking the sum of weighted inputs is biologically plausible, and adapting a weight is a reasonable model for modifiable synaptic efficacy. But the common assumption that the output activation function is fixed is for computational rather than biological reasons. A fixed analytical function facilitates mathematical analysis to a greater degree than an empirical one. Nonetheless, there are some computational benefits to modifiable activation functions, and they may be biologically plausible as well. Recent work in computational neuroscience is beginning to suggest that neuromodulators play a role in learning by modifying the neuron's activation function [1,2]. From a computational point of view, it is surprising that real neurons could be essentially fixed entities with no adaptive aspect, except at their synapses, since such a restriction leads to non-linear responses typically requiring many neurons. MLPs with an appropriate number of nodes are very effective, but if the activation function is not optimal, neither is the number of hiddens which in turn depends on the function. Training, which is typically slow on linearly inseparable data, always requires hidden nodes. In some cases it may help to adapt a slope-related parameter of the activation function, but not if the analytic shape of the function is unsuited to the problem, in which case many hidden nodes may be required. In contrast, with an adaptive function approach it should be possible to learn linear inseparable problems fast, even without hidden nodes. The simplest test case is the two

input exclusive-OR. A single classic artificial neuron is incapable of solving it, yet by choosing or learning the appropriate activation function, the solution is readily Iearnable with one neuron.

2 Related Approaches to Iris Dataset There are many neural network methods for solving linearly inseparable problems, such as the popular Iris plants dataset. Most of them use a network of multiplayer perceptions (MLPs), so they require hidden neurons. A reinforcement learning method called SANE [3] (Symbiotic, Adaptive Neuro-Evolution) uses an evolutionary algorithm. It searches for effective connections and/or connection weights within a neural network. The dynamic components (i.e. weights and/or architecture) of the neural networks are encoded in structures that form the genetic chromosomes. With the Iris dataset, SANE constructs a network of 5 inputs, 3 hidden and 3 output units, and the transfer function of the network is the sigmoid. The level (averaged over 50 runs) of the learning and generalisation abilities is about 90%. Cantu-Paz [5] also achieved about 90% accuracy with 5 hidden unites, and a spiking neural network with 4 hiddens has achieved more than 96% accuracy [6]. It is possible to solve the problem quite well without a hidden layer, by transforming the input space, but this involves many extra input neurons. For example, Eldracher [4] use several units for each of the four measurements, in a form of coarse coding, and with 90 training and 60 test patterns they achieve an overall misclassification rate of 2.5%.

3 A Single Layer Adaptive Function Network In this paper, we describe how to solve linearly inseparable problems using a simple adaptive function neural network (ADFUNN) which is based on linear piecewise function neurons, and has no hidden layer. 3.1 XOR Experiment The XOR is a simple binary example of linear inseparability, and therefore serves as a good basic test to establish that linearly inseparable problems can be solved by ADFUNN. Two weights are needed for the two inputs

and there is one output. Weights are initialized randomly between -1 and 1, they are then normalized. F point values are also initialized randomly between -1 and 1. Each F point is simply the value of the activation function for a given input sum. F points are equally spaced, and the function value between points is on the straight line between them. This network is adapted as follows:

From the above curve, we can see that when input pattern is (0, 1), the corresponding f(x) = 1.0 where x = 0. 80. Similarly, the other three inputs give the expected correct answer. In the region projected onto between (0, 1) and (1,0), the slope of the activation is nearly 0 and f = 1, and beyond this region, the function slopes down towards 0. Thus, we have a continuous, or fuzzy, XOR.

A = input node activation, E = output node error. Step 1: calculate output error Step 2: adapt weights: Aw = learning rate * A * E wf = w + Aw weights normalization Step 3: adapt function: Af = learning rate * E f = f+Af Step 4: repeat step 1 to step 3 until the output error is close to 0.

3.2 Iris Experiments The iris dataset [7] consists of 150 four dimensional data. Four measurements: sepal length, sepal width, petal length, and petal width, were made by Fisher on 50 different plants from each of three species of Iris (Iris Setosa, Iris Versicolor and Iris Virginica). One class is linearly separable from the other two, but the other two are not linearly separable from each other.

The ADFUNN learns the problem very fast with a learning rate of 0.5. Too low a learning rate makes the network learn very slowly, but too high a learning rate makes the weights and objective function diverge. An example of the weights after learning is: wl = 0. 60, w2 = 0.80, and therefore, the summed inputs are: input pattern input pattern input pattern input pattern

(0, 0): wl (0, 1): wl (1, 0): wl (1, 1): wl

* 0 + W 2 * 0 = 0, * 0 + w2 * 1 = 0.80, * 1 + w2 * 0 = 0.60, * 1 + w2 * 1 = 1.40.

5.0 4.0 3.0 2.0 1.0 -1.0 -2.0 -3.0 -4.0 -5.0

Fig. 1. XOR problem solved using ADFUNN

As can be seen in Fig 1, a characteristic curve of XOR is learned. The line between two borders of the curve marks the learned region, within which adaptation has occurred. The data all projects onto this range, so beyond it none of the points are relevant in the final analysis.

Output

Input wll

Fig. 2. ADFUNN for the Iris dataset experiment

Because the weights are initialized randomly between [-1, 1], the £awj has a known range [-11, 11]. The iris dataset has a precision of 0.1, so this range can be coded with a resolution of 0.1 as the F-point interval. Therefore, 221 points ((ll+ll)/0.1+l=221) are sufficient to encode the data precisely. Clearly, every dataset will have its own range. To cope with this, we need two learning constants, WL and FL. WL depends on input data range and the F point interval, whereas FL just depends on the required output range. We use WL = 0.01, FL = 0.1 to adapt weights and functions respectively. Although we did not do so for the simple XOR case, from basic calculus, we should take into account the fact that we have a variable function slope. Hence, Aw = WL*Fslope*A*error, Thus, the algorithm becomes: Step 1: calculate output error Step 2: adapt weights for each output neuron: Aw = WL*Fslope*A*E w1 = w + Aw weight normalization Step 3: adapt function for each output neuron:

Af=FL*E f = f+Af Step 4: randomly select a pattern in Iris-dataset to train Step 5: repeat step 1 to step 4 until the output error goes to 0 or tends to a steady state.

-6.6 -4.4 -2.2 0|0 J2 -1.0-

figure 9. Comparing figures 3, 4, 5 and 6, 7, 8 we can see that there are only slight differences in the functions, showing that they have solved the problem in approximately the same manner, irrespective of the number of training patterns. The functions are clearly non-linear and non-monotonic, illustrating the value of the adaptive function method which is able to acquire these functions.

8:8 11.

Fig. 3. Iris-Setosa function using ADFUNN

Fig. 6. Iris-Setosa function with 120 training patterns.

Fig. 4. Iris- Versicolor function using ADFUNN A

Fig. 7. Iris-Versicolor function with 120 training patterns.

Fig. 5. Iris- Virginica function using ADFUNN Fig. 8. Iris-Virginica function with 120 training patterns.

4 Results In this case, after 25 epochs, the average error for these three classes tends to a steady state, which is always around 0.065. We ran more than 100 simulations (it learns the task in no more than 30 epochs in each run). The average final error is consistently in the range [0.05, 0.15]. In addition, from the weights for each class, which are consistent across the simulations, we can conclude that, for Iris-Setosa, petal length and petal width are the two most important measurements, whilst for IrisVersicolor, petal length is the most important measurement. The same for Iris-Virginica, petal length is the most important measurement. After many comparisons between the weights for the three classes, we found sepal width is the least important measurement. The problem is learned with 100% successful classification when all patterns are used in training, and also with only 120, whilst 90 patterns were insufficient to achieve complete generalization. Results are tabulated in

Performance Average error Generalisation 90 train 60 test

Best 0.05 (55/60) 93.33%

Generalisation 120 train 30 test

(30/30) 100%

Average 0.10 89.17%

Worst 0.15 (51/60) 85%

95%

(27/30) 90%

Fig. 9. Evaluation of ADFUNN for Iris dataset (100 runs) We can interpret the data by considering weights and functions together. To identify Setosa, both petal width and petal length must be taken into account as they both have equally strong weights. In contrast, Iris-Versicolor and Iris-Virginica are most dependent on petal length, but it is their activation functions which show that Virginica has longer petals.

Setosa Versicolor Virginica

Sepal length

Sepal width

Petal length

Petal width

0.32 0.31 0.43

0.19 0.30 0.38

0.42 0.52 0.45

0.40 0.39 0.31

Fig. 10. Typical weight values for the iris dataset

5 Conclusion and Future Work One of the main conclusions is that an effective ADFUNN can be realized using the following general learning rules: Aw = WL*Fslope*A*E wf = w + Aw (normalized) Af=FL*E f = f+Af Thus, ADFUNN performs supervised learning. In related work, we are exploring unsupervised and reinforcement learning regimes that combine complimentary adaptation rules within a single network: namely, the snap-drift algorithm [8]. It is proposed that snap-drift be followed by a supervised phase acting on the activation functions alone, to perform classification. Related work on connectionist natural language parsers [9, 10] will allow ADFUNN to be compared to MLPs and SRNs on phrase segmentation and recognition tasks, and this may lead to the creation of multilayer ADFUNN. Intelligent data analysis with neural networks requires analysis of the weights to establish the most important factors and generate simplified equations to explain network decisions [11]. In ADFUNN, the learned functions offer insights into the data, in addition to the weights. We will investigate algorithms for replacing learned functions with matched analytical ones to automatically generate mechanistic models from trained networks. In complex domains, there will be a need to use hidden nodes in a ML-ADFUNN to represent arbitrary concave decision functions. However, just as the single layer ADFUNN is more powerful than the SLP, so the multilayer ADFUNN will be more powerful in learning than MLPs, and may well require fewer hidden neurons.

References [1] Scheler, G. (2004) Regulation of Neuromodulator Efficacy - Implications for Whole-Neuron and Synaptic Plasticity. Progress in Neurobiology 72(6)

[2] Scheler, G. (2004) Memorization in a neural network with adjustable transfer function and conditional gating. Quantitative Biology [3] Moriarty, D.E. (1997) Symbiotic Evolution of Neural Networks in Sequential Decision Tasks. Ph.D. thesis, Department of Computer Science. The University of Texas at Austin [4] Eldracher, M. (1992) Classification of Non-LinearSeparable Real-World-Problems Using A-Rule, Perceptions, and Topologically Distributed Encoding. In Proceedings of the 1992 ACM/SIGAPP Symposium on Applied Computing. Vol. I, ACM Press, pp 1098-1104 [5] Cantu-Paz, E. (2003) Pruning Neural Networks with Distribution Estimation Algorithms. GECCO 2003. USA, pp. 790-800 [6] Pavlidis, N.G., Tasoulis, D.K., Plagianakos, V.P., Nikiforidis, G. and Vrahatis, M.N. (2004) Spiking Neural Network training Using Evolutionary Algorithms. International Joint Conference on Neural Networks (IJCNN 2004). Budapest, Hungary [7] Fisher, R.A. (1936) The Use of Multiple Measurements in Taxonomic Problems. Annals of Eugenics 7. pp. 179-188 [8] Lee. S.W., Palmer-Brown. D. and Roadknight, C. M. (2004). Performance-guided Neural Network for Rapidly Self-Organising Active Network Management. Neurocomputing, 2004 (in press) [9] Palmer- Brown, D., Tepper, J. and Powell, H. (2002). Connectionist Natural Language Parsing, Trends in Cognitive Sciences, 6(10) pp.437 - 442 [10] Tepper, J., Powell, H. and Palmer-Brown D. (2002). A Corpus-based Connectionist Architecture for Largescale Natural Language Parsing, Connection Science, 14(2) [11] Roadknight, CM., Balls, G., Mills, G. and PalmerBrown, D. (1997). Modelling Complex Environmental Data. IEEE Transactions on Neural Networks, 8(4): pp.856 - 862

Certain comments on data preparation for neural networks based modelling Bartlomiej Beliczynski Warsaw University of Technology Institute of Control and Industrial Electronics Koszykowa 75, 00-662 Warsaw Email: [email protected] Abstract The process of data preparation for neural networks based modelling is examined. We are discussing sampling, preprocessing and decimation, finally urguing for orthonormal input preprocessing. 1 Introduction In a very first step of input/output modelling of a physical plant one selects several variables named inputs and others outputs. Those variables are functions of time and they are usually called signals. The process of modelling requires expressing of dependencies between the input and output variables. Usual building blocks for modelling consist of specially selected static or linear dynamical systems and universal function approximators. Contemporary, the role of funtion approximators is usually fulfilled by neural networks. Those mentioned preselected systems may be viewed as preprocessors for neural networks [1]. The neural feedforward architecture is know to be universal function approximator. Even with only onehidden layer, it can approximate any function (from a class) with any degree of accuracy, provided that sufficiently large number of hidden units is used, [2]. So a natural question arises: why input preprocessing is needed when the feedforward neural architecture is used? There are several answers to it. Traditionally input preprocessing was used to scale input signals. There were some intuitive and formal arguments for that, but practical results was that with scaled signals, the learning process was significantly faster. This kind of preprocessing is widely used and is implemented via linear transformation. But really important input preprocessing has to be done if the input and output signals have significantly different spectrum and they cannot be effectively sampled with the same sampling time [3]. The third role of input preprocessing is to transform a set of input signals which cannot span desired output

(the output is not described by a function of inputs), into a new set of signals which exactly or approximately can. The input signals are preprocessed thus by a dynamical system. The simplest and the most widely used is a line of delays, others could be considered [1], [4], [5] as well. In this paper data preparation for neural based modelling is considered. We will be examining sampling, preprocessing and decimation processes. The concept of almost linearly dependent signals will be introduced and used. Finally some arguments for orthonormal input preprocessing will be presented. This paper is organized as follows. In Section 2 general scheme of data preparation for neural modelling is presented. In Section 3 almost linearly dependent signals concept is formulated and discussed, then in Section 4 sampling process is examined. Dynamic preprocessing is briefly discussed in Section 5. Finally conclusions are drawn in Section 6. 2 Neural network teaching data preparation Majority of processes modelled by using neural networks are continuous in time. Process variables carrying information about their inputs u(t) and outputs y(t) are continuous-time signals. Throughout this paper we will be considering only one output signal. Continuous-time signals before being used must be sampled with sufficiently high sampling rate. Since that a finite number of samples represent the signals. The process of data preparation for neural networks based input/output modelling is shown in Fig. 1. The ll\ ll' , Y\ U and Y are matrices of numbers. Along each column ofU and Y' matrices the samples of input and output signals are located. Thus, the preprocessing can be applied to the matrix U 'columns resulting with Jj'. The signal samples contained in the columns of U and Y' matrices are characterized by the sampling time T. For U and Y the sampling time is n times larger, where n is a decimation constant. Number of columns in Uf is usually larger than in £/". This is due to pre-

r ' ' } N from the set < (u (i),y (i)) > tion does not exist either.

are deleted, such situa•

3 Continuous-time signals: almost linearly dependent signals Let assume that the system to be modelled is excited by the input continuous signals denoted as u(t), t G [0, tmax] being collection of m measurable signals u(t) =

Fig. 1. The process of data preparation for neural modelling.

(2)

Ui(i) £ £ 2 , i = 1,..., 77i.Similarly assume that the output signal is y(t), y(t) e £ 2 . Let construct the following matrix p — \] c

processing. In order to be able to approximate the relationship between Uf and Y columns by a feedforward neural networks the relationship must be static i.e. be described by a function U' —> Y'. More precisely let U' £ ft"+1'm, Y' G MN+1>1 such that u(0) U =

andY = u(N) _

(i)

L y(N) J

then the set of input/output pairs I (u (i),y (i) should be a function. The role of preprocessing block is to ensure that. At least for some type of preprocessing (for instance line of delays), allowed sampling time is bounded from both sides [6], [7]. Too large and too small sampling time should be avoided. So usually after preprocessing, the samples are taken with much lower sampling rate. This is done via so called decimation process. An important property of this process is that, it does not destroy static relationship between the data set as demonstrated in the following Proposition. Proposition 1 If the relationship between U' and Y is static then the relationship between U and Y is also static. Proof: The U and Y matrices are created from Uf and Y by deleting certain rows from both or in other words deleting certain pairs from the set ^ iv

{ (u (i),y (i))>

f

f

• If relationship between U and Y

J i=0

is static i.e. it is a function, it means that for any i and 3, 0 < i,j < N, there does not exist such situation that u(i) 7^ u (j) —> y (i) — y'(j), so if some pairs

L^-

ii

j

(3} v

H,j=l,...,m

'

where < .,. > denotes scalar product which for C2 space is usually defined as — / '

Ui(t)uj{t)dt,

i,j =

(4) Matrix F c property is used to define linear independence of signals. Is said that the signals iii(t),..., um(t) are linearly independent if matrix (3) is nonsingular i.e.

rank(Tc) = m. In practice signals are measured with errors and computer operations are error prone. If two signal samples are located well within measurement error they can be treated as indistinguishable [6] reducing rank of the matrix F c . Here it is a certain generalization of that concept and further development. Let select only two input signals. Two opposite situations might be distinguished. The first one takes place when two signals are very different in their amplitudes. We may put such case into mathematical framework I \u>211 < < | \u\ 11. In engineering practice U2 is then often neglected. In the second case the signals are the same magnitude say ||iX2|| — ll^ill — 1 a n d thy are very closed one to another, say ||ui — U2II < £» where e is a small number. If ||ui — 1x211 = O7 then u\ = u2 and both of signals are carrying the same information and the one is useless. If however distance between two signals is not zero but a very small, in practice the two signals could be treated as indistinguishable from measurement point of view. One may say e—indistinguishable [8]. For tapped delay neural architecture these two signals fulfill the following condition u2(t) = u\(t + T) where T is a sampling time. Thus for small sampling time u\ (t) and 1x1 (t + T) become indistinguishable. If one considers general input preprocessing involving various dynamic transformations, also u2(t) = —ui(t)

Example 1 Let ui(t) = ^=sin(§?£) and u2(t) = 4= sin(|l L (t 4- T)).where Tp denotes obserwationalperiod and T is the sampling time. Let denote also f = jr.

should be avoided. In such case of course the two signals are distinguishable, but linearly dependent. Let note that 0 whenwi = u2, = 1 2 when < u\, u2 >— 0 4 when wi = — u 2 , < ^1,^2 > = —1

Thus IMOII = ll«2(*)ll = 1, < «i(*W«) >= cos(2?r/T) and the two signals will be linearly independent with margin \ > 1 - 8. Note that for normalized signals ui,u2, the scalar product < u\,u2 > expresses cosines of an angle between u\ and u2. So if two signals are approximately linearly dependent with accuracy 5, it means that the angle (p between the two fulfills condition |cos 1 — If one denotes 4 — 28. ad 2) Because F c matrix is symmetric positive semidefinite then its singular values are the same as its eigenvalues. Eigenvalues of F c matrix in this case are the following Ai)2 = 1 ± |< ui,u2 >| and A max = 1 + |< u\,u2 > | and Amin = 1 — |< u\,u2 > | . Thus Amax > 2 - 8, A min < 8 and cond(Fc) > ^ . And vice versa, starting with the last condition we obtain easily definition property. • One can acknowledge from Proposition 2 that approximately linearly dependent signals will give large condition number of F c matrix. An interesting and important case is when two signals are shifted by sampling time i.e. u2{t) =u\{t + T). Of course, when T is equal to zero both signals are indistinguishable.

u(k) = [iii(/t),...,

i — 1,..., m and k = 0,..., AT. Similarly ' 2/(0) '

(9)

. y(N) _ Let define F^ matrix

rd = [ = i v t l = j^fiUTU and Note that ronA:(£/). The following two Propositions demonstrate that while sampling one can loose linear independence of continuous-time variables. Proposition 3 [8] rank(Td) < rank{Tc) Proposition 4 lim Yd = Tc T—>0

8 Proof: Let distinguish N + 1 points in the range of time [0,t max ] enumerated 0,1,..., AT as in (8). Thus the scalar product (4) can be approximated through sampled signals < Ui,Uj > =

lim j^fiJ2^=oUi(kjf^)uj(k^^-) T~^°0 as N oo.

and note that

5 Dynamic preprocessing A feedforwad neural network being the universal approximator may approximate well some complicated nonlinear functions. But a simple, single input/ single output lag to be approximated by such network - it is an impossible task. In such case a dynamic transformation of the input signals has to be used. This is shown in Fig. 2

G3

ooo

3

Fig. 2. A line of dynamic transformations of the input signal.

Instead of the input signal u only, the signal X — (it, it 1 , u2,..., un) will be an input to the neural net. We hope that now y could well be described as a function of X. In each case however it must be checked that a static relationship between given X and y exists [9]. As an input preprocessing for dynamic systems a line of delays could be used which together with the neural network form so called tapped delay neural architecture. In this case Gi from Fig. 2 is

and T denotes sampling time. Such tapped delay neural architecture has number of performance limitations [6]. A better way of input preprocessing is to introduce allpass transfer functions chain with the following cells 1— 1+jo/Zi

_

e_j2arctan(u;T0

(or its discrete time version). Note that the magnitude of Gi(juj) is always equal to l.The time delay of each cell could now be controlled. There are several reasons why to use input signals orthonormalization, so the signals from the blocks shown in Fig. 2 are mutually orthogonal i.e. < it*, wJ > = O.But from our considerations, it comes out directly that according to formula (6), for two signals orthonormality

ensures the largest angular distance (i.e. §— 0

10

A A / \MAA 20

30

M - .VI

Ar

AAA 50

40

2

60

70

V

80

90

80

90

100

A

AWU

0

/\

A

-2 0

-

60

A

0 -2

x"

M A i A' vAAV » \

2

10

20

JW

30

A,

A fliAi

/^/ V 40

v^

50

/V

\Aj V

60

70

V

100

WA

Figure 4 Example on the model fit on the test data using five non-zero weights in the lower part of the network ( observations, model).

Figure 2. Inputs and output in a training set of the reference case.

Remaining lower-layer weights Remaining lower-layer weights

Figure 3. Evolution of the errors with the number of remaining weights (excluding biases) in the lower part of the network ( training set, — test set).

Figure 5. Evolution of the training errors with the number of remaining weights (excluding biases) in the lower part of the networks from four random weight matrices.

Figure 5 shows the errors on the training set for networks evolved from four random weight matrices. Even though the levels of the errors are seen to differ between the networks, the general features are very similar, especially as far as the minimum connectivity for achieving a good fit is concerned. A noteworthy fact is also that the fits for the largest networks are worse than those of the somewhat pruned networks, even on the training set. Thus, superfluous lower-layer connections can be directly detrimental for the model. This is the result of the fact that the lower-layer weights are not retrained, but it still

3.2 Detection of relevant inputs The capability of the model to detect and remove irrelevant inputs is next evaluated by creating a data set with identical parameters to those of the previous subsection, except a = 0.1. This gives rise to a function y that only slightly depends on x\. The algorithm was used to detect the one of the inputs that would be first eliminated, i.e., would lose all its connections to the hidden layer.

12 Starting from 20 random initial weight matrices, Wo, the first input variable to be eliminated was always x\. Table 1 shows the "frequency" of the number of remaining weight connections at the point where the final connection to JCI was excluded. Even though there is some scattering, it is interesting to note that the required lowerlayer complexity at this point corresponds quite well to the one required for solving the task with a = 0.5. Another interesting observation is that the generalization ability of the networks at these very points turns out to be close to optimal for many of the runs. Figure 6 illustrates this behavior for four of the runs, where the test set errors have been depicted. The arrows in the figure indicate the network complexity where the first input variable (x\) was excluded from the model. Moving from left to right, these are the points where a further added complexity does not improve the function approximation provided by the network. Table 1. Frequency of remaining lower-layer weights at the point where the first input (x\) was eliminated by the pruning method for the function (3) with a = 0.1. Remaining weights 3 4 5 6 7 8 9

Frequency 2 5 4 6 1 1 1

4 Conclusions The paper has described a systematic method for selection of both input variables and a constrained connectivity of the lower-layer weights in MLPs. It also provides initial guesses for a gradient based training. The method has been illustrated on a class of test problems, where it has shown promising performance.

REFERENCES [I] Principe J. C, N. R. Euliano and W. C. Lefebvre, (1999) Neural and adaptive systems: Fundamentals through simulations, John Wiley & Sons, New York. [2] Sridhar, D.V., E. B. Bartlett and R. C. Seagrave, (1998) "Information theoretic subset selection for neural networks", Comput Chem. Engng. 22, 613-626. [3] Bogler, Z., (2003) "Selection of quasi-optimal inputs in chemometrics modeling by artificial neural network analysis", Analytical Chimica Acta 490, 31-40. [4] Sarle, W.S., (2000) "How to measure importance of inputs", ftp://ftp.sas.com/pub/neural/importance.html [5] Frean, M., (1989) "The Upstart Algorithm. A method for Constructing and Training Feed-forward Neural Networks", Edinburgh Physics Department, Preprint 89/469, Scotland. [6] Fahlman, S.E. and C. Lebiere, (1990) "The CascadeCorrelation Learning Architecture", in Advances in Neural Information Processing Systems II, (Ed. D.S. Touretzky), pp. 524-532. [7] Le Chun, Y., J. S. Denker and S. A. Solla, (1990) "Optimal Brain Damage", in Advances in Neural Information Processing Systems 2, ed. D.S. Touretzky, pp. 598-605, (Morgan [8] Thimm, G. and E. Fiesler, (1995) "Evaluating pruning methods", Proc. of the 1995 International Symposium on Artificial Neural Networks (ISANN'95), Hsinchu, Taiwan, ROC.

lowe r—(sycr wdQhts

Figure 6. Test set errors of four networks trained on y generated by Eq. (3) with a = 0.1. Arrows denote the points where JCI was excluded from the model.

[9] Maniezzo, V., (1994) "Genetic Evolution of the Topology and Weight Distribution of Neural Networks", IEEE Transactions on Neural Networks 5, 39-53. [10] Gao, F., M. Li, F. Wang, B. Wang and P. Yue, (1999) "Genetic Algorithms and Evolutionary Programming Hybrid Strategy for Structure and Weight Learning for Multilayer Feedforward Neural Networks", Ind. Eng. Chem. Res. 38, 4330-4336. [II] Pettersson, F. and H. Saxen, (2003) "A hybrid algorithm for weight and connectivity optimization in feedforward neural networks", in Artficial Neural Nets and Genetic Algorithms (eds. Pearson, D. et al.), PP- 47-52, SpringerVerlag. [12] Golub, G., (1965) "Numerical methods for solving linear least squares problems", Numer. Math. 7, 206-216.

13

The Concept and Properties of Sigma-if Neural Network M. Huk, H. Kwasnicka Department of Computer Science, Wroclaw University of Technology, Poland E-mail: {maciej.huk, halina.kwasnicka}@pwr.wroc.pl Abstract Our recent works on artificial neural networks point to the possibility of extending the activation function of a standard artificial neuron model using the conditional signal accumulation technique, thus significantly enhancing the capabilities of neural networks. We present a new artificial neuron model, called Sigma-if, with the ability to dynamically tune the size of the decision space under consideration, resulting from a novel activation function. The paper discusses construction of the proposed neuron as well as training Sigma-if feedforward neural networks for well known sample classification problems. 1 Introduction The basic constituent of a classic artificial neural network (ANN) is the neuron, used to process signals presented as input, through an activation function and a nonlinear output (threshold) function. The former function, hereafter referred to as A, determines the activation level of the neuron, while the latter (F) bases on the result returned by A to construct the neuron's output value. [1,2] The importance of the threshold function as well as of the weights attached to individual interneural connections is well discussed in numerous publications. [1,3,4,5] However almost all of them assume the activation function to be a simple weighted sum of neuron input values and connections weights. In fact, analyzing the behavior of artificial neurons with nonstandard activation functions may lead us to develop new classification structures, with interesting and useful properties. [6-13] 2 Sigma-if Neuron The authors propose a special type of neuron, whose activation function can be clearly interpreted from a biological perspective. In the case of real neurons, individual dendrites differ in length, allowing a biological neural network to associate incoming signals with particular connections and processing areas. Such a property can be incorporated into classic feedforward neural network models through connections grouping and conditional signals accumulation technique. [11,14] More specifically, the M dendrites of a Sigma-if neu-

ron are divided into K distinct groups, by complementing each i-th input connection with an additional integer parameter 0;€ {0,1,..,lf-1}, determining membership in one of the groups. This allows us to divide the process of signals accumulation into K steps, where K is a function of neuron's grouping vector 8T=\6\, #2>..., 0MV-

K{0) =

(1)

During each step k (from 0 to KA) the neuron accumulates data belonging to one selected group, such that 0% — k.

(2)

Within each k-th group, partial activation Ag(k) is determined as a weighted sum of input signals and the appropriate Kronecker's delta: M

, 0) = where Wi and X{ are coefficients of the neuron's weight vector w and input vector x. This process is repeated until the activation derived from respective groups exceeds a preselected activation threshold net*. It can be described by the following recursive formula (vectors w, JC and 8 are omitted for clearness):

,(Ag(k)H(net*-net(k-l))+net(k-l)

: fc>0

M

:fc R from scattered samples (xt; y{ - f(x)) i=l n, where: S we have little a priori knowledge on the unknown function f which lives in some infinite dimensional smooth function space, S the function approximation process is performed iteratively: each new measure on the function (x;; f(Xj)) is used to compute a new estimate f as an approximation of the function f. Simulation results are demonstrated to validate the generalization ability and efficiency of the proposed Beta wavelet network.

can be regarded as a function approximator which estimates an unknown functional mapping: y =f(x) +£ (2) where / i s the regression function and the error term s is a zero-mean random variable of disturbance. There are a number of approaches to WNN construction (a brief survey is provided in [9-12] ), we pay special attention on the model proposed by Zhang [1, 6, 8].

2 The Beta wavelet The Beta function [13] is defined as: if p>0, q>0, (p, q) e IN r

P(x) =

Xc-XO

(3)

1 Introduction

else

Combining the wavelet transform theory with the basic concept of neural networks [1-3], a new mapping network called wavelet neural network or wavenets (WNN) is proposed as an alternative to feedforward neural networks for approximating arbitrary nonlinear functions. Kreinovich proved in [14] that if we use a special type of neurons (wavelet neurons), then the resulting neural networks are optimal approximators in the following sense: as e —> 0, the number of bits that is necessary to store the results of a 3-layer wavelet neural network approximation, increases slower than for any other approximation scheme. Wavelets occur in a family of functions and each is defined by dilation at which controls the scaling parameter and translation tt which controls the position of a single function, named the mother wavelet \|/(x). Mapping functions to a time-frequency phase space, WNN can reflect the time-frequency properties of function more accurately than the RBFNN. Given an nelement training set, the overall response of a WNN is: y(w) = wo +

i

\xo,x\

}

(1)

where Np is the number of wavelet nodes in the hidden layer and wt is the synaptic weight of WNN. A WNN

where, xc = P + 9 2.1. The derivatives of Beta function We proved in [4, 5] that all derivatives of Beta function e L2(IR) and are of class C00. The general form of the nth derivative of Beta function is: "W

dxn

where: px(x)=-I.

SL.

x-xo

and

x\-x n\p n Pn(x)=(-l) r

n\q

The first (BW1), second (BW2) and third (BW3) derivatives of Beta wavelet are shown graphically in Figure 1.

19

Fig.2. Result of interpolation by Mexican hat.

Fig, 1. First, second and third derivatives of Beta function.

2.2. Proposition if p = q , for all n e IN and 0 < n < p the functions

dxn

are wavelets [4, 5].

(5)

3 Experiments In this section, we present two experimental results of the proposed Beta Wavelet Neural Networks (BWNN) on approximating two functions using the Stepwise selection by orthogonalization training algorithm. First, simulations on the 1-D function approximation f(x)=:0.5xsin(2x)-^cos2(2x) are conducted to validate and compare the proposed BWNN with some other wavelets. The input x is constructed by the uniform distribution on [-2.5, 2.5], and the corresponding output y is functional of y = f(x) and is artificially contaminated by random errors. The training and test data are composed of 50 points and 500 points, respectively. Beta wavelet is chosen as the mother wavelet for training network. Second, the two-dimension function: ~[{x\ 16

- 0.5)2 + ( x2 - 0.5)2]

Fig.3. Result of interpolation by BW2.

Approximated functions are displayed in Figures 2 and 3. The Normalized Root Mean Square Error NRMSE of the Mexican hat WNN is 0.0138434 compared to 0.009344716 the BW2 WNN achieved. From these simulations we can deduce the efficiency of Beta wavelet in term of function interpolation. The table below gives the normalized square root mean square error and mean square error using traditional wavelets and Beta wavelet: Stepwise selection by orthogonalization algorithm Wavelets

IS

approximated to illustrate the robustness of the proposed wavelets family. The training set D contains 11x11 uniform spaced points, and 1 lxl 1 stochastic points. The test set V is constructed by evenly spaced 21x21 grid on [-1.1M-1.1]. 3.1 1-D interpolation using the stepwise selection by orthogonalization algorithm These results are given, using the Stepwise selection by orthogonalization algorithm, on a Neural Wavelet Networks using 9 wavelets, 4 levels decomposition, 500 iterations, 50 points for training and a uniform spaced points. f(x)=0.5xsin(2x)+cos2(2x).

NSRMSE

MSE (e-005)

Mexican hat 0.0138434 Beta 1 0.0295078 Beta 2 0.00934716 Table. 1 Comparison of NSRMSE and MSE for and some others in term of 1 -D approximation.

10.7061 48.6430 4.88097 Beta wavelets

3.2 1-D interpolation of noisy data using the stepwise selection by orthogonalization algorithm These results are given, using on a Neural Wavelet Network using 9 wavelets, 4 levels decomposition, 500 iterations, 50 points for training, uniform spaced points.

J{x)=0.5xsin(2x)+cos2(2jc)+4x)

(6)

20

Fig.5. Result of interpolation of noisy data by Mexican hat wavelet.

Fig.8. Result of 2-D interpolation by Mexican hat wavelet after training using uniform spaced input patterns.

Fig.6. Result of interpolation of noisy data by BW2 wavelet.

Fig.9. Result of 2-D interpolation by BW2 wavelet after training using uniform spaced input patterns.

We display in figure 5 the result of approximation of a noisy signal using Mexican hat WNN and in figure 6 the Beta WNN one. From these simulations we can see that Beta 2 WNN is more efficient than the Mexican hat wavelet on noisy data approximation. 3.3 2-D interpolation using the stepwise selection by orthogonalization algorithm These results are given, using the Stepwise selection by orthogonalization algorithm on a Neural Wavelet Networks using 4 wavelets, 4 levels decomposition, 200 iterations, 11 xl 1 points for training. ° O1 -fi[(xl-0.5) 2 +(;c2-0.5) 2 ] (*. f(xi Xl) = r (7)

F 10 f - R e s u l t o f 2 " D ' " R a t i o n by Mexican hat wavelet after training using randomly input patterns.

Fig.l 1. Result of 2-D interpolation by BW2 wavelet after training using randomly input patterns. Fig.7. 2-D data to be interpolated.

21 We display in Figure 7 the 2-D data we use in our tests, in figure 8 the result of 2-D interpolation using Mexican hat WNN on which we see some distortion in amplitude and at the edges. In figure 9 using Beta 2 WNN we reduce the amplitude distortion. In figure 10 we display the result of interpolation using Mexican hat WNN on which distortion becomes greater than its homolog using Beta 2 WNN displayed in figure 11.

4 Conclusion We present two experimental results of the proposed Beta Wavelet Neural Networks (BWNN) on approximating two functions using the Stepwise selection by orthogonalization training algorithm. First, simulations on the 1 -D function approximation on which we prove the superiority of Beta wavelets in term of NSRMSE. Second, the two-dimension function is approximated with the second derivative of Beta wavelet and the Mexican hat wavelet to illustrate the robustness of the proposed wavelets family. The training set D contains 11x11 uniform spaced points, and 11x11 random points, the test set V is constructed by evenly spaced 21x21 grid on [-1, l]x[-l, 1]. So the new Beta wavelets family has the superiority of approximation in the 1-D and the 2-D case. This new wavelet family can be used to approximate volume using the 2-D 1-D 2-D technique. A cknowledgmen t The authors would like to acknowledge the financial support of this work by grants from the General Direction of Scientific Research and Technological Renovation (DGRSRT), Tunisia, under the ARUB program 01/UR/l 1/02.

References [1] Q. Zhang, (1997) Using Wavelet Network in Nonparametric Estimation, IEEE Trans. Neural Network, Vol. 8, pp.227-236. [2] S.T. Li and E. Leiss, (2001) On Noise-immune RBF Networks, in Radial Basis Function Neural Networks: Recent Developments in Theory and Applications, Editors: Howlett, R. J. and Jain, L.C., Springer Verlag, pp. 95-124. [3] P. Cristea, R. Tuduce, and A. Cristea, (2000) Time Series Prediction with Wavelet Neural Networks, Proceedings of IEEE Neural Network Applications in Electrical Engineering, pp. 5-10. [4] W. Bellil, C. Ben Amar et M. A. Alimi, (2003) Beta Wavelet Based Image Compression, International Conference on Signal, System and Design, SSD03, Tunisia, Mars, vol. 1, pp. 77-82.

[5] W. Bellil, C. Ben Amar, M. Zaied and M. A. Alimi, (2004) La fonction Beta et ses derivees: vers une nouvelle famille d'ondelettes, First International Conference on Signal, System and Design, SCS'04, Tunisia, Mars, vol. 1, PP. 201-207. [6] Q. Zhang and A. Benveniste, (1992) Wavelet Networks, IEEE Trans, on Neural Networks 3 (6),PP 889-898. [7] J. Zhang, G. G. Walter, Y. Miao and W. N. Wayne Lee, (1995) Wavelet Neural Networks For Function Learning, IEEE Trans, on Signal Processing 43 (6), PP1485-1497. [8] Q. Zhang (1997) Using Wavelet Network in Nonparametric Estimation. IEEE Trans, on Neural Networks, Vol. 8, No. 2, pp. 227-236. [9] Y. Oussar, I. Rivals, L. Personnaz (1998) Training Wavelet Networks for Nonlinear Dynamic InputOutput Modeling Neurocomputing. [10] C.S.Chang, Weihui Fu, Minjun Yi, (1998) Short term load forecasting using wavelet networks Engineering Intelligent Systems for Electrical Engineering and Communications 6, 217-23. [11] J.Echauz, (1998) Strategies for Fast Training of Wavelet Neural Networks, 2nd International Symposium on Soft Computing for Industry, 3rd World Automation Congress, Anchorage, Alaska, May 10-14, 1-6. [12] V.Kruger, Happe, A., Sommer, G (1999) Affine real-time face tracking using a wavelet network Proc. Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems, IEEE, 1418. [13] M. A. Alimi, (2000) The Beta System: Toward a Change in Our Use of Neuro-Fuzzy Systems. International Journal of Management, Invited Paper, pp. 15-19. [14] V. Kreinovich, O. Sirisaengtaksin. S. Cabrera, (1992) Wavelets compress better than all other methods: a 1-dimensional theorem. University of Texas at El Paso, Computer Science Department, Technical Report.

22

Speeding up backpropagation with Multiplicative Batch Update Step Pedro Cruz Department of Mathematics, University of Aveiro, Portugal E-mail: [email protected] Abstract Updating steps in a backpropagation neural network with multiplicative factors u > 1 and d < 1 has been presented by several authors. The istatistics field of Stochastic Approximation has a close relation with backpropagation algorithms. Recent theoretical results in this field show that for functions of one variable, different values of u and d can produce very different results: fast convergence at the cost of a poor solution, slow convergence with a better solution, or produce a fast move towards a solution but without converging. To speed up backpropagation in a simple manner we propose a batch step adaptation technique for the online backpropagation algorithm based on theoretical results on simple cases. 1 Introduction Speeding up backpropagation has been a constant challenge and several techniques has been applied like using second order information [4]. Also several works on multiplicative step update where developed [5,6,8] and each of them uses the common update algorithm It =

It-i It-i

if condition C otherwise

where u > 1 and 0 < d < 1 are real parameters. Possibly variable j t is guaranteed to be limited 0 < Yd < it < r u . Condition C could be a gradient condition [5] or a condition on the quadratic error [6,8]. In both cases the quadratic error Et = E(Wt,x,d)

then step must be incremented otherwise reduced. Different updates are going to occur for each weight w in the net parameters W = (w0, wi,...,wn). Another type of condition is the error condition which is based on increases or decreases of global error AEt causing the step to change. We propose an algorithm that uses a gradient condition for step update. Next we present theoretic results that are guides to the proposed algorithm. 2 Stochastic Approximation and New results Many problems in Stochastic Approximations are described as the following. Consider the problem of searching for the zero point of a function, according to the stochastic approximation procedure

Vt =

i l

(2) is to be minimized, with (x, d) being the pattern presented at the iteration, Wt is the weight vector that describes net state, and NET; is the ith output of the neural network. Gradient condition uses change of signs: if SEt (3) Sw Sw

(4) (5)

where