Mechatronics and Machine Vision in Practice

John Billingsley • Robin Bradbeer (Eds.) Mechatronics and Machine Vision in Practice John Billingsley • Robin Bradbeer...

Author: John Billingsley | Robin Bradbeer

168 downloads 1712 Views 10MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

John Billingsley • Robin Bradbeer (Eds.) Mechatronics and Machine Vision in Practice

John Billingsley • Robin Bradbeer (Eds.)

Mechatronics and Machine Vision in Practice

With 245 Figures

123

Prof. Dr. John Billingsley Faculty of Engineering and Surveying University of Southern Queensland Toowoomba, QLD Australia [email protected]

Prof. Dr. Robin Bradbeer Department of Electrical Engineering City University of Hong Kong 88 Tat Chee Avenue Kowloon, Hong Kong P.R. China [email protected]

ISBN 978-3-540-74026-1

e-ISBN 978-3-540-74027-8

DOI 10.1007/978-3-540-74027-8 Library of Congress Control Number: 2007933848 © 2008 Springer-Verlag Berlin Heidelberg This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable for prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Cover design: Erich Kirchner, Heidelberg Printed on acid-free paper 987654321 springer.com

Foreword

Research papers on the subject of mechatronics cover a great variety of topics. Among them are those that explore new techniques and applications, but all too often there are those others that seek to paint old, tired techniques with a patina of new jargon. You will find none of the latter here. There is a heavy emphasis of the ‘in Practice’ that completes the title of the conference series from which these papers have been drawn. The papers were originally reviewed as full manuscripts and now a selection of authors have been invited to rewrite their work for inclusion in this volume. In the first section, papers with an educational theme have been gathered. Most of them focus on practical experiments that will reinforce a mechatronics course. A variety of techniques for vision analysis form the next section, again stressing a practical emphasis. The third section focuses on practical applications of machine vision, several of which have been implemented in industry, while the fourth is concerned with techniques within robotics other than vision. Some of the medical applications of the fifth section might not be for the squeamish. The book is completed with a number of applications that have an agricultural theme. University of Southern Queensland, Australia City University, Hong Kong

John Billingsley Robin Bradbeer November 2007

Contents

Education Emergent Behaviour Real-time Programming of a Six-Legged Omni-Directional Mobile Robot: Planning of Viennese Waltz Behaviour ..... Frank Nickols

3

The Hong Kong Underwater Robot Challenge ............................................... Robin Bradbeer

17

Dynamics and Control of a VTOL Quad-Thrust Aerial Robot....................... Joshua N. Portlock and Samuel N. Cubero

27

Project-oriented Low Cost Autonomous Underwater Vehicle with Servo-visual Control for Mechatronics Curricula................................... C. A. Cruz-Villar, V. Parra-Vega, and A. Rodriguez-Angeles Coordination in Mechatronic Engineering Work............................................ James Trevelyan

41 51

Vision Techniques A Vision System for Depth Perception that Uses Inertial Sensing and Motion Parallax........................................................................................ Vlatko Bečanović and Xue-Bing Wang Rate Shape Identification Based on Particle Swarm Optimization ................. P.W.M. Tsang and T.Y.Y. Yuen

65 77

Advanced 3D Imaging Technology for Autonomous Manufacturing Systems .................................................................................. A. Pichler, H. Bauer, C. Eberst, C. Heindl, J. Minichberger

87

Vision Based Person Tracking and Following in Unstructured Environments ........................................................................ Mahmoud Tarokh and John Kuo

99

VIII

Contents

Simple, Robust and Accurate Head-Pose Tracking Using a Single Camera............................................................................................... Simon Meers, Koren Ward and Ian Piper

111

Vision Applications Machine Vision for Beer Keg Asset Management .......................................... Michael Lees, Duncan Campbell, Andrew Keir Millimetre Wave Radar Visualisation System: Practical Approach to Transforming Mining Operations ................................................................ E. Widzyk-Capehart, G. Brooker, S. Scheding, A. Maclean, R. Hennessy, C. Lobsey and M. Sivadorai

125

139

An Underwater Camera and Instrumentation System for Monitoring the Undersea Environment .............................................................................. Kenneth K.K. Ku, Robin Bradbeer and Katherine Lam

167

Visual Position Estimation for Automatic Landing of a Tail-Sitter Vertical Takeoff and Landing Unmanned Air Vehicle.................................... Allen C. Tsai, Peter W. Gibbens and R. Hugh Stone

181

Minutiae-based Fingerprint Alignment Using Phase Correlation.................... Weiping Chen and Yongsheng Gao

193

Robotic Techniques A Snake-like Robot for Inspection Tasks ........................................................ Bin Li, Li Chen and Yang Wang

201

Modelling Pneumatic Muscles as Hydraulic Muscles for Use as an Underwater Actuator .............................................................................. Kenneth K.K. Ku and Robin Bradbeer

209

Automated Tactile Sensory Perception of Contact Using the Distributive Approach................................................................................ X. Ma, P. Tongpadungrod and P.N. Brett

219

Blind Search Inverse Kinematics for Controlling All Types of Serial-link Robot Arms ............................................................................... Samuel N. Cubero

229

Contents

IX

Distributive Tactile Sensing Applied to Discriminate Contact and Motion of a Flexible Digit in Invasive Clinical Environments.................................... Betty Tam, Peter Brett, David Holding, and Mansel Griffiths

247

Medical Applications

Intelligent Approach to Cordblood Collection................................................ S.L. Chen, K.K. Tan, S.N. Huang and K.Z. Tang

255

An Autonomous Surgical Robot Applied in Practice ..................................... P.N. Brett, R.P. Taylor, D. Proops, M.V. Griffiths and C. Coulson

261

Development of an Intelligent Physiotherapy System .................................... S.L. Chen, W.B. Lai, T.H. Lee and K.K. Tan

267

Visual Prostheses for the Blind: A Framework for Information Presentation .................................................... Jason Dowling, Wageeh Boles and Anthony Maeder Computer-based Method of Determining the Path of a HIFU Beam Through Tissue Layers from Medical Images to Improve Cancer Treatment............................................................................................ E. McCarthy and S. Pather

275

289

Agricultural Applications On-the-go Machine Vision Sensing of Cotton Plant Geometric Parameters: First Results .................................................................................................... Cheryl McCarthy, Nigel Hancock and Steven Raine

305

Robotics for Agricultural Systems .................................................................. Mario M. Foglia, Angelo Gentile, and Giulio Reina

313

More Machine Vision Applications in the NCEA .......................................... John Billingsley

333

Authors ..........................................................................................................

345

Index...............................................................................................................

347

Education

Mechatronics education has been a constant source of interest to nearly all engineering conferences, and the first chapter in this book is dedicated to this most important subject. The first paper discusses a very original idea from one of the more innovative pioneers of mechatronics education. This looks at the movement and control of a six legged omni-directional mobile robot, and determines how it can be programmed to perform the movements needed for a Viennese waltz. This is followed by something completely different! A description of an underwater robot competition held in Hong Kong, mainly for school students. The winners would attend the World Championships in the USA. From under the sea into the air; the next paper describes an aerial robot propelled by four rotors. It contains details of the control system needed to keep the platform stationary, and in position. It’s a pity that videos can’t be shown in books (yet?) as the demonstration of this robot is very impressive. Then it's back underwater again, with a paper from Portugal about a small project-based vehicle which is used to teach the basics of mechatronics syllabus. Finally, James Trevelyan has an interesting opinion piece concerning mechatronics engineering work, and how some sort of coordination between those institutions offering mechatronics courses may be beneficial.

Emergent Behaviour Real-time Programming of a Six-Legged Omni-Directional Mobile Robot: Planning of Viennese Waltz Behaviour

Frank Nickols Mechanical Engineering Dept. Dhofar University, Salalah, Sultanate of Oman, Middle East.

1

Introduction

Viennese waltz behaviour means simultaneous translation and rotation of a robot body when viewed from above. Emergent behaviour is concerned with how complex motion is computed in robots. The author has designed a six-legged, omnidirectional robot, Fig. 1, in order to investigate emergent behaviour real-time algorithms. The robot is programmed to carry out behaviour patterns using nine on-board, parallel processing, 16-bit microcomputers. The group of microcomputers is programmed to control independently the leg step amplitude and step angle for each of the robot’s six legs. The Viennese waltz is a very good starting point to demonstrate emergent behaviour due to, (i) its usefulness and, (ii) understanding the complexity in programming an omnidirectional, six-legged robot to carry out such motion. The property of omnidirectionality is imparted to the robot body due to each robot leg possessing three degrees of freedom (3dof) that are actuated by three servos. One microcomputer is dedicated to each leg in order to compute the displacement of

Fig. 1. Six-legged, 3dof each leg, omni-directional robot designed to investigate emergent behaviour algorithms

4

Frank Nickols

Fig. 2. Planar sheet of elemental mathematical functions, (emf’s), computing one output value as a function of inputs and coefficients. The idea is to produce continuous computation

each of the three servos. The remaining three microcomputers compute the Viennese waltz motion. This paper is the first of three papers concerned with (i) the planning (this paper), (ii) the strategy [1] and, finally, (iii) the real-time computational implementation (to be written) of Viennese waltz behaviour into the six-legged omnidirectional walking robot. The three papers are concerned with emergent behaviour and how complex robot motion behaviour patterns are computed from flat sheet standardised computer architectures, Fig. 2. The computer architecture uses a minimum number of elementary mathematical functions, (emf’s), which are multiplication, division, addition and subtraction. The four emf’s enable low level elementary computing that can be executed quickly and give the promise of realtime computing with minimum computing power. The challenge is how to carry out complex mathematical equations modeled with just these four emf’s. The solution will be described in the third and final of the papers which is currently work in progress. The robot body can be thought of as a mobile Stewart platform where, alternately, three legs of the six legs are used to locomote the body. It is known that the solutions of the servomechanisms that displace such a platform require significant computational power to solve in real-time (in this case every 20 msecs), e. g. [2]. The reader can see this if the following points are appreciated. 1. The robot has six legs. 2. Each leg is actuated by three servos so there are 18 servos. 3. Each servo is to be updated with a new angle demand at a rate of 50 times per second to place each leg tip in the correct 3dof cylindrical coordinate, (R, θ, Z). 4. The inverse kinematics equations for the leg tip placement are non-linear and complex. So there are 18 servos × 50 equations/second-servo = 900 complex inverse kinematics equations to be solved every second, i. e. one equation to be solved approximately every 1 msec.

Six-Legged Omni-Directional Mobile Robot: Planning of Viennese Waltz Behaviour

5

5. Each leg tip must follow a plane rectangular locus that gives locomotion. This locus needs additional computation power and to make things worse, the locus is subject to change both in height and length and worse still the locus has to be a curved plane whose curvature is variable when the robot is turning a corner or rotating. 6. The leg tips should be able to keep the body of the robot level whilst walking on uneven terrain. 7. The leg gait should be able to walk and change between any of the gaits, i. e., 1 leg up 5 legs down gait, 2 legs up 4 legs down, or 3 legs up 3 legs down, i. e. double tripod gait. 8. The body of the robot should possess 6dof motion albeit with limited displacement about some axes. In other words the body of the robot should be able to rotate about an instantaneous axis of rotation located at any position and at any direction in 3dof space. The rotation is continuous or limited dependent on the leg tip limited range of motion. For example, rotation can be continuous if the axis of rotation is perpendicular to the ground plane because the robot can keep walking, e. g. continuous rotation on the spot about a central vertical axis. However, if the axis of rotation is parallel to the ground plane then the robot can only rotate by a limited amount about that axis, e. g. limited roll left/right, limited pitch up/down, limited yaw left/right. Thus Viennese waltz behaviour presents an interesting challenge and even more so because, in the interests of computational efficiency, the author wishes to use no trig functions, no inverse trig functions, no square root, no log functions, no piece-wise linear functions, no look-up tables, no floating point numbers, no callable subroutines, no “if-then-else” instructions and no functions that are enclosed in parentheses. Instead, a significant number of coefficients are used together with 16 bit/32 bit integer numbers and integer power series to replace traditional mathematical functions. The advantages of such computation relate to efficiency and standardisation of computing hardware which can be designed to implement emergent behaviour. Such behaviour can be the Viennese waltz or many other behaviour patterns all of which emerge from an identical computational structure but with different coefficients. Figure 2 shows that the emf’s are arranged in a regular standardised physical layout such that information flows, wave-like, through a parallel pipe-lining processing architecture from one end of the layout to the other resulting in one computed output value at the other end. It is to be noted that the equation being computed by the standardised planar sheet of emf’s is changed by changing the values of the coefficients and not by changing the wiring or format of emf’s because this is a fixed standardised sheet. Biological systems may work like this which is quite different from digital computers that solve mathematical equations with a serial stepping computer program. There should be an efficient method by which biological systems elegantly compute behaviour patterns utilising a minimum number of elemental mathematical functions, (emf’s). Furthermore Nature would have evolved a computational architecture that is common to all biological computational processes. For example the biological computational process for (i) processing vision information from a retina, (ii) processing acoustic information from a cochlea, (iii) causing the heart

6

Frank Nickols

to beat, (iv) the gut muscles to massage food through the digestive system and (v) solving inverse kinematics equations for muscle contraction that enable limb movement and body locomotion. In fact the driving force behind this research is to show if it is possible that a standardised, generic, biological-like, parallel processing computer architecture can be used to solve most robot computational problems more efficiently than a serial digital computer. Inverse kinematics equations for the computation of each 3dof robot leg system have already been worked out. These equations will be described in forthcoming publications. The next step is to extend the concept to higher level computational intelligence processes, e. g. the Viennese waltz. These equations are almost complete. It is intended to implement these equations in a standardised field programmable gate array and use pre-calculated coefficients to obtain the required behaviour pattern. We now move on and use the important omnidirectional robot behaviour pattern, i. e. the Viennese waltz, as an application example in order to create an efficient emergent behaviour computing architecture.

2

The Viennese Waltz (see reference [3] for a movie clip)

The Viennese Waltz is a ballroom dance that requires a human dancing couple, joined as a coordinated four-legged paired unit, to rotate quickly whilst simultaneously translating, Fig. 3. Furthermore the paired unit has to negotiate the extremities of the dance floor so, added to the fast rotation and simultaneous translation, there is a less intense rotation around the dance floor. The less intense rotation means that the translation vector is constantly changing its direction. In fact ballroom dancing represents a very interesting study for the analysis and synthesis of legged and wheeled robot behaviour patterns. So also does the study of motion behaviour patterns and motion strategies of players in a premier league football match. It will be shown in this paper that the Viennese waltz behaviour pattern is a highly useful motion, possibly a fundamental motion, for omnidirectional robots. For example the behaviour pattern can be used to program

Fig. 3. Plan view of dancing couple doing the Viennese Waltz


7

Fig. 4. Sun and planet wheel modeling robot Viennese Waltz behaviour

Fig. 5. Planet wheel on outside of sun wheel producing opposite rotation of planet wheel

a robot to back out of a tight corner whilst rotating, or, for a robot football player to circulate around an opponent whilst tracking the translating opponent. The classic “sun and planet wheel”, Fig. 4, is used to model Viennese Waltz behaviour. The planet wheel is a disc attached to the robot. There is no slip between the planet and the sun wheels. The planet wheel could be on the outside of the sun wheel, Fig. 5, in which case the rotation of the planet wheel is reversed.

3

Applications of Viennese Waltz Behaviour

3.1

Retreating Motion in a Corridor

The robot is shown, Fig. 6, backing out of a corridor representing retreating motion, i. e. translating, and simultaneously rotating such that the head of the robot is ready to face an opponent in the direction of its exit path.

8

Frank Nickols

Fig. 6. Application of Viennese Waltz behaviour. Retreating motion in a corridor

3.2

Rotating on the Spot

If the radius of the virtual planet wheel is set to zero then the robot will turn on the spot, Fig. 7.

Fig. 7. Application of Viennese Waltz behaviour. Rotating on the spot

3.3

Rotating About a Fixed Point

If the radius of the sun wheel is set to zero and the planet wheel rotates on the outside of the sun wheel then the robot will rotate about a fixed point which is the sun wheel, Fig. 8.

Fig. 8. Application of Viennese Waltz behaviour. Rotating about a fixed point


3.4

9

Motion in a Straight Line with no Rotation, i.e. Pure Translation

This occurs when the radii of the sun and planet wheels are set to infinite radius, Fig. 9.

Fig. 9. Application of Viennese Waltz behaviour

4

Analysis of Sun and Planet Wheel Model

Viennese Waltz motion is programmed into an omnidirectional robot by using a model of a virtual sun and planet wheel, Fig. 10. The radii of each wheel depend on the required behaviour pattern. The contact point between the sun and planet wheels is an instantaneous centre of rotation, IC of R.

Fig. 10. Sun and planet wheels that are used as a model to produce Viennese Waltz behaviour in an omnidirectional robot

10

Frank Nickols

In order to program Viennese waltz behaviour it is only necessary to specify the following three variables; (see Fig. 11 for explanation): 1. The range coordinate, RP, of the instantaneous centre of rotation, IC of R, with respect to the robot body. 2. The angle coordinate, θP, of the instantaneous centre of rotation, IC of R, with respect to the robot body. 3. The radius of the sun wheel, RS.

Legend for figure 11 RP = radius of planet wheel RS = radius of sun wheel θplanet = incremental rotation of planet wheel about contact point between sun and planet wheels, Contact point is the IC of R of planet wheel. θICR = incremental rotation of IC of R w.r.t. the robot body. θsun = incremental rotation of instantaneous centre of rotation about sun wheel centre. θP1 , θP2 = angular positions of IC of R w.r.t robot body datum at positions 1 and 2 respectively. Note that: θP2 – θP1 = θICR Fig. 11. Analysis of Viennese Waltz motion with a sun and planet wheel. In this diagram the planet wheel is on the inside of the sun wheel but the opposite could be the case


11

Fig. 12(a). Simplified analysis of the sun and planet wheels; (i) planet wheel on inside of sun wheel

Fig. 12(b). Simplified analysis of the sun and planet wheels; (ii) planet wheel on outside of sun wheel

Figure 11 above represents a detailed analysis of an incremental rotational displacement of the planet wheel. However it is rather complicated so the important angles featured in Fig. 11 are shown in Figs. 12(a) and 12(b). Analysing Fig. 12(a), we obtain: Note minus sign (minus indicates CCW) δθICR

⎛ ⎜ 1 = −δθplanet × ⎜ ⎜ RP ⎜ 1− R S ⎝

⎞ ⎟ ⎟ ⎟ Minus sign indicates planet ⎟ ⎠ wheel on inside of sun wheel.

(1)

12

Frank Nickols

Analysing Fig. 12(b), we obtain: Note plus sign δθICR

⎛ ⎜ 1 = +δθplanet × ⎜ RP ⎜ ⎜ 1+ R S ⎝

⎞ ⎟ ⎟ ⎟ Plus sign indicates planet ⎟ ⎠ wheel on outside of sun wheel.

(2)

Equations (1) and (2) are used to compute the coordinates of the new position, newθICR, of the IC of R with respect to the robot body, like this: newθICR = oldθICR + δθICR

(3)

where, δθICR, is given by Eq. (1) or Eq. (2) RP, remains the same unless a new trajectory is required. In practice the very small angles, δθICR, and, δθplanet, are not small. In fact, δθICR, enlarges to, ΔθICR, and, δθplanet, enlarges to, Δθplanet. Hence Eqs. (1) and (2) lead to errors in the robot body following the required path. However, these errors are compensated by inserting a gain factor, k > 1, into the Eq. (1) like this:

ΔθICR

⎛ ⎜ 1 = ±Δθplanet × k ⎜ RP ⎜ ⎜ 1± R S ⎝

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

(4)

where, k > 1.

5

Application of Sun and Planet Wheel Model to the Robot

A schematic view of the six-legged omni-directional robot is shown below in Fig. 13. Locomotion of the robot is produced by the leg tips moving in rectangular curved plane shapes. A plan view of the robot is shown in Fig. 14, which shows the scale of the computational problem to achieve Viennese waltz behaviour because for each leg the value of the leg tip walking angles, (P), and the amplitude of step, (amp), must be computed in real time. The computational problem is made more challenging because the leg tip loci are curved planes whose radii of curvature are the distances, L, from the IC of R.


13

Fig. 13. Illustration showing the six-legged robot and the leg tip locus plane shapes that will achieve robot body rotation about an instantaneous centre of rotation, IC of R

14

Frank Nickols

Fig. 14. Plan view of the robot during rotation about an instantaneous centre of rotation, IC of R


6

15

Summary

A plan has been worked out using a sun and planet wheel for obtaining Viennese waltz behaviour which means combined translation and rotation of the robot body. Equations have been developed using this model such that a strategy can be worked out for the eventual real-time computational implementation of Viennese waltz behaviour in the robot. Further details are worked out in two forthcoming papers which are, firstly reference [1] which outlines the strategy of obtaining Viennese waltz behaviour and, because this is work in progress, a paper is yet to be written concerning the real-time computational implementation of Viennese waltz behaviour.

References [1] [2] [3]

Nickols F.M.J. Emergent Behaviour Real-time Programming of a Six-Legged OmniDirectional Mobile Robot: Strategy of Viennese Waltz behaviour. Forthcoming conference paper ICARCV, Singapore, 5–8 December 2006 Parallel Robots: Open Problems Jean-Pierre MERLET http://www-sop.inria.fr/coprin/equipe/merlet/Problemes/isrr99-html.html For a movie clip:- http://www.franknickolsrobots.com/movie1.htm and click on “Baby beetle performing Viennese Waltz” (allow time to download)

The Hong Kong Underwater Robot Challenge

Robin Bradbeer Director, Hoi Ha Wan Marine Science and Engineering Laboratory, Department of Electronic Engineering, City University of Hong Kong, Kowloon Tong, Hong Kong

1

Introduction

The aims of the project were: • To introduce technological concepts into marine environment and conservation education programmes. • To publicise the work currently being carried out by WWF and City University of Hong Kong in this area. • To enhance the awareness of Hong Kong teachers and students to marine conservation. • To provide a platform for design and technology students to partake in a practical design exercise with observable objectives. • To promote the development of technical, problem solving, critical thinking, and teamwork skills. The contest in Hong Kong was advertised widely in the press in December 2005, and 20 schools applied to join; 16 were accepted after some preliminary discussions to ascertain whether the students would be capable of performing in the contest, as well as determining the commitment of the school and its teachers. This was part of an international contest organised by the Marine Advanced Technology Education Center (MATE), Monterey, CA, USA in cooperation with the Marine Technology Society (MTS) [1].

2

The Robot Kit

All teams were supplied with a kit of parts to build a very simple robot. The concept of an inexpensive kit of parts, made available without charge, was based upon an original idea developed by MATE Center and a number of other educational

18

Robin Bradbeer

groups in the US [2]. There are also resources on the Internet with information about building underwater robots, and a book by Harry Bohm [3]. Providing some of the basic parts to the student teams makes entering the competition less daunting. As many of the components in the original rov kits designed in the US were not available in Hong Kong (as well as being specified in non-metric units!), a new kit was designed which synthesised some of these other ideas. A completed robot built with these parts can be seen in Figs. 1 and 2. The cost of each kit came to around HK$894, or US$115. This did not include a camera module, and each robot would need at least one to be able to complete the tasks in the competition. These would be one of the extra components that the students would need to buy to do so. To introduce the students and their teachers to the contest a series of workshops was help at CityU in the Underwater Systems Laboratory to allow each team to build the rov from their kit. As most schools in Hong Kong do not have access to a mechanical workshop all tools and accessories, as well as access to a test tank were made available. Figures 3 and 4 show the workshop activities. Fourteen of the sixteen teams were able to build their rov and test it in the water tank within the three hour period allocated.

Fig. 1. Constructed ROV made from kit of parts supplied – rear view

Fig. 2. Side view of built ROV

Fig. 3. Workshop in the CityU lab

Fig. 4. Testing in the lab water tank


3

19

The Competition

The competition was based upon that designed by MATE Center for the MATE/MTS ROV Contest 2006. The teams from Hong Kong, being school students, were competing in the Ranger class contest. There is also an Explorer class contest, which is more advanced, and is for experienced schools and/or colleges. This year the Ranger mission tasks that the robots had to perform included: • Transporting the electronics module from the surface to the trawl-resistant frame. • Placing the electronics module in the frame. • Opening the door of the trawl-resistant frame adjacent to the submarine cable. • Retrieving the submarine power/communications cable connector from the seafloor. • Inserting the power/communications connector into the appropriately labelled open port on the electronics module. [5] There was also another task which involved locating and attaching to the acoustic transponder’s release loop and removing the release loop from the acoustic transponder to free the instrument. These two tasks had to be carried out in 20 minutes. Teams were allowed two attempts, and the highest scoring run was the one marked. The rovs had to be able to work at a depth of 5 m and at a distance of 7 m from the pool edge. They could only be powered by 12 v dc and with a maximum total current of 25 A.

4

The Workshops

A series of workshops to introduce the concept of designing and building an underwater robot were organised for the teams, as shown in the photos above. This not only allowed the teams to complete the kit robot which had been designed by CityU, based on a combination of two designs from Bohm’s book, and the availability of parts in Hong Kong. However, it was not only the robot construction that interested the students. As the contest was sponsored by WWF Hong Kong there was an environmental aspect that needed to be addressed too. Therefore a number of workshops were held at the Marine Life Centre in Hoi Ha Wan Marine Park – a marine environmental centre run by WWF, and where CityU also has a Marine Science and Engineering Laboratory As well as having an introduction to the marine environment around Hong Kong, the teams also took a ride in the Centre’s glass bottom boat to see the coral reef at first hand. Then they were given the opportunity to drive the laboratories commercial rov. Figures 5 to 7 show some of these workshop activities.

20

Robin Bradbeer

Fig. 5. On the glass bottom boat

Fig. 6. Driving the commercial ROV

Fig. 7. In the Marine Science and Engineering lab

The commercial rov is used at the laboratory to survey the coral reefs in the marine park. The laboratory also has permanently deployed video cameras and instrumentation on the reef for monitoring purposes. The students were able to witness real-life applications in these workshops, which gave them a better understanding of why they were taking part in the competition.

5

The Hong Kong Finals

The Hong Kong finals, to select the team to go to Houston, TX for the International Contest was held in the swimming pool at CityU. 14 teams made it through to the final, all with rovs that were based on the original kit, although some had changed considerably. As the original design was scalable, most had been made larger to accommodate the electronics module, which was around 400 mm × 400 mm × 550 mm, and weighed 0.5 kg in water. The final contest was held in the 3.3 m deep middle of the pool. The mission props were placed around 3 m from the side of the pool. Each team was given 15 minutes to complete the two mission tasks. At the same time, each team was given a notice board to display a poster of their work, and table and power supply to work on their robots before and/or after their runs. They also had to exhibit an


21

Fig. 8. A pool full of robots – showing the underwater mission props

Fig. 9. Judging the engineering and documentation

Fig. 10. Making final adjustments

Fig. 11. Manoeuvring the modules

engineering report which was judged during the contest for content, as well as their robot being graded for its engineering. The engineering and report/poster marks made up half the total marks. Extra marks could be gained for completing the tasks in the time allowed. No team was able to finish the tasks, but two came very close. Figures 8 to 11 show some of the robots and how they performed in the pool.

6

The Hoi Ha Wan Trials

The three top teams in the swimming pool contest were taken to the WWF Marine Life Centre in Hoi Ha Wan Marine Park two weeks later. They were then given the chance to operate their robots in the sea, and try and locate an object suspended from the glass-bottom boat. This allowed them to see the difference between operating a robot in the swimming pool/ test tank and the sea. All three teams were able to locate the object successfully in a short time. In fact, it was surprising how such simple rovs could perform so well in a real life environment! Figures 12 to 14 show the Hoi Ha Wan trials.

22

Robin Bradbeer

Fig. 12. Dropping the robot over the side of the GBB

Fig. 13. Looking for the object in the sea

Fig. 14. Searching

7

The International Finals, Houston, Texas

The winning team from Hong kong waqs invited to attend the International Finals held at the NSA Neutral Buoyancy Lab., Houston, Texas in June 2006. Twenty five teams from all over USA and Canada took part in the Ranger contest. The Hong Kong team was the sole representative from outside N America. The team had rebuilt their robot so that it consisted of two parts – a ‘lander’ module that would be easily controlled and that would open the cage door and insert the cable connector, and a larger ‘carrier’ module that was streamlined and could carry the lander very fast, as it had 6 motors. It was less manoeuvrable, but could release the pin of the buoy. The team took three of the major 13 prizes including best poster and highest scoring rookie. Figures 15 and 16 show this rov.


Fig. 15. The HK ROV and Spacelab

8

23

Fig. 16. The final assembled ROV

The Educational Objectives

The main objective of holding the competition was to introduce technology students to the concept of marine conservation, as well as to give them an opportunity to design an underwater robot. These two main objectives were fully met, and feedback from the teachers involved has been very positive. None of the students had seen an underwater robot before the contest started, so the fact that 14 of the 16 original teams were not only able to build a fully functioning robot, but one that, in most cases, could also carry out at least one of the mission tasks, was beyond our initial anticipations. The availability of the basic kit certainly helped, as did the hands-on workshops staffed with graduate students working in underwater robotics research. The contestants could easily relate to what they were trying do, as it was not an abstract task. Certainly, the visits to the Marine Park at Hoi Ha Wan gave an insight into practical applications of using technology to assist marine scientists, as well as seeing real rovs in action. The relationships between technology and conservation are not usually obvious – especially when technology is sometimes seen as causing many of our environmental problems. However, the close collaboration between the Department of Electronic Engineering, and the Department of Biology and Chemistry at CityU, especially with the use of rovs to monitor the reef at Hoi Ha Wan [7], as well as the development of underwater instrumentation for ocean observation [8] [9], meant that the real-life and practical aspects could be demonstrated. At the same time the competitive nature of the contest, with schools competing to go to the USA for the International Finals in Houston, TX, added some excitement. This was shown by the number of teams who came to the Underwater Systems Lab at CityU during the periods when there were no workshops, so that they could test their robots, as well as use some of the specialist facilities. Again, the presence of graduate students working in underwater robotics and instrumentation meant that much was learned by the contestants, which probably could not be found in books or on the web!

24

9

Robin Bradbeer

Conclusions

The First Hong Kong Underwater Robot Challenge was run from January to April 2006. It was a joint effort between WWF Hong Kong and the City University of Hong Kong and was designed to not only raise awareness of marine conservation issues amongst technologically oriented senior school students, but also to give them a competitive situation in which to design, build and operate an underwater vehicle. These aims were successfully accomplished with 14 out of the initial 16 teams entering the finals of the contest. During the competition the students not only learned about the marine environmental around Hong Kong, they also learned how technology is being used to conserve that environment. At the same time, they learned about underwater robotics, and, initially using a simple kit supplied by the organisers, eventually designed quite complex robots to carry out a series of tasks stipulated by the organisers of the International ROV Contest in the USA. The students experienced how to work as part of a team, and how to organise a complex project. The judging of the contest combined not only the completion of the tasks and the speed at which they were completed but also the documentation and engineering aspects. For further information please look at the contest web site www.ee.cityu.edu.hk/rovcontest.

Acknowledgements I would like to thank WWF Hong Kong for their financial and logistic support for the Hong Kong Underwater Robot Challenge 2006, as well as the Deputy President of City University of Hong Kong, the Dean of the Faculty of Science and Engineering, and the Head of Department of Electronic Engineering for their financial and logistics support. Also, the team from Student Development Services Sports Centre for making the swimming pool available and putting up with our strange requests. Katherine Lam and Paul Hodgson for the photos. Jill Zande at MATE Center for her patience in answering our questions (and for documenting all the rules/regulations etc so clearly), and finally, Cyrus Wong and Kenneth Ku, without whom none of this would have been possible.

References [1] [2] [3] [4] [5]

http://www.wwf.org.hk/eng/index.php http://www.cityu.edu.hk http://www.wwf.org.hk/eng/hoihawan/ http://robotchallenge.com/index1.html http://www.marinetech.org/rov_competition/index.php

The Hong Kong Underwater Robot Challenge [6] [7]

[8] [9]

25

H. Bohm and V. Jensen, Build Your Own Underwater Robot and Other Wet Projects. Vancouver, Canada:Westcoast Words, 2003 K.Y. Lam, P.K.S. Chin, R.S. Bradbeer, D. Randall, K.K. Ku, P. Hodgson, and S. G. Cheung, “A comparison of video and point intercept transect methods for monitoring subtropical coral communities”, Journal of Experimental Marine Biology and Ecology, (in publication) R.S. Bradbeer, K.K.Y. Lam, L.F. Yeung, K.K.K. Ku, “Real-time Monitoring of Fish Activity on an Inshore Coral Reef”, Proceedings, OCEANS 2005, Washington, DC, USA 19–23 October 2005 Paper #050216–04; IEEE NJ K.K. Ku, R.S. Bradbeer, K.Y. Lam, L.F. Yeung and Robin C.W. Li, “An underwater camera and instrumentation system for monitoring the undersea environment”, Proceedings 10th IEEE International Conference on Mechatronics and Machine Vision in Practice, pp.189–194, Macau, December 2004

Dynamics and Control of a VTOL Quad-Thrust Aerial Robot

Joshua N. Portlock and Samuel N. Cubero Mechanical & Mechatronic Engineering, Curtin University of Technology, Perth

1

Introduction

Some possible useful applications for Vertical Take-Off & Landing (VTOL) Unmanned Aerial Vehicles (UAVs) include remote video surveillance by security personnel, scouting missions or munitions delivery for the military, filming sports events or movies from almost any angle and transporting or controlling equipment. This paper describes the design, control and performance of a low-cost VTOL quadrotor UAV, known as the QTAR (Quad Thrust Aerial Robot). The QTAR is capable of stationary hover and omnidirectional flight; whereby pitch angle, roll angle, yaw rate and thrust can be controlled independently, while translation is subsequently controlled by these primary four inputs (tilting the thrust vector in the desired direction). The QTAR project had succeeded in developing and implementing a novel “attitude estimator” controller using very low cost components which provide sufficiently accurate tilt angles and state information for very responsive closed-loop feedback control of all flight degrees of free-

Fig. 1. QTAR Prototypes built at Curtin University of Technology

28

J.N. Portlock and S.N. Cubero

dom. The Attitude Control System (ACS) of the QTAR serves to automatically control all four motor thrusts simultaneously to stabilize all the main flight degrees of freedom (translation forwards and backwards, left and right and rotating on the spot yaw) except for altitude control. Thus, the QTAR saves a remote operator a great deal of adjustment and control effort, allowing the user to focus more on navigation and performing tasks rather than on continuously adjusting several motor speeds to maintain stability and control, manually. The quadrotor configuration employs four independent fixed-pitch rigid propellers for both propulsion and control. Each propeller is powered by its own electric motor, symmetrically positioned on each end of a “+” shape. The photos in Fig. 1 show two prototypes of the QTAR UAV that were designed, built, programmed and successfully flown at Curtin University of Technology, Western Australia, in 2005. A demonstration video can be viewed online [10].

2

Current “State of the Art” in Quadrotor UAVs

Triple and quadrotor configurations are the only types of VTOL UAV that employ rotor speed for control. Therefore control is actuated with no extra mechanical complexity, weight penalty or energy losses, commonly associated with swash plates, control surfaces or tail rotors. A triple rotor VTOL UAV, like the Tribelle [5], is the mechanically simplest configuration, however it cannot achieve independent control over roll and yaw moves, as they are coupled. At the time of initiating this project in 2005, the DraganflyerTM by RC Toys [5] was the only commercially available quad-rotor, selling at over $1300 AUD [12]. Many other VTOL UAV researchers have used this platform for their research [1], [2], [11], [3], [7]. In early 2005, the Draganflyer only had rate-gyro feedback for damping its attitude rates and little attitude stabilization or correction capabilities, hence a human operator had to focus much attention on maintaining stability and control. Later in 2005, RC Toys released their Ti (Thermal Intelligence) system. This performs some angular feedback to level out the Draganflyer when no user input is given; however the thermal horizon sensors are only accurate outdoors, at altitudes above the urban canopy [12]. As well as this attitude control limitation, the Draganflyer was limited to only 10 minutes of flight time, a small 1.5:1 thrust to weight ratio and a payload of less than 100 grams. These performance limitations were the key motivators to develop a more powerful quadrotor platform with an attitude control system capable of functioning indoors. Using low-cost commercially available “off the shelf” components, the goals of the QTAR project were to achieve a 2:1 thrust/weight ratio for improved control, flight endurance greater than 15 minutes and a 200 gram payload capacity, enough to carry an onboard wireless camera and additional equipment or sensors. These capabilities would satisfy many VTOL UAV applications.

Dynamics and Control of a VTOL Quad-thrust Aerial Robot

3

29

Design of the Propulsion System

Electrical DC motor drives were chosen in preference to Internal Combustion (IC) engines, which are quite noisy and involve high maintenance and operating costs. It was desirable to keep the maximum span of the QTAR within the width of a typical doorway to allow flight transitions into and through buildings. Therefore a propeller diameter of 10" (10 inches, or 254 mm) was selected to maintain a maximum span under 750 mm. Dual-bladed propellers (prop) were selected because they have much lower inertia and thus respond faster to thrust command signals than four-bladed props. Two different 10" diameter props, one with an 8 inch pitch and another with a 4.5" pitch were compared in tests. It was found that the 4.5" prop was more efficient, as it produced more thrust for the same amount of power. The GWS 380 brushed motor (rated at 70 Watts continuous) with a 5.33:1 gearbox was determined to be suitable for the 10" by 4.5" prop. This was compared with two different types of brushless motors (a gear-boxed in-runner and a direct-drive out-runner). The brushless motors both performed marginally better than the brushed motor, however, the brushed motors were chosen to simplify the controller and minimize costs. The thrust verses voltage (duty cycle) relationship for the brushed motor was close to linear, making simple open-loop speed control possible. The following plot in Fig. 2 illustrates the QTAR propulsion performance compared to two other commercially available quadrotor aircraft: the RC Toys DraganflyerTM and the Silverlit X-UFOTM. This data illustrates QTAR’s superior efficiency, while the Draganflyer and X-UFO both had similar, lower thrust/power characteristics. This plot also illustrates the maximum collective thrusts of 510 grams for the X-UFO and 620 grams for the Draganflyer. The QTAR system was capable of producing more than 2 kg

Fig. 2. Quadrotor Propulsion Performance Comparision

30


of collective thrust. The final QTAR prototype weighs about 450 grams. The energy density of Lithium Polymer (Li-Po) batteries at the time was calculated to be 145 mWh/gram, so the maximum battery capacity while retaining a 2:1 thrust/ weight ratio and carrying a 200 gram payload was 2490 mAh. This gave a theoretical endurance of 18 minutes. Even using the 2100 mAh battery we received from [13], the QTAR achieved flight times greater than 15 minutes.

4

Dynamic Modelling of Attitude

The dynamic attitude model was derived from Newton’s Laws. The gyroscopic procession of each rotor cancels out due to the counter-rotating pairs, which removes any coupling between the pitch and roll dynamics. Due to the low rotor inertia relative to the craft’s rotational inertia, the response of the electric motor was significantly faster than the attitude dynamics, so the motor response was assumed negligible in this model. The total collective thrust, FT , is the sum of all four rotor forces. (subscripts are T = Total, F = Front, B = Back, L = Left, R = Right )

FT = FF + FB + FL + FR

(1)

This collective thrust is nominally equal to the gravitational force when hovering, however, it can be varied by the pilot with the throttle input up to a maximum of 2×Fg, due to the 2:1 thrust/weight ratio. When QTAR is in a stationary hover, FT equals the weight force of the entire aircraft due to gravity. 4.1

Yaw Dynamics

A quadrotor has two sets of counter-rotating propellers, therefore, the net yaw moment generated from aerodynamic drag is cancelled out in neutral flight. This eliminates the need for a tail rotor that normally wastes 12% of the power in a conventional helicopter [4], [9]. Furthermore, a yaw moment is induced on a quadrotor by proportionally varying the speeds of the counter-rotating pairs, as illustrated in Fig. 7. The thrust variation, Vψ, is given by

Vψ ≤ From

τ max k

(k = 2 or 4 to avoid motor saturation, τmax = max. Torque)

(2)

τ ψ = I z ×ψ , where Iz is the mass moment of inertia, yaw acceleration

is

ψ =

τψ Iz

(where ψ = Yaw angle)

(3)


31

Yaw moment is the sum of all rotor torques (CW = Clockwise, CCW = counter CW)

τ ψ = ∑τ r = τ CW − τ CCW = (τ L + τ R ) − (τ F + τ B )

(4)

In Fig. 3, the magnitudes of thrust forces are set so that FL = FR are both larger than FF = FB. The increased drag of the motors with higher thrust will create a net reaction moment that will rotate the body in one yaw direction. Similarly, the body can be rotated in the opposite yaw direction by reversing the relative magnitudes of the above pairs of thrust forces, where the thrusts of FF = FB are greater than the thrusts of FL = FR. Note that during yaw movement of the QTAR τψ ≠ 0 (net torque on the body), i.e. the sum of reaction moments is non-zero. Note that the size of each thrust is proportional to the size of each arrow, where the largest arrow represents a high thrust, the medium sized arrow represents a medium thrust (idling thrust for zero net rise or fall for each motor) and the smallest arrow represents a weak thrust. When the QTAR body is not rising or dropping in altitude, the sum of all thrusts equals the weight force due to gravity. The torque on each rotor, caused by aerodynamic drag, is proportional to the thrust by a scalar constant kτ. Therefore, Eq. (4) becomes

τ ψ = (kτ Vψ + kτ Vψ ) − (−kτ Vψ − kτ Vψ ) = 4kτ Vψ

(5)

The z-axis “Moment Of Inertia” (MOI) of the QTAR is the sum of all point mass inertias about the z-axis (assuming battery and controller inertia is negligible due to their masses being located predominantly at the “Centre Of Gravity”, or COG).

I z = ∑ I m = 4mm l 2 (where mm is a single motor & arm mass)

Fig. 3. Plus “+” configuration for flight control (size of arrow is proportional to thrust)

(6)

32


Therefore, substituting Eqs. (5) and (6) into (3), gives the equation of motion for yaw acceleration:

ψ =

4.2

τψ

=

Iz

4kτ Vψ 4m m l 2

=

kτ Vψ mm l 2

(7)

Pitch and Roll Dynamics

Due to the symmetrical nature of the quadrotor configuration, pitch and roll can be represented by the same model. Figure 3 illustrates the thrust variations required to induce a moment about the y-axis for rolling. The yaw deviation limit is thus

Vφ ,θ ≤

Fmax (k = 2 or 4 to avoid motor saturation, Fmax = maximum Force) (8) k

The equation of motion for this pitching or rolling moment is derived from the sum of moments about the y-axis:

∑τ θ = I

y

× θ

(9)

The thrust deviation for one motor can be calculated as

Vθ = ( FB − FF ) / 2

(10)

Therefore the sum of the moments is

∑τ θ

= 2Vθ l

(11)

The y-axis moment of inertia of QTAR is the sum of the two point mass inertias

I y = ∑ I m = 2m m l 2

(12)

We now substitute Eqs. (11) and (12) into (9) to find pitch acceleration.

∑τ θ

= I y × θ

2Vθ l = 2mm l 2 × θ

θ =

(13)

2Vθ l V = θ 2 mm l 2m m l

Due to symmetry of the QTAR body, this also represents pitch dynamics. The dynamic equations discussed so far have treated the QTAR as a flying “+” structure. Alternatively, Professor John Billingsley from the University of Southern Queensland proposed a different control strategy involving the aircraft controlled as a flying “X” structure, whereby pairs of motors are controlled. Figure 4 shows another method for controlling the thrusts of the QTAR. Note that motors “a” and


33

Fig. 4. “X” configuration for flight control (size of arrow is proportional to thrust)

“b” are at the “front” side, “c” and “d” are at the “back” side, “a” and “c” are on the “left” side and “b” and “d” are on the “right” side of the vehicle (imagine this as a form of “diagonal flying” for the “+” structure in Fig. 3). Either “+” or “X” configurations can be used to control the QTAR successfully. For both configurations, the dynamic equation for vertical altitude acceleration will be the same, but the equation for pitch acceleration will be slightly different due to pairs of motors being controlled for the “X” configuration control method.

5

Attitude Controller Design and Simulation

It is evident from the developed models that the pitch/roll dynamics are linear (if actuator saturation is avoided), time-invariant and 2nd order. Furthermore, aerodynamic drag is assumed to be negligible, therefore this system model has no natural damping, no zeros and only one pole at the origin. This means that open-loop systems will always be unstable without feedback.

Fig. 5. Pitch/Roll Controller Block Diagram

34


Fig. 6. Yaw Rate Controller Block Diagram

With no natural damping, a proportional-only feedback controller would not adequately stabilise the attitude of the system, rather the system will require active damping. The pitch/roll controller illustrated in Fig. 5 was implemented. Yaw required a different controller configuration because it was a 1st order system without a global bearing angle reference. The user input was angular-rate, which was adequate for remotely piloted control. This control method dampens the yaw rate and thus maintains a relatively constant bearing while the user yaw input is left neutral. The yaw rate controller is illustrated below in Fig. 6. After establishing appropriate attitude controller designs, simulations were performed using MATLAB™ (by Mathworks) to evaluate the dynamic response of these controllers and sensor requirements such as states, ranges and resolutions.

Fig. 7. QTAR Control Electronics Block Diagram


6

35

Control Electronics

Each system module illustrated in the block diagram of Fig. 7 either has its own removable circuit board or sub-assembly making it modular and upgradeable.

6.1

Inertial Measurement Modules

Since beginning the QTAR project, two different quadrotor aircraft have become available with attitude sensing onboard, however, they both have their limitations. The mechanical reference gyro on the SilverlitTM X-UFO is suitable for a toy, but for a UAV, it cannot operate for extended periods of time without drifting or becoming unstable. The thermal sensors on the DraganflyerTM only operate outdoors above the urban canopy in Visual Meteorological Conditions (VMC). To avoid these limitations and operate both indoors and outdoors, the QTAR system implemented low-cost inertial sensors and fused their complementary characteristics to estimate attitude using software. Micro Electro-Mechanical Sensor (MEMS) gyroscopes (gyros) measure angular rate/velocity around one axis. Theoretically a set of three orthogonally mounted gyros could be recursively integrated to continuously track attitude. Unfortunately, sensor bias drift and signal noise behaviour for low-cost gyros make this unrealisable. High-performance Ring-Laser gyros are far more accurate, however, their cost and weight make them impractical for the QTAR system. The result of integrating (discretely summing) MEMS gyros is an accumulation of the bias and signal noise errors, consequently increasing the uncertainty of the attitude estimation. Without bounding this accumulated error, the estimation becomes

Fig. 8. Electronics Modules Laid Out Prior to Mounting

36


unstable or unusable. The magnitude of this uncertainty is linearly proportional to the integration time, making these gyro sensors only good for short term estimation (i.e. high frequency performance). The Tokin gyros were used in the QTAR (metal rectangular prism sensors in Fig. 8) because they were the cheapest angular rate sensors at the time. MEMS accelerometers are implemented to compliment the gyros and bound the estimation error. Accelerometers measure both static acceleration due to gravity and dynamic acceleration due to forces on the vehicle. In steady state, a set of three orthogonally mounted accelerometers can accurately measure the pitch and roll tilt angles relative to the gravity vector. In mid flight, they also measure the collective thrust and any external disturbances like wind. Significant acceleration due to gearbox chatter and vibration introduces severe signal noise. At the time of developing the QTAR Inertial Management Unit (IMU), the Analog DevicesTM biaxial ADXL202 (±2 g range) accelerometers were the best solution. Two were mounted perpendicularly to sense all three axes. The maximum angles of incline were relatively small (±15°) so a linear approximation was used to relate the horizontal accelerators to the respective tilt angles, thus avoiding complex and time-consuming trigonometric functions in firmware.

7

Attitude Estimation

As mentioned before, performing integration on the gyros to estimate tilt angle is only accurate for a short period before eventually drifting. Accelerometer data is not always a precise measurement of tilt but remains stable and bounded over an extended period of time. Therefore, a discrete recursive complementary filter was implemented in software to benefit from both sensor characteristics and estimate the tilt angle states. Since this was being performed on a microcontroller, it was developed using scaled integer arithmetic and without the aid of matrix operations in order to minimise processing time. Figure 9 illustrates the final angular tilt state estimator, including the integer scaling factors used to maintain high accuracy without using floating point arithmetic. (Accelerometer output a = acceleration) The result of this compensating process is a calculated angle, dominated on the short term by the gyro sensor and bounded over the long term by the accelerometer data, where Kest determines these time scales. Mathematically, this recursive discrete state estimator can be written as

⎡∑ a

⎤ − (θ previous + Δθ gyro )⎥ K est ⎢⎣ 2 ⎥⎦

θ est = (θ previous + Δθ gyro ) + ⎢

(16)


37

Fig. 9. Tilt Angle State Estimator

This angle estimator was simulated in MATLAB™ using inertial sensor data from flight tests. It was then implemented in firmware on the microcontroller with the scaled integer arithmetic. The experimental angle estimation data plotted in Fig. 10 demonstrates the effectiveness of the estimator in practice.

Fig. 10. Experimental Angle Estimation Data

38


The test was performed with the motors running to ensure the accelerometers were experiencing typical vibration noise. The plot compares the estimated angle with the uncompensated integrated gyro angle and the raw low-passed accelerometer. It can be seen that a steady state error will occur on the gyro integration if not bounded by the accelerometers. Also, the estimator rejects most high frequency accelerometer disturbances while also responding faster than the latent low-pass filtered accelerometer. An adaptive gain was implemented on the estimator gain Kest in Fig. 9. It was determined that higher rates of change in acceleration (Jerk) meant that the accelerometer was predominantly sensing dynamic acceleration. To improve the tilt angle estimation, the estimator gain was adapted to give less credibility to the accelerometer when jerk was high, but more when jerk was low.

8

Attitude Controller Implementation

The tilt angle controller gains determined from simulation were experimentally evaluated first with a 15° step input for the tilt command, but the response was underdamped. The rate gain was increased slightly and the proportional gain was lowered for improved performance. After tuning, step responses for tilt angle commands like those shown in Fig. 11 were obtained. With these tuned controller gains it was found that the system would no longer overshoot or oscillate, however, greater stability or damping comes at the cost of slower response times. High level code for the QTAR ACS was written to target the AtmelTM AVR AT mega 32 8-bit microcontroller using the signals shown in Fig. 12. A 4-channel (2-joystick) radio transmitter was used to send Pulse Position Modulated (PPM) signals for yaw, pitch, thrust and roll to QTAR’s 6-channel radio receiver in Fig. 8.

Fig. 11. Step Response with Tuned Controller Gains


39

Fig. 12. Signal flow diagram for the QTAR ACS (Attitude Control System)

9

Conclusions

The QTAR attitude control system successfully estimated and controlled attitude both indoors and outdoors, allowing stable hover and easily controllable omnidirectional flight as described in Fig. 3. To the best of the authors’ knowledge at this time of writing, the Jerk-based adaptive tilt estimator gain method described in this paper had not been described in previous attitude estimation literature. The final QTAR prototype was capable of carrying a 200 gram payload, while maintaining a 2:1 thrust/weight ratio and achieving flight times of around 15– 20 minutes. The total cost of parts and materials for the QTAR was about AUD$870 (Australian), making it suitable for mass production and many lightweight VTOL UAV applications. The authors would like to thank Andre Turner from www.radiocontrolled.com.au [13] for sponsoring the QTAR project.


Altug, E., J. P. Ostrowski, et al. (2002). Control of a quadrotor helicopter using visual feedback. Robotics and Automation, 2002. Proceedings. ICRA ’02. Altug, E., J. P. Ostrowski, et al. (2003). Quadrotor control using dual camera visual feedback. Robotics and Automation, 2003. Proceedings. ICRA ’03. IEEE International Conference on. Castillo, P., A. Dzul, et al. (2004). “Real-time stabilization and tracking of a fourrotor mini rotorcraft.” Control Systems Technology, IEEE Transactions on 12(4): 510–516.

40 [4] [5] [6] [7] [8] [9] [10] [11] [12] [13]

J.N. Portlock and S.N. Cubero Coleman, C. P. (1997). A Survey of Theorectical and Experimental Coaxial Rotor Aerodynamic. California, Ames Research Center. Dienlin, D. S., S. Dolch. (2002). “TriBelle – The Innovative Helicopter.” from http://braunmod.de/etribelle.htm. Innovations, D. (2005). “RC Toys Website.” http://www.rctoys.com. McKerrow, P. (2004). Modelling the Draganflyer four-rotor helicopter. Robotics and Automation, 2004. Proceedings. ICRA ’04. 2004 IEEE International Conference on. Microdrones GmbH (2006). http://www.microdrones.com. Petrosyan, E. (2003, 27 March 2003). “Aerodynamic Features of Coaxial Configuration Helicopter.” 2006, from http://www.kamov.ru/market/news/petr11.htm. Portlock, J. (2005). QTAR: Quad Thrust Aerial Robot 2005 Video. Perth. http://www.youtube.com/watch?v=MLxe3FuQ3v0. Suter, D., T. Hamel, et al. (2002). Visual servo control using homography estimation for the stabilization of an X4-flyer. Decision and Control, 2002, Proceedings of the 41st IEEE Conference on. Taylor, B., C. Bil, et al. (2003). Horizon Sensing Attitude Stabilisation: A VMC Autopilot. 18th International UAV Systems Conference, Bristol, UK. Turner, A. (2006). “Radio Controlled” website. www.radiocontrolled.com.au.

Project-oriented Low Cost Autonomous Underwater Vehicle with Servo-visual Control for Mechatronics Curricula

C. A. Cruz-Villar 1, V. Parra-Vega2, and A. Rodriguez-Angeles 1 1

Center for Research and Advanced Studies (CINVESTAV I.P.N.), Electrical Engineering Department, Mechatronics Group 2 CINVESTAV I.P.N., Saltillo, Robotics and Advanced Manufacture Group

1

Introduction

This paper describes a project oriented Autonomous Underwater Vehicle (AUV) that integrates three different courses in a Mechatronics MSc. academic program. The postgraduate program in Mechatronics at CINVESTAV includes the 64 hrs. courses: Real Time Programming, CAD/CAM/CAE, and Modeling and Simulation of Mechatronics Systems which are pursued simultaneously. It is intended that the students develop a final project per course, however during the first term of 2006; it was proposed to integrate a single final project for the three courses. For mechatronics integration purposes, it was suggested to take a radio control system which could be modified to apply reverse engineering and real time control, as well as advanced modeling techniques. A micro submarine was selected, as it requires a small working area and presents challenging issues such as hydrodynamic effects and under-actuation in some degrees of freedom (Gianluca 2003), on the other hand the remote control unit included at the commercial system can be modified to be interfaced to a PC through the parallel port to implement real time control. A major advantage of a radio control system is the autonomy yielded by absence of cables and physical connections between the control and controlled systems, looking to keep this autonomy; it was chosen a servo visual closed loop system. Thus the original “human” radio control system was modified to the servo visual control layout of Fig. 1.

42

C. A. Cruz-Villar, V. Parra-Vega and A. Rodriguez-Angeles

Fig. 1. Human and visual servoing RC systems

The project was presented to the students with the commitment to be accomplished in 8 weeks, it was divided into several subsystems, and each subsystem was assigned to a group of students among 10 enrolled students. The groups were encouraged to be multidisciplinary groups according to their undergraduate background (electronics, mechatronics, systems and mechanics engineers). It was also provided a minimum number of goals to be fulfilled as individual evaluation of each course, as well as of the whole project.

2

General Layout of the Project

The challenges and goals of the project were designed to include the three involved courses. The minimum project’s goals requested to the students were: • Reverse engineering of the commercial radio control AUV system to characterize it and be able to propose modifications. • Modeling, design and validation of the servo visual control using MATLAB as well as CAD/CAE tools. • Implementation of a real time controller in C-language on a Debian GNU/Linux RTAI gcc version 3.3.5-1 platform. • Modification of the electronics of the AUV to design an interface with the PC. • Position and orientation regulation of the AUV system in the X-Y plane. Due to the limited performance of the commercial AUV system and the goals of the project the students faced the following challenges: • • • • •

Image processing to determine position and orientation of the AUV. Under-actuation and limited response and control of the motors at the AUV. Limitations at the commercial radio control system provided with the AUV. Design of the multi task real time control implementation. Servo-visual problems in fluid environments, such as reflection and distortion.

The project was divided into the following subsystems: AUV, Radio-control and electronics, real time control and visual servoing.

Underwater Vehicle with Servo-visual Control

3

43

AUV Visual Servoing Project Description

The main component of the project was chosen as a micro radio control submarine Sea Scout of the brand HobbicoTM, formed by 53 assembled pieces, see the CAD disassembly of Fig. 2, which has been obtained by reverse engineering.

Fig. 2. CAD disassembly of the HobbicoTM, Sea Scout mini-submarine

The Sea Scout has a radio control unit working at 49 Mhz, and is provided with one DC motor to generate the up/down motion, and two coupled DC motors to provide the left/right and forward/backward motions, thus the AUV is underactuated. The power electronics in the AUV locks the signals such that only one of the left/right and forward/backward motions can be generated at a time. Another major constraint is that the DC motor power electronics works as an on/off controller, which limits the resolution and accuracy of the closed loop system. Thus, the performance and trajectories are highly constrained. To keep water resistance at the AUV, it was not modified, and for project purposes it was modeled and characterized by disassembling a second AUV purchased for reverse engineering purposes.

3.1

Model of the AUV

One of the major goals of the project was to model the AUV. The kinematic and dynamic models of the AUV were used for control design and validation through simulations with MATLABTM SIMULINKTM and Visual NastranTM. At the course

44


Modelling and Simulation of Mechatronics Systems, several works related to the modelling and parameter identification of underwater vehicles were reviewed, comparing different approaches and simplification hypothesis, and particularly for modelling of the hydrodynamic effects. In the next subsections the model of a general AUV and the particular model of the Sea Scout AUV are presented. AUV's General Kinematics and Dynamics Model A rigid body that moves in a 3D space has 6 degrees of freedom denoted by T q = [ x y z φ θ ψ ] where x, y, and z are the position coordinates of the mass center with respect to a fixed coordinate frame; φ , θ and ψ are the Euler angles, which define the roll-pitch-yaw, see Fig. 3. Notice that the X axis is defined along the movement of the AUV and Z is defined downward, which follows the notation of the SNAME (Society of Naval Architects and Marine Engineers). By considering the Newton–Euler approach, see (Gianluca 2003), (Smallwood et al. 2004), the 3D body dynamics is given by

Mq + C (q )q = τ ,

M , C ∈ \ 6×6 , τ ∈ \ 6×1

(1)

where M is the constant, symmetric and positive definite inertia matrix, C (q ) is antisymmetric and represents the Coriolis and Centripetal forces, τ is the vector of external forces and torques. However, the AUV being immerse in a water environment is subjected to hydrodynamic and buoyancy effects (Jordán et al., 2005). For the study of a rigid body moving in an incompressible fluid an analysis based on the Navier–Stokes equations is considered. Since the density of the fluid is comparable to that of the body, then additional inertia effects must be taken into account, for that an added inertia matrix MA > 0 is introduced, which depends on the geometry of the body and the velocities of the system, this matrix is generally represented as follows

{

M A = − diag X x , Yy , Z z , Kφ , M θ , Nψ

}

Fig. 3. Generalized coordinates and reference frames

(2)


45

where the entries X x , Yy , Z z , Kφ , M θ , Nψ depend on the geometry of the particular submarine and its velocities. Due to the fluid there is also a centripetal and Coriolis contribution, that is repT resented by the added Coriolis matrix C A ( q ) = −C A ( q ) , see (Gianluca 2003)

0 0 0 − Z z z ⎡ 0 ⎢ 0 0 0 0 Z z z ⎢ ⎢ 0 0 0 −Yy y X x x CA = ⎢ 0 − Z z z Yy y − Nψψ ⎢ 0 ⎢ Z z z 0 0 − X x x Nψψ ⎢ 0 − M θθ Kφφ ⎢⎣ −Yy y X x x

Yy y ⎤ − X x x ⎥⎥ 0 ⎥ ⎥ M θθ ⎥ − Kφφ ⎥ ⎥ 0 ⎥⎦

The viscosity of the fluid generates dissipative effects on the AUV, such as hydrodynamic damping and dragging forces, which are represented by a damping matrix D( q ) , (Jordán et al., 2005). The hydrodynamic damping and dragging forces acts against the movement of the AUV and are collinear to the fluid direction. For simplicity only dragging forces are considered, furthermore the effects are assumed to be decoupled along the degrees of freedom of the AUV, such that the matrix D( q ) is diagonal. Since the dragging forces are external, they are included like the vector of external torques τ in a form of a dragging vector force τ D . Therefore the dynamics of the AUV of Eq. (1) can be rewritten as

Mq + C (q )q = τ + τ D where τ D = ⎡⎣τ Dx

τ Di = −12

(3)

τ Dy τ Dz τ Dφ τ Dθ τ Dψ ⎤⎦

with

Ai μ ui 1 − CDi Aρ ui ui 2 di

(4)

ui is the incidence velocity (Olguin, 1999) at the direction i = x, y, z , φ ,θ ,ψ ; Ai and di are the transversal section and characteristic dimension perpendicular to ui respectively. The fluid density is denoted by ρ , CDi is a damping coefficient depending on Reynolds number, and μ denotes the

where

fluid dynamic viscosity. The hydrodynamic effects depend on the fluid regime and thus on the incidence velocity ui . The first term on (4) corresponds to laminar regime and the second term to turbulent regime (Olguin, 1999), being one of them zero depending on the regime. Then, in matrix form τ D is rewritten as the matrix D given by

D = − diag ⎡⎣τ Dx τ Dy τ Dz τ Dφ τ Dθ

τ Dψ ⎤⎦

(5)

46


Finally the fluids effects related to buoyancy and gravity of an immerse body have to be taken into account. Considering the gravity acceleraT tion g = 0 0 9.81 , it follows that the buoyancy is given by B = ρV g , where V represents the body volume, and the body weight is W = m g , where m is the body mass. The gravity forces are external forces included the same way as the external torques τ through the vector G ( q ) given by

[

]

(W − B) sin(θ ) ⎡ ⎤ ⎢ ⎥ −(W − B) cos(θ ) sin(φ ) ⎢ ⎥ ⎢ ⎥ −(W − B ) cos(θ ) cos(φ ) G (q) = ⎢ ⎥ (6) ⎢ −( yGW − yB B) cos(θ ) cos(φ ) + ( zGW − z B B) cos(θ ) sin(φ ) ⎥ ⎢ ⎥ ( zGW − z B B) sin(θ ) + ( xGW − xB B) cos(θ ) cos(φ ) ⎢ ⎥ ⎣⎢ −( xGW − xB B) cos(θ ) sin(φ ) − ( yGW − yB B) sin(θ ) ⎦⎥ Thus, the general dynamics (1) transforms into

M ′q + C ′( q )q + D( q) + G ( q) = τ ,

M ′ = M + M A , C′ = C + CA

(7)

Dynamic Model of the Sea Scout AUV The Sea Scout is limited to three DC motors, which allow the up/down, forward/backward and left/right turn motions. So the allowed movements are highly constrained. Moreover only one camera for visual feedback is considered, such that only X-Y plane movements (horizontal position and yaw orientation) are to be covered and controlled, i.e. the camera stands over the working area of the AUV, see Fig. 4. Thus, the generalized coordinates become q = [ x y ψ ] . There are some assumptions about the modeling of the Sea Scout. First of all it has a maximum forward velocity of 0.25 m/s approximately, and its body is completely immersed, thus laminar regime is considered. For geometry and parameter estimation a cylindrical approximation of the AUV is taken, see Fig. 5. Finally, since the camera covers the X-Y plane, the Z-motion is left out, so that the AUV

Fig. 4. Generalized coordinates and reference frames


47

Fig. 5. Cylindrical approximation

works at its neutral buoyancy point, thus neglecting the gravity and buoyancy effects G(q) , given by the vector (6).

The control forces acting on the AUV, i.e. τ , are related to the forces exerted by the DC actuators by a Jacobian B (Morel et al. 2003). Because of construction of the AUV, there are two coupled DC motors implying that the left/right turn depends on the backward/forward motors, as a result the yaw angle ψ and only one motion direction X or Y can be controlled at any time t . Along this work, the X direction is considered independent, and Y depends and it is result of a combination of x and ψ motions. So, it is obtained that the dynamics (7) is reduced for the Sea Scout to ⎡ m + X x −2my ⎤ ⎡ x ⎤ ⎡ τ Dx x ⎤ 0 0 ⎤ ⎡ x⎤ ⎡ 0 0 ⎡τ x ⎤ ⎢ ⎥⎢ ⎥ ⎢ ⎢ ⎥ m + Yy 0 ⎥⎢ y⎥ + ⎢ 0 0 2my ⎥⎥ ⎢⎢ y ⎥⎥ + ⎢ τ Dy y ⎥ = B ⎢ ⎥ ⎢ 0 ⎣τ y ⎦ ⎢ 0 0 I zz + Nψ ⎥⎦ ⎢⎣ψ ⎥⎦ ⎢⎣ 2my −2my 0 ⎥⎦ ⎢⎣ψ ⎥⎦ ⎢⎣τ Dψψ ⎥⎦ ⎣ where m=0.065 [Kg] is the AUV mass, L=0.118 and R=0.019 [m] are the dimensions of the cylindrical approximation, see Fig. 5, following (Olguin-Díaz 1999) it is obtained that

I zz =

1 1 m(3R 2 + L2 ), X x = −0.1m, Yy = −πρ R 2 L, Nψ = − πρ R 2 L3 12 12

ρ = 1000 ⎡⎣ Kg/m3 ⎤⎦ is the water density, the Jacobian B has been determined experimentally and by geometric projections, and it is given by 0.182 ⎤ ⎡ 1 B = ⎢⎢ −0.153 0.116 ⎥⎥ ⎢⎣ −0.223 L ⎥⎦

The dragging forces τ Dx , τ Dy and τ Dψ are obtained from the Eq. (4) taking into account the geometric dimensions Ax = π R 2 , Dx = 2 R, 3 Ay = 2 LR, Dy = L, Aψ = 2 L R / 8, Dψ = L / 2 and the water dynamic viscosity μ = 0.001 [ Pa sec ] .

48

3.2


Radio Control and Electronics System

The commercial AUV includes a 49 MHz amplitude modulated radio control transmitter based on the integrated circuit TX6 ATS306T, which sends codified 2 Hz PWM signals corresponding to the on/off control of the actuators. The receptor in the AUV works with the integrated circuit RX6 ATS306R, which demodulates the PWM signals to the drivers of the motors that are based on transistors working in cut off and saturation regimes. The transmitter was modified to be interfaced to the PC through the parallel port. Also an amplification stage based on a bipolar transistor 2N3866 and an operational amplifier NE5539N was implemented to increase the power of the transmitted signal. 3.3

Real Time Control System

At this stage of the curriculum, the students only have background on linear controllers design; thus, it is proposed that PD controllers be implemented at each actuation motor. A strong limitation is that a single motor can be controlled each time interval of 250 ms. Moreover, the on/off limitation on the DC motors power electronics led the students to implement Pulse Width Modulation (PWM) based controllers, which are fully digital in the sense that one PWM output for each motor is obtained as a bit of the parallel port. To obtain a well synchronized carrier signal, it was decided to use a hard real time system which was implemented on a RTAI platform. The real time system was designed to include two threads in real- time -one for each DC motor controller-, and two user tasks; one for image acquisition and another for image processing. The controllers were validated through simulations in SIMULINK (MATLAB) and Visual Nastran. In Fig. 6, it is presented simulation results of the switched PD controllers for position error and yaw angle error respectively.

Fig. 6.

x -position [pixels] and yaw angle ψ

[rad] errors for a simulated scenario


3.4

49

Visual Servoing Control

A SONY DFWVL500 CCD camera of 640 × 480 pixels and an 8 bits gray scale resolution was used for visual feedback of the position and orientation. This camera was given to the student with UNIX routines based on the libdc 1394 library for capturing and processing images, therefore no particular knowledge of visual control was required. Once the image was captured and processed to differentiate the tones, an algorithm to compute the centroid and orientation of the AUV was implemented. The PD control for regulation of position and orientation of the camera was programmed in pixels. The camera covers an area of 86 [cm] × 64.5 [cm], such that there is a ratio of 7 [pixels/cm] approximately.

4

Results

Notice that the AUV has singularities at ψ = 90°, 270°, that correspond to the AUV being parallel to the X axis. These singularities were experimentally confirmed and they presented the larger position error for the system, 50 pixels (6.5 cm) at position and 3° at orientation; meanwhile, far from these singularities the average errors were of 25 pixels (3.25 cm) at position and 3° at orientation, which are in the limits of resolution and performance of the servo visual AUV system. In Fig. 7 a snapshot composition of an experiment video with x(0) = 67 pixels and ψ (0) = 182° as initial conditions is presented. The desired values are xd = 400 pixels, ψ = 20°, while the achieved values are x = 425 pixels, ψ = 39°. The above results are highly satisfactory since the goal was to integrate the AUV visual-servoing platform, rather than to design a high performance controller, which nonetheless could be considered as an extension for further development of the AUV system. Concerning to the original goals of the project, they are fully satisfied since the integration of the AUV, the real time control and visual servoing are achieved. Furthermore, the students went farther than the original

Fig. 7. Results from an experiment and GUI for the AUV system

50


goals by introducing an algorithm for determining the minimum error direction when regulating the position of the AUV, and designing a GUI (graphical user interface) for the system, that is shown in Fig. 7.

5

Conclusions

From the results it is concluded that this project enforced the integration of three courses in a Mechatronics curricula. The project is low cost and gives the opportunity to the students to practice the theory reviewed in the classroom. The students commented that the project was very useful and gave suggestions to improve it, such as modifying the electronics of the AUV to obtain proportional control actions at the DC motors, and even to design and build their own AUV. They also concluded that the servo-visual feedback loop could be applied to other non invasive autonomous applications like mobile robots and ship systems.

References [1] [2] [3] [4] [5] [6]

Gianluca A (2003) Underwater robots. Motion and force control of vehicles-manipulator systems, Springer. Olguin-Díaz E (1999) Modelisation et Commande d'un Systeme Vehicule/Manipulateur Sous-Marin, These pour le grade de Docteur de l'Institute National Polytechnique de Grenoble, Grenoble, France. Morel Y and Leonessa A (2003) Adaptive Nonlinear Tracking Control of an Underactuated Nonminimum Phase Model of a Marine Vehicle Using Ultimate Boundedness, Proceedings of the 42nd IEEE CDC, Maui, Hawaii, USA. Jordán MA et al (2005) On-line identification of hydrodynamics in underwater vehicles, Proceedings of the IFAC World congress, Prague. Smallwood A and Whitcomb LL (2004) Model-Based Dynamic Positioning of Underwater Robotic Vehicles: Theory and Experiment, Journal of Oceanic Engineering, vol. 29, No. 1, January. SNAME, Society of Naval Architects and Marine Engineers, http://www.sname.org/.

Coordination in Mechatronic Engineering Work

James Trevelyan School of Mechanical Engineering, The University of Western Australia E-mail: [email protected]

1

Abstract

This paper shows that little has been written on the roles that people actually perform in the course of mechatronics engineering work. The paper reports empirical results of interviews with several engineers working in mechatronics: this is part of a larger study of engineering work in several disciplines. The paper argues that coordinating the work of other people is the most significant working role, both in mechatronic engineering and also other engineering disciplines. This role has not been explicitly identified before. While coordination appears to be a generic nontechnical role, close examination reveals that technical knowledge is very important for effective coordination in engineering work. The coordination role is not mentioned in engineering course accreditation criteria. The absence of explicit references to this role in previous literature suggests that the research methods used for this study could provide better guidance for engineering educators on course requirements.

2

Introduction

This paper presents some of our research on what industrial mechatronic engineers really do in their work: part of a larger study of many aspects of engineering work. Most engineering researchers are interested in seeing and learning about new technology and hardware, novel ideas and how these can be applied to solve specific technical problems or used for improving existing processes and work practices. A study of the behaviour of engineers and technicians working in industry might seem to be irrelevant in this context, belonging more in a non-technical conference on psychology or management. Yet researchers rely on engineers and technicians for the application of their ideas, mostly in private firms. Whether we

52

J. Trevelyan

like it or not, people are an essential part of nearly all engineering processes and work practices. Many engineering researchers work for private firms and research tells us that private industry R&D can be a frustrating career for many (Manners, Steger et al. 1983; Allen and Katz 1995; Lam 1997; Vinck 2003; Lam 2005). Some social scientists have studied this, but it is difficult for them to understand engineering work (the paper reviews some of their results). This paper argues that engineers can gain useful insights into R&D work practices using contemporary social science research methods combined with their personal experience, particularly to understand why, for example, Japanese engineers can be particularly successful with certain mechatronics products. We can also learn how technical experts can be used effectively to maintain high productivity in engineering design and development. Most other engineering researchers are also educators. It is valuable for educators to know something about the work their students will be expected to perform and the environment in which they will do that. First they have a duty to convey this to students. Second, engineering students will be more likely to work harder learning methods they know will be useful in their career. Third, engineering students need to learn how human behaviour constrains engineering processes. It is tempting to draw a line in the sand and dismiss all human behavioural issues as an issue for social scientists, of no concern to engineers and technologists. Many people, especially social scientists, see engineering as applied science. Engineering academics have recently described engineering practice in terms of “specialist technical problem solving” (Sheppard, Colby et al. 2006). Yet we cannot separate people from engineering. Our research shows that young engineers have to work with and through other people and it is human behaviour that constrains their work right from the start of their careers. Given the intrinsic involvement of people in engineering processes, it is surprising that so little attention has been devoted to this in engineering research. One explanation for this gap is that it is difficult for engineering researchers to acquire appropriate research methods and background knowledge. It is possible that many engineering researchers assume that the necessary research is already being done by social scientists. This paper shows that this is not the case: a background knowledge of engineering is just as essential as social science methods for effective research on this issue. Engineering work is largely unknown except by engineers themselves and much of their know how is knowledge that they do not know they have (Polyani 1962). While there are many anecdotal accounts, the research literature has remarkably little to offer that would help an inquiring mind learn more about it. Only a handful of reports have appeared in the last three decades. Mechatronic engineering, as a relatively new discipline, is even less known. Few companies, even now, would admit that they employ mechatronic engineers. Instead they know them as instrumentation engineers, systems engineers, control engineers, automation engineers and several other titles. A recent survey of about 100 research publications (Tilli and Trevelyan 2005) reveals two major groups of reports. Investigations on engineering education have stimulated about half of all the reports on engineering work. The other half has


53

emerged from researchers who have been interested in engineering management. However, the vast majority of these results were obtained using presumed lists of engineering roles which do not seem to be based on empirical observations (e.g. Deans 1999; Bons and McLay 2003). Only a small number of empirical studies can be relied on, mostly using anthropological approaches based on qualitative research. Most of these were written by social scientists or students with little engineering experience. There are significant industrial problems in mechatronics that this research could help to solve. Recently published data from Scandinavia confirms considerable anecdotal evidence that major industries incur costs ranging between 10% and 50% of turnover resulting from maintenance and operator failures. Research on the roles performed by people in maintenance work is almost non-existent (Trevelyan, Gouws et al. 2006). (Orr 1996) is the only report we have uncovered so far, coincidentally of significant relevance in mechatronics as the focus was photocopier repair technicians.

3

Empirical Research Method

The empirical research followed well-established qualitative research methods used by contemporary social science researchers (e.g. Zussman 1985). Most data came from transcripts of semi-structured interviews performed by the author who has 20 years of engineering work experience in different fields of engineering. The sampling was partly opportunistic and partly purposeful for typical cases and maximum variation. About 50 engineers happily agreed to be interviewed, including several mechatronic engineers with experience ranging from 1.5 to 35 years. Three of the interview subjects were female and most have engineering degree qualifications. Each interview took between one and two hours. Open-ended questions encouraged the respondent to talk about the details of the work he or she performs. Field studies were also part of the survey: a limited number of subjects were shadowed for 1–2 days to test interview data. We use standard ethnographic analysis techniques on interview transcripts, field notes and other reference texts (e.g. Strauss 1987; Patton 1990; Miles and Huberman 1994; Huberman and Miles 2002). Working papers at our web site provide full details on interview questions and analysis methods (http://www.mech.uwa.edu.au/jpt/pes.html).

4


The coordination role emerged from the interview data unexpectedly. Initially we considered formal and informal “supervision” as an engineering work role, and there were two questions in the interview to explore supervision relationships. However the huge number of references was unexpected. These were mostly oneon-one situations involving a coordination or supervision role. Most references

54

J. Trevelyan

were in response to questions unrelated to supervision. Finally an insightful firsthand comment about C. Y. O’Connor, the engineer responsible for the Perth to Kalgoorlie pipeline in 1895–1899, led us to understand that obtaining willing cooperation is an important part of this role. “He was neither dogmatic nor arrogant [and] possessed an extraordinary capacity to win the interest and cooperation of men.” (Evans 2001, p 135). Coordination in this context means influencing other people so that they conscientiously perform some necessary work at an agreed time. This usually requires three different interactions. First it is necessary to reach an agreement on what has to be done and when it has to be performed. Usually it is necessary to be present, at least intermittently, while the work is being done to check that the results (perhaps partial) agree with expectations. The final result needs to be checked to make sure no further work or rectification is needed. The coordinator may not know how to do the work, nor even what the final result might look like, and may learn some of this through the coordination process. In other situations the coordinator may need to provide some of the knowledge and help the worker develop necessary skills. Willing cooperation increases the chance that the work will be performed conscientiously with less chance of mistakes and re-work. A small selection of quotes from interviews with mechatronic engineers (or mechatronics related engineering disciplines) is a useful way to illustrate some aspects of the role. Explicit references to particular companies or individuals in quotations from the interviews have been changed to preserve the anonymity of the interview subjects. Quotation 1. A recently graduated mechatronic engineer on his first assignment, working on site with little real authority, and learning about progress reporting: On my previous job I had to work with electricians and technicians and tradesmen. They were reporting to me on the previous job, not on this job. [How many?] Maybe three to five. They also reported to other engineers. For example I had to show them how particular wiring is done. I had to make sure that things are done the way we specified in the cable schedules… and in the design diagrams and carried out in a timely manner. [How did you know how the cabling was supposed to be done?] That was easy. You just follow the cable diagrams. Actually, one of my first tasks was to look at the cable schedules and make some modifications. That sort of got me to know how the cabling is done. It wasn’t that difficult to pick up. The cabling is installed by electricians but the engineer is responsible for making sure the installation is correct. The electricians probably can read the drawings, but the engineer needs to take them through each aspect shown in the drawings and relate the drawings to the actual situation on-site. The engineer needs to know how the cabling should be installed, the meanings of all the symbols used in the drawings (documentation standards and methods), and he also needs to understand how the cabling is part of a larger system so he can understand the function performed by the cables and hence


55

help to identify any mistakes in the drawings. This is one aspect of technical coordination work, working as a site engineer. He continues…. [And how did you know how long it should take, how could you tell when they were going slowly?] Okay, I was doing a very hands-on job when I started, I was working alongside the electricians, I was working as an electrician. Like a tradesman. I know how long it takes to commission a bus station. In a bus station we have PA systems, the communication system, we have 32 stations along the busways and they are all similar. So once you have done one you know how long each one should take. His initial first hand experience would have involved learning about working methods and techniques from the tradesmen. He may not have reached their proficiency levels, but he has learned how long the work should take under normal conditions. You can also know by the quality of their work. They always make mistakes because some of them just cannot be bothered to follow the schedules. Some of them don’t mark the cables correctly, others do not terminate the cables correctly. You can just go in and find out the quality of their work when you just start turning things on. Sometimes you can spend just as much time fixing up their mistakes. But that was all very valuable experience on how you handle this kind of commissioning work because you learn from their mistakes. Spotting mistakes is frequently mentioned in reports of site engineering experience. Rectifying the mistakes means discussing problems with the tradesmen and arranging for the rework to be done. …you would ask them “Hey, have you done this task?” They would say “yes, yes, it’s fine, it’s all done.” And then it you would go out there and you would find that it is 30% completed. There is very little control over the electricians because they are unionised and they get paid a lot more than us. They are direct employees but because they are governed by union guidelines they are not really controlled or managed. Quotation 2. A highly experienced control systems engineer, commenting on what graduate engineers need to learn early in their career: The business of working with other people. It is amazing how much you don’t know about that. Like the fact that you can’t order people to do things, you have to ask people to do things and they will tell you to get stuffed if they don’t like it. This illustrates the importance of winning willing cooperation. The next sentence reveals that winning cooperation may require some sharing of information… The fact that when you share information there is currency involved and the fact that to the extent that you will be taken to the extent that you will contribute…

56

J. Trevelyan

He goes on to describe his own view of the engineering process that graduates need to learn. This is useful because it reveals how extensive coordination is needed to make the whole process work. The progressive process of creation, how does it work, how do you progress a vision through to a strategy through to a design through to a plan, how you get the work done, how you control the quality how you control the safety and how you liaise with clients, the whole process. Teaching people about differential equations and thermodynamics….. about finance and that sort of thing, the whole business how you translate a vision into a finished structure and a whole number of processes involved, how you involve other people, how you do it within an economic framework, all these sorts of things, that’s what engineering is about. That is the core of engineering. And, how you do that process and at the same time preserve some magic in it, without changing it into a completely mechanistic process, because there is a lot of magic all the way through. Quotation 3. A recently graduated mechatronic engineer in the building services industry: With sub contractors the person who gets the best service is the one who shouts the loudest. If you’re nice to them you drop to the bottom of the stack. You have to scream and yell. We prefer a different technique. We monopolise each sub contract or so he has no time to work for anyone else. If he tries to work for somebody else he will have to shuffle his own work around and squeeze it in and then he will have to keep on changing it as we change our schedule. Then he will do the job but find he has to go down to Albany to do one of our jobs first. In recent years, there has been a transformation away from large organisations employing all their required labour to a widespread use of small subcontractors, sometimes a single individual. Building a cooperative relationship takes time and we see one approach that combines the advantage of a long term relationship with the flexibility of a contracting arrangement. Most sub contractors are very good. But, there are some guys coming in just for the money. Our reputation and standing is very high. For example, we make sure that our wiring is installed in rigid conduit because it looks good. Flexible conduit looks bad, scrappy. We win work just for that reason alone. That means that we have to make sure that our sub contractors don’t cut corners. Again, technical requirements lie behind this as well: rigid conduit eliminates repeated bending stresses. Quotation 4. A project manager in a small specialist mechatronic engineering company: I had a group of electronic engineers who were doing the electronics and a group of mechanical engineers who were doing the mechanical aspects and a software engineer who was senior and who was waiting until everything was ready so he could do it, so that he could do his software. But, one of the items


57

that was assigned to a young engineer, which is... he was a graduate, I think he was a second-year graduate and he hasn’t got the sort of experience to say for this isolation you need this and you need the other. So, here we were on the last day and we were actually having a review and we said okay let’s have a look at this equipment, is it going to work? And we found on the last day that it wouldn’t because we were getting these echoes which were wrong from what we were expecting so we had to actually... I had to ask that young engineer to get his senior guy to help him to fix it but then I had to go to my customer who is expecting this kit to sort of arrive and buy ourselves time. So, here on one hand I am at the design level helping this young engineer by saying, look, we got noise, we need some isolation, we need this, sort of screened etc and on the other hand now I need to sort of jump a few ranks not to tell the customer, “Look I have noise etc etc, I can’t do it” but to give an excuse which is plausible which will then gain time and actually it was good because he couldn’t provide the sort of vehicle which we wanted so it was him who said “Look, I can’t get you the vehicle, is it okay if you come in a weeks time instead?” So I said “well, it might be a bit hard...” (laughs heartily) Here the project manager has to coordinate the graduate engineer and provide some guidance on reducing electromagnetic noise in a circuit, and at the same time coordinate a client’s schedule. The client needs an understandable reason for a delivery delay. Quotation 5. An instrumentation project manager working on process plant construction: So, yeah, there aren’t really any direct technical (responsibilities) but there is still the requirement that you have a good working knowledge of everything that’s going on in the project and, to a certain extent, everything that all the other disciplines are doing as well. Because you have got to be able to interface with what everyone else is doing so it’s a lot of coordination, a certain amount of handholding with individuals, once you’ve started getting a group of people together you get all the different personalities, different traits, different people with different strengths and different problems that all have to be addressed. Here the engineer uses the term “hand holding” to describe the second interaction in coordination: intermittent face to face contact to make sure that the work proceeds as expected and to provide guidance and help when needed. The engineer will normally anticipate problems and provide timely advice to help avoid them. Coordination is the most commonly referenced engineering work role in our interviews, both across all engineering disciplines and mechatronics. Qualitative analysis software provides an easy method to count interview references to particular roles. This does not imply a corresponding proportion of work time, but it does provide strong evidence for the significance of coordination roles in engineering work. In the interviews analysed so far, the most commonly referenced role is coordinating people in the same organisation. On average, each interview produced 24 references to technical roles, and 70 references to generic roles in-

58

J. Trevelyan

Fig. 1. Average number of role references in each interview, generic, non-technical roles. Average number of references to technical roles was 24

cluding coordination, financial, legal, project and operations management, business development and personal career development. Of the generic non-technical roles, coordinating people dominates as shown in the graph below (Fig. 1). One must be careful not to place too much significance on the numerical results. The numerical values can be strongly influenced by the way the interviews and analysis were conducted. Even though only a small number of interview quotations have been included in this paper, they illustrate several significant factors. First, even recently graduated engineers report significant coordination roles, mostly without real authority. Most respondents mention the importance of maintaining good working relationships and some comment directly on the counterproductive results of resorting to authoritarian or confrontational relationships. This aspect of engineering work does not seem to be mentioned in well-known works on engineering management (e.g. Badawy 1995) except in the context of cross-functional teams. Most reports of coordination work in the interviews did not refer to team work. A good example is coordination with clients, either directly or through staff employed by the client. A further factor is the significance of technical expertise in the coordination role. Again this is illustrated in some of the quotations reproduced above. Technical knowledge can be a significant factor that confers ‘informal authority’ in working relationships. Spotting mistakes in drawings and specifications is an important aspect of site engineering work: this draws on technical expertise. Engineering work is commonly separated into “real engineering” or “hard technical” working roles and “soft skills” including communication. The results of this study tend to support a different view: that coordinating people, gaining their willing cooperation, is the most significant working role for many engineers, and that this role relies on technical knowledge and expertise as much as interpersonal communication skills. The level of interview analysis has been far more detailed than reported in this paper. As well as engineering working roles, we have also analysed many aspects of technical knowledge used in engineering work and we plan to report on this in forthcoming publications.


5

59

Implications for Engineering Education

Engineering education is traditionally viewed as necessarily technical with “some management” or “soft skills” content. The prevailing social view reflects this and sees engineering as a technical discipline, or as applied science. Some engineering academics view the discipline as mainly specialized technical problem-solving (Sheppard, Colby et al. 2006). The data from this study shows that coordination work is an important aspect in the work of engineers, even at the start of their careers. Engineers devote little of their time to hands-on technical work: that is largely performed by other people. What we see in the evidence is that engineering work is coordinated and driven by engineers, but the end results are delivered through the hands of other people. The strength of this evidence leads one to conclude that engineering needs to be treated as a technical and a social discipline at the same time. Engineering educators rely strongly on accreditation criteria such as the ABET engineering education outcomes (ABET 2005) and Engineers Australia generic attributes (Engineers Australia 2005) to provide guidance in course design. These accreditation criteria are supposed to be based on the attributes that graduates need for engineering work. Neither mentions coordination or gaining willing cooperation specifically. While one might interpret “team work” to mean the same thing, and there are obvious parallels, working in teams is a different experience. Most of the engineer’s coordination work reported in the interview data occurs outside the context of a particular team. There is no doubt that effective communication is required to win willing cooperation and this probably explains why accreditation criteria and job advertisements place strong emphasis on communication skills. However, defining “communication skills” as an educational objective can lead to many different interpretations, possibly not relevant in coordination roles. Powerpoint presentations can be useful on occasions, but will probably not be very helpful in arranging well-timed concrete deliveries at a construction site. We can therefore raise legitimate questions on whether current accreditation criteria mentioning team work and communication skills are promoting effective acquisition of coordination skills in engineering students. Of course, effective coordination relies on a hierarchy of several other groups of more fundamental skills such as interpersonal verbal and non-verbal communication, written communication (verbal and visual), selecting appropriate communication strategies, mentoring, and informal leadership. It also relies on an accurate appreciation of both individual and shared technical knowledge domains and ways to represent technical knowledge in the working context. Traditionally, engineering schools rely on industry advisory committees with experienced engineers to provide guidance on course content and feedback on graduate attributes. The author has many years of experience working with such committees. This research has produced results that have not emerged either from these committees or national engineering accreditation requirements. This research, therefore, shows significant shortcomings in the way that most engineering

60

J. Trevelyan

schools gain their knowledge of engineering work requirements to design appropriate courses for their graduates.

Acknowledgements This work would not have been possible without the support of my family, in Australia, the UK and Pakistan. Nor would it have been possible without enthusiastic support from my colleague Sabbia Tilli. Thanks are also due to my faculty colleagues Ruza Ostrogonac and Lee O’Neill and students Leonie Gouws, Sally Male, Adrian Stephan, Ernst Krauss, Emily Tan, Katherine Custodio, Nathan Blight, Tim Maddern and Brad Parks. Finally I need to provide anonymous thanks to all the engineers and others who have contributed, knowingly and unknowingly, through their interview responses, comments, voluntary contributions and suggestions.

References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13]

ABET (2005). Criteria for Accrediting Engineering Programs 2006–2007. Allen, T. J. and R. Katz (1995). “The project-oriented engineer: A dilemma for human resource management.” R&D Management 25(2): 129–140. Badawy, M. K. (1995). Developing Managerial Skills in Engineers and Scientists: Succeeding as a Technical Manager, Van Nostrand Reinhold. Bons, W. and A. McLay (2003). Re-engineering Engineering Curricula for Tomorrow’s Engineers. 14th Annual Australasian Association for Engineering Education Conference, Melbourne, Australia. Deans, J. (1999). “The Educational Needs of Graduate Mechanical Engineers in New Zealand.” European Journal of Engineering Education 24(2): 151–162. Engineers Australia (2005). Accreditation Criteria Guidelines. A. Bradley, Engineers Australia. 2006. Evans, A. G. T. (2001). C. Y. O’Connor: His Life and Legacy, University of Western Australia Press. Huberman, A. M. and M. B. Miles, Eds. (2002). The Qualitative Researcher’s Companion. Thousand Oaks, California, Sage Publications. Lam, A. (1997). “Embedded Firms, Embedded Knowledge: Problems of Collaboration and Knowledge Transfer in Global Cooperative Ventures.” Organization Studies 18(6): 973–996. Lam, A. (2005). “Work Roles and Careers of R&D Scientists in Network Organisations.” Industrial Relations 44(2): 242–275. Manners, G. E., J. A. Steger, et al. (1983). Motivating Your R&D Staff. Managing Professionals in Innovative Organizations. R. Katz. Cambridge, Massachusetts, Ballinger: 19–26. Miles, M. and A. Huberman (1994). Qualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks, California, Sage Publications Inc. Orr, J. (1996). Talking About Machines: An Ethnography of a Modern Job. Ithaca, New York, Cornell University Press.


61

[14] Patton, M. Q. (1990). Qualitative Evaluation and Research. Newbury Park, California, Sage. [15] Polyani, M. (1962). Personal knowledge: towards a post-critical philosophy. New York, Harper Torchbooks. [16] Sheppard, S., A. Colby, et al. (2006). “What is Engineering Practice?” International Journal of Engineering Education 22(3): 429–438. [17] Strauss, A. (1987). Qualitative Analysis for Social Scientists, Cambridge University Press. ]18] Tilli, S. and J. Trevelyan (2005). Published Research on the Nature of Engineering Work, School of Mechanical Engineering, The University of Western Australia. [19] Trevelyan, J. P., L. Gouws, et al. (2006). The nature of engineering maintenance work: a blank space on the map. 2006 World Conference on Engineering Asset Management, Surfer’s Paradise, Queensland, Maintenance Engineering Society of Australia. [20] Vinck, D., Ed. (2003). Everyday Engineering: An Ethnography of Design and Innovation. Inside Technology. Boston, MIT Press. [21] Zussman, R. (1985). Mechanics of the Middle Class: Work and Politics Among American Engineers. Berkeley, University of California Press.

Vision Techniques

Estimation of distance by means of binocular vision is a well trodden area. Even the use of a single camera with techniques such as visual streaming is well known. However the authors of the paper from Japan that opens this section have gone one step further. They have implemented their method on an analogue VLSI chip. Indeed, two versions are presented. The first gives a linear motion estimate that are virtually independent of contrast and illumination. The second one estimates motion in a spatially coarser manner, favoring certain directions as it mimics the receptive fields of the retina. To put the methods to a practical test, the sensors have been used in the RoboCup robot soccer competition. Some Hong Kong research concerns the use of a genetic algorithm for determining shapes. The ‘particle swarm optimisation’ technique can identify contours within the image of an object; the research aims to reduce the number of iterations required for convergence. The following article is inspired by altogether more practical considerations. Autonomous operations such as spray painting or sandblasting need a 3D vision system. The work is set in the context of an Austrian firm that specialises in spray painting. It concerns the recognition of identity and orientation of a part from a library of parts. The paper includes results from numerous practical tests. What sort of vision system will enable your mobile robot to follow you around? A paper from San Diego investigates special applications of techniques such as frame-differencing, color thresholding, region growing and shape identification. The aim is to cause the robot to follow at a more-or-less constant distance with no other source of information. Video of the system in action was shown when the paper was presented. An Australian paper concerns the tracking of position and orientation of a head in real time. Rather than indulge in computationally intensive methods of feature recognition, the authors have taken the simple expedient of attaching LED beacons to the head, mounted on a special pair of spectacles. A single camera can provide the data for all the necessary calculations.

A Vision System for Depth Perception that Uses Inertial Sensing and Motion Parallax

Vlatko Bečanović1 and Xue-Bing Wang1 1

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu, Kitakyushu, Japan

1

Introduction

There are several ways for a vision system to obtain depth information from a complex visual scene. Some extract absolute distance information and others only relative information from the scene. Commonly used methods that give absolute distance measurements are triangulation, suitable for medium to far distances, and binocular disparity used in stereo vision, which is a robust method for determining distance information at close range. Other cues that are commonly used, but that only give relative distance information, are scale, perspective, precedence of non-occlusion etc. There is however another important cue for distance, or depth perception, not as commonly used as stereo vision, and that can reveal depth information from a monocular motion field. It is referred to as structure from motion or kinetic depth effect when the object that is observed is moving. When the observer is moving it is referred to as motion parallax (Mallot 1998, p 201). The former gives rise to local depth information and the latter can determine distances to objects. There is however an important requirement for determining absolute distances to objects, i.e. that the motion of the observer needs to be known. The motion of the observer will in our case correspond to the measured inertial change of the observer. The distance to an object point can then be obtained by augmenting the additional inertial cue, i.e. the observer motion, with the perceived motion of any object point present in the scene (Huster 2003, Karri et al. 2005). In a practical scenario the object points belong to image structures of anything from object contours, corners, to local textured regions, e.g. patches of the scene that can be distinguished from the background. An experiment was performed where a static high contrast object was the primary target and the observer was accelerated with the help of gravity. The experimental data was compared with a Monte-Carlo simulation of the algorithm in order to help us build a noise distribution model of the system dynamics. The hardware setup in our experiments consisted of an inertial sensor, a high contrast imager and a sliding pad mounted on a rail. The design of the particular vision

66

V. Bečanović and X.-B. Wang

sensor was inspired by the early vision processing stages of the vertebrate retina (Kameda et al. 2003). This sensor will calculate a contrast enhanced input image together with a temporal change output based on difference images, i.e. it is a brain inspired neuromorphic sensor. We plan to implement this approach in an embedded system that consists of a hardware based on the Intel PXA27x processor described in previous work (Misawa 2006, Bečanović et al. 2006).

2

Experiental Set-up

The experiment is performed by letting the vision system fall along a sliding rail with a constant acceleration of up to 9.4±0.4 m/s2. The sliding rail is equipped with a braking mechanism that will dampen the fall after an initial period of acceleration in order to protect the vision system from getting damaged during the de-acceleration (cf. Fig. 1). Data was collected continuously during each experimental trial that consisted of a trajectory of up to seven position estimates, giving up to five perceived acceleration estimates at 20 Hz. The response of the vision sensor was given along a trajectory row for the two parallel images provided; contrast enhanced (sustained) image and difference image as presented in Figs. 2 and 3. The contrast enhanced image will improve object centroid estimation and the temporal image is not used at present. The observer acceleration is measured simultaneously at a higher update rate and then down sampled (low-pass filtered) to fit the update rate of our imager. A characteristic example of the inertial information corresponding to the image sequence is presented in Fig. 4. The inertial sensor used is presently considered to be one of the smallest commercially available inertial sensors measuring only 5.6 × 5.6 × 1.4 mm and giving inertial measurements in

Fig. 1. The vision system mounted on the sliding rail. A view of the experimental setting can be seen in the background


67

Fig. 2. Sustained image output along a single row. The parameter xC is the image centroid position as calculated from the corresponding 2D image

Fig. 3. Difference image output where bright to dark edges give a positive, and dark do bright edges a negative contribution (cf. Fig. 3)

the range from –2G to 2G. Priced at reasonable 25 USD per piece and consuming only about 5 mW it is our preferred choice for performing inertial measurements1. It is a true 3D sensor, although in our measurements only the component in the direction of motion is considered. 1

cf. model: HAAM-313B, HDK Hokuriku company web page: http://www.hdk.co.jp/

68


Fig. 4. The inertial sensor output for the transversal movements in vertical (x-), horizontal (y-) and depth (z-) direction. The period of constant acceleration is marked with an arrow

3

Distance from Motion Parallax

There are several ways to obtain depth information. Motion parallax has the advantage that the depth can be determined to objects by using monocular vision only. For the case that the observer motion is unknown only relative depth can be estimated, but if the observer motion is known the absolute distance (as seen by the observer) can be estimated. If the velocity of the observer is known the distance to static objects can be determined. If also the acceleration of the observer is known the distance can be estimated to objects moving with constant velocity, that is, if the observer motion is known to a higher derivate than the object motion then the absolute distance to the object can be estimated. It should be noted that only translational motion will contribute to the motion parallax effect, the rotational component of the observer will not contribute at all. The distance algorithm will here be derived for the special case when the observer has constant acceleration in the plane perpendicular to the optical axis (that corresponds to our inertial sensor measurements) and where the object is moving with constant velocity or is static (the motion derivative is assumed to be constant). The depth formula can then be derived for an object point that is moving along a line in the x-y plane perpendicular to the optical axis of the observer, i.e. the z-axis (cf. Fig. 5). By assuming that the thin lens approximation holds: 1 1 1 zf = + ⇒ z′ = = M ( z) z f z z′ z− f

Fig. 5. Thin lens derivation of the distance from motion parallax formula

(1)


and M ( z ) =

69

f , then for an object at distance z = d the perceived object point z− f

is projected as: (x', y') = (M(d) · x, M(d) · y)

(2)

where the prime indicates coordinates as perceived on the focal plane image. The relative velocity in the direction of the observer motion is Δv = vobs + v0 and the acceleration difference is Δa = aobs + a0, where a0 and v0 are components of object motion, aobs and vobs are components of observer motion, that all are parallel with the direction of observer motion. Both parameters are assumed to be zero, i.e. vobs= a0 = 0, meaning that the initial velocity of the accelerating observer is zero and that the perceived object is moving with constant velocity. The relative velocity to the object can be obtained by integration of the relative acceleration: t

Δv (t ) = ∫ Δa (t ′)dt ′ = aobs t + v0

(3)

0

The relative object position along the direction of observer motion s(t) is a linear path on the x-y plane, i.e. s(t):= ||(x(t), y(t))||. It can be obtained by integration of the velocity difference: t 1 s(t ) = ∫ Δv(t ′)dt ′ = aobs t 2 + v0t + s0 0 2

(4)

where the perceived position (at the focal plane) can be calculated from (4) by using the relation (2) so that:

⎛1 ⎞ s p (t ) = M d ⎜ aobs t 2 + v0t + s0 ⎟ ⎝2 ⎠

(5)

The acceleration as it is perceived by the observer can be obtained by first calculating the velocities at the two instances t + Δt and t + 2Δt:

v p (t + Δt ) ≈

s p (t + Δt ) − s p (t ) Δt

= (6)

⎛1 ⎞ = M d ⎜ a obs ( 2t − Δt ) + v 0 ⎟ ⎝2 ⎠

v p ( t + 2 Δt ) ≈

s p ( t + 2 Δt ) − s p ( t + Δ t ) Δt

=

⎛1 ⎞ = M d ⎜ a obs ( 2t + Δt ) + v 0 ⎟ ⎝2 ⎠

(7)

Then, by using the perceived velocities at t + Δt and t + 2Δt the perceived acceleration can be calculated as:

a p (t + 2Δt ) ≈

v p (t + 2Δt ) − v p (t + Δt ) Δt

= M d a obs

(8)

70


which in turn can be written as:

ap =

f a obs d− f

(9)

and finally the distance as perceived by the observer can be obtained by solving for d so that:

d=

aobs f a f + f ≈ obs ap ap

(10)

The perceived distance is thus obtained as a quotient between the inertial and optically perceived accelerations. Both are measured quantities that introduce considerable amounts of imprecision to the estimate. Especially the perceived acceleration is critical for distant objects, e.g. when ap « 1, since d ∝ 1/ap.

4

Results from Simulation and Experiment

The first task was to simulate the algorithm in order to validate its performance for our inertial and vision sensors. The algorithm was simulated with a Monte-Carlo approach where two kinds of errors where introduced, the first being inertial measurement error which was estimated to be ±0.4 m/s2 and the second perceived pixel error that was estimated to be ±0.223 mm, e.g. half the inter-pixel distance of 0.445 mm. The errors were both simulated as normally distributed noise sources and a quantization artefact was modelled for the resolution of 40 pixels which is the focal plane resolution of our vision sensor prototype in the direction of object motion. The distance estimate was improved by the use of a Kalman filter and the corresponding model matrix F, the state vector X and the mixture matrix H were selected as follows:

⎛ 1 Δt ⎜ ⎜0 1 F =⎜ 0 0 ⎜ ⎜0 0 ⎝

Δt 2 / 2 0 ⎞ ⎟ 0⎟ Δt , 1 0⎟ ⎟ 0 1 ⎟⎠

⎛ s p (t k ) ⎞ ⎜ ⎟ ⎜ v p (t k +1 ) ⎟ , X (k ) = ⎜ a p (tk +2 ) ⎟ ⎜⎜ ⎟⎟ ⎝ a obs (t k + 2 ) ⎠

⎛1 ⎜ ⎜0 H =⎜ 0 ⎜⎜ ⎝0

0 0 0⎞ ⎟ 1 0 0⎟ 0 ε 0⎟ ⎟ 0 0 1 ⎟⎠

(11)

where the iteration step was tk = t0 + kΔt and ε « 1. Note that n observations correspond to k + 2 position estimates, since the acceleration is calculated from the position estimate. The simulated results are shown in Fig. 6 where the single observation points are presented together with the corresponding Kalman and mean estimates after 3 observations.


71

Fig. 6. Distance estimates for a simulated run, where individual estimates, Kalman estimates and averages over observations in each trajectory are shown

There is a rich literature describing the Kalman filter and a more detailed description will not be given here. A good review of the Kalman filter together with sample code can, for example, be found in the book by Grewal and Andrews (Grewal et al. 2001). Experimental data were obtained for 10 runs at each distance, comprising trajectories of up to five observations for objects at far distances and down to a single observation in the close range. This was expected since the field of view decreases at close range together with the fact that the perceived object motion is high. This makes the error in the experimental data decrease with greater distance, since at far distances the imprecision due to low perceived velocity (close to sub-pixel) is compensated by the increase of observational points. At close range there are very few observations which will instead increase the spread of the estimates. At average the relative error for the Kalman estimate is about 3% for distances greater than 2.0 m, 6% for distances in the range between 1.0 and 2.0 m and up to 15% in the closer range. The experimental data is presented in Fig. 7 and Tab. 1 for a freely falling observer with an acceleration of 9.4±0.4 m/s2. The experiment was repeated twice with decreasing observer accelerations of 7.3±1.0 m/s2 and 6.2±1.3 m/s2 respectively. The experimental set-up was not deemed to be appropriate for lower accelerations because of too high friction between the sliding pad and rail at low inclinations, cf. Figs. 8 and 9.

72


Fig. 7. Distance estimates for the experimental data. The number of observations per trajectory increases with distance. Estimates at close range might only have a single observation Table 1

d [m]

n *2

a p [cm/s2]

σa [cm/s2]

d p [m]

σ d [m]

σ d /di*3

0.5

1.1

21.8

4.41

0.55

0.08

0.12

0.7

2.3

16.9

2.66

0.68

0.05

0.06

0.9

2.8

12.7

3.13

0.94

0.14

0.15

1.1

3.5

11.0

2.90

1.11

0.14

0.06

1.3

3.9

10.1

3.37

1.25

0.13

0.06

1.5

3.3

8.29

2.26

1.50

0.10

0.02

1.7

3.7

6.34

1.82

2.07

0.20

0.05

1.9

3.9

5.79

1.19

2.07

0.15

0.06

2.1

4.2

5.42

0.88

2.15

0.12

0.03

2.3

4.1

4.89

0.54

2.32

0.08

0.03

2.5

4.6

4.53

0.63

2.52

0.06

0.02

3.0

4.1

3.68

0.73

3.16

0.11

0.04

*2 *3

Average number of observations along unique trajectories. Relative error obtained from standard deviation.


Fig. 8. Distance estimates for an observer accelerated with 7.3±1.0 m/s2

Fig. 9. Distance estimates for an observer accelerated with 6.2±1.3 m/s2

73

74


Fig. 10. Relative error as a function of distance for an observer accelerated with 9.4±0.4 m/s2 (freely falling), 7.3±1.0 m/s2 (falling at 30 degrees inclination) and 6.2±1.3 m/s2 (falling at 45 degrees inclination) respectively

For lower accelerations the uncertainty increases somewhat for farther distances (greater than 2.0 m), as can be seen in the result presented in Fig. 10. At close distances the best result was obtained with an observer acceleration of 7.3±1.0 m/s2 (sliding pad falling at 30 degrees inclination). Thus the result seems to depend not only on the acceleration but also on the actual inclination of the sliding rail. This could be explained as effects of non idealities in the experimental set-up. The friction and vibrations introduced when the pad slides along the rail are highly non-linear and give effects not accounted for in the simple 1D geometry and dynamics of our model. This is something that needs to be addressed in future designs. Nevertheless, the simple approach used at present gives surprisingly accurate results, which shows that it is a fairly robust principle deserved to be considered in vision systems, especially real-time systems that utilize motion information and where several sensing strategies are combined, e.g. inertia, vision, etc.

5

Conclusions

A monocular optical distance sensing system was investigated that was based on an algorithm that utilizes motion parallax. The algorithm was suitable for low precision hardware and showed promising results in simulation and experiments.


75

The simulation showed that the distance could be robustly calculated when the rotational component of the observer motion was very small, and this was experimentally confirmed using a relatively ideal experimental set-up. Most probably this would not be the case in a real application, e.g. on a wheeled robot, vehicle, etc., thus the efficiency of the method would need to be validated further, especially under the influence from rotational motion. It would also be advantageous to complement this method with other distance sensing strategies when the observer is static, because this is the case when the system would be blind in the sense that no distance cues can be calculated. All in all, exploiting the mechanisms of motion parallax by fusing visual and inertial cues could prove to be an efficient way to sense the depth in a scene with lower cost, less weight and with lower power consumption compared to other alternatives.

Acknowledgements This work is supported by the Kyushu Institute of Technology 21st Century Center of Excellence Program financed by the Ministry of Education, Culture, Science and Technology in Japan which is gratefully acknowledged.

References [1] [2] [3] [4] [5] [6] [7] [8]

Bečanović, V., Matsuo, T., Stocker, A.A. (2006), An embedded vision system based on an analog VLSI optical flow vision sensor, ICS 1291: Brain-Inspired IT II, pp. 249–252. Grewal, M.S. Andrews, P.S. (2001), Kalman Filtering Theory and Practice using Matlab, John Wiley and Sons Inc. Huster, A. (2003), Relative Position Sensing by Fusing Monocular Vision and Inertial Rate Sensors, PhD. Dissertation, Stanford University, USA. Kameda, S., Yagi, T. (2003), A silicon retina system that calculates direction of motion. ISCAS pp 792–795. Karri, S.S., Titus, A.H. (2005), Toward an analog very large scale integration system for perceiving depth through motion parallax. Optical Engineering, vol. 44, no. 5. Mallot, H.A. (1998), Computational Vision – Information Processing in Perception and Visual Behavior, MIT Press (2000), 2nd Ed (translated by J.S. Allen). Misawa, H. (2006), An embedded vision system based on analog VLSI motion sensors. COE Report, Kyushu Institute of Technology, Japan. Wang, X.B. (2006) Determining distance from motion parallax with a silicon retina. Master thesis, Kyushu Institute of Technology, Japan.

Rate Shape Identification Based on Particle Swarm Optimization

P.W.M. Tsang1 and T.Y.Y. Yuen2 1,2

Department of Electronic Engineering, City University of Hong Kong, Hong Kong

1

Introduction

When a near planar object is viewed from different directions with a camera placed sufficiently far away, its images can be mathematically related with the Affine Transformation as given by

⎡ d si ( x ) ⎤ ⎡a b ⎤ ⎡ d ri ( x ) ⎤ ⎡ e ⎤ ⎢ d ( y )⎥ = ⎢ ⎥+⎢ ⎥ ⎥⎢ ⎣ si ⎦ ⎣ c d ⎦ ⎣d ri ( y )⎦ ⎣ f ⎦

(1)

where (dsi(x), dsi(y)) and (dri(x), dri(y)) are the co-ordinates of pixels in the scene Oscene and reference Oref objects, respectively. The set of parameters A = {a,b,c,d,e,f} are coefficients of the Affine Transformation relating the two images. As a result, matching of object shapes can be encapsulated as a process of searching the existence of a mapping between the pixels of each pair of images to be compared. The method can be easily achieved by defining an objective function E to be the area of non-overlapping region between the scene and the transformed reference object images, followed by a search algorithm to locate the Affine Transform that gives the smallest value for E (the global minimum) in the vector space. The effectiveness of this simple approach is severely limited by the enormous amount of time taken to locate the correct state (if it exists) in the six dimensional search space governed by the transform parameters. In addition, as there are numerous local minima in the objective function, fast searching algorithms based on Hill Climbing or Gradient Descent are not applicable, as the search path can be easily trapped in sub-optimal solutions. To overcome this drawback, attempts have been made using Genetic Algorithms to determine the optimal state in the vast search space. Amongst these techniques [1] has exhibited satisfactory performance in identifying isolated object shapes and well defined contours. The method is also applicable in shape alignment [2] as well as matching of broken contours [3].

78

P.W.M. Tsang and T.Y.Y. Yuen

An important assumption that has been made in [1] is that some of the chromosomes in the initial population should carry partial characteristics of the optimal descendant. By exchanging and modifying these useful qualities through crossover, mutation and reproduction, the majority of members in the population will finally evolve to a state which exhibits high fitness value for a given objective function. Despite the success of this method, little has been mentioned on how the algorithm will behave if the entire population is ill-formed (i.e., there is no genetic information in the population that is related to the targeted solution). Experimental results reveal that the evolution, which depends heavily on mutation, can be very long if the initial states are far away from the optimal solution in the search space. In the extreme case, the process will fail to converge or trapped in local minimum with moderate fitness values. A straightforward approach is to repeat the genetic search several times with different initial populations [4]. However, application of this simple method will increase the computation loading significantly. An alternative method is to build an initial population with individuals that exhibit fitness values that are above a pre-defined threshold. Although this selection criteria may be effective for other engineering applications, it is not always effective in object shape matching where the search space is unpredictable (depending on the pair of images to be matched), highly non-linear and generally contains large amount of local minima. A chromosome with large fitness value does not necessary imply the presence of genes that are related to the optimal solution. Recently, the above problem have been partially alleviated by integrating the “Migrant Principle” [5][6] to the genetic algorithm. However, the average number of generations required in accomplishing the task is considerably higher than the original scheme and complete success rate is not achieved in general. In this paper we propose a novel technique to overcome the above problems with the use of Particle Swarm Optimization (PSO). Experimental evaluations have reflected that for the majority of cases, our method is capable of identifying contours of the same object within a small number of iterations. Based on this important finding we have further enhanced the method by conducting multiple trials on a reduced evolution timeframe. The revised algorithm has demonstrated 100% success rate with only slight increase in the computation time. Organization of the paper is given as follows. In section 2 a brief summary of the Affine Invariant Object Matching scheme reported in [1] is outlined. Next, the Basic principle of PSO is described in section 3. Our method in applying PSO in object shape matching is presented in section 4. This is followed by a report on the experimental results and a conclusion summarizing the essential findings.

2

Outline of Genetic Algorithm Based Affine Invariant Object Matching

Details of the genetic algorithm for object shape matching have been described in [1] and only a brief outline will be given in this paper. Given a reference contour Oref = {pr1, pr2, ....prM} and a scene contour Oscene = {ps1, ps2, ....psM} the task of the genetic search was to determine whether an Affine Transform existed that maps


79

one contour to the other. The algorithm is summarized as follows. To start with three seed points S = [s1, s2, s3] are selected on the reference contour. An initial population Pop of I individuals are generated each formed by a triplet of randomly selected test points T = [t1, t2, t3] on the scene contour. Each seed point or test point is represented in the form of an index with reference to a fixed start point on the corresponding object contour. The pair of triplet point sets define an Affine Transform mapping S to T, which is applied to the reference contour to give a normalized contour Oref. The fitness value of the chromosome is determined by the fraction of overlapping between Oref and Oscene. Individuals from the initial population are selected with probabilities according to their fitness value into a mating pool at which they are allowed to crossover or mutate to produce the next generation of offspring. The reproduction cycle is repeated until a number of epochs have elapsed or the maximum fitness value in the population has exceeded a predefined threshold. If Oref and Oscene belong to the same object, a set of test points corresponding to S can be found which define an Affine Transform mapping the majority of points from Oref to Oscene, resulting in a maximum fitness value that is close to unity.

3

Particle Swarm Optimization (PSO)

Particle Swarm Optimization (PSO) was first initiated by Kennedy and Eberhart [7][8] to study and model the social behaviour of a population of individuals. In their theory, they suggest that when individuals are grouped together in a transparent community, each member will tend to improve its condition by learning from past experience and following the trend of other successful candidates. This postulation is found to provide a good description on the movement of birds’ flock, and could also be extended to unravel solutions for many real world problems. The generalization could be explained as follows. Suppose a problem is represented by an objective function f (x1 , x 2 ,........, x N −1 ) with N independent variables, defining a multidimensional space with each point corresponding to a state of the problem. The task is to deduce the set of variables that constitute to the best solution reflected by the highest (in certain cases the lowest) outcome of the objective function. When a group of individuals are placed in the space, each of them will be located at position that may score differently from others. Suppose the individuals all have the urge to migrate to better positions, they will move around with the hope that one or more of them will finally reach the best point in the space. In the paradigm of Particle Swarm, each individual will conduct such kind of search by referencing to the results achieved by their neighbours and also the population at large. The success of this kind of collaborative effort has instigated the development of PSO in determining global solutions for hard problems in science and engineering applications. PSO is similar to SGA in many ways. A problem is modelled as an objective function defined by a set of variables, forming a multi-dimensional parametric

80


space. A state (or position) in the latter is encoded into a mathematical expression such as a binary string. Initially, a Swarm of individuals known as particles each representing a state of a problem is established. The population is then evolved into the next generation through the repositioning of its individuals. The process repeats when certain number of epochs has elapsed, or when the objective function represented by the best candidate has exceeded a threshold indicating successful location of the optimal solution. Evolution of the population Gfrom one generation to the next is attained with two major steps. First, suppose xi n denotes the K position of the ‘ith’ particle at the nth generation, its velocity vi is given by

()

G K K v i (n ) = w × v i (n ) + c1 × rand (1) × [ pbest [i ] − x i (n )] G + c 2 × rand (1) × [ pbest [G best ] − x i (n )]

(2)

where

K vi (n ) = [− Vmax ,Vmax ] is the velocity, G xi (n ) = [0, X max ] is the current position of the particle, w = [wmin , wmax ] is the time varying inertia weight,

rand(1) is a random real number between the range [0,1], pbest i is the best position attained so far of particle ‘I’,

[]

Gbest is the particle that has attained the best position pbest [Gbest ] in the whole

swarm, Second, the velocity determines the direction and displacement on where the particle will be relocated in the next instance, as given by

G G K xi (n + 1) = xi (n ) + vi (n )

(3)

A new child population is generated after all the parent individuals have migrated to their new positions. It can be seen from Eq. (2) that we have adopted the “Global Variant” of PSO. In this approach, each particle will move towards its best previous position and towards the best particle in the entire swarm.

4

Proposed Method: Affine Invariant Object Matching Based on PSO

Our method is based on the same matching scheme in [1], but with the Genetic Algorithm replaced with the PSO. An initial population P0 of M particles is established each defining a randomly selected triplet of test points. Following the terminology in section 3, we have

G G G G Pn = {x 0 (n ), x1 (n ), x 2 (n ),....., x M −1 (n )}

(4)


81

The parameter n denotes the epoch of evolution which is set to 0 for the initial population. Subsequent generations are established based on repeated application of Eqs. (2) and (3). The pair of objects is considered as matched with each other if the maximum fitness value in the population has attained a value of 0.65, reflecting a similarity of no less than 65%. If the fitness value fails to reach the threshold within the maximum allowable number of generations the objects are considered as different. The expression in (2) though simple, involves a number of variables that have to be preset properly to effectuate the optimization. To begin with we have adopted the time varying inertia weight suggested in [9] to induce global to local exploration. As for the rest of the unknown parameters, we have conducted repetitive trials in matching different pairs of object contours and arrived at a set of values that give favourable result as listed in Table 1. Table 1. Parameter settings for the PSO scheme

Vmax Vmin c1 c2 wmin wmax Population Size(M) Maximum allowable generation

20 –20 4 4 0.9 1.4 30 100

Experimental results reveal that our method is capable of identifying matched object shapes within 30 generations for about 90% of test cases (i.e. a success rate of 90%). As for the remaining samples, the PSO generally fails within the maximum allowable number of iterations. The success rate is similar to that employing the Migrant Principle [6], but the computation time is significantly shortened. In view of the above observation, we have further enhanced the method by conducting multiple trials on a reduced evolution timeframe. The maximum allowable iteration is set to 30 instead of 100. As for the object shapes that fail to match after 30 generations, the PSO is conducted again for a maximum of two more times. Following the analysis in [4], the success rate is given by:

[1 − (0.1) ]× 100% = 99.9% 3

(5)

The revised algorithm has demonstrated almost 100% success rate. In the best case, a pair of object shapes could be identified in 30 generations or less which is significantly faster than that attained in [6]. A slight decrease in computation time is also noted under the worst condition when all three repeated trials (a maximum of 90 epochs) are required.

82


5

Experimental results

The performance of the proposed scheme in Affine invariant object recognition as compared with the Migrant Principle was illustrated with the use of the four model object contours: a scissors, a spanner, a wrench and a hammer. Two Affine transformed scene contours are associated with each model. The models together with its pair of transformed contours are shown in Figs. 1a to 1d. For a fair comparison all the contours adopted in the experiments are identical to that employed in [6].

Fig. 1a. Model and Transformed Scissors Contours

Fig. 1b. Model and Transformed Spanner Contours


83

Fig. 1c. Model and Transformed Wrench Contours

Fig. 1d. Model and Transformed Hammer Contours

The method reported in [6] is employed to match each model contours to its pair of scene contours with settings as given in Table 2. Table 2. Setting of the Genetic Algorithm

Maximum generations Total population size (P) Mutation rate (m) Condition of success match Number of trials

100 30 0.2 Fitness ≥ 0.65 30

The number of successful match for each model contours after 30 trials are shown in the third row of Table 3. It shows that the successful rate is high but less than 100% for the contours Sci-A, Sci-B and Span-A. The results obtained with the proposed scheme after 30 trials are shown in the fourth row of the same table. If can be seen that 100% successful rate is achieved for all cased based on the same population size.

84


Table 3. Successful Match results based on [6] and the proposed scheme

Migrant Principle Proposed Scheme

Model contour Scene contour P=30, m=0.2 P=30

Scissors

Spanner

Wrench

Hammer

SciA

SciB

SpanA

Span- Wre- Wre- Ham- HamB A B A B

27

29

28

30

30

30

30

30

30

30

30

30

30

30

30

30

At first sight it seems that the improvement is small. However, most of the success match in our scheme is attained in less than 30 generations whereas in [6] the same conclusion could rarely be made within 60 generations.

6

Conclusion

Recently it has been demonstrated that Genetic Algorithm, together with the incorporation of Migrant Principle, could attain good success rate in matching pairs of object contours that are grabbed from different viewpoints. In this paper we have explored the feasibility of further enhancing this benchmark with the use of Particle Swarm Optimization. Our initial finding was that under the same population size and maximum allowable generations, both approaches have similar performance. However, we also observed that in the majority of cases, PSO could identify a pair of matched contours with considerable smaller number of iterations. Based on this important finding we have further enhanced the method by conducting multiple trials on a reduced evolution timeframe. The revised algorithm had demonstrated 100% success rate with reasonable reduction in computation load as compare with the work reported in [6]. At present we are unable to explore further usage of PSO in other different context of contour matching. However, we believe that our favourable findings could be taken as a useful basis for overcoming more complicated problems in computer vision.


P.W.M. Tsang, “A Genetic Algorithm for Affine Invariant Object Shape Recognition”, Proc. Instn. Mech. Engrs., vol. 211, part 1, pp. 385–392, 1997. P.W.M. Tsang, “A Genetic Algorithm for Aligning Object Shapes”, Image and Vision Computing, 15, Elsevier, pp. 819–831, 1997. P.W.M. Tsang, “A genetic algorithm for invariant recognition of object shapes from broken boundaries”, Pattern Recognition Letters, vol. 18, issue 7, pp. 631–639, 1997.

Rate Shape Identification Based on Particle Swarm Optimization [4] [5] [6] [7] [8] [9]

85

S.Y. Yuen, “Guaranteeing the Probability of Success using Repeated Runs of Genetic Algorithm”, Imag. Vis. Comput., 2001, 19, pp. 551–560. P.W.M. Tsang (2001), “A Novel Approach to Reduce the Effects of Initial Population on Simple Genetic Algorithm”, PDPTA ‘2001 , Las Vegas, pp. 457–462. P.W.M. Tsang, “Enhancement of a Genetic Algorithm for Affine Invariant Planar Object Shape Matching using the Migrant Principle”, IEE Proc. Vis. Image Sig. Process., vol. 150, no. 2, pp. 107–113, April 2003. J. Kennedy and R.C. Eberhart, “Particle Swarm optimization”, Proc. IEEE Int. Conf. on Neural Networks, pp. 1942–1948, Perth, 1995. Kennedy, J., “The particle swarm: Social adaptation of knowledge”, Proc. Int. Conf. on Evol. Comput., Indianapolis, pp. 303–308, 1997. Y Shi and R.C. Eberhart (1998). “Parameter selection in particle swarm optimization”, In Evolutionary Programming VII: Proc. EP98, New York: Springer-Verlag, pp. 591–600.

Advanced 3D Imaging Technology for Autonomous Manufacturing Systems

A. Pichler, H. Bauer, C. Eberst, C. Heindl, J. Minichberger PROFACTOR Research, Im Stadtgut A2, 4407 Steyr-Gleink, Austria. {apichl,hbauer,cebers,cheindl,jminich}@profactor.at

1

Introduction

Today’s markets and economies are becoming increasingly volatile, unpredictable, they are changing radically and even the innovation speed is accelerating. Manufacturing and production technology and systems must keep pace with this trend. The impact of novel innovative 3D imaging technology to counter these radical changes is exemplarily shown on the robot paint process. One has to keep in mind that investments in automatic painting lines are considerably high and as the painting line often is the bottleneck in production, it is imperative to prevent nonproductive times and maximize the use of the expensive equipment. Highly flexible, scalable and user-friendly production equipment is needed, including robotic systems for painting – a common process in production. The presented work argues that an intelligent 3D imaging system is mandatory to step forwards towards an autonomous painting system or production system generally. As the shop floor (scenes in the context of computer vision) in industrial environments are of high complexity traditional vision systems are constrained to well-defined tasks lacking adaptation and flexibility. To counter this issue a highly dynamic 3D vision system is required. The paper gives an overview of a novel innovative vision system used for manufacturing applications. In opposite to high-level uncertainties (incl. non-stable and non-deterministic production disturbances such as rush orders) that challenge scheduling algorithms within the (digital) factory, low-level uncertainties can widely be managed locally at the equipment by incorporation of sensory information. While vision-based control to compensate small pose deviations (usually in 2D or 2 1/2D) and calibration errors are state of the art, large deviations in object shape or 3D/6DOF pose corrections at complex objects are beyond. The range image processing to compensate missing data or pose uncertainties described in this paper includes segmentation, feature extraction and recognition/localisation. Related work on segmentation and is presented in [9], on finding features with defined geometric

88

A. Pichler et al.

properties in [3,16]. Recognition and localisation of objects in 3D based on range images has been described in [20]. Related approaches to compensate large uncertainties using planning based on sensory data have been presented in the original “FlexPaint” project [1], the “Fibrescope” Project (Flexible inspection of Bores with an robotic Endoscope) and in [21], where a sensory approach is used to deal with uncertainties in turbine-blade repair was introduced. There are are numerous 3D object recognition methods that are either global, like eigenpictures [17] or eigenshapes [4], or that rely on an initial segmentation of the object [8, 5]. Those methods obtain good results on noise free images, but there deficiencies on global properties which makes them vulnerable to occlusions. A more generic way of approaching the object recognition problem pose spin images [12], which have been shown to yield striking results with cluttered or occluded objects. As this method is based on finding correspondences between image and model regions, it is rather time intensive, though. [11] gives a good overview about current global and local approaches on range images. The first part of the paper describes an autonomous robot painting system, and its evaluation in industrial manufacturing lines. The second part describes 3D imaging workflow integrating adaptive vision, object recognition and localization – and demonstrate its impact on compensating incomplete data and disturbances that are induced by the industrial environment.

2

System Overview

The main concept foresees to derive the robot-process programming (in this paper the painting process) automatically even if the product is unspecified [19, 23]. The proposed robot-system analyses product sensor data of the products to be painted. The system processes a chain of subsequent modules as seen in Fig. 1: (i) 3D image processing, (ii) process planning and (iii) generation of collision free robot motions. As the scene in the task space is unknown 3D sensors are integrated to build a 3D world model. A novel 3D imaging system has been developed combining 3D sensors, object modeling and data analysis to create a 3D model of a robot working area. Next the idea of the employed paint process planning is to link geometries to a process model. This link is established with a flexibility that considers that the precise painting strategy that is mapped to the geometries may vary from customer to customer. Scheduling of individual strokes follows specific criteria, as cycle time or others [23]. Feature-specific paint-strokes for the feature-sets and sequences are converted to collision free tool and robot motions and complemented with air motions to a full speed-optimized paint-application. Automatically generated executable programs are first simulated and than executed on the paint-robots.


89

Fig. 1. System overview of sensor-based robot painting system

Fig. 2. Integrated 3D sensor cell and products hooked up in conveyor system (left); automatic paint process planning result showing paint strokes generated on retrieved 3D world model (right)

3

Inline 3D Image Processing

3.1

Overview

The proposed system combines 3D sensors, object modeling and data analysis to create a 3D model of a robot working area. This paradigm, known as task space scene analysis, provides a much richer understanding of complex, interior work environments than that gleaned from conventional 2-D camera images, allowing a process planning tool to plan and simulate robotic actions within the virtual world model. In order to retrieve a world model the 3D imaging system proceeds as follows (Fig. 3): Range images of the products scanned during their transport in the conveyor are taken using 3D laser range scanners. To retrieve most of the object geometry several sensor systems are necessary. Alignment of the data into a common world coordinate system is achieved using 3D registration method [11]. A meshing algorithm transforms aligned points data fragments into a surface model. In a final step reconstructed data is analysed by a 3D object recognition procedure. Resorting to a product database containing prepared CAD data the system identifies product

90

A. Pichler et al.

Fig. 3. Flow Diagram 3D Image Processing

models in the reconstructed sensor data. Additionally the algorithm implies accurate retrieval of the corresponding position and orientation of the products.

3.2

3D Scanning Process

In response to small product lot size and to compensate uncertainties (position uncertainties, non- stable and non-deterministic production disturbances such as rush orders) a highly flexible sensor cell has been integrated in the production line. As dimension and shape complexity of products is varying over a large scale an adaptive approach to set the viewpoints of the sensors has been chosen. Figure 4 shows a standard setup of the proposed system. Products are carried on a conveyor system which has a defined process flow direction. The scanning process is carried out iteratively. Starting from a standard position all systems are triggered to take initial scans. Aligning the output to a common world coordinate system and transforming it into a surface model retrieves boundary points or holes in the data which give raise to missing data points in the original data set. These positions are fed back in the loop to the scanning process. As the sensor systems are equipped with separate kinematics on their own all system are positioned to a new scanning position to compensate incomplete surface models.

Fig. 4. Inline 3D sensor cell, setup made up of 5 sensor systems each of those is capable of reconfiguring its kinematics configuration in order to adapt to the product geometry


3.3

91

Registration Process

Given the correct sensor calibration (intrinsic + extrinsic parameters) resulting from geometric configuration of laser plane and sensor the sensor data is transformed to 3D coordinates in a local sensor coordinate system. In order to give the sensor system a certain kind of freedom for adapting themselves to the product geometry a simple robot kinematics has been adapted. Using forward kinematics and registration algorithm for fine alignment such as the Iterative Closest Point (ICP) a perfectly aligned data set can thus be retrieved.

3.4

Reconstruction Process

Having an aligned 3D point’s cloud one of the mayor issues is to find a realistic topological structure. That means finding a transformation from point data to surface representation. Numerous of approaches have been developed over the last decades. The marching triangle algorithm [24] gives excellent meshing results on fragmented data at low computational costs. The idea is to polygonize the implicit surface by growing a triangulated mesh according to the local geometry and topology.

3.5

Object Recognition Process

Recognition of three dimensional objects in noisy and cluttered scenes is challenging problem. A successful approach in past research has been the use of regional shape descriptors. Given a reconstructed 3D surface model of the scene produced by a set of 3D sensors the goal is to identify objects in scene (in this case e.g. products in the production line) by comparing them to a set of candidate objects (product CAD database). There are several issues making the process of 3D recognition challenging: (1) the target object is often obscured due to self-occlusion or occluded by other objects. (2) Close objects in the scene act as background clutter, and thus interfere the recognition process, (3) many objects (same product spectrum) have quite the same similarity in dimension and shape, and (4) laser range scanners have limited resolution to produce detailed discriminative features. Generally the recognition task is considered as matching task between two surfaces. The proposed 3D object recognition scheme is based on regional shape descriptors (e.g. spin images, shape context, point signatures) which do not impose a parametric representation on the data, so they are able to represent surfaces of general shape. The identification of products in the task space of a production line requires the knowledge of product CAD data. This kind of representational data stored in the model library must be prepared for the recognition task. In an offline step all models are re-meshed to the same mesh resolution. Shape descriptor signatures of all objects are established and the most distinctive ones of each object are stored in the model library.

92

A. Pichler et al.

Finding correspondences using the correlation coefficient is computationally expensive, and therefore, a different way of managing the information conveyed by shape descriptor signatures is needed. In order to make the process of matching efficient, dimensionality reduction was achieved by projecting shape signatures represented as n-tuples to a space of dimension d < n, using principal component analysis (PCA). Shape descriptor signatures provide a comfortable way of handling generic 3D data. Unfortunately in spite of matching PCA compressed 3D data becomes an extensive task when dealing with hundreds of 3D models as common in industry. Furthermore regional shape descriptor signatures are quite redundant and tend to provide ambivalent results. These known issues in mind the algorithm has been extended by a distinctive point detection algorithm to find the most representative shape descriptor signatures of a 3D model. The approach measures the retrieval performance of all shape descriptor signatures of a model against all remaining models in the database. Identified distinctive shape descriptor signatures showing high retrieval performance to the correct model are selected. A common way of determining similarity between surfaces is the Mahalanobis distance which is used to calculate the likelihood of each descriptor. Shape descriptor signatures of a model are assigned distinctive if their similarity is least likely to the models in the database. As the likelihood is computed with respect to all shape descriptors in an entire database similarities measures are accumulated in a histogram. Basically, distinctive feature are the outliers of the histogram. Normally, histograms of similarity measure will typically have a single mode corresponding to similar shape descriptor signature matches. Distinctive shape descriptor signature matches will be upper outliers in the histogram of similarity measure. For well behaved distributions (i.e., single mode), a standard statistical way of detecting outliers is to determine the fourth spread of the histogram (fs = upper fourth – lower fourth = median of largest N/2 measurements – median of smallest N/2 measurements) [25]. Statistically moderate outliers are 1,5fs units above (below) the upper (lower) fourth, and extreme outliers are 3fs units above (below) the upper (lower) fourth. Figure 5 shows a histogram of distinctiveness measures. Four extreme outliers (points with similarity measure greater than 3fs above the upper fourth) are detected. Through detection of extreme outliers in the histogram of distinctiveness measures, an automatic method for finding distinctive shape descriptor signatures has been created. The actual recognition process is carried out in an online step. The database contains a set of models. Each of them is represented by a vector of n distinctive shape descriptor signatures. Recognition consists of finding the model which correspondents to a given scene that is the model which is most similar to this scene. Randomly selected points are taken from the scene and nearest neighbours are evaluated in the model database. If the distance of a model to a scene point is below a threshold the corresponding model gets a vote. The idea of the voting algorithm is to sum the number of times each model is selected. The model that is selected most often is considered to be the best match. In a verification step all selected models are tested against the scene by matching labeled scene points to corresponding models using Iterative Closest Point [11]. The Hausdorff distance


93

Fig. 5. Distinctiveness computation: distinctiveness histogram (left), distinctive points on 3D model (right)

Fig. 6. Flow diagram 3D object recognition process

between scene and model points has been used as measure of the quality of the recognition result.

4

Experiments

The system has been tested on a variety of industrial use cases. The database contains 200 industrial parts mainly from automotive sector. Figure 7 shows an excerpt of the database. In an offline stage all models have been prepared to homogeneous mesh resolution and distinctive points of every model have been retrieved. Experiments have been carried out on scenes where objects are hooked up in a conveyor system which is a common way in industry. Two representative experiments are shown in the paper. The result obtained from experiment 1 and 2 are listed in Table 1 and graphically presented in Fig. 8.

94

A. Pichler et al.

Fig. 7. Excerpt of model database (200 CAD models of industrial parts)

Fig. 8. Recognition result: scene 1 (left) and scene 2 (right); reconstructed data (green, solid) and superimposed recognized CAD model (grey, wireframe) Table 1. Experimental results of 3D recognition performance Exp. 1

2

Object 1 2 3 4 5 6 7 8 9 1 2 3

Model Overlap 0.41 0.46 0.51 0.45 0.67 0.53 0.59 0.65 0.61 0.42 0.35 0.49

Match Correlation 0.93 0.92 0.95 0.94 0.92 0.93 0.96 0.94 0.93 0.98 0.97 0.99

Recognition Time [ms] 954 865 856 765 967 841 745 618 831 432 218 342


95

The scenes feature the typical amount of clutter and background noise. They have been processed as described in above sections. The experiments consist of acquiring 3D sensor data, running the recognition algorithm on the scenes and then measuring the recognition rate in each scene. The first experiment has been carried out on a scene with 11 objects. Figure 8 (left) shows the simultaneous recognition of 9 models from the library of 200 models. The reconstructed mesh is visualized as solid and the superimposed CAD models are displayed as wire-frame model. The scene addresses a few interesting problems in object recognition. Part of the conveyor system is visible in the scene and is handled as background noise. There are obscured objects due to occlusion and self occlusion. Basically the recognition algorithm performs excellent. In Table 1 the recognition rate for the detected objects is above 90%. The recognition time for each of the objects is less than 1 second on a Pentium 4, 3 GHz, 1 GB Ram. The recognition algorithm failed when only a small portion of the model was visible. The second experiment shows a smaller scene. The recognition rate for all objects is above 95%. Again, recognition time is below 500 ms for each object. As can be seen from the results the system shows excellent performance in an industrial setup. A mayor step towards more robustness, stability and minimum computational time makes the system applicable for industrial applications.

5

Conclusion

In this paper a novel 3D imaging system has been proposed to be integrated in a production system in the shop floor. It enables 3D recognition and localization of products to be processed inline. Having a world model of the products in the task space sophisticated process-specific planning systems are used to generate robot motions. The developed 3D imaging workflow comprising adaptive sensing, reconstruction and recognition is tightly coupled to compensate uncertainties and disturbances which are ubiquitous in an industrial surrounding. To handle arbitrary object shapes a highly flexible 3D sensor cell has been integrated. The sensor systems adapt to product geometry based on initial scans to cover most of the object geometry. The computation of distinctive shape descriptor signatures contributes to more robustness and less computation costs in the recognition process. Furthermore it assigns unique complex feature points to CAD models getting rid of redundant data. Robust 3D object recognition and pose estimation contributes to the realization of small lotsize robotic applications if part or scene complexity is very high. Having a flexible 3D imaging system paired with automatic process planners the approach has proven to be a promising alternative to conventional teach in and OLP based programming and has shown feasible especially for high variant, low volume parts. Additionally it contributes to the realization of autonomous manufacturing systems.

96

A. Pichler et al.

The 3D imaging systems has been integrated into several robotic paint lines in North America and Europe. Future work will focus on classification of 3D models allowing recognizing new product variants on the basis of prototypical product shapes. Furthermore other fields of automatic process planning (inspection, sand blasting) will be investigated.

References [1] [2] [3] [4] [5] [6] [7] [8] [9]

[10] [11] [12] [13] [14] [15]

“Flexpaint.” [Online]. Available: www.flexpaint.org. Autere, “Resource allocation between path planning algorithms using meta a*,” in ISRA, 1998. N.W.C. Robertson, R.B. Fisher and A. Ashbrook, “Finding machined artifacts in complex range data surfaces,” in Proc. ACDM2000, 2000. R.J. Campbell and P.J. Flynn, “Eigenshapes for 3D object recognition in range data,” pp. 505–510. [Online]. Available: citeseer.ist.psu.edu/137290.html. O. Camps, C. Huang, and T. Kanungo, “Hierarchical organization of appearancebased parts and relations for object recognition,” 1998. [Online]. Available: citeseer.ist.psu.edu/camps98hierarchical.html E. Freund, D. Rokossa, and J. Rossmann, “Process-oriented approach to an efficient off-line programming of industrial robots,” in IECON 98: Proceedings of the 24th Annual Conference of the IEEE Industrial Electronics Society, 1998. P. Hertling, L. Hog, L. Larsen, J. Perram, and H. Petersen, “Task curve planning for painting robots – part i: Process modeling and calibration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, no. 5, pp. 324–330, April 1996. R. Hoffman and A.K. Jain, “Segmentation and classification of range images,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 9, no. 5, pp. 608–620, 1987. A. Hoover, G. Jean-Baptiste, X. Jiang, P.J. Flynn, H. Bunke, D.B. Goldgof, K.K. Bowyer, D.W. Eggert, A.W. Fitzgibbon, and R.B. Fisher, “An experimental comparison of range image segmentation algorithms,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 18, no. 7, pp. 673–689, 1996. [Online]. Available: citeseer.csail.mit.edu/hoover96experimental.html N. Jacobsen, K. Ahrentsen, R. Larsen, and L. Overgaard, “Automatic robot welding in complex ship structures,” in 9th Int. Conf. on ComputerApplication in Shipbuilding, 1997, pp. 410–430. R.J.Campbell and P.J. Flynn, “A survey of free-form object representation and recognition techniques,” Comput. Vis. Image Underst., vol. 81, no. 2, pp. 166–210, 2001. A. Johnson and M. Hebert, “Using spin images for efficient object recognition in cluttered 3D scenes,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 21, no. 5, pp. 433–449, May 1999. T. Kadir and M. Brady, “Scale, saliency and image description,” International Journal of Computer Vision, vol. 45, no. 2, pp. 83–105, 2001. K.K. Gupta and A.D. Pobil, “Apartical motion planning in robotics: Current approaches and future directions,” 1998. K. Kwok, C. Louks, and B. Driessen, “Rapid 3-d digitizing and tool path generation for complex shapes,” in IEEE International Conference on Robotics and Automation, 1998, pp. 2789–2794.


97

[16] D. Marshall, G. Lukacs, and R. Martin, “Robust segmentation of primitives from range data in the presence of geometric degeneracy,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 3, pp. 304–314, 2001. [17] H. Murase and S.K. Nayar, “Visual learning and recognition of 3-d objects from appearance,” Int. J. Comput. Vision, vol. 14, no. 1, pp. 5–24, 1995. [18] M. Olsen and H. Petersen, “A new method for estimating parameters of a dynamic robot model,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 17, no. 1, pp. 95–100, 2001. [19] A. Pichler, M. Vincze, O. M. H. Anderson, and K. Haeusler, “A method for automatic spray painting of unknown parts,” in In IEEE Intl. Conf. on Robotics and Automation, 2002. [20] F.R.B., F. A.W., M. Waite, O.M., and E. Trucco, “Recognition of complex 3-d objects from range data,” in CIAP93, 1993, pp. 509–606. [21] X. Sheng and M. Krmker, “Surface reconstruction and extrapolation from multiple range images for automatic turbine blades repair,” in IEEE IECON Conference, 1998, pp. 1315–1320. [22] W. Tse and Y. Chen, “A robotic system for rapid prototyping,” in IEEE International Conference on Robotics and Automation, 1997, pp. 1815–1820. [23] C. Eberst, H. Bauer, H. Nöhmeyer, J. Minichberger, A. Pichler, G. Umgeher, “Selfprogramming Robotized Cells for Flexible Paint-Jobs”, International Conference on Mechatronics and Robotics 2004, Aachen, Germany. [24] A. Hilton, A.J. Stoddart, J. Illingworth and T. Windeatt, Marching Triangles: Range Image Fusion for Complex Object Modelling’, IEEE 1996 International Conference on Image Processing. [25] J. Devore, Probability and Statistics for Engineering and Sciences. Brooks/Cole, Belmont, CA, 1987.

Vision Based Person Tracking and Following in Unstructured Environments

Mahmoud Tarokh and John Kuo Department of Computer Science, San Diego State University, San Diego, CA 92124, USA.

1

Introduction

Vision based tracking and following a person by a robot equipped with a vision system has many applications such as surveillance and motion capture, and detection and following intruders. Such a robot can also be used as a human assistant for carrying tools and equipment and helping elderly. The major requirement in these applications is the ability to track and follow a moving person through nonpredetermined, unstructured and often rough environments. Vision based robotic person following consists of two main tasks – providing sensory (visual) feedback about the location of the person relative to the robot, and issuing signals to robot actuators, e.g. steering and wheel motors, to follow the person. When the camera is fixed, the simplest and fastest method for detecting moving objects is frame differencing, which compares consecutive image frames (Cai 1995; Richards 1995). However, the major challenge in the tracking task is the detection of person’s motion by a camera mounted on a moving robot as these two motions are blended together. A number of approaches have been proposed to address this issue, e.g. tracking features (Censi 1999; Zoghlami 1997, Foresti 2003) and computing optical flow (Srinivasan 1997; Irani 1994). In (van Leeuwen 2002) a method is proposed to track cars in front using a camera mounted on the pursuing car. A color based tracking system capable of tracking color blobs in real time is implemented on a mobile robot (Schlegel 2000), but requires the person to wear a shirt of specified color and does not consider shape. An approach to recognition of a moving person by a camera mounted on a robot is provided in (Tanawongsuwan 1999) which also uses color recognition. These approaches are only effective in environments that do not contain objects whose color is similar to that of the person to be tracked. More recently, a probabilistic approach is proposed which is based on frame differencing with a compensation for the robot mounted camera motion (Jung 2004).

100

M. Tarokh and J. Kuo

Considerable research work has been reported in the area of autonomous robot navigation, but very few addressed person following. In particular numerous fuzzy-logic based approaches have been developed for navigation (e.g. see Saffiotti 1997 for a review). Fuzzy logic has been applied to the wall following and obstacle avoidance problem (Braunstingl 1995). Research reported in (Weng 1998) uses vision to guide a mobile robot by comparing images to a database of images that are created during an initialization tour of the environment. Regardless of the approach, navigation and tracking using maps require that the environment be known prior to application, which limits flexibility and is not a valid approach to person following. We recently reported a simple vision based robotic person following for flat environments using a grey-scale camera that was fixed to a mobile robot platform (Tarokh and Ferrari 2003). The purpose of the present paper is to enable robust person following in rough terrain. In this work, we employ color and shape for person identification and an articulated camera which pan and tilt actuators for robust person following.

2

Person Identification

One of the main tasks in person following is the detection and segmentation of the person from the scene. This task consists of two subtasks, namely, training a detection system and recognition of the person as he/she moves in the environment. Both these subtasks employ color and shape characteristics. In our system, the person appears in front of the camera at the start of a tour, and images of the person are captured automatically when the person takes several poses, i.e. back and side view of the person. The system is then trained to recognize the shape and color of the person’s upper body. We use H (hue or color), S (saturation or color depth), B (brightness or lightness) color model, as HSB is based on direct interpretation of colors and provides a better characterization compared to other color models such as RGB for this application. The averages of H, S and B components for the poses are recorded, which provide the nominal values H nom , S nom and B nom. However these values will go through changes during the motion due to the change in lighting. We, therefore, allow deviations ΔH , ΔS , ΔB from the nominal values, which are found experimentally. Thus during the person following, if an object in the image has color components within the reference values H ref = H nom ± ΔH, S ref = S nom ± ΔS and Bref = Bnom ± ΔB, then the object will be a candidate for the person’s image, and its shape measures are checked. The shape identification system is trained with the above mentioned poses. Shape measures must be invariant to the mass (area) changes of the person’s image since the mass changes with the distance of the robot to the person. The three measures that satisfy this requirement are compactness C, circularity Q and eccentricity E. Equations for computing these shape measures are given in (Tarokh


101

2003), where the normalized values of the three measures are between 0 and 1. During the training, each of these measures is evaluated for the person in each of the above two poses (k = 1, 2) and their values C k, ref , Q k ,ref and E k ,ref are stored for the person following phase. This completes the training of the system, which takes a few seconds on a standard PC, and can be considered as an off-line phase. During person following, the camera takes images of the scene and the system performs several operations to segment (isolate) the person from other objects in the image. The first operation is to scan every pixel and mark the pixel as belonging to the person image, e.g. set it is to white if all its three color components are within the reference color ranges H ref , S ref and B ref . This process of checking all pixels is time consuming, and therefore we speed it up by considering two observations. First, since the person’s image occupies a large portion of the image, it will be sufficient to check pixels on every other row and every other column for color verification. In this manner, only a quarter of the pixels are checked and marked white if they satisfy the color range. The skipped pixels will be marked white if the checked pixels around them have been marked white. The second observation is that there is a maximum distance that the person can move between two consecutive frames. As a result, the person’s pixels in the current frame must all lie within a circle centered at the centroid (to be defined shortly) of the previous frame. These two observations limit the number of pixels to be checked and speed up the marking of the pixels that belong to the person’s image. The final operation is to perform a standard region growing on the marked pixels so that connected regions can be formed. Regions smaller in area than a specified value are considered noise and are removed. The shape measures values C i , Q i and E i for the remaining regions are computed, where i = 0,1,2,…,m-1 denote the region numbers. Rather than checking each shape parameter with its corresponding reference value, we define a single measure for the closeness of the detected region to the reference region, i.e. the person’s image during the training. A possible function σ is given in Tarokh (2003). The closeness function produces 1 if all shape measures of the region are the same as the reference value, and approaches zero if the region shape measures are completely different. It is noted that for each detected region, two shape measures are found, i.e. one for each pose. The region that has the largest value of closeness σ is selected, and if this value is close to 1, the selected region is assumed to represent the person. If all the regions have small values of σ , then none is chosen and another image is taken and analyzed. The above method of distinguishing the region corresponding to the person from other detected regions in the image is simple and yet quite effective. There are several reasons for this effectiveness. One is that the robot is controlled reasonably close to the person being followed and in the direction of person’s motion, as will be seen in the next section. This allows only few objects in the camera’s view making the person identification reasonably easy. Furthermore, the simplic-

102


ity of image processing tasks allows fast computation, making it possible to achieve relatively high sample rates. We must now determine several characteristics of the detected region representing the person in the image. These characteristics will be used for the robot control. The area or the mass of the region is important since it gives a measure as to how close the person is to the camera mounted on the robot. A large mass is indicative of a person that is close to the camera, whereas a small mass implies that the person is far away. The mass (area) M is simply equal to the total number of pixels in the region. The coordinates of the center of the mass, denoted by x c , y c is defined as xc =

1 M

⎡ ⎤ ⎤ 1 ⎡ ⎢ ∑ y ⎥ ; yc = ⎢ ∑x ⎥ M ⎢⎣∀( x , y)∈R ⎥⎦ ⎢⎣∀( x , y )∈R ⎥⎦

(1)

where x, y are the coordinates of a pixel in the region, and the summation is taken over all pixels in the region R. It is noted that we assign the x-axis across the field of camera view, and the y-axis along the field of view, i.e. along the path of the person. The center of mass is of importance for person tracking because it provides the coordinates of the point to be tracked by the robot.

3

Fuzzy Tracking and Following Control

The objective of the robot control is to follow the person and keep a reasonably constant distance to him/her. There are four actuation quantities to be controlled, as shown in Fig. 1. These are camera pan or yaw angle β, camera tilt or pitch angle θ, robot forward/backward speed v, and robot steering angle ϕ . We explored standard PID controllers for these tasks. However, due to ambiguities and imprecision in the image information, PID controllers required frequent gain tuning, which were not practical. We therefore, use a fuzzy control paradigm which is effective in dealing with such imprecision and provides for a natural and intuitive rules base specification for this application. The image information, namely the person’s mass M, the center of the mass ( x c , y c ) and their derivatives ( dx c / dt , dy c / dt ), are the sensed/computed quantities. Note that the derivative (e.g. dx c / dt ) is computed as a change in the quantity between two samples (e.g. Δx c ) divided by the sample time, which is taken as the unit time. Thus in what follows, we use the derivative and the difference interchangeably. For reasons that will become clear shortly, instead of the current values β, θ and ϕ, the changes in these quantities from the last values, i.e. Δβ, Δθ and Δϕ are implemented. Each of the sensed and actuation quantities is treated as a fuzzy (linguistic) variable with five normalized membership functions as given in Fig. 2. The steering is not included in this table, and its value will be determined using the average of the camera pan (yaw), as will be described later. The fuzzy sets Set 1, … Set 5 are given specific names for each fuzzy variable as listed in Table 1, where the


103

Fig. 1. Robot actuation quantities

Fig. 2. Normalized membership functions

fuzzy variables are shown with a tilde. For example, the fuzzy sets for the x-axis x c that describes motion across the field of the center of the mass fuzzy variable ~ of view of the camera are named Far Left, Center, etc. Similarly, the fuzzy sets for the y-axis of the mass are called Down, Up, etc. depending on where the person appears in the image. Each of the sensed and actuation quantities is treated as a fuzzy (linguistic) variable with five normalized membership functions as given in Fig. 2. The steering is not included in this table, and its value will be determined using the average of the camera pan (yaw), as will be described later. The fuzzy sets Set 1, Set 2,…, Set 5 are given specific names for each fuzzy variable as listed in Table 1, where the fuzzy variables are shown with a tilde. For example, the fuzzy sets for the xaxis of the center of the mass fuzzy variable ~ x c that describes motion across the field of view of the camera are named Far Left, Center, etc. Similarly, the fuzzy sets for the y-axis of the mass are called Down, Up, etc. depending on where the person appears in the image.

104


Table 1. Definition of fuzzy variables and associated sets

We propose the following scheme that decomposes the control task into three controllers for pan, tilt and speed. Steering control will be discussed later. The main tasks of the camera pan and tilt controllers are to position the camera so that the person is in the camera’s sight from which the person’s whereabouts can be deduced. The purpose of the robot speed controller is to keep a nearly constant distance between the robot and the person. Consider first the pan (yaw) controller. When the person moves to the left, the image of the person will be shifted to the left of the frame along the image x-axis if the camera and the robot are stationary. Thus the person’s center of mass in the x-direction, x c , is an indication of the location of the person across the field of view. Furthermore, Δx c = x c (k ) − x c (k − 1) gives the amount and direction of the change from the last sample, where k denotes the current sample (frame) value and (k–1) denotes the previous value of x c . The speed controller takes two inputs, namely the person’s image mass M and the change in the camera tilt Δθ. The mass is a measure of the person’s distance to the camera and the larger this mass, the closer the person will be to the camera, and vice versa. The tilt is used to account for hilly terrain When Δθ is positive as in the case of the person starting to climb a hill, the robot must slow down and when Δθ is negative, as in the case of the person starting to descend a hill, it must speed up. These considerations lead to the rule matrix in Table 4. The center of gravity defuzzification is used to determine the crisp value of the camera pan and tilt, and robot speed. The final control quantity is the steering. Although it is possible to employ fuzzy rules for determining the steering control similar to the other three quantities, it is simpler and more reasonable to base the robot steering on the pan (yaw) of the camera. This is due to the observation that the camera rotates to keep the person in its view and thus essentially follows the


105

person’s turning motions, which must eventually cause the rotation (steering) of the robot. It will be unnecessary and undesirable to steer the robot at the same rate as the camera pan. In other words, the camera must track relatively fast and fine motions of the person, whereas the robot must follow the gross motion of the person which is the average motion taken over a time period. As a result of this averaging, the steering is computed as ϕ = K ∫ θ dt where K is the proportionality constant. Table 2. Fuzzy rule matrix for camera pan control.

Table 3. Fuzzy rule matrix for camera tilt control.

106


Table 4. Fuzzy rule for robot speed control

4

Indoor and Outdoor Experiments

The robot base used in the experiments was an ActiveMedia Pioneer2 All-Terrain rover, as shown in Fig. 3. The robot dimensions are 50 × 49 × 26 cm, it has four motorized wheels, and can travel at a top speed of 1.6 m/s. The Pioneer 2 is capable of holding 40 kg and has a battery life of 10–12 hours. The robot has a sonar ring with 8 sensors, which has an operation range of 15 cm to 7 m. The sonar sensors, seen in Fig. 3 as circles, are used for obstacle detection. If obstacles are detected in the robot’s path, a collision avoidance procedure is executed. This procedure will not be discussed in this paper for the sake of brevity. A Canon VC-C4 camera installed on the Pioneer 2 (Fig. 3), and permits color image capture at maximum resolution of 640 horizontal lines and 480 vertical lines in the NTSC format. It is connected to a laptop computer through an Imperx VCE-B5A01 PCMCIA frame gabber, which is specifically designed for laptops. The frame grabber can achieve capture rates of 30 frames/second at the lowest resolution of 160 × 120 in NTSC format, and 5 frames per second at the highest resolution of 640 × 480 . The laptop mounted on the base (Fig. 3) is an IBM T40 with Windows XP operating system. It contains an Intel Centrino processor running at 1.5 MHz.

Fig. 3. Pioneer 2 all-terrain rover used in experiments


107

Fig. 4. Examples of person following in unstructured environments

The application uses a variety of software libraries written by third-party for creating interface and enabling device control. The libraries for the user interface are written in Java, whereas libraries for low motor control are in C++. As a result our person following code was written both in Java and C++. The person following application uses the client server, distributed callback, model view controller. The cycle (sample) time for performing various tasks is found to be 0.13 s, or about 8 frames per second. Extensive indoor and outdoor trials were conducted with the person following system. Indoor trials included passing through a door (Fig. 4a) and hallway (Fig. 4b). Outdoor trials included following up a steep and winding dirt trail (Fig. 4c), a rocky terrain (Fig. 4d) that involved shaking of the rover, and follow-

108


ing with partial occlusion and identification of person to be followed from another person (Fig. 4e and Fig. 4f). The successful experiments in rough terrain and partial occlusion, demonstrate that the person detection and fuzzy controllers are able to cope with shaky images and imprecise or incomplete information. The system even handles full occlusion in cases where the person does not quickly change directions or disappear behind other objects for an extended period of time.

5

Summary and Conclusions

The paper has presented an intelligent control method for person following in previously unknown environments. It consists of simple person identification using both color and shape, and fuzzy controllers for the camera and the robot. It is shown through various experiments that the system can function in both indoors and outdoors. The system has a number of features, which include robustness to noise due to rough terrain traversal, and to partial occlusion. It can perform well in difficult locations such as hallways with tight turns, and winding hilly outdoor trails. A video has been prepared showing person following in various environments, and can be viewed or downloaded from (Tarokh 2005). The system has two limitations. First, it is unable to perform satisfactory person following when the person moves fast. The main bottlenecks are image capture/save and thresholding routine that in combination take more than half of the total cycle. The other limitation is that in bright outdoor lights with distinct shadows, the person identification system can get confused since it treats the shadows as objects/obstacles. We are currently investigating these issues to improve the robustness of the system.

References [1] [2] [3] [4] [5] [6]

Braunstingl, R., P. Sanz and J.M. Ezkerra (1995). Fuzzy logic wall following of a mobile robot based on the concept of general perception, Proc. 7th Int. Conf. on Advanced Robotics, pp. 367–376, Spain. Cai, Q., A. Mitchie and J.K. Aggrarwal (1995). Tracking human motion in an indoor environment, 2nd Int. Conf. on Image Processing. A. Censi, A. Fusiello, and V. Roberto, (1999). Image stabilization by feature tracking, Proc. 10th Int. Conf. Image Analysis and Processing, pp. 665–667, Venice, Italy. G.L. Foresti . and C. Micheloni, (2003). A robust feature tracker for active surveillance of outdoor scenes, Electronic Letters on Computer Vision and Image Analysis, vol 1, No. 1, 21–34. M. Irani, B. Rousso, and S. Peleg, (1994). Recovery of ego-motion using image stabilization, Proc. IEEE Computer Vision and Pattern Recognition, pp. 454–460. B. Jung, and G. Sukhame,( 2004). Detecting moving objects using a single camera on a mobile robot in an outdoor environment, Porc. 8th Conf. Intelligent Autonomous Systems, pp. 980–987, Amsterdam, The Netherlands.

Vision Based Person Tracking and Following in Unstructured Environments [7] [8] [9] [10] [11] [12] [13] [14] [15] [16]

109

C. Richards, C. Smith and N. Papaikolopoulos, (1995). Detection and tracking of traffic objects in IVHS vision sensing modalities, Proc. 5th Annual Meeting of ITS America. A. Saffiotti (1997). “The uses of fuzzy logic in autonomous robot navigation: a catalogue raisonn’e”, Technical Report 2.1, IRIDIA, Universite Libr’e de Bruxelles, Brussels, Belgium. C. Schlegel, J. Illmann, H. Jaberg, M. Schuster, and R. Worz, (2000). Integrating vision based behaviors with an autonomous robot, Journal of Computer Vision Research, Videre, 1 (4), pp. 32–60. S. Srinivasan and R. Chellappa, 1997. Image stabilization and mosaicking using overlapped basis optical flow field. Proc. IEEE Int. Conf. Image Processing. R. Tanawongsuwan, A. Stoytchev and I. Essa (1999). Robust tracking of people by a mobile robotic agent. College of Computing Report, Georgia Institute of Technology. M. Tarokh and P. Ferrari, (2003). Robotic person following using fuzzy logic and image segmentation, J. Robotic Systems, vol. 20, No. 9, pp. 557–568. M. Tarokh, (2005) www-rohan.sdsu.edu/~tarokh/lab/research-person following.html. M.B. Van Leeuwen, F.C. Greon, (2002). Motion interpretation for in-car vsion system, Proc. IEEE/JRS Conf. Intelligent Robots and Systems, Lausanne, Switzerland. J. Weng and S. Chen (1998)Vision-guided navigation using SHOSLIF. Neural Networks, 1, pp.1511–1529. I. Zoghami, O. Faugeras and R. Deriche, (1997). Using geometric corners to build a 2d mosaic from a set of images, Proc. IEEE Conf. Vision Patter Recognition, pp 420–42.

Simple, Robust and Accurate Head-Pose Tracking Using a Single Camera

Simon Meers, Koren Ward and Ian Piper University of Wollongong, Australia.

1

Introduction

Tracking the position and orientation of the head in real time is finding increasing application in avionics, virtual reality, augmented reality, cinematography, computer games, driver monitoring and user interfaces for the disabled. While developing a computer interface for blind computer users, we encountered the need for a robust head-pose tracking system for accurately monitoring the gaze position of the user on a virtual screen. Although many head-pose tracking systems and techniques have been developed, we found most existing systems either added considerable complexity and cost to our application or were not accurate enough for our requirements. For example, systems described in (Horprasert et al. 1996), (Kaminski et al. 2006) and (Newman et al. 2000) use feature detection and tracking to monitor the position of the eyes, nose and/or other facial features in order to determine the orientation of the head. Unfortunately these systems require considerable processing power, additional hardware or multiple cameras to detect and track the facial features in 3D space. Although monocular systems like (Horprasert et al. 1996), (Kaminski et al. 2006) and (Zhu et al. 2004) can reduce the cost of the system, they generally performed poorly in terms of accuracy when compared with stereo or multi-camera tracking systems (Newman et al. 2000). Furthermore, facial feature tracking methods introduce inaccuracies and the need for calibration or training into the system due to the inherent image processing error margins and diverse range of possible facial characteristics of different users. To avoid the cost and complexity of facial feature tracking methods a number of head-pose tracking systems have been developed that track LEDs or infrared reflectors mounted on the user’s helmet, cap or spectacles (see (NaturalPoint 2006), (Foursa 2004), (Foxlin et al. 2004), and (Hong et al. 2005)). However we found the pointing accuracy of systems utilising reflected infrared light (NaturalPoint 2006) to be insufficient for our application. The other LED-based systems, like (Foursa 2004), (Foxlin et al. 2004), and (Hong et al. 2005), still require multi-

112

Simon Meers, Koren Ward and Ian Piper

ple cameras for tracking the position of the LEDs in 3D space which adds cost and complexity to the system as well as the need for calibration. In order to overcome much of the cost and deficiencies in existing head-pose tracking systems we have been developing accurate methods for pinpointing the position of infrared LEDs using an inexpensive USB camera and low-cost algorithms for estimating the 3D coordinates of the LEDs based on known geometry. Our system is comprised of a single low-cost USB camera and a pair of spectacles fitted with three battery-powered LEDs concealed within the spectacle frame. Judging by our results, we believe our system to be the most accurate low-cost head-pose tracking system developed. Furthermore, our system is robust and requires no calibration. Experimental results are provided demonstrating a headpose tracking accuracy of less than 0.5 degrees when the user is within one meter distance from the camera.

2

Hardware

The prototype of our infrared LED-based head-pose tracking spectacles is shown in Fig. 1(a). Figure 1(b) shows our experimental rig for testing the system, which incorporates a laser pointer (mounted below the central LED) for testing the ‘gaze’ accuracy. The baseline distance between the outer LEDs is 147 mm; the perpendicular distance of the front LED from the baseline is 42 mm. Although the infrared light cannot be seen with the naked eye, the LEDs appear quite bright to a digital camera. Our experiments were carried out using a lowcost, standard ‘Logitech QuickCam Express’ USB camera (Logitech 2006), providing a maximum resolution of 640×480 pixels with a horizontal lens angle of approximately 35°. The video captured by this camera is quite noisy, compared with more expensive cameras, though this proved useful for testing the robustness of our system. We filtered out most visible light by fitting the lens with a filter comprising several layers of developed, fully-exposed colour photographic negative. We found it unnecessary to remove the camera’s internal infrared filter. The filtering, combined with appropriate adjustments of the brightness, contrast and exposure settings of the camera, allowed the raw video image to be completely black, with the infrared LEDs appearing as bright white points of light. Consequently the image processing task is simplified considerably.

Fig. 1. (a) Prototype LED Spectacles (b) LED testing hardware


113

The requirement of the user to wear a special pair of spectacles may appear undesirable when compared to systems which use traditional image processing to detect facial features. However, the advantage of being a robust, accurate and lowcost system which is independent of individual facial variations, plus the elimination of any training or calibration procedures can outweigh any inconvenience caused by wearing special spectacles. Furthermore, the LEDs and batteries could be mounted on any pair of spectacles, headset, helmet, cap or other head-mounted accessory, provided that the geometry of the LEDs is entered into the system.

3

Processing

The data processing involved in our system comprises two stages: 1) determining the two-dimensional LED image blob coordinates, and 2) the projection of the two-dimensional points into three-dimensional space to derive the real-world locations of the LEDs in relation to the camera.

3.1

Blob Tracking

Figure 2(a) shows an example raw video image of the infrared LEDs which appear as three white blobs on a black background. The individual blobs are detected by scanning the image for contiguous regions of pixels over an adjustable brightness threshold. Initially, we converted the blobs to coordinates simply by calculating the centre of the bounding-box; however the sensitivity of the three-dimensional transformations to even single-pixel changes proved this method to be unstable and inaccurate. Consequently we adopted a more accurate method – calculating the centroid of the area using the intensitybased weighted average of the pixel coordinates, as illustrated in Fig. 2(b). This method provides a surprisingly high level of accuracy even with low-resolution input and distant LEDs.

Fig. 2. (a) Raw video input (showing the infrared LEDs at close range – 200 mm). (b) Example LED blob (with centroid marked) and corresponding intensity data

114

3.2


Head-Pose Calculation

Once the two-dimensional blob coordinates have been calculated, the points must be projected back into three-dimensional space in order to recover the original LED positions. This problem is not straightforward. Fig. 3 illustrates the configuration of the problem. The camera centre (C) is the origin of the coordinate system, and it is assumed to be facing directly down the z-axis. The ‘gaze’ of the user is projected onto a ‘virtual screen’ which is also centred on the z-axis and perpendicular to it. The dimensions and z-translation of the virtual screen are controllable parameters and do not necessarily have to correspond with a physical computer screen, particularly for blind users and virtual reality applications. In fact, the virtual screen can be easily transformed to any size, shape, position or orientation relative to the camera. Figure 3 also displays the two-dimensional image plane, scaled for greater visibility. The focal length (z) of the camera is required to perform the three-dimensional calculations. The LED points are labelled L, R and F (left, right and front respectively, ordered from the camera’s point of view). Their two-dimensional projections onto the image plane are labelled l, r and f. L, R and F must lie on vectors from the origin through their two-dimensional counterparts. Given our knowledge of the model, we are able to determine exactly where, on the projection rays, the LEDs lie. We know that the front LED is equidistant to the outer LEDs, thus providing Eq. (1).

d ( L, F ) = d ( R, F )

(1)

We also know the ratio r between these distances and the baseline distance.

d ( L, F ) = rd ( L, R )

(2)

These constraints are sufficient for determining a single solution orientation for the model. Once the orientation has been calculated, we can also derive the exact physical coordinates of the points, including the depth from the camera, by utilising our model measurements (provided in Section 2).

Fig. 3. Perspective illustration of the virtual screen (located at the camera centre), the 2D image plane, the 3D LED model and its projected ‘gaze’


115

The distance of the model from the camera is irrelevant for determining the model’s orientation, since it can simply be scaled in perspective along the projection vectors. Thus it is feasible to fix one of the points at an arbitrary location along its projection vector, calculate the corresponding coordinates of the other two points, and then scale the solution to its actual size and distance from the camera. We use parametric equations to solve the problem. Thus the position of point L is expressed as: Lx = tlx

(3a)

Ly = tl y

(3b)

Lz = tz

(3c)

Since z is the focal length, a value of 1 for the parameter t will position L on the image plane. Thus there are only three unknowns – the three parameters of the LED points on their projection vectors. In fact one of these unknowns is eliminated, since we can fix the location of one of the points – we chose to fix the location of R to be at depth Rz = z, thus making its x- and y-coordinates equal to rx and ry respectively. The position of the point F is expressed as: Fx = uf x

(4a)

Fy = uf y

(4b)

Fz = uz

(4c)

Substituting these six parametric coordinate equations for L and F into Eq. (1) yields:

( tlx − uf x )

2

+ ( tl y − uf y ) + ( tz − uz ) = 2

2

( rx − uf x )

2

+ ( ry − uf y ) + ( z − uz ) 2

2

(5)

which can be rewritten as: u (t ) =

z 2 ( t 2 − 1) + lx2 t 2 + l y2 t 2 − rx2 − ry2

2 ( z 2 ( t − 1) + lx f x t + l y f y t − rx f x − ry f y )

(6)

Figure 4 shows a plot of Eq. (6). It should be noted that the asymptote is at: t=

rx f x + ry f y + z 2 lx f x + l y f y + z 2

(7)

and that the function has a root after the asymptote. Now we can calculate the point on the front-point projection vector which is equidistant to L and R, given a value for t. Of course, not all of these points are valid – the ratio constraint specified in Eq. (2) must be satisfied. Thus we need to also calculate the dimensions of the triangle formed by the three points and find the parameter values for which the ratio matches our model.

116


Fig. 4. Relationship between parameters t and u

Fig. 5. Triangle Baseline Distance

The baseline distance of the triangle is given by Eq. (8) and plotted in Fig. 5. b (t ) =

( rx − tlx )

2

+ ( ry − tl y ) + ( z − tz ) 2

2

(8)

The height of the triangle is given by:

h (t ) =

((u (t ) f − tl ) + (u (t ) f x

y

)

− tl y ) + ( u ( t ) z − tz ) − ( b ( t ) / 2 ) 2

2

x

2

2

(9)


117

Fig. 6. Triangle Height

Figure 6 shows a plot of Eq. (9). It should be noted that this function, since it is dependent on u(t), shares the asymptote defined in Eq. (7). At this stage we are not interested in the actual baseline distance or height of the triangle – only their relationship. Figure 7 shows a plot of h(t)/b(t). The function has a near-invisible ‘hump’ just after it reaches its minimum value after the asymptote (around t=1.4 in this case). This graph holds the key to our solution, and can tell us the value of t for which the triangle has a ratio which matches our model. Unfortunately, it is too complex to be analytically inverted, so we must resort to root-approximation techniques to find the solution. Thankfully, we can reduce the solution range by noting two more constraints inherent in our problem. Firstly, we know that we are looking for a solution in which the head is facing toward the camera. Rearward facing solutions are considered to be invalid as the user’s head would obscure the LEDs. Thus we can add the constraint that: Fz < M z

(10)

where M is the midpoint of line LR. This can be restated as: u ( t ) f z < ( tlz + z ) / 2

Fig. 7. Triangle Height/Baseline Ratio

(11)

118


Fig. 8. z-coordinates of F and M

Figure 8 shows the behaviour of the z-coordinates of F and M as t varies. It can be seen that Eq. (10) holds true only between the asymptote and the intersection of the two functions. Thus these points form the limits of the values for t which are of interest. The lower-limit allows us to ignore all values of t less than the asymptote, while the upper-limit crops the ratio function nicely to avoid problems with its ‘hump’. Hence we now have a nicely behaved, continuous piece of curve on which to perform our root approximation. The domain could be further restricted by noting that not only rearward-facing solutions are invalid, but also solutions beyond the rotational range of the LED configuration; that is, the point at which the front LED would occlude one of the outer LEDs. Our prototype LED configuration allows rotation (panning) of approximately 58° to either side before this occurs. The upper-limit (intersection between the Fz and Mz functions) can be expressed as: t≤

− S − −4 ( − lx2 − l y2 + lx f x + l y f y )( rx2 + ry2 − rx f x − ry f y ) + S 2 2 ( − lx2 − l y2 + lx f x + l y f y )

(12)

where S = f x ( lx − rx ) + f y ( l y − ry ) . Note that this value is undefined if lx and ly are both zero (l is at the origin) or one of them is zero and the other is equal to the corresponding f coordinate. This follows from the degeneracy of the parametric equations which occurs when the projection of one of the control points lies on one or both of the x- and yaxes. Rather than explicitly detecting this problem and solving a simpler equation for the specific case we have chosen instead to jitter all two-dimensional coordinates by a very small amount so that they will never lie on the axes. We have determined that the lower-limit is bounded by the asymptote; however we can actually further restrict the domain by noting that all parameters should be positive so that the points cannot appear behind the camera. Note that the positive


119

Fig. 9. Triangle ratio graph with limits displayed

root of Eq. (6) (illustrated in Fig. 4) is after the asymptote. Since u must be positive, we can use this root as the new lower-limit for t. Thus the lower-limit is now: t≥

rx2 + ry2 + z 2 lx2 + l y2 + z 2

(13)

Figure 9 illustrates the upper and lower limits for root-approximation in finding the value of t for which the triangle ratio matches the model geometry. Once t has been approximated, u can be easily derived using Eq. (6), and these parameter values substituted into the parametric coordinate equations for L and F. Thus the orientation has been derived. Now we can simply scale the solution to the appropriate size using the dimensions of our model. This provides accurate three-dimensional coordinates for the model in relation to the camera. Thus the user’s ‘gaze’ (based on head-orientation) can be projected onto a ‘virtual screen’ positioned relative to the camera.

4

Experimental Results

Even using as crude a method of root-approximation as the bisection method, our prototype system implemented in C++ on a 1.3 GHz Pentium processor took less than a microsecond to perform the entire three-dimensional transformation, from two-dimensional coordinates to three-dimensional head-pose coordinates. The t parameter was approximated to ten decimal place precision, in approximately thirty bisection approximation iterations. To test the accuracy of the system, the camera was mounted in the centre of a piece of board measuring 800 mm × 600 mm. A laser-pointer was mounted just below the centre LED position to indicate the ‘gaze’ position on the board. The system was tested over a number of different distances, orientations and video resolutions. The accuracy was monitored over many frames in order to measure the system’s response to noise introduced by the dynamic camera image. Table 1 and Fig. 10 report the variation in calculated ‘gaze’ x- and y-coordinates when the

120


position of the spectacles remained static. Note that this variation increases as the LEDs are moved further from the camera, because the resolution effectively drops as the blobs become smaller (see Table 2). This problem could be avoided by using a camera with optical zoom capability providing the varying focal length could be determined. Table 1. Horizontal and vertical ‘gaze’ angle (degrees) resolution Resolution Distance (mm) Avg. x-error Max. x-error Avg. y-error Max. y-error

320×240 pixels 500 1000 0.09° 0.29° 0.13° 0.40° 0.14° 0.32° 0.22° 0.46°

1500 0.36° 0.57° 0.46° 0.69°

2000 1.33° 2.15° 2.01° 2.86°

640×480 pixels 500 1000 0.08° 0.23° 0.12° 0.34° 0.10° 0.20° 0.15° 0.29°

1500 0.31° 0.46° 0.38° 0.54°

2000 0.98° 1.43° 1.46° 2.15°

Fig. 10. Horizontal and vertical ‘gaze’ angle (degrees) resolution graphs Table 2. LED ‘blob’ diameters (pixels) at different resolutions and camera distances 640×480 pixels 320×240 pixels

500 mm 20 7

1000 mm 13 5

1500 mm 10 4

2000 mm 8 3

To ascertain the overall accuracy of the system’s ‘gaze’ calculation, the LEDs were aimed at fixed points around the test board using the laser pointer, and the calculated gaze coordinates were compared over a number of repetitions. The test unit’s base position, roll, pitch and yaw were modified slightly between readings to ensure that whilst the laser gaze position was the same between readings, the positions of the LEDs were not. The averages and standard deviations of the coordinate differences were calculated, and found to be no greater than the variations caused by noise reported in Table 1 and Fig. 10 at the same distances and resolutions. Consequently it can be deduced that the repeatability accuracy of the system


121

is approximately equal to, and limited by, the noise introduced by the sensing device. As an additional accuracy measure, the system’s depth resolution was measured at a range of distances from the camera. As with the ‘gaze’ resolution, the depth resolution was limited by the video noise. In each case, the spectacles faced directly toward the camera. These results are tabulated in Table 3. Table 3. Distance from Camera Calculation Resolution Distance from Camera Accuracy at 320×240 pixels Accuracy at 640×480 pixels

5

500 mm ±0.3 mm ±0.15 mm

1000 mm ±2 mm ±1.5 mm

1500 mm ±5 mm ±3 mm

2000 mm ±15 mm ±10 mm

Conclusion

The experimental results demonstrate that the proposed LED-based head-pose tracking system is very accurate considering the quality of the camera used for the experiments. At typical computer operating distances the accuracy is within 0.5 degrees using an inexpensive USB camera. If longer range or higher accuracy is required a higher quality camera could be employed. The computational cost is also extremely low, at less than one microsecond processing time per frame on an average personal computer for the entire three-dimensional calculation. The system can therefore easily keep up with whatever frame rate the video camera is able to deliver. The system is independent of the varying facial features of different users, needs no calibration and is immune to changes in illumination. It even works in complete darkness. This is particularly useful for human-computer interface applications involving blind users as they have little need to turn on the room lights. Other applications include scroll control of head mounted virtual reality displays or any application where the head position and orientation is to be monitored.

Acknowledgements Equations (6), (7), (12) and (13) were derived with the assistance of the Mathematica (Wolfram 2006) software package.

References [1]

Foursa, M. (2004) Real-time infrared tracking system for virtual environments. In Proceedings of the 2004 ACM SIGGRAPH international conference on Virtual Reality continuum and its applications in industry, pages 427–430, New York, USA. ACM Press.

122 [2]


Foxlin, E., Altshuler, Y., Naimark, L. and Harrington, M. (2004) FlightTracker: A novel optical/inertial tracker for cockpit enhanced vision. In ISMAR ‘04: Proceedings of the Third IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR’04), pages 212–221,Washington, DC, USA. IEEE Computer Society. [3] Hong, S.K. and Park, C.G. (2005) A 3d motion and structure estimation algorithm for optical head tracker system. In American Institute of Aeronautics and Astronautics: Guidance, Navigation, and Control Conference and Exhibit. [4] Horprasert, T., Yacoob, Y. and Davis, L.S. (1996) Computing 3-d head orientation from a monocular image sequence. In Proceedings of the Second International Conference on Automatic Face and Gesture Recognition, pages 242–247. [5] Kaminski, J.Y., Teicher, M., Knaan, D., and Shavit, A. (2006) Head orientation and gaze detection from a single image. In Proceedings of International Conference Of Computer Vision Theory And Applications. [6] Logitech (2006) QuickCam Express. http://www.logitech.com. [7] NaturalPoint Inc. (2006) TrackIR. http://www.naturalpoint.com/trackir. [8] Newman, R., Matsumoto, Y., Rougeaux, S. and Zelinsky, A. (2000) Real-time stereo tracking for head pose and gaze estimation. In Proceedings. Fourth IEEE International Conference on Automatic Face and Gesture Recognition, pages 122–128. [9] Wolfram Research Inc. (2006) Mathematica 5.2. http://www.wolfram.com. [10] Zhu, Z. and Ji, Q. (2004) Real time 3d face pose tracking from an uncalibrated camera. In First IEEE Workshop on Face Processing in Video, in conjunction with IEEE International Conference on Computer Vision and Pattern Recognition, Washington DC.

Vision Applications

This section contains five papers on various applications of machine vision. The advent of low cost cameras and high speed video processing has led to a plethora of devices and applications in this area. The ones described in this chapter are some of the more interesting to come along. The first is a most novel application, using machine vision to determine the state of aluminium beer kegs as they arrive at the brewery, determines what their usage has been, and then feeds the information into the keg management system. Next, we go underground to look at the use of millimetre wave radar being used as a vision system to aid mining operations. Machine vision does not necessarily mean using visible light. Then, from underground to underwater again, to see how underwater cameras and instrumentation can aid marine biologists in monitoring fish behaviour on a coral reefs. Back up into the air for a paper on how machine vision can be used to estimate the position, and then the automatic landing of a vertical take off aircraft. Finally, at a different scale altogether, we see how the latest techniques in fingerprint identification are performed.

Machine Vision for Beer Keg Asset Management

Michael Lees1, Duncan Campbell2, Andrew Keir2 1

Foster’s Australia, Yatala Operations School of Engineering Systems, Queensland University of Technology

2

1

Abstract

A typical large brewery could have a keg fleet size in the order of hundreds of thousands. For some breweries, the asset value of this fleet is second only to the fixed plant. The annual rate of attrition within the fleet can range from 5% to 20%, a sizable figure in dollar terms with a stainless steel keg costing around USD100. There is a business case for a keg asset management system that can help to reduce the annual rate of attrition and supply chain cycle time. Established solutions such as bar codes and RFID tags are costly as they require a modification to every keg. The feasibility of a machine vision tracking system based on the optical character recognition (OCR) of the keg’s existing serial number is explored. With prospective implementation in the keg filling line, a process is proposed which is based on neural network OCR. A recognition rate of 97% was achieved for kegs with non-occluded serial numbers, with realistic scope for further improvement.

2

Introduction

The modern day stainless steel keg has proven to be a robust and reliable means of transferring large quantities of beer from the brewery to the various destinations of consumption. However, it has been estimated that for every pouring tap, a total of up to eight kegs are required in support of each tap to cater for supply chain latencies and keg losses (Bryson 2005). A typical large brewery could have a fleet size in the hundreds of thousands, but it is easy to see how fleet sizes can range from anything from tens of thousands (Till 1996) to millions (Schneider 2003) of kegs. In terms of asset value this can be very high. For some breweries it will be second only to the fixed plant (Perryman 1996). Despite the kegs being able to survive upward of 20 to 30 years (Clarke and Simmons 2005), the annual rate of attrition (including loss and theft) can range from 5% to 20% depending on location (Bry-

126

Michael Lees, Duncan Campbell, Andrew Keir

son 2005). With a stainless steel keg costing around USD100 each, this becomes a considerable annual financial burden for the brewery (Bryson 2005). Clearly there is a business case for a keg asset management system that can help to reduce the annual rate of attrition and supply chain cycle time. Due to a keg’s cycle time in the trade, it will only spend a fraction of its life on the actual company premises. These relatively brief moments are primarily for activities such as cleaning, filling and repair (when necessary). This is the only time available for collecting information that is required to effectively manage the fleet. The type of information required includes the current size of the fleet, and the average age and the condition of the kegs. Currently these are estimates based on visual assessments and counts of kegs moving through the plant. The brewery’s ability to manage keg assets could be significantly improved if each keg could be automatically identified once it enters the keg line. Due to the financial dimensions of keg fleet management, the concept of keg tracking has become popular (Till 1996, 1997). In recent times a variety of different techniques and solutions have been proposed. These include regular bar codes, advanced laser-etched two dimensional bar codes (Clark and Simmons 2005), as well as RFID tags (Perryman 1996; Pankoke et al. 2005). The RFID tag option has proven to be popular with third party keg management companies.1 However most of the proposed techniques, including these mentioned, require each and every keg to be modified in some way. For a typical fleet size, this means that the application of even the cheapest of barcodes or RFID tags (which can cost up to USD52 depending on the type of transponder and keg) to each and every keg would present a significant cost. It is a task that at best would only ever asymptote towards completion. Another proposed solution to the challenge of keg asset management is the use of non-returnable disposable kegs.3 As each keg already has a unique ID number stamped into the metal dome (the top of the keg), machine vision techniques have attracted particular attention in addressing this problem. Syscona4 offer a product that is designed to read the ID number on the top of brand new kegs. This system has a niche market. It is required by customs in some countries to provide a human-readable but automated verification of the keg’s RFID tag. This is an example of where the application of these two technologies can be complementary. Despite Syscona being able to read the ID number on new kegs, there is still a need for a machine vision system that can read the ID number on the regular (often aged and weathered) kegs in circulation in the trade. By identifying ID numbers in real-time through optical character recognition, coupled to a national keg asset database, a meaningful keg audit could be conducted at any time simply by querying the database. This system would provide the mechanism to establish the keg fleet age distribution, the history of each keg, identify kegs not seen for a prolonged time, identify and remove foreign kegs from the production line, and to enable preventative maintenance to be carried out on 1

www.trenstar.com www.schaeferkegs.com 3 www.ecokeg.com 4 www.syscona.de 2


127

each keg. An automated visual inspection of the keg dome with the filler valve will identify many of the faults that would lead to product loss. This offers a significant saving in lost beer and the transport costs for replacing a customer’s leaking keg. Most importantly, a machine vision OCR solution does not require costly alterations to each and every keg.

3

Problem Statement

The primary research objective is to develop machine vision based techniques which non-invasively identify individual beer kegs prior to filling on the production line. The nature of the problem and the solution methodology offers the further prospective benefit (a secondary objective) of detecting visible keg deformations which are likely to lead to subsequent product loss. Deformed kegs must be detected in real-time so that they can be automatically removed from the production line prior to filling. The feasibility of machine vision techniques is investigated as a potential solution to the serial number identification problem in the first instance and is the subject of this paper. One key measure for the determination of success in terms of serial number recognition is set as a successful identification rate of 98%.

4

Methodology

The challenges in addressing this problem are captured through a set of constraints and parameters, some of which are pre-defined due to the physical properties of the kegs, and some defined in consideration of the operational requirements within the context of a production line. In consideration of such, the first stage of the research methodology was that of an off-line configuration (laboratory) used to determine lighting configurations, adequate image capture requirements and the development of the machine vision techniques using a sample collection of kegs.

4.1

Key Characteristics and Constraints

The keg fleet of interest comprises four different brands of kegs: Spartanburg, Rheem and Thielmann (all of which have a spoked pattern on the dome), and Blefa (which does not have a spoked pattern). Two examples are shown in Fig. 1. They demonstrate two different ID number positions and one dome which has spokes, and one which does not.

128


Fig. 1. ID number positioning for two different keg brands: Blefa (left) and Thielmann (right)

The kegs have the following relevant properties: • • • •

Each keg has a unique ID number stamped on the top dome Each brand of keg has the number stamped in the same orientation Different brands of kegs use different fonts and have different orientations ID numbers can be obscured by foreign substances (dust, dirt, etc.) or even partly corroded Within the context of a production line, the following considerations are made:

• Placement on the production line (prior to filling) • Successful serial number identification and the determination of keg removal from the production line prior to filling must be made within one second (the definition of real-time in this instance)

4.2

Illumination and Image Capture

A key consideration in machine vision applications, is appropriate illumination of the area under inspection. In this instance, the need is to establish adequate illumination to best highlight the ID numbers and to minimise the impact of dome sheen and potential soiling. The proposed illumination configuration for a production line implementation is shown in Fig. 3 (left). The initial laboratory configuration used is shown in Fig. 3 (right). The lighting array modules are Nerlite 51 mm×203 mm, 636 nm wavelength LED arrays. Images were captured with an effective spatial resolution of 4 pixels per millimetre across the keg dome.


129

Fig. 2. Schematic of lighting array concept (left) and simulation of lighting via domed array setup (right)

The following is a list of the key equipment used: • • • •

Sony DFW-700 Firewire Camera Avenir 3.5 mm F1.4 lens (HTCH0354) Navitar Zoom 7000 lens Canon PowerShot G2 Digital Camera

The two lenses were required to examine two widely different aspects of the kegs. The Avenir is a wide angle lens with a field of view, when coupled with the Sony Camera, of (D H V) 112.0° 94.6° 73.5°. This allows imaging of the keg rim and dome from a distance of not more than 280 mm directly above the surface. Alternatively, the Navitar is a zoom lens capable of providing sufficiently resolved images of areas such as the filler valve assembly. Both lenses have manual iris adjustments. The two different cameras were used to examine the resolution differences and subsequent requirements for the process. The Sony Firewire camera integrates with the software used (Halcon) and can be controlled via commands given within the program. However, with limited dynamic range, and a resolution of only 728×1024 pixels, it was found that it did not meet the requirements for clarity when it came to the application of the OCR algorithms. In order to eliminate camera resolution as an inhibiting factor, still shots were taken with a Canon PowerShot G2 digital camera, which has a resolution of 2272×1704 pixels. These images formed the basis of the subsequent processing in this developmental phase, and were treated as equivalent to images captured in real-time by a large format camera.

5

Keg ID Number Recognition

The following image processing methodology was developed and shown in Fig. 3. The pre-processing and classification is discussed in greater detail in this section.

130


Fig. 3. Image capture, conditioning and serial number recognition process

5.1

Pre-processing

Due to the symmetric nature of the kegs, and the inability to know the specific orientation of the kegs on a production line, it was necessary to develop a mechanism whereby the serial number region could be located on the dome, and then targeted for application of the OCR algorithm for classification. Following initial image pre-processing to convert to greyscale (and therefore reducing the pixel depth of the image), global edge enhancement and noise attenuation, a two step approach to segment the image and to extract serial numbers is applied. Reduced computational time was a consideration in selecting the processes within these steps and also in consideration of the computing architecture that would ultimately be commissioned. The first step is one of locating text areas on the dome and extracting likely candidate regions for serial numbers. The second step establishes and extracts serial numbers within the candidate regions. A Laplace of Gauss (LoG) operation (σ = 2) is applied prior to segmentation for noise suppression. This filter uses the two-dimensional derivatives of the Gauss-function together with a Gaussian low pass filter for noise suppression (Klette and Zamperoni 1996). The LoG operator computes the Laplacian Δg(x,y) for an arbitrary smoothing of the Gauss-function σ, giving an expression for the kernel: Δ G σ (x , y ) =

1 2 πσ

4

⎛ x2 + y ⎜⎜ 2 ⎝ 2σ

2

⎞⎡ − 1 ⎟⎟ ⎢ exp ⎠⎣

⎛ x2 + y ⎜⎜ − 2σ 2 ⎝

2

⎞⎤ ⎟⎟ ⎥ ⎠⎦

(1)


131

Edges are defined by zero-crossings following application of the LoG filter. This operator tends to be insensitive to noise and provides faithful representation of edge positions even with low-edge gradients (Gonzalez and Wintz 1987; Russ 1995). A dilation circle is applied to the thresholded image following the LoG filtering operation. Areas containing stamped text, or those with significant scratching, are broadened by a relatively large radius dilating circle. The circle radius is chosen such that sequences of digits making up a serial number appear as a single, elongated area as seen in Fig. 4. The filler valve assembly is an easily locatable reference point corresponding to the centre of the dome from a longitudinal perspective. This effectively addresses any translational shift of the image from true centre along the longitudinal axis. The serial number stamp location varies from brand to brand, however the radial location is relatively consistent within each brand, and there are not a large number of brand types. In each case, a mask is constructed corresponding to the serial number stamping location. The mask(s) can then be digitally rotated about the keg centre point, as defined by the centre of the filler valve, such that as the mask passes over the broadened serial number, a positive detection is made. Analysis of the size of the text present on the dome of the keg showed that the serial number would return a clearly distinguishable pattern quite unlike that of any of the other areas. The Spartanburg, Rheem and Thielmann brands have a spoked pattern with the serial number stamped consistently in relation to the spokes. The spoke pattern is easily distinguishable and can be used as a landmark for six discrete rotational steps of the mask. The serial number must appear within one of the six segments contained within the arms of the spokes (see Fig. 4). This decreases the number of rotational steps required to locate the serial number. This spoke detection can be used as a first pass stage for image rotation and serial number detection. Should the spoke pattern not be detected, the keg is either a Blefa or it is overblown. Assuming it is a Blefa, the serial number location mask is incrementally rotated at relatively small angles (eg. 2.5°). The broadened serial number is detected as it

Fig. 4. Auto-location of serial number using density profiles (left) and location of serial number within a spoked segment (right)

132


passes through the mask. If a serial number is not detected through a complete digital rotation of the mask (either in the spoked case or non-spoked case) then it is poorly discriminated from the background. This can be either due to extreme damage such as widespread abrasions, foreign substances coating the dome such as paint, or the keg may be overblown. This sequence of operations and decisions is summarised in Fig. 6. It is noted that a section of rust/scratches could return a false positive for the serial number section. Whilst requiring further investigation to exclude such an occurrence from interfering with the data, the likelihood of such a section being contained to a small area and not enveloping the whole of the keg dome would be quite small. If a full keg is placed in a cellar that is so cold that the beer freezes, it will expand and stretch the actual keg. This could impact upon brand determination and serial number extraction. Since this stretching almost always results in an increase in the height of the top dome of the keg, and the keg would then be classified as being overblown (see Fig. 5). Given that this condition creates some complications for the machine vision process, it prompted some further consideration as to how to measure whether or not a keg was indeed overblown. It is proposed that three or four high intensity light sources could be focused at predefined spots on the keg dome. The illuminated “spots” would intersect within a pre-defined region of interest, and thereby give an indication as to whether the radius of curvature of the keg is within acceptable bounds.

Fig. 5. Overblown Thielmann keg (left) and planar diagram of an overblown dome (right)


133

Fig. 6. Flow chart of serial number detection and extraction

5.2

Classification

Serial number digits are classified using optical character recognition methods. Two approaches were investigated. One is based on the traditional template matching approach and one using neural networks. Template Matching The basis of the template matching approach to OCR is to match the input pattern image to a template either derived from many exemplars or one which is artificially created. Font tolerance is easily catered for through the appropriate construction of the template. The degree of coincidence (or the total sum of differences) between the pattern and the template give a metric by which a specific character can be identified.

134


Table 1. Experimental recognition rates of template matching based OCR Overall true recognition rate False positives False negatives

72% 25% 3%

The template was created specifically for the Blefa font type, Blefa’s being the newer of the kegs, so as to determine the accuracy of the template matching approach with minimal impact from variations in sheen, scratches and surface damage. Template matching based recognition achieved a 72% overall successful recognition rate – an unsatisfactory result. The false positive cases are defined as those for which a digit was classified and either it did not exist (9%), or it did exist but was incorrectly classified (16%). The false negative cases are those where a valid digit was not recognized at all (3%). This recognition performance appears to be strongly related to three factors. 1. Visually malformed digits that would defy human visual. This was due to instances of poor embossing (stamping) of the digit, and markings occluding digits. The former instance will always provide a challenge. The latter instance can in part be dealt with through caustic cleaning of the keg prior to inspection. 2. The digit registration method to delineate digits relied on the existance of clear space between digits. There were instances where extraneous pixels (byproducts of the pre-processing) effectively joined adjacent digits and thereby eliminated the required inter-digit spacing. Some further tuning of the preprocessing stage as well as using predefined digit pitch spacing may assist in improving the false classifications. 3. Occassionally, extraneous pixels with space on either side, were taken as being candidate digits. This contributed to false positives. Template matching relies heavily on spatial correlation (or coincidence) and is therefore very sensitive to translational and rotational variations. Given the nature of the keg serial number identification problem, variations in translation and rotation are expected. A recognition method with high degrees of tolerance is preferable. Neural Network Computationally intelligent (CI) methods, which are more likely to successfully extract digits that are visually discernable, and are less dependent on digit registration processes, are suggested to be more appropriate. Candidate CI methods include neural networks, fuzzy logic and hybrids thereof. Neural networks were examined as a precursor to potential future development of computationally intelligent keg serial number OCR. A three layer feed-forward neural network (one input layer, one hidden layer and one output layer) was used (Hagan et al. 1996). The number of input layer nodes corresponded to the number of pixels in the scanning pixel array. The number of output neurons corresponds to the number of output classes. It was found that a single hidden layer was sufficient to achieve the results described below.


135

The main consideration in selecting the number of hidden layers and the number of hidden neurons is that of having sufficient neurons to achieve acceptable performance whilst maintaining generalisation within the solutions. In all, the neural network had 6000 input neurons (corresponding to a 60×100 pixel block), 10 output neurons, to identify the ten digits per font, and a single hidden layer of 32 neurons. The activation function for all neurons was a logSigmoid function. The network was trained using a template of the digits arranged such that each numeral was represented by a 60×100 array corresponding to the input digit block. A set of idealised data vectors was firstly used to train the network until it reached a predefined and acceptably low sum-squared error. The training vectors were then augmented with a set of noisy vectors to assist with the generalisation process and facilitating correct identification of noisy digits as well as clean digits. These noisy vectors were created via the addition of random noise with standard deviations of 0.1 and 0.2 to the idealised vector. Table 2. Experimental recognition rates of neural network based OCR Overall true recognition rate False Positives False Negatives

92% 8% 0%

The neural network achieved an improved recognition rate of 92%. No classifications were made for non-existing digits and 8% of existing digits were incorrectly classified. Taking into consideration those digits which were malformed in the stamping process, and those occluded by dirt and markings (as cited above in the template matching case), presenting significant challenges for human visual inspection, a recognition rate of 97% is arguable.

5.3

Discussion

Neural network classification appears to provide reduced sensitivity to digit translation and rotation. Template matching is particularly susceptible to the lack of, or extraneous, inter-digit spacing. Given the nature of the keg ID number recognition problem, and production line implementations, it is apparent that neural network classification is the more robust method. Multiple fonts could be used in the training process of neural networks and therefore rely on the training to tolerate variations. This does place a greater emphasis on correctly configuring and training of the neural network. It also relies on developing an extensive list of exemplars for training to ensure the entire solution space is solved for. Neuro-fuzzy systems have the advantage that the internal configuration of such systems can be defined by humans based on knowledge rather than the somewhat black-box approach with neural networks. Neuro-fuzzy systems can be trained using exemplars to either assist in exploring the ideal configuration, or to tune the internal parameters. In the context of multiple fonts, a neuro-fuzzy system could

136


be manually configured to directly represent all digits in all fonts and then tuned with exemplar data.

5.4

Conclusion

Given the justified need to implement a beer keg tracking and management system, machine vision based techniques offer a solution that does not require tagging or other intrusive modifications to be made to the keg fleet. Conceivably, a vision based inspection station could be placed after the keg external cleaning stage and prior to filling. A vision based system also offers the ability to inspect kegs for visible deformities providing an opportunity to remove the keg from circulation for repair or discarding without loss of product. It is proposed that the inspection station comprises adequate keg handling to ensure keg travel within a field of view encompassing the keg dome. The dome should be illuminated using a circular arrangement of red LED arrays angled such that the contrast between the embossed ID numbers and the dome sheen is greatest. A digital camera with an appropriate lens could be mounted inside the centre of the domed light array. The camera should have a resolution of at least 1704×1704 pixels with the keg dome filling the field of view (hence the need to mechanically guide the keg through the inspection station). The camera should have a dynamic range of at least 16 bits and can be monochrome. Neural network based recognition provides a feasible classification method albeit it did not achieve the target 98% classification rate required in the first instance. It does however demonstrate greater robustness to artifacts (rotation and translation variations) which will be inherent in a production line installation. Classification rates can be further improved through refined image pre-processing techniques, refinement of the neural network architecture and the employment of hybrid techniques such as neuro-fuzzy based classification.

Acknowledgements The authors wish to thank: Dr Peter Rogers and the management of Foster’s Australia for both support of this project and for permission to publish this material, and Queensland University of Technology (QUT) for the loan of camera equipment and lenses that were used within this project.

References [1] [2]

Bryson L (2005) Brewers, Do You Know Where Your Kegs Are? The New Brewer. Sept/Oct, http://www.beertown.org Clark D, Simmons K (2005) A dynamic 2D laser mark, Industrial Laser Solutions, Aug, pp 19–21

Machine Vision for Beer Keg Asset Management [3]

137

Gonzalez RC, and Wintz P (1987) Digital Image Processing, 2nd edn. AddisonWesley, Reading, Massachusetts [4] Hagan MT, Demuth HB, and Beale MH (1996), Neural Network Design, PWS Publishing, Boston, MA [5] Klette R, Zamperoni P (1996) Handbook of Image Processing Operators, John Wiley, Chichester, England [6] Pankoke I, Heyer N, Stobbe N, Scharlach A, and Fontaine J (2005) Using RFID technology to optimize traceability, Proceedings of the European Brewing Convention, Prague, pp 1–7 [7] Perryman M (1996) Practical use of RF tags for tracking beer containers. Brewer’s Guardian, Nov, pp 29, 33 [8] Russ JC (1995) The Image Processing Handbook, 2nd edn. CRC Press, Boca Raton, Florida [9] Schneider M (2003) Radio Frequency Identification (RFID) Technology and its Application in the Commercial Construction Industry. M.Sc. Thesis, University of Kentucky [10] Till V (1996) Keg Tracking – a Method to Control the Keg Fleet. Proceedings of the Twenty Fourth Convention of the Institute of Brewing Asia Pacific Section, pp 170– 173 [11] Till V (1997) Keg tracking – a method to control a keg fleet; experiences and advantages. Proceedings of the European Brewing Convention, pp 737–746

Millimetre Wave Radar Visualisation System: Practical Approach to Transforming Mining Operations

E. Widzyk-Capehart1, G. Brooker2, S. Scheding2, A. Maclean2, R. Hennessy2, C. Lobsey2 and M. Sivadorai1 1

CSIRO, Brisbane, Australia (formerly CRCMining, The University of Queensland, Brisbane, Australia) 2 CRCMining and Australian Centre for Field Robotics, University of Sydney, Sydney, Australia.

1

Introduction

Over the last three decades, mining operations have undergone massive transformation to mechanized, semi- and automated mining systems; manual labor has been gradually replaced by machine operations and processes have become more integrated. This change was possible due to technological advances in sensing techniques, improved excavation methods, bigger and more reliable mining machines and better understanding of geological conditions. Yet, with all the technological advances, majority of mining operations still rely on human “operator” to achieve production goal, whose performance, in turn, is influenced by the accuracy of information provided by various data gathering systems and by the variable, sometimes unmanageable or unpredictable environmental conditions. In order to achieve and maintained high level of performance of man-machine systems, the information acquired using various technologies must be accurate and must be provided in time for uninterrupted operation. One such technology, which promises to revolutionalise the way mining is conducted and bring it even closer to automation, is the millimetre wave radar visualisation system which, by mapping the working environment of mining equipment, acquiring production data and supplying information to operator, is well positioned to improve safety, increase productivity and reduce operational and maintenance costs. Depending on the application, the radar visualization systems, developed by the CRCMining in conjunction with the Australian Centre for Field Robotics at the University of Sydney, are currently at various stages of development, from proto-

140

E. Widzyk-Capehart et al.

type to fully functional commercial units. These systems are continually being improved through ongoing research. The implementation and performance verification of the radar sensor for range and 3D profiling were undertaken in underground and surface mines. Cavity, stope fill and orepass monitoring as well as dragline environmental mapping and rope shovel bucket and dig face imaging were the main areas of radar testing. Excellent performance results show the mm wave radar unaffected by dust, vapor, high temperature, vibration or corrosive environment encountered in mining operations. The benefits of the radar technology are associated with an increased productivity, decreased maintenance costs, reduction in machine wear and tear and improved safety.

1.1

Application

Underground applications of the mm wave radar system include range measurements in ore-passes and imaging of the internal structures of large mined out cavities. In open cut mines, the uses of radar extend from 3D surface mapping for volume reconciliation and slope stability monitoring to measurements of machine position, bucket fill and bucket position. Most underground mines operate by moving rock from the higher levels, where the mining takes place, through near vertical tunnels called ore-passes or silos to crusher stations at the lowest levels. This is illustrated schematically in Fig. 1a, which shows rock being dumped into the pass through a coarse lattice called a grizzly that restricts the maximum size to less than one meter across. In some mines, the rock travels through an inclined finger pass that feeds into the side of the main pass (Fig. 1a), in others, it goes through a grizzly that covers the top of the main pass. Pass diameters vary between 3 m and 6 m while silo and stope diameters can be up to ten times this size. An accurate measurement of the range to the ore in a pass allows for checking “hung” passes, monitoring the volume of ore store, maintaining a rock buffer above the loading station by the operator and automating some of the ore flow process. The “hung” pass occurs when the rock plugs the pass, creating an ever widening void below the hung rock as broken material is drawn from the bottom of the pass. If this condition is not detected in good time, thousands of tonnes of rock can fall onto the loading station, when the plug releases, with potentially catastrophic consequences. For example, a kinetic energy of a single 2 t boulder after falling 100 m is 2 MJ. Considering that 1 m of rock in a 4 m diameter pass has a mass of 30 t, the resulting kinetic energy would be 30 MJ when the rock is dislodged 100 m above the draw point. This could have disastrous consequences with damage infrastructure and production stoppages. Many underground mines, where massive deposits of ore are found, produce large underground voids called stopes. Generally, geotechnical constraints limit the stope’s size in plan to about 40×40 m but vertically the limitation is determined by the orebody outline, which could reach up to 300 m in extent. The void created by the mining operation must be filled with a competent material capable

Millimetre Wave Radar Visualisation System

141

of supporting the walls and the roof while the adjacent stope is being mined. The materials of choice are Cemented Aggregate Fill CAF (a weak concrete) and/or rock, depending on the duty required. The cost difference between CAF and rock fill makes it desirable to maximize the amount of rock-fill without compromising the strength of the filled stope. Since the cavities are often filled with dust and water vapor, an ideal tool to monitor filling process and fill levels is a “real-time” remote device that can see through the mist and vapor, as shown in Fig. 1b. Surface coal mining operations rely heavily on the performance of mining equipment, especially, draglines and rope shovels. The primary function of draglines is to remove the overburden and uncover the coal seam, which can then be excavated by another dragline, shovel-truck system or front-end loader and transported to crushers for further processing. The excavation process is complicated by the requirement that these massive machines, some weighing in excess of 3,500 tonnes, retain access to and from the working area. The environments around draglines and rope shovels are often dusty or, in some cases, shrouded in mist, smoke or even steam, to the extent that the operator is unable to see the working area. This makes digging impossible for significant periods of time, which is extremely expensive for a machine that costs up to $500,000 per day to operate. Once again, the ideal tool for viewing the dig and fill process is a “real-time” remote device that can see through the opaque environment and can present the operator with an image of the terrain surface or the bucket contents, as illustrated in Fig. 1c and 1d. These visual feedback sensors can also be used for the partial and, ultimately, for the complete automation of the mining process.

2

Sensor Requirements

2.1

Signal Dynamic Range

All sensors described in Fig. 1 must be capable of measuring the range to the rock or backfill through an extremely dusty volume to accuracy better than 1% of the maximum range. Some stopes and passes are longer than 300 m and the range requirements for open-cut visualisation easily exceed this distance. Therefore, for a minimum range requirement of 3 m, the ratio of maximum to minimum range will be at least 100. To determine the sensor’s relative received power, a simplified version of the radar range equation can be applied,

⎛ λ2 ⎞ ⎟ + σ dB − LdB − 40 log10 R S dB = PtdB + 2GdB + 10 log10 ⎜⎜ 3 ⎟ ⎝ (4π ) ⎠

(1)

where: SdB – Received power (dBW), PtdB – Transmitted power (dBW), GdB – Antenna Gain (dB), λ – carrier wavelength (m), σdB – Target radar cross section (dBm2), LdB – Losses (dB), R – Range (m).

142


Most of these parameters are determined by the radar system design with the exception of the target radar cross section, σdB, which is a function of the terrain reflectivity, σ°, and the beam footprint. The reflectivity of a typical distributed target (rock surface) can be estimated using rough surface scattering models (Beckmann and Spizzichino 1987) or the following well known relationship (Nelson 2001)

σ = o

cosθ − ε rock − sin 2 θ cosθ + ε rock − sin 2 θ

(2)

where θ is the angle of incidence and the complex permittivity of the rock is εrock. For a rough surface, where the average incidence angle θ = 0° the equation reduces to

σo =

1 − ε rock 1 + ε rock

= 0.34

(3)

where εrock = 4.21-j0.156 for coal at X-Band (Nelson 2001). From measurements, it has been found that the target structure, which can vary from a huge boulder to piles of blasted rock and paste, not the rock permittivity is dominant in determining the “effective” reflectivity. The measure shown in (3) is, therefore, not accurate in isolation and the radar cross-section, σdB, is usually determined experimentally. Figure 2 shows the results of more than 200 measurements made over a 24 hour period in a working pass using a pulsed 94 GHz radar in which variations of 30 dB to 40 dB in target reflectivity can be seen (Brooker et al. 2005) for dry rock with a reasonably constant permittivity.


143

Fig. 1. Radar applications for (a) Ore-pass showing the rock fill (b) Stope showing the CAF and crushed rock fill, (c) Dragline monitoring of the dig area and the bucket position and (d) Rope shovel monitoring of the bucket fill and tooth integrity

144


Fig. 2. Measured 94 GHz reflectivity histogram made in a vertical ore-pass showing the large dynamic range that can be expected in signal level

As theradar cross section is the product of the target reflectivity and the area of the beam footprint, Afoot, for a symmetrical antenna with a half power beamwidth, θ, at range, R, it is

σ dB = 10 log10 (σ o Afoot ) = 10 log10 σ o

π 4

(Rθ )2 = σ dBo + 10 log10 ⎜⎜ πθ

⎞ ⎟⎟ + 20 log10 R ⎝ 4 ⎠ ⎛

2

(4)

Substituting into (1) results in

⎛ λ2 πθ 2 ⎞ o ⎟⎟ − LdB + σ dB S dB = PtdB + 2GdB + 10 log10 ⎜⎜ . − 20 log10 R 3 ⎝ (4π ) 4 ⎠

(5)

It is clear from (5) that a combination of 40 dB of variation in the reflectivity and a range ratio of 100 to 1 (40 dB) results in a signal dynamic range of 80 dB. To process this received power, the signal is generally digitised and hence an analog to digital converter (ADC) with a dynamic range of at least 80 dB is required. As most common ADCs with the required throughput are 12 bit devices, with a dynamic range of less than 72 dB, some form of gain control prior to conversion is required.


2.2

145

Attenuation Through Clouds of Dust and Water Droplets

Atmospheric attenuation through clear air is determined by frequency dependent molecular interactions with the electromagnetic radiation. In Fig. 3 (Preissner 1978), it can be seen that the attenuation increases with increasing frequency in the microwave and millimetre wave bands before dropping off sharply into the IR and visible bands. In the millimetre wave band, windows of relatively low attenuation occur at 35 and 94 GHz between oxygen absorption lines. It is within these windows that most radar activity occurs, with the frequency around 77 GHz earmarked for automotive sensors and that around 94 GHz reserved for defence and experimental applications. For the relatively short range applications considered here, atmospheric attenuation is not significant; however, signal attenuation is affected by the particulates suspended in the atmosphere. Although no quantitative studies of millimetre wave signal propagation through dust on mines have been undertaken, some work has been done on propagation through dust storms (Gillett 1979), which showed that the visibility can be related to the mass of dust per cubic meter of air (Table 1). Since the wavelengths of most radar signals, including the millimetre wave band, is much larger than the diameter of the dust particles, Rayleigh scattering is used to determine signal attenuation as a function of the visibility (Goldhirsh 2001). This relationship (Fig. 4) can be used to determine the performance of the radar in dusty or misty conditions (Brooker 2005).

Fig. 3. Atmospheric attenuation of electromagnetic radiation in clear air as a function of frequency

146


Table 1. Relationship between Visibility and Mass Loading Visibility (m) 0.5 1 2 4 8 12.75

Mass Loading (g/m3) 78 37 17.8 8.5 4.0 2.44*

*Average for sand storms

Fig. 4. Relationship between visibility and attenuation for coaldust and water droplets at 94 GHz

In contrast to the poor transmission at visible and IR wavelengths, attenuation at a wavelength of 3.2 mm (94 GHz) is practically negligible for short range operation through the dust, though the losses do become significant in a water vapor environment if the visibility is very poor. For a visibility of 4 m (extremely thick fog), which corresponds to a mass loading of 8.5 g/m3, an attenuation of about 12 dB/km can be expected.

2.3

Backscatter from Clouds of Dust and Water Vapour

The effectiveness of a laser or radar system is dependent not only on the actual signal level returned from the target of interest, but also the relative level of this signal in comparison to other competing returns at the same range. These compet-


147

Fig. 5. Backscatter from coal dust and water with identical particle size distributions as a function of the visibility at 94 GHz

ing signals are generally referred to as clutter. The most common sources of clutter are returns from the dust or water droplets within the radar beam or from large (high reflectivity) returns that enter through the sidelobes of the antenna. Figure 5 shows the reflectivity plotted for coal dust and water spray with identical particle/droplet size distributions. It can be seen that the magnitude of the backscatter at 94 GHz is very small even at extremely high dust or water droplet levels.

2.4

Other Considerations

In some applications, spurious targets must be discriminated against as partial blockages may occur in ore passes and falling rocks and/or falling water in both passes and cavities may be present. The environmental effects of high ambient temperature, shock, vibration and dramatic changes in pressure as a result of the blast concussion, must be taken into considerations during sensor development while dust and water ingress as well as potential rock falls are given special consideration when designing protective housing. The configuration setup for the system should be kept to a minimum. Systems should be light, versatile and easy to install and align as they are often mounted in inaccessible and dangerous areas. The sensors extreme reliability (MTBF>1 year

148


of continuous operation) and long term accuracy are necessary requirements to reduce/eliminate large component of system maintenance. Multi sourcing of components with robust electrical design allow for component tolerances of up to 20% and provide for module replacement without recalibration. Simple and quick assembly as well as fast testing and calibration of modules and of the complete units should be ensured. From safety perspective, any radiation produced at the operational frequency should be well within statutory limits. This is a tall order given the environmental extremes encountered in mining operation; blast concussion, machine vibration and the likelihood of rock falls.

3

Selection of Technology

For many years, laser, acoustic, visual and microwave radar systems have been the workhorse technologies for industrial level measurement and imaging. However, since the availability of moderately low-cost K and millimetre wave band components, these higher frequency options have become more popular (Kielb et al. 1999). Using the results presented here and data from numerous studies on electromagnetic (Bhartia and Bahl 1984; Comparetto 1993; Perry and Baden 2000; Goldhirsh 2001) and acoustic (Goodsit 1982; Kue 1984) propagation through dust, criteria can be presented that can be used to select the most cost-effective candidate technology for a particular application. The pros and cons of the four sensor types are summarized in Table 2 while visual system issues are addressed in the next paragraph. It should be noted that, in addition to properties listed in Table 2, the performance of acoustic sensors are affected by extraneous noise and air currents while laser technology, though robust and well established, exhibits poor sensitivity to low reflectivity materials, such as, coal. As with laser system, CCTV systems are affected by environmental conditions; dust, rain as well as variable illumination (due to changing weather conditions), which makes the visualisation and object recognition (rocks) difficult to perform. CCTV would require special illumination for night vision and multiple units for stereovision for real-time data processing and operator feedback. Target recognition using video cameras is a very difficult problem, compounded with the need to reliably estimate small differences in shape, when dealing with face/bucket imaging. Shape recognition is a largely unexplored field in this context. In their current application in bucket tooth detection systems (Motion Metrics, 2006), the image acquisition process is supported by high power illumination assembly and the imaged objects are large and distinctive. Even then there is minimal or no data available on their performance to substantiate their application in other areas of object recognition. In many above ground installations, the size requirement is not an issue and a number of quite large microwave and millimetre wave systems have been constructed for imaging and slope stability applications (Reeves et al. 2000; Macfarlane et al. 2004). Thus, for dragline and shovel based imaging systems, both millimetre wave and microwave frequencies can be used. However, the


149

smaller aperture requirements, and hence lower overall size and weight, make the millimetre wave option more practical, albeit, at present, more expensive. In the ore-pass application, reliable and repeatable measurements can only be made if the beam is sufficiently narrow that it does not illuminate the walls even at the longest operational range (Brooker et al. 2005). Similarly, to produce accurate measurements in a cavity requires an extremely narrow beam. This is only available from either a large aperture or high frequency operation. As large aperture devices are generally cumbersome and heavy, the only alternative is to use the highest possible frequency. Ultimately, millimetre wave radar is the only viable candidate for these applications. Table 2. Comparison of sensors for mining application Sensor

Laser

Acoustic

Microwave radar

Millimetre wave radar

Beam width

Narrow

Wide

Wide

Narrow

Beam function in enclosed regions

Good

Too wide

Too wide

Good

Ease of scanning with mirrors

High

Impossible

Low

High

Dust/water vapor penetration

Poor

Poor

Good

Good

Dust operational effect on antenna/transducer

High

Moderate

Very low

Very low

Cost

Low

Low

Low

High

Property

4

Radar Operational Technique and Specifications

A number of different millimetre wave radar techniques have been applied to measure range in industrial processes (Brooker et al. 2005). Of those, the Frequency Modulated Continuous Wave (FMCW) was proved to be conceptually simple, most robust and a lowest cost technique despite chirp linearity and sensitivity with reflected power issues, which have to be addressed during operation. The FMCW technique has only recently been adapted for longer range applications by increasing the operational frequency to the 24 GHz ISM band (Zimmermann et al. 1996). Most FMCW radars operate by transmitting a linear frequency chirp of long duration. At any time, the received echo is shifted in frequency from the transmitted signal by the product of the roundtrip time to the target, and the rate of change of frequency. If the received signal is mixed with a portion of the transmitted signal and filtered, the resulting output will be a constant beat frequency. Two

150


factors limit the range resolution of these systems: the first is a function of the chirp bandwidth and the second is the actual linearity that is achieved for the transmitted chirp. These limitations notwithstanding, it has been confirmed that transmitted powers of only 10 mW are sufficient for all of the application discussed here. The basic structure of the radar systems and operational principle are shown in Fig. 6, a and b, respectively. The specifications of the radar systems discussed here are summarized in Table 3, with the Stope and Dragline radars having similar properties.

Fig. 6. FMCW radar (a) schematic diagram and (b) operational principles


151

Table 3. Radar specifications Radar type Ore-pass Properties Transmit Power 10 mW Centre Frequency 94 GHz

Bucket-Fill

Stope Fill & Dragline

10 mW

10 mW

94 GHz

77 GHz 600 MHz

Sweep Bandwidth

250 MHz

1 GHz

Sweep Linearity

Mechatronics and Machine Vision in Practice

Machine vision

Machine vision

Understanding and Applying Machine Vision

Handbook of machine vision

Handbook of Machine Vision

Mechatronics

Machine Vision: Theory, Algorithms, Practicalities

Mechatronics in Action: Case Studies in Mechatronics - Applications and Education

Machine vision: theory, algorithms, practicalities

Mechatronics

Computer and Machine Vision. Theory, Algorithms, Practicalities

Machine Vision: Theory, Algorithms, Practicalities

Machine vision: Theory, algorithms, practicalities

Machine Vision Beyond Visible Spectrum

ISE Image Processing, Analysis and Machine Vision

Bézier and Splines in Image Processing and Machine Vision

Recent Advances in Mechatronics

Recent Advances in Mechatronics

Mechatronics: Principles and Applications

Mechatronics: Principles and Applications

Recent Trends in Mechatronics

Mechatronics '98

Understanding and Applying Machine Vision (Manufacturing Engineering and Materials Processing)

Mechatronics in Engineering Design and Product Development

Computer Vision for Human-Machine Interaction

Intelligent Mechatronics

Template matching techniques in computer vision: theory and practice

Mechatronics '98

Channel-coded feature maps for computer vision and machine learning

Image Processing, Analysis, and Machine Vision, 3rd Edition

Mechatronics and Machine Vision in Practice

Machine vision

Machine vision

Understanding and Applying Machine Vision

Handbook of machine vision

Handbook of Machine Vision

Mechatronics

Machine Vision: Theory, Algorithms, Practicalities

Mechatronics in Action: Case Studies in Mechatronics - Applications and Education

Machine vision: theory, algorithms, practicalities

Mechatronics

Computer and Machine Vision. Theory, Algorithms, Practicalities

Machine Vision: Theory, Algorithms, Practicalities

Machine vision: Theory, algorithms, practicalities

Machine Vision Beyond Visible Spectrum

ISE Image Processing, Analysis and Machine Vision

Bézier and Splines in Image Processing and Machine Vision

Recent Advances in Mechatronics

Recent Advances in Mechatronics

Mechatronics: Principles and Applications

Mechatronics: Principles and Applications

Recent Trends in Mechatronics

Mechatronics '98

Understanding and Applying Machine Vision (Manufacturing Engineering and Materials Processing)

Mechatronics in Engineering Design and Product Development

Computer Vision for Human-Machine Interaction

Intelligent Mechatronics

Template matching techniques in computer vision: theory and practice

Mechatronics '98

Channel-coded feature maps for computer vision and machine learning

Image Processing, Analysis, and Machine Vision, 3rd Edition

Recommend Documents