TIME-VARYING IMAGE PROCESSING AND MOVING OBJECT RECOGNITION, 4
This Page Intentionally Left Blank
TIME-VARYING IMAGE PROCESSING AND MOVING OBJECT RECOGNITION, 4 Proceedings of the 5th InternationalWorkshop Florence, Italy, September 5-6, 1 996 Edited by V. CAPPELLINI Department of Electronic Engineering University of Florence Florence, Italy
1997 ELSEVIER Amsterdam
- Lausanne - New York- Oxford - Shannon - Tokyo
ELSEVIER SCIENCE B.V. Sara Burgerhartstraat 25 P.O. Box 211,1000 AE Amsterdam, The Netherlands
ISBN: 0 444 82307 7 91997 Elsevier Science B.V. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of the publisher, Elsevier Science B.V., Copyright & Permissions Department, P.O. Box, 521,1000 AM Amsterdam, The Netherlands. Special regulations for readers in the U.S.A. - This publication has been registered with the Copyright Clearance Center Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923. Information can be obtained from the CCC about conditions under which photocopies of parts of this publication may be made in the U.S.A. All other copyright questions, including photocopying outside of the U.S.A., should be referred to the copyright owner, Elsevier Science B.V., unless otherwise specified. No responsibility is assumed by the publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. pp. 69-76, 184-189, 190-196: Copyright not transferred. This book is printed on acid-free paper. Printed in The Netherlands.
PREFACE The area of Digital Image Processing is of high actual importance in terms of research and applications. Through the interaction and cooperation with the near areas of Pattern Recognition and Artificial Intelligence, the specific area of "Time-Varying Image Processing and Moving Object Recognition" has become of increasing interest. This new area is indeed contributing to impressive advances in several fields, such as communications, radar-sonar systems, remote sensing, biomedicine, moving vehicle tracking-recognition, traffic monitoring and control, automatic inspection and robotics. This book represents the Proceedings of the Fifth International Workshop on Time-Varying Image Processing and Moving Object Recognition, held in Florence, September 5-6, 1996. Extended papers reported here provide an authorative and permanent record of the scientific and technical lectures, presented by selected speakers from 10 nations. Some papers are more theoretical or of review nature, while others contain new implementations and applications. They are conveniently grouped into the following fields: m. B.
C. D. E. F. G. H. I.
Digital Processing Methods and Techniques Pattern Recognition Computer Vision Image Coding and Transmission Remote Sensing Data and Image Processing Digital Processing of Biomedical Images Motion Estimation Tracking and Recognition of Moving Objects Application to Cultural Heritage.
New digital image processing and recognition methods, implementation techniques and advanced applications (television, remote sensing, biomedicine, traffic, inspection, robotics, etc.) are presented. New approaches (i.e. digital filters, source coding, neural networks .... ) for solving 2-D and 3-D problems are described. Many papers are concentrated on the motion estimation and tracking-recognition of moving objects. The increasingly important field of Cultural Heritage is also covered. Overall the book presents - for the above outlined area - the state of the art (theory, implementation, applications) with the next-future trends. This work will be of interest not only to researchers, professors and students in university departments of engineering, communications, computers and automatic control, but also to engineers and managers of industries concerned with computer vision, manufacturing, automation, robotics and quality control. V. Cappellini
vi
WORKSHOP CHAIRMAN
V. CAPPELLINI, University of Florence, Florence, Italy
STEERING COMMITTEE
J.K. A GGARWAL, University of Texas, Austin, U.S.A. M. BELLANGER, Conservatoire National des Arts et Mgtiers, Paris, France J. BIEMOND, University of Delft, The Netherlands M. BRA CALE, University of Naples, Italy A.G. CONSTANTINIDES, Imperial College, London, England T.S. DURRANI, University of Strathclyde, Glasgow, Scotland G. GALATI, H University of Rome, Italy G.H. GRANLUND, University of Link6ping, Sweden T.S. HUANG, University of Illinois at Urbana-Champaign, U.S.A. G. IMMOVILLI, University of Modena, Italy M. KUNT, Ecole Polytechnique Fe'd~rale de Lausanne, Switzerland A.R. MEO, Polytechnic of Turin, Italy S.K. MITRA, University of California, Santa Barbara, U.S.A. F. ROCCA, Polytechnic of Milan, Italy A. ROVERI, University of Rome "La Sapienza", Italy G. L. SICURANZA, University of Trieste, Italy A.N. VENETSANOPOULOS, University of Toronto, Canada G. VERNAZZA, University of Cagliari, Italy
o~
VII
Sponsored by: European Association for Signal Processing (EURASIP) IEEE Central & South Italy Section European Association of Remote Sensing Laboratories (EARSeL) International Center for Signal & Image Processing (ICESP), Florence Centro d'Eccellenza Optronica (CEO) Dipartimento di Ingegneria Elettronica, University of Florence Istituto di Ricerca sulle Onde Elettromagnetiche (IROE) "Nello Carrara" - C.N.R., Florence Fondazione Ugo Bordoni Fondazione IBM ITALIA Fondazione per la Meteorologia Applicata Associazione Italiana di Telerilevamento (AIT) Gruppo Nazionale Telecomunicazioni e Teoria dell 'lnformazione (T. T.I.) - C.N.R. Sezione di Firenze dell'A.E.I. Associazione Italiana di Ingegneria Medica e Biologica (A.I.LM.B.) CESVIT- Agenzia per l' Alta Tecnologia Regione Toscana - Giunta Regionale
Co sponsored by: Alenia Spazio Alinari AXIS Esaote Biomedica Nuova Telespazio OTE SAGO S.M.A - S istemi per la Meteorologia e l'Ambiente Syremont Telecom Italia Telesoft Ente Cassa di Rispaxmio di Pistoia e Pescia
This Page Intentionally Left Blank
ix
CONTENTS
AO
DIGITAL PROCESSING METHODS AND TECHNIQUES
A.1
"On 3-D Space-time Interpolation and Data Compression of Digital Image Sequences Using Low-order 3-D IIR Filters" H.-L.M. CHENG and L.T. BRUTON
A.2
"Flicker Reduction in Old Film Sequences" (Invited) P.M.B. VAN ROOSMALEN, R.L. LAGENDIJK and J. BIEMOND
A.3
"Multichannel Filters in Television Image Processing"
19
K.N. PLATANIOTIS, S. VINAYAGAMOORTHY, D. ANDROUTSOS and A.N. VENETSANOPOULOS
BO
PATTERN RECOGNITION
25
B.1
"Blotch and Scratch Detection in Image Sequences based on Rank Ordered Differences" (Invited)
27
M.J. NADENAU and S.K. MITRA B.2
"Feature Matching by Optimization using Environmental Constraints"
36
A. BRANCA, E. STELLA, G. ATTOLICO and A. DISTANTE B.3
"@stem Identification for Fuzzy Controllers"
42
G. CASTELLANO, G. ATTOLICO, T. D'ORAZIO, E. STELLA and A. DISTANTE
Co
COMPUTER VISION
49
C.1
"Computer Vision for Autonomous Navigation: from Research to Applications"
51
G. GARIBOTTO, P. BASSINO, M. ILIC and S. MASCIANGELO C.2
"An Optimal Estimator of Camera Motion by a Non-Stationary Image Model" G. GIUNTA and U. MASCIA
57
C.3
"A Simple Cue-Based Camera Calibration Method for Digital Production of Moving Images"
63
Y. NAKAZAWA, T. KOMATSU and T. SAITO C.4
"Exploration of the Environment with Optical Sensors Mounted on a Mobile Robot"
69
P. WECKESSER, A. VON ESSEN, G. APPENZELLER and R. DILLMANN
DO
IMAGE CODING AND TRANSMISSION
77
D.1
"Time-Varying Image Processing for 3D Model-Based Video Coding" (Invited)
79
T.S. HUANG, R. LOPEZ and A. COLMENAREZ D.2
"A New Arbitrary Shape DCT for Object-Based Image Coding"
87
M. TANIMOTO and M. SATO D.3
"Picture Coding Using Splines"
93
M. BUSCEMI, R. FENU, D.D. GIUSTO and G. LIGGI D.4
"A 10 kb/s Video Coding Technique Based on Spatial Transformation"
99
S. BONIFACIO, S. MARSI and G.L. SICURANZA D.5
"Image Communications Projects in ACTS" (Invited)
105
F. BIGI D.6
"Conveying Multimedia Services within the MPEG-2 Transport Stream"
115
L. AZTORI, M. DI GREGORIO and D.D. GIUSTO D.7
"A Subband Video Transmission Coding System for ATM Network"
121
M. EYVAZKHANI D.8
"A High Efficiency Coding Method"
127
K. KAMIKURA, H. JOZAWA, H. WATANABE, H. KOTERA and K. SHIMAMURA D.9
"A Sequence Analysis System for Video Databases"
133
M. CECCARELLI, A. HANJALIC and R.L. LAGENDIJK D.10
"Subjective Image Quality Estimation in Subband Coding: Methodology and Human Visual System Application" Z. BOJKOVIC, A. SAMCOVIC and B. RELJIN
139
xi
EO
REMOTE SENSING DATA AND IMAGE PROCESSING
145
E.1
"Neural Networks for Multi-Temporal and Multi-Sensor Data Fusion in Land Cover Classification"
147
A. CHIUDERI E.2
"Influence of Quantization Errors on SST Computation Based on A VHRR Images"
153
P.F. PELLEGRINI, F. LEONCINO, E. PIAZZA and M. DI VAIA E.3
"Study of Ecological Condition Based upon the Remote Sensing Data and GIS"
159
M. ZHANG, J. BOGAERT and I. IMPENS E.4
"PEICRE PROJECT: a Practical Application of Remote Sensing Techniques for Environmental Recover and Preservation"
165
M. BENVENUTI, C. CONESE, C. DI CHIARA and A. DI VECCHIA E.5
"A Wavelet Classification Chain for Rain Pattern Tracking from Meteorological Radar Data"
171
P. GAMBA, A. MARAZZI and A. MECOCCI E.6
"Frequency Locked Loop System for Doppler Centroid Tracking and Automatized Raw Data Correction in Spotlight Real-Time SAR Processors"
176
F. IMPAGNATIELLO and A. TORRE E.7
"Use of Clutter Maps in the High Resolution Radar Surveillance of Airport Surface Movements"
184
G. GALATI, M. FERRI and M. NALDI E.8
"Simulation of Sequences of Radar Images for Airport Surveillance Applications"
190
F. MARTI, M. NALDI and E. PIAZZA E.9
"Data Fusion and Non Linear Processing of E.L.F. Signal for the Detection of Tethered Satellite System" 197 S. MONTEVERDE, R. RUGGERONE, D. TRAVERSO, S. DELLEPIANE and G. TACCONI
Fo F.1
DIGITAL PROCESSING OF BIOMEDICAL IMAGES
203
"A Simple Algorithm for Automatic Alignment of Ocular Fundus Images"
205
L. B ALLERINI, G. COPPINI, G. GIACOMELLI and G. VALLI
xii F.2
"Automatic Vertebrae Recognition throughout a Videofluoroscopic Sequence for Intervertebral Kinematics Study"
213
P. BIFULCO, M. CESARELLI, R. ALLEN, J. MUGGLETON and M. BRACALE F.3
"An Evaluation of the Auditory Cortex Response to Simple Non-Speech Stimuli through Functional MRI"
219
A. PEPINO, E. FORMISANO, F. DI SALLE, C. SAULINO and M. BRACALE
GO
MOTION ESTIMATION
225
G.1
"Temporal Prediction of Video Sequences Using a Region-Based Image Warping Technique" (Invited)
227
N. HERODOTOU and A.N. VENETSANOPOULOS G.2
"High Performance Gesture Recognition Using Probabilistic Neural Networks and Hidden Markov Models"
233
G. RIGOLL, A. KOSMALA and M. SCHUSTER G.3
"Image Segmentation Using Motion Estimation"
238
K. ILLGNER and F. MOLLER G.4
"A Phase Correlation Technique for Estimating Planar Rotations"
244
L. LUCCHESE, G.M. CORTELAZZO and M. RIZZATO G.5
"Tracking by Cooccurrence Matrix"
250
L. FAVALLI, P. GAMBA, A. MARAZZI and A. MECOCCI G.6
"Robust Pose Estimation by Marker Identification in Image Sequences"
256
L. ALPARONE, S. BARONTI, A. BARZANTI, A. CASINI, A. DEL BIMBO and F. LOTTI G.7
"Markov Random Field Image Motion Estimation Using Mean Field Theory"
262
A. CHIMIENTI, R. PICCO and M. VIVALDA G.8
"Moving Object Detection in Image Sequences Using Texture Features"
268
F. MOLLER, M. HOTTER and R. MESTER G.9
"Determining Velocity Vector Fields from Sequential Images Representing a Salt-Water Oscillator" A. NOMURA and H. MIIKE
274
xiii
HO
TRACKING AND RECOGNITION OF MOVING OBJECTS
281
H.1
"'Long-Memory' Matching of Interacting Complex Objects from Real Image Sequences"
283
A. TESEI, A. TESCHIONI, C.S. REGAZZONI and G. VERNAZZA H.2
"Spatial and Temporal Grouping for Obstacle Detection in a Sequence of Road Images"
289
S. DENASI and G. QUAGLIA H.3
"Attitude of a Vehicle Moving on a Structured Road"
295
A. GUIDUCCI and G. QUAGLIA H.4
"An Algorithm for Tracking Pedestrians at Road Crossing"
301
M. LORIA and A. MACHI
1.1
APPLICATION TO CULTURAL HERITAGE
307
"Cultural Heritage: The Example of the Consortium Alinari 2000-SOM"
309
A. DE POLO, E. SESTI and R. FERRARI 1.2
"Color Certification"
313
A. ABRARDO, V. CAPPELLINI, A. MECOCCI and A. PROSPERI 1.3
"Image Retrieval by Contents with Deformable User-Drawn Templates"
319
A. DEL BIMBO and P. PALA 1.4
"Synthesis of Virtual Views of Non-Lambertian Surface through Shading-Driven Interpolation and Stereo-Matched Contours"
325
F. PEDERSINI, A. SARTI and S. TUBARO
AUTHOR INDEX
331
This Page Intentionally Left Blank
A DIGITAL PROCESSING METHODS AND TECHNIQUES
This Page Intentionally Left Blank
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.)
9 1997 Elsevier Science B.V. All fights reserved.
On 3-D Space-time Interpolation and Data Compression of Digital Image Sequences Using Low-order 3-D IIR Filters H.-L. Margaret Cheng and Leonard T. Bruton Department of Electrical and Computer Engineering, The University of Calgary, Calgary, Alberta, CANADA A bstract--A method is proposed for the data compression and spatio-temporal interpolation of temporally sub-sampled digital image sequences using a first-order 3-D Linear Trajectory (LT) IIR filter.
1. I N T R O D U C T I O N Data compression of image sequences can be achieved by spatio-temporal sub-sampling. In this contribution, we propose a method for recovering a sequence of digital images from the temporally sub-sampled version using a low-order spatio-temporal 3-D IIR (infinite impulse response) filter to perform the required spatio-temporal interpolation. A firstorder 3-D Linear Trajectory (LT) IIR filter [1] is employed for this purpose, followed by a smoothing operation performed in the direction of the motion vector. Experimental results suggest that high compression ratios may be possible. We assume for simplicity that, in each spatio-temporal sub-image sequence, the 3-D spatio-temporal signal contains only one object moving with a constant velocity. This assumption is valid for many practical situations and is the underlying assumption of MPEG-2 and other compression methods.
2. R E V I E W
O F S P A T I O - T E M P O R A L S U B - S A M P L I N G OF I M A G E
SEQUENCES A 3-D LT signal pc(x, y, t), (x, y, t) E ~3 is defined as a continuous-domain space-time signal having a value that is everywhere constant in the direction of the motion vector ~ = (v~'x + vyey-4-vte.t), where ~x, C-u,C~tare the unit basis vectors in the spatial and temporal directions, respectively. The region of support (ROS) of the 3-D Fourier transform of a LT signal is the plane passing through the origin and perpendicular to ~, i.e. w~v~ + wuv u + cotvt 0. The 2-D spectrum on this plane represents the spatial frequency components of the intersection of the 3-D signal with the plane perpendicular to ~ [1]. We assume that this continuous-domain LT signal pc(z, y, t) is 3-D rectangularly sampled at a sufficiently high 3-D sampling frequency that aliasing is negligible. However, temporal sub-sampling of pc(x, y, t) by M introduces aliased replicated 3-D frequency planes (referred to as replica hereafter), at locations w~v~ + wyvy + wtvt = -t-2rrvtj/M, j C [1,..., M-l]. These replica must be completely eliminated by an ideal interpolator. To achieve close-to-ideal interpolation, we employ motion-compensated (MC) interpolation (lower part of Figure 1), where the orientation of the interpolator's passband =
,~, ,~/~ [:?i!~i~!~
(a)
co x
-
(b)
':::~....
:.........
li :: :': :: : :l~i i: :: :IV
(c)
Figure 1: Spectral representation of temporal (upper) and motion-compensated (lower) interpolation, shown for the 2-D case. Dashed lines show aliased replicated signal planes under temporal sub-sampling by M=2. Shaded regions represent passbands of interpolators. The problem of aliasing is shown in (a), and its solution (i.e. pre-filtering) is shown in (b). Interpolation of properly pre-filtered signals is shown in (c). Adapted from [2]. is adapted to that of the spectrum of the sub-sampled signal. In Figure 1 we review the advantage of using this method by comparing it with temporal (upper part of Figure 1) interpolation [2]. For ease of illustration, a 2-D signal that has been temporally sub-sampled by M = 2 is used. Its spectrum is shown in Figure l(a), where the solid line represents the original spectrum of the signal prior to sub-sampling, the dashed lines represent replica introduced by sub-sampling, and the shaded regions represent the passbands of the interpolators. Clearly, the temporal interpolator transmits the undesirable replica and, therefore, fails. To avoid such aliasing in the case of the temporal interpolator, the high-frequency components may be eliminated by separably pre-filtering the signal prior to sub-sampling (Figure l(b)). This seriously attenuates the 3-D planar spectrum of the signal, causing spatio-temporal blurring. However, MC interpolation ideally eliminates the replica and, therefore, does not require pre-filtering. In Figure l(c) we show the two interpolators operating on appropriately pre-filtered and sub-sampled sequences. Aliasing is avoided in both cases. However, because M C interpolation is performed in the direction of the motion vector ~, it does not attenuate the 3-D planar spectrum resolution of the signal and is, therefore, much more effective than simple temporal (or spatial) interpolation. A D E S I G N T E C H N I Q U E TO O B T A I N T H E 3-D LT I I R DISCRETE-DOMAIN FILTER FOR MC INTERPOLATION
3.
To achieve motion-compensated interpolation, we wish to design a stable 3-D IIR discretedomain LT filter having a 3-D passband that is approximately planar, where this passband closely surrounds the planar ROS of the 3-D LT signal. The design process commences with a suitable continuous-domain 3-D frequencyplanar filter [1] having a 3-D Laplace transform transfer function of the form [1]
T(s:,sy, st) = R/[R + s:L: + syLy + stL,l
(1)
The passband of T(s:, sy, st) closely surrounds a 3-D plane [1] passing through the origin and having a normal fi = +(n:~: + Ly~y + Lt~t)/l[LII2. The parameters R, L:, Ly, Lt
Figure 2: Resonant plane of the first-order 3-D LT IIR filter determine the orientation of the passband, and the "thickness" of the passband is determined by its 3-D bandwidth B3 = 2R/llLll~ (Figure 2) [1]. The proposed 3-D discrete-domain interpolating filter is obtained from the above continuous-time prototype by applying the triple s-to-z domain transform [3],
si =
1+
2
ai z i - 1 --,
O et) /x ( e f > et)
(6)
otherwise
where I n (r~) is the pixel intensity at the location r~ in the n-th image frame and the motion c o m p e n s a t i o n vectors in forward and backward direction are Vn, n - 1 (r), Vn. n + 1 (r) . A n examined location is marked as corrupted, if the forward and backward squared frame difference (ef, eb ) are greater than a certain threshold el. 3.3. MRF
- Detector
The M R F detector is based on a Markov random field (MRF) model. This model is not applied to the image itself, but it is used to model the blotches of an image by creating a blotch detection frame D. This can be considered as an additional virtual frame between two real image frames of the sequence, containing only the blotch and not any real image information. For a possible configuration (D = d) of the complete detection frame D the presence of a
32 blotch at position ~r is indicated by d (r~) = 1 , while d (r~) = 0 represents an uncorrupted location. In [ 1] the following equation for an a posteriori joint distribution for the detection frame D is given P (D = d l I = i) = ~exp -
[a(1-d(r~))
(i(~r)-i(Nc)) 2
_
[31f(d(~r)) + J 3 2 6 ( 1 - d ( r ~ ) ) ]
where 0~, J3~ an J32 are certain parameters, i (~r) is the pixel intensity at the location r~ in the current frame, t(Nc) is the pixel intensity of the motion compensated other real image, the function f ( d (r~) ) gives the number of the four neighbors of d (r~) with the same value as d (~-), 6 ( ) is the delta function and S describes the possible area for r~ , namely the whole image frame. With Eq. (7) the probability for a certain configuration of D can be evaluated. The Gibbs sampler with annealing is used to find the maximum a posteriori (MAP) configuration of the detection frame D, given the data and the model for blotches. First, this technique is aimed to the current frame and preceding frame; next, it is applied to the current frame and succeeding frame. Only at those sites, where both times a discontinuity is estimated, the location is classified as corrupted. The search for the MAP is carried out in an iterative manner. After approximately 5 iterations the algorithm is assumed to have converged. 4. SIMULATIONS To compare the efficiency of the three detectors we use the same black and white image sequence, WESTERN, which has been used in [ 1]. The images have a size of 256 x 256 pixels and contain gray values in the range between 0 and 255. The sequence is artificially corrupted with blotches of random gray values, quite realistic in size and shape. This makes it possible to compute the false alarm and correct detection rate. First, we discuss the detector efficiencies by applying all three methods to the whole sequence of 64 image frames. We then show a typical frame to provide a visual demonstration of the detection algorithm. The motion in the image sequence is estimated from the degraded frames using a four-level estimation process, described above in Section 2. It is a different motion estimation process than used in [ 1], therefore the results for the MRF and SDIa detector might be slightly different. Figure 4 shows a plot of the correct detection rate versus false alarm rate for the ROD, MRF and SDIa detector, applied to the whole sequence. The probability of false alarm and correct detection are defined as:
nfa = Number of false detections
nfa Pfa = "~
nco P co =
nco + nmi
nmi
-
"
Number of missing detections
nco = Number of correct detections
(8)
N = Number of pixels per frame We used 1 < T 1 < 38, T2 = 39, T 3 = 55 as parameters for the ROD detector. For the MRF detector the best results out of the parameter range of 6 < e 1 < 34, 14 < e 2 < 54 have been chosen and the SDIa curve has been generated by measurements for 50 < e I < 2000 in steps of 25.
33
i::~
0.9
M.F .
~
0.
0.6 10 -4
.
.
.
.
.
1 0 -3 Probability
.
.
.
.
: - : .
.
.
.
1 0 -2
of false alarm
Figure 4: Performance of detectors applied to the whole sequence WESTERN
.
1 0 -1
.!
~0.7~ .
!i!i,! :~-r::
!
i
.
0.6l
i
.......
! . i~i i
.
10 -3
~ ! i!ii!i
./
: .... :
.
.
:
.
.
:
::::
..........
.
.
i
i ! i i!!
" ........................
ROD
.
:-
[ :
.
10 -2
Probability of false alarm
10
Figure 5: Performance of detectors applied to frame 49
Obviously the performance of the MRF detector and the SDIa detector are very similar. A slightly better result is obtained with the MRF detector, but the difference is really marginal. The ROD curve shows the fundamental improvement of the new detector versus the other approaches. For a correct detection rate of 80% the ROD detector has about 2.5 times less false detections, which provides a much more feasible basis for the restoration process to be carried out in the second step. The MRF detector, although a very complex algorithm, does not show a better performance versus the SDIa or the ROD approaches. Figure 5 provides a comparison of the detector performances as applied to frame 49 of the sequence W E S T E R N shown in Figure 6. This frame contains more marginally contrasted blotches on the average than all others. In this special case the MRF detector is able to take advantage of its better capability to detect poorly contrasted blotches by its spatial connectivity. In fact, the MRF detector performs perceptually better than the SDIa detector, but the new ROD approach provides a still superior performance. For a correct detection rate of 93%, the false alarm rates are - ROD 0.096% - MRF 0.81% - SDIa 0.95%. That means the performance of the ROD detector is 10 times better than the performance of the SDIa detector. Also the difference between the ROD and MRF detector is almost of one order. To visualize the difference of detection Figure 7 - 9 show the detector results. Green colored pixels indicate correct detections, red colored pixels mark missing detections and brown colored pixels represent false detections. The chosen bias point of the algorithms provide the same correct detection rate, that is, the number of green marked areas should be almost equal. To compare the detector performances, attention should be focused on the number of brown pixels. All three detectors produce false alarms in the area of the white coat lining, because it appears only in frame 49. In the preceding frame and succeeding frame it is covered. For a three frame based approach, this demonstrates the limit of an automatic detection process. In the same way all detectors miss the blotch on the right shoulder of the main person. This blotch is too marginally contrasted for all algorithms. While the ROD detector provides nearly perfect detection with the fewest number of false detections, the other two detectors result sometimes in small coherent areas of false detections.
34 A restoration process applied to these locations definitely causes a degradation of the fine details of the picture. From the implementation point of view, the computational complexity of these algorithms is of greater interest. We define the cost for an addition, subtraction, multiplication or division as 1, the cost for an exp-function as 20, according to the numbers used in [ 1]. The SDIa detector uses only 6 operations, while the MRF approach needs about 60 operations in forward and 60 operations in backward direction. For 5 iterations, this results in about 600 operations per pixel. That is a difference of two orders to the SDIa approach. The ROD detector requires only 24 operations and is thus a very easily implementable algorithm. 5. CONCLUSIONS In this paper we introduced an very efficient blotch and scratch detector of quiet low computational complexity. The proposed detector delivers a very solid basis for the next steps in image restoration. If it will be necessary to increase the efficiency of the ROD detector, even when this causes a higher computational load, the spatial information of the current frame has to be used in a sophisticated way, combined with the algorithm presented in this paper. ACKNOWLEDGEMENT This research is part of an ongoing Alexandria digital library project being carried out at the University of California, Santa Barbara under NSF Grant Number IR194-11330. REFERENCES [ 1] A. C. Kokaram, R. D. Morris, W. J. Fitzgerald and P.J.W. Rayner, "Detection of Missing Data in Image Sequences", IEEE Trans. Image Processing, Vol. 4, No. 11, pp. 1496-1508, Nov 1995 [2] W. Enkelmann, "Investigations of Multigrid Algorithms for the Estimation of Optical Flow Fields in Image Sequences", Computer Vision Graphics and Image Processing, Vol. 43, pp. 150-177, 1988 [3] J. Boyce, "Noise reduction of image sequences using adaptive motion compensated frame averaging", IEEE ICASSP, vol. 3, 1992, pp. 461-464 [4] E. Abreu and S. K. Mitra, "A Signal-Dependent Rank Ordered Mean (SD-ROM) Filter- A New Approach for Removal of Impulse from Highly Corrupted Images", IEEE ICASSP, Detroit, MI, USA, 9-12 May 1995, pp. 2371-4 vol.4 [5] A. C. Kokaram and P. J. Rayner, "A system for the removal of impulsive noise in image sequences", SPIE Visual Communication Image Processing, 1990, pp. 122-133 [6] R. D. Morris, "Image sequence restoration using Gibbs distributions", Ph.D. thesis at University of Cambridge, UK, May 1995
35
Top left illustration - Figure 6: Corrupted frame 49 of sequence WESTERN Top fight illustration - Figure 7: SDIa detector applied to frame 49 Bottom left illustration - Figure 8: MRF detector applied to frame 49 Bottom right illustration - Figure 9: ROD detector applied to frame 49
Time-Varying Image Processing and Moving Object Recognition, 4 - V. Cappellini (Ed.) 36
9 1997 Elsevier Science B.V. All rights reserved.
Feature Matching by Optimization using Environmental Constraints A. Branca, E.Stella, G.Attolico, A. Distante a aIstituto Elaborazione Segnali ed Immagini- C.N.R. Via Amendola, 166/5- 70126 Bari- ITALY Phone (39) 80-5481969 Fax (39) 80-5484311 bran ca ~ie si. ha. cnr.i t Matching is the capability to find correct correspondences among features extracted in two images of the scene acquired from different points of view or after TV camera motion. 3D stereo reconstruction and optical flow estimation are contexts of image understanding where matching has a fundamental rule. We describe a feature-based approach to solve the correspondence problem. Our goal is to correct initial matches, obtained by correlation, minimizing an appropriate energy function using as constraint the invariance of the cross ratio evaluated among coplanar points. 1. I n t r o d u c t i o n
Time-varying images of real-world scenes can provide kinematical, dynamical, and structural information of the world. To estimate the 3D motion and the structure of objects from image sequences, it is often necessary to establish correspondences between images, i.e., to identify in the images the projections corresponding to the same physical part of the sensed scene. The existing techniques for general two-view matching roughly fall into two categories: continuous and discrete. In this work a general method to perform discrete feature matching between images acquired in different times or from two different views is proposed. Generally, discrete matching techniques proposed in literature are implemented by direct methods, using local constraints on features [8], [1], or through optimization methods, using 'global constraints on features [5], [2],[10] to formulate an energy function to be minimized. While the direct methods are fast but more sensitive to noise, the optimization based techniques are more reliable though the drawback to require a burdensome processing. The energy minimization based approaches have been extensively used in literature [6][9] and most of them formulate the energy functional using constraints, determined considering feature characteristics, such as: uniqueness, ordering, disparity continuity. An unexplored field is that to consider projective geometrical invariance constraints in the optimization process computing features correspondences. In this paper an optimization method including cross-ratio invariant constraint of five coplanar points is proposed to solve the correspondence problem. The geometric invariance of cross-ratio of five coplanar
37 points has been used in literature as constraint for optimal matches selection in tracking algorithms, planar region detection or object recognition using probabilistic analysis [3],[11],[4],[12]. The performance of probabilistic approaches depends on the choice of rule for deciding whether five image points have a given cross-ratio [7]. In our method projective invariant constraints are included directly in the optimization process. We propose a feature-based approach to solve the correspondence problem by minimizing an appropriate energy function where constraints on radiometric similarity and projective geometric invariance of coplanar points are defined. The method can be seen as a correlation based approach which take into account the projective invariance of coplanar points in computing the optimal matches. In the following sections the algorithm used for optimal match selection (section 2) and the minimization technique implemented (section 3) to correct all mismatches are described. The experimental results (section 3) show as the approach provides good estimates of visual correspondences.
2. Raw Match C o m p u t a t i o n and Mismatch Selection Our aim is to define a new optimization algorithm for solving the correspondence problem using perspective invariance of cross ratio. Displacement vectors should be estimated only for features of "high" interest (using the algorithm proposed in [8]) which are salient points that can be more easily matched than other points. Initially, raw matches are computed maximizing the radiometric similarity among windows in the first image cenetered on high variance features and second image candidate features. Such matches represent the initial guess will be improved through an optimization process. Our idea is to use the geometric invariance of cross ratio C R ( P ) of five coplanar points P = (Pl, P2, Ps, P4, ps)
CR(P) - sin(a13)* sin(a24)
(1)
(where sin(ao) is the sin of the angle p(p-ff'pj) to verify the goodness of matches estimated through radiometric similarity and at same time to correct all mismatches. This require to satisfy the constraint that for all matches among neighboring points, given five points Pi~kl~ = {Pi,Pj,Pk,Pt,P~} in the first image and the corresponding points in the second image Qokl,.,, = {qi, qj, qk, qz, q,.,,}, the cross ratio computed for each subset must be the same. Previous works proposed in literature use the value of cross ratio computed on a group of five points to verify their coplanarity or match correctness. Evaluation performed on a single group must involve the use of thresholds, actually a small error in locating points can determine large variations in cross ratio values. These problems can be overcome if many combinations of five image points are considered. A mismatch or a point not coplanar with its neighborhood will be easily identified by considering the cross ratio similarity computed on all groups containing it.
38 3. Mismatch Correction through Optimization Cross ratio similarity computed on a large number of groups can be used as constraints to correct all mismatches generated using radiometric similarity. We propose to solve the correspondence problem by imposing that the sum of all differences between the cross ratio computed for each considered subset of five features of the first image and the cross ratio computed for the matched points in the second image, must be minimized. The energy function to be minimized to solve the correspondence problem is: E =
x"lvsaa ~,=1 IICR(P,)
-
IVFeattl C R ( Q , ) I I ~ + V, z_.,=~ ,~ _ R4)
(2)
where P,~ and Q,, are the n t h subsets of five points from the first and second image respectively, and the term P~ imposes that corresponding features in the first and the second image must have a radiometric similarity. The norm E will be minimized only when its partial derivatives with respect to all points qi in the second image equal zero. Satisfying this condition for each of the qi then a system on N F e a t simultaneous equation in N F e a t unknowns is generated. Since the problem is nonlinear, we use an iterative approach to compute the optimal solution. Each match is update iteratively by an amount given by the partial derivative of energy function with respect the same point scaled by a parameter fl determined at each iteration using the method of conjugate gradient:
Vi, q,+=Z ~ -~qi k--l,qh~qi
(3)
Since the partial derivatives of cross ratio estimated for a subset Qn with respect to a point qi G Qn depends to all points qk, k = 1...5 of Qn, the update should depend to radiometric similarity of other points {qk C Qn, qk ~ qi; k = 1...4}. Correct matches (with high radiometric similarity) will influence positively the update, on the other hand, mismatches (with a low radiometric similarity) will avoid any update depending of them. Starting from some approximate matches, the algorithm improve the solution until a predetermined convergence criterion is satisfied. Due to the nonlinearity of the system more than one solution can exist. The success to reach a global minimum, and not be trapped in local minimum, depends on having a good first-guess for the solution. The goal is to reject the noise introduced from correlation measurements. The approach we propose converges through iteration upon the desired correspondence points {qi} by implementing gradient descent along the E(q~) surface, which expresses the quadratic cost function's dependency on all of the {qi} points. The correct matches are not changed because the computed adapting signal control is zero due to the satisfaction of the geometrical constraint. On the other hand, mismatches are influenced by correct matches determining the noise rejection. When a stable state is reached, the energy function value in each subset will provide useful information to identify coplanar features.
39 4. Experimental Results The experimental results reported in this paper have been obtained from tests performed on time varying image sequences. The tests have been performed by considering pairs of image relative to a same static scene and acquired in different times while the TV camera is moving forward along its optical axis (the resulting optical flow must have a radial topology). Once highest interest features are extracted in the first image of a sequence, the raw matches, computed imposing the radiometric similarity, have been considered to define a large number of five point groups between neighboring features, in order to select all mismatches or to correct all mismatches through the optimization process. In the reported results we can observe as the performances of the approach are independent of the planarity of the scene. Actually, to satisfy the cross-ratio constraint it is sufficient that near features are coplanar, but it is not necessary that all extracted features to be coplanar. We can compare the results obtained from a sequence of coplanar image features in fig.(1) and those obtained from the image sequence in fig.(2) where features are extracted on different planes: the algorithm recovers the correct matches in both sequences. Finally, the ability of the system to select all correct matches from the raw measurements obtained through correlation, without apply the optimization process is shown in fig.(3). 5. Conclusions In this paper, we have proposed a new approach to solve the correspondence problem between sparse features of images acquired at different times or from two different point of view. The approach is based on cross-ratio similarity between five coplanar points. Cross ratio invariance constraint computed on a large number of combinations of five image points can provide an useful mean to identify mismatches generated by radiometric measurements and at same time to correct all mismatches through an optimization process. REFERENCES 1. N.Ayache, "Artificial Vision for Mobile Robots",MIT Press, 1991 2. N.Ayache and B.Faverjon, "Efficient Registration of Stereo Images by Matching Graph Descriptions of Edges Segments" The Int. Journal of Comp. Vision 1(2):107-131, April 1987. 3. H.Chabbi, M.O.Berger "Using Projective Geometry to Recover Planar Surfaces in Stereovision" Pattern Recognition Vol.29, No.4, pp.533-548, 1996. 4. S.Carlsson "Projectively Invariant Decomposition and recognition of Planar Shapes" International Journal of Computer Vision Vol 17 No 2 pp 193-209 1996. 5. Y.Ohta and T.Kanade, "Stereo by Intr- and Inter-Scanline Search. IEEE Trans. on Pat. Anal, and Mach. Intell.,7,No.2:139-154,1985. 6. J.J.Lee, J.C.Shim,Y.H.Ha "Stereo correspondence using the Hopfield Neural Network of a new energy function", Pattern Recognition, Vol.27,No.ll,1994.
40
Figure 1. (a)Start image of the sequence with extracted feature over-imposed. (b)Optical flow estimated after matching with correlation. (c)Optical flow estimated after matching with our optimization technique.
Figure 2. (a)Start image of the sequence with extracted feature over-imposed. (b)Optical flow estimated after matching with correlation. (c)Optical flow estimated after matching with our optimization technique. 7. S.J.Maybank "Probabilistic Analysis of the Application of Cross Ratio to Model Based Vision" International Journal of Computer Vision Vol 16 pp 5-33 (1995). 8. H.P.Moravec "The Stanford Cart and the CMU Rover",Proc. IEEE,1983. 9. J.P.Pascual Starink, E. Backer "Finding point correspondences using simulated annealing", Pattern Recognition, Vol.28,No.2,1995. 10. L.Robert and O.D.Faugeras "Curve-based Stereo: Figural Continuity and Curvature" In CVPRgl, 57-62. 11. D.Sinclair, A.Blake "Qualitative Planar Region Detection" International Journal of Computer Vision Vol 18 No 1 pp 77-91 (1996). 12. C.A.RothweU,A.Zisserman,D.A.Forsyth,J.L.Mundy"Planar Object Recognition using Projective Shape Representation" International Journal of Computer Vision Vol 16 pp 57-99 1995.
41
Figure 3. (a)Start image of the sequence with extracted feature over-imposed. (b)Second image of the sequence with features computed through correlation over-imposed. (c)Second image of the sequence with features corrected through optimization overimposed. (d)Optical flow estimated after matching with correlation. (e)Matches selected from the flow in (d). technique.
42
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All fights reserved.
System identification for fuzzy controllers G. Castellano, G. Attolico, T. D'Orazio, E. Stella, A. Distante " Istituto Elaborazione Segnali ed Immagini- C.N.R. Via Amendola, 166/5 - 70126 Bari- ITALY at t olico @iesi.ba. cnr.it Several robotic applications can be accomplished through a direct mapping between perceptual situations and control commands. Fuzzy logic is a useful tool for realizing such a mapping. It allows either explicit programming or automatic learning of control rules from suitable training data. A fuzzy control system for wall-following has been developed studying the problem of automatic extraction of rules from training data. Using a machine learning technique we build a compact rule base by estimating the relevance of each input signal in the control decisions. The derived fuzzy rules have successfully driven a TRC Labmate inside an indoor environment. 1. I N T R O D U C T I O N Some tasks involved in autonomous mobile vehicle navigation can be solved efficiently without using plan-based methods, that need some internal representations to be built and updated. Obstacle detection and avoidance, wall-following, door-crossing are examples of low-level strategies that can be realized by a direct mapping between the sensory input and the output control spaces, thus avoiding the delay introduced by the update of an environment model and enabling the use of simple and cheap sensors, as the ultrasonic ring used in our experiments. To describe mathematically the mapping between perceptual situations and control commands may be not easy or desirable. Therefore the use of techniques for designing good approximation of the desired behaviors is preferred. Both neural networks ([1], [2]) and fuzzy controllers ([3], [4]) have proved to give good results as function approximators in robot navigation applications. The learning process for a neural network is generally slow and the final knowledge is represented in a way difficult to evaluate, integrate and refine. The linguistic representation of fuzzy rules is easily understandable instead, allowing validation and correction at any time using information provided by human experts. The learning process for a fuzzy controller is quicker and scales gracefully with the size of the problem. The design of fuzzy systems requires the choice of input and output data, the definition of linguistic values (with the associated membership functions) for each fuzzy variable and finally the derivation of rules from the available knowledge (human experts and real data). To make automatically these choices means to improve the autonomy of the system in learning the initial strategy and in its tuning during on-the-job runs in a dynamic environment.
43 In [5] and [6] the rules are obtained by iteratively dividing the input and the output space into regions to which numerical input-output data are assigned Also machine learning approaches [7], neural networks [8] and genetic-based learning algorithms [9] have been used to derive a feasible set of rules. In [10] we developed a rule construction algorithm to automatically build a fuzzy wall-follower. The resulting rule base is efficient but contains a large number of fuzzy rules depending on the number and the granularity of input variables. This paper addresses the problem of automatically building a compact rule base by estimating the relevance of each input signal in the control decisions. Using a well known machine learning technique the number of produced fuzzy rules is drastically reduced with respect to [10] without weakening the controller. Experimental results will be shown using a fuzzy controller for the wall-following task. 2. T H E F U Z Z Y W A L L - F O L L O W E R 2.1. N o t a t i o n s and definitions Let us consider a fuzzy system with n inputs xl,...,xn and a single output y. Each input linguistic variable Ak E Ux, k = 1, ..,n, is characterized by Nk linguistic terms Akl, Ak~., ..., AkN~.. The linguistic variable B E Uo is characterized by M linguistic terms B1,B2,...,BM. A fuzzy set is associated with a crisp representative value, that we define as the modal point for triangular or gaussian shaped sets. Trapezoidal sets instead will be represented by the midpoint of the range of points having membership value 1.0. We will denote by aki and bj the representative value for sets Aki and Bj respectively. Let the rule that maps the i th multivariate fuzzy input variable A i to the jth univariate output set be labelled by rij, i.e.: rij: IF (xl is A~) A N D . . . AND (x~ is A~)THEN (y is B j) where A~ (respectively B j) is the linguistic value of the input fuzzy variable Ak (respectively the output fuzzy variable B) in rule rij. 2.2. Fuzzification, Rule Inference and Defuzzification We have adopted a nonsingleton fuzzifier, which is more adequate than singleton fuzzification when dealing with noisy data [11]. Our fuzzifier maps an input crisp value x into the fuzzy set A* for which pa*(x) > 0.5. A product-max inference has been applied. Rule evaluation by product operator retains more input information than using the min operator and generally gives a smoother output surface [11], a desiderable attribute for a controller. Given an input x = (zl,z2,..,zn), the firing strength of a rule is n
#,'o (x) - #A' (X) -- II /Za~ (Xk) k=l
(1)
while the final maximum membership value for the output set Bj, j = 1...M after the inference of all rules is given by: #-Bj (Y) - minCPej(y), im.a.~(p,,j (x))) where Y represents the support values of the output membership functions.
(2)
44 The center of area defuzzification method has been adopted, since it yelds a better performance of the controller than the mean of maxima method [11]. However, to reduce the computational cost of the method, we have considered the crisp representative value bj of the set Bj instead of its centroid. Thus the crisp control value is obtained as:
M y*-- Ej=I - ~ B j ( bj )bj Eg=I~Bj (bj)
(3)
-
2.3. R u l e C o n s t r u c t i o n Following the machine learning approach of [7], we have developed a rule construction method, which applies the ID3 algorithm to construct a compact rule base by selecting only the most relevant input variables. The rule construction algorithm involves building a decision tree by recursively taking as root of each subtree the input variable with greater information content and with less homogeneous branches (in terms of values of that variable in the training set). Rules are obtained by crossing all possible branches of the tree from the main root the leaf nodes, representing the output fuzzy values. At each level of the decision tree we define:
nt the total number of training samples at that point of the tree nsj the number of the nt samples with y in Bj nAhi is the number of the nt samples with xk in Aki nAkiBj is the number of the nt samples with x~ in A~ and y in Bj In order to evaluate the importance of each input variable, the Quinlan's Gain Ratio has been adopted as information measure. For an input variable ~ this is defined as: GR(xk) -
INF(y)-
Mk
INF(x~)
where Mk -- ~ nAh______A (_i i nt j
nAh,Bj 1og2nAkiBj ) nBj nBj
is the information content if xk is selected as root of the current subtree, and N~
INF(=k) - - ~
z9
nab, 1og2nAh, nt
,
M
I N F ( y ) - - ~ , :~J log2 n ' i
nt
39
nt
are the total information content for the input variable xk and the output variable y respectively. In addition, in order to avoid useless details in the decision tree (normally producing about 100 rules) at each step we create a sub-tree only if it produces a relevant reduction of the error rate, that is if
N
E eAk~ ~ et--s i=l
where
45
~q - - 4 V nt nAhi eAki -
et
-
-
-
nt
-
TI,AhiBrna~ TI,Bma,. ~
-
0.5 0.5 - - maxj=l,...M n A k i B i maxj-1,...M TI,B j -
-
7~AkiBrna~ -~
nB,~~
+
3. E x p e r i m e n t a l R e s u l t s The Sistema AUtonomo RObotizzato (AUtonomous RObotic System) SAURO, an autonomous vehicle oriented to transportation tasks in indoor environments [fig.l), has been used for collecting the training data and for testing the fuzzy wall-follower. SAURO is a LABMATE mobile base provided with a VME bus processing system and a ring of 18 ultrasonic sensors.
Figure 1. The mobile robot SAURO.
Figure 2. Arrangement of ultrasonic sensors and sensor suits.
Input to the fuzzy controller are the ultrasonic sensors measures grouped into suits, according to the required spatial resolution, each one providing a single value to the control system (fig.2). Also the number of linguistic labels associated to each fuzzy variable depends on the position (and relevance for the wall-following task) of the corresponding suit (fig.3). The motion control of the mobile robot is realized by setting its steering velocity w (fig.4). For simplicity a constant forward speed has been assumed. Training data (sensory input and corresponding control output) have been collected during navigation sessions of the vehicle driven by a human operator along a wall on its right-hand side. Fig.5 shows the training environments and the corresponding trajectories of SAURO. The rule construction method used in [10] has derived about 170 fuzzy rules, with a visible degree of redundancy. With the application of the ID3 method, only 15 rules have been extracted, without decreasing the performance of the controller. Each rule does not
46
N
F
V
~. ( m m )
150
300
10000
500 (a)
..
150 200 300
500
600
NB
NM NS t PS PM
PB
-15
30
,. (mm) 10000
(b)
Figure 3. Membership functions of the input variables (a) LeftSack, LeftFront, Front and (b) RightFront and RightBack.
-60
-30
-5 0 5 15
60
Figure 4. Membership functions of the output variable.
use necessarily all the 5 input data, therefore exploiting the real relevance of each input on control commands. The final number of rules is comparable with the size of hand-written fuzzy rule-base for similar tasks. The compact controller has successfully driven SAURO to follow unknown configurations of walls in both simple (fig. 6) and complex (fig. 7) environmental situations. It can be noted that the robot is also able to avoid unespected obstacles by correctly changing its trajectory and still following the wall.
STTS
ST~__&T
3
START
~\
Figure 5. Situations used for collecting the training data.
4. C O N C L U S I O N S Fuzzy navigation controllers can be an effective solution for the implementation of navigation behaviors that do not require internal representations of the environment, so hard to acquire and to update, necessary for conventional plan-based techniques. Automatic learning and continuous adaptation of the control strategy by representative real data can produce fuzzy rules, that experts can then evaluate and tune with their skills. A first automatic derivation of the fuzzy rule base, has produced redundant rule bases. By estimating the relationship between input and output data, we have build a fuzzy wall-f011ower with a reduced number of rules. Simplifying the fuzzy controller is especially important in prospect of the extension of the control system to the complete set of behaviors required by a safe navigation in indoor environments.
47
~,~
START/
Figure 6. SAURO's trajectory in a simple environment.
STAR
Figure 7. SAURO's trajectory in a complex environment cluttered with obstacles.
REFERENCES
1. D.A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3, 1991. 2. H. Meng and P.D. Pincton. Neural network for local guidance of mobile robots. In Proc. of Inter. Conference on Automation, Robotics and Computer Vision, pages 1238-1242, Singapore, November 1994. 3. K.T. Song and J.C. Tai. Fuzzy navigation of a mobile robot. In Proc. of I E E E / R S J Inter. Conference on Intelligent Robots and Systems, volume 1, pages 621-627, Raleigh, NC, July 1992. 4. W. Li. Fuzzy logic based robot navigation in uncertain environments by multisensor integration. In Proc. of the 1994 IEEE International Conference on Multidensor Fusion and Integration for Intelligent Systems (MFI '94), pages 259-265, Las Vegas, NV, October 1994. 5. M. Lan S. Abe. Fuzzy rules extraction directly from numerical data for function approximation. IEEE Transactions on Systems, Man and Cybernetics, 25(1):119129, January 1995. 6. L.X. Wang and J.M. Mendel. Generating fuzzy rules by learning from examples. IEEE Transactions on Systems, Man and Cybernetics, 22(6):1414-1427, November 1992. 7. J.Y. Hsu S.C. Hsu and I.J. Chiang. Automatic generation of fuzzy control rules by machine learning methods. IEEE Proc. of lnt. Conference on Robotics and Automation, pages 287-292, 1992. 8. Y. Lin and G. A. Cunningham III. A new approach to fuzzy-neural system modeling. IEEE Transactions on Fuzzy Systems, 3(2):190-197, May 1995. 9. A. Homaifar and Ed McCormick. Simultaneous design of membership functions and rule sets for fuzzy controllers using genetic algorithms. IEEE Transactions on Fuzzy Systems, 3(2):129-138, May 1995. 10. G. Castellano, G. Attolico, E. SteUa, and A. Distante. Learning the rule base for a fuzzy controller. In 4th IEEE Mediterranean Symposium on Control and Automation (MSCA '96), Crete, Greece, June 1996. 11. M. Brown and C. Harris. Neurofuzzy Adaptive Modelling and Control. Prentice Hall, 1994.
This Page Intentionally Left Blank
C COMPUTER VISION
This Page Intentionally Left Blank
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All fights reserved.
Computer Vision Applications
for
Autonomous
51
Navigation" from
Research
to
G.Garibotto, P.Bassino, M.Ilic, S.Masciangelo Elsag Bailey -TELEROBOT, via Hermada 6, Genova, Italy The results described in this paper represent the conclusion of a cycle of researches carried out in the last few years in the field of Artificial Vision applied to mobile robotics. A first prototype system was experimented in the field of service robotics, including light material transportation (mail, documents, medicines, clinical data), and museum guide services. This technology has been recently applied to the driving control of standard transportation vehicles for palletised goods and materials. A prototype system named ROBOLIFT has been developed and is presently in the stage of industrial exploitation. The referred results are a concrete demonstration of technology transfer from basic research studies to the application domain and the level of maturity of Computer Vision technology for industrial use in Service Robotics. 1. INTRODUCTION TO ROBOTIC VISION In this section we briefly refer the main steps of our research efforts, from the beginning of the '80s with a strong industrial investment in robotics applications. At that time the driving force of the research effort was the development and integration of prototypes of Vision systems for a manipulating robot to perform flexible assembly operations [1 ]. Beside modelbased image recognition and object positioning and orientation, the main contribution was the effective integration of the system in a fully autonomous robotic assembly cell. The achieved performance were satisfactory in terms of flexibility and processing speed, but the high costs of the parallel implementation on the proprietary EMMA2 architecture [1] and the lack of a standardisation prevented a wider industrial exploitation of these results. In the mid of the '80s an international project has been established to better investigate 3D stereovision and motion analysis and realise a special hardware machine to perform such image processing functions almost at video rate. The main result of this European project (ESPRIT P940)[2] has been a multi-DSP parallel processing architecture based on M E - b u s which allowed 5 Hz 3D stereo reconstruction, using trinocular (three cameras) stereo arrangement and tracking of linear segment features at a rate of 10 Hz. The limited market size, as well as the not yet proved reliability and robustness of the on-going research did not allow a consolidation of the system as an industrial product. On the other hand the P940 machine has represented for many years (since the end of the project in 1992) a strong competitive advantage for the European industry in Computer Vision and a very powerful advanced research environment for real time experiments in the field of Computer Vision [3]. We have successfully applied this technology in different contexts (quality inspection and control), and
52 in robotic metrology, using camera calibration [4] for the 3D reconstruction of surface patches of object models. Anyway, one of the most challenging problem for Computer Vision was clearly identified, since the end of the '80s, in the development of intelligent sensors for autonomous navigation control. This is the context where almost all features of Vision research can be fully exploited, in terms of adaptability, dynamic response, visual servoing, learning and understanding of the environment, perception of global features (self-orientation) and local features (obstacle detection) [5]. From 1987 to 1992 our team has participated to an international project, ESPRIT P2502 [6], aimed to develop vision technologies for mobile robotics, together with the most qualified European Research centres in the field. The final demonstration in Genova was a combination of interactive teleguidance, and stereo-based obstacle detection, using off-board processing with a special hardware workstation. Moreover, a strong experience was gained in monocular vision with the development of perspective inversion tools and geometric reasoning by 3D model based techniques. In 1993, to demonstrate the maturity of vision-based navigation, using on-board low-cost PC-based processing hardware, a fully integrated system, SAM (Autonomous Mobile System), was realised, to address a wide class of autonomous navigation and transport tasks, in an indoor environment, in presence of people. In the following section 2 a brief description of this first prototype system is referred. More recently, at the beginning of 1994, an industrial oriented project has been started, to put into practice Vision technology to achieve competitive results both in terms of performance and costs. The goal of this project was the automation of an existing, conventional fork-lift carrier using an intelligent on-board control system, driven by Computer Vision. The reference application was the transportation of self-supporting palletised goods in a warehouse. Section 3 is devoted to briefly recall the Vision techniques which has been developed and used in this project as well as the current experimental results of the engineered version ofRoboLift. A more detailed description of the system can be found in [7]. 2. DESCRIPTION OF THE MOBILE ROBOT SAM The logic architecture of SAM was implemented as a series of almost independent layers of competencies, each one in charge of a single, well defined task, such as obstacle avoidance and global position maintenance. The obstacle avoidance strategy is reflexive, that is the trajectory is heuristically determined on the basis of sensor readings rather than accurately planned starting from a reconstructed local environmental map. The suboptimality of the obtained trajectory is largely compensated by the fast response time which allows to navigate safely at an acceptable speed also in presence of cluttered environments. The hardware solution is based on a PC platform as the main computational infrastructure to reduce costs, minimise the development time and take advantage of the wide choice among a great variety of add-on boards which can be integrated to improve the system functionality. The navigation system needs a periodic position and orientation estimation coming from an external sensor in order to reset drifts of odometry. This is provided through a Vision system able to detect and recognise navigation landmarks placed in known position along the robot routes [8], and to recover the robot position and orientation with respect to them.
53
Figure 1: The mobile robot SAM
Figure 2: The artificial landmark
The selected artificial landmark consists of a black annulus on a white background, as depicted in Fig.2. The 3D position and attitude of the camera with respect to the landmark reference system is obtained from a single image through model based perspective inversion.
2.1. Summary of experimental results Beside extensive laboratory experiments, the system SAM has been tested for two years in our office environment, for document transport and guest accompany service, in normal operating conditions with a lot of people wandering around. Later, the robot SAM has been equipped with a Soundblaster board for sound generation and a radio modem providing a link to an host computer at a remote control station. The remote computer is able to select the most appropriate sound or voice file according to the robot position or the navigation status (presence of obstacles, landmark search, and so on) as communicated by the robot navigation system. The robot in such a configuration was installed in an historical building during the Christmas '93 exhibitions, as reported in [9]. The results has been very encouraging in terms of performance and good acceptance by the people who visited the museum during the exhibition days. 3. ROBOLIFT: AUTONOMOUS FORK-LIFT IN LOGISTIC SERVICES The problem of autonomous transport of material in workshops and warehouses has been traditionally approached through the use of automated guided vehicles (AGV). They have been introduced into the market in the early '70s and have provided significant improvements in terms of efficiency, precision and co-ordination of material flow in manufacturing, as compared to conveyor belts, single track railways, etc. The main drawbacks of the consolidated AGV technology [10] comes from the heavy installation requirements (inductive guides buried into the floor), the need of continuous central control, the rigidity of fixed navigation pathways, the requirement to design specialised machines for each different application. Moreover there is a severe limitation in flexibility, and
54 the position of all the palletised loads in the warehouse is supposed to be known in advance with high precision (within the range of 10 mm.). Our answer is RoboLift the first Autonomous Fork Lift developed jointly by Elsag Bailey Telerobot and Fiat OM Carrelli Elevatori SpA (Patent pending). This system is based on Vision technology both for autonomous navigation and for the recognition of the pallets to be transported. 3.1 Main Characteristics of the vehicle The selected basic vehicle is the classical frontal fork lift carrier (from the well known EU family of Fiat OM Carrelli Elevatori SpA, operating in the range of 1.2, 1.5 ton), being one of the most commonly used in the market. The kinematics of the vehicle is made of three wheels (two driving and one steering). A schematic drawing of the vehicle and the list of sensors which have been introduced for autonomous control is referred in figure 3.
Figure 3 Sensor arrangement in ROBOLIFT
Fig.3 Model based vision and 3D recognition and positioning of the pallet
3.2: Computer Vision for Autonomous Navigation Vision processing is performed primarily to support autonomous navigation. Through the recognition and 3D location of some artificial landmarks (H shaped) placed on the floor along the planned navigation path, the Vision system is able to self-localise the vehicle in the scene, by integrating this information with the odometric values coming from sensors on the wheels (both drive and steering).
55 3D model based vision is used to identify and recognise the H-shaped landmarks placed onto the floor, by exploiting all a priori information available to simplify image processing analysis. To avoid errors and ambiguities caused by other features in the scene or noise effects, geometric reasoning is performed directly in 3D, by reprojecting all features onto the 3D floor. Using extended geometric features, this process is proved to be extremely robust also when the landmarks is partially occluded or damaged by stains. Computer Vision is performed on-line, during the motion of the vehicle passing through these landmarks, along the navigation path. The success of Artificial Vision is strongly related to the accuracy of camera calibration, computed with respect to the odometry of the vehicle, to obtain a homogeneous data representation suitable for the navigation commands (steering and driving). 3.3" Computer Vision for the recognition of the pallet pose A second fundamental vision function which is implemented in ROBOLIFT is pallet detection and recognition and is performed by a camera placed within the forks and rigidly connected to them, so that it can move up and down, searching for the different position of the palletised load. A model-based Vision algorithm has been implemented, to search for the central cavities of the pallets and compute the size and shape of these holes. A prediction-verification paradigm has been implemented. It consists in projecting onto the image the geometry of the pallet model, from the expected position in the 3D world. An adaptive estimation of the contrast is performed into the expected hole position in the image, followed by a controlled region growing process aimed to propagate this grey level up to the border of the hole, with a constraint on the expected size and shape. A schematic example is referred in figure 3. Once the holes are correctly identified and localised, the current 3D position of the pallet is computed. If this new position is within the tolerance bounds to be carried by the fork-lift, the forks will be properly shifted to left or right of the appropriate amount to take the load in a centred position. The project has been developed by taking as a reference the standard Europallett of size 1200 x 800 mm., with loading side 1200. The Computer Vision system has proved to be able to recognise the presence of the pallet, also in a wide range of operating conditions, from intense sun-light, artificial light and shadows, and compute the current distance and orientation of the pallet with respect to the vehicle.
4. RESULTS AND STATUS OF THE PROJECT The ROBOLIFT project started in 1994, with the definition of the basic architecture and the selection of the necessary modifications implemented on a first prototype. During 1995 a first laboratory prototype has been integrated and a first release of control software has been made available in July '95, followed by an extensive experimentation phase in a suitably equipped warehouse environment. A second engineered prototype has been integrated at the beginning of'96, and it has been officially presented at the Hannover Fair in April '96. Further engineering work is in progress to improve the system performance, its robustness and reliability as well as the level of integration with the application field.
56 5. CONCLUSIONS One of the most important objective of advanced research institutions, including EEC programmes, is the promotion of industrial exploitation of research results with particular attention to A.I. technologies which received significant research funds in the last few years. The paper describes our recent experience in exploiting Computer Vision technology in transport service automation, following the necessary fundamental steps of basic research development, laboratory prototype implementation, and the final acquisition of a strong integration knowledge and expertise. The achieved results consists in the integration of an autonomous fork-lilt carrier, which can be also driven in the conventional way by the human operator. The system makes use of model based passive vision techniques, without any external active lighting support. Computer Vision represents the main sensory component for both autonomous navigation and pallet recognition. The possibility to use a standard PC-based multiprocessing architecture allows the implementation of a competitive industrial system. The extensive experimental results collected during many hours of test demonstrate a high maturity of Vision technology in advanced mobile robotics, to come out from the research labs an be used as an established and accepted technology in Industrial Automation. REFERENCES 1. L.Borghesi, et al. "A Modular Architecture for a flexible real-time Robot Vision System", Proc. of the Int. Conference on Digital Signal Processing, Firenze, Sept. 1987. 2. G.Garibotto, S.Masciangelo, "Depth and Motion Analysis P940: development of a realtime Computer Vision System", ESPRIT Workshop at ECCV'92, S.Margherita, May 1992. 3. Faugeras, "Three-Dimensional Computer Vision; a geometric viewpoint", The MIT press, 1993. 4. E.Bruzzone, F.Mangili, "Calibration of a CCD Camera on a Hybrid Coordinate Measuring Machine for Industrial Metrology", International Symposium on Industrial Vision Metrology, Winnipeg, Manitoba, Canada, July 1991. 5. G.Garibotto, S.Masciangelo, "3D Computer Vision for Navigation/Control of Mobile Robots, in Machine Perception, AGARD Lecture Series, 185, 1992. 6. B.Buxton, et al. "The Transfer of Vision Research to Vehicle Systems and Demonstrations", Proc. of the ESPRIT Conference, 1991, Brussels. 7. G.Garibotto, "ROBOLIFT: Vision-guided Autonomous fork-lilt", Service Robot, An International Journal, Vol.2, n.3, 1996, pp.31-36. 8. Garibotto, M. Ilic, S. Masciangelo, "An Autonomous Mobile Robot Prototype for Navigation in Indoor Environments", Proc. of the Int. Symposium on Intelligent Robotic Systems '94, Grenoble (France), July 1994. 9. Garibotto, S. Masciangelo, M Ilic, "Vision Based Navigation in Service Robotics", pp. 313-318, Image Analysis and Processing, Lecture Notes in Computer Science, Springer, 1995 10. Warnecke, C.Schaeffer, J.Luz, "A driverless and free-ranging fork-lift carrier, Proc. of the 24th ISIR, session El, Nov. 1993.
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All rights reserved.
57
An Optimal Estimator of Camera Motion by a Non-Stationary Image Model G. Giunta
U. Mascia
INFO-COM Department, University of Rome "La Sapienza", Rome, Italy e-mails:
[email protected] [email protected] Camera zooming can be regarded as a 2D Doppler effect. Techniques for Doppler estimation from 1D signals, based on data partition and linear regression on a set of time-delay measurements, were presented in the literature. Such basic idea is here extended to fine motion estimation. The devised algorithms, estimating four global motion parameters (viz: horizontal and vertical translation, zooming, and rotation), are based on a non-stationary model. They have been validated by both synthetic and experimental tests. 1. I N T R O D U C T I O N The analysis and estimation of motion are very important in time-varying image processing. Many algoritms have been developed to estimate 2D motion for different applications [ 1]-[ 12], such as object tracking, image segmentation, environment sensing for autonomous vehicle navigation, image sequence coding, object-oriented analysis and synthesis coding, TV standard conversion, frame rate conversion, bandwidth compression for HDTV, very low bit-rate video coding for audiovisual services, 2D motion parameter acquisition from image sequences, camera motion estimation and compensation, etc.. Fast discrete-time techniques for time-delay estimation with sub-sample accuracy, based on a parabolic interpolation of estimated cross-correlation samples, were devised and analysed for random 1D signals [ 13]. Camera zooming (or radial motion) causes an isotropic change of the scale in the whole image, that can be regarded as a 2D Doppler effect on the magnitude of spatial polar coordinates. Moreover, any rotation can be modeled as a 2D Doppler effect on the phase of spatial polar coordinates. Techniques for Doppler estimation of 1D signals were proposed in the literature [14]-[16]. Among them, an indirect estimation method based on data partition and linear regression on a set of time-delay measurements (linearly related to the actual Doppler coefficient) was devised [ 17]. Such basic idea is here extended to a 2D fast estimator of spatial position. Motion compensation brings about a saving in bit-rate, due to the smaller prediction error as well as to the reduction in motion-vector coding. A fine estimation with sub-pixel accuracy can take account of more information, giving better results in the prediction of the picture. Stationarity is widely assumed in image modeling for the sake of simplicity, while it is well known that this assumption is far from reality. A motion estimation procedure with a subpixel accuracy is here presented. It is based on a non-stationary model depending on local properties. This approach can be usefully employed to reach our aim, by extending the method devised in [ 17], for estimating the four global motion parameters (viz: horizontal and vertical translation, zoom, and rotation). In particular, a 2D paraboloid is used for interpolating the inter-pixel cross-correlation estimates. This method can be applied to 2D Doppler estimation, after a block partition of the whole image, by a linear regression in the complex domain of the spatial dispacements. Such measurements can be also weighted according to a proper error function, derived from a confidence measure, accounting for the local statistics of each block. Such a non-stationary method is then based on the minimization of the weighted mean square error.
58 2. I M A G E MODEL AND MOTION ESTIMATOR
2.1 Time-varying image model Let z = x + j y be the Cartesian coordinates, expressed in the complex domain. Let us consider a pair of sequential image frames, say R and P. Let us assume the following model of instantaneous motion: R(x,y) = R(Z) = S(Z) (1) (referencepicture) P(x,y) = P~) = S[~-~i) / ~ + E~) (2) (moved picture) where E(/,) is the model error image, that also accounts for the two noises. In particular, Re{fi} and Im{fi} represent the horizontal and vertical displacements, while the term o~=pexp [j 0] accounts for both the zoom factor (p) and the rotation angle (0). It is interesting to point out that any rotation can be directly included in the MSE estimation in the complex domain: in fact, a complex change of scale takes zoom as well as rotation into account (i.e. modulus represents zoom, and phase represents angle of rotation).
2.2 Displacement estimation Our method performs a fine (sub-pixel) estimation by means of a fast digital algorithm. The whole picture is divided into small blocks and the relative position displacements are extracted by a conventional matching algorithm based on cross-correlation measurements. The estimated displacements are linearly related in the complex domain and the four parameters can be derived by performing a complex linear regression. The relationships so obtained can be weighted according to their accuracy, which depends on the contents of each block of data. In paticular, we divide the whole reference image into small blocks. For each block, we search for the best matching block in the moved picture (like accomplished in several widely used standard), by evaluating the magnitude of displaced frame difference, i.e.: MDFD(.-I) = { ~k I R(_zk) - P(_zk+~) I } (3) where the sum extends over all the pixels of the considered block. We then estimate the linear motion between the considered pair of blocks as the displacement ld, that minimizes the magnitude of displacedframe difference, i.e.: R = "c 9arg MDFD(Id,) = arg mini: MDFD~) (4) by performing a sub-pixel parabolic interpolation of the square of the estimated MDFD(.TJ: MDFD2(.~) = MDFD2(~,~) _=a ~2 + b ~2 + c ~ + d ~ + e ~ + f (5) with :t = ~ + j ~ . The displacement ld, is determined as the minimum argument of the parabolic function, fitted by six samples of the squared MDFD(jO chosen around its coarse minimum.
2.3 MSE global motion estimation Let /5 be the center of the i-th reference block. We may write a (usually overdetermined) set of linear equations in the complex domain:
o~zi + ~ = R i
(6)
In fact, there are 2 complex unknowns (8 and ~ accounting for 4 real parameters (viz: horizontal and vertical displacement, zoom factor and rotation), while the number of equations is equal to the number of considered blocks in the whole picture.
59 If we chose the origin of the coordinates at the exact center of the reference frame and we take a symmetic arrangement of N blocks into account, a standard pseudo-inverse solution, based on the mean square error (MSE), can be employed: N N
~5= 1 ~
i=l
~l,i (7/
zi Ri
o~=~
(8)
N
i=l
~ z i zi i=l
2.4 WMSE global motion estimation
If we have no particular symmetries or if we wish to take the available equations into account with different weights, a weighted mean square error (WMSE) can be defined: /5=~-1 I [ ~ w i2 z i*zi][~ w i2 i]~l,- [ ~ w i2 *i l [ ~z w i2 z i ~l,i] 1 i=l
i=l
i=l
ct= 1 I[i__~1 w2l [i__~1w2z~Ril-[i__~1 w2z~l [i__~1w2Ril I with
A= ~ i=l
~w2zi z
~wiz i
i=l
i=l
(9)
i=l
(10,
wiz i=l
In fact, while the MSE criterion minimizes (6), the solutions (9)-(10) minimize the set of equations: Wi [ (t ~ + ~5 ] = Wikl, i (11) We have employed a parabolic approximation of the squared MDFD (5) near the minimum in the displacement estimation. It is well known the dependence of time-delay estimation performance on such a parameter (its second derivative provides the asymptotic error variance). The curvature of the same squared MDFD function is the simplest local confidence measure that we may use. In particular, we have employed the curvature of the estimated squared MDFD (depending on the cross-correlation function) along the direction of the local motion gi estimated for each block, i.e.: --
2
wi
M D F D 2(.~i)
~ ~i 2
I
(12)
,~i= Ri
MDFD (depending on the cross-correlation coefficient), divided by the variances 0..Kz- and o.. 2 of the reference and the moved blocks, i.e." t" and the normalized
1 . wi2 OROp
i
~,i2
MDFD 2r
(13)
~i=ai with Y'i= ~'i exp[Jei]' evaluated along the same direction (i.e. ei=angle{Ri}).
60
3. SYNTHETIC AND EXPERIMENTAL RESULTS 3.1 Synthetic tests with known global motion parameters A number of standard still images (viz: "Airplane", "Barbie", "Baboon", "Boats", and "Gold") have been considered. The test images were reduced by the factor 4 from the original size of 512.512 pixels, becoming 128.128 pixels sized. Each image was compared to a copy of itself, deformed by a known set of global motion parameters. The parameter values were randomly chosen, each one with a uniform probability density function, in the range [-0.5,0.5] pixels for the horizontal and vertical displacements, in [0.95, 1.05] for the zoom factor, and in [-3,3] degrees for the rotation angle. Each image was tested 500 times, and a total of 2500 test images was then obtained. The motion vectors were estimated from 8.8 pixels sized blocks. Three different weights were used: NOW = no weight; DCCF = directional convexity of the cross-correlation function; DCCC = directional convexity of the cross-correlation coefficient. The numerical results of accuracy of the four estimates are reported in tabs. 1-4 for the cases of few (16) and many (195) blocks used in the performed tests. H.Dis.(pels) NOW DCCF DCCC
few (16) blocks Bias.10 Var.ol02 MSE.102 0.13 1.46 1.48 -0.09 0.27 0.27 -0.12 0.29 0.30
many (195) blocks Bias.10 Var..102 MSE.102 0.04 0.17 0.17 0.01 0.33 0.33 0.02 0.30 0.30
V.Dis.(pels) NOW DCCF
Bias.10 0.63 0.14
Var.~ 1.78 0.65
MSE.102 2.17 0.67
Bias.10 0.08 0.04 0.04
Var..102 0.15 0.14 0.12
MSE.102 0.16 0.14 0.12
Zoom factor NOW DCCF DCCC
Bias.103 4.88 1.37 1.31
Var.ol06 68.1 21.0 21.1
MSE.106 91.9 22.9 22.8
Bias.103 0.11 0.17 0.11
Var..106 1.02 1.54 1.78
MSE.106 1.03 1.57 1.79
Rot.angle (o) NOW DCCF DCCC
Bias.10 0.97 0.36 0.37
Var..102 43.5 9.8 9.7
MSE.102 44.4 9.9 9.9
Bias.10 -0.05 0.01 -0.05
Var..102 0.31 0.22 0.23
MSE.102 0.31 0.22 0.23
Tables 1-4. Estimation accuracy of the synthetic tests for a small and a large number of blocks.
3.2 Experimental tests with unknown global motion parameters Pairs of frames from two standard test image sequences (viz: "Foreman" and "Tabletennis") were extracted, because of their global motion characteristics. The subsequences were alternately cut forward and back. The motion vectors were estimated from 8-8 pixels sized blocks and collected to simultaneously estimate the four motion parameters. Some significant results of the three estimates (NOW, DCCF, and DCCC) of the rotation angle from the "Foreman" sequence (320 blocks per frame, 141 frames) and of the zoom factor from the "Table-tennis" sequence (1176 blocks per frame, 93 frames) are shown in the figs. 1-2.
61
Frames
w
from
"Foreman"
3
i!
L,
N0W [XX~
~ t 9
R
ii
il~.
l i i i~ t~
-
l/,J
//!
l
ol c t~
c o
in
-1
I I , .... ,.h,
m o
-2
9
90
i
9
100
l
9
110 frames
i
120
130
Fig. 1. Estimates of the rotation angle from an experimental sub-sequence. Frames
from
.
#
"Table-tennis"
1,03 L_
o r
1,02
N,-
E o o N
, t -"
,, x...~,....,,f .~
~\
1,01 DCCC 1,00
50
.
,
60
.
,
70
frames
9
,
80
. 90
Fig. 2. Estimates of the zoom factor from an experimental sub-sequence. 4. C O N C L U D I N G
DISCUSSION
The results of the synthetic tests, performed on actual standard images to validate the devised algorithm, have shown that the WMSE-based method is suited in the presence of a small number of blocks, while its accuracy is comparable to a simpler MSE-based method for a larger number of available data. No significant difference appears between the results obtained with weights derived from the local cross-correlation function or from the local crosscorrelation coefficient. The method has been also applied to standard test image sequences. It visually appears that the WMSE criterion enhances the dynamic properties of the algorithm (this can be useful for fast tracking the camera motion), while MSE-based estimates are usually more smoothed.
62 The specific criterion (namely: uniform MSE, CCF-based WMSE, CCC-based WMSE) should be chosen in practice according the particular application. As a general remark, a simple MSE-based technique is suited for estimating the camera motion of high resolution image sequences, while both the examined WMSE-based methods should be preferred on small images, such as a region of interest extracted by a segmentation algorithm and containing a moving object. Future research investigations will include the case of multiple objects moving on a background. The mathematical problem then becomes a multi-linear regression, that can be resolved after a proper clustering of the available measurements. REFERENCES
[ 1] J.K. Aggarwal and N. Nandhakumar, "On the computation of motion from sequences of images - A review", Proc. IEEE, Vol. 76, No. 8, 1988, pp. 917-935. [2] G.J. Keesman, "Motion estimation based on a motion model incorporating translation, rotation and zoom", in Signal Processing IV: Theory and Applications, 1988, pp. 31-34. [3] S.F. Wu and J. Kittler, "A differential method for simultaneous estimation of rotation, change of scale and translation", Signal Processing: Image Communication, Vol. 2, 1990, pp. 69-80. [4] J.H. Moon and J.K. Kim, "On the accuracy and convergence of 2D motion models using minimum MSE motion estimation", Signal Processing: Image Communication, Vol. 6, 1994, pp. 319-333. [5] Z. Eisips and D. Malah, "Global motion estimation for image sequence coding applications", Proc. 17th Conv. of Elec. and Electronics Eng., Israel, 1991, pp. 186-189. [6] Yi Tong Tse and R.L. Baker, "Global zoom/pan estimation and compensation for video compression", Int. Conf. Acoust. Speech Signal Proc., 1991, Vol. 4, pp. 2725-2728. [7] G. Giunta, T.R. Reed and M. Kunt, "Image sequence coding using oriented edges", Signal Processing: Image Communication, Vol. 2, No. 4, 1990, pp. 429-440. [8] M. Bierling and R. Thoma, "Motion compensating field interpolation using a hierarchically structured displacement estimator", Signal Processing, Vol. 11, 1986, pp. 387-404. [9] Y. Ninomiya and Y. Ohtsuka, "A motion compensated interframe coding scheme for television pictures", IEEE Trans. Commun., Vol. COM-30, 1982, pp. 201-211. [ 10] A. Amitay and D. Malah, "Global-motion estimation in image sequences of 3-D scenes for coding applications", Signal Processing: Image Communication, Vol. 6, 1995, pp. 507-520. [11] M. Hoetter, "Differential estimation of the global motion parameters zoom and pan", Signal Processing, Vol. 16, 1989, pp. 249-265. [ 12] P. Migliorati and S. Tubaro, "Multistage motion estimation for image interpolation", Signal Processing: Image Communication, Vol. 7, 1995, pp. 187-199. [ 13] G. Jacovitti and G. Scarano, "Discrete time techniques for time delay estimation", IEEE Trans. on Signal Proc., Vol. 41, No. 2, 1993, pp. 525-533. [14] C.H. Knapp and G.C. Carter, "Estimation of time delay in the presence of source or receiver motion", J. Acoust. Soc. Am., Vol. 61, No. 6, 1977, pp. 1545-1549. [15] J.W. Betz, "Comparsion of the deskewed short-time correlator and the maximum likelihood correlator", IEEE Trans. on Acoust. Speech Signal Proc., Vol. ASSP-32, No. 2, 1984, pp. 285-294. [ 16] J.W. Betz, "Effects of uncompensated relative time companding on a broad-band cross correlator", IEEE Trans. on Acoust. Speech Signal Proc., Vol. ASSP-33, No. 3, 1985, pp. 505-510. [17] E. Weinstein and D. Kletter, "Delay and Doppler estimation by time-space partition of the array data", IEEE Trans. on Acoust. Speech Signal Proc., Vol. ASSP-31, No. 6, 1983, pp. 1523-1535.
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All fights reserved.
63
A simple c u e - b a s e d c a m e r a calibration m e t h o d for digital production of m o v i n g images Y. Nakazawa, T. Komatsu
and
T. Saito
Department of Electrical Engineering, Kanagawa University 3-27-1 Rokkakubashi, Kanagawa-ku, Yokohama, 221, Japan
One of the keys to new-generation digital image production applicable even to domestic uses is to construct simple methods for estimating the camera's motion, position and orientation from a moving image sequence observed with a single domestic video camera. For that purpose, we present a method for camera calibration and estimation of focal length. The method utilizes four definite coplanar points, e.g. four corner points of an A4 size paper, as a cue. Moreover, we apply the cuebased method to the digital image production task of mixing real and CG moving image sequences. The cue-based method works well for the digital image mixing task.
1. INTRODUCTION
- BACKGROUND AND MOTIVATION -
Recently some research institutes have started studying digital production of a panoramic image sequence from an observed moving image sequence, construction of a virtual studio with the 3-D CG technology and so on, with intent to establish the concept and the schema of the new-generation digital image production technology. Such an image production technology, utilizing information about the camera's motion, position, orientation and so on, integrates consecutive image frames to produce such an enhanced image as a high-resolution panorama or mixes a synthetic 3-D CG image sequence and a real moving image sequence taken with a video camera. The key to the newgeneration digital image production applicable even to domestic uses is to develop simple methods for estimating the camera's motion, position and orientation from a real moving image sequence observed with a single video camera [1 ]-[4]. In this paper, to render it feasible to pertbrm such 3-D estimation of the camera's motion, position and orientation when we use a single do~nestic handy video camera whose camera parameters are not given in advance, we present a method for performing camera calibration along with estimation of focal length of the camera accurately by using four definite coplanar points, which usually correspond to four vertices of a certain quadrilateral plane object such as an A4 size paper, as a cue. The practical computational algorithms for the cue-based method of camera calibration are composed of simple linear algebraic operations and arithmetic operations, and hence they work so well as to provide accurate estimates of the camera's motion, position and orientation stably. Furthermore, in this paper, we apply the cue-based camera calibration method to the image production task of mixing a synthetic 3-D CG image sequence and a real moving image sequence taken with a video camera according to the recovered estimates of the camera's motion, position and orientation.
64 2. CUE-BASED C A M E R A C A L I B R A T I O N In this paper, we assume the following situation. The situation is as follows: while moving the single video camera arbitrarily by hand, we image the scene which includes not only the objects of interest but also four definite coplanar points P~ -- P4 whose relative positions are known in advance and which usually correspond to four vertices of a certain quadrilateral plane object with known shape and are used as a cue for camera calibration. We perform camera calibration, that is to say, determination of the camera's position and orientation at each image frame, from 2-D spatial image coordinates of the four definite coplanar cue points, which are detected with our recently presented active line-contour model [5] and tracked temporally over consecutive image frames. Under such conditions, we perform camera calibration and estimate the focal length f o f the camera at the same time.
2.1. Image Coordinate System Here for each image frame we define the 3-D viewing coordinate system o'-x'y'z' which is associated with the 2-D image coordinate system O-XY as shown in figure 1. In figure 1, we represent the 3-D viewing coordinates and the 2-D image coordinates with ( x' y' z' ) and ( X Y ) respectively. We represent the 3-D viewing coordinates of the four coplanar cue points Pl " P4 with { p~' = ( x / y / z i' )t ; i = 1,2, 3, 4 }, and we represent the 2-D image coordinates of the imaged coplanar cue points, perspectively projected onto the image plane, with { Pi - ( X Y )" i = 1, 2, 3, 4 }. 2.2. Camera Calibration The problem of camera calibration is to recover geometrical transformation of 3-D world coordinates of an arbitrary point in the imaged scene into its corresponding 2-D image coordinates, from given multiple pairs of the 3-D world coordinates and their corresponding 2-D image coordinates. The camera calibration problem is concisely formulated with the homogeneous coordinate systems. Given both 4-D homogeneous world coordinates a - ( x y z 1 )' of an arbitrary point in the imaged scene and their corresponding 3-D homogeneous image coordinates b = h. ( X Y 1 )', then the foregoing transformation will be represented as the linear transformation, which is
m3 3-D world coordinate ,
Y
~
ml
X
Figure I. Coordinate systems.
system
65 defined as follows:
(x'
y
,
z
,)t
-M'(x
v
=(m I
m2
)t 1
z
m3
m4).(x
v z. 1)t
=x.m 1 +y.m 2+z.m 3+m 4
(1)
Xt
X=--7.f
,
Y= y ' . /
Z
(2)
.7
where the focal length )"is explicitly handled. Here the camera calibration problem is defined as the problem to recover the 3 x 4 matrix M and the focal length f o f equation 1 from given multiple pairs of the homogeneous world coordinates and their corresponding homogeneous image coordinates. Equation 1 means that the 3-D viewing coordinates ( x' y' z' ) are expressed as a linear combination of the three vectors { in 1 rnz in 3 }, and hence we may regard the three vectors { m I in 2 m 3 } as the basis vectors of the 3-D world coordinate system o-xyz. On the other hand, the vector m 4 means the displacement vector shifting from the origin of the 3-D viewing coordinate system to that of the 3-D world coordinate system. Here we imagine a plane quadrilateral whose four vertices are given by the four definite coplanar cue points, and we refer to the plane quadrilateral as the cue quadrilateral. As a common coordinate system to all image frames, we define the 3-D world coordinate system o-xyz whose x-y cross section contains the cue quadrilateral, that is to say, whose z-axis is normal to the cue quadrilateral. Moreover, without loss of generality, we put the origin of the 3-D world coordinate system o-xyz at one of the coplanar cue points, e.g. P l In this case, we can represent the 3-D world coordinates of the four coplanar cue points P~ - P4 with
I,-(xi
}
y, :.,),-(-,, y, o)'-o Pi-(xi
Yi
zi
Yi
0) t
"i-2,3,4
(3)
Assuming that the focal length f o f the camera is accurately estimated some way or other, which will be described in the next section, we can easily recover the 3 x 4 transformation matrix M of equation 1 from the four pairs of the 3-D world coordinates Pi - ( x; Yi 0 )' of each cue point P; and its corresponding image coordinates P ~ - ( X; Y; )'. Substituting the four coordinate pairs into equation 1, then we will reach the simultaneous equations:
, (x'i
Yi
xi/z'i
,)t zi
=N.(xi
Yi
1) t =
in,1n,2 n21
n22
n24
n31
n32
n34
"
i
/
t
Xi -
,
Yi - Yilz'i
" i - l, 2, 3, 4
(4)
where the focal length f i s implicitly included in the expression of the 3 x 3 matrix N and the matrix
66 N is related with the matrix M as follows" / mll m21
m12 m22
m24
m31
m32
m34
ml411 =
nillf n21/f
nl2/f
nl41f]
n22/f
n24/f
tt31
n32
(5)
t/34 J
The simultaneous equations of given by equation 4 are linear with respect to the nine unknown matrix components n's, and we can easily solve them. However, their solution should be expressed with one scale factor, and hence here we set the value of the matrix component n~ to one. Moreover, given the focal length f o f the camera, then we can recover the column vectors { m 1 m z m 4 }of the matrix M by applying the relation of equation 5. With regard to the column vector m 3 o f the matrix M, we should employ a vector which is normal to both the two column vectors { m 1 m z }, e.g. m3 =
{Im~l/Iml x m2]}.
(m I x m2)
(6)
Thus we can recover the 3 x 4 transformation matrix M of equation 1. 2.3. E s t i m a t i o n
of Focal Length
Once we recover the foregoing transformation matrix N of equation 4, we can estimate the relative depth z i' of each coplanar cue point Pi as follows: t
zi = m31 "xi + m32 ' Yi + m34 = n31 "xi + n32 "Yi + n34 Thus we get an estimate of the 3-D viewing coordinates Pi' - ( P. as follows"
,( x'i Yi) tZi( = Xi "z.i l l
I
Pi =
Yi "z i l l
Xi' Yi' Zi' )' o f
;
(7) each coplanar cue point
(8)
zi
The lengths of the four sides of the cue quadrilateral are assumed to be known in advance, and furthermore taking account of the fact that the ratio of lengths of two sides arbitrarily chosen out of the four sides is invariant irrespective of the definition of the 3-D coordinate system, we get the relation:
I'
'1
P2 - P l I / I P 4 - P l
= IP2
-
Pl
12/Ip4-p,I
2
=
r
(9)
Substituting equation 8 along with equation 7 into equation 9, then we will obtain the quadratic equation with respect to the focal length f. The solution is given by (10)
f = ~/(r.C-A)/(B-r.D) -
- x,. z;)
+
-
z;)
C = (X 4'2:.'4 - X l'z' 1)2+ (Y4"z4-YI'z'I)2,
,)2
67
3. DIGITAL MOVING IMAGE MIXING We have imaged the scene in our laboratory while moving a 8-mm domestic handy video camera arbitrarily by hand, and then we have applied the foregoing cue-based camera calibration method to the moving image sequence, each image frame of which is composed of 720 • 486 pixels. In the imaged scene we have put an A4 size paper on the table in the scene, and we have used the four comer points of the A4 size paper as cue four coplanar points. Moreover, we have put a book on the table as a still obstacle. We have detected the four edges of the A4 size paper with our recently presented active line-contour model [5], and identified its four comer points as their intersections, and thus we obtain estimates of the image coordinates of the four comer points.
Figure 2. Image frames chosen from the resultant mixed moving image sequence.
68 We have performed the digital image production task of mixing a synthetic 3-D CG image sequence of moving two toy cars and a moving toy robot and the real moving image sequence of our laboratory, according to the recovered estimates of the camera's motion, position and orientation. Figure 2 shows some image frames chosen fiom the resultant mixed moving image sequence. As shown in figure 2, we can hardly identify any artificial distortions in the mixed image sequence, which demonstrates that the cue-based camera calibration method works well for the foregoing digital moving image mixing task. 4. CONCLUSIONS In this paper, we present a method for performing camera calibration along with estimation of focal length of the camera accurately by using four definite coplanar points as a cue. The practical computational algorithms for the cue-based method of camera calibration are composed of simple linear algebraic operations and arithmetic operations, and hence they work so well as to provide accurate estimates of the camera's motion, position and orientation stably. Moreover, in this paper, we apply the cue-based camera calibration method to the digital moving image production task of mixing a synthetic 3-D CG image sequence and a real moving image sequence taken with a domestic video camera according to the recovered estimates of the camera's motion, position and orientation. Experimental simulations de~nonstrate that the cue-based camera calibration method works well for the digital moving image mixing task. The key to the accurate cue-based camera calibration is how to detect the feature points used as a cue in an input image highly accurately. Sub-pixel accuracy will be possibly required for the detection task. To detect the feature points with sub-pixel accuracy, in advance we should enhance spatial resolution of the image region containing the feature points. It seems that we can apply our recently presented temporal-integration resolution-enhancement method [6] to this purpose. Moreover, to complete a practical image processing algorithm for the digital moving image mixing task, in addition to the camera calibration, we should take account of many other points, that is to say, occlusion between real objects and synthetic CG objects, 3-D shape of real objects, and so on. Further studies on these points will be required. REFERENCES
1. K. Deguchi, "Image of 3-D Space : Mathematical Geometry of Computer Vision", Shoukodo Press, Tokyo, Japan, 1991. 2. C. Longuet-Higgins , "A Computer Algorithm for Reconstructing a Scene from Two Projections", Nature, .293 ( 1981) 133. 3. R. Horaud, et al., "An Analytic Solution for the Perspective 4-Point Problem", Computer Vision, Graphics, and hnage Processing, 47 (1989) 33. 4. C. J. Poelman and T. Kanade, "A Paraperspective Factorization Method for Shape and Motion Recovery", Lecture Notes in Computer Science, 801 (1994) 97. 5. Y. Nakazawa, T. Komatsu and T. Saito, "A Robust Object-Specified Active Contour Model for Tracking Smoothly Deformable Line-Features and Its Practical Application to Outdoor Moving Image Processing", IEEE 1996 International Conference on Image Processing, 17P8.13, 1996. 6. Y. Nakazawa, T. Komatsu and T. Saito, "Temporal Integration Method for Image-ProcessingBased Super-High-Resolution Image Acquisition", lEE Proc. Vis. Image Signal Process., 143 (1996) in press.
Time-Varying Image Processing and Moving Object Recognition, 4 - V. Cappellini (Ed.) 69
1997 Elsevier Science B.V. All fights reserved.
E x l ) h ) r a t i o n of t h e e n v i r o n m e n t w i t h o p t i c a l s e n s o r s m o u n t e d on a m o b i l e robot P. Weckesser, A. von Essen, G. Al)l)enzeller, R. Dillmann Institute for Real-tiine Computer Systems & Robotics Prof. Dr. U. Reinhold, Prof. Dr. R. Dillmann University of Karlsruhe, Department for Comt)uter Science I(arlsruhe, Germany The exI)loration of unknown enviroimmnt is an iInportant task for the new generation ()f mol)ile service robots. These rol)ots are supposed to operate in dynalnic and changing envir()nlnents together with human beings an(1 other static or moving objecl, s. Sensors tha,t are capable of providing tile quality of information that is require(t for the described scenario are optical sensors like digital cameras and laserscanners. In this I)al)er sensor integration and fusion for sli(:h sens()rs is described. Complementary sensor information is transformed into a coInmOn representation in order to achieve a coot)crating sensor system. 1. I n t r o d u c t i o n
In this Imper an apt)roach to fuse sensor infl)rnmtion fr(ml (:{mlplementary sensors is I)resented. 'File mobile rol)()t PlqlAMOS (figure 1) was used as an experimental testlmd. A multisensor systmn supports the vehicle witll o(lometric, sonar, visual and laserscanner information. This work is part of a large project with tile goal of lnaking r()bot navigation safer, faster, more rdiable and l~lore stalJle under changing environtnental conditions. An architecture for actiw; and ta.sk-driw'~l processing of sens()r data is presented in [10]. With this architecture it is p()ssit)le to control the sensor system acc()r(ling to envir(mmental conditions, per(:eiw;d sensor infornlation, a1)riori knowledge and the task of the robot. The system's 1)erfl)rmance is demonstrate(l for tile task of exploring an llnkn()wn environment and incrementally lmihling up a geometrical model of it.
Figure 1. PRIAMOS
70 Sensor fusion is perh)rmed by matching the local perception of a laserscanner and a camera system with a global model that is being built up incrementally. The Mahalanobis distance is used as matching criterion and a Kalman filter is used to filse matching features. A common representation inclu(ling the uncertainty and the confidence is used for all scene features. 1.1. Mobile robot navigation Navigation tasks of a mobile robot can be subdivided into three subproblems.
1. collision avoidance: this is the basic requirement for safe navigation. The problem of collision avoidance is solve(1 for dynamic environments with different kinds of sensors like sonars or laserscanners.
2. mobile robot positioning: if geometrical a priori information about the environment is available to the robot the folh)wing questions can be asked. 'Where am I?', Where am I going?' and 'How do I get there' [7]. With today's sensors these questions can be answered for static environments [6]. Though the problem is not solved in general for dynamic and changing environments.
3. exploration and environmental modelling: The problem of exploring an unknown environment was apI)roached by various groups [4,1,9] but is by far not solved. Most approaches aim at building up a 2-dimensional map of a static environment. In this paper a 3-dimensional map of the environment is built up with an integrated use of a laserscanner and a trinocular vision system. The laserscanner only provides 2dimensional information. The vision system is capable of perceiving the environment 3-dimensionally. The goal of this paper is to develop and to apply sensor fusion techniques in order to improve the system's performance for 'mobile robot positioning' and 'ext)loration of unknown environments'. The approach is able to deal with static as well as dynamic environments. On different levels of t)rocessing geometrical, topological an(l semantical mo(lels are generated (exploration) or can be used as a priori information (positioning). The system's l)erformance is demonstrated for the task of building a geometrical model of an unknown environment. 2. Obtaining 3D descriptions of the scene Ill this section the reconstruction of the 3-dimensional scene with the trinocular vision system and the laserscanner is descril)e(|. As scene featurcs linear edge segments are used. These edge segments are reI)resented by nlidI)oint, direction-vector and half-length. The uncertainty of the segments is rel)resented by a covariance matrix [3]. The xz-I)lane is the ground-plan of the coordinate system and the y-axis represents the h(:ight. As the laserscanner only provides 2-(limensional data the y-coordinate is always equal to the height in which the sensor is mounted on the robot.
71
2.1. 3 D r e c o n s t r u c t i o n f r o m t r i n o c u l a r s t e r e o
The process to reconstruct scene features from camera images is relatively complex but it is possible to derive a 3-(limensional (lescription of the scene. The first step of stercoima,ging is the calibration of the, cameras. In [12] a photogrammetric aI)t)roach to highly accurate camera calibration of zo()m lenses is developed. The result of the calil)ration is the matrix MDLT which (lescril)es the transformation from scene- to image-coordinates for a camera. In homogeneous coor(linates this tra, nsformation is given by ( 'W"'U?' ) "W~'U~
i
ix/ y
"
-- MDLT
wi
Z
(1) "
1
In the presented system linear edge segments which are extracted fi'om the camera images in real-time are used as image and scene features. It is possible to reconstruct scene features if corresponding image features in at least two camera images are known. This means that the stereo correspondence 1)rol)lenl has to be solved. In [8] a trinocular stereo-matching algorithm using the eI)ipolar c()nstraint combined with a local n()rmalized cross-correlation technique has been developed. The stereo matching algorithm provides c()rresl)onding image I)oints (u i, v i) in the camera images. For the presented system it was ext)erimentally proved that the uncertainty of the matches can generally be estimate(t to be below one pixel. F()r the stereo-reconstruction of a scene point the following overconstrained linear system has to be solved: A
y
- b
~
(,)
p-
y
z
-(ATA)-'ATb.
(2)
z
In [5] it is shown that the uncertainty for the reconstruction of a scene point can be written in a first order approximation by a covariance matrix E p -- J E,,,,,
jT
with
J
0((ATA)-'ATb)
0(',,,~, v')
(3)
Are, asonable estimation for E~,,~ is given by Eu,,, = 1. In order to reconstruct a line segment the endpoints of the lille Pl and P2 are reconstructed. The equations 4 to 8 (lescril)e the representation of a line segment by midpoint, normalized dire, ctiou-vector and halflength and tile correspon(ling covariance matrices for the representation of the m~certainty. For a minimal reI)resentation there is no uncertainty ret)resente(1 for the halflength. m -
midpoint
(4)
r -
m-v2 [[PI --O2[[
normalized direction
(5)
l =
[[Pl-P2112
ha lflength
(6)
E m ---
Epl +Ep:~.4
(:()variance of midI)oint
(7)
Er -
r..~ +r.p22 [[Pl -p2ll
covariance of direction
(8)
-
-
p~+p22
Tlle state vector for a line segment in rol)ot coordinates is given by k r - (m, r, l) T
72 3D descriptions obtained by a laser scanner Figure 2 shows a laserscan that is ac(luired in a corridor environment. The sensor (lata 1)r()vi(led by the laserscanner are 2-(limensional (ground plan) so y-coordinate is always 0. In order to show the quality of the laserscan the cad-model of this environment is overlayed in grey lines. 2.2.
_:__=_.-_ .....................................~a~t.a1~,m.=~r
_-::-
:-=_-
,-.1
_ _ ..............I
Figur(; 2. raw data from laserscan
Figure 3.
.....
example for edge extraction by
iterative end point fit combined with leastsquare approximation
An experimental evaluation for tile accuracy of the laserscanner measurements was carried out with the result that within a distance of 10 meters the variance of the distance measurement and tile variance, 1)erp(',ndicular to the measuring direction can 1)e estimated to be
a~ - (2cm)2 - 4cm 2
'
a~ - d2 tan2 (0"25~ 2
"
(9)
From the scanner's polar ret)resentation of the measurements a cartesian rcpresentation is computed which results in the following covariance matrix for the uncertainty of a single scan point of the laserscanner.
((~) + alcos2((~,) (a'Z_L- a~)cos(t~)sin(c~)
)
The next step of processing is the extraction of linear edge segments fronl the laserscan. This is done by using the iterative end-point ,fit algorithm for the determination of points belonging to a line segment. A least square solution is applied to compute a symbolic representation as defined 1)y the equations 4 to 8 (see also [11]). This is displayed in figure 3.
73
2.3. T r a n s f o r m a t i o n t o w o r l d c o o r d i n a t e s The line segments are so far rel)reseute(l in robot coordinates. The state vector of the r()b()t is given by x~ - (x,., z,., qS) 't' For a c o m m o n robot indeI)endent ret)resenta.ti()u the t.ransformation to world coor(linates is necessary. This transformation is given t)y the t()llowing rotation and translatiou:
R -
0 - sin(C)
1 0 0 cos(qS)
,
T -
() zr
.
(1())
The transforination of the state vector in rol)ot coordinates k r = (m, r, l) t() the state v(,ct()r in world coordinates k w 1)e('.()lu(;s k w = ( R m + T, R r, 1). The 1)r()t)agatioIl of the uncertainty of a r a n d o m vector x with covariance matrix Ex lln(l(;r the transformation y = f(x) is in a first order aI)proximation given by ]~-] y
z
Of(x) Ox
~x X--~
Of(x)"' Ox
(11) X-=X
\Vitll this equation the ulw(;rtainty ()f the state vector iu worl(t coor(tinates [5] 1)eCOllW.s
y]~,,w __ I t Y],rn R T -~-
-0~-I11
~-]R
-0-~ Ill
-4- Y]T
(12)
and r. w - R ~ r R " ' +
r
~
(oR,), ,, -bT~
(13)
with E R a Il(1 ET beeing the llu(:('rl,aiuties in R and T. 3. E x p l o r a t i o n
II~ ()r(ler to explore an ellvir()llillent the rol)()t is 1)rovi(le(t with a topologicaJ m()(lel and a, (:('rtaiu)uission ((lirectiou and (listau(~(, t() travel) is sI)ecified. The geomet, ri('al worl(t m()(lel is 1)lfilt uI) incremeutally. The lo(,al 1)('.rcet)tion is matched with the gl()l)al m()(lel a u(1, if 1)()ssit)le, fused acc()r(liug t() ttl(', f()ll()willg section. 3.1. F u s i o n o f s y m b o l i c e d g e s e g m e n t s Iu this work linear edge segnlents a r(' rei)reseuted t)y midI)oint, nornlalized (lire('ti()n vcct.()l a u(l llalltength 1)e(:a.us('. this reI)r(,seutati()xl is advantageous for the fusiou of segments. 'l'his lIl(~/,IIS ~tll edge seguw.ut is (l(;Iine(1 1)y a state vector k = (71~.:~,mu, 7/~.z,r:,;, r.~, r~, l) '1' aTl(l l,h(' uucertainty is givelt 1)y the (:()variau(:r lua.trix Ek. I~l ()r(ler to lind ('.orresi)()u(ling s(:eue f(,atllres (nearest neighbor matching) in tll(; h)ca] 1)cr(:Cl)ti()u ;tnd the gl()t)al mo(lel the Matlalan()t)is distance is apt)lied. The Mahalall()l)is (lisl:al~(:(' is a distance crit,eri()u f()r tw() stat, e ve(:tors normalized by the sllm ()f their (:()Val'iance lnatIiC('s. The s(tlmre(l Nlahalall()l)is (lista, n(:e is given by
Sec
0
Figure 1. Acquired signal ss(t)
50
Sec
)
100
Figure 2. Filtered signal s(t)
The average-slope method has been utilised for the identification of trend parameters. Windows of various size were employed to avoid side-effects. The random nA(t) noise component is also cancelled out after identification of main impulses. Despite nx(t) shows a white spectrum over a long period time (more than 1 hour), it mainly represents a non-white noise source over a shorter period of time (few minutes), considered as appropriate for the required analysis. The improvement in the signal due to the filter is proved by the spectral analysis conducted over the filtered data, as compared with spectral analysis on the original acquired data, reported in Fig.3. 80
~~~OI .
.
.
.
.
dBm 501
2G 0
30
,
L
i
i
:
Frequency, Hz
Figure 3. Power Spectral Density of acquired data
6O
%-
Frequency, Hz
9
> 6O
Figure 4. Power Spectral Density after the random trend has been deleted
It is clear that, by cancelling the SQUID acquisition noise, a better estimation of low frequencies is achieved, since exponential components due to the random trend have been deleted as it can be noticed by comparing Figs. 3 and 4.
200 In addition, cancelling nA(t) impulses allows a better resolution of peaks present in the estimated spectral density function (Fig.5).
Figure 5. Power Spectral Density of filtered data
Figure 6. Daily evolution of the first Schumann's peak
The correctness and usefulness of such a filtering process is also proved by the conformity of background noise nB(t) with the theoretical model [4]. In fact Schumann's resonance peaks, due to the resonant cavity formed by the ionosphere and the Earth, are sharp and clearly visible in the spectrum at frequencies around 8, 14, 21 Hz and higher. This was achieved by analysing few minutes interval, a much shorter one than that required by formerly proposed methods [6]. 4. DATA FUSION The so obtained filtered data So(t) are then used as a reference input for the dynamic calibration of coils. These instruments can be represented by a Transfer Function (called He(f)) and its module can be estimated by
_
so(f)
(3)
The spectnun of the filtered SQUID signal So(t), So(f), plays the role of reference input signal and So(f) is the spectrum of coil acquired signal se(t). After identification of coil frequency characteristics by applying equation (3), and after phase identification, the frequency distortion introduced by coils is eliminated by an inverse filter. Finally, such a compensated signal represents an additional data source available for data fusion and a better understanding of the signal's dynamic behaviour at low frequencies.
201 5. RESULTS By the method presented in the paper, a precise analysis of the ELF signal and its temporal evolution can be performed. The dynamic behaviour of Schumann's resonance peaks can be observed in a time-frequency space like that in Fig. 6, where the daily evolution of the 8 Hz resonance is shown. By analysing the signal corresponding to the time interval going from 4 a.m. to 5 p.m. (local hour) a power increase due to the sun's electromagnetic emissions [7] can be visualised thanks to the growth in width and height of the peak corresponding to the resonance. In dealing with target detection we can exploit the periodical characteristic due to the orbital movement of the target system. If we identify different portions of x(t), corresponding to the closest points of approach of the TSS to the ground-based receiver we can cut and fold them in order to increase the detection probability. In fact, it is possible to consider these portions of x(t) to be different detection opportunities of one single process in such a way that an increase in SNR and detection probability will occur.
6. CONCLUSIONS The proposed method allows to process the acquired signal in an adaptive and automatic way. The signal-to-noise ratio improvement estimated by means of synthetic signals is about 15-29 dB. The proposed pre-processing method has been applied to signals recorded by different acquisition set-up in different signal-to-noise conditions and has always proved to achieve good results. REFERENCES
[1] [2] [3] [4]
[s] [6]
[7]
C.B. Powers, C. Shea, T. McMahan " The first mission of the tethered satellite system "ed. Essex Corporation, 1992, Huntsville, Alabama J.Clarke, "Gli SQUID",Le scienze, No 314, Ottobre 1994 S.A.Kassam, "Signal detection in Non-Gaussian noise ",ed. Springer Verlag, USA, 1988 J.E.Evans, A.S.Griffiths, "Design of a Sanguine noise processor based upon world-wide Extremely Low Frequency (ELF) recordings ",IEEE Transactions on Communications, Vol. Com-22, No. 4, p. 528-539, April 1974 E & G S.V.Czarnecki, J.B.Thomas, "Nearly optimal detection of signal in non-Gaussian noise",Department of Electrical Engineering and computer science, Princeton, 1994 G.Tacconi, S.Dellepiane, L.Minna, C.Ottonello, S.Pagnan, "Campaigns of ground listening to the e.m. emission expectedfrom spaceborne electrodynamic tethered system ", Conference paper, 4th Int. Conf. Tethers, Washington, April 1995 Ya.L.Al'pert, "The near-earth and interplanetary plasma",ed. Cambridge University Press, UK, 1983
This Page Intentionally Left Blank
F DIGITAL PROCESSING OF BIOMEDICAL IMAGES
This Page Intentionally Left Blank
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All rights reserved.
205
A Simple Algorithm for Automatic Alignment of Ocular Fundus Images L. Ballerini a and G. Coppini b and G. Giacomelli r and G. Valli a ~Department of Electronic Engineering, University of Florence, Italy bCNR- Institute of Clinical Physiology, Pisa, Italy r
di Clinica Oculistica, University of Florence, Italy
This paper describes an automatic alignment algorithm for registration of ocular fundus images. In order to enhance vessel structures, we used a spatially oriented bank of filters designed to match the properties of the objects of interest. To evaluate interframe misalignment we adopted a fast cross-correlation algorithm. The performances of the method have been estimated by simulating shifts between image pairs and by using a cross-validation approach. We also propose temporal integration techniques of image sequences so as to compute enhanced pictures of the overall capillary network. 1. I N T R O D U C T I O N Retinal vessels are the only vascular network directly observable from the outside of our body. Many systemic diseases, such as hypertension, arteriosclerosis, and diabetes mellitus, are studied using retinal vessels as index of diagnostic staging and therapeutic efficacy [1,2]. The Scanning Laser Ophthalmoscope (SLO) is an instrument that allows observation of a retinal image on a video monitor [3]. Digital video fundus angiography has several advantages over conventional angiography, including real time access to retinal images, the possibility of computer processing, and increased light sensitivity that makes indocyanine green angiograpy possible [4]. The use of SLO for digital video fundus angiography may improve spatial resolution with respect to conventional photography and provides much more temporal information than conventional angiography [4,5]. This explains the interest of several research groups for SLO imaging. Some authors have studied retinal circulation using fluorescein angiography [3,4,6]. Wolf et al.[6] present quantitative measurements of blood velocities in retinal capillaries and propose a method to evaluate vessel morphology. Tanaka et al.[3] observed the transit of numerous fluorescent dots in the perifoveal capillaries, and used them to identify direction and velocity of blood flow in the retinal capillaries. Nasemann et al.[7] demonstre new diagnostic possibilities in fluorescein angiography obtained with SLO such as the computation of circulation times and the imaging of erythrocytes and leucocites. Van de Velte et al.[8] describe some applications of SLO to microperimetry that attempt to correlate anatomical features or pathologic findings in the fundus with retinal function. Rehkopf et al.[9] developed a
206 method based on indicator diluition theory and image processing technology for estimating total retinal arteriovenous circulation time and the transit time in individual arteries. Alignment of temporal sequences of retinal images is crucial for quantitative analysis. In the analysis of fundus images previous investigators have used several different registration methods, which can be classified into two broad groups: interactive and automated. Automated registration methods may be divided into local and global methods. Local methods use a subset of the image information by extracting distinctive features; registration is performed only on the extracted features. Global methods use all pixel values in order to determine a single best set of transformation parameters for a given image pair [10]. Automated image registration methods that use local information commonly extract the ocular blood vessels and/or their crossing. For example Yu et al.[ll] used the branching points of retinal vessels as registration templates. Sequential fundus image alignment is done by using the sum of the absolute values of the differences method. Hart and Goldbaum [12] describe a method for identifying control points automatically using the branching and crossing points in the retinal vessel network. They propose to use a matched filter to exctact blood vessel segments. Registration is performed with an affine transformation that is computed using the control points. Cideciyan [10] describe a global registration method based on the cross-correlation of triple invariant image descriptors. One of such descriptors is the log-polar transform of the Fourier magnitude, which removes the effects of translation and converts rotation and uniform scaling into independent shifts according to orthogonal directions. Our approach is a global registration method based on image cross-correlation following spatially oriented filtering. 2. F U N D U S I M A G E S Retinal images were taken by a SLO, with a frequency of 25 frames per second following the injection of a bolus of fiuorescein. These images were digitized into 256 • 256 pixel matrices with 256 gray levels per pixel. The retinal region is approximatly 20 • 20 degrees. In a fundus image (see Figure 1) the darker region is the macula and the lighter curvilinear structures are the retinal blood vessels; they branch, cross and become smaller the farther they are traced from the optic nerve. The optic nerve stands out from the retina as a bright, vertically-oval disk with a sharpe edge. In theory the complete macular network of capillaries can be observed. 3. A L I G N M E N T M E T H O D The misalignment is due to changes in the acquisition geometry which may occur in a few milliseconds in the case of sequential frames of a fluorescein (or indocyanin green) angiogram. Misalignment is due both to eye movement and SLO equipment movement. As the patient head is kept fixed during SLO acquisition, we consider constant scaling and no rotation, so we can assume only translatory movement between two subsequentail frames. Images we deal with are projections of a spherical surface, but it can be easily shown that geometrical distortion has a negligible effect.
207
Reference image
Extractsub-imageI
Other images Extract
1
[ Filtersub-image ]
L
Binarizesub-image[
sub-image I
1
Filter
sub-image I
1
Binarizesub-imageI
Computecross-correlation
Figure 1. SLO image of ocular fundus: the darker region is the macula and the lighter structures are the retinal blood vessels.
Figure 2. Flow-chart of our automatic image alignment algorithm.
3.1. Algorithm description We have developed a procedure (summarized in Figure 2) based on the automatic extraction of a vascular feature map amenable to a binary image representation. A simple global threshold is not adeguate to extract the blood vessels from the retinal background because of noise, background variability, the low and space-varying contrast of vessels. Thus we resorted to using spatial filtering to enhance and detect vessel structure. Filtered fundus images were segmented by a trainable threshold unit. The cross-correlation was used as index of similarity between binarized images to compute the needed realignment shift.
3.2. Filtering technique Our filtering technique is based on the optical and spatial properties of the objects to be recognized. We can observe that blood vessels have typical properties such as small curvature and they appear lighter relative to other retinal surfaces. The two edges of a vessel always run parallel to each other; such objects may be represented by piecewise linearly directed segments of finite width. On this ground, we studied two different kinds of filters. In the first case, source images were filtered by a Laplacian of Gaussian (LOG) kernel: 1
(
2
x2+y2) x2+~,2
(1)
where a is the standard deviation. The half-width of the central lobe of LoG(x,y) is w = 2x/~cr and can be adopted as measure of the filter scale [13]. Despite the well known propertiers of LoG filter [14], we observed that, in our case, it tends to produce noisy outputs unless high a's are used. However, in this case vessels
208 are strongly blurred. Thus, we used a different method based on the observation that the grey-level profile of the cross section of a blood vessel is well approximated by a Gaussian shaped curve. Consequently, we used oriented matched filters [15] which model the shape of the intensity profile of the blood vessels by a Gaussian bar:
K(x, y)
= exp(-x2/2o 2)
for
L/2
l Y I-
_ 0 0_ 3 always. Therefore, the shape recognition stage is not critical: objects that are not markers are discarded by simply thresholding S. Further experiments confirmed that the position / of the marker with respect to the optical axis does ._~~ \,--~-~m\ o /.I.."//, not affect the recognition performance. This fact expedites the task of verifying the results achieved ~o~ F" " by the algorithm concerning the estimation of slant ~= .... "-. ./ ./ / and tilt angles and distance, since the marker can be t ,,. ,,.," / / 0.1 l ' " . . . . . . . . . "" " - " " Y"" placed at the same height as the optical axis of the camera, without losing generality. With this ol assumption the tilt angle is 0 ~ and the slant angle 50 60 70 90 110 1 0 1 0 2 0 290 can be directly obtained by comparing the apparent marker-camera distance (cm) length in the image plane of horizontal and vertical segments of known real length. The apparent length Figure 3 - Standard deviation in slant angle estimation as a function of camera-marker of vertical segments yields a prompt estimate of the distance for several acquisition angles. distance of the objects from the imaging point. Comparison of the results obtained with the procedure described in [7], and with those deriving from the simplifying assumptions of Eq.(2) allows the performance of the proposed recognition procedure to be evaluated with respect to two possible error sources: the first due to the discrete nature of the images used to recover geometry information; the second to be charged to misfunctioning of the recognition procedure. Throughout the case study, the two different estimations have always been in accordance with a maximum difference of 3 ~ for slant. The recognition algorithm always under-estimates the slant; this is a systematic effect which occurs in all tests, probably due to the quadratic approximation of the variations of angles and lengths with the slant cy. Also the tilt angle was always correctly estimated as 0 ~ In Figure 4 the slant estimated for various ~.t (dq.) 67.5 . . . . . ,, , ~ acquisition angles is plotted versus the distance | ~ . . . . .;,. . . . . ~. . . . . _ ~ ~ . . ~ .......... between camera and marker. Estimation errors ~.5' .....:......:..... ......- .....: .....i.......... -L-...., .....:......:..... are independent of the distance, while there is a certain dependency on the acquisition angle, , 0 1 ~ . ~ . . . . ~ . . . . . . . . . . ~ . . . . . . . . . . , . . . . . ~ . . . . . ~. . . . . . ~. . . . which is roughly the same for each distance. A further remark concerns the value of the 7.s ..... "..... slant used to construct the object C necessary for the application of the method. Pizlo and 50 70 90 110 130 150 170 190 210 230 2SO 270 290 d ~ (cm) Rosenfeld [7] suggest using a value of r e f e r e n c e slant (Jc = 65~ We investigated this point by examining the performance of the algorithm Figure 4 - Estimated slant angle as a function of the when this value is varied from 3 7 . 5 ~ to 70 ~. camera-marker distance for several acquisition angles. While marker recognition is still correct, the error affecting the estimation of slant and tilt angles becomes larger. Tilt angle results to be in some cases x = -5 ~ The trend of slant error to the reference angle is shown in Figure 5. Indeed, crc = 6 5 ~ is the best value, except for an acquisition angle of 75 ~ representing the limit case [7], due to perspective bindings. ,.-
[
03
._o
N
O0
45
30
15
'/":/
~ l .
~.5:
.....
: ...........
: .....
: .....
22.5
.....
:. . . . . .
:.....
" .....
" ...........
" .....
i .....
! .....
9
..... i .... ---.---4-
: ...........
i .....
~. . . . . .
: .....
? .....
i......
i....
:. . . . . . . . . . .
" .....
:" . . . . .
:......
:....
i .....
i .....
- - i
" .....
261 For what concerns the accuracy in distance estimation, we have found that the maximum error ranges from 0.5 to 2.5 cm at distances of 50 and 290 cm, respectively, and lower than i% in most of cases, as shown in Figure 6. Since the percentage error is somewhat independent of the distance, the absolute error is approximately proportional to the distance. Note that at the distance of 170 cm, and for angles ranging between 15 ~ to 45 ~ the error is zero, since these measures have been taken as references to calibrate the camera. 24
slant error (deg.) .
1 .
.
~0.6o ~- 0 . 4 r 0,2-
20 16
0-
12
/ ,.
8
~-o.4 -o,6;-0,8 -
0
O_
-1
-
.4
-1,2 -1.4 -
-8
-1.6 -1.8 -
15
3O
45
6O
75
/:
W
\
50
lqr--~
/ /,; I/\/
\
//,,/
~'-0,2 -
4,
-12 0
/\
0.8-
.
,...,?"
-~,
...~.........
z. "-.,.-\/ \
\
/
" 1,0
t
o
9
,5
60 60
70 90 110 marker-camera
130 170 210 290 distance (cm)
nominal angle (deg.)
F i g u r e 5 - Slant error versus acquisition angle, at a distance D=110 cm for different reference slant angles 6c-
F i g u r e 6 - Percentage error in distance estimation varying with distance D (cm) and acquisition angle (degrees).
REFERENCES
1. J. W. Courtney, J. K. Aggarwald, "Robot guidance using computer vision", Pattern Recognition, Vol. 17, pp. 585-592 (1984). 2. S.Y. Chen, W. H. Tsai, "Determination of Robot Locations by Common Object Shapes", IEEE Trans. Robotics Automat., Vol. 7(1), pp. 149-156 (1991). 3. M. R. Kabuka, A. E. Arenas, "Position Verification of a Mobile Robot Using Standard Patterns", IEEE Trans. Robotics Automat., Vol. 3(6), pp. 505-516 (1987). 4. D.H. Ballard, C. M. Brown, Computer Vision, Englewood Cliffs, NJ: Prentice Hall (1982). 5. R. M. Haralick, L. G. Shapiro, Computer and Robot Vision, Vol. I, Reading, MA: Addison-Wesley (1992). 6. M. F. Augusteijn, C. R. Dyer, "Recognition and recovery of the three-dimensional orientation of planar point patterns, CVGIP, Vol. 36, pp. 76-99 (1986). 7. Z. Pizlo, A. Rosenfeld, "Recognition of planar shapes from perspective images using contour-based invariants", CVIP: Image Understanding, Vol. 56(3), pp. 330-350 (1992). 8. T. S. Huang, A. N. Netravali, "Motion and structure from feature correspondences", Proc. IEEE, Vol. 82(2), pp. 252-268 (1994). 9. C. H. Teh, R. T. Chin, "On the detection of dominant points on digital curves", IEEE Trans. Pattern Anal. Machine Intell., Vol. 11(8), pp. 859-872 (1989). 10. L. Alparone, S. Baronti, A. Casini, "A novel approach to the suppression of false contours originated from Laplacian-of-Gaussian zero-crossings," Proc. ICIP, I, pp. 825-828 (1996).
262
Time-Varying Image Processing and Moving Object Recognition, 4 - V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All fights reserved.
Markov random field image motion estimation using mean field theory A. Chimienti, R. Picco, M. Vivalda Television Study Centre of the National Research Council Strada delle Cacce 91, 10135 - Torino - Italy
Abstract The estimation of a dense displacement field from image sequences is an ill-posed problem because the data supply insufficient information, so constraints are needed to obtain a unique solution. The main advantages of Markov random field modelling of the displacement field are its capacity to regularize the motion vector field, smoothing it while preserving motion discontinuities, and its power to easily integrate information derived from gradient based and feature based motion constraints, obtained by the introduction of other fields in the model. The configuration of the fields are computed by a deterministic iterative scheme derived from the mean field theory and saddle point approximation. The algorithm is well suited for a multigrid approach to obtain more regular results and to speed up the convergence of the iterations.
1. Markov random fields A random field is a family of random variables defined on a lattice, a regular set of sites with a neighbourhood system. In this work the nearest-neighbour system was considered, so the neighbourhood of the position (i,j) is the set ~(i,j) = { (i,j - 1), (i,j + 1), (i - 1,j), (i + 1,j)} . A subset of sites formed or by only one site or by more sites each of which is neighbour of all the others is called a clique. A random field is said a Markov random field [ 1] if each of its variables defined over a site is influenced directly only by variables defined on neighbour sites, that is p(xn = anlxm = a m , V m # n) = p(xn = anlXm = a m , m ~ ~n) 9 The Markov property causes the probability distribution of the field to be a Gibbs distribution, p = ~1 e -flu where fl is a constant, Z, called partition function, is the probability distribution normalization constant and U, called potential function, is a sum of basic potential functions defined on cliques.
2. Mean field approximation of the optimal configuration of variables In the problems expressed by Markov random fields, the solution lies in computing the configuration of variables which maximizes the probability distribution. The classical algorithm for this kind of problems is the simulated annealing [ 1], a stochastic procedure which converges statistically to the optimal solution, but presents a great computational complexity. To overcome this problem other techniques have been developed; one of these is the iterative conditional mode [2], a deterministic procedure that converges more quickly but can also be stuck in a suboptimal solution, a local minimum of the potential U. Another technique, known as mean field approximation [3, 4], solves the problem in a different way: instead of computing the optimal solution, which is the configuration k that mimimizes the potential U, the mean field is computed
263 x 2 e1 -~U(x) ,
~=
{x} where the sum is over all the values taken by each variable defined on each site. The mean field 2is an approximation of J~, in fact lim ~ = J~ ; in pratice ~is a good approxi-
fl--.
+oo
mation of J~ for sufficiently high values offl.
3. Model for the motion vector field determination The computation of the motion vector field from image sequences is an ill-posed problem because images supply insufficient information. Some constraints are so needed to regularize the solution; the classical constraint [5] is the smoothing of the displacement field, but it produces poor results at objects boundaries. With the Markov random field framework [2, 3] it is possible to handle motion discontinuities, smoothing the displacement field inside the objects while preserving motion boundaries, by the introduction of the motion discontinuity fields in the model; furthermore this model can easily integrate information derived from gradient based motion constraints, valid in the regular regions of the images, with other knowledge brought by feature based constraints, valid in the edges of the images. The motion vectors and the other variables introduced in the model to handle motion discontinuities and the uncovered areas of the image are considered Markov random fields. The displacement field and the uncovered background field are defined on the lattice of the positions of the pixels, while the motion discontinuities are defined on lattices formed by intermediate positions. The fields of unknown variables are: d = (a,b), the displacement field; h and v, the so called line processes, pointing out the motion discontinuities of the d field in horizontal and vertical directions respectively; they are binary fields, but for reasons that will be explained later they are considered real fields taking values in the interval [0, 1]; s, a field that shows the uncovered areas of the image, the points for which there is not a correspondence with the previous image; it is a binary field too, but is considered real as h and v. The fields of known variables, called observations, are: x t- 1 and x t, the two images at time t - 1 and t ; f t - 1 = (V__~t-l, V_ty-1),the gradientof the i m a g e a t t i m e t _ 1; d, a field defined only on the edges of the image x t, formed by the projection of the displacement d along the direction normal to the edge, computed in an image preprocessing phase. The probability distribution of all the fields is a Gibbs distribution, p ( a , b , h , v,s) = 21e -flU(a,b,h,v,s) with the potential U is so defined
U = 2{~f.. tj
fij
(1-sij)(1-~ij)+ +
(l -
+
The potential
.. __ (xt
F j
(1-hij)+D vij (1-vij)]+
2hrelij Hij ~,ij+ l~d[D h
ij
X(_I
t-aijj-bij
-
)2
causes the minimization of the displaced difference between the two images x t and x t/
1in the
regular areas and creates un uncovered area indicator (s ij = 1 ] in a site where the displaced dif\ /
264 ference is greater then the threshold T, a parameter that balances Fij, the price paid to set sq = 1. This potential is not applied in those sites of the image, the luminance edges, identified by the parameter binary field Yij = 1. In these sites the second potential is enabled, Hij = (aijnxij+bijnYij-
-dij) 2
which causes the minimization of the difference between the projection of the motion vector
(aij•bij)•n••the•nit•e••••(nxij•n••j)•••ma•t•theedgeandthep•e••••s••••••••a•edp••je•tion d-ij ; this potential is weighted by relij, a parameter taking value in the interval [0, 1] that indicates the reliability of the estimated dij. The potential
Dh..tj = 1 - 2e
-
,
where fld is a scalar, and its vertical companion D.V. smooth the d field in absence of motion distj - - 0 and vii = 0 respectively) and set motion discontinuities when the poten-
continuities
(hij
tials values are high. These potentials take values in the interval [ - 1, 1], so they are self-balanced. The horizontal potential Eq.= 1 IJ
2
ij
'
t - lj
where e is a small value that prevents E.h. to go to infinity, and the vertical one E~. inhibit the cretJ tJ ation of motion discontinuities between almost equal intensity pixels. They occur in fact at object boundaries, the edges of an image. The horizontal potential G.h. is so defined tJ
Gh.. = 1 - 2e-fl'(s,,-s,-,,)2 lJ
;
for this potential and the vertical one G v. the same considerations made for D h. and D v. hold; as IJ lJ lJ only difference they smooth the s field. The weights/~h a n d / t v. ,J inhibit (if positive) or excite (if negative) the creation of the motion discontinuities everywhere; in this work they are set to zero but they are considered for their usefulness in the analytic approximation that follows. The scalars 2f, 2 h , 2 d , a/and2s are weights for the related potentials. The introduction of the potential Hij, which offers a complementary information regarding to
Fq ' is derived from [2]"' the other potentials are taken from [3], but G.h.and GV.have been moditJ tJ fled because s is now considered a real field and other potentials that cause the self-interaction of h and v have been eliminated to make possible the analytic computations present in the mean field approximation.
4. Edge matching The optimal configuration of the motion vector field is the one which minimizes the displaced difference between the two images x t and x t- 1. The gradient ~ t - 1 is used to compute the motion field. Given an estimate of the vector dij, the new estimate is reached moving in the same direction of the gradient in the position (i, j) - dij or in the opposite one, depending on the sign of
265 the image displaced difference. This procedure is generally correct, but it suffers for two drawbacks. The first one, known as the aperture problem, is the updating of dij in the direction of the gradient that causes the determination of the displacement vector projection along this direction instead of the right vector. Furthermore the choice of the direction based on the sign of the displaced difference may cause to move in the opposite way of the actual one in the luminance peaks or dips of the images. The first disadvantage is an intrinsic limit of the displacement field computation, which is an ill-posed problem, and is overcome, at least partly, by the regularization constraint that considers the co-operations of adiacent positions, the second one however does not get sufficient benefit from this procedure. Since this problem occurs at the luminance edges of the images, in points that can be put in correspondence in two consecutive images because they are insensitive enough to the noise, it is necessary to find these correspondences. Two discontinuity maps, one for each image, are created, formed by the points whose gradient norm is higher than a suitable threshold. The positions of these points are intermediate with respect to those of the image pixels. For each of these points the direction of the gradient is quantized and stored; eight directions have been chosen, the four of the two Cartesian axes and the four of the two bisectors of the main quadrants. For each point (i, j) of the map of x t, the correspondent one is searched in an area of x t - 1 centered in (i, j) that belongs to the map of x t - 1 and is characterized by the same gradient direction. At this point the reliability of the correspondence is evaluated, computing the energy of the difference between a block of the image x t near (i, j) and the block of x t - 1 positioned in the same way respect to the correspondent point of (i, j). To handle motion discontinuities, for each point of the map of x t two blocks are considered, set in opposite sites respect to (i, j) in the direction of the gradient, because areas on opposite sites respect to an edge may have different motion. In this way the image points near (i, j) are characterized by two motion vectors, perhaps different. Each vector is the more reliable one among all the candidates. If the reliability is higher than a suitable threshold, the edge match is considered valid and for the image points near (i, j) the projection -dij of the motion vector along the gradient direction is stored. In these points (n, m) the potential Humis enabled, as said before, instead of F n m which is unreliable there. To improve the precision with which dij is known, it is possible to consider displacement taking real values, measured in distances between pixels.
5. Optimal configuration computation For the line processes h and v, the mean field approximation is used. The partition function Z is the normalization constant of the probability distribution, so the following equation holds
Z
-- Z Z Z Z Z e-fl U(a,b,s,h,v); {a} {b} Is} {hi
Iv}
then h ij can be expressed as 10lnZ
-
hu =
.
~ o~ h
tj
the equation above shows the analytic usefulness of the weight/t,h. tj even if it is set to zero, as said before. Therefore the partition function must be calculated; the two sums Z ~ puted because h and v are non interactive discrete fields and so Z becomes
can be com-
{h } Iv}
Z = Z Z ~ e -flV(a'b's) , {a} {b} {s}
where V is a suitable potential; the three sums Z Z ~ {a} {b} Is}
are not computable analytically, so an
266 approximation is made, called saddle point approximation: each sum is replaced by the product of a constant and the contribution brought to it by the saddle point (perhaps the minimum point) of the potential V
Z = ~ ~ ~ e-#
V(a,b,s)~'k
e-~ v(?,,~,~),
{a} {b} {s}
where k is a scalar and ~, b and ~ satisfy the equations
oV - 0 , c9~.. tj dV - 0 , Obij ix
OV a?~ij
o
vij.
Then the mean value h~j becomes
this equation shows the reason why h and v are considered real fields. The saddle point is assumed as an approximation of the optimal configuration for a, b and s fields. It leads to the iterative scheme 1 0V , b!n + 1) __ b!n.) 1 0 V , s!n. + 1) = s!n.) 1 OV a!n + 1) __ a!n)
tj
tj ka da ij tj tj k b db ij tj lj ks OS ij ' where ka ,k b and ks are suitable scalars. The values hij and v~j are updated after each iteration m
of the previous equation system. A detailed description of the algorithm can be found in [6]. i
Implementation improvements and results 0 V ~0 V and Osij OV the displaced difference between In the derivatives -~/j,
Xt
and
X t-
1 and the ele-
ments of the gradient ~ t - 1 appear. These variables in the edges of the images may take values greater by one or two magnitude orders than the values assumed in the regular areas, and this causes poor motion estimation and the creation of wrong motion discontinuities. So, to limit their dynamic range, the components of the gradient are replaced by their signs and the displaced difference of the images is clipped by an exponential-like compression function. This procedure improves greatly the computation of the motion vectors and avoids the creation of wrong motion discontinuities and uncovered area indicators. The algorithm is well adapted for a multigrid approach useful to obtain a more regular displacement field in images where wide motion is present. The image is split in levels: level 0 corresponds to the original image, each successive level corresponds to an image obtained from the one at previous level by low pass filtering and subsampling. The algorithm is applied first to the images at the coarsest level, then the displacement field obtained, appropriately adapted, constitutes the starting configuration for the iterations at the finer level. This strategy gives more regular results and speeds up the convergence of the algorithm. To improve the uncovered areas handling it is possible to make a little change in the edge matching procedure, useful especially in synthetic images. The edge matching algorithm gives two motion vectors for each edge luminance point, each with its own reliability; if one is greater than
267 the other, the lower one is no longer considered valid and uncovered area indicators with motion discontinuities are set in a suitable region near the point, because this region could belong to an uncovered area. If this hypothesis is wrong, the uncovered area indicator and the motion discontinuities will disappear during the iterations without consequences for the final configurations. The algorithm was tested on a subsampled version of the sequence Mobile & Calendar, rich of details and complex motion, on a subsampled version of the sequence Brazil, of medium complexity, on the sequence Miss America, of low complexity and on a synthetic sequence formed by a square moving with known uniform motion on a steady background. Two different criteria have been used to evaluate the correctness of the computed motion fields. For the real sequences, since the true motion field is unknown, it has been considered the difference between the actual image and the displaced previous one. The results are compared with the ones obtained by the standard technique used for image difference energy minimization, the block matching, with 8 • 8 pixels wide blocks and with precision of 0.5 pixel in the motion vector components. For the synthetic images, on the contrary, the mean squared error of each component of the motion field with respect to the true motion field is considered to evaluate the results; it is computed on all the image with the exception of the uncovered areas. The results obtained are shown in the following tables. In both cases they show a great improvement given by the dynamic compression mentioned above. Mobile & Calendar
Brazil
Miss America
block matching
30.76
10.34
4.19
MRF
14.42
5.00
2.16
MRF; no compression
33.15
8.85
5.27
synthetic images
m.s.e, dx
m.s.e, dy
m.s.e, tot
MRF
0.11
0.05
0.16
MRF; no compression
0.24
0.17
0.41
mean squared error
References [1]
[2] [3] [41
S. Geman, D. Geman, Stochastic relaxation, Gibbs distributions, and the bayesian restoration of images, IEEE Trans. Pattern Anal. Machine Intell., vol. 6, no. 6, pp. 721-741, Nov. 1984. E Heitz, P. Bouthemy, Multimodal estimation ofdiscontinous opticalflow using Markov random fields, Artif. Intell., IEEE Trans. Pattern Anal. Machine Intell., vol. 15, no. 12, pp. 1217-1232, Dec. 1993. J. Zhang, G. G. Hanauer, The application of mean field theory to image motion estimation, IEEE Trans. Image Processing, vol. 4, no. 1, pp. 19-33, Jan. 1995. D. Geiger, E Girosi, Parallel and deterministic algorithms from MRF's: surface reconstruction, IEEE Trans. Pattern Anal. Machine Intell., vol. 13, no. 5, pp. 401-412, May 1991.
[5]
B. K. P. Horn, B. G. Schunck, Determining optical flow, Artif. Intell., vol. 17, pp. 185203, Aug. 1981.
[6]
M. Vivalda, Calcolo del moto tra immagini mediante modellizzazione con campi stocastici di Markov, technical report CSTV-CNR, 96.04, Aug. 1996.
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 268
9 1997 Elsevier Science B.V. All fights reserved.
M o v i n g O b j e c t D e t e c t i o n in I m a g e S e q u e n c e s U s i n g T e x t u r e F e a t u r e s Frank Miiller ~, Michael HStter b and Rudolf Mester ~ ~Institut fiir Elektrische Nachrichtentechnik, Aachen University of Technology (RWTH) D-52056 Aachen, Germany bRobert Bosch GmbH, Research Institute for Communications (FV/SLH) P.O. Box 777777, D-31132 Hildesheim, Germany r for Applied Physics, Johann Wolfgang Goethe-University Frankfurt Robert-Mayer-Str. 2-4, D-60054 Frankfurt am Main, Germany In this paper, an algorithm for detection of moving objects in image sequences is presented. The proposed method calculates so called clique texture features for each block (detector cell) of a still image and detects significant changes between corresponding features of successive frames. When a moving object enters a detection cell, the texture feature will change its value and this change can be detected. The method is insensitive to small movements of strongly textured areas like trees moving in the wind. This property is advantageous compared to methods which calculate the pixelwise difference of successive frames or fit the image contents blockwise with low order polynomials. The computational load to calculate the texture features is low. For detection, i.e. decision whether a significant change has occurred, the features are considered as samples of a random process in time. The statistics of these processes (one for each detector cell) are estimated online and statistical hypothesis testing is employed for the decision. 1. I N T R O D U C T I O N Detection of moving objects in video sequences is an important task in a variety of applications. The employed methods vary depending on the specific requirements of the application. The method for object detection presented in this paper is characterized by low computational complexity (allowing inexpensive real time application) and robustness against false alarms caused by diffuse motion is typical for outdoor scenes. The presented algorithm is therefore well suited for surveillance applications. While automatic visual object detection in buildings can often be performed by simple change detection algorithms, the situation is more difficult in case of outdoor scenes. There, the illumination can not be controlled; wind may cause camera vibrations and diffuse motion of non-significant objects. Conventional change detection algorithms will often yield false alarms in these situations, for instance in areas which contain leaves moving in the wind. In a typical outdoor environment, a moving object detection algorithm must deal with varying weather conditions as snow, rain and sudden illumination changes as well as
269 with influences from wind (moving trees and leaves, camera vibrations). Hence, the desired algorithm must have the ability to discriminate between significant changes in the scene caused by moving objects and normal scene changes which can be explained by the aforementioned environment. In other words, a classification of presumptive detection events into significant and non-significant ones is required in order to suppress false alarms. Previous approaches to object detection can be categorized into mainly two classes. First, there are temporal difference based algorithms which evaluate differences between a local luminance value of corresponding areas in successive frames. These approaches differ in the way how the features employed for detection are obtained. They can be obtained pixelwise by low pass filtering [1], or blockwise by simple averaging. Another method is least squares fitting of polynomials (usually second order) to the contents of a block [2]. In the latter case, the parameters obtained from the fitting are regarded as local luminance features. Subsequently, we will denote these types of object detection as
change detection (ChD). Secondly, there are motion based algorithms, in which a motion flow field between successive frames is estimated and subsequently segmented into regions [3]. Regions containing motion are then regarded as belonging to moving objects; the other regions are classified as background. Previous approaches to introduce more robustness into ChD approaches have achieved this on the cost of higher computational complexity. These improved algorithms either need extensive pixelwise computations, or postprocessing of the change mask [1][2][4]. The key idea of our approach is that the proposed algorithm uses tezture features which are obtained blockwise from the frames of the image sequence. The object detection itself is essentially a temporal change detection algorithm, detecting changes of corresponding texture features between successive frames. It suffices to use simple features, which reflect the local covariance structure of the scene. The subsequent change detection algorithm operates on a small number of these features computed per block; i.e. on a significantly reduced amount of data. This approach results in an efficient object detection scheme with low computational complexity. Since the features are related to the texture of a block, the detection scheme is insensitive against pixelwise luminance changes as they might be caused e.g. by slow motion. For example, in case of vegetation in front of the sky, the texture (and the related feature values) will essentially remain unchanged even if wind moves the branches and leaves. An object entering or leaving a block (detector cell) however will cause a change of the feature in almost any cases. Thus the reliability of the object detection can be increased by using suitable texture features. 2. T E X T U R E
FEATURES
Besides being easy to compute, the features should be shift invariant to a certain extent. Then the value of the feature will change only slightly in case of small movements of strongly textured objects as e.g. bushes or trees. The features we use are closely related to the autocovariance function of the image signal inside a block.
270
2.1. Description of Texture Features We use features which are computed from the gray value signal values of pixel pairs whose position relative to each other is fixed. Such a pixel pair is usually called a clique, therefore we denote the features clique texture features. Let us denote with i and j two sites with the coordinates (xi, yi) and (xj, yj) respectively. A clique set is defined as a set of sites for which xi - xj - dx and yi - Y j - d y holds for a fixed displacement d - (dx, dy) under the condition that both sites are located in the regarded block. The vector d determines the clique type. For instance, if dz - O, the clique is denoted as a vertical clique; if dy - 0 it is a horizontal clique. If the sites i and j belong to a clique, we call the difference of the signal s~ 8j at the sites i, j a clique differential, and the squared term -
z-
(1)
forms a squared clique differential. The summation of all squared clique differentials of a given clique type inside of a given image block yields a texture feature
z-
(2)
that characterizes (partially) the texture content of the regarded block. Regarding the image signal as realization of a temporal random vector, the feature Z becomes a random variable, whose statistical distribution can be computed from the distribution of the original gray value vector"
where E[.] denotes expectation.
Assuming stationarity of the image signal in the regarded area, E[si] becomes ms and E[s~] becomes a s2+ ms2 for all i. The covariance cov(si, sj) between two pixels belonging to a clique depends only on the clique type" cov(si, sj) = cij,
(4)
where cij denotes the (i, j ) - t h element of the covariance matrix. For the expectation of the squared clique differential we obtain" E [ ( s i - s j ) ~ ] - 2 (a~ - cij).
(5)
2.2. Properties of Texture Features The described features are invariant to a constant luminance shift, since only differences of the values of certain pixels are involved. As a result, the algorithm is much more insensitive against illumination changes, than a grey level based change detection. The features efficiently discriminate between different textures. In figure 1, a test image (256 • 256 pixels) containing patches of Brodatz textures are given on the left side. In the middle and on the right side, corresponding texture features (based on 8 • 8 detector cells) are depicted.
271
Figure 1. Test image and texture features. Horizontal clique feature.
Middle: Vertical clique feature.
Right:
3. D E T E C T I O N A natural scene usually contains areas of differing image content. Therefore the features of different detection cells are processed individually, allowing for spatially varying texture characteristics of the scene. The time sequences of feature values (one sequence per cell) are regarded as realization of a (vector valued) stochastic process indexed by time. Consequently, the decision rule to tell, whether the scene content inside a particular detection cell hassignificantly changed at a particular time, is based on statistical analysis of the corresponding stochastic process. For each detector cell, the feature statistics are approximated by a mean value and a variance for each feature. Both parameters (mean and variance) are estimated recursively from the past. An IIR lowpass filter accomplishes this task with low computational and memory requirements. Using such an IIR filter results in a time recursive estimator for the mean and variance, which handles slowly varying scene conditions automatically. The estimated values rh, 6 2 are continuously updated as long as no object detection occurs. If the current feature value belongs to the interval [rh - a5 2, rh + a6 2] with a predefined constant a, the detector will decide that no object is present; otherwise a detection event will occur. For a symmetric unimodal distribution of the regarded feature this decision rule is equivalent to significance testing using the estimated values of mean and variance. A more detailed description of the used time recursive estimators can be found in [5]. 4. S I M U L A T I O N
RESULTS
In figure 2 two images from a test sequence are shown together with a corresponding difference image. The sequence shows an outdoor scene with a road and trees in the foreground and in the background. On the second image a person can be seen who entered the scene during the time interval between both exposures. A human observer easily detects the person walking along the road. However, the images differ from each other as well in regions where tree branches move due to wind. A human observer can detect these changes only by close examination of the pictures.
272
Figure 2. Two images from the test sequence "walk" and corresponding difference image
An object detection algorithm based on the evaluation of temporal pixel differences will almost always "see" these differences and output a false alarm.
Figure 3. Absolute temporal texture feature differences for sequence "walk" with varying detector cell size. Whereas in such regions particular pixel values may change drastically between different frames, texture features show much less variation. At the same time, the temporal variations of the features are high in the detector cells, which the person enters. To show this, in figure 3 absolute texture feature value differences are depicted for varying detector cell sizes. Even if cell sizes of 32 x 32 pixels are used, the difference between the features is much higher at the person's location than in the other areas. The difference of the feature values between successive frames just gives an idea of the performance of the algorithm. In the "tree" areas, the time recursive estimation procedure estimates much higher values of the variance than in the area where the person walks. As result the detection algorithm in all cases detected the person without any false alarms. The presented method operates with low computational complexity. Calculation of
273 the texture features involves computations of the same order as calculation of a squared difference criterion (as is used in plain ChD algorithms). The time recursive estimation of mean and variance of the features takes less than computation of the features, due to the data reduction in the feature extraction step. Sharing the low complexity with earlier detection methods, the presented algorithm additionally can deal with complex textured scenes and temporarily varying image signal statistics. Therefore it is very well suited for outdoor applications, where weather conditions and illumination change continuously and the characteristics of the observed scene cannot be controlled. The robustness and emciency of the proposed method has been extensively tested in offline simulations as well as in (realtime) online processing of numerous scenes; for further information regarding these evaluation, the reader is referred
to [5]. REFERENCES
1. T. Aach, A. Kaup, R. Mester: Statistical model-based change detection in moving video. Signal Processing 31 (1993) 165-180. 2. Y.Z. Hsu, H.-H. Nagel and G. Rekers: New likelihood test methods for change detection in image sequences. Computer Vision, Graphics and Image Processing. 26 (1984) 73-106. 3. J.H. Duncan and T.-C. Chou: On the detection of motion and the computation of optical flow. IEEE Transactions on Pattern Analysis and Image Processing 14 (1992) 346-352. 4. T. Aach, A. Kaup, R. Mester: Change detection in image sequences using Gibbs random fields: A Bayesian approach. Proceedings of International Workshop on Intelligent Signal Processing and Communication Systems, Sendal, Japan (1993) 56-61. 5. M. H5tter, R. Mester and F. Miiller: Detection and description of moving objects by stochastic modelling and analysis of complex scenes. Signal Processing: Image Communication 8 (1996) 281-293.
274
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All fights reserved.
Determining velocity vector fields from sequential images representing a salt-water oscillator A.Nomura ~ and H.Miikeb ~Department of Cultural and International Studies, Yamaguchi Prefectural University, Sakurabatake 3-2-1, Yamaguchi, 753 Japan bDepartment of KANSEI Design and Engineering (Department of Perceptual Sciences and Design Engineering), Yamaguchi University, Tokiwadai 2557, Ube, 755 Japan With the gradient-based method we can determine 2 dimensional velocity vector fields from an image sequence. The method employs a basic constraint equation and additional constraint equations. In this paper, as an additional constraint equation, periodic behavior of salt-water oscillation is taken into account to determine its flow fields. The proposed method is applied to an artificially synthesized image sequence and to the real one representing the salt-water oscillation. Through analysis of the image sequences the usefulness of the proposed method is confirmed. 1. I N T R O D U C T I O N An image sequence represents projection of time-varying 3 dimensional (3D) scene onto an image plane. Since various kinds of visualized phenomena are captured by a TV camera and an image acquisition board, digital image processing has broad applications in the fields of scientific measurement, computer vision (the study to realize an artificial vision system), medical image analysis, industrial measurement and so on. For example, the determination of an optical flow (an apparent velocity vector field of a brightness pattern) brings meaningful information on 3D scene as follows. In computer vision, shape and structure of a 3D object in a static scene is recovered from an optical flow by the theory of motion stereo. Horn and Schunck proposed a famous gradient-based method to determine an instantaneous optical flow [1], since their goal was to realize an artificial or robotic vision system. In fluid measurement, Imaichi and Ohmi applied conventional image processing techniques to an image sequence representing fluid dynamics visualized by small particles and slit light illumination [2]. Because of the slit light illumination, depth information is known already. Thus, with the techniques 2 dimensional distributions of physical variables of a fluid flow are obtained. Our interest in this paper is focused on to determine 2D fluid flow fields. The methods to determine the velocity fields are divided into two categories: the matching-based method and the gradient-based one. The former method determines a velocity vector by tracing a brightness pattern between two sequential images. On the other hand, the latter determines a velocity vector field by minimizing an error function
275 employing a basic constraint equation and additional ones. The basic constraint equation represents a relationship between spatial and temporal partial derivatives of brightness distribution of the sequence and two unknown components of a velocity vector. Several additional constraint equations such as local constancy of a velocity vector field [3] and smoothness of that [1] have been proposed to determine the unknown variables. While the basic constraint equation can be used under most situations except for special situations such as non-uniform illumination, the additional constraint equation should be selected appropriately for each situation. In this paper, our goal is to determine a 2D velocity vector field with high accuracy. Especially, we focus on the determination of 2D slice of a three dimensionally distributed velocity vector field observed in a salt-water oscillator. We already confirmed that it had a periodic nature and its oscillation period was almost constant. Since we can utilize the periodic characteristic as an additional constraint equation in the gradient-based method, we propose a new method to determine the optical flow field having a rigid periodic behavior. We confirmed the usefulness of the proposed method through the analysis of an artificially synthesized image sequence and real one. 2. A D D I T I O N A L C O N S T R A I N T E Q U A T I O N TO THE GRADIENT BASED METHOD
AND ITS A P P L I C A T I O N
Horn and Schunck derived the following basic constraint equation by tracing a brightness pattern [1],
Of
-b7
Of
Of
+ v N = o,
(1)
where f(x, y, t) is spatio-temporal brightness distribution of an image sequence and (u, v) are two components of a velocity vector. Since the brightness distribution is measured by a TV camera, f ( x , y , t ) is a known variable. On the other hand, u and v are unknown variables to be estimated. In spite of two unknown variables only one constraint equation is available at a pixel in principle. Therefore, more additional constraint equations are necessary for obtaining full solution on the velocity vector components. Fukinuki proposed an additional constraint equation assuming spatial local constancy of a velocity vector field [3] and Horn and Schunck did a smoothness constraint equation [1]. Nomura et al. proposed one assuming temporal constancy of velocity vector fields (stationary velocity vector fields) [4] as an additional constraint equation. Consequently, the basic constraint equation obtained at a fixed spatial point along a time coordinate is assumed to have the same velocity vector (the temporal optimization method). Since several additional constraint equations have been proposed, we have to select proper method representing characteristics of an image sequence from all of them. Let us focus on the image sequence representing a salt-water oscillator. Its velocity vector field is oscillating at a constant period. Temporal changes of velocity vector components observed in simple oscillating and translating velocity vector fields is shown in Figure 1. From the figure, same velocity values are observed at the constant interval L(frame) as follows,
u(t) = u(t + L) = u(t + 2L) = u(t + 3L) . . . . .
u(t + N . L),
(2)
276
v(t) = v(t + L) = v(t + 2L) = v(t + 3L) . . . . .
v(t + N . L),
where the oscillation period L is assumed to be constant over N(cycle) and time variable is denoted by an integer number t which is defined in the range of 0 < t < (L - 1). (In this case, t corresponds to phase.) Equation (2) can be used as an additional constraint equation. Since eq.(2) does not spatially constrict the field, the field determined by the proposed method employing eq.(2) will be expected to have high spatial resolution. The period is assumed to be constant and to be known as previous knowledge. From this, we need a method to estimate the period. This additional constraint equation is developed from the one of temporal constancy we proposed [4]. Now we propose a new gradient-based method employing the basic constraint equation eq.(1) and the additional one eq.(2). If an observation point is fixed, basic constraint equations obtained at that point at the time intervals L(frame) have the same velocity components. Consequently, we can estimate the two velocity components by minimizing the set of the obtained basic constraint equations as follows, E=~
-~n--O
+UTxx +v x,y,t+nL x,y,tTnL ~
,
x,y,tTnL
(3)
where partial derivatives are evaluated by the method described in the literature [1] and the least squares method is utilized for its minimization. In the matching-based method the additional constraint equation eq.(2) is difficult to be taken into account. This is the reason why we use eq.(2) as an additional constraint equation of the gradient-based method. Velocity[pixel/frarne]
0.4 L
1.0
L
L
0.8
0.6
t)
02
0.0
0
10
20
30
Time [framel
40
Figure 1. Temporal changes of velocity vector components u(t) and v(t). oscillating at a period of L(frames).
They are
3. A N A L Y S I S O F A N A R T I F I C I A L L Y S Y N T H E S I Z E D I M A G E S E Q U E N C E To confirm usefulness of the proposed method, it is applied to an image sequence. The image sequence is synthesized artificially by translating a brightness pattern on an image plane. The pattern is synthesized by,
f o ( x , y ) = A{1 + sin ( ~ - ~ ) . sin ( ~ - ~ ) },
(4)
277 where 2A = 254 is the maximum brightness value of the pattern and A = 30(pixel) is the wave length of that. The two components of the translating velocity vector are modulated with time t as,
---
~
+
U2COS
~
(5)
v
where T = 10(frame) is the period of the temporal sinusoidal changes of the velocity components and other parameters are as follows: u0 = 0.4, U 1 - - " 0 . 6 , U 2 - - - 0 . 0 , V 0 - 0.3, Vl -- 0.1, v2 = 0.1(pixel/frame). The image sequence having 100 frames (10 cycles) with the spatial resolution of 100• 100 (pixel) is analyzed by the proposed method and by the ordinary temporal optimization method [4] assuming constancy of velocity vector fields during 10 frames. A velocity vector field determined at every frame is spatially averaged. Temporal changes of two components of the averaged velocity vector are shown in Figures 2 and 3. From the figures the two components determined by the proposed method are almost on the true lines. Consequently, the usefulness of the method is basically confirmed through the analysis. 1.2 Velocity[pixel/frame]
0.8
. . . . . .
§
1.2 Velocity!pixel/frame]
"
0.8
§
0.4
0.4
0.0
0.0 0
2
4 6 Time[frame]
8
10
Figure 2. Temporal changes of two velocity components. The solid lines represent true velocity. Symbols represent velocity components determined by the ordinary temporal optimization method from the artificially synthesized image sequence.
0
~
2
4 6 Time[frame]
8
10
Figure 3. Temporal changes of two velocity components. The solid lines represent true velocity. Symbols represent velocity components determined by the proposed method from the artificially synthesized image sequence.
278 4. A N A L Y S I S OF A R E A L I M A G E S E Q U E N C E R E P R E S E N T I N G WATER OSCILLATOR
A SALT-
The proposed method is applied to the real image sequence representing a salt-water oscillator. Its experimental setup is shown in Figure.4. Around the small hole in the bottom of the inner vessel an oscillating fluid flow is observed [5,6]. The fluid flow is visualized by small particles and laser slit light illumination. The visualized 2D slice of the flow is captured by a TV camera system and an image acquisition system. Sampling frequency is 30 Hz, the spatial resolution of an image plane is 40 • 40 (pixel) and brightness is quantized into 256 levels. The acquired image sequence consists of 7923 frames (19 cycles). The image sequence is analyzed by the proposed method and the spatio-temporal optimization one. The latter one assumes constancy of the velocity vector field during 19 frames. In addition, both methods assume local constancy of a velocity vector field, where its local size is 7 • 7 (pixel). When we can use more grid points (more basic constraint equations) in the optimization, it is expected to reduce noise. Temporal changes of two velocity components determined at the center (x = 20, y = 20) by the proposed method are shown in Figure 5. The u(t) component is almost zero, while, the v(t) component has a down flow. The peak of v(t) is observed at around t = 100 (frame). After the passage of the peak, the flow velocity is decreasing with time. These changes are consistent with what we observe in the real experiments. On the other hand, such characteristics is not clear in the temporal changes of velocity vector fields determined by the spatio-temporal optimization. 5. C O N C L U S I O N S In this paper, an additional constraint equation representing the characteristics of periodic velocity vector fields was proposed. The gradient-based method employing the basic constraint equation eq.(1) and the proposed one eq.(2) was applied to an artificially synthesized image sequence and to the real one representing salt-water oscillation. Through the analysis of two kinds of image sequences the usefulness of the proposed additional constraint equation was confirmed. The proposed method focused on the analysis of salt-water oscillation. However, we often observe oscillatory flow in several situations. For instance, Karman vortex [2] is one of the typical examples of the oscillatory flow field. Consequently, we can expect that the proposed method is effective to fluid flow field analysis. We have proposed the generalized gradient-based method [7]. The method introduced a generalized basic constraint equation representing the effect of non-uniform illumination. Since we can apply the additional constraint equation proposed in this paper to the generalized gradient-based method, we can determine oscillatory velocity vector fields under non-uniform illumination frequently observed. ACKNOWLEDGEMENTS The authors thank to Prof.K.Yoshikawa (Nagoya University) and his student Miss.M.Okamura for their experimental help. This work was partly supported by the Grant-in-Aid of Ministry of Education, Science and Culture of Japan.
279
Figure 4. Experimental setup of a salt-water oscillator.
Figure 5. Temporal changes of two velocity components determined at the point (x, y) = (20, 20) from the real image sequence (see Figure 4). They are determined by the proposed method employing local constancy of the velocity vector field in addition to the assumption of eq.(2), where the local size is 7 x 7 pixels.
REFERENCES
1. 2. 3. 4. 5. 6.
B.K.P.Horn and B.G.Schunck, Artificial Intelligence 17 (1981) 185. K.Imaichi and K.Ohmi, J. Fluid Mechanics 129 (1983) 283. T.Fukinuki, Technical Report of IECE, IE78-67 (1978) 35 (in Japanese). A.Nomura, H.Miike and K.Koga, Pattern Recognition Letters 12 (1991) 183. S.Martin, Geophysical Fluid Dynamics 1 (1970) 143. K.Yoshikawa, S.Nakata, M.Yamanaka and T.Waki, J. Chemical Education 66 (1989) 205. 7. A.Nomura, H.Miike and K.Koga, Pattern Recognition Letters 16 (1995) 285.
This Page Intentionally Left Blank
H
TRACKING AND RECOGNITION OF MOVING OBJECTS
This Page Intentionally Left Blank
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All rights reserved.
283
"LONG-MEMORY" MATCHING OF INTERACTING COMPLEX OBJECTS FROM REAL IMAGE SEQUENCES A. Tesei*, A. Teschioni*, C.S. Regazzoni*, and G. Vemazza**
*Department of Biophysical and Electronic Engineering (DIBE), University of Genova, Via all'Opera Pia 11A, Genova, Italy **Department of Electrical and Electronic Engineering (DIEE), University of Cagliari, Piazza d'Amfi 1, Cagliari, Italy
1.
INTRODUCTION
Computer-assisted surveillance of complex environments is getting more and more interesting, thanks to recent significant improvements ha real-time signal processing. The main role of automatic computation in such systems is to support the hulnan operator to perform tasks such as detecting, interpreting, logging or giving alarms. In the surveillance research field applied to public areas, crowding monitoring is very useful but presents particulm'ly complex problems. Recognizing objects and persons and tracking their movements in complex real scenes by using a sequence of images are among the most difficult tasks in computer vision [1][2]. Object and human motion tracking in 3D real scenes can be achieved by means of Kahnan filtering [3][4]; a suitable lnathematical model for describing objects and persons and a refined dynamic model for tracking them while moving and reciprocally interacting are needed. Such approaches can provide accurate and robust results even in uncontrolled real-life working conditions. In [4] a method for tTacking only a single moving person was presented."
"The work has been partially supported by the European Communities under Conlracl ESPRIT-P8433PASSWORDS
284 In this work, this limithag assumption decays, and morn general and more complex situations " are considered, as several objects (persons and not) moving and interacting in real scenes are treated. It is mainly addressed to present the two main phases at the basis of object recognition and tracking: 9
the selection of a set of image features characterizing each detected mobile object or group of objects, and consequently allowing the system to distinguish an object with respect to another; the matching procedure which allows one to recognize a certain object even after various frames in which it disappeared completely or pm'tially ("long-memory" matching).
Thanks to real-time functioning, accuracy and robustness, the method can be used in real-life surveillance systems.
11
SELECTION AND EXTRACTION OF BLOB INTERNAL CHARACTERISTICS
From each image of the sequence to be analyzed, the lnobile areas of the image (i.e., the blobs) are detected by a frame-background difference [4] and analyzed by extracting numerical characteristics (e.g., geometrical and shape properties). Blob analysis is performed by the following modules:
1.
Change detection (Fig. l b): by using statistical morphological operators [5], it identifies the mobile blobs present in the original b/w image (Fig. l a) that exhibit remarkable differences with respect to the background.
.
Focus of attention (Fig. lc): by means of a fast image-segmentation algorithm [6], it detects the minimum rectangle bounding each blob in tile image (corresponding to single or multiple objects, or parts of an object).
285
Figure 1. (a) Original image, (b) Change detection image, (c) Focus of attention image (surveillance of a metro station).
3.
Measure extractor: it extracts from each blob its perimeter, area, bounding box area, height and width, mean grey level value, bounding box centre 2D position on the image plane.
3.
MOBILE BLOB MATCHING
The module labels with the same number each blob corresponding to the same object, object part or group present in the sequence over time. On the basis of the matching result, each blob can be tracked over time, hence providing a further blob characterization by means of cinematic parameters. Matching is perfonned in two steps: A. A first rough result is roached by comparing the lists of blob characteristics refening to the current (time step k) and previous (time step k-i) frames. Blob correspondences are organized as a graph: all nodes of each level are the blobs detected in each tYame, and the relationship among blobs belonging to different adjacent levels are represented as arcs between the nodes. Arcs are inserted on the basis of the superposition of blob areas on the image plane: if a blob at step (k-l) overlaps a blob at step k, then a lhak between them is created, so that the blob at step (k-l) is called "father" of the blob at time step k (its "son"). Different events can occur: 1)
If a blob has only one "father", its type is set "one-overlapping" (type o), and father
label is assigned to it.
286 2)
If a blob has more than one "father", its type is set to "merge" (type m), and a new
label is assigned. 3)
If a blob is not the only "son" of its father, its type is set to "split" (type s), and a new
label is assigned. 4)
If a blob has no "father", its type is set to "new" (type n) and a new label is assigned.
B. Blob matching is refined by substituting, if possible, the new labels with the labels of some blob either belonging to a time step previous than (k-I), or belonging to the (k-l) frame and erroneously labelled in phase A. This processing phase is based on the comparison between each current blob not labelled with "o" with the set of recent previous blobs, whose label was inherited by no successive blob, collected in a "long-memory" blob graph. This approach should be useful for recover some situations of temporal~j wrong splitting of a blob (corresponding to a single object) into more blobs, because of image noise or static occlusions, or of temporary merging of two overlapping objects. Comparison is performed on the basis of the blob shape characteristics which have been tested to be approximately time/scale-invariant. Blob matching provides as output the final graph of blob con'espondences over time, in which matched blobs have the same label over time, and each blob is classified with one of the described types.
4.
EXPERIMENTAL RESULTS
Extensive tests on real image sequences were performed in the context of CEC-ESPRIT Project PASSWORDS, in which two main surveillance application sites were selected: a supermarket and an underground station. Figure 2 shows the result of the blob matching algorithm on a test sub-sequence: each image contains the detected blobs (resulting from blob detection) with their numerical label and type (obtained from blob matching). This exmnple points out in particular the capability of the module to:
287 1. assign the same label to two blobs before and after their temporary overlapping on the image plane (hence to consider them as the same mobile object) even after several frames (see frames 2 and 5); 2. assign the correct label of a blob which was erroneously classified as new during the matching phase A (see frames 2 and 3).
Figure 2. A sequence of images showing critical cases of blob splitting, merging and displacement.
REFERENCES 1. W. Kinzel, and Dickmanns ED, "Moving humans recognition using sl)atio-temporal
models", XVII-th Congress Int. Soc. Photogrammetry and Remote Sensing, 1992. 2. F. Cravino, M. Delucca and A. Tesei, "DEKF Ls3'stemfor crowding estimation by a
multiple-model approach", Electronics Letters, no. 30, vol. 5, 1994, pp. 390-391.
288 3. Y. Bar-Shalom, and T.E. Fortmann, "Tracking and Data Association", Acade~rfic Press, New York, 1988. 4. A. Tesei, G.L. Foresti, and C.S. Regazzoni, "Human Body Modelling for People
Localization and Tracking j)'om Real Image Sequences", IPA'95, Heriot-Watt University, UK, July 1995, pp. 806-809. 5. J. Serra, "Image Analysis and Matt~ematical Molphology. 2: Theoretical Advances", Academic Press, London, 1988. 6. D.H. Ballard, and C.M. Brown, "Computer Vision", Prentice Hall, New York, 1982.
Time-Varying Image Processing and Moving Object Recognition, 4- V. Cappellini (Ed.) 9 1997 Elsevier Science B.V. All rights reserved.
Spatial and temporal
289
g r o u p i n g for o b s t a c l e d e t e c t i o n in a s e q u e n c e of
road images Sandra Denasi and Giorgio Quaglia ~ ~Istituto Elettrotecnico Nazionale "Galileo Ferraris" Strada delle Cacce 91, 10135 Torino, Italy E-mail:
[email protected] Computer vision systems devoted to driving assistance of vehicles moving on structured roads require the fulfillment of two essential tasks: the localization of the road boundaries and the detection of obstacles on the road. The present paper proposes an algorithm for early detection and tracking of vehicles based on active perception of the most significant structures pointed out in correspondence of the road in the image sequence. Perceptual persistence of some structures is used to start up object hypotheses, then a model based grouping process integrates these vague hypotheses along the sequence until a real obstacle is recognized by the description of its main structures. 1.
INTRODUCTION
Among the different contributions that computer vision equipments can provide for increasing the safety in car driving, lane keeping is the most feasible, and is helpful mainly in long highway trips. However this aid could be the cause of lack of attention, so lane keeping equipments become more complete if they are coupled with obstacle detection devices. Regarding as "obstacles", in a first instance, vehicles moving on the same lane, two kinds of obstacles can be met on this lane: faster overtaking vehicles that will disappear on the horizon and slower or still vehicles that will be reached after a lapse of time. The first ones can be pointed out analyzing their salient structures, while the latter are hardly perceivable when they appear on the horizon and become more evident as they approach, as shown in figure 1. In the present paper we face the problem of detecting obstacles as soon as possible. However, early detection conflicts with reliable detection, because structures of vehicles far away from the camera mingle with the background. So we propose an approach based on integration of edge segmented images along the sequence and on perceptual and geometrical grouping of segments to form meaningful structures for obstacle recognition. Perceptual persistence of peculiar structures is used to start up object hypotheses, then a model based grouping process searches for meaningful groups of segments related to the outline and to parts of a vehicle in order to distinguish an approaching obstacle from other patterns such as road signs, patches or shadows.
290 2.
THE PROPOSED
APPROACH
The obstacle detection process focuses its attention on the area of the image that corresponds to the road. Details about the road boundary detection algorithm can be found in [1]. Since only the rear or front side of cars can be seen from a vehicle moving on the same road, parts such as the windscreen or the rear window, the number plate, the bumper, the lights, the wheels, and also the shadow under the vehicle between the wheels, could be looked for to detect and recognize the vehicles. These parts are usually pointed out by the analysis of horizontal and vertical contours [2,3]. However edge segmentations of road scenes are strongly cluttered, namely in those areas where an early detection of obstacles is important: that is, far away, near the horizon. Moreover, because of segmentation noise, edges corresponding to parts of the vehicle appear and disappear along the sequence and obstacles are nearly imperceptible. Different strategies must, then, be used to analyze different situations. While approaching a vehicle, three phases can be distinguished: the attention phase monitors the end of the road and detects far away objects that appear, then the tracking phase follows these object hypotheses in order to verify their persistence in successive frames, finally, when the objects are closer and their shape is better visible, the recognition phase validates the hypotheses searching for structures of the objects that could be matched with known models of vehicles, estimates their position and warns about obstacles.
Figure 1. Images of far and close vehicles and their segmented lines.
3.
DETECTION
OF OBSTACLE HYPOTHESES
Analyzing segmented images frame by frame, as soon as the likely vehicle appears, does not allow its recognition. Structures of vehicles are too small and confused with other structures to give reliable indications. Instead, something that suggests an object appears when we observe the entire sequence of frames. Therefore, an object cannot be identified considering a snapshot of its structures, but taking into account the persistence of these structures along the sequence of images. That is because, when a vehicle is far away, at the end of the visible road, its position and dimension do not change significantly from a frame to the next one. An initial area of attention (AOI) is centered around the end of the road, at the intersection of the left and right road borders previously localized (figure 2a). Because a "loose" model of a vehicle is sufficient to detect it, a far away vehicle is described simply by a group of short horizontal lines, as shown in figure 2b. Persistence
291 of structures is computed using a grid mapped to the image and considering a sequence of Natt frames. All the horizontal lines in the AOI of each image are rasterized and their pixels contribute with a vote to the total count on the grid. A circular frame buffer is used to update the sum along the sequence, removing votes of the oldest image and adding votes of the current frame. When one or more pixels in the grid reaches a sufficient amount of votes MINatt, a good probability does exist that an object has appeared in that area, then a first hypothesis of obstacle is instantiated. At present, the following values are considered to be suitable thresholds for detecting 10 and MINatt = 8. Figure 2c shows the reliable persistence of structures: Natt resulting integration. =
Figure 2. (a) Road borders and AOI, (b) H and V lines, (c)line persistence.
4.
TRACKING
OF THE HYPOTHESES
The detected structures must be tracked for a while, to verify their reliability to belong to a single object and not to be part of the background. Very noise sensitive features, such as corners, and small elements, such as vertical line segments, hardly can give reliable indications. Then again a rough model is used in this phase. In particular, an obstacle is modeled as a parallelepiped, whose projection on the image plane is a rectangle, which must include enough structures to characterize a vehicle. For each frame, firstly the position and size of the AOI are updated according to the detected obstacle position and the known road boundaries. Since lines belonging to a vehicle are peculiar for their symmetry, a likely vehicle can be pointed out by a peak on their projection profile. Therefore, the nearly horizontal lines in the AOI are projected and accumulated to a horizontal buffer and the maximum value of the resulting profile is computed. Let Xl and x2 be the endpoints of a line, and its horizontal projection be defined as
P(x)-
1 ifxl
