Tactile Perception of Textiles in a Virtual-Reality System (Cognitive Systems Monographs)

Cognitive Systems Monographs Volume 10 Editors: Rüdiger Dillmann · Yoshihiko Nakamura · Stefan Schaal · David Vernon

Dennis Allerkamp

Tactile Perception of Textiles in a Virtual-Reality System

ABC

Rüdiger Dillmann, University of Karlsruhe, Faculty of Informatics, Institute of Anthropomatics, Humanoids and Intelligence Systems Laboratories, Kaiserstr. 12, 76131 Karlsruhe, Germany Yoshihiko Nakamura, Tokyo University Fac. Engineering, Dept. Mechano-Informatics, 7-3-1 Hongo, Bukyo-ku Tokyo, 113-8656, Japan Stefan Schaal, University of Southern California, Department Computer Science, Computational Learning & Motor Control Lab., Los Angeles, CA 90089-2905, USA David Vernon, Khalifa University Department of Computer Engineering, PO Box 573, Sharjah, United Arab Emirates

Author Dipl.-Math. Dennis Allerkamp Gottfried Wilhelm Leibniz Universität Hannover Institut für Mensch-Maschine-Kommunikation Fachgebiet Graphische Datenverarbeitung Welfengarten 1 30167 Hannover Germany E-mail: [email protected]

ISBN 978-3-642-13973-4

e-ISBN 978-3-642-13974-1

DOI 10.1007/978-3-642-13974-1 Cognitive Systems Monographs

ISSN 1867-4925

Library of Congress Control Number: 2010929202 c 

2010 Springer-Verlag Berlin Heidelberg

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable for prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typeset & Cover Design: Scientific Publishing Services Pvt. Ltd., Chennai, India. Printed on acid-free paper 543210 springer.com

Preface

This monograph originates from my work on the HAPTEX project. In December 2004 Prof. Franz-Erich Wolter, the head of the Institute of Man– Machine Communication of the Leibniz Universit¨ at Hannover, offered me the opportunity to participate in that EU funded project. Being a mathematician I had only very little experience in the field of haptic simulation in those days, but Prof. Wolter trusted in my ability to become acquainted with new fields of research in a very short time. I am still thankful for the confidence he has shown me. Since then I indeed learned and found out a lot. With this monograph I try to pass on the knowledge I gained. Having a reader in mind who—like me at the beginning of the project—has no background in psychophysics, neurophysiology or textile engineering I will provide the necessary basics. The skilled reader may safely skip these parts. Nevertheless I presume some basic knowledge in mathematics. I hope that this thesis might help a newcomer to discover the fascinating field of tactile simulation. This work would not have been possible without the funding of the project “HAPtic sensing of virtual TEXtiles” (HAPTEX) under the Sixth Framework Programme (FP6) of the European Union (Contract No. IST-6549). The funding is provided by the Future and Emerging Technologies (FET) Programme, which is part of the Information Society Technologies (IST) programme and focuses on novel and emerging scientific ideas. For inventing this project and accordingly requesting the funding I thank Prof. Nadia Magnenat-Thalmann, Dr. Harriet Meinander, Prof. Franz-Erich Wolter, Dr. Ian Summers and P.Eng. Fabio Salsedo. Special thanks go to my thesis advisor Prof. Franz-Erich Wolter for his guidance and support throughout my work on my PhD thesis which has greatly benefited from all his feedback. I hope I inherited some of his good sense for interesting and profound research. My research was influenced a lot by Dr. Ian Summers who originally worked on the subject of tactile simulation. He not only provided several

VI

Preface

tactile displays for the experiments but also shared a lot of his knowledge and experience. I really appreciate I was able to work with him. I also thank Prof. Nadia Magnenat-Thalmann for the coordination of the HAPTEX project. She always applied very high standards to our research work. This was sometimes very stressful but the project would not have been nearly as successful as without her leadership. I appreciate her willingness to be a member of my examination committee. I really enjoyed the interdisciplinary collaboration within the HAPTEX project. I thank all the team members for the interesting discussions and the positive working atmosphere. I would like to give a special mention to Guido B¨ottcher; we worked closely together on the haptic rendering software and complemented each other very well. It was a pleasure to work with him. I had the pleasure of working with some very talented students who contributed to my research: I thank (in alphabetical order) Steffen Blume, Daniel Gl¨ockner, Tjard K¨ obberling, Karin Matzke and Natalya Obydenna. I also thank Steffen Blume, Wiebke Frey and Iris Lieske for proofreading the manuscript and for the helpful discussions. The experiments conducted for my research were only possible because numerous persons voluntarily participated as probands. I thank all of them for their support. Finally, I really appreciate the moral support of my colleagues, friends and especially of my family. Thank you! Hannover, March 2010

Dennis Allerkamp

Contents

1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 4

2

Human Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Introduction to Psychophysics . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.1 Just Noticeable Differences and Weber’s Law . . . . . . . 2.1.2 Weber-Fechner Law and Stevens’ Power Law . . . . . . . . 2.1.3 Measuring Psychometric Distances . . . . . . . . . . . . . . . . 2.1.4 Statistical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.5 Signal Detection Theory . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.6 Multidimensional Scaling . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 The Human Nervous System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Neurons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2 Neural Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.3 Sensory Reception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Human Tactile Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.1 Mechanoreceptive Afferents . . . . . . . . . . . . . . . . . . . . . . . 2.3.2 Perception of Form and Texture . . . . . . . . . . . . . . . . . . . 2.3.3 Perception of Fine Textures . . . . . . . . . . . . . . . . . . . . . . . 2.3.4 Perceptional Model of Tactile Simulation . . . . . . . . . . . 2.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5 5 6 7 8 9 10 12 15 15 17 17 18 18 19 20 20 22 22

3

Devices for Tactile Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Survey of Existing Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Electromagnetic Displays . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.2 Pneumatic Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.3 Displays with Shape Memory Alloys . . . . . . . . . . . . . . . 3.1.4 Piezoelectric Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . .

25 25 26 27 27 27

VIII

Contents

3.1.5 Other Actuator Mechanisms . . . . . . . . . . . . . . . . . . . . . . 3.2 The Tactile Displays Used . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 The Piezoelectric Effect . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Mechanical Behaviour . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Geometrical Configurations . . . . . . . . . . . . . . . . . . . . . . . 3.3 Drive Electronics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 USB Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 Data Bus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.3 Variable Voltage Supply . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.4 Digital–Analogue Conversion . . . . . . . . . . . . . . . . . . . . . 3.3.5 Amplification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Force-Feedback Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 The Haptic Loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Modeling the Contact Forces . . . . . . . . . . . . . . . . . . . . . . 3.4.3 Different Degrees of Freedom . . . . . . . . . . . . . . . . . . . . . 3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

28 28 29 30 31 33 34 37 39 39 41 42 43 43 44 44 45

4

Generation of Virtual Surfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Sample Fabrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 Fibres, Yarn and Fabrics . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.2 Selection of Sample Fabrics . . . . . . . . . . . . . . . . . . . . . . . 4.2 Kawabata Evaluation System for Fabrics . . . . . . . . . . . . . . . . . 4.2.1 KES-F Roughness Test . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 KES-F Friction Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Spatial Frequency Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.1 Selection of Appropriate Sections . . . . . . . . . . . . . . . . . . 4.3.2 Discrete Fourier Transform . . . . . . . . . . . . . . . . . . . . . . . 4.3.3 Two-Dimensional Composition . . . . . . . . . . . . . . . . . . . . 4.4 The Correlation–Restoration Algorithm . . . . . . . . . . . . . . . . . . 4.4.1 Data Reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.2 Two-Dimensional Composition . . . . . . . . . . . . . . . . . . . . 4.5 Surface Reconstruction from an Optical Surface Scan . . . . . . 4.5.1 Symmetry Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5.2 Shape from Shading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Artificial Surfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6.1 Brownian Surfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

49 49 50 52 53 54 55 56 57 58 60 61 62 63 64 65 67 68 69 72 73

5

Tactile Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Rendering Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.1 Position Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Experiment with a Force-Feedback Device . . . . . . . . . . . . . . . . 5.2.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

75 76 77 80 80

Contents

5.2.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Vibrotactile Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.1 Computation of Resulting Vibrations . . . . . . . . . . . . . . 5.3.2 Decomposition of Vibrations into Base Frequencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.3 First Results with Brownian Surfaces . . . . . . . . . . . . . . 5.3.4 Further Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Integration with a Force-Feedback Device . . . . . . . . . . . . . . . . . 5.4.1 Physical Simulation and Haptic Rendering . . . . . . . . . . 5.4.2 Integrated Interface Hardware . . . . . . . . . . . . . . . . . . . . 5.4.3 Software Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.4 Evaluation of the Integrated System . . . . . . . . . . . . . . . 5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

IX

81 85 85 86 88 89 91 93 94 95 96 98 99

6

Summary and Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

A

Fabrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

Acronyms

ACM CRA DAC DFT DIP DOF FFT fps IC IPC jnd KES-F lpi MDS OPA PC RA rms SA1 SA2 SFM SFS USB VR

autocorrelation matrix correlation–restoration algorithm digital–analogue converter discrete Fourier transform dual in-line package degrees of freedom fast Fourier transform frames per second integrated circuit inter-process communication just noticeable difference Kawabata evaluation system for fabrics lines per inch multidimensional scaling operational amplifier Pacinian mechanoreceptive afferent rapidly adapting mechanoreceptive afferent root mean square slowly adapting type 1 mechanoreceptive afferent slowly adapting type 2 mechanoreceptive afferent statistical feature matrix shape from shading Universal Serial Bus virtual reality

Chapter 1

Introduction

Virtual Reality (VR) has a lot of applications ranging from entertainment to mechanical design and medical training. VR systems are often used in training situations where training in a real environment would be inappropriate and possibly even dangerous. Pilots, for example, often practise on a flight simulator before flying with a new type of aeroplane. Interestingly, flight simulators are also sold as games for personal computers. Today, computer games are probably the most common VR systems. VR systems can be categorised by the modalities they support. In today’s systems the modalities of seeing and hearing are the most commonly employed as these are also the modalities in which we as human beings mostly exchange information. They require least effort in terms of energy transfer, the corresponding sensory receptors are concentrated in the retina and the cochlea and can be excited remotely with light and sound waves respectively. In contrast to seeing and hearing the creation of appropriate haptic stimuli demands very sophisticated hardware. Firstly, the skin with its size of 1.5–2 m2 is a very large organ. Therefore most haptic devices focus on a rather small part of the human body—usually the fingertip. Secondly, forces cannot be transmitted contactfree with current technology. Thus haptic devices always need direct contact to the parts of the skin where the forces are applied. Thirdly, the amount of energy is relatively high compared to other modalities, e.g. if one wants to simulate the lifting of an object with a mass of 500 g the haptic device has to create a force of approximately 5 N. All these properties make haptic simulation a complex task still presenting a lot of problems awaiting a good solution. Today’s VR systems provide haptic interaction with virtual objects only indirectly via tools like a thimble or a pen-like probe or more specialised tools like, for example, the yoke of a plane, the handle of a scalpel or the handles of a pair of scissors. These tools usually appear twice in a VR system: as end effector transmitting the forces from the haptic device and as a virtual representation of the tool in the simulated virtual environment. The VR system is responsible of matching the physical state of the end effector and its virtual representation. Therefore, these tools can be seen as link between the real and the virtual world. However, this rather simple solution has two drawbacks. Firstly, special end effectors have to be designed for D. Allerkamp: Tactile Perception of Textiles in a Virtual-Reality System, COSMOS 10, pp. 1–4. c Springer-Verlag Berlin Heidelberg 2010 springerlink.com 

2

1 Introduction

different applications, e.g. an end effector imitating the handle of a scalpel is probably useless for flight training of aviators. Secondly, it is not possible to directly touch the virtual objects with the hand, which reduces the possible realism of a VR system. In order to simulate direct touch the parts of the skin that are in contact with virtual objects have to be appropriately stimulated with a tactile display. Although there exists a large variety of tactile displays only very little research has been done on the tactile simulation of real objects. The latter is the topic of the work at hand. In order to further narrow the research question, the virtual objects to be simulated have been restricted to textiles and only one of the many different types of tactile displays was investigated. This work was part of the EU funded HAPTEX project, which aimed at developing a VR system for the visual and haptic presentation of textiles (cf. [4, 1, 2]). The project was coordinated by the MIRALab at the University of Geneva which also contributed the physical based simulation system of the fabrics. The Biomedical Physics Group at the University of Exeter developed the tactile stimulator hardware and was responsible for the multimodal integration and validation. The PERCRO Laboratory at the Scuola Superiore di Studi Universitari e di Perfezionamento Sant’Anna in Pisa developed the force-feedback hardware which also carried the tactile stimulator hardware. The SmartWearLab at the Tampere University of Technology provided a selection of fabrics together with measurements appropriate for the project. And the Welfenlab at the Leibniz Universit¨at Hannover developed the haptic rendering software computing appropriate signals for the force feedback and the tactile devices. The target scenario of the project is depicted in Fig. 1.1. The virtual cloth is attached to a fixed stand. The user can touch, squeeze, rub and stretch the fabric with the thumb and index finger, feeling appropriate tactile stimulation at the fingertips. Reaching the target scenario is a very challenging task, because force-feedback and tactile simulation have to be integrated, posing more than only mechanical Fig. 1.1 HAPTEX final scenario from [1] (courtesy of PERCRO Laboratory)

1 Introduction

3

difficulties. Therefore, an intermediate scenario has been defined as a test bed for the tactile rendering. Here the fabric is completely fixed on a rigid support. The user can tactually explore the fabric’s surface with one fingertip. Force feedback is only available to define the plane of the surface. This research has been conducted at the Welfenlab of the Leibniz Universit¨at Hannover. The workgroup can draw upon many years of experience in the field of haptics in VR environments. Already in the Nineties a cooperation with the Virtual Reality Research Laboratory at the Volkswagen AG in Wolfsburg existed. From this cooperation the doctoral theses [3] and [5] emerged. Figure 1.2 shows the structure of this monograph. Chapter 1 provides an introduction to the problem tackled in this work and describes the context of each chapter. Chapters 2, 3 and 4 build the foundation for the following chapters. In Chap. 2 human perception is described from two different angles: in psychophysics perception is regarded as a black box and reactions to stimuli are

!"#$%&'( )*$&+,-.$/+*

!"#$%&': ;-