Digital Human Modeling: Second International Conference, ICDHM 2009, Held as Part of HCI International 2009 San Diego, CA, USA, July 19-24, 2009. (Proceedings. Lecture Notes in Computer Science)

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris...

Author: Vincent G. Duffy (ed.)

17 downloads 1113 Views 26MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

5620

Vincent G. Duffy (Ed.)

Digital Human Modeling Second International Conference, ICDHM 2009 Held as Part of HCI International 2009 San Diego, CA, USA, July 19-24, 2009 Proceedings

13

Volume Editor Vincent G. Duffy Purdue University School of Industrial Engineering 315 North Grant Street, Grissom Hall West Lafayette, IN 47907-2023, USA E-mail: [email protected]

Library of Congress Control Number: Applied for CR Subject Classification (1998): H.5, H.1, H.3, H.4.2, I.2-6, J.3 LNCS Sublibrary: SL 3 – Information Systems and Application, incl. Internet/Web and HCI ISSN ISBN-10 ISBN-13

0302-9743 3-642-02808-X Springer Berlin Heidelberg New York 978-3-642-02808-3 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2009 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12712076 06/3180 543210

Foreword

The 13th International Conference on Human–Computer Interaction, HCI International 2009, was held in San Diego, California, USA, July 19–24, 2009, jointly with the Symposium on Human Interface (Japan) 2009, the 8th International Conference on Engineering Psychology and Cognitive Ergonomics, the 5th International Conference on Universal Access in Human–Computer Interaction, the Third International Conference on Virtual and Mixed Reality, the Third International Conference on Internationalization, Design and Global Development, the Third International Conference on Online Communities and Social Computing, the 5th International Conference on Augmented Cognition, the Second International Conference on Digital Human Modeling, and the First International Conference on Human Centered Design. A total of 4,348 individuals from academia, research institutes, industry and governmental agencies from 73 countries submitted contributions, and 1,397 papers that were judged to be of high scientific quality were included in the program. These papers address the latest research and development efforts and highlight the human aspects of the design and use of computing systems. The papers accepted for presentation thoroughly cover the entire field of human–computer interaction, addressing major advances in knowledge and effective use of computers in a variety of application areas. This volume, edited by Vincent G. Duffy, contains papers in the thematic area of Digital Human Modeling, addressing the following major topics: • • • • • • •

Face, Head and Body Modeling Modeling Motion Modeling Behavior, Emotion and Cognition Human Modeling in Transport Applications Human Modeling Applications in Health and Rehabilitation Ergonomic and Industrial Applications Advances in Digital Human Modeling

The remaining volumes of the HCI International 2009 proceedings are: • • • • •

Volume 1, LNCS 5610, Human–Computer Interaction––New Trends (Part I), edited by Julie A. Jacko Volume 2, LNCS 5611, Human–Computer Interaction––Novel Interaction Methods and Techniques (Part II), edited by Julie A. Jacko Volume 3, LNCS 5612, Human–Computer Interaction––Ambient, Ubiquitous and Intelligent Interaction (Part III), edited by Julie A. Jacko Volume 4, LNCS 5613, Human–Computer Interaction––Interacting in Various Application Domains (Part IV), edited by Julie A. Jacko Volume 5, LNCS 5614, Universal Access in Human–Computer Interaction––Addressing Diversity (Part I), edited by Constantine Stephanidis

VI

Foreword

• • • • • • • • • • •

Volume 6, LNCS 5615, Universal Access in Human–Computer Interaction––Intelligent and Ubiquitous Interaction Environments (Part II), edited by Constantine Stephanidis Volume 7, LNCS 5616, Universal Access in Human–Computer Interaction––Applications and Services (Part III), edited by Constantine Stephanidis Volume 8, LNCS 5617, Human Interface and the Management of Information––Designing Information Environments (Part I), edited by Michael J. Smith and Gavriel Salvendy Volume 9, LNCS 5618, Human Interface and the Management of Information––Information and Interaction (Part II), edited by Gavriel Salvendy and Michael J. Smith Volume 10, LNCS 5619, Human Centered Design, edited by Masaaki Kurosu Volume 12, LNCS 5621, Online Communities and Social Computing, edited by A. Ant Ozok and Panayiotis Zaphiris Volume 13, LNCS 5622, Virtual and Mixed Reality, edited by Randall Shumaker Volume 14, LNCS 5623, Internationalization, Design and Global Development, edited by Nuray Aykin Volume 15, LNCS 5624, Ergonomics and Health Aspects of Work with Computers, edited by Ben-Tzion Karsh Volume 16, LNAI 5638, The Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience, edited by Dylan Schmorrow, Ivy Estabrooke and Marc Grootjen Volume 17, LNAI 5639, Engineering Psychology and Cognitive Ergonomics, edited by Don Harris

I would like to thank the Program Chairs and the members of the Program Boards of all thematic areas, listed below, for their contribution to the highest scientific quality and the overall success of HCI International 2009.

Ergonomics and Health Aspects of Work with Computers Program Chair: Ben-Tzion Karsh Arne Aarås, Norway Pascale Carayon, USA Barbara G.F. Cohen, USA Wolfgang Friesdorf, Germany John Gosbee, USA Martin Helander, Singapore Ed Israelski, USA Waldemar Karwowski, USA Peter Kern, Germany Danuta Koradecka, Poland Kari Lindström, Finland

Holger Luczak, Germany Aura C. Matias, Philippines Kyung (Ken) Park, Korea Michelle M. Robertson, USA Michelle L. Rogers, USA Steven L. Sauter, USA Dominique L. Scapin, France Naomi Swanson, USA Peter Vink, The Netherlands John Wilson, UK Teresa Zayas-Cabán, USA

Foreword

Human Interface and the Management of Information Program Chair: Michael J. Smith Gunilla Bradley, Sweden Hans-Jörg Bullinger, Germany Alan Chan, Hong Kong Klaus-Peter Fähnrich, Germany Michitaka Hirose, Japan Jhilmil Jain, USA Yasufumi Kume, Japan Mark Lehto, USA Fiona Fui-Hoon Nah, USA Shogo Nishida, Japan Robert Proctor, USA Youngho Rhee, Korea

Anxo Cereijo Roibás, UK Katsunori Shimohara, Japan Dieter Spath, Germany Tsutomu Tabe, Japan Alvaro D. Taveira, USA Kim-Phuong L. Vu, USA Tomio Watanabe, Japan Sakae Yamamoto, Japan Hidekazu Yoshikawa, Japan Li Zheng, P.R. China Bernhard Zimolong, Germany

Human–Computer Interaction Program Chair: Julie A. Jacko Sebastiano Bagnara, Italy Sherry Y. Chen, UK Marvin J. Dainoff, USA Jianming Dong, USA John Eklund, Australia Xiaowen Fang, USA Ayse Gurses, USA Vicki L. Hanson, UK Sheue-Ling Hwang, Taiwan Wonil Hwang, Korea Yong Gu Ji, Korea Steven Landry, USA

Gitte Lindgaard, Canada Chen Ling, USA Yan Liu, USA Chang S. Nam, USA Celestine A. Ntuen, USA Philippe Palanque, France P.L. Patrick Rau, P.R. China Ling Rothrock, USA Guangfeng Song, USA Steffen Staab, Germany Wan Chul Yoon, Korea Wenli Zhu, P.R. China

Engineering Psychology and Cognitive Ergonomics Program Chair: Don Harris Guy A. Boy, USA John Huddlestone, UK Kenji Itoh, Japan Hung-Sying Jing, Taiwan Ron Laughery, USA Wen-Chin Li, Taiwan James T. Luxhøj, USA

Nicolas Marmaras, Greece Sundaram Narayanan, USA Mark A. Neerincx, The Netherlands Jan M. Noyes, UK Kjell Ohlsson, Sweden Axel Schulte, Germany Sarah C. Sharples, UK

VII

VIII

Foreword

Neville A. Stanton, UK Xianghong Sun, P.R. China Andrew Thatcher, South Africa

Matthew J.W. Thomas, Australia Mark Young, UK

Universal Access in Human–Computer Interaction Program Chair: Constantine Stephanidis Julio Abascal, Spain Ray Adams, UK Elisabeth André, Germany Margherita Antona, Greece Chieko Asakawa, Japan Christian Bühler, Germany Noelle Carbonell, France Jerzy Charytonowicz, Poland Pier Luigi Emiliani, Italy Michael Fairhurst, UK Dimitris Grammenos, Greece Andreas Holzinger, Austria Arthur I. Karshmer, USA Simeon Keates, Denmark Georgios Kouroupetroglou, Greece Sri Kurniawan, USA

Patrick M. Langdon, UK Seongil Lee, Korea Zhengjie Liu, P.R. China Klaus Miesenberger, Austria Helen Petrie, UK Michael Pieper, Germany Anthony Savidis, Greece Andrew Sears, USA Christian Stary, Austria Hirotada Ueda, Japan Jean Vanderdonckt, Belgium Gregg C. Vanderheiden, USA Gerhard Weber, Germany Harald Weber, Germany Toshiki Yamaoka, Japan Panayiotis Zaphiris, UK

Virtual and Mixed Reality Program Chair: Randall Shumaker Pat Banerjee, USA Mark Billinghurst, New Zealand Charles E. Hughes, USA David Kaber, USA Hirokazu Kato, Japan Robert S. Kennedy, USA Young J. Kim, Korea Ben Lawson, USA

Gordon M. Mair, UK Miguel A. Otaduy, Switzerland David Pratt, UK Albert “Skip” Rizzo, USA Lawrence Rosenblum, USA Dieter Schmalstieg, Austria Dylan Schmorrow, USA Mark Wiederhold, USA

Internationalization, Design and Global Development Program Chair: Nuray Aykin Michael L. Best, USA Ram Bishu, USA Alan Chan, Hong Kong Andy M. Dearden, UK

Susan M. Dray, USA Vanessa Evers, The Netherlands Paul Fu, USA Emilie Gould, USA

Foreword

Sung H. Han, Korea Veikko Ikonen, Finland Esin Kiris, USA Masaaki Kurosu, Japan Apala Lahiri Chavan, USA James R. Lewis, USA Ann Light, UK James J.W. Lin, USA Rungtai Lin, Taiwan Zhengjie Liu, P.R. China Aaron Marcus, USA Allen E. Milewski, USA

Elizabeth D. Mynatt, USA Oguzhan Ozcan, Turkey Girish Prabhu, India Kerstin Röse, Germany Eunice Ratna Sari, Indonesia Supriya Singh, Australia Christian Sturm, Spain Adi Tedjasaputra, Singapore Kentaro Toyama, India Alvin W. Yeo, Malaysia Chen Zhao, P.R. China Wei Zhou, P.R. China

Online Communities and Social Computing Program Chairs: A. Ant Ozok, Panayiotis Zaphiris Chadia N. Abras, USA Chee Siang Ang, UK Amy Bruckman, USA Peter Day, UK Fiorella De Cindio, Italy Michael Gurstein, Canada Tom Horan, USA Anita Komlodi, USA Piet A.M. Kommers, The Netherlands Jonathan Lazar, USA Stefanie Lindstaedt, Austria

Gabriele Meiselwitz, USA Hideyuki Nakanishi, Japan Anthony F. Norcio, USA Jennifer Preece, USA Elaine M. Raybourn, USA Douglas Schuler, USA Gilson Schwartz, Brazil Sergei Stafeev, Russia Charalambos Vrasidas, Cyprus Cheng-Yen Wang, Taiwan

Augmented Cognition Program Chair: Dylan D. Schmorrow Andy Bellenkes, USA Andrew Belyavin, UK Joseph Cohn, USA Martha E. Crosby, USA Tjerk de Greef, The Netherlands Blair Dickson, UK Traci Downs, USA Julie Drexler, USA Ivy Estabrooke, USA Cali Fidopiastis, USA Chris Forsythe, USA Wai Tat Fu, USA Henry Girolamo, USA

Marc Grootjen, The Netherlands Taro Kanno, Japan Wilhelm E. Kincses, Germany David Kobus, USA Santosh Mathan, USA Rob Matthews, Australia Dennis McBride, USA Robert McCann, USA Jeff Morrison, USA Eric Muth, USA Mark A. Neerincx, The Netherlands Denise Nicholson, USA Glenn Osga, USA

IX

X

Foreword

Dennis Proffitt, USA Leah Reeves, USA Mike Russo, USA Kay Stanney, USA Roy Stripling, USA Mike Swetnam, USA Rob Taylor, UK

Maria L.Thomas, USA Peter-Paul van Maanen, The Netherlands Karl van Orden, USA Roman Vilimek, Germany Glenn Wilson, USA Thorsten Zander, Germany

Digital Human Modeling Program Chair: Vincent G. Duffy Karim Abdel-Malek, USA Thomas J. Armstrong, USA Norm Badler, USA Kathryn Cormican, Ireland Afzal Godil, USA Ravindra Goonetilleke, Hong Kong Anand Gramopadhye, USA Sung H. Han, Korea Lars Hanson, Sweden Pheng Ann Heng, Hong Kong Tianzi Jiang, P.R. China

Kang Li, USA Zhizhong Li, P.R. China Timo J. Määttä, Finland Woojin Park, USA Matthew Parkinson, USA Jim Potvin, Canada Rajesh Subramanian, USA Xuguang Wang, France John F. Wiechel, USA Jingzhou (James) Yang, USA Xiu-gan Yuan, P.R. China

Human Centered Design Program Chair: Masaaki Kurosu Gerhard Fischer, USA Tom Gross, Germany Naotake Hirasawa, Japan Yasuhiro Horibe, Japan Minna Isomursu, Finland Mitsuhiko Karashima, Japan Tadashi Kobayashi, Japan

Kun-Pyo Lee, Korea Loïc Martínez-Normand, Spain Dominique L. Scapin, France Haruhiko Urokohara, Japan Gerrit C. van der Veer, The Netherlands Kazuhiko Yamazaki, Japan

In addition to the members of the Program Boards above, I also wish to thank the following volunteer external reviewers: Gavin Lew from the USA, Daniel Su from the UK, and Ilia Adami, Ioannis Basdekis, Yannis Georgalis, Panagiotis Karampelas, Iosif Klironomos, Alexandros Mourouzis, and Stavroula Ntoa from Greece. This conference could not have been possible without the continuous support and advice of the Conference Scientific Advisor, Prof. Gavriel Salvendy, as well as the dedicated work and outstanding efforts of the Communications Chair and Editor of HCI International News, Abbas Moallem.

Foreword

XI

I would also like to thank for their contribution toward the organization of the HCI International 2009 conference the members of the Human–Computer Interaction Laboratory of ICS-FORTH, and in particular Margherita Antona, George Paparoulis, Maria Pitsoulaki, Stavroula Ntoa, and Maria Bouhli. Constantine Stephanidis

HCI International 2011

The 14th International Conference on Human–Computer Interaction, HCI International 2011, will be held jointly with the affiliated conferences in the summer of 2011. It will cover a broad spectrum of themes related to human–computer interaction, including theoretical issues, methods, tools, processes and case studies in HCI design, as well as novel interaction techniques, interfaces and applications. The proceedings will be published by Springer. More information about the topics, as well as the venue and dates of the conference, will be announced through the HCI International Conference series website: http://www.hci-international.org/

General Chair Professor Constantine Stephanidis University of Crete and ICS-FORTH Heraklion, Crete, Greece Email: [email protected]

Table of Contents

Part I: Face, Head and Body Modeling Static and Dynamic Human Shape Modeling . . . . . . . . . . . . . . . . . . . . . . . . Zhiqing Cheng and Kathleen Robinette

3

An Advanced Modality of Visualization and Interaction with Virtual Models of the Human Body . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lucio T. De Paolis, Marco Pulimeno, and Giovanni Aloisio

13

3D Body Scanning’s Contribution to the Use of Apparel as an Identity Construction Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marie-Eve Faust and Serge Carrier

19

Facial Shape Analysis and Sizing System . . . . . . . . . . . . . . . . . . . . . . . . . . . . Afzal Godil

29

Facial Gender Classification Using LUT-Based Sub-images and DIE . . . . Jong-Bae Jeon, Sang-Hyeon Jin, Dong-Ju Kim, and Kwang-Seok Hong

36

Anthropometric Measurement of the Hands of Chinese Children . . . . . . . Linghua Ran, Xin Zhang, Chuzhi Chao, Taijie Liu, and Tingting Dong

46

Comparisons of 3D Shape Clustering with Different Face Area Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jianwei Niu, Zhizhong Li, and Song Xu

55

Block Division for 3D Head Shape Clustering . . . . . . . . . . . . . . . . . . . . . . . . Jianwei Niu, Zhizhong Li, and Song Xu

64

Joint Coupling for Human Shoulder Complex . . . . . . . . . . . . . . . . . . . . . . . . Jingzhou (James) Yang, Xuemei Feng, Joo H. Kim, Yujiang Xiang, and Sudhakar Rajulu

72

Part II: Modeling Motion Development of a Kinematic Hand Model for Study and Design of Hose Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Thomas J. Armstrong, Christopher Best, Sungchan Bae, Jaewon Choi, D. Christian Grieshaber, Daewoo Park, Charles Woolley, and Wei Zhou

85

XVI

Table of Contents

Generation of Percentile Values for Human Joint Torque Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Florian Engstler and Heiner Bubb

95

Adaptive Motion Pattern Recognition: Implementing Playful Learning through Embodied Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anja Hashagen, Christian Zabel, Heidi Schelhowe, and Saeed Zare

105

A Multi-functional Visualization System for Motion Captured Human Body Based on Virtual Reality Technology . . . . . . . . . . . . . . . . . . . . . . . . . . Qichang He, Lifeng Zhang, Xiumin Fan, and Yong Hu

115

Augmented Practice Mirror: A Self-learning Support System of Physical Motion with Real-Time Comparison to Teacher’s Model . . . . . . . . . . . . . . Itaru Kuramoto, Yoshikazu Inagaki, Yu Shibuya, and Yoshihiro Tsujino

123

Video-Based Human Motion Estimation System . . . . . . . . . . . . . . . . . . . . . Mariofanna Milanova and Leonardo Bocchi

132

Virtual Human Hand: Grasping and Simulation . . . . . . . . . . . . . . . . . . . . . . Esteban Pe˜ na-Pitarch, Jingzhou (James) Yang, and Karim Abdel-Malek

140

Harmonic Gait under Primitive DOF for Biped Robot . . . . . . . . . . . . . . . . Shigeki Sugiyama

150

Problems Encountered in Seated Arm Reach Posture Reconstruction: Need for a More Realistic Spine and Upper Limb Kinematic Model . . . . . Xuguang Wang Intelligent Motion Tracking by Combining Specialized Algorithms . . . . . . Matthias Weber

160 170

Part III: Modeling Behavior, Emotion and Cognition Ambient Compass: One Approach to Model Spatial Relations . . . . . . . . . Petr Aksenov, Geert Vanderhulst, Kris Luyten, and Karin Coninx

183

A Comprehension Based Cognitive Model of Situation Awareness . . . . . . Martin R.K. Baumann and Josef F. Krems

192

A Probabilistic Approach for Modeling Human Behavior in Smart Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Christoph Burghardt and Thomas Kirste

202

PERMUTATION: A Corpus-Based Approach for Modeling Personality and Multimodal Expression of Affects in Virtual Characters . . . . . . . . . . . Céline Clavel and Jean-Claude Martin

211

Table of Contents

XVII

Workload Assessment in Field Using the Ambulatory CUELA System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rolf Ellegast, Ingo Hermanns, and Christoph Schiefer

221

Computational Nonlinear Dynamics Model of Percept Switching with Ambiguous Stimuli . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Norbert F¨ urstenau

227

A Computational Implementation of a Human Attention Guiding Mechanism in MIDAS v5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Brian F. Gore, Becky L. Hooey, Christopher D. Wickens, and Shelly Scott-Nash

237

Towards a Computational Model of Perception and Action in Human Computer Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pascal Haazebroek and Bernhard Hommel

247

The Five Commandments of Activity-Aware Ubiquitous Computing Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nasim Mahmud, Jo Vermeulen, Kris Luyten, and Karin Coninx

257

What the Eyes Reveal: Measuring the Cognitive Workload of Teams . . . . Sandra P. Marshall

265

User Behavior Mining for On-Line GUI Adaptation . . . . . . . . . . . . . . . . . . Wei Pan, Yiqiang Chen, and Junfa Liu

275

Modeling Human Actors in an Intelligent Automated Warehouse . . . . . . . Davy Preuveneers and Yolande Berbers

285

Bridging the Gap between HCI and DHM: The Modeling of Spatial Awareness within a Cognitive Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . Bryan Robbins, Daniel Carruth, and Alexander Morais Behavior-Sensitive User Interfaces for Smart Environments . . . . . . . . . . . . Veit Schwartze, Sebastian Feuerstack, and Sahin Albayrak Non-intrusive Personalized Mental Workload Evaluation for Exercise Intensity Measure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . N. Luke Thomas, Yingzi Du, Tron Artavatkun, and Jin-hua She

295 305

315

Incorporating Cognitive Aspects in Digital Human Modeling . . . . . . . . . . Peter Thorvald, Dan H¨ ogberg, and Keith Case

323

Workload-Based Assessment of a User Interface Design . . . . . . . . . . . . . . . Patrice D. Tremoulet, Patrick L. Craven, Susan Harkness Regli, Saki Wilcox, Joyce Barton, Kathleeen Stibler, Adam Gifford, and Marianne Clark

333

XVIII

Table of Contents

Part IV: Human Modeling in Transport Applications A Simple Simulation Predicting Driver Behavior, Attitudes and Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Aladino Amantini and Pietro Carlo Cacciabue

345

Nautical PSI - Virtual Nautical Officers as Test Drivers in Ship Bridge Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ulrike Br¨ uggemann and Stefan Strohschneider

355

Determining Cockpit Dimensions and Associative Dimensions between Components in Cockpit of Ultralight Plane for Taiwanese . . . . . . . . . . . . . Dengchuan Cai, Lan-Ling Huang, Tesheng Liu, and Manlai You

365

Multilevel Analysis of Human Performance Models in Safety-Critical Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jeronimo Dzaack and Leon Urbas

375

Development of a Driver Model in Powered Wheelchair Operation . . . . . . Takuma Ito, Takenobu Inoue, Motoki Shino, and Minoru Kamata A Model of Integrated Operator-System Separation Assurance and Collision Avoidance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Steven J. Landry and Amit V. Lagu Modeling Pilot and Driver Behavior for Human Error Simulation . . . . . . Andreas L¨ udtke, Lars Weber, Jan-Patrick Osterloh, and Bertram Wortelen

384

394 403

Further Steps towards Driver Modeling According to the Bayesian Programming Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Claus M¨ obus and Mark Eilers

413

Probabilistic and Empirical Grounded Modeling of Agents in (Partial) Cooperative Traffic Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Claus M¨ obus, Mark Eilers, Hilke Garbe, and Malte Zilinski

423

A Contribution to Integrated Driver Modeling: A Coherent Framework for Modeling Both Non-routine and Routine Elements of the Driving Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Andreas Mihalyi, Barbara Deml, and Thomas Augustin The New BMW iDrive – Applied Processes and Methods to Assure High Usability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bernhard Niedermaier, Stephan Durach, Lutz Eckstein, and Andreas Keinath Method to Evaluate Driver’s Workload in Real Road Context . . . . . . . . . . Annie Pauzié

433

443

453

Table of Contents

Intelligent Agents for Training On-Board Fire Fighting . . . . . . . . . . . . . . . Karel van den Bosch, Maaike Harbers, Annerieke Heuvelink, and Willem van Doesburg

XIX

463

Part V: Human Modeling Applications in Health and Rehabilitation Eprescribing Initiatives and Knowledge Acquisition in Ambulatory Care . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ashley J. Benedict, Jesse C. Crosson, Akshatha Pandith, Robert Hannemann, Lynn A. Nuti, and Vincent G. Duffy Using 3D Head and Respirator Shapes to Analyze Respirator Fit . . . . . . Kathryn M. Butler Hyperkalemia vs. Ischemia Effects in Fast or Unstable Pacing: A Cardiac Simulation Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ioanna Chouvarda and Nicos Maglaveras Learning from Risk Assessment in Radiotherapy . . . . . . . . . . . . . . . . . . . . . Enda F. Fallon, Liam Chadwick, and Wil van der Putten Simulation-Based Discomfort Prediction of the Lower Limb Handicapped with Prosthesis in the Climbing Tasks . . . . . . . . . . . . . . . . . . Yan Fu, Shiqi Li, Mingqiang Yin, and Yueqing Bian Application of Human Modeling in Health Care Industry . . . . . . . . . . . . . Lars Hanson, Dan H¨ ogberg, Daniel Lundstr¨ om, and Maria W˚ arell A Simulation Approach to Understand the Viability of RFID Technology in Reducing Medication Dispensing Errors . . . . . . . . . . . . . . . . Esther Jun, Jonathan Lee, and Xiaobo Shi Towards a Visual Representation of the Effects of Reduced Muscle Strength in Older Adults: New Insights and Applications for Design and Healthcare . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . David Loudon and Alastair S. Macdonald A Novel Approach to CT Scans’ Interpretation via Incorporation into a VR Human Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sophia Sakellariou, Vassilis Charissis, Ben M. Ward, David Chanock, and Paul Anderson The Performance of BCMA-Aided Healthcare Service: Implementation Factors and Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Renran Tian, Vincent G. Duffy, Carol Birk, Steve R. Abel, and Kyle Hultgren

475

483

492

502

512

521

531

540

550

560

XX

Table of Contents

On Improving Provider Decision Making with Enhanced Computerized Clinical Reminders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sze-jung Wu, Mark Lehto, Yuehwern Yih, Jason J. Saleem, and Bradley Doebbeling Facial Shape Variation of U.S. Respirator Users . . . . . . . . . . . . . . . . . . . . . . Ziqing Zhuang, Dennis Slice, Stacey Benson, Douglas Landsittel, and Dennis Viscusi

569

578

Part VI: Ergonomic and Industrial Applications Method for Movement and Gesture Assessment (MMGA) in Ergonomics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Giuseppe Andreoni, Marco Mazzola, Oriana Ciani, Marta Zambetti, Maximiliano Romero, Fiammetta Costa, and Ezio Preatoni

591

Complexity of Sizing for Space Suit Applications . . . . . . . . . . . . . . . . . . . . . Elizabeth Benson and Sudhakar Rajulu

599

Impact of Force Feedback on Computer Aided Ergonomic Analyses . . . . . H. Onan Demirel and Vincent G. Duffy

608

A Methodology for Modeling the Influence of Construction Machinery Operators on Productivity and Fuel Consumption . . . . . . . . . . . . . . . . . . . . Reno Filla Human Head 3D Dimensions Measurement for the Design of Helmets . . . Fenfei Guo, Lijing Wang, and Dayong Dong

614 624

Realistic Elbow Flesh Deformation Based on Anthropometrical Data for Ergonomics Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Setia Hermawati and Russell Marshall

632

Database-Driven Grasp Synthesis and Ergonomic Assessment for Handheld Product Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Keisuke Kawaguchi, Yui Endo, and Satoshi Kanai

642

Within and Between-Subject Reliability Using Classic Jack for Ergonomic Assessments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Brian McInnes, Allison Stephens, and Jim Potvin

653

Human Head Modeling and Personal Head Protective Equipment: A Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jingzhou (James) Yang, Jichang Dai, and Ziqing Zhuang

661

Part VII: Advances in Digital Human Modeling HADRIAN: Fitting Trials by Digital Human Modeling . . . . . . . . . . . . . . . . Keith Case, Russell Marshall, Dan H¨ ogberg, Steve Summerskill, Diane Gyi, and Ruth Sims

673

Table of Contents

The Pluses and Minuses of Obtaining Measurements from Digital Scans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ravindra S. Goonetilleke, Channa P. Witana, Jianhui Zhao, and Shuping Xiong

XXI

681

Auto-calibration of a Laser 3D Color Digitization System . . . . . . . . . . . . . Xiaojie Li, Bao-zhen Ge, Dan Zhao, Qing-guo Tian, and K. David Young

691

Virtual Task Simulation for Inclusive Design . . . . . . . . . . . . . . . . . . . . . . . . Russell Marshall, Keith Case, Steve Summerskill, Ruth Sims, Diane Gyi, and Peter Davis

700

Data Mining of Image Segments Data with Reduced Neurofuzzy System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Deok Hee Nam and Edward Asikele

710

The Impact of Change in Software on Satisfaction: Evaluation Using Critical Incident Technique (CIT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Akshatha Pandith, Mark Lehto, and Vincent G. Duffy

717

Validation of the HADRIAN System Using an ATM Evaluation Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Steve J. Summerskill, Russell Marshall, Keith Case, Diane E. Gyi, Ruth E. Sims, and Peter Davis A 3D Method for Fit Assessment of a Sizing System . . . . . . . . . . . . . . . . . . Jiang Wu, Zhizhong Li, and Jianwei Niu Analyzing the Effects of a BCMA in Inter-Provider Communication, Coordination and Cooperation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gulcin Yucel, Bo Hoege, Vincent G. Duffy, and Matthias Roetting Fuzzy Logic in Exploring Data Effects: A Way to Unveil Uncertainty in EEG Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fang Zheng, Bin Hu, Li Liu, Tingshao Zhu, Yongchang Li, and Yanbin Qi Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

727

737

744

754

765

Static and Dynamic Human Shape Modeling Zhiqing Cheng1 and Kathleen Robinette2 1

Infoscitex Corporation, 4027 Colonel Glenn Highway, Suite 210, Dayton, OH 45431, USA 2 711th Human Performance Wing, Air Force Research Laboratory, 2800 Q Street, Wright-Patterson AFB, OH 45433, USA {zhiqing.cheng,kathleen.robinette}@wpafb.af.mil

Abstract. Recent developments in static human shape modeling based on range scan data and dynamic human shape modeling from video imagery are reviewed. The topics discussed include shape description, surface registration, hole filling, shape characterization, and shape reconstruction for static modeling and pose identification, skeleton modeling, shape deformation, motion tracking, dynamic shape capture and reconstruction, and animation for dynamic modeling. A new method for human shape modeling is introduced. Keywords: Human body, shape modeling, pose, animation.

1 Introduction From the perspective of the motion status of the subject to be modeled, human shape modeling can be classified as either static or dynamic. Static shape modeling creates a model to describe human shape at a particular pose and addresses shape description, registration, hole filling, shape variation characterization, and shape reconstruction. Dynamic shape modeling addresses the shape variations due to pose changes (pose identification, skeleton modeling, and shape deformation) or while the subject is in motion (motion tracking, shape capture, shape reconstruction, and animation). Extensive investigations have been performed on human shape modeling [1-10]. Recent developments of human shape modeling, in particular, static shape modeling based on range scan data and dynamic shape modeling from video imagery, are reviewed in this paper. A new method for human shape modeling based on body segmentation and contour lines is introduced.

2 Static Shape Modeling 2.1 Shape Description Shape description is a fundamental problem for human shape modeling. Traditional anthropometry is based on a set of measurements corresponding to linear distances between anatomical landmarks and circumference values at predefined locations. These measurements provide limited information about the human body shape [11]. With the advances in surface digitization technology, a 3-D surface scan of the whole body can be acquired in a few seconds. While whole body 3-D surface scan provides V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 3–12, 2009. © Springer-Verlag Berlin Heidelberg 2009

4

Z. Cheng and K. Robinette

very detailed description of the body shape, the verbose scan data cannot be used directly for shape analysis. Therefore, it is necessary to convert 3-D scans to a form of compact representation. For searching and mining from a large 3-D scan database, Robinette [12] investigated 3-D shape descriptors where the Paquet Shape Descriptor (PSD) developed by Paquet and Rioux [13] was examined in detail. While PSD is able to discriminate or characterize different human shapes, it is not invertible. In other words, it is impossible to reconstruct a human shape from PSD. An ideal human shape descriptor should be concise, unique, and complete for human shape description, efficient for shape indexing and searching, and invertible for shape reconstruction. Finding such a descriptor still remains a challenge. Alternatively, various graphic elements or graphic representation methods can be used to describe the human shape. For instance, Allen et al [2] and Anguelov et al [7] basically dealt directly with the vertices or polygons of a scanned surface for shape description. Allen et al [1] used subdivision surface in their pose modeling. Ben Azouz et al [6] utilized volumetric representation to convert vertices to voxels in their human shape modeling. While these methods guarantee reconstruction, they are not quite efficient for shape identification, discrimination, and searching. 2.2 Surface Registration Surface registration or point-to-point correspondence among the scan data of different subjects is essential to many problems such as the study of human shape variability [2, 14] and pose modeling and animation [1, 7] where multiple subjects or multiple poses are involved. One methodology for establishing point-to-point correspondence among different scan data sets or models is usually called non-rigid registration. Given a set of markers between two meshes, non-rigid registration brings the meshes into close alignment while simultaneously aligning the markers. Allen et al [1, 2] solved the correspondence problem between subjects by deforming a template model which is a hole-free, artist-generated mesh to fit individual scans. The resulting individually fitted scans or individual “models” all have the same number of triangles and point-to-point correspondences. The fitting process relies on a set of anthropometric landmarks provided in the CAESAR (Civilian American and European Surface Anthropometry Resource) database [15]. Anguelov et al [16] developed an unsupervised algorithm for registering 3-D surface scans of an object among different poses undergoing significant deformations. The algorithm called Correlated Correspondence (CC) does not use markers, nor does it assume prior knowledge about object shape, the dynamics of its deformation, or scan alignment. The algorithm registers two meshes with significant deformations by optimizing a joint probabilistic model over all point-to-point correspondences between them. This model enforces preservation of local mesh geometry as well as global constraints that capture the preservation of geodesic distance between corresponding point pairs. To obtain the markers for the non-rigid registration, Anguelov et al [7] then used the CC algorithm to compute the consistent embedding of each instance mesh (the mesh of a particular pose) into the template mesh (the mesh of a reference pose). Ben Azouz et al [14] used a volumetric representation of human 3-D surface to establish the correspondences between the scan data of different subjects. By converting their polygonal

Static and Dynamic Human Shape Modeling

5

mesh descriptions to a volumetric representation, the 3D scans of different subjects are aligned inside a volume of fixed dimensions, which is sampled to a set of voxels. A human 3-D shape is then characterized by an array of signed distances between the voxels and their nearest point on the body surface. Correspondence is achieved by comparing for each voxel the signed distances attributed to different subjects without using anatomical landmarks. 2.3 Hole Filling Surfaces acquired with scanners are typically incomplete and contain holes. Filling a hole is a challenging problem in its own right, as discussed by Davis et al. [18]. A common way to complete a hole is to fill it with a smooth surface patch that meets the boundary conditions of the hole. While these methods fill holes in a smooth manner, which is reasonable in some areas such as the top of the head and possibly in the underarm, other areas should not be filled smoothly. Therefore, Allen et al [2] developed a method that maps a surface from a template model to the hole area. Alternatively, hole-filling can be based on the contour lines of a scan surface [14]. 2.4 Shape Variation Characterization The human body comes in all shapes and sizes. Characterizing human shape variation is traditionally the subject of anthropometry—the study of human body measurement. The sparse measurements of traditional anthropometric shape characterization curtail its ability to capture the detailed shape variations needed for realism. While characterizing human shape variation based on a 3-D range scan could capture the details of shape variation, the method relies on three conditions: noise elimination, hole-filling and surface completion, and point-to-point correspondence. Also, whole body scanners generate verbose data that cannot be used directly for shape variation analysis. Therefore, it is necessary to convert 3-D scans to a compact representation that retains information of the body shape. Principal components analysis (PCA) is a potential solution to the problem. Allen et al [2] captured the variability of human shape by performing PCA over the displacements of the points from the template surface to an instance surface. Anguelov et al [7] also used PCA to characterize the shape deformation and then used the principal components for shape completion. Ben Azouz et al [14] applied PCA to the volumetric models where the vector is formed by the signed distance from a voxel to the surface of the model. In order to explore the variations of the human body with intuitive control parameters (e.g., height, weight, age, and sex), Allen et al [2] showed how to relate several variables simultaneously by learning a linear mapping between the control parameters and the PCA weights. Ben Azouz et al [6,14,21] attempted to link the principal modes to some intuitive body shape variations by visualizing the first five modes of variation and gave interpretations of these modes. While PCA is shown to be effective in characterizing global shape variations, it may smear local variations for which other methods (e.g., wavelets) may be more effective.

6


2.5 Shape Reconstruction Given a number of scan data sets of different subjects, a novel human shape can be created that has resemblance to the samples but is not the exact copy of any existing ones. This can be realized in four ways. • Interpolation or morphing. One shape can be gradually morphed to another by interpolating between their vertices or other graphic entities [2]. In order to create a faithful intermediate shape between two individuals, it is critical that all features are well-aligned; otherwise, features will cross-fade instead of moving. • Reconstruction from eigen-space. After PCA analysis, the features of sample shapes are characterized by eigen-vectors or eigen-persons which form an eignespace. Any new shape model can be generated from this space by combining a number of eigen-models with appropriate weighting factors [14]. • Feature-based synthesis. Once the relationship between human anthropometric features and eigen-vectors is established, a new shape model can be constructed from the eigen-space with desired features by editing multiple correlated attributes, such as height and weight [2] or fat percentage and hip-to-waist ratio [4]. • Marker-only matching. Marker-only matching can be considered as a way of reconstruction with provided markers [2]. This is important for many applications such as deriving a model from video imagery, since marker data can be obtained using less expensive equipment than a laser range scanner.

3 Pose Change Modeling During pose changing or body movement, muscles, bones, and other anatomical structures continuously shift and change the shape of the body. For pose modeling, scanning the subject in every pose is impractical; instead, body shape can be scanned in a set of key poses, and then the body shapes corresponding to intermediate poses are determined by smoothly interpolating among these poses. The issues involved in pose modeling include pose definition and identification, skeleton model derivation, shape deformation (skinning), and pose mapping. 3.1 Pose Definition and Identification The human body can assume various poses. In order to have a common basis for pose modeling, a distinct, unique description of difference poses is required. Since it is impossible to collect the data or create template models for all possible poses, it is necessary to define a set of standard, typical poses. This is pose definition. A convention for pose definition is yet to be established. One approach is to use joint angle changes as the measures to characterize human pose changing and gross motion. This means that poses can be defined by joint angles. By defining poses and motion in such a way, the body shape variations caused by pose changing and motion will consist of both rigid and non-rigid deformation. Rigid deformation is associated with the orientation and position of segments that connect joints. Non-rigid deformation is related to the changes in shape of soft tissues associated with segments in motion, which, however, excludes local deformation caused by muscle action alone. One


7

method for measuring and defining joint angles is using a skeleton model. In the model, the human body is divided into multiple segments according to major joints of the body, each segment is represented by a rigid linkage, and an appropriate joint is placed between the two corresponding linkages. Given a set of scan data, imagery, or photos, the determination or identification of the corresponding pose can be done by fitting a skeleton model to the data set. The skeleton model derivation will be discussed in the following section. Alternatively, there are several methods for pose identification that are not based on skeleton models. Mittal et al [22] studied human body pose estimation using silhouette shape analysis. Cohen and Li [23] proposed an approach for inferring the body posture using a 3-D visual-hull constructed from a set of silhouettes. 3.2 Skeleton Model Allen et al [1] constructed a kinematic skeleton model to identify the pose of a scan data set using markers captured during range scanning. Anguelov et al [24] developed an algorithm that automatically recovers from 3-D range data a decomposition of the object into approximately rigid parts, the location of the parts in the different poses, and the articulated object skeleton linking the parts. Robertson and Trucco [25] developed an evolutionary approach to estimating upper-body posture from multi-view markerless sequences. Sundaresan et al [26] proposed a general approach that uses Laplacian eigen-maps and a graphical model of the human body to segment 3-D voxel data of humans into different articulated chains. 3.3 Body Deformation Modeling Body deformation modeling is also referred to as skinning in animation. Two main approaches for modeling body deformations are anatomical modeling and examplebased modeling. The anatomical modeling is based on an accurate representation of the major bones, muscles, and other interior structures of the body [27]. These structures are deformed as necessary when the body moves, and a skin model is wrapped around the underlying anatomy to obtain the final geometry of the body shape. The finite element method is the primary modeling technique used for anatomical modeling. In the example-based approach, a model of some body part in several different poses with the same underlying mesh structure can be generated by an artist. These poses are correlated to various degrees of freedom, such as joint angles. An animator can then supply values for the degrees of freedom of a new pose and the body shape for that new pose is interpolated appropriately. Lewis et al [28] and Sloan et al [29] developed similar techniques for applying example-based approaches to meshes. Instead of using artist-generated models, recent work on the example-based modeling uses range-scan data. Allen et al [1] presented an example-based method for calculating skeleton-driven body deformations. Their example data consists of range scans of a human body in a variety of poses. Using markers captured during range scanning, a kinematic skeleton is constructed first to identify the pose of each scan. Then a mutually consistent parameterization of all the scans is constructed using a posable subdivision surface template. Anguelov et al [7] developed a method that incorporates both articulated

8


and non-rigid deformations. A pose deformation model was constructed from training scan data that derives the non-rigid surface deformation as a function of the pose of the articulated skeleton. A separate model of shape variation was derived from the training data also. The two models were combined to produce a 3D surface model with realistic muscle deformation for different people in different poses. The method (model) is referred to as the SCAPE (Shape Completion and Animation for People). For pose modeling, it is impossible to acquire the pose deformation for each person at each pose. Instead, pose deformation can be transferred from one person to another for a given pose. Anguelov et al [7] addressed this issue by integrating a pose model with a shape model reconstructed from eigen-space. As such, they were able to generate a mesh for any body shape in their PCA space in any pose.

4 Shape Modeling of Human in Motion 4.1 Motion Tracking Human motion tracking or capturing is an area that has attracted a lot of study and investigation. Today’s performance of off-the-shelf computer hardware enables marker-free, non-intrusive optical tracking of the human body. For example, Theobalt et al [30] developed a system to capture human motion at interactive frame rates without the use of markers or scene-introducing devices. The algorithms for 2-D computer vision and 3-D volumetric scene reconstruction were applied directly to the image data. A person was recorded by multiple synchronized cameras, and a multilayer hierarchical kinematic skeleton was fitted into each frame in a two-stage process. 4.2 Dynamic Shape Capture During dynamic activities, the surface of the human body moves in many subtle but visually significant ways: bending, bulging, jiggling, and stretching. Park and Hodgins [8] developed a technique for capturing and animating those motions using a commercial motion capture system with approximately 350 markers. Supplemented with a detailed, actor specific surface model, the motion of the skin was then computed by segmenting the markers into the motion of a set of rigid parts and a residual deformation. Sand et al [5] developed a method (a needle model) for the acquisition of deformable human geometry from silhouettes. Their technique uses a commercial tracking system to determine the motion of the skeleton and then estimates geometry for each bone using constraints provided by the silhouettes from one or more cameras. 4.3 Shape Reconstruction from Imagery Data • From Photos. Seo et al [31] presented a data-driven shape model for reconstructing human body models from one or more 2-D photos. A data-driven, parameterized deformable model acquired from a collection of range scans of a real human body is used to complement the image-based reconstruction by leveraging the quality, shape, and statistical information accumulated from multiple shapes of rangescanned people.


9

• From Video Sequences. One recent work was done by Balan et al [10] that proposed a method for recovering human shape models directly from images. Specifically, the human body shape is represented by the SCAPE [7] and the parameters of the model are directly estimated from image data. A cost function between image observations and a hypothesized mesh is defined and the problem is formulated as an optimization. 4.4 Animation The animation of the subject can be realized by displaying a series of human shape models for a prescribed sequence of poses. Hilton et al [3] built a framework for construction of animated models from the captured surface shape of real objects. Seo et al [4] developed a synthesizer where for any synthesized model, the underlying bone and skin structure is properly adjusted, so that the model remains completely animatable using the underlying skeleton. Aguiar et al [9] developed a novel versatile, fast and simple framework to generate high quality animations of scanned human characters from input motion data. The method is purely mesh-based and can easily transfer motions between human subjects of completely different shape and proportions.

5 A New Method In the static human shape modeling based on 3-D laser scan data, polygons/vertices are usually used as the basic graphic entities for the representation of a human body shape. Usually approximately 20,000 ~ 500,000 vertices are required to describe a full body shape, depending upon surface resolution. This way of surface representation incurs a large computational cost and cannot ensure point-to-point correspondence among the scans of different subjects. Thus we developed a new method that uses contour lines as the basic entities for the shape modeling. The entire procedure of the method is as follows. (1) Joint center calculation The human body is treated as a multi-segment system where segments are connected to each other by joints, which in turn, are defined by respective landmarks. (2) Skeleton model building A skeleton model is formed by connecting respective joint centers to represent the articulated structure and segments of the human body, as shown in Fig. 1. (3) Segmentation The entire body scan is divided into segments according to the skeleton model with some special treatment in certain body areas. (4) Slicing The scan of each segment is sliced along the main axis of each segment at fixed intervals, which produces the contour lines of the segment. Figure 2 displays the segmentation and slicing of a whole body scan. (5) Discretizing Each contour line is discretized with respect to a polar angle. As such, the two-dimensional contour curve is represented by a vector. (6) Hole-filling The hole-filling is performed on contour lines for each segment. Figure 3 shows the original surface and filled surface of the abdomen segment. (7) Parameterization The vector of each discretized contour line is represented by a set of wavelet coefficients. (8) Registration The point-to-point correspondence between the scans of two bodies is established with respect to the contour lines of each segment. (9) Shape description and PCA The assembly of the wavelet coefficients of all segments is used as the shape description vector. Principal component analysis (PCA) is performed on a

10


selection of subjects from the CAESAR database. (10) Shape reconstruction A 3D human shape model is reconstructed in the following way: (a) From the shape description vector to wavelet coefficients; (b) From wavelet coefficients to contour lines; (c) From contour lines to 3D surface model; and (d) Part blending as needed.

Fig. 1. Landmarks, joint centers, and the skeleton model

Fig. 2. Segmentation and slicing

250

200

150 300 200 100 0 -100 -200 -300

-300

-200

-100

0

100

200

Fig. 3. Hole filling based on contour lines

6 Concluding Remarks Human shape modeling spans various research areas from anthropometry, computer graphics and computer vision to machine intelligence and optimization. It simply would not be possible to present a full survey of the related work. Instead, this paper just intended to provide an indication of the current state of the art. In addition to traditional uses, human modeling is finding many new applications with great challenges, such as virtual environment, human identification, and human-borne threat detection.

References 1. Allen, B., Curless, B., Popovic, Z.: Articulated Body Deformation from Range Scan Data. In: ACM SIGGRAPH 2002, San Antonio, TX, USA, pp. 21–26 (2002) 2. Allen, B., Curless, B., Popvic, Z.: The space of human body shapes: reconstruction and parameterization from range scans. In: ACM SIGGRAPH 2003, San Diego, CA, USA, 2731 July (2003)


11

3. Hilton, A., Starck, J., Collins, G.: From 3D Shape Capture to Animated Models. In: Proceedings of First International Symposium on 3D Data Processing Visualization and Transmission, pp. 246–255 (2002) 4. Seo, H., Cordier, F., Thalmann, N.M.: Synthesizing Animatable Body Models with Parameterized Shape Modifications. In: Eurographics/SIGGRAPH Symposium on Computer Animation (2003) 5. Sand, P., McMillan, L., Popovic, J.: Continuous Capture of Skin Deformation. ACM Transactions on Graphics 22(3), 578–586 (2003) 6. Ben Azouz, Z., Rioux, M., Shu, C., Lepage, R.: Analysis of Human Shape Variation using Volumetric Techniques. In: Proc. of 17th Annual Conference on Computer Animation and Social Agents Geneva, Switzerland (2004) 7. Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., Davis, J.: SCAPE: Shape Completion and Animation of People. ACM Transactions on Graphics 24(3) (2005) 8. Park, S.I., Hodgins, J.K.: Capturing and Animating Skin Deformation in Human Motion. In: ACM Transaction on Graphics (SIGGRAPH 2006), vol. 25(3), pp. 881–889 (2006) 9. Aguiar, E., Zayer, R., Theobalt, C., Magnor, M., Seidel, H.P.: A Framework for Natural Animation of Digitized Models. MPI-I-2006-4-003 (2006) 10. Balan, A., Sigal, L., Black, M., Davis, J., and Haussecker, H.: Detailed Human Shape and Pose from Images. In: IEEE Conf. on Comp. Vision and Pattern Recognition (2007). 11. Robinette, K.M., Vannier, M.W., Rioux, M., Jones, P.: 3-D surface anthropometry: Review of technologies. In: North Atlantic Treaty Organization Advisory Group for Aerospace Rearch & Development, Aerospace Medical Panel (1997) 12. Robinette, K.M.: An Investigation of 3-D Anthropometric Shape Descriptors for Database Mining, Ph.D. Thesis, University of Cincinnati (2003) 13. Paquet, E., Rioux, M.: Content-based access of VRML libraries. In: Ip, H.H.-S., Smeulders, A.W.M. (eds.) MINAR 1998. LNCS, vol. 1464. Springer, Heidelberg (1998) 14. Ben Azouz, Z.B., Shu, C., Lepage, R., Rioux, M.: Extracting Main Modes of Human Body Shape Variation from 3-D Anthropometric Data. In: Proceedings of the Fifth International Conference on 3-D Digital Imaging and Modeling (2005) 15. Robinette, K., Daanen, H., Paquet, E.: The Caesar Project: A 3-D Surface Anthropometry Survey. In: Second International Conference on 3-D Digital Imaging and Modeling (3DIM 1999), Ottawa, Canada, pp. 380–386 (1999) 16. Anguelov, D., Srinivasan, P., Pang, H.C., Koller, D., Thrun, S., Davis, J.: The correlated correspondence algorithm for unsupervised registration of nonrigid surfaces. Advances in Neural Information Processing Systems 17, 33–40 (2005) 17. Ben Azou, Z.B., Shu, C., Mantel, A.: Automatic Locating of Anthropometrics Landmarks on 3D Human models. In: Proceedings of the Third International Symposium on 3D Data Processing, Visualization, and Transmission (2006) 18. Davis, J., Marschner, S., Garr, M., Levoy, M.: Filling holes in complex surfaces using volumetric diffusion. In: Proceedings of the First International Symposium on 3D Data Processing, Visualization and Transmission. Padua, Italy (2002) 19. Curless, B., Levoy, M.: A volumetric method of building complex models from range images. In: Proceedings of SIGGRAPH 1996, 303–312 (1996) 20. Liepa, P.: Filling holes in meshes. In: Proc. of the Eurographics/ACM SIGGRAPH symposium on Geometry processing, pp. 200–205 (2003) 21. Ben Azouz, B.Z., Rioux, M., Shu, C., Lepage, R.: Characterizing Human Shape Variation Using 3-D Anthropometric Data. International Journal of Computer Graphics 22(5), 302– 314 (2005)

12


22. Mittal, A., Zhao, L., Davis, L.S.: Human Body Pose Estimation Using Silhouette Shape Analysis. In: Proceedings of the IEEE Conference on Advanced Video and Signal Based Surveillance (2003) 23. Cohen, I., Li, H.X.: Inference of Human Postures by Classification of 3D Human Body Shape. In: Proceedings of the IEEE International Workshop on Analysis and Modeling of Faces and Gestures (2003) 24. Anguelov, D., Koller, D., Pang, H.C., Srinivasan, P., Thrun, S.: Recovering Articulated Object Models from 3D Range Data. In: Proceedings of the 20th conference on Uncertainty in artificial intelligence, pp. 18–26 (2004) 25. Robertson, C., Trucco, E.: Human body posture via hierarchical evolutionary optimization. In: BMVC 2006 (2006) 26. Sundaresan, A., Chellappa, R.: Model driven segmentation of articulating humans in Laplacian Eigenspace. IEEE Transaction: Patter Analysis and Machine Intelligence (2007) 27. Aubel, A., Thalmann, D.: Interactive modeling of the human musculature. In: Proc. of Computer Animation (2001) 28. Lewis, J.P., Cordner, M., Fong, N.: Pose space deformations: A unified approach to shape interpolation and skeleton-driven deformation. In: ACM SIGGRAPH, pp. 165–172 (2000) 29. Sloan, P.P., Rose, C., Cohen, M.F.: Shape by example. In: Proceedings of 2001 Symposium on Interactive 3D Graphics (2001) 30. Theobalt, C., Magnor, M., Schüler, P., Seidel, H.P.: Combining 2D Feature Tracking and Volume Reconstructions for Online Video-Based Human Motion Capture. International Journal of Image and Graphics 4(4), 563–583 (2004) 31. Seo, H., Yeo, Y.I., Wohn, K.: 3D Body Reconstruction from Photos Based on Range Scan. In: Pan, Z., Aylett, R.S., Diener, H., Jin, X., Göbel, S., Li, L. (eds.) Edutainment 2006. LNCS, vol. 3942, pp. 849–860. Springer, Heidelberg (2006)

An Advanced Modality of Visualization and Interaction with Virtual Models of the Human Body Lucio T. De Paolis1,3, Marco Pulimeno2, and Giovanni Aloisio1,3 1

Department of Innovation Engineering, Salento University, Lecce, Italy 2 ISUFI, Salento University, Lecce, Italy 3 SPACI Consortium, Italy {lucio.depaolis,marco.pulimeno,giovanni.aloisio}@unile.it

Abstract. The developed system is the first prototype of a virtual interface designed to avoid contact with the computer so that the surgeon is able to visualize models of the patient’s organs more effectively during surgical procedure. In particular, the surgeon will be able to rotate, to translate and to zoom in on 3D models of the patient’s organs simply by moving his finger in free space; in addition, it is possible to choose to visualize all of the organs or only some of them. All of the interactions with the models happen in real-time using the virtual interface which appears as a touch-screen suspended in free space in a position chosen by the user when the application is started up. Finger movements are detected by means of an optical tracking system and are used to simulate touch with the interface and to interact by pressing the buttons present on the virtual screen. Keywords: User Interface, Image Processing, Tracking System.

1 Introduction The visualization of 3D models of the patient’s body emerges as a priority in surgery both in pre-operative planning and during surgical procedures. Current input devices tether the user to the system by restrictive cabling or gloves. The use of a computer in the operating room requires the introduction of new modalities of interaction designed in order to replace the standard ones and to enable a non-contact doctor-computer interaction. Gesture tracking systems provide a natural and intuitive means of interacting with the environment in an equipment-free and non-intrusive manner. Greater flexibility of action is provided since no wired components or markers need to be introduced into the system. In this work we present a new interface, based on the use of an optical tracking system, which interprets the user’s gestures in real-time for the navigation and manipulation of 3D models of the human body. The tracked movements of the finger provide a more natural and less-restrictive way of manipulating 3D models created using patient’s medical images. Various gesture-based interfaces have been developed; some of these are used in medical applications. V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 13–18, 2009. © Springer-Verlag Berlin Heidelberg 2009

14

L.T. De Paolis, M. Pulimeno, and G. Aloisio

Grätzel et al. [1] presented a non-contact mouse for surgeon-computer interaction in order to replace standard computer mouse functions with hand gestures. Wachs et al. [2] presented ”Gestix”, a vision-based hand gesture capture and recognition system for navigation and manipulation of images in an electronic medical record database. GoMonkey [3] is an interactive, real time gesture-based control system for projected output that combines conventional PC hardware with a pair of stereo tracking cameras, gesture recognition software and customized content management system. O’Hagan and Zelinsky [4] presented a prototype interface based on tracking system where a finger is used as a pointing and selection device. The focus of the discussion is how the system can be made to perform robustly in real-time. O’Hagan et al. [5] implemented a gesture interface for navigation and object manipulation in the virtual environment.

2 Technologies Used In the developed system we have utilized OpenSceneGraph for the construction of the graphic environment and 3D Slicer for building the 3D models starting from the real patient’s medical images. OpenSceneGraph [6] is an open source high performance 3D graphics toolkit used by application developers in fields such as visual simulation, computer games, virtual reality, scientific visualization and modeling. The toolkit is a C++ library and is available on multiple platforms including Windows, Linux, IRIX and Solaris. 3D Slicer [7] is a multi-platform open-source software package for visualization and image analysis, aimed at computer scientists and clinical researchers. The platform provides functionality for segmentation, registration and threedimensional visualization of multi-modal image data, as well as advanced image analysis algorithms for diffusion tensor imaging, functional magnetic resonance imaging and image-guided therapy. Standard image file formats are supported, and the application integrates interface capabilities with biomedical research software and image informatics frameworks. The optical tracking system used in this application is the Polaris Vicra of the NDI. The Polaris Vicra is an optical system that tracks both active and passive markers and provides precise, real-time spatial measurements of the location and orientation of an object or tool within a defined coordinate system. The system tracks wired active tools with infra-red light-emitting diodes and wireless passive tools with passive reflective spheres. With passive and active markers, the position sensor receives light from marker reflections and marker emissions, respectively. The Polaris Vicra uses a position sensor to detect infrared-emitting or retroreflective markers affixed to a tool or object; based on the information received from the markers, the position sensor is able to determine position and orientation of tools within a specific measurement volume. In this way each movement of the marker, or marker geometry, attached to the specific tool in the real environment is replicated in the corresponding virtual environment. The markers outside of the measurement volume are not detected.

An Advanced Modality of Visualization and Interaction with Virtual Models

15

The system is able to track up to 6 tools (maximum 1 active wireless) with a maximum of 32 passive markers in view and the maximum update rate is 20 Hz. The systems can be used in a variety of surgical applications, delivering accurate, flexible, and reliable measurement solutions that are easily customized for specific applications.

3 The Developed Application The developed system is the first prototype of a virtual interface designed to avoid contact with the computer so that the surgeon can visualize models of the patient’s organs more effectively during the surgical procedure. A 3D model of the abdominal area, reconstructed from CT images, is shown in Figure 1. The patient suffers from a pathology in the liver which causes notable swelling.

Fig. 1. A 3D model reconstructed from CT images

In order to build the 3D model from the CT images some segmentation and classification algorithms were utilized. The Fast Marching algorithm was used for the image segmentation; some fiducial points were chosen in the interest area and used in the growing phase. After a first semi-automatic segmentation, a manual segmentation was carried out. All of the interactions with the models happen in real-time using the virtual interface which appears as a touch-screen suspended in free space in a position chosen by the user when the application is started up.

16


When starting the user has to define a space area where the interface is located and to decide on the positions of the four vertexes of the virtual screen. In this way a reference system is also defined; this is necessary to fix the interaction plane. In front of this region the marker is moved around and, in order to choose the different interaction modalities and the organs to be visualized, you press the virtual buttons which are present on the interface. In addition a scaling operation is carried out in order to adapt the size of the virtual interface to the real screen of the computer. Finger movements are detected by means of an optical tracking system and are used to simulate touch with the interface where some buttons are located. In figure 2 the interaction with the user interface by means of the tracking system is shown.

Fig. 2. The interaction with the virtual user interface

The interaction with the virtual screen happens by pressing these buttons, which make it possible to visualize the different organs present in the built 3D model (buttons on the right) and to choose the possible operations allowed on the selected model (buttons on the left). For this reason, when using this graphical interface, the surgeon is able to rotate, to translate and to zoom in on the 3D models of the patient’s organs simply by moving his finger in free space; in addition, he can select the visualization of all of the organs or only some of them. At the bottom of the screen the interaction modality chosen is visualized and in the top left-hand corner the cursor position is shown in the defined reference system. In figure 3 is shows the virtual user interface.

An Advanced Modality of Visualization and Interaction with Virtual Models

17

cursor position

action buttons selection buttons

interaction modality

Fig. 3. The virtual user interface

To build the virtual scene a scene graph has been used and 2D and 3D environments are included. The 2D environment allows the cursor, some text and the buttons to be visualized, updating the active interaction modality and the cursor position. The 3D environment allows the model of the organs to be visualized and provides the interaction operations. The lighting conditions are important and can cause problems because some external light could be interpreted as other IR reflectors with the creation of false cursors in the scene.

4 Conclusions and Future Work The described application is the first prototype of a virtual interface which provides a very simple form of interaction for navigation and manipulation of 3D virtual models of the human body. The virtual interface created provides an interaction modality with models of the human body, a modality which is similar to the traditional one which uses a touch screen, but in this interface there is no contact with the screen and the user’s finger moves through open space. By means of an optical tracking system, the position of the finger tip, where an IR reflector is located, is detected and utilized first to define the four vertexes of the virtual interface and then to manage the interaction with this. The optical tracker is already in use in computer aided systems and, for this reason, the developed interface can easily be integrated in the operation room. Taking into account a possible use of the optical tracker in the operating room during surgical procedures, the problem of possible undesired interferences due to the detection of false markers (phantom markers) will be evaluated.

18


The introduction of other functionalities of interaction with the models is in progress, after further investigation and consideration of surgeons’ requirements. Another improvement could be to provide the visualization of CT images in addition to the 3D models and give surgeons the opportunity of navigating into the set of CT slices. In this way surgeons are provided with the traditional visualization modality as well as the new one and are able to compare them.

References 1. Grätzel, C., Fong, T., Grange, S., Baur, C.: A non-Contact Mouse for Surgeon-Computer Interaction. Technology and Health Care Journal 12(3) (2004) 2. Wachs, J.P., Stern, H.I., Edan, Y., Gillam, M., Handler, J., Feied, C., Smith, M.A.: Gesturebased Tool for Sterile Browsing of Radiology Images. The Journal of the American Medical Informatics Association 15(3) (2008) 3. GoMonkey, http://www.gomonkey.at 4. O’Hagan, R., Zelinsky, A.: Finger Track - A Robust and Real-Time Gesture Interface. In: Sattar, A. (ed.) Canadian AI 1997. LNCS, vol. 1342, pp. 475–484. Springer, Heidelberg (1997) 5. O’Hagan, R., Zelinsky, A., Rougeaux, S.: Visual Gesture Interfaces for Virtual Environments. Interacting with Computers 14, 231–250 (2002) 6. OpenSceneGraph, http://www.openscenegraph.org 7. 3D Slicer, http://www.slicer.org

3D Body Scanning’s Contribution to the Use of Apparel as an Identity Construction Tool Marie-Eve Faust1 and Serge Carrier2 1

The Hong Kong Polytechnic University, Institute of Textiles & Clothing, Hung Hom, Kowloon, Hong Kong, China 2 Université du Québec a Montréal, C.P. 8888, Succ. Centre-ville, Montréal, Québec, H3P 3P8, Canada [email protected], [email protected]

Abstract. Humans use apparel as an artifact to construct their identities and present it to the outside world. Beyond textiles and clothing style, garment fit contributes to this image presentation. This research, conducted on the Hong Kong market shows that women view 3D body scanning technology positively and that it therefore could prove an effective and efficient tool, both from a consumer’s and from a seller’s point of view, in facilitating the body image creation. Keywords: Body image, 3D body scan, apparel, fashion.

1 Introduction Throughout the ages clothing has not only fulfilled a need for protection but has also played a role in defining the wearer’s personality and determined his or her status in society. Yet selecting a garment that advantages one’s silhouette and projects the right image often, for most, proves a difficult if not impossible task. Style, color and textile are, to a large extent, subjective decisions: de gustibus et coloribus non est disputandum. Yet fit is a very objective criterion. Until a few years ago, the only way to identify a fitting garment was to try it on. The invention of 3D body scanning technology is rapidly changing this situation. Not only is it a possible first step toward mass customization, but it may also be used, by retailers, as an added customer service tool helping them identify the best fitting garments, thereby improving their satisfaction with the shopping experience.

2 Literature Review The following section first discusses the importance of clothing in the individual’s image formation. It then proceeds to a brief presentation of 3D body scanning technology and its potential use in helping women selecting the best fitting and most advantaging pieces of clothing. 2.1 Personal Identity and Styling Clothing is worn daily yet it serves more than a mere protection need. Leroi-Gourhan [11], Wolfe [18] and Boucher [3] state that clothing has always served the same three V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 19–28, 2009. © Springer-Verlag Berlin Heidelberg 2009

20

M.-E. Faust and S. Carrier

basic human needs: (1) protection (physical need), (2) adornment and identification (psychological needs), and (3) modesty and status (social needs or role). For Kefgen and Touchie-Specht [9] clothing forms a nonverbal communication. The way people dress tells others, what kind of a person they are or would like to be perceived as. Johnson and Foster [7] point out that clothing has a language of its own providing. It may not be appropriate to judge a book by its cover but many argue that the cover certainly helps selecting the book. Body image can be positive or negative, accurate or inaccurate, particularly as we form this image in comparison with others and in relation to cultural views of fashion. The evaluative dimension of body image is known as body cathexis [8]: the indication of an individuals’ satisfaction or dissatisfaction with their different body parts [10; 16]. Body cathexis is closely related to the person’s global self-image, self esteem, and self concept [16; 17]. Rasband [14] showed that an accurate and objective body image is necessary as it plays a significant part in clothing selection and appearance. Clothing becomes part of your body image, a second skin in establishing new physical boundaries for yourself. Over the last few decades, various studies have focused on five elements of a garment important in achieving a better clothing message: line, shape, color, texture and pattern. In order to understand how clothing can impact on the way someone looks, body figures must be understood even if standards or beliefs change with times or who makes the decision. Authors generally recognize eight more or less standard female body shapes: • Ideal figure: the shoulders and hips have similar width, the bust size is medium the waist is small. The abdomen is flat to slightly curved, the buttock is moderately curved and thighs are slim. The figure is well balanced. The weight is just enough to cover the bones. • Hourglass figure type: the hourglass shape appears full-rounded in the bust and hip, with a small waist. The bust is more often larger than average as well as the hips. The waist is well indented waist. Hips and buttocks are smoothly rounded. • Triangular figure type: the triangular figure seen from the front looks narrower above the waist and wider below. The excess of weight appears on the buttocks, the low hips, and the thighs. Women with this type of figure appear unbalanced from top to bottom with the shoulders narrower than the hips. The bust and the waist are usually small to medium. • Inverted triangular figure type: the inverted triangle gives the opposite look. It appears wider above the waist and narrower below. The shoulder, the upper back, and the bust look prominent. • Rectangular figure type: women with a called rectangular figure seams to have nearly the same width at shoulders, waist, and hips. Their waist line doesn’t seem well defined and their body lines look straight. • Tubular figure type: similar to the rectangular, the weight is considerably below the average or ideal range. • Diamond figure type: points up with narrow shoulders and hips in combination with a wide midriff and waist. The midriff and upper hips do not appear to taper inward towards the waist.

3D Body Scanning’s Contribution to the Use of Apparel

21

• Oval or full-rounded figure: the weight is noticeably above the average and larger throughout the figure where body lines are full-round curves. (Rasdand and Liechty, 2006) Numerous fashion articles are written every year focusing either on the body or parts of the body such as “C’est moi, ma personalite, mon style” [5] or InStyle [1]. Each describes and shows two dimensional figures and ways to improve upon them. Instyle, for example, talks about “curvy women” stating that these women should showcase their waist and curves without over emphasizing them; steer clear of anything to tight or clothes that are cut straight un-and-down and fabrics that are thin. It provides tips to select garments that flatter the body, or part of it, for each body figure: short, narrow shoulders or broad shoulders, full bust or small bust, heavy arms, well defined tummy, short-waisted/long legs or long-waisted/short legs, bottom heavy, etc. For Rasband and Liechty [15], a garment can change the visual of the body figure, even in areas where it may appear difficult. According to Rasband and Liechty [15] a garment line creates shape and form. Yet to fully take advantage of this wisdom, women need to know their body shapes. 2.2 3D Body Scanner A 3D body scanner is the size of a fitting room. It uses cameras or safe lasers to capture up to 300,000 data points for each person’s scan. The scanning process takes only a few seconds. Within a few minutes the software automatically extracts hundreds of body measurements. Data on body shape and body volume can also be automatically extracted. The resolution of the final scan is quite accurate. Data can be transferred directly from the scanner over local networks or the web (Shape Analysis Limited, 2008). In the early stages of the 3D body scanning technology many argued that this technology would mostly be used to provide custom fitting services. Many thought that it would bring consumers into the design and production stages, resulting in well-fitting, made-to-measure garments at competitive prices and turnaround times. Although it has not quite reached this point yet, 3D scanning has come to play an important for some apparel retailers and producers. It contributes to mass customization by enabling retailers to rapidly collect three-dimensional (3D) data and to send it to manufacturers who tailor the garment to fit individuals [2]. In addition to custom fitting, 3D body scanning technology also improves the body measurement data used in traditional mass production [4]. Industry and academic researchers are beginning to use large amounts of anthropometric (body measurement) data captured by body scanners to adjust the sizing systems of ready-to-wear clothing lines in order to provide better fitted garments ([TC]2, 2004). Another application of 3D body scanning is the Virtual Try-on. Consumers can now virtually try garments on. An individual's scan is visualized on a computer while clothing of various sizes is superimposed (in 3D) on a rotatable image (http://www.bodyscan.human.cornell.edu/scene0037.html, [2]). The computer application highlights areas of good and bad fit, helping the user to select the most appropriate product according to his or her body size and shape. Body scanning data will also increase the number and accuracy of measurements used in size prediction (match between one’s body and garments on offer). The

22


combination of virtual try-on with size prediction not only provides consumers with the brands and sizes that fit their measurements and proportions best, but also lets them virtually view garments on their scan and choose the design they prefer. This process combines objective fit information and with fit preference. Locker and Ashdown [12] noticed that Commercial applications of body scanning (mass-customized clothing, improved ready-to-wear sizing systems, and virtual tryon) will only be viable if consumers agree to being scanned. They surveyed a group of women they scanned in the course of one of their studies enquiring about their level of comfort with, and interest in, body scanning. The answer was resoundingly positive on both counts regardless of size, age, or their satisfaction with the fit of available ready-to-wear pants. Almost all were willing to be scanned again and many were willing to be scanned every year or whenever their weight changed. They also found very positive reactions to commercial applications and research using body scan data. Participants found the virtual try-on application more appealing than custom-fit clothing or patterns, size prediction, or personal shopper applications. Women also selected virtual try-on as the most likely to influence them to buy more clothing on the Internet. Virtual try-on, custom-fitted clothing, and the creation of a “personal shopper” were rated highest in their potential contribution to find clothing that looks good on the body; custom-fit and size prediction were rated highest in helping to find clothing that fits them best. Participant confidence was also extremely high in the body scan data’s applications as an effective way to obtain body measurements, an effective means to arrive at a good fit, and in improving the trustworthiness of an online screen image (figure 1) of their own body over an idealized body shape (avatar).

Fig. 1. Adapted from Cornell Body Scan Research Group (http://www.bodyscan.human. cornell.edu/scene0037.html)

Another approach to provide the consumer with a “personal shopper” is the avatar such as those being offered by Lands' End and My Virtual Model (figure 2). Lands' End customers enter their body measurements and select a virtual model with a similar body shape in order to visualize clothing styles through their on-line store. My Virtual Model supplies the virtual try-on web interface for Lands' End, Levi's, Kenneth Cole, and other on-line retailers (My Virtual Model Inc., 2008). According to Istook and Hwang [6], there is no doubt that scanners will become an important component of the shopping experience.


23

Fig. 2. My Virtual Model, Adapted from: http://www.mvm.com/brandme.php?id=10&lang_id=en

2.3 Styling Service Online Styling services are now offered by cutting edge Internet companies such as myShape.com which boast of having more than 20,000 women’s measurements on file. This company offers women the possibility to shop from personalized clothing collections matching their style, fit preferences, and body shapes and sizes (myShape, 2008).

Fig. 3. The 7 body shapes Adopted from: www.myshape

In Wannier (2006), myShape’s chief executive states that the method seems to be working, particularly among women 35 and older. On the other hand, Mulpuru (2006), an analyst from Forrester Research states that myShape’s approach fails to gain a mass audience because the measuring process is too complicated. Only a small percentage of women would accept the site’s offer to mail them a free tape measure, and fewer would go through the process of taking the measurements and logging them into the myShape system. One thing that might help myShape to reach a mass audience would be if the company somehow offered interested people to be 3D body scan in malls or other locations, saving them the trouble of measuring themselves [13].

3 Methodology Many previous researches looked into the consumers’ reaction to 3D body scanning; yet few researches studying the combination of 3D body scanning with a styling

24


service were found. Moreover, most were merely based on western countries. A questionnaire was therefore developed to determine if a potential market combining these two areas may exist in Asia and more specifically Hong Kong. 3.1 Questionnaire Design The questionnaire was comprised of 26 questions divided into seven sections. The first section looked into the consumers’ expectations in clothing; a better understanding of their thoughts and behavior providing a first input as to the need for the type of service this research is interested in. The second section tried to evaluate the consumers’ awareness and knowledge of their body measurements and figure. In the third section, the questionnaire focused on time as a factor in clothing selection. The fourth section evaluated the consumers’ difficulties in selecting clothing. The next section dealt with shopping habits. The next to the last section investigated the consumers’ interest in using the 3D scanning-styling service should it be offered. Finally, the seventh section pertained to our respondents’ socio-demographic characteristics. 3.2 Sampling and Data Analysis A total of 128 women answered our questionnaire. The sample was a convenience one as the questionnaire was distributed to teachers, classmates, friends and relatives over the first 3 months of 2008. SPSS and Excel were used to process and analyze the data collected. Besides using descriptive statistics and frequency distributions to describe the sample population, cluster analysis were used to break it into smaller more homogenous groups.

4 Results and Findings The following section presents some of our findings on women’s purchases and perception of 3D body scanning technology. 4.1 Women’s Garment Purchases As our literature review revealed clothing serves different purposes. Figure 4 shows that second to fulfilling a basic need the HK women who participated in our survey stated that clothing should reflect their personalities and help them look beautiful. More than half of them believe that clothing helps build their self-esteem. This validated Leroi-Gourhan [11] and Wolfe’s [18] study that people use clothes for three major reasons: physical, psychological and social. It also confirms that, as is the case with Westerners, clothing is not only fulfilling a need but also answers a “want” [15]. When asked where they purchase their clothes 99 women chose mall stores and boutiques, 68 answered that they bought them in stand-alone stores. None of them mentioned shopping (the impossibility to feel the material and see the garments being mentioned as the main reasons whereas the long store hours in HK reduced the need for on-line shopping).


150 100

25

Q1. What purpose does clothing server? Selection) A basic need 105 (Mulitiple 99 83 71 57 To get

50

attention Reflect lit

0

0

Fig. 4. Justification for clothing purchases

Q5. Did you ever purchase a garment that you never or hardly ever wore? Y… No

35% 65%

Fig. 5. Garment use

Interestingly 35% of the women surveyed admitted to having purchased a garment which they hardly ever wore as they felt it did not look good on them, did not advantage their body figure, or they did not feel confident when wearing it. All concerns mentioned had to do with image and psychological needs; none mentioned concerns about fit or comfort. While 80% of our sampled admitted to searching for the garments most advantageous to their silhouette, 70% admitted they found it difficult to identify the style that accomplished this objective. To the question of perceived time consumption to find fitting clothes (1 being the lowest and 5 the highest) over 50% of women scored a 4 or 5 and 37% scored a 3; a result which clearly shows that women find the process to be time consuming. When asked how much time they spend to choose a garment, 49% stated that they spend 5 to 15 minutes to choose a casual garment and 28% that they spend 15 to 30 minutes. For the selection of party clothes, 49% of our sample stated spending between 15 and 30 minutes and 23% between 30 to 60 minutes. Almost 40% of the women stated they take between 15 to 30 minutes to decide what to wear for an interview. As one could expect, the time to choose a garment for a special event (such as a wedding) increases with 56% spending 2 hours and 18% up to three hours. When trying to identify if women were concerned with their body measurement and shapes (body figure) we found 50% of women scoring 4 and 5 (on a Likert scale where 1 identified a small extent and 5 a great extent). Our results also revealed that 62% of our sample had never heard about the “standard” body shapes before being shown the pictures taken from the literature review. Surprisingly 77were not sure about which body shape represent them best.

26


4.2 3D Body Scanning As we expected women were curious about body figures and body scanning although we were not sure if Hong Kong females were as “opened” as westerners and willing to be scanned. To the question as to how they would react to the possibility of being scanned in a retail store, 64% of women stated they would accept to then compare their body shape to the “standard” ones. Nearly 35% of our sample stated that they would visit a retail store more often if it offered styling recommendations. Nearly 40% believed that styling recommendation would reduce the risks of buying “unsuitable” garments. Through a clustering analysis, we found that 50.8% of our sampled women clustered on the first group (aged 19 to 25, single, low income level, secondary or tertiary education level) which cared the most about their own body measurement. Our second group (15.8% of our sample, aged 26 to 32, single or married without children, middle income level, secondary or tertiary education level) cared much less. The third group (12.5% of our sample, aged 33 to 40, married with or without children, high income level, tertiary education level) did not seem to care about their body measurement. Lastly the fourth group (20.8% of the sample, aged 26 to 35, married with children, high income level, tertiary education level) cared only marginally about their own body measurement. A clustering analysis on the perceived time consumption of apparel shopping showed that 36.7% did not perceive apparel shopping as particularly time consuming (aged 19 to 25, single, low income level secondary or tertiary education). The second group (29.2% of sample, aged 19 to 30, single or married without children, middle income level, tertiary education) stated that apparel shopping was time consuming. A third group (20.8% of sample, aged 26 to 40, married with children, high income level, and tertiary education level) also finds apparel shopping to be time consuming. The fourth group (13.3% of sample, aged 19 to 32, married with or without children, high income, tertiary education) finds it highly time consuming. A third clustering analysis was performed to identify willingness to go through a scanning process. In this case we identifier a first group (39.2% of sample, aged 19 to 32, single, low income, secondary to tertiary education) expressed no interest in trying the 3D body scanner. The second group (17.5% of sample, aged 19 to 25, single, low income level, tertiary education) expressed willingness to try the 3D body scanner. A third group (30.8% of sample, aged 26 to 40, married with or without children, high income, secondary to tertiary education) also expressed interest. The last group (12.5% of sample, aged 19 to 32, married with or without children, high income level, tertiary education) also expressed interest in trying the 3D body scanner. Our fourth clustering analysis focused on the interest of our sample to pay for styling recommendations. Only one group (17.5% of sample, aged 26 to 40, married with or without children, high income, and tertiary education) stated that the provision of styling recommendations by a retail store would not influence them in their shopping patterns. Lastly we conducted a clustering analysis to try and understand the relationship between the wish for styling recommendations and the willingness to try the 3D body scan. A first group (39.2% of sample) finds apparel shopping time consuming and


27

spends relatively little on fashion would like to be offered styling recommendations but expresses no interest in 3D body scanning. A second group 10% of sample) does not perceive shopping as being time consuming spends very little on fashion yet is interested in trying the body scanner as well as being offered styling recommendations. A third group (25% of sample) was comprised of those who find shopping time consuming, spend moderately on fashion products, and are interested in trying the body scanner but will patronize a retail store because it offers styling recommendations. The last group (25.8% of sample finds shopping time consuming yet spends relatively more on fashion; they are uncertain about their interest in trying the body scanner but will patronize a retail offering styling recommendations.

5 Conclusions and Recommendations Our results show that the group of consumers most interested in the body scan technology and in patronizing stores offering styling recommendations is comprised of individuals at the lower end of the spending spectrum on fashion. This finding begs the question: is it worthwhile investing in 3D scanning technology and in providing styling recommendations. Unfortunately our research does not enable us to determine whether a “free body scanning / styling recommendations” offer would impact the amount of money these customers spend on fashion items. Yet it clearly indicates that 82,5% of women would appreciate styling recommendations and 35% would agree to an in-store 3D body scan. This clearly represents an opportunity which should be investigated further.

References 1. Arbetter, L.: Style, Secrets of Style. In: Style (eds.) The complete guide to dressing your best every day, p. 191. Melcher Media, New York (2005) 2. Ashdown, S.P.: Research Group, Cornell University, About the Body Scanner (2006), http://www.bodyscan.human.cornell.edu/scene60df.html 3. Boucher, F.: Histoire du Costume en Occident des origines a nos jours, p. 478. Flammarion, Paris (1996) 4. Faust, M.-E., Carrier, S.: Discard one size fits all labels! New Size and Body Shapes labels are coming! Way to achieved Mass Customization in the apparel industry. In: Extreme Customization Mass Customization World Conference (MIT) Cambridge/Boston & (HEC) Montreal. Conference dates (October 2007) (Book chapter TBP, 2009) 5. Hamel, C., Salvas, G.: C’est moi, ma personnalité, mon style, Québec: Éditions Communiplex, p. 310 (1992) 6. Istook, C.L., Hwang, S.-J.: 3D body scanning systems with application to the apparel industry. Journal of Fashion Marketing and Management 5(2), 120–132 (2001) 7. Johnson, J.G., Foster, A.G.: Clothing image and impact. South-Western Publishing Co. (1990) 8. Jourard, S.M.: Personal adjustment; an approach through the study of healthy personality. Personal adjustment. Macmillan, New York (1958) 9. Kefgen, M., Touchie-Specht, P.: Individuality in clothing selection and personal appearance, 3rd edn. MacMillan Publishing Company, Basingstoke (1986)

28


10. LaBat, K.L., Delong, M.R.: Body Cathexis and Satisfaction with Fit Apparel. Clothing and Textiles Research Journal 8(2), 43–48 (1990) 11. Leroi-Gourhan, A.: Milieu et techniques, Évolution et Techniques. Paris: Éditions Albin Michel, p. 475, 198–241 (1973) 12. Locker, S., Cowie, L., Ashdown, S., Lewis, V.D.: Female consumers reactions to body scanning. Clothing and Textiles Research Journal 22(4), 151–160 (2004) 13. Powell, T.: Body-scanning kiosk wows apparel shoppers. Seifserviceworld.com (2006), http://www.selfserviceworld.com/article.php?id=16541 14. Rasband, J.: Fabulous Fit. Fairchild Publications, New York, p. 176 (1994) 15. Rasband, J.A., Liechty, E.L.G.: Fabulous fit Speed Fitting and Alteration, 2nd edn., p. 432. Fairchild Publications, New York (2006) 16. Secord, P.F., Jourard, S.M.: The appraisal of body-cathexis: Body-cathexis and the self. Journal of Consulting Psychology 17(5), 343–347 (1953) 17. Wendel, G., Lester, D.: Body-cathexis and self esteem. Perceptual and Motor Skills, p. 538 (1988) 18. Wolfe, M.G.: Fashion! The Goodheart-Willcox Company, Inc., West Chester Pennsylvania (2002)

Websites − − − −

INTELLIFIT (2007). http://www.it-fits.info/IntellifitSystem.asp MyShape (2008). http://www.myshape.com/content/body_shapes My Virtual Model (2008): http://www.mvm.com/cs/ Seifserviceworld.com (2006). http://www.selfserviceworld.com/article.php?id=16541 − Shape Analysis Limited (2008). http://www.shapeanalysis.com/prod01.htm − Wannier (2006). http://www.myshape.com/content/body_shapes

Facial Shape Analysis and Sizing System Afzal Godil National Institute of Standards and Technology, 100 Bureau Dr, MS 8940, Gaithersburg, MD 20899, USA [email protected]

Abstract. The understanding of shape and size of the human head and faces is vital for design of facial wear products, such as respirators, helmets, eyeglasses and for ergonomic studies. 3D scanning is used to create 3D databases of thousands of humans from different demographics backgrounds. 3D scans have been used for design and analysis of facial wear products, but have not been very effectively utilized for sizing system. The 3D scans of human bodies contain over hundreds of thousand grid points. To be used effectively for analysis and design, these human heads require a compact shape representation. We have developed compact shape representations of head and facial shapes. We propose a sizing system based on cluster analysis along with compact shape representations to come up with different sizes for different facial wear products, such as respirators, helmets, eyeglasses, etc. Keywords: Anthropometry, shape descriptor, cluster analysis, PCA.

1 Introduction The understanding of shape and size of the human head and faces is vital for design of facial wear products, such as respirators, helmets, eyeglasses and for ergonomic studies. With the emergence of 3D laser scanners, there have been large scale surveys of humans around the world, such as the CAESAR anthropometric database. The 3D scans of human bodies contain over hundreds of thousand grid points. To be used effectively for analysis and design, these human bodies require a compact shape representation. We have developed two such compact representations based on human head shape by applying Principal Component Analysis on the facial surface and in the second method the whole head is transformed to a spherical coordinate system expanded in a basis of Spherical Harmonics. Then we use cluster analysis on these shape descriptors along with the number of cluster that come up with the sizing system for different product; such as, facial respirators, eyeglasses, helmets, and so on. Cluster analysis is a technique for extracting implicit relationship or patterns by grouping related shape descriptors. A cluster is a collection of objects that are similar to one another and are dissimilar to the objects in other clusters. There are a number of clustering techniques, but we have only tried k-mean and k-median techniques. Paquet et al. [9] have used cluster analysis for adjusting the sizes of virtual mannequin using anthropometry data. V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 29–35, 2009. © Springer-Verlag Berlin Heidelberg 2009

30

A. Godil

Facial respirators are used by millions of people around the world to reduce their risk to diseases, harmful and hazardous airborne agents. At the heart of the effectiveness is the seal of the respirator which mainly depends on the fit, which prevents harmful gases and particulates from enter into the wearer’s respiratory system. The Los Alamos National Laboratory (LANL) fit test panel developed in the 1970’s is based on an anthropometric survey conducted in 1967 of Air Force personnel and still the standard for today’s respirator fit tests. The National Institute for Occupational Safety and Health (NIOSH) conducted a new survey in 2001 entitled the NIOSH Head-and-Face Anthropometric Survey of U.S. Respirator Users to produce a more accurate picture of the civilian workforce. Following analysis of the survey revealed that the LANL panels were in fact not representative of most respirator users in the U.S. Out of the 3997 respirator users in the survey, 15.3% of them were outside of the LANL fit test panel. Although the fit test panel is in the process of being updated, the core of these fit tests is still traditional anthropometric measures, which simplify the complexity of the shape of the human face. Today most manufactures supply half and full facial mask respirators based on the facial grouping that are based on the above surveys. However, many researchers have shown that there are little or no correlations between facial dimensions and the fit of half mask respirator [10]. Hence the best respirator shape for the best seal fits can only be archived by using the full 3D facial data. In this paper, we describe the CAESAR database. Then the different compact shape descriptors to represent the facial and head shape. Then finally, we discuss cluster analysis along with the shape descriptors for sizing system for facial wear products.

2 CAESAR Database The CAESAR (Civilian American and European Surface Anthropometry Resource) project has collected 3D Scans, seventy-three Anthropometry Landmarks, and Traditional Measurements data for each of the 5000 subjects. The objective of this study was to represent, in three-dimensions, the anthropometric variability of the civilian populations of Europe and North America and it was the first successful anthropometric survey to use 3-D scanning technology. The CAESAR project employs both 3-D scanning and traditional tools for body measurements for people ages 18-65. A typical CAESAR body is shown in Figure 1. The seventy-three anthropometric landmarks points were extracted from the scans as shown in Figure 2. These landmark points are pre-marked by pasting small stickers on the body and are automatically extracted using landmark software. There are around 250,000 points in each surface grid on a body and points are distributed uniformly.

Facial Shape Analysis and Sizing System

31

Fig. 1. A CAESAR body with three postures

Fig. 2. A Caesar body with landmark numbers and positions

3 Head Shape Descriptor We now describe two methods for creating descriptors based on human head shape. 3.1 PCA Based The first shape descriptor is based on applying principal component analysis (PCA) on the 3D facial grid and the most significant eigenvectors is the shape descriptor. PCA is a statistical technique to reduce the dimensionality of the data set and it has also have been applied to face recognition. First we use four anthropometric landmark points on the face from the database to properly position and align the face surface and then interpolate the surface information

32

A. Godil

on a regular rectangular grid whose size is proportional to the distance between the landmark points. The grid size is 128 in both directions. Next we perform principal component analysis (PCA) on the set of regular 3D surface grid to create the PCA based shape descriptor. The facial grid is cut from the whole CAESAR body grid using the landmark points 5 and 10 as shown in Figure 3 and listed in Table 1. Table 1 list all the numbers and names of landmark points used in our 3D face shape descriptor. The new generated facial grid for some of the subjects with two different views is shown in Figure 3. The facial grid is very coarse for some of the subjects in the seated pose.

Fig. 3. Landmark points 1, 2, 3, 4, 5 and 10. Vertical and horizontal lines are the cutting plane.

Table 1. Numbers and names of landmark points used in our 3D face 1 3 5 7 10

Sellion Lt Infraobitale Rt.Tragion Lt. Tragion Rt. Clavicale

2 Rt Infraobitale 4 Supramenton 6 Rt. Gonion 8 Lt. Gonion 12 Lt.Clavicale

Next, we use four anthropometric landmark points (L1, L2, L3, L4) as shown in Figure 5, located on the facial surface, to properly position and align the face surface using an iterative method. There is some error in alignment and position because of error in measurements of the position of these landmark points. Then we interpolate the facial surface information on a regular rectangular grid whose size is proportional to the distance between the landmark points L2 and L3 ( d=| L3 - L2 | ) and whose grid size is 128 in both directions. We use a cubic interpolation and handle missing values with the nearest neighbor method when there are voids in the original facial grid. For some of the subjects there are large voids in the facial surface grids. Figure 4, shows the facial surface and the new rectangular grid.


33

Fig. 4. Shows the new facial rectangular grid for two subjects

We properly positioned and aligned the facial surface and then interpolated the surface information on a regular rectangular grid whose size is proportional to the distance between the landmark points. Next we perform Principal Component Analysis (PCA) on the 3D surface and similarity based descriptors are created. In this method the head descriptor is only based on the facial region. The PCA recognition method is a nearest neighbor classifier operating in the PCA subspace. To test how well the PCA based descriptor performs, we studied the identification between 200 standing and sitting subjects. The CMC at rank 1 for the study is 85%. More details about this descriptor are described in the paper by [3, 5] 3.2 Spherical Harmonics Based In the second method the 3D triangular grid of the head is transformed to a spherical coordinate system by a least square approach and expanded in a spherical harmonic basis as shown in Figure 5. The main advantage of the Spherical Harmonics Based head descriptor is that it is orientation and position independent. The spherical harmonics based descriptor is then used with the L1 and L2 norm to create similarity measure. To test how well the Spherical Harmonics Based head descriptor performs, we studied the identification of the human head between 220 standing and sitting subjects. The CMC at rank 1 for the study is 94%.

Fig. 5. 3D head grid is mapped into a sphere

34

A. Godil

4 Cluster Analysis We have used the compact face descriptors for clustering. Clustering is the process of organizing a set of faces/heads into groups in such a way that the faces/heads within the group are more similar to each other than they are to other bodies belonging to different clusters. We use k-mean cluster analysis along with the shape descriptors and the number of clusters to come with sizes for products design such as respirator. The k-means algorithm is an algorithm to cluster n objects based on attributes into k partitions, k < n. It is similar to the expectation-maximization algorithm for mixtures of Gaussians in that they both attempt to find the centers of natural clusters in the data, where there are k clusters. k

V =∑

∑ (x

j

− μi ) 2

(1)

i =1 x j ∈S i

5

Results

For this initial study, we have used the facial surface of the first 200 standing subjects from the CAESAR database. The PCA based shape descriptor is calculated for these faces and then K-means clustering is applied and the number of cluster selected is four. Figure 6, shows the 40 faces out of the 200 faces that were used for clustering. The four different colors show the different clusters. When we are doing cluster analysis with the facial shape descriptor we can vary the emphasis on the shape verses the size. We should emphasis that these results based on cluster analysis are preliminary.

Fig. 6. Shows the 40 faces out of the 200 faces that were used for clustering. The four different colors show the different clusters.


35

6 Conclusion We have developed compact shape representations of head and facial shape. We have proposed a sizing system based on cluster analysis along with compact shape representations to come up with different sizes for different facial wear products; such as, respirators, helmets, eyeglasses, etc. We also present our preliminary results based on clustering analysis. Disclaimer. The identification of any commercial product or trade name does not imply endorsement or recommendation by the National Institute of Standards and Technology (NIST). Also the findings and conclusions in this report are those of the authors and do not necessarily represent the views of the NIST.

References 1. Allen, B., Curless, B., Popovic, Z.: Exploring the space of human body shapes: data-driven synthesis under anthropometric control. In: Proc. Digital Human Modeling for Design and Engineering Conference, Rochester, MI, June 15-17, SAE International (2004) 2. CAESAR: Civilian American and European Surface Anthropometry Resource web site, http://www.hec.afrl.af.mil/cardlab/CAESAR/index.html 3. Godil, A., Ressler, S.: Retrieval and Clustering from a 3D Human Database based on Body and Head Shape. In: SAE Digital Human Modeling conference, Lyon, France, 02 July - 04 August (2006) 4. Godil, A., Grother, P., Ressler, S.: Human Identification from Body Shape. In: Proceedings of 4th IEEE International Conference on 3D Digital Imaging and Modeling, Banff, Canada, October 6-10 (2003) 5. Godil, A., Ressler, S., Grother, P.: Face Recognition using 3D surface and color map information: Comparison and Combination. In: Godil, A., Ressler, S., Grother, P. (eds.) The SPIE’s symposium on Biometrics Technology for Human Identification, Orlando, FL, April 12-13 (2004) 6. Ip, H.H.S., Wong, W.: 3D Head Model Retrieval Based on Hierarchical Facial Region Similarity. In: Proc. of 15th International Conference on Visual Interface (VI 2002), Canada (2002) 7. Paquet, E.: Exploring Anthropometric Data Through Cluster Analysis. In: Digital Human Modeling for Design and Engineering (DHM), Oakland University, Rochester, Michigan, USA, NRC 46564, June 15-17 (2004) 8. Paquet, E., Rioux, M.: Anthropometric Visual Data Mining: A Content-Based Approach. In: IEA 2003 - International Ergonomics Association XVth Triennial Congress. Seoul, Korea. NRC 44977 (Submitted, 2003) 9. Paquet, E., Viktor, H.L.: Adjustment of Virtual Mannequins through Anthropometric Measurements, Cluster Analysis and Content-based Retrieval of 3-D Body Scans. IEEE Transactions on Instrumentation and Measurement 56(5), 1924–1929 (1924) NRC 48821 10. Yang, L., Shen, H.: A pilot study on facial anthropometric dimensions of the Chinese population for half-mask respirator design and sizing. International Journal of Industrial Ergonomics 38(11-12), 921–926 (2008) 11. Zheng, R., Yu, W., Fan, J.: Development of a new Chinese bra sizing system based on breast anthropometric measurements. International Journal of Industrial Ergonomics 37(8), 697–705 (2007)

Facial Gender Classification Using LUT-Based Sub-images and DIE Jong-Bae Jeon, Sang-Hyeon Jin, Dong-Ju Kim, and Kwang-Seok Hong School of Information and Communication Engineering, Sungkyunkwan University, 300, Chunchun-dong, Jangan-gu, Suwon, Kyungki-do, 440-746, Korea [email protected], [email protected], [email protected], [email protected]

Abstract. This paper presents a gender classification method using LUT-based sub-images and DIE (Difference Image Entropy). The proposed method consists of three major steps; extraction of facial sub-images, construction of a LUT (Look-Up table), and calculation of DIE. Firstly, extraction of sub-images of the face, right eye, and mouth from face images is conducted using Haar-like features and AdaBoost proposed by Viola and Jones. Secondly, sub-images are converted using LUT. LUT-based sub-regions are constructed by calculation of one pixel and near pixels. Finally, sub-images are classified male or female using DIE. The DIE value is computed with histogram levels of a grayscale difference image which has peak positions from -255 to +255, to prevent information sweeping. The performance evaluation is conducted using five standard databases, i.e., PAL, BioID, FERET, PIC, and Caltech facial databases. The experimental results show good performance in comparison with earlier methods. Keywords: Gender Classification, Difference Image Entropy.

1 Introduction Biometrics such as facial structure, fingerprints, iris structure, and voice can be used in many applications in fields such as human computer interaction, multimedia, security systems, and gate entrances. As time goes by, the research of facial image processing has become the focal point of many researchers’ attention. Facial images have a lot of information including information about gender, age, expression, and ethnic origin. In this paper a method for gender classification is proposed. Not surprisingly, a lot of research on facial gender classification has been done by researchers from in the field of computer science. Our gender classification method consists of three major steps; extraction of facial sub-images, construction of LUT, and calculation of DIE. We propose a new gender classification system using Shannon’s entropy-based DIE and LUT-based sub-images. The difference images are computed with pixel subtraction between input images and average images from reference images. For performance evaluation of the proposed method, we use five standard facial databases, i.e., PAL [1], BioID [2], FERET [3], PIC [4], and Caltech [5]. In addition, the proposed method is compared to the methods of using Euclidean with PCA and sobel image among the edge detection methods. V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 36–45, 2009. © Springer-Verlag Berlin Heidelberg 2009

Facial Gender Classification Using LUT-Based Sub-images and DIE

37

This paper is organized as follows. In section 2, we review some related work on the research of gender classification. Section 3 describes the extraction method of sub-images of the face, eye, and mouth, basic concepts of the Difference Image Entropy (DIE), LUT-based sub-images and the method of the proposed facial gender classification. The experimental results are described in Section 4. Finally, we draw conclusions in Section 5.

2 Related Work Earlier work on gender classification mainly originated in psychology and cognition research. Recently people began to consider this problem more technically. Several methods have been proposed for solving the gender classification problem. Among them, systems based on neural networks, PCA, decision trees, SVM, and AdaBoost classifiers can be mentioned [6]. Shakhnarovich et al. [7] proposed an AdaBoostbased gender classification method that achieved even better performance than SVM. In [8], a gender classification system is proposed based on the use of the SVM classifier. Other work includes Wu et al.'s LUT-based AdaBoost method that implemented a real-time gender classification system with comparative performance [9]. In 1948, Shannon introduced a general uncertainty-measure on random variables that takes different probabilities among states into account [10]. Given events occurring with probability P, the Shannon entropy is defined as Eq. (1). m

H = ∑ pi log i =1

m 1 = − ∑ pi log pi pi i =1

(1)

Shannon’s entropy can also be computed for an image, where the probabilities of the gray level distributions are considered in the Shannon Entropy formula. A probability distribution of gray values can be estimated by counting the number of times each gray value occurs in the image and dividing those numbers by the total number of occurrences. In this method, Shannon entropy is also a measure of dispersion of a probability distribution [11], however this system is an entropy-based method for face localization. Recently, we proposed DIE-based teeth verification [12] and DIE-based teeth recognition [13].

3 A Proposal for Facial Gender Classification The architecture for a DIE-based facial gender classification system using facial images consists of three steps. First, sub-images are extracted using Haar-like features and the AdaBoost algorithm. Second, extracted sub-images are made into LUT-based sub-images. Third, DIE is computed with the accepted input sub-image and average sub-image from male and female. Finally, the minimum value is selected via comparison processing of the DIE value and the facial gender result is returned to the user. The system flow-chart of a DIE-based biometric verification system using an LUT-based image is shown in Fig. 1.

38

J.-B. Jeon et al.

Fig. 1. Block Diagram of gender classification

3.1 Extraction of Sub-images from Original Image In this paper, we extract three sub-regions from facial images as illustrated in Figure 2. The detection process for face, right eye and mouth regions use Haar-like features and the AdaBoost method introduced by Viola and Jones. Extracted subimages are resized. The face is resized to 80ⅹ80 pixels, the right eye to 40ⅹ40 pixels, and the mouth to 50ⅹ30 pixels. 3.2 LUT-Based Sub-images LUT is a data structure, usually an array or associative array, used to replace a runtime computation with a simpler array indexing operation. The LUT used in the proposed method is defined as the following Eq. (2) – (5). LUT is computed using pixel subtraction between a grayscale value of one pixel and the average value for three pixels around one pixel.


39

Fig. 2. Extraction of three sub-images from original image

w −2 h−2

∑ ∑ LUT [ y ][ x ] = OI [ y ][ x ] − x =0 y =0

(2)

{OI [ y ][ x + 1] + OI [ y + 1][ x ] + OI [ y + 1][ x + 1]) / 3} h− 2

∑ LUT[ y][w − 1] = OI[ y][w − 1] − y =0

(3)

{OI[ y][w − 2] + OI[ y + 1][w − 2] + OI[ y + 1][w − 1]) / 3} w −2

∑ LUT[h − 1][x] = OI[h − 1][x] − x =0

(4)

{OI[ h − 2][ x] + OI [h − 2][ x + 1] + OI[ h − 1][ x + 1]) / 3} LUT [ h − 1][ w − 1] = OI[ h − 1][ w − 1] − {OI[ h − 2][ w − 2] + OI[ h − 2][w − 1] + OI[ h − 1][ w − 2]) / 3}

(5)

In above equation, OI is the original sub-image, y, x indicates height and width indexes, and w, h indicates the width and height of sub-images respectively. Finally, the pixel of LUT-based sub-images is added to a 100-gray value as shown in Eq. (6). The 100-gray value is an experimental value which is most suitable for gender classification. w −1 h −1

∑ ∑ Im g[ y ][ x] = LUT [ y ][ x] + 100 x=0 y =0

(6)

40

J.-B. Jeon et al.

3.3 Difference Image Entropy Difference Image Entropy is computed with histogram levels of a grayscale difference image which has a peak position from -255 to +255, to prevent information sweeping. The average image from the M reference sub-images is given in Eq. (7).

Saverage = In Eq. (7), S m ( x, y ) means the

1 M ∑ S m ( x, y ) M m=1

(7)

mth reference image. The difference image ( Ddiff )

is defined as Eq. (8). where Ddiff is computed by pixel subtraction between input subimages,

I input and average sub-images, and Saverage on random-collected gender

reference images.

Ddiff = I input − Saverage The DIE,

(8)

E g is defined as Eq. (9), and Pk means probabilities of the frequency

of histogram in difference images.

Eg = −

255

∑ Pk log 2 Pk =

k =−255

In addition, a probability

255

∑ P log

k =−255

k

2

1 Pk

(9)

Pk is defined as Eq. (10). Where a k indicates the fre-

quency of histogram from the -255 histogram levels to +255 histogram levels, and the sum and total of each histogram in the difference images G(T ) is given in Eq. (11).

Pk =

ak G(T )

G(T ) =

(10)

255

∑a

k =−255

k

(11)

4 Experiments and Results 4.1 Facial Databases Experiments for gender classification involved five standard databases of facial images, i.e., FERET, PIC, BioID, Caltech, and PAL. The sample images of frontal faces are shown in Fig. 3. We used a total of 3,655 frontal face images, 2,078 males and 1,577 females. We used 660 images from PIC with 438 males and 222 females. We used 705 images from FERET, with 400 males and 305 females. We used 1,270 images from BioID, with 746 males and 524 females. We used 440 images from Caltech, with 266 males and 174 females. And, we used 580 images from PAL, with


(a)

(b)

(c)

(d)

41

(e)

Fig. 3. Samples from facial databases: (a) PAL, (b) FERET, (c) PIC, (d) BioID, (e) Caltech

228 males and 352 females. To make the average image for males and females, we used 1,040 and 790 images of men and women, respectively. Also, we use 1,038 images of males and 787 images females to evaluate the performance of the gender classification system. Extraction of face, right eye, and mouth regions from facial images is performed by using Haar-like features and the AdaBoost algorithm. Facial regions are detected using the frontal face cascade. This face detector is scaled to the size of 24 × 24 pixels. The right eyes are extracted by our own training. The training process is implemented in a Microsoft Visual C++ 6.0 environment, simulated on a Pentium 2.6GHz machine. This right eye detector is scaled to the size of 24 × 12 pixels. We used 5,254 images for positive images of the right eye region and 10,932 images for negative images of the background. The training set consisted of five face databases and our own facial images acquired by webcam as shown figure 4.

Fig. 4. Left: Right eye images (positive). Right: Non-right eye images (negative).

In order to perform gender classification experiments, the program needs to preprocess original facial images. The preprocessing step is described below. For grayscale images, extracted sub-images are converted into grayscale images and normalized using histogram equalization to minimize the effect of illumination. Next, sobel edge images and LUT-based images are made and converted in the gray level. Figure 5 shows male and female grayscale sub-images and the sub-images for sobel edge detected images and LUT-based images.

42

J.-B. Jeon et al.

(a)

(b)

(c)

Fig. 5. The preprocessed sub-images: (a) grayscale images, (b) sobel edge detected images, (c) LUT-based images

Figure 6 shows the average image of sub-images for grayscale images, sobel edge detected images and LUT-based images. These average images are used to compute the DIE values with input images.

(a)

(b)

(c)

Fig. 6. The average images of sub-images:(a) grayscale images, (b) sobel edge detected images, (c) LUT-based images

4.2 Comparative Experiments We tested four methods of facial gender classification. The first method was conducted using grayscale based sub-images and DIE for each sub-region. The second method was conducted using sobel edge images of detected sub-images and DIE. Thirdly, the proposed LUT-based sub-images and DIE method in the paper was tested for each sub-region. Lastly, the method of Principal Component Analysis (PCA) and Euclidean for each sub-region is conducted as comparative experiment. The result of facial gender classification for sub-images is shown in Table 1 to Table 3. Table 1, Table 2, and Table 3 are results for the facial region, right eye region, and mouth region, respectively.


43

Table 1. Gender classification results for facial images

Male Grayscale image + DIE Female Sobel Edge Detected image + DIE

Male Female Male

LUT-based image +DIE Female Male PCA + Euclidean Female

Male

Female

646/1038 (62.23%) 126/787 (16.1%) 203/1038 (19.5%) 27/787 (3.5%) 815/1038 (78.5%) 246/787 (31.3%) 487/1038 (46.9%) 150/787 (19.1%)

392/1038 (37.77%) 661/787 (83.9%) 835/1038 (80.5%) 760/787 (96.5%) 223/1038 (21.5%) 541/787 (68.7%) 551/1038 (53.1%) 637/787 (80.9%)

Total

71.6%

52.7%

74.3%

61.6 %

Table 2. Gender classification results for right eye images


Male Female Male


Male

Female

743/1011 (73.4%) 354/765 (46.2%) 870/1011 (86.5%) 658/765 (86%) 691/1011 (68.3%) 369/765 (48.2%) 589/1011 (58%) 350/765 (45.7%)

268/1011 (26.6%) 411/765 (53.8%) 141/1011 (13.5%) 107/765 (14%) 320/1011 (31.7%) 396/765 (51.8%) 422/1011 (42%) 415/765 (54.3%)

Total

64.9%

55%

61.2%

56.5%

From the experiment results, it can be seen that the advantage of DIE is demonstrated. Also, LUT-based sub-images are better than grayscale sub-images and sobel edge detected sub-images. This method indicated classification rates with an overall classification rate of 74.3%. And the first, the second, and fourth methods showed a classification rate of 71.6%, 52.7%, and 61.6% for facial regions respectively. For the right eye region and mouth region, grayscale based images and the DIE method is 64.9% and 64.8% respectively. We can confirm two main results. First, the facial region indicated better performance than the right eye region and the mouth region. Secondly, the proposed LUT-based sub-images and the DIE method are generally better than three methods.

44

J.-B. Jeon et al. Table 3. Gender classification results for mouth images


Male Female Male


Male

Female

606/1038 (58.3%) 209/787 (26.5%) 1030/1038 (99.2%) 777/787 (98.7%) 241/1038 (23.2%) 320/787 (40.6%) 473/1038 (45.5%) 174/787 (22%)

432/1038 (41.7%) 578/787 (73.5%) 8/1038 (0.8%) 10/787 (1.3%) 797/1038 (76.8%) 467/787 (59.4%) 565/1038 (54.5%) 613/787 (78%)

Total

64.8%

56.9%

38.7%

59.5%

5 Conclusions In this paper, the method to classify whether an input image is male or female was proposed by using DIE and LUT-based sub-images. We conducted gender experiments for sub-images and four gender classification methods. In classification results for comparative experiments, the proposed LUT-based sub-images and the DIE method showed better performance than remainder methods with a classification rate of 74.5% for facial region. From this result, we confirm that DIE-based methods give reliable gender classification results. The gender classification system is expected to be applied to a live application in the field. In the future, it will be necessary to research more robust gender classification techniques including more variation in rotation, illumination, and other factors. Although we discuss the method of DIE on gender classification, it can be used in other facial expressions and age classifications. Also, more efforts should be paid on the combination of DIE and other pattern recognition algorithms.

Acknowledgment This research was supported by MIC, Korea under ITRC IITA-2008-(C1090-08010046), and the Korea Science and Engineering Foundation (KOSEF) grant funded by the Korean government (MEST) (No. 2008-000-10642-0).

References 1. Minear, M., Park, D.C.: A lifespan dataset of adult facial stimuli. Behavior Research Methods, Instruments Computers 36(4), 630–633 (2004) 2. http://www.bioid.com/downloads/facedb/index.php

Facial Gender Classification Using LUT-Based Sub-images and DIE 3. 4. 5. 6. 7. 8.

9.

10. 11. 12.

13.

45

http://www.frvt.org/FERET/default.htm http://PICS.psych.stir.ac.uk http://www.vision.caltech.edu/html-files/archive.htm Amin, T., Hatzinakos, D.: A Correlation Based Approach to Human Gait Recognition. In: Biometrics Symposium, pp. 1–6 (2007) Moghaddam, B., Ming-Hsuan, Y.: Learning Gender with Support Faces. IEEE Trans. Pattern Analysis and Machine Intelligence 24(5), 707–711 (2002) Shakhnarovich, G., Viola, P., Moghaddam, B.: A Unified Learning Framework for Real Time Face Detection and Classification. In: IEEE conf. on Automatic Face and Gesture Recognition 2002, pp. 14–21 (2002) Wu, B., Ai, H., Huang, C.: LUT-based AdaBoost for Gender Classification. In: Kittler, J., Nixon, M.S. (eds.) AVBPA 2003. LNCS, vol. 2688, pp. 104–110. Springer, Heidelberg (2003) Shannon, C.E.: A Mathematical Theory of Communication. The Bell Systems Technical Journal 27, 379–423 (1948) Alirezaee, S., Aghaeinia, H., Faez, K., Askari, F.: An Efficient Algorithm for Face Localization. International Journal of Information Technology 12(7), 30–36 (2006) Jong-Bae, J., Kim, J.-H., Yoon, J.-H., Hong, K.-S.: Teeth-Based Biometrics and Image Selection Method Using Difference Image Entropy. In: The 9th International Workshop on Information Security Applications (2008) Jeon, J.-B., Kim, J.-H., Yoon, J.-H., Hong, K.-S.: Performance Evaluation of Teeth Image Recognition System Based on Difference Image Entropy. In: IEEE conf. on ICCIT 2008, vol. 2, pp. 967–972 (2008)

Anthropometric Measurement of the Hands of Chinese Children Linghua Ran, Xin Zhang, Chuzhi Chao, Taijie Liu, and Tingting Dong China National Institute of Standardization, Zhichun Road, 4, Haidian District, Beijing 100088, China [email protected]

Abstract. This paper presents the results of a nationwide anthropometric survey conducted on children in China. Eight hand anthropometric dimensions were measured from 20,000 children with age ranged from 4 to 17 years old. Mean values, standard deviations, and the 5th, 95th percentile for each dimension were estimated. The dimension difference between age, gender and difference between Chinese and Japanese were analyzed. It was found that the mean values of the dimensions showed a gradual increase by age. The dimensions had no significant difference between genders for the children from 4 to 12, but the difference became significant for the children from 13 to 17. Comparison between Chinese and Japanese children showed that Chinese children tended to have relatively longer and broader hands than Japanese children. These data, previously lacking in China, can benefit the children’s products design. Keywords: Hand; anthropometric measurement; Chinese children.

1 Introduction Anthropometric data are essential for the correct design of various facilities. Without such data, the designs can not fit people properly. This is especially true for children. The comfort and functional utility of the workspace, equipments and products which designed based on the anthropometric data are related with children’s health and safety. Many anthropometric studies had been undertaken to determine the size of children [1][2][3][4]. In china, a nationwide anthropometric survey project for children from 4 to 17 was completed from 2005 to 2008. This survey measured more than 100 anthropometric dimensions, including body size as well as head, foot and hand size. The hand anthropometric data for the children are presented in this paper. The purpose is to determine hand dimensions in different age groups to facilitate the design of such products as toys, gloves and other components in their daily life.

2 Methods 2.1 Subjects China is a vast country with an area of over 96 million square kilometers. Children in different regions have large difference in body development status and body shape. To V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 46–54, 2009. © Springer-Verlag Berlin Heidelberg 2009

Anthropometric Measurement of the Hands of Chinese Children

47

make the anthropometric survey more representative, a stratified cluster sampling method was used to determine the distribution of the samples. The whole country was divided into six geographical areas, which was in accordance with the adult anthropometric survey in 1988[5]: north and northeast area, central and western area, the lower reaches of the Changjiang River area, the middle reaches of the Changjiang River area, Guangdong-Guangxi-Fujian area, Yunnan-GuizhouSichuan area. From the statistical point of view, the people within each area have similar body shape and body size, but body shape and size for the people in different areas are different with other. The sample size in each area was determined based on the distribution of children’s population reported by China National Bureau of Statistics [6]. One or two cities in each area were selected and some kindergartens, primary schools and high schools were taken from these cities. Within each kindergarten, primary school or high school selected, a number of classes were taken and all the children in which were measured until the number of children desired in per age group was met. According to Report on the Physical Fitness and Health Surveillance of Chinese School Students (2000) [7] and Report on the Second National Physical Fitness Surveillance (2000) [8], the children were subdivided into five age groups: preschool (4-6), lower primary (7-10), upper primary (11-12), middle school (13-15), high school (16-17). In this survey, for example, 10 years old means ones whose age is from 9.5 to 10.5 years old. The sample size in each age group was distributed according to the children’s body development status. The sample size of preschool age group may be smaller. Within lower primary and middle school age group, sample size should be increased, and for upper primary and high school age group the sample size can be reduced appropriately. Based on this sampling plan, body dimension data were obtained from more than 20,000 children in ten provinces distributed in the six geographical areas. 2.2 Dimension Measurements Instead of traditional Martin type anthropometer, a two-dimensional color scanner was adopted for hand anthropometric survey. The ratio of image size with the real hand size was 1:1 with a resolution of 150.They were kept in BMP format. The advantages of such system are that it would be much faster than Martin method of collecting hand data and it is applicable for a large-scale anthropometric survey. And it would provide a permanent record from which any measurement dimensions can be taken as needed. To achieve a greater scientific uniformity, measurements were always carried out on the right hand. Every subject was scanned with two hand postures. The first was with four fingers closing together and the thumb naturally outreached, putting on the scanning plane lightly. The second was with the five fingers outreached as far as possible, putting on the scanning plane lightly. After each scanning, a view to the scanning results was required to prevent the scanning failure caused by finger shifting. In each area, before starting the survey the measurement team was specially trained in anthropometric techniques and checked for consistency in their procedures to ensure the data reliability. The parents or teachers were asked to fill a form including their

48

L. Ran et al.

child’s name, sex, birth date and place, nationality, the school and grade, etc. The whole survey was completed in a period of about two years. 2.3 Data Processing and Statistical Analysis Hand Dimension Calculating Computer Software was developed. This programme allows the user to select anatomical points on both hand images on screen by means of a cursor. Once the points in each image have been identified, the programme can calculate hand length and breadth dimensions automatically. In this paper, eight anthropometric measurement dimensions were taken: hand length, hand breadth at metacarpals, palm length perpendicular, index finger length, thumb length, middle finger length, index finger breadth (proximal) and index finger breadth (distal). Except thumb length and middle finger length, the definitions of other six hand dimensions were taken from ISO 7250:2004[9]. The dimension values obtained were categorized according to sex and age groups and abnormity data examination was conducted. The extreme outliers and unreasonable results were identified and eliminated carefully by using 3σ test, peak value test and logical value test. The Statistical Package for the Social Sciences (SPSS) for Windows version 16.0 was used in the following statistical analysis. The descriptive statistics, including arithmetic means (M), standard deviations (SD), and percentiles (5th and 95th) of the above measurements were calculated for both boys and girls.

3 Results The statistical data of eight hand anthropometric dimensions are presented in table 1-6, including the number of subjects, gender (boys and girls) and age (4 to 17 years old). Estimates of mean, standard deviation (SD) and the 5th, 95th percentile are included. All dimensions are reported in mm.

4 Discussion 4.1 Differences between Age Groups From table 1 to 6, it can be found that all mean values for the eight dimensions increase gradually by age. Because hand length and breadth are the basis to establish hand sizing system [10], hand length and breadth are further analyzed to show the difference between age groups. Table 6 and table 7 show the interclass increase value and relative odds ratio of the mean values. Both length and breadth show a trend for significant dimension increase by age in boys and girls and there are clear differences between the five age groups. For boys, the difference of hand length between (4-6) and (7-10) age group is 20.1mm. From (7-10) to (11-12), the length of boys increased by 16.8mm, and from (11-12) to (13-15) the increase value is 18.1mm. Also for girls, the increase of mean values of hand length are 20.7mm,18.6mm, 7.8mm and 0.9mm respectively for the age group from (4-6) to (16-17) .


49

Table 1. The statistical values of hand anthropometric dimensions (4–6 years old) Dimensions(mm)

Boys(N=1138)

Girls(N=1140)

M 124.1

SD 9.3

P5 110.1

P95 138.7

M 122.0

SD 8.4

P5 108.2

P95 136.5

Hand breadth at metacarpals Palm length perpendicular Index finger length

58.4

4.0

52.3

64.7

56.5

3.7

50.7

62.6

71.0

5.6

62.7

80.4

69.3

5.2

61.0

78.2

48.2

4.0

42.2

55.1

48.0

3.8

41.6

53.9

Thumb length

39.2

3.7

33.9

45.2

38.5

3.5

32.9

44.3

Hand length

Middle finger length

53.8

4.5

47.2

61.3

53.6

4.1

47.0

60.5

Index finger breadth, proximal Index finger breadth, distal

14.3

1.4

12.1

16.5

13.8

1.3

11.7

16.0

12.7

1.3

10.8

14.9

12.3

1.2

10.5

14.5


Boys(N=2239) M

Girls(N=2115)

SD

P5

P95

M

SD

P5

P95

Hand length

144.2

9.9

128.7

161.9

142.7

10.5

126.3

161.2


65.5

4.5

58.5

73.2

63.4

4.3

56.6

71.1

82.3

6.0

72.8

92.8

80.7

6.2

70.8

91.8

56.0

4.4

49.0

63.8

56.2

4.6

48.9

64.2

Thumb length

45.9

4.0

39.9

52.9

45.9

4.3

39.1

53.4

Middle finger length Index finger breadth, proximal Index finger breadth, distal

62.4

4.8

54.9

70.8

62.6

5.0

54.9

71.5

15.7

1.3

13.6

18.0

15.1

1.3

13.1

17.4

14.1

1.2

12.2

16.3

13.6

1.1

11.9

15.7

Table 6 and 7 also reveal that for both boys and girls, there is a stage in which the hands have a relatively fast growth rate. For boys, it is in the 4 to 15 years old, but for girls it is the 4 to 12 years old. When the boy grows up to 15, girls to 12, the hand growth rate slows down. According to the Report on the Physical Fitness and Health Surveillance of Chinese School Students (2000), children have a sudden increase in youth period. During this period, their physical size has an obvious change. In that report, the periods are 12-14 and 10-12 for the boys and girls. It can be found that there is a certain degree of correlation between the hand dimension changes and age group. This exact relationship may be verified through future research.

50

L. Ran et al. Table 3. The statistical values of hand anthropometric dimensions (11–12 years old)

Dimensions(mm) Hand length Hand breadth at metacarpals Palm length perpendicular Index finger length Thumb length Middle finger length Index finger breadth, proximal Index finger breadth, distal

Boys(N=2098)

Girls(N=2019)

M

SD

P5

P95

M

SD

P5

P95

161.0

10.9

144.6

180.7

161.3

9.3

145.9

176.5

71.8

5.1

64.3

81.1

70.0

4.1

63.5

76.9

91.8

6.5

81.6

103.6

90.9

5.6

81.5

100.1

62.3

4.7

55.0

70.6

63.5

4.5

56.3

71.0

51.7

4.4

45.0

59.6

52.3

4.0

45.9

59.2

69.6

5.3

61.7

79.3

70.9

4.8

62.9

78.8

17.0

1.5

14.6

19.5

16.4

1.4

14.3

18.7

15.1

1.3

13.1

17.5

14.8

1.3

12.8

17.0


Boys(N=2942)

Girls(N=2795)

M 179.1

SD 10.9

P5 159.6

P95 196.1

M 169.1

SD 7.8

P5 156.4

P95 181.7


79.5

5.2

70.5

87.8

73.0

3.7

67.1

79.2

101.6

6.5

90.4

112.1

95.2

5.0

87.2

103.5

69.3

5.1

60.7

77.5

66.8

4.0

60.5

73.2

Thumb length

57.5

4.6

49.9

64.9

54.4

3.6

48.6

60.6

Middle finger length Index finger breadth, proximal Index finger breadth, distal

77.7

5.5

68.2

86.6

74.2

4.2

67.5

81.3

18.6

1.7

15.7

21.3

17.3

1.4

15.2

19.6

16.4

1.5

14.0

18.8

15.4

1.2

13.4

17.5

Hand length

4.2 Gender Differences The differences between boys and girls can be obtained in table 1 to 6. In table 1 to 3 most of the boys’ dimensions have a litter higher than girls’, but the differences are not obvious. The differences of mean values range from -1.3mm (index finger length and middle finger length in 11-12 age group) to 2.1mm (hand length in 4-6 age group and hand breadth in 7-10 age group). In table 4 and 5, the gender differences have become significant. In the age group of (13-15), the mean differences range from 1.0mm (index finger breadth, distal) to 10.0mm (hand length).The differences keep increasing in the 16-17 age group by a range from 1.3mm (index finger breadth, distal) to 14.7 mm (hand length).


51


Boys(N=1840)

Girls(N=1910)

M 184.7

SD 8.9

P5 170.2

P95 198.8

M 170.0

SD 8.0

P5 157.2

P95 183.2


82.0

4.4

75.2

89.2

73.4

3.6

67.4

79.4

105.0

5.8

95.7

114.9

96.0

5.1

88.2

104.5

71.7

4.2

65.1

78.6

67.0

3.9

60.5

73.8

Thumb length

59.2

4.0

52.8

66.2

54.4

3.8

48.2

60.6

Middle finger length

80.2

4.6

72.6

87.5

74.4

4.3

67.5

81.7

Index finger breadth, proximal Index finger breadth, distal

19.2

1.5

16.8

21.5

17.6

1.3

15.5

19.7

16.8

1.4

14.7

19.2

15.6

1.2

13.7

17.6

Hand length

Table 6. Mean value increase of hand length and breadth in different age groups (for boys)

(4-6) to (7-10)

Hand length Interclass increase (mm) 20.1

Relative odds ratio (%) 116.2

Hand breadth at metacarpals Interclass Relative odds increase (mm) ratio (%) 7.1 121.2

(7-10) to (11-12)

16.8

111.7

6.3

109.6

(11-12) to (13-15)

18.1

111.2

7.7

110.7

(13-15) to (16-17)

5.6

103.1

2.5

103.1

Age group

Table 7. Mean value increase of hand length and breadth in different age groups (for girls)

(4-6) to (7-10)

Hand length Interclass increase (mm) 20.7

Relative odds ratio (%) 117.0

Hand breadth at metacarpals Interclass Relative odds increase (mm) ratio (%) 6.9 112.2

(7-10) to (11-12)

18.6

113.0

6.6

110.4

(11-12) to (13-15)

7.8

104.8

3.0

104.3

(13-15) to (16-17)

0.9

100.5

0.4

100.5

Age group

The significance of the differences between boys and girls was also examined by Mollision’s method [11] [12] across age groups. The formula is as followed:

S=

A1 − A11 × 100 S A11

(1)

52

L. Ran et al.

A1 — Arithmetic mean of boys in each age group; A11 — Arithmetic mean of girls in each age group; S A11 — Standard deviation of girls in each age group; Differences between the means of boys and girls are expressed in each measurement by percentage deviation. When the indicator of mean deviation is positive, then the value of the mean of boys is bigger than the mean of girls. The situation is reversed when the indicator is negative. If the result exceeds 100, then it shows that there is a significance difference between the two groups. The indicator of mean deviation was calculated. The results showed that from 4 to 12, no significant differences were found between boys and girls in all the eight hand dimensions. In (13-15) age group, the differences were significant in the hand length, hand breadth and the palm length. In the (16-17) age group, all the eight dimensions had significant difference between boys and girls, especially the hand length, breadth and palm length. The results showed that the hand dimensions have very little differences between boys and girls for the children from 4-12 years old, which may imply that it was not necessary to consider gender difference in the design of some hand related products for children younger than 12 years old. But for children older than 12 years old, the difference should be taken into consideration. 4.3 Differences between Chinese and Japanese Children China and Japan are both in the eastern of Asia, with similar ethnic characteristics and cultural traditions. It is meaningful to find out whether there are significant differences in body dimensions of the children in these two groups. Table 8. Comparison of mean values between China and Japan in four age groups ( boys) Dimensions(mm)

Hand length Hand breadth at metacarpals Palm length perpendicular Index finger length Thumb length Middle finger length Index finger breadth, proximal Index finger breadth, distal

(7-10)

(11-12)

(13-15)

(16-17)

China

Japan

China

Japan

China

Japan

China

Japan

144.2

141.6

161.0

157.0

179.1

175.8

184.7

182.6

65.5

61.7

71.8

68.5

79.5

76.1

82.0

79.2

82.3

80.2

91.8

88.4

101.6

99.3

105.0

103.5

56.0

54.1

62.3

60.4

69.3

67.5

71.7

69.8

45.9

46.0

51.7

51.5

57.5

58.0

59.2

60.2

62.4

61.4

69.6

68.6

77.7

76.5

80.2

79.2

15.7

16.1

17.0

17.3

18.6

18.7

19.2

19.8

14.1

14.0

15.1

15.0

16.4

16.2

16.8

17.3


53

Table 9. Comparison of mean values between China and Japan in four age groups( girls) Dimensions(mm)

Hand length Hand breadth at metacarpals Palm length perpendicular Index finger length Thumb length Middle finger length Index finger breadth, proximal Index finger breadth, distal

(7-10)

(11-12)

(13-15)

(16-17)

China

Japan

China

Japan

China

Japan

China

Japan

142.7

140.9

161.3

158.8

169.1

167.5

170.0

167.7

63.4

60.4

70.0

67.5

73.0

70.1

73.4

70.8

80.7

79.5

90.9

89.1

95.2

94.0

96.0

94.3

56.2

54.3

63.5

61.9

66.8

65.9

67.0

65.6

45.9

45.4

52.3

51.5

54.4

54.3

54.4

55.3

62.6

61.5

70.9

69.6

74.2

73.4

74.4

73.4

15.1

15.5

16.4

16.9

17.3

17.6

17.6

17.8

13.6

13.5

14.8

14.6

15.4

15.1

15.6

15.3

The Japanese data were collected during 1992 to 1994 by the Institute of Human Engineering for Quality of Life (HQL).More than 5,000 children from 7 to 17 years old were included in the survey. Because there’s no hand data for children from 4 to 6 in Japan, only the hand mean values in four age groups are displayed in table 8 and 9. It is found that both Chinese boys and girls have greater values in hand length, hand breadth palm length in the four age groups. It appears that Chinese children have longer and broader hands than Japanese children. As to the three finger length dimensions, most of the Chinese children have a relatively higher value than Japanese children. Only one finger breath dimension was compared and it showed that Japanese children had wider 2nd joint and narrower1st joint than Chinese children. Whether there are differences between other fingers’ breadth, more data should be extracted from the Chinese children hand images.

5 Conclusion This study was conducted to provide hand anthropometric information of Chinese children from 4 to 17 years old, which could be used for the ergonomic design of workspace and products. A total of eight hand anthropometric dimensions extracted from 20,000 children are listed in the forms of mean, standard deviation and percentile values. The differences among age groups, between boys and girls groups, and between Chinese and Japanese are discussed. The results showed that the differences between the age groups were significant. In (13-15) age group, the gender difference was significant in the hand length, hand breadth and the palm length and in the (16-17) age group; all the eight dimensions had significant difference between boys and girls. Chinese children had longer and broader hands than Japanese children, whereas Japanese children had wider 2nd joint and narrower 1st joint than Chinese children. In

54

L. Ran et al.

this study, the hand dimensions were extracted from 2-D images. The thickness and girth data about hands and fingers have not been obtained. Nevertheless, this survey provides the first hand anthropometric database of Chinese children.

References 1. Wang, M.-J.J., Wang, E.M.-y., Lin, Y.-C.: The Anthropometric Database for Children and Young Adults in Taiwan. Applied Ergonomics 33, 583–585 (2002) 2. Kayis, B., Ozok, A.F.: Anthropometry Survey Among Turkish Primary School Children. Applied Ergonomics 22, 55–56 (1991) 3. Steenbekkers, L.P., Molenbroek, J.F.: Anthropometric Data of Children for Non-specialist Users. Ergonomics 33(4), 421–429 (1990) 4. Prado-Leon, L.R., Avila-Chaurand, R., Gonzalez-Munoz, E.L.: Anthropometric Study of Mexican Primary School Children. Applied Ergonomics 32, 339–345 (2001) 5. Chinese National Standard, GB10000-1988: Human Dimension of Chinese Adults. Standards Press of China, Beijing (1988) 6. National Bureau of Statistics: Chinese Demographic Yearbook. Statistics Press, China (2003) 7. Ministry of Education of the People’s Republic of China, General Administration of Sports of China, Ministry of Health of the People’s Republic of China, Ministry of Science and Technology, Sports and health study group of Chinese Students Allocation: Report on the Physical Fitness and Health Surveillance of Chinese School Students. Higher Education Press, Beijing (2000) 8. General Administration of Sports of China: Report on the Second National Physical Fitness Surveillance. Sports University Press, Beijing (2000) 9. International Standard, ISO 7250: Basic Human Body Measurements for Technological Design. International standard Organization (2004) 10. Chinese National Standard, GB16252-1996: Hand Sizing system-Adult. Standards Press of China, Beijing (1996) 11. Hu, H., Li, Z., Yan, J., Wang, X., Xiao, H., Duan, J., Zheng, L.: Anthropometric Measurement of the Chinese Elderly Living in the Beijing Area. International Journal of Industrial Ergonomics 37, 303–311 (2007) 12. Nowak, E.: Workspace for Disabled People. Ergonomics 32(9), 1077–1088 (1989)

Comparisons of 3D Shape Clustering with Different Face Area Definitions Jianwei Niu1,∗, Zhizhong Li2, and Song Xu2 1

School of Mechanical Engineering, University of Science and Technology Beijing, Beijing, 100083, China Tel.: 86-131-61942805 [email protected], [email protected] 2 Department of Industrial Engineering, Tsinghua University, Beijing, 100084, China [email protected], [email protected]

Abstract. The importance of fit for face-related wearing products has introduced the necessity for better definition of face area. In this paper, three definitions of face area are compared on the context of Three dimensional (3D) face shape similarity based clustering. The first method defines the face area by spanning from the whole head grid surface by the front π/2 wedge angle along a line going through the centroid and pointing to the top of the head. The second method defines the face area as the grid surface enclosed by several anthropometric landmark points (sellion, both zygions, and menton) on the facial surface. The zonal surface where the respirator interferes with the wear’s face is taken as the third alternative definition for the comparative study. By utilizing the block-distance measure, each face was converted into a compact block-distance vector. Then, k-means clustering was performed on the vectors. 376 3D face data sets were tested in this study. One-way ANOVA on the block distance based vectors was conducted to evaluate the influence on clustering results by utilizing different face area definitions. No difference was found at the significant level of 0.05. However, the cluster membership shows great difference between different definitions. This emphasizes the value of the selection of face area in 3D face shape-similarity-based clustering. Keywords: 3D anthropometry; face area; shape comparison; clustering.

1 Introduction Of all the biometrics features, face is among the most common ones [1]. Face anthropometry is a focused issue over the past years. It has applications in clinical diagnostics, cosmetic surgery, forensics, arts and other fields. For example, comparison with patient’s face anthropometric data can help to indicate the existence of deformities, possibly leading to discovery of an illness [2]. If size and shape of a deformity are quantifiable, the surgeon can make more exact statements about necessary corrections [3]. Before 3D digitalizing technology emerged, traditional anthropometry was based on one-dimensional (1D) dimensions. Fortunately, with the wide availability of 3D ∗

Corresponding author.

V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 55–63, 2009. © Springer-Verlag Berlin Heidelberg 2009

56

J. Niu, Z. Li, and S. Xu

scanning technologies, it is convenient to acquire the 3D data of the human body. Understanding the 3D shape variation is essential to many scientific activities such as personal identification, population accommodation, human computer interaction, and image retrieval, etc [4]. Extraction of biologically important information on shape variance from 3D digitized samples has been developed as geometric morphometrics, which has now found extensive applications to 3D human data, such as pathology, archaeology, primatology, paleoanthropology, and reconstructive craniofacial surgery [5]. For example, Hennessy et al. [6] made an effort by using 3D face shape to establish a relationship between facial morphogenesis and adult brain function as a basis for conducting subsequent studies in schizophrenia. How to use 3D anthropometry to obtain proper fit of wearing products has been excessively addressed [7-11], while how to use 3D face anthropometry for the fit design purpose has not been well investigated yet. As an example, Mochimaru and Kouchi [12] used Free Form Deformation (FFD) method in the analysis of 3D human face forms for spectacle frames design. As typical face-related wearing products, respirators have been widely used across numerous fields. The current sizing for respirators is based on some linear measurements. In USA, the respirator RFTP with the proper facial anthropometric dimensions should specify tightness of fit satisfactorily for >95% of the targeted race group [13, 14]. However, NIOSH’s research indicated that the LANL panel for full-facepiece respirators accommodated only 84% of current civilian subjects [15]. Utilizing 3D facial shape information appears to be a promising avenue to overcome some of the limitations of current 1D-measurement based sizing systems and widened the opportunities for improving the design of face-related products. However, unlike some other biometrics features such as iris, retina, and fingerprint, it’s usually difficult to define the face area strictly, especially in 3D form. Various definitions of face area have been introduced in the past. There is considerable interest in the assessment of 3D shape clustering with different face area definitions. In this paper, three definitions of face area are compared on the context of 3D face shape similarity based clustering. The remainder of this paper is organized as follows. In Section 2, we introduce the method. Section 3 reports the results and gives some discussions. Finally, Section 4 summarizes this study.

2 Methods 2.1 Different Face Area Definitions The raw 3D head data of 376 young male Chinese soldiers (aged from 19 to 23) are used [16]. All faces are aligned by translating the origin of the Cartesian coordinate system to a specified point. The y and z axis values of the new origin are the average values of the y and z axis values of sellion, both zygions, both cheilions, and menton, and the x axis value of the new origin equals the x axis value of sellion. The landmarks, defined in accordance with 1988 Anthropometric Survey of the U.S. Army Personnel Project [17], were located manually by the same experienced investigator.

Comparisons of 3D Shape Clustering with Different Face Area Definitions

57

Three definitions of face area are then introduced. The first method defines a face as the surface spanned from the whole heads by the front π/2 wedge angle along a line going through the centroid and pointing to the top of a head. Here the centroid was computed as the point with averaged coordinates of all points. This selection criterion of the front π/2 wedge angle is based on the observation of the average angle spanned forward of all samples in this study, based on which almost the whole face coverage could be obtained. The second method defines a face as a grid characterized by four facial landmarks, i.e. sellion, both zygions and menton. The top of the face area lies 50mm above the sellion. This is based on the subjective judgment of the position of a full-face respirator on the forehead. The zonal surface on the face where a certain level of compression force will be applied around is the third definition of face area for our study, since it is the actual interfacing area between equipment and face. If the surface of the contacting strip is not well consistent with the zonal surface, the compression force will be unevenly distributed and cause discomfort. In our previous study [18], a block-division method was proposed to convert each 3D surface into a block-distance based vector. In the current case study, each face surface was divided into 30 (6X5) blocks, and the zonal surface consists of the peripheral 18 blocks. 2.2 Comparison between Different Face Area Definitions For each face area definition, k-means clustering was applied to the block distance-based vectors referring to the inscribed surface of all samples. Wang and Yuan [19] presented a new oxygen mask sizing system where they partitioned the Chinese face samples into four sizes, namely small size, medium-narrow size, medium-wide size and large size. For comparison with their method in the future, the number of K for the clustering was also set as four in this case study. The representative face surface of each cluster is obtained by calculating the average coordinates of the points of the samples belonging to the cluster. Then the block distance between a sample surface and the representative surface can be constructed as S1' and S2' .

S1' can be calculated as,

S1' =

1 n ∑ dis( p j ) n j =1

(1)

where pj is the jth point, n represents the number of points of a face, and Euclidean distance between two corresponding points on the sample and the representative surface, dis (pj), was computed. S2' can be calculated as,

58


S2' =

1 n dis( p j ) − S1' ∑ n j =1

(2)

S2 describes the local shape variation between the sample and the representative surface. Tests for normality are conducted on all S1' and S2' values using the One-Sample Kolmogorov-Smirnov test. Tests for the homogeneity-of-variance of the variables are conducted using the Levene test. Finally, multiple comparisons of means between the three face area definitions were conducted by using One-way ANOVA.

3 Results 3.1 Face Area Definitions The landmarks labeled manually are illustrated in Fig. 1. Considering the difficulty in identifying landmarks on a virtual image without the ability of feeling the force feedback to palpate and locate bony landmarks as in traditional anthropometry, the landmark-label result has passed visual check from several views of the 3D head under CAD software Unigraphics. The face areas according to the three definitions are shown in Figs. 2-4, respectively. For the zonal surface, the average value of the side length of each peripheral block is about 25mm. This is consistent with the width of the contacting strip of full-face respirator in real application.

L5

L1

L6

L2

L3

L4

(a) front view

(b) side view

Fig. 1. Interactive manual identification of landmarks (pink dots, L1: sellion; L2: right zygion;

；

L3: right cheilion; L4: menton L5: left zygion; L6: left cheilion)


(a) front view

(b) side view

Fig. 2. The first definition of face area

(a) front view

(b) side view

Fig. 3. The second definition of face area

(a) front view

(b) side view

Fig. 4. The third definition of face area

3.2 Comparison between Different Face Area Definitions The average face area of each cluster was generated, as shown in Fig. 5.

59

60


First definition

Second definition

Third definition

Face definition

Front view

Side view

Bottom view

Fig. 5. Different views of the merged average faces of clusters

Tests for normality of S1' and S2' values showed p values less than 0.05, resulting in rejection of the null hypothesis. Afterwards each S1' and S2' values were transformed into their corresponding natural logarithmic values, denoted as ln S1' and ln S2' respectively. One-Sample Kolmogorov-Smirnov test on the ln S1' and ln S2' values resulted in p values greater than 0.05 (p=0.566 and 0.106 respectively). Levene test on the ln S1' and ln S2' values showed p values of 0.138 and 0.000, respectively. This indicated that the homogeneity-of-variance for ln S1' was satisfied at the significance level of 0.05, while for ln S2' the homogeneity-of-variance was not satisfied. So when multiple comparisons in One-way ANOVA was conducted, Least-significant difference (LSD) test was used for ln S1' , while Tamhane's T2 was used for ln S2' .


61

The descriptives of block-distance measures, i.e., ln S1' and ln S2' , for different face definitions are shown in Table 1. It can be seen that for the average values of ln S1' , the difference between the first two face definitions is almost ignorable. While, the average value of ln S1' for the zonal face area is greater than the first two alternatives. In contrast, the average value of ln S2' for the zonal face area is smaller than the first two alternatives. This can be explained from the definition of S1' and S2' which reflect the local size and shape differences, respectively. The zonal face area only covers a small portion of the whole face, i.e., the peripheral blocks of the face. For the whole face, the distance is averaged over a big surface, thus the effect of S1' is weakened. Since the shape variation of the whole face is greater that that of the peripheral portion of the face, S2' values of the whole face are greater. What is more, compared with the center face area consisting nose, mouth, and eyes, the zonal face area usually demonstrates more regular geometry. Therefore, the shape variation of the zonal face area is smaller, and the effect of S2' is weakened. Table 1. Descriptives of block-distance measures (N=376)

ln S1'

ln S2'

Face definition 1 2 3 1 2 3

M 1.21 1.21 1.25 0.09 0.10 0.07

SD 0.317 0.324 0.346 0.383 0.381 0.455

As shown in Table 2, One-way ANOVA results demonstrated p values greater than 0.05. Such results lead to no rejection of the null hypothesis at the significance level of 0.05. However, the p values for ln S1' between the first and third definition (0.129), and between the second and third definition (0.070), both show marginally significant difference. Cluster membership variation with different face areas was investigated and summarized in Table 3. Compared with the second definition, the numbers of samples whose cluster membership has changed with the first and the third definition are 83 and 30, respectively. Whereas, the number of samples with changed membership between the first and third definition is 79. It can be seen that the the membership variation between the second and third definition is much smaller than that between the first and second definition. These membership differences may indicates that the face area definition should be considered according to design requirements when developing a sizing system for face-interfaced products.

62

J. Niu, Z. Li, and S. Xu Table 2. Multiple comparisons in One-way ANOVA

Dependent Variable

Group I

Group J

Mean Difference (I-J)

Std. Error

Sig.

ln S1'

1 2 0.01 0.024 0.767 1 3 -0.04 0.024 0.129 2 3 -0.04 0.024 0.070 ' 1 2 -0.02 0.028 0.894 ln S2 1 3 0.02 0.031 0.935 2 3 0.03 0.031 0.610 Note: Group 1, 2, and 3 are the first, second and third face definitions, respectively. Table 3. Cluster membership change (N=376)

Cluster ID 1 2 3 4 number of change

Second definition 40 114 48 174 -

Sample size First definition 37 160 48 131 83

Third definition 35 137 49 155 30

4 Conclusions This study investigates the influence of face area definition on 3D face shape clustering. Though no significant difference is found for the block-distance measures between these three face definitions, the cluster membership shows remarkable difference between the first definition and the latter two alternatives. This underlines the potential value of the selection of face area for assessing the face shape variation among the population and designing better fitted face-related products.

Acknowledgements The study is supported by the National Natural Science Foundation of China (No.70571045).

References 1. Zhou, M.Q., Liu, X.N., Geng, G.H.: 3D face recognition based on geometrical measurement. In: Li, S.Z., Lai, J.-H., Tan, T., Feng, G.-C., Wang, Y. (eds.) SINOBIOMETRICS 2004. LNCS, vol. 3338, pp. 244–249. Springer, Heidelberg (2004)


63

2. McCloskey, E.V., Spector, T.D., Eyres, K.S., Fern, E.D., O’Rourke, N., Vasikaran, S., Kanis, J.A.: The assessment of vertebral deformity: A method for use in population studies and clinical trials. Osteoporosis International 3(3), 138–147 (1993) 3. Kaehler, K.: A Head Model with Anatomical Structure for Facial Modeling and Animation. Max-Planck-Institut für Informatik in Saarbrücken, Germany (2003) 4. Godil, A.: Advanced human body and head shape representation and analysis. In: Duffy, V.G. (ed.) HCII 2007 and DHM 2007. LNCS, vol. 4561, pp. 92–100. Springer, Heidelberg (2007) 5. Hennessy, R.J., McLearie, S., Kinsella, A., Waddington, J.L.: Facial surface analysis by 3D laser scanning and geometric morphometrics in relation to sexual dimorphism in cerebral-craniofacial morphogenesis and cognitive function. Journal of Anatomy 207(3), 283–295 (2005) 6. Hennessy, R.J., McLearie, S., Kinsella, A., Waddington, J.L.: Facial Shape and Asymmetry by Three-Dimensional Laser Surface Scanning Covary With Cognition in a Sexually Dimorphic Manner. The Journal of Neuropsychiatry and Clinical Neurosciences 18, 73–80 (2006) 7. Whitestone, J.J., Robinette, K.M.: Fitting to maximize performance of HMD systems. In: Melzer, J.E., Moffitt, K.W. (eds.) Head-Mounted Displays: Designing for the User, pp. 175–206. McGraw-Hill, New York (1997) 8. Meunier, P., Tack, D., Ricci, A., Bossi, L., Angel, H.: Helmet accommodation analysis using 3D laser scanning. Applied Ergonomics 31, 361–369 (2000) 9. Hsiao, H.W., Bradtmiller, B., Whitestone, J.: Sizing and fit of fall-protection harnesses. Ergonomics 46(12), 1233–1258 (2003) 10. Witana, C.P., Xiong, S.P., Zhao, J.H., Goonetilleke, R.S.: Foot measurements from three-dimensional scans: A comparison and evaluation of different methods. International Journal of Industrial Ergonomics 36, 789–807 (2006) 11. Au, E.Y.L., Goonetilleke, R.S.: A qualitative study on the comfort and fit of ladies’ dress shoes. Applied Ergonomics 38(6), 687–696 (2007) 12. Mochimaru, M., Kouchi, M.: Proper sizing of spectacle frames based on 3-D digital faces. In: Proceedings: 15th Triennial Congress of the International Ergonomics Association (CD ROM), Seoul, Korea, August 24-29 (2003) 13. National Institute for Occupational Safety Health NIOSH, DHEW/NIOSH TR-004-73. In: McConville, J.T., Churchill, E., Laubach, L.L. (eds.) Cincinnati, OH: National Institute for Occupational Safety and Health. pp. 1–44 (1972) 14. Zhuang, Z.: Anthropometric research to support RFTPs. In: The CDC workshop on respiratory protection for airborne infectious agents, Atlanta, GA (November 2004) 15. Federal Register/Notice: Proposed Data Collections Submitted for Public Comment and Recommendations 67(16) (2002) 16. Chen, X., Shi, M.W., Zhou, H., Wang, X.T., Zhou, G.T.: The "standard head" for sizing military helmets based on computerized tomography and the headform sizing algorithm (in Chinese). Acta Armamentarii. 23(4), 476–480 (2002) 17. Cherverud, J., Gordon, C.C., Walker, R.A., Jacquish, C., Kohn, L.: Northwestern University of EVANSTON IL, 1988 Anthropometric Survey of U.S. Army Personnel: Correlation Coefficients and Regression Equations, Part 1 Statistical Techniques, Landmark and Measurement definition (TANICK/TR-90/032), pp. 48–51. U.S. Army Natick Research, Development and Engineering Center Evanston, Natick, MA (1990) 18. Niu, J.W., Li, Z.Z., Salvendy, G.: Multi-resolution shape description and clustering of three-dimensional head data. Ergonomics (in press) 19. Wang, X.W., Yuan, X.G.: Study on type and sizing tariff of aircrew oxygen masks. Journal of Beijing University of Aeronautics and Astronautics 27(3), 309–312 (2001)

Block Division for 3D Head Shape Clustering Jianwei Niu1, Zhizhong Li2, and Song Xu2 1

School of Mechanical Engineering, University of Science and Technology Beijing, Beijing, 100083, China 2 Department of Industrial Engineering, Tsinghua University, Beijing, 100084, China [email protected]

Abstract. In our previous Three Dimensional (3D) anthropometric shape clustering study, block-division technique is adopted. The objective of this study was to examine the sensitivity of clustering results on block-division. Such a block-division technique means to divide each 3D surface into a predefined number of blocks. Then by using a block-distance measure, each surface is converted into a block-distance based vector. Finally, k-means clustering is performed on the vectors to segment a population into several groups. Totally 447 3D head samples have been analyzed in the case study. The influence of block division number on clustering was evaluated by using One-way ANOVA. No significant difference was found between the three block division alternatives. This means the adopted method is robust to block division. Keywords: Three dimensional anthropometry; block-division; clustering; sizing.

1 Introduction In the past decades, several large scale 3D anthropometric surveys have been conducted, such as Civilian American and European Surface Anthropometry Resource (CAESAR) [1], SizeUK [2], SizeUSA [3], etc. An international collaboration named World Engineering Anthropometry Resource (WEAR) brings together a wealth of different anthropometric data collected across the world [4]. 3D anthropometric shape analysis has found many applications such as in clinical diagnostics, cosmetic surgery, forensics, arts, and entertainment as well as in other fields. How to utilize 3D anthropometric data to improve the fitting level of wearing products has also gained great attention from the ergonomics and human factors [5-14]. An effective way to design fitting products is to analyze the shape of human body forms and classify a specific population into homogeneous groups. Traditionally, some One Dimensional (1D) measurements were usually selected as key dimensions for the analysis of human body variation [15]. Unfortunately, there are some drawbacks in such traditional sizing methods. The most important is that geometric characteristics and internal structure of human surface are not adequately considered, which may lead to design deficiency on fitting comfort [16]. For example, studies have disclosed that foot length and width measures are insufficient for proper fit though most consumers usually select footwear based on the two measurements [17, 18]. V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 64–71, 2009. © Springer-Verlag Berlin Heidelberg 2009

Block Division for 3D Head Shape Clustering

65

Considering the inherent abundant information contained in 3D anthropometric data, sizing methods based on 3D anthropometric data may be able to overcome the drawbacks of traditional sizing methods. However, this seems not an easy task. In our previous study [19], a block-division method was proposed to convert each 3D surface into a block-distance based vector, which reflects both size and shape difference. Such vectors were then used as the input of k-means clustering algorithm. The influence of the block division number on the 3D shape clustering is further studied in this paper. The remainder of this paper is organized as follows. Section 2 introduces the proposed method. A case study of 446 3D head samples is then presented in Section 3. Finally, Section 4 concludes this paper.

2 Methods 2.1 Block Division of Head Data The raw 3D data of 446 head of young male Chinese soldiers (aged from 19 to 23) were collected by a Chinese military institute in 2002 [20]. The data we received are row points of outer surface in each slice. All samples were properly positioned and oriented according to a predefined alignment reference [19]. Once the alignment of a 3D head is done, a ‘vector descriptor’ is established. A vector descriptor consists of a number of block distances. Here the term of block means a regular patch on the 3D surface. First the inscribed surface of all the samples is calculated. Then the inscribed surface and all sample surfaces can be divided into m blocks. Let P, Q denote the number of the control knots of a surface in u and v directions respectively, and p, q denote the desirable number of the control knots of a block in u and v directions respectively. The control knots of a surface were partitioned into P/p uniform intervals in u direction and Q/q uniform intervals in v direction. Thus the surface was converted into m = P/p * Q/q blocks. The distance between two corresponding blocks on a sample surface i and the inscribed surface, namely S(i), can then be constructed with two parts, namely S1(i) and S2(i), that reflect macro (size) and micro (shape) differences respectively. S1(i) can be calculated as, ni

S1 (i ) = ∑ dis ( pi , j ) , i=1, 2, 3,…, m

(1)

j =1

where pi,j is the jth point, and ni represents the number of points falling into the ith block, and Euclidean distance was used to calculate the distance, dis (pi,j), between two corresponding points on the sample and the inscribed surfaces. S2(i) can be calculated as, ni

S2 (i ) = ∑ dis ( pi , j ) − j =1

S1 (i ) , i=1, 2, 3, …, m ni

(2)

66


S2 describes the shape variation in the corresponding local areas of two 3D surfaces. Different local areas on a surface can have different shape and geometry characteristics; therefore they contribute differently in the whole shape dissimilarity between a sample surface and the inscribed surface. For example, the geometry of the nose is irregular, so the geometric dissimilarity may play an important role in the total shape dissimilarity between two noses. While the geometry of the upper head is very smooth and quite similar, so the size dissimilarity may play a dominant role in the total shape dissimilarity between two upper heads. By the above method, the shape of a surface can be characterized by a vector (S1(1), S2(1), S1(2), S2 (2),…, S1(m), S2(m)). The vectors are the input of the following k-means clustering algorithm. 2.2 Comparison between Different Block Division Numbers In the case study, each head surface was divided into 20 (5X4), 30 (6X5), and 90 (16X6) blocks, respectively. Then k-means clustering was applied to the block distance-based vectors with different block division number. In this case study, the number of K for the clustering was set as seven. An evaluation of the influence of the block division number on the clustering results is demonstrated by using One-way ANOVA on the block distance based vectors. The representative head sample of each cluster is obtained first by calculating the average coordinates of the points of the head samples belonging to the cluster. Then the distance between a sample surface and the representative surface can be constructed for S1' and

S2' , respectively. As a prerequisite step of ANOVA, the first examination is whether the S1' and S2' display a normal distribution. Tests for normality are conducted on all

S1' and S2' values using the One-Sample Kolmogorov-Smirnov Test. Another prerequisite step of ANOVA Levene test is to test the homogeneity-of-variance of the variables. Finally, multiple comparisons of means between different block divisions were conducted by using One-way ANOVA.

3 Results and Discussions 3.1 Block Division with Different Block Numbers The block division results of a head are shown through Fig. 1-3 with block numbers of 20 (5X4), 30 (6X5), and 90 (16X6), respectively. Each block is illustrated with different colors to distinguish from each other. No anatomical correspondence is taken into consideration during the block division. Consequently, as depicted in Fig. 1, it’s not surprising that the nose is divided into one block, while both eyes are divided into two different blocks. With the increase of the block division number, the area covered by each block decreases. That’s to say, more block division number means fewer points falling into each block.


(a) front view

67

(b) bottom view (c) side view

Fig. 1. Block division of a head (20 blocks)

(a) front view



(a) front view



3.2

Clustering Results under Different Block Division Numbers

As shown in Fig. 4, when representative surfaces of clusters are merged together, it is easy to acquire a visual image of the size and shape difference from each other.

68


Block number

Front view

Views Side view

Bottom view

20

30

90

Fig. 4. Different views of the merged average heads of clusters

3.3 Comparison of Results between Different Block Division Numbers Tests for normality of S1' and S2' values showed p values less than 0.05, resulting in rejection of the null hypothesis. Afterwards each S1' and S2' values were transformed into their corresponding natural logarithmic values, denoted as ln S1' and ln S2' respectively. One-Sample Kolmogorov-Smirnov test was conducted on the ln S1' and

ln S2' values, resulting in p values greater than 0.05 (p=0.393 and 0.193 respectively). Levene test on the ln S1' and ln S2' values showed p values of 0.570 and 0.492, respectively. As shown in Table 1, One-way ANOVA results demonstrated p values greater than 0.05. Such results lead to no rejection of the null hypothesis. Thus no significant


69

differences were found between the block division numbers. In other words, this reveals the robustness of the 3D clustering method with block division. Table 1. Multiple comparisons in One-way ANOVA

Dependent Variable

Group I

Group J

Mean Difference (I-J)

Std. Error

Sig.

1 1 2 1 1 2

2 3 3 2 3 3

-.0007 .0156 .0163 .0001 .0004 .0004

.01967 .01967 .01967 .02102 .02102 .02102

.971 .429 .408 .997 .984 .986

ln S1'

ln S2'

Note: Block division numbers of Group 1, 2, and 3 are 20, 30, and 90, respectively.

Cluster membership variation on different block division number of each sample was investigated and summarized in Table 2. When the block number is changed from 20 to 30 and 90, the numbers of samples whose cluster membership has changed are 10 and 86, respectively (sample size is 446). Whereas, the number of samples with changed membership between 30 and 90 blocks is 88. It can be seen that when the block division number changes within a medium range, the difference of clustering results is almost ignorable. However, when the block division number becomes big, such as 90 in this case study, the membership variation turns to big. This can be explained from the definition of S1 and S2 which reflect the local size and shape differences, respectively. When the surface is divided into many blocks, the effect of S2 is weakened, since for each small block the shape variation is small. Instead, when the block division number is small, the distance of each block is averaged over a big surface, thus the effect of S1 is weakened; while the shape variation of a big surface is greater, and the effect of S2 under this situation will be emphasized. Thus too small or great block division number will cause biased consideration of local size and shape. Table 2. Cluster membership change

Cluster ID 1 2 3 4 5 6 7 number of change

Sample size 20 blocks 40 71 66 77 63 23 106 -

30 blocks 39 72 62 78 60 25 110 10

90 blocks 41 81 76 75 68 27 78 86

70


4 Conclusions This paper is a further study of our previous 3D shape clustering method based on block division technique [19]. Clustering results of three alternatives of the block division number were compared. One-way ANOVA and cluster membership variation results showed the robustness of the block division method for the k-means 3D shape clustering when the block division number changes within a medium range. However, Extreme block division numbers may lead to greater membership variation.

Acknowledgements This study is supported by the National Natural Science Foundation of China (No.70571045).

References 1. Robinette, K.M., Blackwell, S., Daanen, H., Fleming, S., Boehmer, M., Brill, T., Hoeferlin, D., Burnsides, D.: CAESAR, Final Report, Volume I: Summary. AFRL-HE-WP-TR-2002-0169. United States Air Force Research Lab., Human Effectiveness Directorate, Crew System Interface Division, Dayton, Ohio (2002) 2. Bougourd, J., Treleaven, P., Allen, R.M.: The UK national sizing survey using 3d body scanning. In: Proceedings of Eurasia-Tex Conference in association with International Culture Festival, Donghua University, Shanghai, China (March 2004) 3. Isaacs, M.: 3D fit for the future. American Association of Textile Chemists and Colorists Review 5(12), 21–24 (2005) 4. WEAR, http://ovrt.nist.gov/projects/wear/ 5. Whitestone, J.J., Robinette, K.M.: Fitting to maximize performance of HMD systems. In: Melzer, J.E., Moffitt, K.W. (eds.) Head-Mounted Displays: Designing for the User, pp. 175–206. McGraw-Hill, New York (1997) 6. Elliott, M.G.: Methodology for the sizing and design of protective helmets using three-dimensional anthropometric data. Thesis (PhD). Colorado State University, Fort Collins, Colorado, USA (1998) 7. Meunier, P., Tack, D., Ricci, A., Bossi, L., Angel, H.: Helmet accommodation analysis using 3D laser scanning. Applied Ergonomics 31, 361–369 (2000) 8. Mochimaru, M., Kouchi, M.: Proper sizing of spectacle frames based on 3-D digital faces. In: Proceedings of 15th Triennial Congress of the International Ergonomics Association (CD ROM), Seoul, Korea, August 24-29 (2003) 9. Witana, C.P., Feng, J.J., Goonetilleke, R.S.: Dimensional differences for evaluating the quality of footwear fit. Ergonomics 47(12), 1301–1317 (2004) 10. Zhang, B., Molenbroek, J.F.M.: Representation of a human head with bi-cubic B-splines technique based on the laser scanning technique in 3D surface anthropometry. Applied Ergonomics 35, 459–465 (2004) 11. Witana, C.P., Xiong, S.P., Zhao, J.H., Goonetilleke, R.S.: Foot measurements from three-dimensional scans: A comparison and evaluation of different methods. International Journal of Industrial Ergonomics 36, 789–807 (2006) 12. Hsiao, H.W., Whitestone, J., Kau, T.Y.: Evaluation of Fall Arrest Harness Sizing Schemes. Human Factors 49(3), 447–464 (2007)


71

13. Lee, H.Y., Hong, K.H.: Optimal brassiere wire based on the 3D anthropometric measurements of under breast curve. Applied Ergonomics 38, 377–384 (2007) 14. Rogers, M.S., Barr, A.B., Kasemsontitum, B., Rempel, D.M.: A three-dimensional anthropometric solid model of the hand based on landmark measurements. Ergonomics 51(4), 511–526 (2008) 15. Gouvali, M.K., Boudolos, K.: Match between school furniture dimensions and children’s anthropometry. Applied Ergonomics 37, 765–773 (2006) 16. Li, Z.Z.: Anthropometric Topography. In: Karwowski, W. (ed.) The 2nd edition of the International Encyclopedia of Ergonomics and Human Factors, pp. 265–269. Taylor and Fancis, London (2006) 17. Goonetilleke, R.S., Luximon, A., Tsui, K.L.: The Quality of Footwear Fit: What we know, don’t know and should know. In: Proceedings of the Human Factors and Ergonomics Society Conference, San Diego, CA, vol. 2, pp. 515–518 (2000) 18. Goonetilleke, R.S., Luximon, A.: Designing for Comfort: A Footwear Application. In: Das, B., Karwowski, W., Mondelo, P., Mattila, M. (eds.) Proceedings of the Computer-Aided Ergonomics and Safety Conference (Plenary Session, CD-ROM), Maui, Hawaii, July 28-August 2 (2001) 19. Niu, J.W., Li, Z.Z., Salvendy, G.: Multi-resolution shape description and clustering of three-dimensional head data. Ergonomics (in press) 20. Chen, X., Shi, M.W., Zhou, H., Wang, X.T., Zhou, G.T.: The “standard head” for sizing military helmets based on computerized tomography and the headform sizing algorithm (in Chinese). Acta Armamentarii 23(4), 476–480 (2002)

Joint Coupling for Human Shoulder Complex Jingzhou (James) Yang1, Xuemei Feng2, Joo H. Kim3, Yujiang Xiang3, and Sudhakar Rajulu4 1

Department of Mechanical Engineering Texas Tech University, Lubbock, TX79409, USA 2 Wuhan University of Technology, Wuhan, Hubei, China 3 Center for Computer-Aided Design, University of Iowa, Iowa City, USA 4 NASA Johnson Space Center, Houston, TX77058, USA [email protected]

Abstract. In this paper, we present an inverse kinamtics method to determining human shoulder joint motion coupling relationship based on experimental data in the literature. The joint coupling relationship is available in the literature, but it is an Euler–angle-based relationship. This work focuses on transferring Eulerangle-based coupling equations into a relationship based on the DenavitHartenberg (DH) method. We use analytical inverse kinematics to achieve the transferring. Euler angles are obtained for static positions with intervals of 15 degrees, and the elevation angle of the arm varied between 0 and 120 degrees. For a specific posture, we can choose points on clavicle, scapula, and humerus and represent the end-effector positions based on Euler angles or DH method. For both systems, the end-effectors have the same Cartesian positions. Solving these equations related to end-effector positions yields DH joint angles for that posture. The new joint motion coupling relationship is obtained by polynomial and cosine fitting of the DH joint angles for all different postures. Keywords: Keywords: Human shoulder; joint motion coupling; joint limit coupling; shoulder rhythm; Euler angles; DH method.

1 Introduction Human shoulder complex consists of three bones—the clavicle, scapula, and humerus—and more than 20 muscles. The shoulder complex model is the key to correctly simulating human posture and motion. So far, various kinds of kinematics shoulder models are available and are based on various methods. Among those methods, the Denavit-Hartenberg (DH) method is an effective way to control the digital human movement in the virtual simulation field [18]. In the literature, two categories can be found: open-loop chain systems and closed-loop chain systems [3]. There are different types of models within each category. Also, we proposed a closed-loop chain model [4] for the shoulder complex. This model is high–fidelity, and the digital human system operates in real-time. To correctly model the movement of the human shoulder complex, a high-fidelity kinematic model is not enough; a phenomenon called shoulder rhythm should be considered. Shoulder rhythm includes joint motion coupling and joint limit coupling. V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 72–81, 2009. © Springer-Verlag Berlin Heidelberg 2009

Joint Coupling for Human Shoulder Complex

73

Joint limit coupling has been investigated by Lenarcic and Umek [13], Klopcar and Lenarcic [10, 11], Klopcar et al., [12], and Lenarcic and Klopcar [14]. Joint motion coupling was obtained using an experiment [8]; however, this relationship is Eulerbased, and we cannot use it in the DH-based digital human environment. In previous work [4], we proposed one method for transferring Euler-based coupling equations into DH-based relationships based on a shoulder model in Virtools®. That method is tedious and depends on the model in Virtools. This paper presents an analytical inverse kinematics based method for mapping between the two systems. We first summarize the new proposed shoulder complex model. Next, we briefly discuss the joint coupling equations depicted in Euler angle. For an end-effector position, we can define it as a function of Euler angles or DH joint angles. Solving this analytical inverse kinematic problem, we can obtain a set of DH joint angles. Repeating this procedure for all end-effector positions yields different sets of data. Then these separate data are fitted into a set of functional equations accordingly. Finally, we plot the coupling equations based on polynomial and cosine fitting and compare the different fitting results.

2 Shoulder Kinematic Model In previous section, we summarized that there are open-loop chain and closed-chain systems. Within the first one, there are five different models (the 5-DOF models I and II, the 6-DOF model, the 7-DOF model, and the 9-DOF model). For a closed-loop chain, several models are also available [3]. We propose a new shoulder model that has 8 DOFs: (1) two revolute joints ( q1 , q 2 ) in the sternoclavicular (SC) joint, denoting clavicle vertical rotation and horizontal rotation; (2) three revolute joints ( q3 ,

q4 , q5 ) in the acromioclavicular (AC) joint, denoting rotations in three orthogonal directions of the scapula movement with respect to the local frame; (3) three revolute joints ( q6 , q7 , q8 ) in the glenohumeral (GH) joint, denoting the movement of the humerus with respect to the local frame.

3 Euler-Angle-Based Joint Coupling Equations In the past two decades, much research has been done on shoulder complex motion because shoulder pain constitutes a large portion of musculoskeletal disorders in the general population as well as among industrial workers. Shoulder rhythm is one important characteristic to study. Different approaches have been used for this study [1, 2, 6, 7, 8, 16, 17 and 19]. Among these, Hogfors et al. [8] obtained ideal shoulder rhythm solutions. In the study by Hogfors et al. [8], three healthy, right-handed male volunteers (mean age, 24 yr; mean body mass, 70 kg; mean stature 183 cm) were used. They also used numerical evaluation of low roentgen stereophotogrammetric motion pictures of subjects with radiation dense implantations in the bones. Interpolation between measured positions makes it possible to simulate shoulder motions within the normal working range. In the experiment, the orientation angles of the

74

J. Yang et al.

Fig. 1. Shoulder kinematic model

Fig. 2. The elevation angle experiment

θ

used in the

scapula bone and the clavicle bone were measured when the arm was elevated at the angle θ in the scapular plane, which has 45-degree angle with respect to the coronal plane in Figure 2, where θ varied between 0 and 120 degrees. Three bones’ (clavicle, scapula, and humerus) body-fixed coordinate frames are used to define the orientations of the bones in Figure 3. One global coordinate system is attached on the sternum. The global frame ( x, y, z ) and the clavicle local frame

( x1 , x2 , x3 ) have the same origin is located at point

Ω . The origin of the scapula local frame (ξ1 , ξ 2 , ξ3 )

Ω . The origin of the humerus frame (κ1 , κ 2 , κ 3 ) is at point Ω h . s

Fig. 3. Coordinate systems for shoulder bones [8]

The Euler angle system α , −β , and γ shown in Figure 4 was used to depict the movement of the shoulder bones. The transformation matrix is defined by REul = RX (γ ) RY (− β ) RZ (α ) , where ⎛ cos(α ) − sin(α ) 0 ⎞ ⎜ ⎟ Rz (α ) = ⎜ sin(α ) cos(α ) 0 ⎟ , ⎜ 0 0 1 ⎟⎠ ⎝

0 ⎛1 ⎛ cos( β ) 0 − sin( β ) ⎞ ⎜ ⎟ R (γ ) = ⎜ 0 cos(γ ) Ry (− β ) = ⎜ 0 1 0 ⎟ x ⎜ ⎜ 0 sin(γ ) ⎜ sin( β ) 0 cos( β ) ⎟ ⎝ ⎝ ⎠

0 ⎞ ⎟ − sin(γ ) ⎟ , cos(γ ) ⎟⎠

while the XYZ frame can be any one of the body-fixed frames for the three bones.


75

The interpolation results from the experimental data are shown in Eqs. 1-3. The indices h, c, and s for α , −β , and γ stand for humerus, clavicula, and scapula, respectively. In these equations, αh and βh are independent variables.

αh

varies from -100

to 900, and β h = −900 + θ varies within 90-300. The angle γ h in the equations refers to the neutral rotation angle of the upper arm [9]. Humerus: (1) γ h = −45 + α h [1 − ( βh + 90) / 360] + (135 − α h /1.1)sin(0.5( βh + 90)(1 + α h / 90)) ⎧α c = −50 + 30cos[0.75( β h + 90)] Clavicle: ⎪⎨ β c = 24{1 − cos[0.75( β h + 90)]}(0.5 + α h / 90) + 9 (2) ⎪γ = 15{1 − cos[0.75( β + 90)]} + 3 ⎩ c h ⎧α s = 200 + 20 cos[0.75( β h + 90)]

Scapula: ⎪⎨ β s = −140 + 94 cos[0.75( β h + 90)(1 − γ h / 270)]

(3)

⎪γ = 82 + 8cos{(α + 10)sin[0.75( β + 90)]} h h ⎩ s

Fig. 4. Euler angles

4 Transferring Joint Coupling Equations from Euler System to DH System In the above section, we review the coupling equations obtained by experiments in Hogfors et al. [8]. However, it is difficult to use these results directly in digital human models using DH representation instead of Euler angles. This section presents the methodology of transferring these equations from the Euler to the DH system, data generation, data fitting, and discussion about the results from different fitting techniques.

5 Methodology for Transferring the Joint Coupling Equations The principle of transferring coupling equations from the Euler to the DH system is that the same posture can be represented by different orientation representation systems, i.e., we can represent the same posture with Euler’s angles, DH joint angles, Euler parameters, etc. The procedure for transferring the coupling equations shown in Figure 5 entails data generation and equation fitting. Within data generation, there are three steps: (1) selecting key postures based on the Euler system; (3) choosing points on clavicle and humerus AC and GH joint as the end-effectors and form equations

76

J. Yang et al. Data Generation Select key postures based on Euler angle system

Choose points on Clavicle and Humerus as end-effectors and form equations

Solve equations to obtain DH joint angles

Equations Determination Fit the joint angles into functional equations

Fig. 5. Methodology for transferring joint coupling equations

(left hand side is the end-effector by Euler system, and right hand side is by DH system); and (3) solving these equations to obtain DH joint angles. 5.1 Data Generation The global coordinate system is the same for both Euler and DH system. However, DH local frames are different from Euler’s frames. Figs. 6 and 7 show the postures corresponding to zero Euler angles and DH angles, respectively.

Fig. 6. Zero Euler angles

Fig. 7. Zero DH joint angles

In this section, we use one example to illustrate the detailed procedure to determine DH joint angles by analytical inverse kinematics method. When we choose

α h = 45o

and

β h = −45o , then, from Eqs. 1-3, all Euler angles are calculated and

shown in Table 1. The posture is shown in Figure 8. Table 1. Euler angles (in degrees)

αh

βh

4 5

45

γh

αc

46.6 491

25.0559

βc 13.0 447

γc 5.52 796

αs 216. 629

βs 56.9405

γs 88. 889

Considering AC joint center for both systems, one has the following equations:


77

0 0 Fig. 8. The shoulder posture when α h = 45 , β h = −45

⎡ ⎛ L2 ⎞⎤ ⎡0⎤ ⎢ ⎜ ⎟⎥ ⎢ ⎥ ⎢ R (α c , β c , γ c ) ⋅ ⎜ 0 ⎟ ⎥ = T 0 (q ) ⋅ T 1 ( q ) ⋅ ⎢ 0 ⎥ ⎜ 0 ⎟⎥ 1 1 2 2 ⎢0⎥ ⎢ ⎝ ⎠ ⎢ ⎥ ⎢ ⎥ 1 ⎣1 ⎦ ⎣⎢ ⎦⎥

(4)

Bringing in all necessary terms in Eq. (4) yields

sin(q2 ) = cos( β c ) sin(α c )

(5)

sin( β c ) = − cos(q2 )sin(q1 )

(6)

⎧ q1 = 0.2504 ⎧ q1 = −0.2504 ⎨ Solving Eqs. (5) and (6), one obtains ⎩q2 = 2.7163 and ⎨ ⎩q2 = −0.4253 (radians). One can bring in these solutions into Figure 8 to select the correct solution

⎧ q1 = −0.2504 ⎨ ⎩q2 = −0.4253 . Choosing GH joint center for both systems, one has ⎡ ⎤ ⎛ L2 ⎞ ⎡0⎤ ⎢ ⎜ ⎟ Local ⎥ ⎢ ⎥ ⎢ R(α c , β c , γ c ) ⋅ ⎜ 0 ⎟ + R (α s , β s , γ s ) ⋅ VGH ⎥ = T 0 (q ) ⋅ T 1 (q ) ⋅ T 2 (q ) ⋅ T 3 (q ) ⋅ T 4 (q ) ⋅ ⎢ 0 ⎥ 1 1 2 2 3 3 4 4 5 5 ⎜ 0 ⎟ ⎢ ⎥ ⎢ L4 ⎥ ⎝ ⎠ ⎢ ⎥ ⎢ ⎥ 1 ⎢⎣ ⎥⎦ ⎣1⎦

(7)

Local where VGH = [ 0.9463 −0.9463 0.4145] is the GH joint center position corresT

ponding to the scapula fixed frame. Solving the first two equations in Eq.(7), one q3 = 0.2394 and q4 = −2.9030 , obtains the following possible solutions: q3 = −2.9234 and q4 = −2.6982 , q3 = 0.2176 and q4 = −0.4434 , or

q3 = −2.9022 and q4 = −0.2386 . This is a redundant problem and we bring all possible solutions in Fig. 8 and the correct solution is q3 = 0.2176 and q4 = −0.4434 . Solving the third equation in Eq.(7) yields q5 = 0.2895 .

78

J. Yang et al.

Similarly we choose a point on the humerus and have the following equations: ⎡ ⎤ ⎛ L2 ⎞ ⎡ −1⎤ ⎢ ⎜ ⎟ Local Local ⎥ ⎢0⎥ R ( , , ) 0 R ( , , ) V R ( , , ) V α β γ α β γ α β γ ⋅ + ⋅ + ⋅ c c c s s s GH h h h HUM ⎥ 0 ⎢ ⎜ ⎟ ⎢ ⎥ T ( q , q , q , q , q , q , q , q ) = ⋅ 8 1 2 3 4 5 6 7 8 ⎜ ⎟ ⎢ ⎥ ⎢1⎥ ⎝ 0 ⎠ ⎢ ⎥ ⎢ ⎥ 1 ⎣1⎦ ⎣⎢ ⎦⎥

(8)

Solve these equations and check the postures in Fig. 8. The final correct solutions q = 0.8612 q7 = 0.6735 q = 0.2528 are 6 , , and 8 . Transferring all radians to degrees, one gets the DH joint angles in Table 2.

Table 2. DH joint angles for

q1

q2

14.3469

q3

24.3679

12.4 676

α h = 450 , β h = −450 (in degrees)

q4 25.4049

q5

q6

q7

q8

16.5 871

49.3 427

38.5 889

14.4 844

Similarly, we can find all DH joint angles for postures with respect to 0

0

10 to 90 and

βh

αh

within -

0

in -90-30 .

5.2 Data Fitting In the above data generation, we find that the joint angle

q8 does not affect the posi-

tions and orientation of the clavicle and scapula bones. Therefore, the joint angle independent. Joint angles

q8 is

q1 to q5 are functions of q6 and q7 . Based on data from

the above section, we can use functional fitting to build up the coupling equations. There are different functions that can be fitted based on the same set of data. In this study, we choose polynomial, cosine series, and sigmoid functions as the coupling equations. Then we compare these functions to summarize the pros and cons. We use Mathematica® to obtain the fitting functions. The final fitting functions are denoted as follows; the unit is radians. i. Polynomial functions q1 = -0.056496q73 - 0.020185q63 - 0.197370q7 2 q6 + 0.188814q7 q6 2 + 0.094692q7 2 0.265677q6 2 - 0.025195q7 q6 - 0.090610q7 +0.587757q6 - 0.481600 q2 = -0.203064q73 - 0.008258q63 + 0.160842q7 2 q6 + 0.085135q7 q6 2 + 0.312579q7 2 0.110429q6 2 - 0.722626q7 q6 +0.179235q7 + 0.753456q6 - 0.877683

q3 = -0.010575q73 + 0.001193q63 + 0.269967q7 2 q6 - 0.005200q7 q6 2 - 0.142453q7 2 + 0.016321q6 2 - 0.707116q7 q6 +0.613453q7 + 0.440594q6 - 0.228151


79

q4 = 0.015404q73 + 0.008550q63 - 0.308488q7 2 q6 - 0.099259q7 q6 2 + 0.375770q7 2 + 0.134428q6 2 + 1.122621q7 q6 -1.082992q7 -1.032156q6 + 0.400115 q5 = 0.019663q7 3 - 0.004599q63 - 0.047076q7 2 q6 + 0.061801q7 q6 2 +0.021243q7 2 0.086331q6 2 + 0.095226q7 q6 - 0.080383q7 -0.006919q6 + 0.328476

(9)

ii. Fourier series functions q1 = - 0.638284 + 0.095253cos(q7 ) - 0.076156 cos(q6 ) + 0.002803sin(q7 ) - 0.746368sin(q6 ) 0.101693cos(2q7 ) + 0.024957 cos(2q6 ) - 0.168160sin(2q7 ) + 0.017257 sin(2q6 ) + 0.287427 cos( q7 ) cos(q6 ) + 1.085183cos( q7 ) sin(q6 ) + 0.045037 sin(q7 ) cos(q6 ) + 0.730106sin(q7 )sin(q6 ) q2 = -1.740112 + 0.010163cos(q7 ) + 0.998839cos(q6 ) + 1.212096sin(q7 ) + 0.955282sin(q6 ) + 0.050952 cos(2q7 ) - 0.005597 cos(2q6 ) + 0.126617sin(2q7 ) + 0.015509sin(2q6 ) 0.332707 cos(q7 ) cos(q6 ) + 0.125642cos(q7 ) sin(q6 ) - 0.973988sin(q7 ) cos(q6 ) 0.949188sin(q7 )sin(q6 ) q3 = -0.568506 - 0.005966 cos(q7 ) + 0.314923cos( q6 ) + 1.087587 sin( q7 ) + 0.896371sin( q6 ) + 0.161975cos(2q7 ) + 0.004556 cos(2q6 ) + 0.010620sin(2q7 ) + 0.002634sin(2q6 ) 0.211634 cos(q7 ) cos(q6 ) - 0.279816 cos(q7 )sin(q6 ) - 0.319026sin(q7 ) cos(q6 ) 0.889261sin(q7 )sin(q6 ) q4 = -0.029221 + 0.659483cos(q7 ) - 0.087511cos(q6 ) - 0.453476sin( q7 ) - 0.870285sin(q6 ) 0.109823cos(2q7 ) - 0.005751cos(2q6 ) - 0.270092sin(2q7 ) + 0.001511sin(2q6 ) 0.026523cos(q7 ) cos( q6 ) - 0.190386 cos(q7 ) sin(q6 ) + 0.103925sin( q7 ) cos(q6 ) + 0.851738sin(q7 ) sin( q6 ) q5 = 0.012476 + 0.328949 cos(q7 ) - 0.198714 cos( q6 ) + 0.335480sin(q7 ) - 0.310143sin(q6 ) 0.012544 cos(2q7 ) + 0.019457 cos(2q6 ) - 0.269965sin(2q7 ) + 0.004378sin(2q6 ) + 0.200568cos(q7 ) cos( q6 ) + 0.193417 cos(q7 )sin(q6 ) + 0.158652sin(q7 ) cos(q6 ) + 0.314105sin( q7 ) sin( q6 )

(10)

5.3 Discussion By means of data fitting (or regression), the joint coupling equations in the DH system have been obtained in the above section. However, different fitting functions have their own characteristics, which are summarized in this section. During the data regression process, two criteria were used. They are coefficient of determination R2, and maximal absolute residuals. The regression criteria values are presented in Tables 3 and 4. According to statistics theory, a larger R2 means a better model. Most R2 in Table 3 are larger than 0.90. The maximal absolute residuals are different for different joints. Smoothness is another factor to consider when choosing regression functions because human shoulder joints should not have any jerk during motion. From Eqs. 1-3, all functions are smooth. Joint limits calculated from Eqs. 1-3 are the last factor to be considered for regression functions. Bringing joint limits for q6 and q7 into these equations yields a

80

J. Yang et al.

possible range of motions for the joints

q1 to q5 . These values should be finite and

within the given range of motions for each joint. Based on all the factors mentioned above, the Fourier series functions are the best choice for the final coupling equations in the DH system. Table 3. Coefficient of Determination R2

Joint angle q1

Polyno. function 0.964169

Trigon. function 0.985975

Fourier function 0.997812

q2

0.964169

0.985975

0.997812

q3

0.993352

0.996666

0.998945

q4

0.991618

0.996725

0.998068

q5

0.900623

0.911096

0.94599

Table 4. Maximum of absolute residuals in regression equations

Joint angle

Polyno. function

Trigo. function

Fourier function

q1

0.239784

0.0791158

0.0724016

q2

0.187342

0.109164

0.0640427

q3

0.064795

0.0363415

0.0307189

q4

0.141521

0.0529489

0.0413061

q5

0.063625

0.0606651

0.0733895

6 Conclusions This paper presents an analytical inverse kinematics method for transferring coupling equations from the Euler system to the DH system. This method is based on the principle that one posture can be depicted by different rotation representation systems. Key postures from the Euler system were used to obtain the DH joint angles. A shoulder kinematic model was set up in Virtools to eliminate wrong postures, a data fitting technique was implemented, and several types of regression functions were constructed and compared. Fourier series functions are the ideal solutions for these coupling equations based on regression criteria, smoothness, and calculated joint ranges of motion. The original coupling equations in the Euler system were from experiments, and the hypothesis is that they do not depend on anthropometry. That means these equations generally represent the shoulder rhythm for humans from all percentiles. However, when we transferred these equations in the Euler system to equations in the DH system, specific link lengths were used. If a different set of link lengths were used, then we would get a different set of coupling equations. They would not be significantly different, however, because the shoulder rhythm is similar for humans of all


81

percentiles [8]. Therefore, these transferred coupling equations in DH system can be approximately used for a human from any percentile.

References 1. Bao, H., Willems, P.Y.: On the kinematic modeling and the parameter estimation of the human shoulder. Journal of Biomechanics 32, 943–950 (1999) 2. Berthonnaud, E., Herzberg, G., Zhao, K.D., An, K.N., Dimnet, J.: Three-dimensional in vivo displacements of the shoulder complex from biplanar radiography. Surg. Radiol. Anat. 27, 214–222 (2005) 3. Feng, X., Yang, J., Abdel-Malek, K.: Survey of Biomechanical Models for Human Shoulder Complex. In: Proceedings of SAE Digital Human Modeling for Design and, Pittsburgh, PA, June 14-16, 2008 (2008a) 4. Feng, X., Yang, J., Abdel-Malek, K.: On the Determination of Joint Coupling for Human Shoulder Complex. In: Proceedings of SAE Digital Human Modeling for Design and, Pittsburgh, PA, June 14-16, 2008 (2008b) 5. de Groot, J.H., Valstar, E.R., Arwert, H.J.: Velocity effects on the scapula-humeral rhythm. Clinical Biomechanics 13, 593–602 (1998) 6. de Groot, J.H., Brand, R.: A three-dimensional regression model of the shoulder rhythm. Clinical Biomechanics 16(9), 735–743 (2001) 7. Herda, L., Urtasun, R., Fua, P., Hanson, A.: Automatic determination of shoulder joint limits using quaternion field boundaries. International Journal of Robotics Research 22(6), 419–436 (2003) 8. Hogfors, C., Peterson, B., Sigholm, G., Herberts, P.: Biomechanical model of the human shoulder joint-II. The shoulder rhythm. J. Biomechanics. 24(8), 699–709 (1991) 9. Karlsson, D., Peterson, B.: Towards a model for force predictions in the human shoulder. J. Biomechanics. 25(2), 189–199 (1992) 10. Klopcar, N., Lenarcic, J.: Kinematic model for determination of human arm reachable workspace. Meccanica 40, 203–219 (2005) 11. Klopcar, N., Lenarcic, J.: Bilateral and unilateral shoulder girdle kinematics during humeral elevation. Clinical Biomechanics 21, S20–S26 (2006) 12. Klopcar, N., Tomsic, M., Lenarcic, J.: A kinematic model of the shoulder complex to evaluate the arm-reachable workspace. Journal of Biomechanics 40, 86–91 (2007) 13. Lenarcic, J., Umek, A.: Simple model of human arm reachable workspace. IEEE Transaction on Systems, Man, and Cybernetics 6, 1239–1246 (1994) 14. Lenarcic, J., Klopcar, N.: Positional kinematics of humanoid arms. Robotica 24, 105–112 (2006) 15. Maurel, W.: 3D modeling of the human upper limb including the biomechanics of joints, muscles and soft tissues. PhD Thesis, Lausanne, EPFL (1995) 16. Moeslund, T.B.: Modeling the human arm. Technical report at Laboratory of Computer Vision and Media Technology, Aalborg University, Denmark (2002) 17. Rundquist, P.J., Anderson, D.D., Guanche, C.A., Ludewig, P.M.: Shoulder kinematics in subjects with frozen shoulder. Arch. Phys Med. Rehabil. 84, 1473–1479 (2003) 18. Sciavicco, L., Siciliano, B.: Modeling and control of robot manipulators. The McGrawHill Companies, Inc., New York (1996) 19. Six Dijkstra, W.M.C., Veeger, H.E.J., van der Woude, L.H.V.: Scapular resting orientation and scapula-humeral rhythm in paraplegic and able-bodied male. In: Proceedings of the First Conference of the ISG, pp. 47–51 (1997)

Development of a Kinematic Hand Model for Study and Design of Hose Installation Thomas J. Armstrong1, Christopher Best1, Sungchan Bae1, Jaewon Choi1, D. Christian Grieshaber2, Daewoo Park1, Charles Woolley1, and Wei Zhou1 1

Center for Ergonomics, University of Michigan, Ann Arbor, MI 48109 Department of Health Sciences, Illinois State University, Normal, IL

2

Abstract. Kinematic hand models can be used to predict where workers will place their fingers on work objects and the space required by the hand. Hand postures can be used to predict hand strength. Kinematic models also can be used to predict tissue stresses and to study work-related musculoskeletal disorders. Study and design of manual hose installation is an important application for kinematic hand models. Hoses are widely used in many mechanical systems such as autos, aircraft and home appliance, which are all mass-produced on assembly lines. Studies of automobile assembly jobs show that hose installations are one of the most physically demanding jobs that workers perform. Hoses are a good starting point for kinematic model development because they can be characterized as simple cylinders. Keywords: Hands, kinematic model, manufacturing.

1 Introduction Manual work continues to be a vital part of our industrial economy. People have many advantages over machines: they are able to compensate for subtle material and process variations; they can quickly learn to perform different jobs in an agile production process; and they don’t require huge upfront capital investments. However, people, like machines, have operating limits and constraints. Job demands must not exceed their strength capacity and sufficient space must be provided to reach for and grasp work objects. Production hose installation is an example of a job that is routinely performed by hand. The external size and shape of hoses often varies slightly from one hose to another. Hoses are often jointed to a flange in confined and obstructed workspaces. Studies by Ebersole and Armstrong [1 and 2] showed that manual hose installation is one of the most demanding auto assembly jobs that workers perform. Static anthropometric data, such as hand length, width and thickness cannot be applied directly to determine if there is sufficient room for the hand. Static anthropometric data, however, can be used with kinematic models to predict possible grip postures and how much space will be occupied by the hand in those postures. Additionally posture can be used to be used to predict hand strength [3]. Kinematic models also can be used to estimate tendon excursions and loads associated with reaching and grasping and to study risk of musculoskeletal disorders in the wrist [4]. V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 85–94, 2009. © Springer-Verlag Berlin Heidelberg 2009

86

T.J. Armstrong et al.

This paper describes the development of a kinematic model for studying and designing manual hose installation tasks. Although the main focus of this work was on hoses, the resulting model has potential applications to tasks that involve gripping of other parts and tools.

2 Methodology 2.1 The Link System The link system used for development of this model was based on that developed by Buchholz, et al. [5]. They studied the relationship between segment lengths and hand lengths. Planner radiographs were obtained for series of joint angles from straight to fully flexed. Reuleaux’s method was used to determine the joint centers. Segment lengths were then computed as the distances between successive joint centers. Segment lengths were found to be highly correlated with hand lengths. Figure 1 shows segments and the coefficients used for computing their length.

Segment 1 2 3 4

1 0.118 0.251 0.196 0.158

2 0.463 0.245 0.143 0.097

Digit 3 0.446 0.266 0.170 0.108

4 0.421 0.244 0.165 0.107

5 0.414 0.204 0.117 0.093

Fig. 1. Relative link lengths from Buchholz, Armstrong, Goldstein [5]

2.2 Hand and Object Surfaces Buchholz and Armstrong [6] in 1992 proposed a series of ellipsoids that were scaled on segment lengths, widths and thicknesses to give the model hand shape. Use of geometric objects made it possible to detect contact among hand segments and external objects in a virtual environment. Model manipulations were quite slow on

Development of a Kinematic Hand Model for Study and Design of Hose Installation

87

Fig. 2. Choi and Armstrong (7) used arrays of points based on hand segment sizes and truncated cones to depict hand surfaces (left). The surfaces filled in using Open GL (right).

computers at that time. Recently Choi and Armstrong [7] utilized arrays of equally spaced points based on segment sizes and truncated cones to depict hand surfaces (see Fig. 2). 2.3 The Graphical User Interface and Manipulation of Model A graphical user interface was designed to facilitate manipulation of the model (see Fig. 3). The program will compute segment sizes for a given percentile hand length using the Buchholz coefficients or the user can specify desired hand sizes. The program also provides a selection of standard object shapes that can be scaled to desired sizes in each of the three dimensions. The user can also place the object at desired locations and orientations. For example, a hose would be represented as a cylinder and could be placed at a right angle to the hand, parallel to the hand, or something in between. The user also has the option of entering other objects as arrays of surface points. The joint angles can be manipulated manually by entering angles. Positioning the fingers on a work surface is a tedious process and the results will vary from one user to another. Still, it is possible to get an idea of how well an object fits the hand, where the fingers might touch the work object and what kinds of postures are possible. It is helpful if the user has some familiarity with what the worker is trying to do. Grieshaber and Armstrong [3] studied the postures that workers used to install hoses in 113 hose installation tasks in twenty-eight auto assembly tasks. They found that workers are more likely to use a power grip than a pinch grip posture as the ratio of hose diameter to hand length increased and the hose insertion forces increased (see Fig. 4). Some workers still use a power grip posture for small hoses and low forces and some use a pinch grip for large hoses and high forces. Other investigators have studied how people grasp objects with the goal of developing a set of primitives for robots [8]. These primitives also provide a guidance manual for manipulation of

88


Fig. 3. Graphical user interface used by Choi and Armstrong [7] to manipulate Kinematic Hand Model

Fig. 4. Hand posture versus hose OD/ hand length ratio (left) and finger flexor activity (normalized 0-10) (right)

kinematic models. Grip postures are no doubt affected by other factors, such as access to the flange and the worker’s behavior. Although kinematic models do not force objects, that does not mean that the users of those models cannot. It must be possible for the hand to achieve a static equilibrium with the grip objects, that is:

r

∑F

i

= 0 and

r

∑M

i

= 0.

(1)

This means that if the fingers press on one side of an object and that object is not constrained externally, e.g., work surface railing, etc., then that object must be supported on the opposite side by the thumb or the palm. Skin deformation and friction will help keep objects from slipping out of the hand if the fingers are not exactly aligned on opposite sides of the grip object.


89

2.4 Posture Prediction Algorithms Some of the work required to manually position the fingers on the surface of a work object can be reduced through the use of posture prediction algorithms. These algorithms either flex the fingers until contact is detected between segments of the fingers and the grip object, or alternatively, they use optimization methods to calculate the best fit between the hand and work object. Buchholz [6] adapted an algorithm from Fleck and Butler [9] to detect contact between ellipsoid representation of the finger segments and the geometric representation of the grip object. The user first specified the geometry of the grip object and its location and orientation with respect to the hand. The posture prediction routine then started with the fist knuckle and rotated that joint until contact occurred between the hand and the work object. The process was then repeated for the second and then the third knuckles. Buchholz reported very good agreement between predicted and observed postures for gripping different sized cylinders perpendicular to the long axis of the hand using a power grip. The need to represent grip objects mathematically, the lack of a graphical user interface, and the slow processors at that time restricted the use of the resulting model. Choi and Armstrong [7] utilized a contact algorithm that computed distances between points representing surfaces of the hand and surfaces of the work object. Although this is computationally intensive, it is within the capacity of most modern desktop computers. A number of studies were performed to evaluate the sensitivity of posture to hand size, skin deformation and cylinder diameter. The contact algorithm made it possible to simulate skin deformation by allowing penetration of the object into the hand (negative object hand distances). Model predictions explained 72% of the observed hand posture variance (R2) for gripping cylinders with diameters between 26 and 114.3 mm. Prediction errors ranged from -16.4º to 18.7º. The model tended to overestimate the third knuckle angles for cylinder sizes (-16.4º ~ -0.4º) and to underestimate first knuckle angles for all cylinder sizes (-2.4º ~ 18.7º). Cylinder size had the most profound effect on finger joint angles. Hand length and width (from small female to large male percentiles) and skin deformation (up to 20% penetration) had only a small effect on joint angle predictions. Subsequent studies examined how predicted joint angles are affected by where the object is placed in the hand and how it is oriented [10]. Hand placement is important especially when posture prediction algorithms are used. If the object is place too close to the wrist, it is possible for the fingers to completely miss the object as the fist closes. If it is placed too close to the fingertips, the hand may not close much at all before contact occurs. Posture predictions were generally pretty consistent as long as they were between the middle of the palm and the first knuckle. Studies of grip behavior, [3; 8; 11; 12], can be used to guide finger placement and determine if the resulting grip postures are feasible. Lee and Zhang [13] proposed an optimization model based on the assumption that the best prehensile configuration of the hand in power grip optimally conforms to the shape of the grip object. Their model simultaneously minimized the distances between joint centers and the surface of the grip object. Their model was tested by comparing predicted and observed postures of twenty subjects gripping vertically oriented cylindrical handles 45 and 50 mm in diameter. Average root mean prediction errors

90


across all conditions were less than 14 degrees. This optimization routine can be extended to other grip objects, but reformulation of the model would be required if the hand is not in continuous contact with the grip object. The advantage of this model is that it does not require iterations to find the final posture. The disadvantage is that it may have to be reprogrammed for application to grip objects with other shapes or other hand postures. Also, it does not allow the user to easily explore subtle variations in object placement and orientation or finger placements. 2.5 Finger Movements Posture predictions are affected by rotation rates of the finger joints. Figure 5 shows the finger tip trajectories based on rotating one joint at a time versus rotating them together at the same rate. It can be seen that rotating them together reduces the reach area of the fingertip. As a practical matter, people don’t close their fist one joint at a time. Neither do they close them at the same rate. Kamper, et al. [14] studied finger motions for 10 subjects grasping different objects. They observed that the average rate of rotation for the second knuckle was only 26 to 72% that of the first knuckle and that the rate for the third knuckle was only 16 to 36% that of the second knuckle. There were significant variations among fingers and subjects. The actual finger trajectory will probably be somewhere between the two extremes shown in Fig. 5.

Fig. 5. Fingertip trajectories based on rotating joint 1 to its maximum, then joint 2, then joint 3 (solid lines) and based on joints 1, 2 and 3 together at the same rate

One of the challenges to developing accurate finger motions models is that the finger motions are usually combined with movement of the hand towards the work object and starts with opening the hand so that the grip object can pass between the thumb and the fingers [15; 16]. Figure 6 shows hand trajectory and wrist and index finger angles for an average of six subjects reaching for a vertical cylinder. Starting from a relaxed posture (25° Extension and 45°, 30° and 10° Flexion) the wrist extends and the hand opens (Δθ = +8°, -22°, -12° and -5°) before closing (Δθ = +7°, +11°, +10° and +10°). It also can be seen that the trajectory of the wrist is curved. It has been shown that how much the hand opens and how much it closes are related to the


91

Fig. 6. Wrist trajectory (left) and the first (1), second (2) and third (3) knuckle angles of the index finger and wrist angle (right) for a 40 cm reach to grasp a 15 cm diameter cylinder

size of the object [15; 17; 18; 16; 7]. Models are needed that describe the finger motions as functions of time so that they can be used with kinematic models. 2.6 Required Hand Space 3-D kinematic hand models can be used to predict the hand space requirement for hose placement tasks (see Fig. 7). Hand space requirements were simulated using a 3D kinematic hand model described by Choi, et al. [19] and compared with experimental data reported by Grieshaber and Armstrong [20]. The simulation results showed good agreement with measured data with an average 17% underestimation of hand space envelopes. Simulations showed that pinch grip required an average of 72% larger space than power grip, the rotation method required an average of 26% larger space than the straight method, and a 95% male hand size required an average of 44% larger space than 5% female hand length. The hand space envelope can give useful information to designers and engineers who design workspace and parts to avoid problems of obstruction. Future work will include the addition of modules to the kinematic model interface for capturing hand space data and validating space predictions for a range of different size and shape grip objects. 2.7 Work-Related Musculoskeletal Disorders Another important use of kinematic models is evaluating risk of work related musculoskeletal disorders. Choi and Armstrong [21] conducted a study to examine the relationship between tendon excursion and wrist movements and MSDs (musculoskeletal disorders) of the hand and wrist. Video tapes were obtained from a previous study by Latko, et al. [22] that showed a strong basis between Hand Activity Level and risk of non-specific hand pain, tendonitis and carpal tunnel syndrome. One medium-risk job and two low-risk jobs were selected from an office furniture manufacturing facility. Two high-risk jobs, one medium-risk job, and one low-risk job were selected from a manufacturing site for industrial containers. Two high-risk jobs and one medium-risk job were chosen from a company manufacturing spark plugs. Time-based analyses were performed for the right hand and the wrist as described by Armstrong, et al. [23].

92


Fig. 7. Space occupied by the hand while inserting a hose using predicted using moments

Tendon excursions of FDP (flexor digitorum profundus) and FDS (flexor digitorum superficialis), projected for one hour, were assessed by using the models developed by Armstrong and Chaffin [24]. Cumulative tendon excursions were computed from angular velocities and peak wrist excursions. First, wrist posture as a function of time, θ(t), can be written as θ(t)=∑ θ 0isin(ωit+Φ) ,

(2)

where θ 0i is peak wrist excursion, ωi is the frequency, Φ is the phase, and t is time. Second, angular velocity, θ’(t), can be calculated as θ’(t)=∑ θ 0iωicos(ωit+Φ) .

(3)

Third, the cumulative tendon excursion is;

∫

T

0

•

r θ (t ) dt = ∫

T

0

∑ rθ

oi

ω i cos(ω i t + φ ) dt

(4)

Where r is the radius of tendon curvature in the wrist, θ’ is the angular velocity of the wrist, and T is work duration of observations. It can be seen that total tendon travel during the work period provides an exposure index that captures frequency, ωi, peak wrist excursion, θ 0i, and work duration, T. Mean velocity and acceleration for wrist flexion-extension and cumulative tendon excursions were significant (p 0.5) in agreement with recent experimental results [2]. Percept transition times of 150 – 200 ms and mean percept dwell times of 3 – 5 s as reported in the literature, are correctly predicted if a feedback delay of 40 ms is assumed as mentioned in the literature (e.g. [21]). Keywords: cognitive bistability, modelling, nonlinear dynamics, perception, attention, Hurst parameter.

1 Introduction In the present work new simulation results of a nonlinear dynamics model of cognitive multistability [3] are presented. Multistable perception is the spontaneous involuntary switching of conscious awareness between the different percepts of an ambiguous stimulus. It is excited with different methods and stimuli such as binocular rivalry [5], perspective reversal, e.g. with the famous Necker cube [6][7][25], and ambiguous motion displays [8]. Bistability provides an unique approach to fundamental questions of perception and consciousness because it allows for the direct measurement of the switching of subjective perception under constant external stimulus (e.g. [9][10][11][12] [13]). Various aspects of the present model were described in previous papers [3][14][15] where results on stability, typical time scales, statistics of perceptual dominance times, and memory effects were compared with experimental results found in the literature. The present simulation results are compared with two different experiments: classical results of Orbach et.al. [1][6] addressing percept stabilization due to periodic interruption of stimulus, and recently discovered long range correlations of the perceptual duration times [2] via determination of the self similarity (Hurst) parameter H (> 0.5) of the dwell time series. V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 227–236, 2009. © Springer-Verlag Berlin Heidelberg 2009

228

N. Fürstenau

Concerning theoretical modeling there is an ongoing discussion on the predominance of stochastic [16] [17] versus deterministic [3] [18][19] background of multistability, and on the importance of neural or attentional fatigue [6][19] versus memory effects [1][17]. The synergetic model of Ditzinger & Haken [19] is based on two separate sets of coupled nonlinear dynamics equations for the two perception state order parameters and the corresponding attention (control) parameters. According to the experimentally supported satiation (neuronal fatigue) hypothesis [6], quasiperiodic transitions between different attractor states of the perception order parameter are induced by a slow time variation of the attention (control) parameter due to perception–attention coupling. Following [19] and supported by recent experimental results in [4][25][29] the present model couples the dynamics of a macroscopic (behavioral) perception state order parameter with an adaptive attention control parameter, corresponding to feedback gain with delay and additive noise [3]. Memory effects are introduced by allowing for the adaptation of the originally constant attention bias parameter which balances the subjective preference of one of the two percepts. By including an additive attention noise term the model explains the experimental finding that deterministic as well as stochastic dynamics determines the measured reversal time statistics for different multistability phenomena. In section 2 the theoretical approach is described. Computer simulations of perception time series are presented in section 3, adressing percept stabilization with interrupted stimulus in 3.1 and predicting long range correlations with adaptive bias under constant stimulus in 3.2. Discussion of results and the conclusion follows in section 4.

2 Theory 2.1 The Recursive Mean Field Interference Model After reviewing important features of the present model I will add some aspects not mentioned in previous papers [3][14][15]. In agreement with the widely accepted view of reentrant synchronous interactions between distant neuronal groups within the thalamo-cortical system leading to conscious perception (e.g. [13][22][25][29]), the present model assumes superimposition of coherent fields a(Φ1(t)), b(Φ2(t) representing the possible percepts P1, P2, and recursive processes to determine the multistable perception dynamics. Like [19] it utilizes perception-attention coupling, however within a delayed reentrant loop modulating the phase difference ΔΦ = Φ1 – Φ2, with attention identified with feedback gain [3][26], and adaptive attention bias balancing preference between percepts via learning and memory. This approach results in a phase dynamics ΔΦ(t) formalized by a recursive cosinuidal mapping function. The architecture is motivated by thalamo-cortical (TC) reentrant loops as proposed within the dynamical core hypothesis of consciousness [13] and within the discussion of bottom-up and top-down aspects of visual attention [26]. The present approach is furthermore motivated by the mean field phase oscillator theory of coupled neuronal columns in the visual cortex [23]. It describes via the circle (sine) map the synchronization of neural self oscillations as the physiological basis of dynamic temporal binding which in turn is thought to be cruical for the selection of perceptually or

Computational Nonlinear Dynamics Model of Percept Switching

229

Fig. 1. Schematic of visual information flow within the thalamo-cortical system, with indication of bottom-up streams and attentional top-down modulation (black arrows) of ventral ("what") and dorsal ("where") pathways resulting in recurrent v-G-vb loops (based on [26][27]) with feedforward and reentrant delay T ≈ 40 ms [21]. Top sketch shows time scales of disambiguation process.

behavioraly relevant information [10][11][12]. Accordingly Figure 1 depicts within a block diagram important modules of the attentionally modulated visual perception system. The diagram is based on classical brain circuit schematics (e.g. [27]) and extends a figure in [26] depicting the attentional top-down modulation of the dorsal ("where") and ventral ("what") streams of information. Within the present model it is assumed that for the emergence of the conscious percept, feedforward preprocessing of the stimulus up to the Primary Visual Cortex V1 as well as the loop via superior colliculi can be neglected. The main processing takes place within recurrent TC-loops under covert attention (e.g. [9][22][26]). The model architecture is suggested to basically represent the ventral ("what") V2/V4– InferoTemporal (IF)–PraeFrontal (PF)–V2/V4 loop and the TC-hippocampal (memory) loop as target structure. Recent experimental evidence on perception–attention coupling with ambiguous stimuli was based on EEG recording of frontal theta and occipital alpha bands [25]and eye blink rate measurement [4]. According to Hillyard et.al. [28] stimulus-evoked neuronal activity can be modified by an attentional induced additive bias or by a true gain modulation (present model parameters vb(t) and g(t)). Increase of gain g(t) is correlated with increased blood flow through the respective cortical areas. Consequently in the present model, like in [19], the feedback gain serves as adaptive control parameter (g ∼ attention parameter G) which induces the rapid transitions between the alternative stationary perception states P1 and P2, through attention fatigue [6][19]. The reentrant coherent field superimposition yields an overdamped feedback system with a first order dynamical equation. The resulting phase oscillator equation (1) is similar to the phase attractive circle map of Kelso et.al. [24]. The complete dynamics is described by three coupled equations for the perception state order parameter (phase difference v(t) = ΔΦ/π), the attention control parameter G(t), and for the attention bias or preference vb(t). The full model is built upon a set of three perception-attention-memory (PAM) equations for each percept Pi,

230

N. Fürstenau

i = 1,2,..., n, with inhibiting (phase) coupling -cij vj , i ≠ j, in the nonlinear mapping functions, comparable to [19]. n ⎡ ⎛ ⎛ ⎞ ⎞⎤ τv& it + T + v it + T = Gti ⎢1 + μ i cos⎜ π⎜⎜ v it − ∑ cij v tj + v B ⎟⎟ ⎟⎥ . ⎟ ⎜ i≠ j ⎢⎣ ⎠ ⎠⎥⎦ ⎝ ⎝

(

) (

)

i & i = v i − v i /γ + G . G mean - G t /τ G + L t t b t

(

)

(

)

v& ib t = vibe − vib t M/τ L + v it − vibt /τ M .

(1)

(2) (3)

In the computer experiments of section 3, however, like in previous publications, for the bistable case a reduced model with a single set of PAM equations will be used. This is justified by the fact that without noise the system behavior is completely redundant with regard to perception states i = 1, 2 (P1, P2) as will be shown in section 2.2 (see also [19]). The advantage of reduced number of parameters has to be payed for by slightly unsymmetric behavior of P1, P2 time series (slightly different mean dwell times with symmetric bias vb) The reduced model system behavior can be understood as follows. An ambiguous stimulus with strength I and difference of meaning μ (interference contrast 0 ≤ μ ≤ 1) of the two possible percepts P1, P2 excites two corresponding hypothetical mean fields [a1, a2] representing percept possibilities, with phase difference ΔΦ. A recurrent process is established by feedback of the output U ∼ |a1 + a2|2 after amplification (feedback gain g) with delay T into ΔΦ via a hypothetical phase modulation mechanism ΔΦ = πU/Uπ. = πv. As a quantitative estimate for T the reentrant (feedback) processing delay of ≈ 40 ms within the association cortex is assumed as mentioned by Lamme [21]. The nonlinear rhs. of equ. (1) describes the conventional interference between two coherent fields. In what follows I assume the phase bias vB = 0 mod 2. In agreement with Itti & Koch [26] the attention parameter G(t) ∼ κ I0 g(t) is the product of feedback gain g(t) and input (stimulus) strength I0 (=1 in what follows). The attention dynamics is determined by the attention bias vb (determining the relative preference of P1 and P2), fatigue time constant γ, recovery time constant τG, and Gmean = 0.5(3 – μ)/(1 – μ2) = center between turning points of stationary hysteresis v*(G) (see below). Following [19], the random noise due to physically required dissipative processes is added to the attention equation G(t) as a stochastic Langevin force L(t) with band limited white noise power Jω. The attention bias or preference dynamics dvb/dt is modelled as the sum of a learning term M(vt,vb,vbe)(vbe – vb)/τL, and of a memory component (– vb)/τM which couples vb to the low pass filtered perception state. Learning of an unfamiliar (weak) percept Pj is active only in the initial phase of the time series if a Pj association is low and a fluctuation induced jump into the weak Pjperception state from Pi occurs, switching M from 0 to 1. 2.2 Stationary Solutions and Self-oscillations Quasiperiodic switching between two attractor states v*1(P1) and v*2(P2) emerges after a node bifurcation of the stationary solution v*(G). It evolves from a monotonous


a)

231

b)

Fig. 2. a) First order stationary solution of a single percept equation (1) with arrows indicating g-v phase space trajectories of perceptual self oscillations (frequency fG, vertical) and externally imposed stimulus oscillation μ(t) = 0.2 ⇔ 0.6: (frequency fS, horizontal). b) Numerical solution of the full model equs. (1) (2) (3): over 3000Ts = 1 min, depicting redundancy due to antiphase of v1, v2. Stimulus μ(t) changes at t = 1000 Ts = 20 s from μ = 0.2 to μ = 0.6.

function into a hysteresis (S-shaped) ambiguous one with increasing μ as can be seen in the first order stationary solution of equ. (1) shown in Figure 2a). The stationary solution supports the proposed catastrophe topology of the cognitive multistability dynamics [18]. At the critical value, μn = 0.18, the slope of the stationary system state v*(G) becomes infinite, with (Gn, vn) ≈ (1.5, 1.5). For μ < μn both percepts are fused into a single meaning. For μ > μn the stationary solution v*(G) becomes multivalued. For maximum contrast μ = 1 the horizontal slope dv/dG = 0 yields v i∞ = 2i − 1 , i = 1,2,3,… as stationary perception levels for G → ∞. Figure 2b) depicts a numerical solution of the set of two coupled PAM equations with identical parameter values T = 2, τ = 1, γ = 60, τG = 500, cij = 0.1, constant attention bias vb = 1.5, noise power Jω = 0 (time units = sample time TS = 20 ms), as obtained with a Matlab–Simulink code using the Runge-Kutta solver "ode23tb" [3][14][15]. Higher order stationary solutions yield period doubling pitchfork bifurcations [3][14][15] (not shown in Fig. 2a)) on both positive slope regions of the hysteresis curve, with the G-values of the bifurcation points converging at the chaotic boundary according to the Feigenbaum constant δ ∞ = 4.6692 . The corresponding P1-, P2-limit cycle oscillations and chaotic contributions can be seen in Figure 2b) which depicts time series of perceptual switching events of the percept vector [v1, v2] for small and large contrast parameter μ. The small-μ self-oscillations change into pronounced switching between percept-on (vi > 2) and –off (vi ≈ 1) with incrasing contrast. In contrast to the quasiperiodic P1-P2 switching the superimposed limit cycle oscillations (> 5 Hz) originate from the finite delay T with the amplitudes corresponding to the pitchfork bifurcation pattern [3][15]. The linear stability analysis of equ.(1) [15] yields Eigenfrequencies β = 2πf via βτ = − tan(βT ) with numerical values f/Hz = 9.1,

232

N. Fürstenau

20.2, 32.2, 44.5 ... for τ = 20 ms, T = 40 ms. This spectrum compares reasonably well with typical EEG frequencies as well as fixational eye movements as related external observables. The percept reversal time period is determined by the slow G(t) dynamics, with fatigue and recovery time constants γ, τG, leading to the quasiperiodic P1→P2 transitions at the G-extrema. An analytic estimate for small μ of the expected perceptual self oscillations between the stationary states v*(P1) ⇔ v*(P2) due to the v – G coupling may be obtained by combination of equations (1) and (2) yielding the reversal frequency fG = f0 1− D2

(4)

with eigenfrequency ω 0 = 1/ γ(τ + T ) = 3.73rad/s or f0 = 0.59 Hz = 36 min-1 or T0 = 1.7 s. The influence of the damping term can be derived after transformation of the timescale into eigentime ϑ = ω 0 t with normalized damping D = (1 − πμG * )/(2ω 0 (τ + T )) , yielding the reversal rate fD = 0.55 Hz = 33 min-1 in exact agreement with the the numerical solution in Fig. 2b). Although the very rough dwell time estimate for a single percept Δ(Pi) = TG / 2 = 1/2fG due to the low hysteresis (μ = 0.2 ) lies at the lower end of the typical experimental results it nevertheless predicts the correct order of magnitude, e.g. [6][7][16][20]. The percept duration time statistics has been shown in numerous experimental investigations (e.g.[7][20][29]) and different theoretical modelling approaches ([3][19][24]) to correspond to a Γ-distribution as a reasonable approximation. Time series of the kind shown in Fig. 2b) obtained with the simplified (scalar) model were analyzed in previous publications [3][14][15] with respect to the relative frequencies of perceptual duration times Δ(P1), Δ(P2). The analysis confirmed the Γ-distribution statistics of percept dwell times as a good approximation, with absolute mean values Δm of some seconds and relative standard deviation σ/Δm ≈ 0.5 [7][20].

3 Computer Experiments In what follows numerical evaluations of the PAM-equations in its reduced scalar form are presented for comparing theoretical predictions with a) experiments addressing fatigue suppression (or percept stabilization) with periodically interrupted ambiguous stimulus [1][6], and b) long range correlations within dwell time series observed under constant stimulus [2]. 3.1 Perception–Attention Dynamics with Interrupted Stimulus In this section numerical evaluations of a single set of PAM equations with periodically interrupted stimulus are presented. Figure 3 shows for the same parameter values as Fig. 2b) over a period of tSim = 2000 TS = 40 s the time series μ(t), G(t) and v(t), however with noise power Jω = 0.001 (noise sample time tc = 0.1), and τM = 10000, τL = 100000, i.e. effectively constant bias. The periodically interrupted stimulus parameter (contrast) μ(t) alternates between 0.6 = stimulus-on and 0.1 = stimulus-off with ton = toff = 300 ms.


233

Fig. 3. Numerical evaluation of PAM-equations (reduced scalar model) for periodic stimulus with ton = toff = 300 ms. From bottom to top: Stimulus parameter μ(t) alternating between μ = 0.6 (on) and 0.1 (off), attention parameter G, perception state v(t). For details see text.

The v(t) dynamics in Fig.3 exhibits the expected quasiperiodic transitions between stationary perception states P1 (near v* ≈ 1) and P2 (near v* ≈ 2.5). During stimulus on periods the expected superimposed fast limit cycle and chaotic oscillations are observed. The transition time between P1 and P2 is of the order of 8 - 10 TS ≈ 150 - 200 ms, in reasonable agreement with the time interval between stimulus onset and conscious perception [21]. Figure 4 shows model based reversal rates 1/Δm as function of toff.

Fig. 4. Reversal rate 1/Δm obtained from computer experiments for ton = 300 ms and 10 ms ≤ toff ≤ 800 ms (circles: 100 time series of tSim = 5000 TS/data point) and experimental values [1] (crosses)

Numerical values are determined by evaluation of time series like in Fig.3 with ton = 300 ms and a range of toff–values corresponding to experiments reported in [1][6]. A surprisingly good agreement is observed between model simulations and experiments, even with regard to the absolut maximum, indicating the fatigue induced phaseoscillator mechanism to capture essential aspects of the cognitive bistability dynamics.

234

N. Fürstenau

3.2 Memory Effects through Adaptive Bias In a recent analysis of perceptual dwell time statistics as measured with Necker cube and binocular rivalry experiments Gao et.al. [2] detected significant long range correlations quantified by the Hurst parameter (H > 0.5), with 0.6 < H < 0.8 for 20 subjects who indicated subjective percept switching by pressing a button. With the present model the coupling of the dynamic bias vb to the perception state leads to long term correlations via memory effects. The left graph of Figure 5 depicts simulated subjective percept switching with dwell times Δ(P2) versus reversal number. Simulation parameters are μ = 0.6, vb0 = vbe = 1.5, T = 2TS, τ = 0.5, γ = 60, τG = 500, Jω = 0.004, dynamic bias (preference) time constants τM = 3000, τL = 100000. The right graph of Fig. 5 depicts the evaluation of H from 100 time series with simulation length 5000 TS by employing the log(variance(Δ(m))) vs. log(sample size m) method with var(Δ(m)) = s2 m2H-1 as used by Gao et.al. [2]. H is determined from the slope of the regression line and includes 95% confidence intervals of parameter estimates.

Fig. 5. Left: Simulated subjective responses to percept switching depicting dwell times Δ(P2). Right: variance(m) vs. sample time (m) plot of the same simulation runs with linear fit (95% conf. intervals) for estimating H via the slope of the regression line.

It shows significant long range correlations due to the memory effect if the time constant for the attention bias vb satisfies τM < 10000 TS = 200 s. The learning component in equ.(3) influences the dynamics only in the initial phase if |vbe – vb(t=0)| > 0 and only if τL < 2000. Large τL,M (vanishing memory change) represent quasi static preference: for τM,L > 10000 the long range correlations vanish, with H ≈ 0.5 corresponding to a random walk process (Brownian motion).

4 Discussion and Conclusion For the first time to our knowledge the percept reversal rate of alternating perception states under periodic stimulus and the memory effect of an adaptive perception bias was derived by computer simulations using a single behavioral nonlinear dynamics phase oscillator model based on perception-attention-memory coupling and phase feedback. The PAM model can be mapped to a simplified thalamocortical reentrant circuit including attentional feedback modulation of the ventral stream [26]. For the


235

bistable case the full vector model with a set of PAM equations per perception state can be approximated by a scalar PAM model due to redundancy of the noise-free case, at the cost of slight unsymmetries between v1, v2 time series statistics. The dynamics of the reentrant self oscillator perception circuit is determined by delayed adaptive gain for modeling attention fatigue, with additive attention noise. The attention in turn is biased by an adaptive preference parameter coupled to the perception state for simulating memory effects. Simulated perceptual reversal rates under periodic stimulus provide surprisingly good quantitative agreement with experimental results of Orbach et al. [1][6]. With memory time constants < 200 s reversal time series exhibit long range correlations characterized by a Hurst (self similarity) parameter H > 0.5 in agreement with experimental results of Gao et.al.[2]. The present model supports the early proposal of Poston & Stewart [18] of a deterministic catastrophe topology as the basis of the perception reversal dynamics. Acknowledgement. I am indebted to Monika Mittendorf for help with the computer experiments and to J.B. Gao and K.D. White of Univ. of Florida for providing an early preprint of their work.

References 1. Orbach, J., Zucker, E., Olson, R.: Reversibility of the Necker Cube: VII. Reversal rate as a function of figure-on and figure-off durations. Percept. and Motor Skills 22, 615–618 (1966) 2. Gao, J.B., Merk, I., Tung, W.W., Billok, V., White, K.D., Harris, J.G., Roychowdhury, V.P.: Inertia and memory in visual perception. Cogn. Process 7, 105–112 (2006) 3. Fürstenau, N.: A computational model of bistable perception-attention dynamics with long range correlations. In: Hertzberg, J., Beetz, M., Englert, R. (eds.) KI 2007. LNCS, vol. 4667, pp. 251–263. Springer, Heidelberg (2007) 4. Ito, J., Nikolaev, A.R., Luman, M., Aukes, M.F., Nakatani, C., van Leeuwen, C.: Perceptual switching, eye movements, and the bus paradox. Perception 32, 681–698 (2003) 5. Blake, R., Logothetis, N.K.: Visual competition. Nature Reviews / Neuroscience 3, 1–11 (2002) 6. Orbach, J., Ehrlich, D., Heath, H.A.: Reversibility of the Necker Cube: An examination of the concept of satiation of orientation. Perceptual and Motor Skills 17, 439–458 (1963) 7. Borsellino, A., de Marco, A., Allazetta, A., Rinesi, S., Bartolini, B.: Reversal time distribution in the perception of visual ambiguous stimuli. Kybernetik 10, 139–144 (1972) 8. Hock, H.S., Schöner, G., Giese, M.: The dynamical foundations of motion pattern formation: Stability, selective adaptation, and perceptual continuity. Perception & Psychophysics 65, 429–457 (2003) 9. Koch, C.: The Quest for Consciousness – A Neurobiological Approach, German Translation. Elsevier, München (2004) 10. Engel, A.K., Fries, P., Singer, W.: Dynamic Predictions: Oscillations and Synchrony in Top-Down Processing. Nature Reviews Neuroscience 2, 704–718 (2001) 11. Engel, A.K., Fries, P., König, P., Brecht, M., Singer, W.: Temporal binding, binocular rivalry, and consciousness. Consciousness and Cognition 8, 128–151 (1999) 12. Srinavasan, R., Russel, D.S., Edelman, G.M., Tononi, G.: Increased synchronization of magnetic responses during conscious perception. J. Neuroscience 19, 5435–5448 (1999)

236

N. Fürstenau

13. Edelman, G.: Wider than the Sky, pp. 87–96. Penguin Books (2004) 14. Fürstenau, N.: Modelling and Simulation of spontaneous perception switching with ambiguous visual stimuli in augmented vision systems. In: André, E., Dybkjær, L., Minker, W., Neumann, H., Weber, M. (eds.) PIT 2006. LNCS (LNAI), vol. 4021, pp. 20–31. Springer, Heidelberg (2006) 15. Fürstenau, N.: A nonlinear dynamics model of Binocular Rivalry and Cognitive Multistability. In: Proc. IEEE Int. Conf. Systems, Man, Cybernetics, pp. 1081–1088 (2003) 16. De Marco, A., Penengo, P., Trabucco, A., Borsellino, A., Carlini, F., Riani, M., Tuccio, M.T.: Stochastic Models and Fluctuations in Reversal Time of Ambiguous Figures. Perception 6, 645–656 (1977) 17. Merk, I.L.K., Schnakenberg, J.: A stochastic model of multistable perception. Biol.Cybern. 86, 111–116 (2002) 18. Poston, T., Stewart, I.: Nonlinear Modeling of Multistable Perception. Behavioral Science 23, 318–334 (1978) 19. Ditzinger, T., Haken, H.: A Synergetic Model of Multistability in Perception. In: Kruse, P., Stadler, M. (eds.) Ambiguity in Mind and Nature, pp. 255–273. Springer, Berlin (1995) 20. Levelt, W.J.M.: Note on the distribution of dominance times in binocular rivalry. Br. J. Psychol. 58, 143–145 (1967) 21. Lamme, V.A.F.: Why visual attention and awareness are different. Trends in cognitive Sciences 7, 12–18 (2003) 22. Tononi, G., Edelman, G.M.: Consciousness and Complexity. Science 282, 1846–1851 (1998) 23. Schuster, H.G., Wagner, P.A.: A Model for Neural Oscillations in the Visual Cortex: 1. Mean field theory and the derivation of the phase equations. Biol. Cybern. 64, 77–82 (1990) 24. Kelso, J.A.S., Case, P., Holroyd, T., Horvath, E., Raczaszek, J., Tuller, B., Ding, M.: Multistability and metastability in perceptual and brain dynamics. In: Kruse, P., Stadler, M. (eds.) Ambiguity in Mind and Nature, pp. 255–273. Springer, Berlin (1995) 25. Nakatani, H., van Leeuwen, C.: Transient synchrony of distant brain areas and perceptual switching in ambiguous figures. Biol. Cybern. 94, 445–457 (2006) 26. Itti, L., Koch, C.: Computational Modelling of Visual Attention. Nature Reviews Neuroscience 2, 194–203 (2001) 27. Robinson, D. (ed.): Neurobiology. Springer, Berlin (1998) 28. Hillyard, S.A., Vogel, E.K., Luck, S.J.: Sensory gain control (amplification) as a mechanism of selective attention: electrophysiological and neuroimaging evidence. In: Humphreys, G.W., Duncan, J., Treisman, A. (eds.) Attention, Space, and Action, pp. 31–53. Oxford University Press, Oxford (1999) 29. Nakatani, H., van Leeuwen, C.: Individual Differences in Perceptual Switching rates: the role of occipital alpha and frontal theta band activity. Biol. Cybern. 93, 343–354 (2005)

A Computational Implementation of a Human Attention Guiding Mechanism in MIDAS v5 Brian F. Gore1, Becky L. Hooey1, Christopher D. Wickens2, and Shelly Scott-Nash2 1

San Jose State University Research Foundation, NASA Ames Research Center, MS 262-4, Moffett Field, California, USA 2 Alion Science and Technology, 4949 Pearl East Circle, Suite 300 Boulder, Colorado, USA {Brian.F.Gore,Becky.L.Hooey}@nasa.gov, {cwickens,sscott-nash}@alionscience.com

Abstract. In complex human-machine systems, the human operator is often required to intervene to detect and solve problems. Given this increased reliance on the human in these critical human-machine systems, there is an increasing need to validly predict how operators allocate their visual attention. This paper describes the information-seeking (attention-guiding) model within the Man-machine Integration Design and Analysis System (MIDAS) v5 software - a predictive model that uses the Salience, Effort, Expectancy and Value (SEEV) of an area of interest to guide a person’s attention. The paper highlights the differences between using a probabilistic fixation approach and the SEEV approach in MIDAS to drive attention. Keywords: Human Performance Modeling, Modeling Attention, MIDAS v5, SEEV.

1 Introduction There is a need for increased realism in human performance models (HPMs) of extreme and potentially hazardous environments. As the fidelity and realism of the HPMs improve, so too does the need for integrating and using complex human cognitive and attention models. HPMs exist that incorporate basic human vision and attention models to drive how and when a human will respond to events in specific environment contexts. Implementing these models computationally has typically taken the form of scripting a sequence of visual fixations points and some apply a probabilistic distribution [1,2]. Few, if any, HPM-attention models today operate in a closed-loop fashion using information from the environment to drive where the operator is going to look next. As automation and advanced technologies are introduced into current operational environments, there is an increasing need to validly predict how and when a human will detect environmental events. This paper summarizes the augmentations to the information-seeking (attention-guiding) model within the Man-machine Integration Design and Analysis System (MIDAS) v5 software from a probabilistic approach to a predictive model that uses four parameters (Salience, Effort, Expectancy and Value; SEEV) to guide an operator’s attention [3]. V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 237–246, 2009. © Springer-Verlag Berlin Heidelberg 2009

238

B.F. Gore et al.

1.1 Man-machine Integration Design and Analysis System (MIDAS) The Man-machine Integration Design and Analysis System (MIDAS) is a dynamic, integrated human performance modeling and simulation environment that facilitates the design, visualization, and computational evaluation of complex man-machine system concepts in simulated operational environments [4,5]. MIDAS combines graphical equipment prototyping, dynamic simulation, and human performance modeling to reduce design cycle time, support quantitative predictions of humansystem effectiveness, and improve the design of crew stations and their associated operating procedures. HPMs like MIDAS provide a flexible and economical way to manipulate aspects of the operator, automation, and task-environment for simulation analyses [4,5,6]. MIDAS can suggest the nature of likely pilot errors, as well as highlight precursor conditions to error such as high levels of memory demand, mounting time pressure and workload, attentional tunneling or distraction, and deteriorating situation awareness (SA). MIDAS links a virtual human, comprised of a physical anthropometric character, to a computational cognitive structure that represents human capabilities and limitations. The cognitive component is comprised of a perceptual mechanism (visual and auditory), memory (short term memory, long term working memory, and long term memory), a decision maker and a response selection architectural component. The complex interplay among bottom-up and top-down processes enables the emergence of unforeseen, and non-programmed behaviors [7]. MIDAS is unique as it can be used as a cognitive modeling tool that allows the user to obtain both predictions and quantitative output measures of various elements of human performance, such as workload and SA, and as a tool for analyzing the effectiveness of crewstation designs from a human factors perspective [4]. This analysis can help point out fundamental design issues early in the design lifecycle, prior to the use of hardware simulators and human-in-the-loop experiments. In both cases, MIDAS provides an easy to use and cost effective means to conduct experiments that explore "what-if" questions about domains of interest. MIDAS v5 has a graphical user interface that does not require advanced programming skills to use. Other features include dynamic visual representations of the simulation environment, support for multiple and interacting human operators, several HPM outputs (including timelines, task lists, workload, and SA), performance influencing factors (such as error predictive performance, fatigue and gravitational effects on performance), libraries of basic human operator procedures (how-to knowledge) and geometries for building scenarios graphically (that leverage heavily from Siemens' JackTM software) [8]1. 1.2 MIDAS Attention and Perception Model MIDAS represents attention as a series of basic human primitive behaviors that carry with them an associated workload level determined from empirical research [9,10,11]. Actions are triggered by information that flows from the environment, through a perception model, to a selection-architecture (that includes a representation of human 1

Additional MIDAS information in [4,5] and http://hsi.arc.nasa.gov/groups/midas/ JackTM is maintained by Siemens PLM Solutions.

A Computational Implementation of a Human Attention Guiding Mechanism

239

attention loads), to a task network representation of the procedures that then feeds back into the environment. Actions carried out by the MIDAS operator impact the performance of the model in a closed-loop fashion. MIDAS represents perception as a series of stages that information must pass through in order to be processed. The perception model includes visual and auditory information. Visual perception in MIDAS depends on two factors – the amount of time the observer dwells on an object and the perceptibility of the observed object. The perception model computes the perceptibility of each object that falls into the operator’s field of view based on properties of the observed object, the visual angle of the object and environmental factors. In the current implementation of MIDAS, perception is a three-stage, time-based perception model (undetected, detected, comprehended) for objects inside the workstation (e.g., an aircraft cockpit) and a fourstage, time-based perception model (undetected, detected, recognized, identified) for objects outside the workstation (e.g., taxiway signs on an airport surface). The model computes the upper level of detection (i.e., undetectable, detectable, recognizable, identifiable for external objects) that can be achieved by the average unaided eye if the observer dwells on it for a requisite amount of time. For example, in a lowvisibility environment, the presence of an aircraft on the airport surface may be ‘detectable’ but the aircraft company logo on the tail might not be ‘recognizable’ or ‘identifiable’ even if he/she dwells on it for a long time. 1.3 MIDAS Probabilistic Scanning Model MIDAS uses a probabilistic scan pattern to drive the perception model. In the current version, probabilistic scan behaviors drive the eyeball towards a particular area of interest (AOI) based on a combination of the model analysts’ understanding of the operators scan pattern and the analysts’ selection of a statistical distribution of fixation times (i.e. gamma, lognormal, linear, etc) characteristic of the specific Table 1. Visual fixation probability matrix in a model of pilot performance (see [12]) Captain's fixation probabilities by context (phase of flight) Displays Primary Flight Display Nav Display/Elect Moving Map left window left-front window right-front window right window Eng. Indicating & Crew Alerting System Mode Control Panel Jepp chart Control Display Unit Total

descent 0.20 0.20 0.05 0.05 0.05 0.05 0.10 0.10 0.10 0.10 1.00

approach 0.30 0.30 0.05 0.10 0.05

land

0.90 0.10

1.00

1.00

0.05 0.05 0.10 1.00

Context exit runway 0.10 0.20 0.20 0.20 0.10 0.10 0.10 0.10

rollout 0.10 0.10 0.10 0.50 0.10

1.00

after land check taxi to gate

arrive at gate

0.20 0.20 0.20 0.20 0.10 0.05 0.05

0.30 0.20 0.20 0.20 0.10

0.10 0.30 0.30 0.20 0.10

1.00

1.00

1.00

First Officer's fixation probabilities by context (phase of flight) Displays Primary Flight Display Nav Display/Elect Moving Map left window left-front window right-front window right window Eng. Indicating & Crew Alerting System Mode Control Panel Jepp chart Control Display Unit Total

descent 0.10 0.20 0.10 0.10 0.10 0.10 0.10 0.05 0.10 0.05 1.00

approach 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.10 1.00

Context exit runway 0.20 0.20 0.10 0.10 0.10 0.10 0.20 0.10 0.20 0.10

land 0.30 0.20 0.10 0.10 0.10 0.10 0.10

rollout 0.40 0.20

1.00

1.00

1.00

after land check taxi to gate 0.10 0.10 0.25 0.10 0.10 0.20 0.20 0.20 0.20 0.15 0.20 0.10 0.10 1.00

1.00

arrive at gate 0.10 0.10 0.10 0.30 0.40

1.00

240

B.F. Gore et al.

environmental context. This approach requires a known scan pattern (in many cases this requires access to eye-movement data from a human-in-the-loop simulation). Models that use probabilities to drive the scan behavior require extensive model development time in order to represent context. An aviation example from a recently completed MIDAS v5 model (for a scenario description see [12]) will illustrate the manner that the information is input into the MIDAS architecture. The modeled pilots scan the displays and out the windows according to a probability matrix, as presented in Table 1. The probabilities were developed and verified by an experienced commercial pilot Subject Matter Expert (SME). The matrix assigns to the Captain (CA) and First Officer (FO) a probability of attending to information sources (shown in rows) for each of eight scenario contexts or phases of flight (shown in columns). 1.4 MIDAS Implementation of the Probability Matrix Within the model, the probability of visual fixation (location) is context specific as illustrated in Fig. 1. For example, during ‘after land checks’, the Captain is primarily scanning the electronic moving map (EMM) and out the window (OTW), while his/her secondary scanning is towards the Engine Indicating and Crew Alerting System (EICAS). The First Officer (FO) is primarily scanning the EICAS and OTW. The Primary Flight Display (PFD) and EMM are secondary. Probabilities are defined in the node to the right of the high level task (e.g. “descent(1_68)”).

Fig. 1. MIDAS implementation of the probabilistic scan pattern – P decision node (circled) is where the analyst enters the context-specific probabilistic values from probability matrix

This probabilistic approach effectively drives attention when the scan behavior is known but is less suitable when an analyst is interested in predicting the scan pattern given the context of the information content in the modeled world. To address this limitation and to improve the cross-domain generalizability of the MIDAS perception and attention model, MIDAS was augmented to include the validated Salience, Effort, Expectancy, Value (SEEV) model of visual attention [13] as will be described next. 1.5 The Salience, Effort, Expectancy, Value (SEEV) Model The SEEV model began as a conceptual model to predict how visual attention is guided in dynamic large-scale environments [13]. SEEV estimates the probability of


241

attending, P(AOI), to an AOI in visual space, as a linear weighted combination of four components (salience, effort, expectancy, and value) as per the following equation: P(AOI) = s*S –ef*EF + ex*EX + v*V.

(1)

Coefficients in the uppercase describe the properties of a display or environment, while those in lower case describe the weight assigned to those properties in the control of an operator’s attention [14]. Specifically, the allocation of attention in dynamic environments is driven by the bottom-up capture of Salient (S) events (e.g., a flashing warning on the instrument panel) and inhibited by the Effort (E) required to move attention (e.g., a pilot will be less likely to scan an instrument located at an overhead panel, head down, or to the side where head rotation is required, than to an instrument located directly ahead on a head-up display (HUD). The SEEV model also predicts that attention is driven by the Expectancy (EX) of seeing a Valuable (V) event at certain locations in the environment. A computational version of this model drives the eyeballs around an environment, such as the dynamic cockpit, according to the four SEEV parameters. For example, the simulated eyeball following the model will fixate more frequently on areas with a high bandwidth (and hence a high expectancy for change), as well as areas that support high-value tasks, like maintaining stable flight [15].2 SEEV has been under development since 2001 and has been extensively validated with empirical human-inthe-loop data from different domains [3,16]. The integration of the SEEV model into MIDAS allows dynamic scanning behaviors by calculating the probability that the operator’s eye will move to a particular AOI given the tasks the operator is engaged in within the multitask context. It also better addresses allocation of attention in dynamic environments such as flight and driving tasks. A description of the implementation of the SEEV model into the MIDAS software follows.

2 Augmenting the MIDAS Visual Scan Mechanism with SEEV In MIDAS, Effort, Expectancy, and Value are assigned values between 0 and 1, while Salience is left unconstrained. As such, Effort, Expectancy, and Value drive the human operator’s eye around the displays. However, if a salient event occurs, then P(AOI) may be offset by the display exhibiting the salient event until the display location of the salient event has been fixated and detected. In order to integrate SEEV into MIDAS, provisions were made for the analysts to estimate values for each of the four parameters. Each will be discussed in turn.

Salience. In MIDAS, salience is associated with an event, not a display or object. An example of salience could be a proximity indicator on the navigation display that flashes when another aircraft comes too close. That is, for example, a cockpit display becomes salient when it is presenting an alert, but otherwise, is not salient. In addition, salience could include the loudness of an utterance (but not the content), the flash rate of an alert, and the color of an indicator (i.e., red to indicate a failure). In 2

The SEEV conceptual model has been refined to include a “to-be-noticed event” [15,16,17].

242

B.F. Gore et al.

Fig. 2. Salience heuristics are provided to guide model development

MIDAS, the time between the onset of the salient event and the time at which perception exceeds “Undetected” is reported [15,16,17]. The analyst must assess the salience of an event and provide a weight from 1 to 4. To aid this process, and in an attempt to establish a consistent set of rules to be applied across models, simple heuristics were developed: 1 = change with no luminance increase, 2 = change with luminance increase, 3 = change in position and luminance increase, 4 = repeated onsets (flashing). Fig. 2 below shows how an analyst sets the salience of an event in the MIDAS software.

Effort. Effort refers to the work that is required to sample the information (distance to the AOI). Effort is the only inhibitory factor in the SEEV equation and impacts the likelihood of traveling from one AOI to another. Since MIDAS knows the location of all displays and objects in the environment, the model can calculate Effort empirically. In MIDAS, an Effort rating between 0 and 1 is calculated for each AOI relative to the currently fixated AOI and is based on the angular difference. Any AOI that is 90 degrees or more from the current AOI is set to the maximum (1.0). The visual angle to any AOI that is less than 90 degrees is divided by 90 degrees. Expectancy. Expectancy, also called bandwidth, is described as the event frequency along a channel (location). This parameter is based on the assumption that if a channel has a high event rate, people will sample this channel more frequently than if the event rate is lower [14]. An example is the frequent oscillation of attitude of a light plane when encountering turbulence. The pilot expects the horizon line on the attitude indicator to change frequently and therefore monitors it closely. In contrast, the pilot expects the altimeter during a controlled descent to descend at a constant rate and therefore has a low expectation of seeing changes in descent rate. Thus, when the rate of change is constant, the bandwidth is zero. In SEEV applications, bandwidth (event rate) is always used as a proxy for expectancy. In MIDAS, Expectancy is implemented as a SEEV primitive (Fig. 3). Different expectancy values on a given display can be set for each context, procedure and operator. The context of events that precede the onset of a given signal will influence the likelihood that operators will bring their attention into the areas that are infrequently sampled.


243

Fig. 3. An example of setting expectancy for First Officer

Expectancy for each AOI is set by the user to ‘none’, ‘low’, ‘moderate’ or ‘high’. When used in the SEEV equation, Expectancy is converted to 0, .333, .666 and 1.0 respectively. Drilling down on the SEEV Expectancy primitive in the task network reveals the setting as shown in Fig. 3. Value. The level of Value denotes the importance of attending to an event or task or the cost of missing it. For example, information that is used to prevent stalling the aircraft (airspeed, attitude, angle-of-attack), is clearly more important than navigational information, such as waypoint location. The sum of the product of the task value and the relevance of each display to the task is used to compute the value (importance) of the display [14] as illustrated in Table 2. Before the SEEV calculation is run, the task set importance is normalized between 0 and 1 (as shown by the values in Table 2) by computing the sum of all the importance values and then dividing each importance by the sum. It can be seen that an increased weight is given to the front window when avoiding collision relative to maintaining speed and heading. Table 2. Task value computation to determine display importance per context

Task

Task Value .8

Avoid collision Maintain .2 speed/ heading Value of AOI

Front Window

Importance of AOI to task Left Window Near PFD

Near ND

.6

.4

0

0

.1

.1

.4

.4

=(.8*.6)+(.2*.1) =.5

=(.8*.4)+(.2*.1) =.34

=(.8*.0)+(.2*.4) =.08

=(.8*.0)+(.2*.4) =.08

244

B.F. Gore et al.

Fig. 4. Example of assigning the value of AOIs to a task

In MIDAS, Value is implemented using SEEV primitives in order to bracket sets of primitives belonging to the most relevant task. The SEEV calculation considers all tasks that are active until they are explicitly ended by a SEEV end task primitive. For each task, an overall importance is set by the user. The user can indicate a relevance of none, low, moderate and high for each AOI. Just as with Expectancy, these are converted to 0, .333, .666, and 1. In addition, the user can specify none, low, moderate and high importance rating for the entire task. In Fig. 4, monitoring out the window (Front Right Window) is of high importance to the task bracketed by the “Monitoring OTW during land – FO” task set.

3 Discussion Few computational models operate in a closed-loop manner when it comes to seeking information within the environmental context. For a HPM to produce valid output, it must accurately model visual attention. Two attention-guiding mechanisms within MIDAS were presented: Probabilistic fixations and the SEEV approach. Probabilistic scan behaviors drive the eyeball towards a particular AOI based on a known scan pattern and a statistical distribution of fixation times. Models that use probabilities to drive the scan behavior are suitable if the analyst wants to replicate a known scan pattern but are less suitable when an analyst is interested in predicting the scan pattern given the context of information in the environment. Further, the probabilistic approach is often limited in that it does not consider dynamic changes to the environment and to the task. The SEEV method overcomes those limitations by breaking down relevant flight deck display features to four parameters (Salience, Effort, Expectancy, and Value). This approach to modeling attention is more consistent with actual human behavior and has previously been validated with


245

empirical human-in-the-loop data (see [14,16]). The SEEV model is also less prone to error introduced by the modeler/analyst, as it does not require adjustment of fixation probabilities each time the task or environment is changed, as the probabilistic method does.

4 Conclusion Incorrectly defining visual scanning behavior and the manner that humans seek information when interacting in a system context can result in devastating outcomes and system inefficiencies if model results are to be relied upon for system design and evaluation purposes. The improved predictive capability of information-seeking behavior that resulted from the implementation of the validated SEEV model leaves MIDAS better suited to predict performance in complex human-machine systems. Acknowledgments. The SEEV model integration into MIDAS v5 is the result of NASA’s Aviation Safety Program’s Integrated Intelligent Flight Deck Technologies (IIFTD), System Design & Analysis project. The reported modeling effort was coordinated by Dr. David C. Foyle (NASA Ames Research Center). The opinions expressed in this paper are those of the authors and do not reflect the opinions of NASA, the Federal government, Alion Science and Technology, or SJSU.

References 1. Landy, M.S.: Vision and attention for Air MIDAS (NASA Final Report NCC2-5472). New York University, Moffett Field, CA (2002) 2. Corker, K.M., Gore, B.F., Guneratne, E., Jadhav, A., Verma, S.: Coordination of Air MIDAS Safety Development Human Error Modeling: NASA Aviation Safety Program Integration of Air MIDAS Human Visual Model Requirement and Validation of Human Performance Model for Assessment of Safety Risk Reduction through the implementation of SVS technologies (Interim Report NCC2-1563): San Jose State University (2003) 3. Wickens, C.D., McCarley, J.M.: Applied Attention Theory. Taylor and Francis/CRC Press, Boca Raton (2008) 4. Gore, B.F.: Chapter 32: Human Performance: Evaluating the Cognitive Aspects. In: Duffy, V. (ed.) Handbook of Digital Human Modeling. Taylor and Francis/CRC Press, NJ (2008) 5. Gore, B.F., Hooey, B.L., Foyle, D.C., Scott-Nash, S.: Meeting the Challenge of Cognitive Human Performance Model Interpretability Though Transparency: MIDAS V5.X. In: The 2nd International Conference On Applied Human Factors And Ergonomics, Las Vegas, Nevada, July 14-17 (2008) 6. Hooey, B.L., Foyle, D.C.: Advancing the State of the Art of Human Performance Models to Improve Aviation Safety. In: Foyle, D.C., Hooey, B.L. (eds.) Human Performance Modeling in Aviation. CRC Press, Boca Raton (2008) 7. Gore, B.F., Smith, J.D.: Risk Assessment and Human Performance Modeling: The Need for an Integrated Approach. In: Malek, K.A. (ed.) International Journal of Human Factors of Modeling and Simulation, vol. 1(1), pp. 119–139 (2006) 8. Badler, N.I., Phillips, C.B., Webber, B.L.: Simulating Humans: Computer Graphics, Animation, and Control. Oxford University Press, Oxford (1993)

246

B.F. Gore et al.

9. McCracken, J.H., Aldrich, T.B.: Analysis of Selected LHX Mission Functions: Implications for Operator Workload and System Automation Goals (Technical note ASI 479-024-84(b)). Anacapa Sciences, Inc. (1984) 10. Hamilton, D.B., Bierbaum, C.R., Fulford, L.A.: Task Analysis/Workload (TAWL) User’s Guide Version 4.0. U.S. Army Research Institute, Aviation Research and Development Activity, Fort Rucker, AL: Anacapa Sciences, Inc. (1990) 11. Mitchell, D.K.: Mental Workload and ARL Workload Modeling Tools. ARL-TN-161. Aberdeen Proving Ground, M.D., Army Research Laboratory (2000) 12. Hooey, B.L., Gore, B.F., Scott-Nash, S., Wickens, C.D., Small, R., Foyle, D.C.: Developing the Coordinated Situation Awareness Toolkit (CSATK): Situation Awareness Model Augmentation and Application. In: HCSL Technical Report (HCSL-08-01) NASA Ames, Moffett Field, CA (2008) 13. Wickens, C.D., Goh, J., Helleberg, J., Horrey, W., Talleur, D.A.: Attentional Models of Multi-Task Pilot Performance using Advanced Display Technology. Human Factors 45(3), 360–380 (2003) 14. Wickens, C.D., McCarley, J.S., Alexander, A., Thomas, L., Ambinder, M., Zheng, S.: Attention-Situation Awareness (A-SA) Model of Pilot Error. In: Foyle, D., Hooey, B.L. (eds.) Human Performance Modeling in Aviation, Taylor and Francis/CRC Press, Boca Raton (2008) 15. Wickens, C.D., Hooey, B.L., Gore, B.F., Sebok, A., Koenecke, C., Salud, E.: Identifying Black Swans in NextGen: Predicting Human Performance in Off-Nominal Conditions. In: Proceeding of the 53rd Annual Human Factors and Ergonomics Society General Meeting, San Antonio, TX, October 19-23 (2009) 16. Gore, B.F., Hooey, B.L., Wickens, C.D., Sebok, A., Hutchins, S., Salud, E., Small, R., Koenecke, C., Bzostek, J.: Identification Of Nextgen Air Traffic Control and Pilot Performance Parameters for Human Performance Model Development in the Transitional Airspace. In: NASA Final Report, ROA 2007, NRA # NNX08AE87A, SJSU, San Jose (2009) 17. McCarley, J., Wickens, C.D., Steelman, K., Sebok, A.: Control of Attention: Modeling the Effects of Stimulus Characteristics, Task Demands, and Individual Differences. NASA Final Report, ROA 2007, NRA NNX07AV97A (2007) (in prep.)

Towards a Computational Model of Perception and Action in Human Computer Interaction Pascal Haazebroek and Bernhard Hommel Cognitive Psychology Unit & Leiden Institute for Brain and Cognition Wassenaarseweg 52, Leiden, 2333 AK The Netherlands {PHaazebroek,Hommel}@fsw.leidenuniv.nl

Abstract. The evaluation and design of user interfaces may be facilitated by using performance models based on cognitive architectures. A recent trend in Human Computer Interaction is the increased focus on perceptual and motorrelated aspects of the interaction. With respect to this focus, we present the foundations of HiTEC, a new cognitive architecture based on recent findings of interactions between perception and action in the domain of cognitive psychology. This approach is contrasted with existing architectures. Keywords: Cognitive Architecture, Perception, Action, HCI, action effect learning, PDP, connectionism.

1 Introduction The evaluation and design of user interfaces often involves testing with human subjects. However, sometimes this is too expensive, impractical or plainly impossible. In these cases, usability experts often resort to analytical evaluation driven by their intuition rather than empirically obtained findings or quantitative theory. In these situations, computational models of human performance can provide an additional source of information. When applied appropriately, these models can interact with user interfaces and mimic the user in this interaction, yielding statistics that enable the usability engineer to quantitatively compare interaction with alternative interface designs or to locate possible bottlenecks in human computer interaction. As more and more aspects of our lives are becoming increasingly 'computerized', even small improvements that slightly facilitate user interaction can scale up to large financial benefits for organizations. In addition, using computational models of human performance may contribute to deeper insights in the mechanisms underlying human computer interaction. Usually, models of human performance are task specific instances of a more generic framework: a cognitive architecture [1]. Such an architecture (e.g., ACT-R, [2]; SOAR, [3]; EPIC, [4]) describes the overall structure and basic principles of human cognition, covering a wide range of human cognitive capabilities (e.g., attention, memory, problem solving and learning). Recently, the focus in Human Computer Interaction is no longer only on purely cognitive aspects, but also on the perceptual and motor aspects of interaction. Computers, mobile phones, interactive toys and other devices are increasingly V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 247–256, 2009. © Springer-Verlag Berlin Heidelberg 2009

248

P. Haazebroek and B. Hommel

equipped with advanced displays and controls, such as direct manipulation GUI's, touch screens, multi-function keys et cetera, that allow for user interfaces that draw on a rich body of real world perceptual-motor experience in the human user [5]. To account for perceptual and action-related interactions, some cognitive architectures have extended their coverage from primarily cognitive processes to perceptual processing and response execution (e.g. EPIC; ACT-R/PM, [6]). Although these approaches have been shown to be quite successful in modeling human performance in a number of specific tasks, they are still too limited to explain more general phenomena that are relevant in the perception-action domain in cognitive psychology. In this paper, we first examine some existing cognitive architectures and discuss their characteristics with respect to a number of challenging findings from cognitive psychology. Next, we present and describe the characteristics of our HiTEC model for perception and action [7]. Finally, we discuss its promise as a cognitive architecture for digital human modeling in HCI.

2 Cognitive Architectures A cognitive architecture can be characterized as a broad theory of human cognition based on a wide selection of human experimental data [1]. Whereas traditional research in cognitive psychology tends to focus on specific theories of a very limited range of phenomena, cognitive architectures are attempts to integrate these theories into computer simulation models. Apart from their potential to compare and contrast various theoretical accounts, cognitive architectures can be useful in creating models for an applied domain like HCI that requires users to employ a wide range of cognitive capabilities, even in very simple tasks. Cognitive architectures define an overall structure and general principles. To model a specific task, certain aspects (e.g., prior knowledge, the task goal) need to be filled in by a cognitive engineer. Only then, ‘running’ the architecture may result in interactions comparable to human behavior. The best known cognitive architectures (e.g., SOAR, EPIC, ACT-R) are theoretically based on the Model Human Processor, the seminal work of Card, Moran, and Newell [8]. According to this theoretical model, the human ‘processor’ is composed of three main modules: perception, cognition and action. It describes cognitive processing as a cyclic, sequential process from stimulus perception to cognitive problem solving to response execution. Note that this closely resembles the seven stage model [9] often used to explain human behavior in HCI: (1) users perceive the state of the world, (2) users interpret their perception, (3) users form evaluations based on these interpretations, (4) users match these evaluations against their goals, (5) users form an intention to act, (6) users translate this intention into a sequence of actions and (7) users execute this action sequence. Executing an action sequence subsequently results in a change in the world state which can again be perceived in stage 1. Traditionally, cognitive architectures are developed to model the middle, cognitive steps of this sequence. It is assumed that the first steps, perceiving and interpreting the world state, are performed relatively easily. The main focus is on comparing the world state with a goal state and deciding upon which action to take next in order to achieve the goal state. It is further assumed that once an action is chosen, its execution is easy, leading to a predictable new world state.

Towards a Computational Model of Perception and Action in HCI

249

The core mechanism used by most architectures is a production rule system. A production rule defines the translation of a pre-condition into an action that is known to produce a desired post-condition. This can be interpreted as “IF (x) THEN (y)” rules. By specifying a set of production rules, a cognitive architecture can be given some prior knowledge resulting in tendencies to choose those actions that eventually realize certain goals. When putting a cognitive architecture, endowed with a set of production rules, in interaction with an environment, conflicts between rules or unexpected conditions may present themselves. Some cognitive architectures have means to cope with these situations. For example, SOAR has a learning mechanism that can learn new production rules [1]. By assuming a set of production rules, a cognitive architecture also assumes a set of action alternatives. However, when a user is interacting with a physical or virtual environment, it is often unclear which actions can be performed. In certain contexts, users may not readily detect all action opportunities and action alternatives may differ in their availability, leading to variance in behavior [10]. This is hard to capture in a cognitive architecture that assumes a predefined set of (re)actions. With the increased interest in the perceptual and action-related aspects of human computer interaction, some of the architectures have been extended with perceptual and motor modules that allow for the modeling of aspects related to ‘early’ and ‘late’ stages of the interaction cycle as well. For example, EPIC contains not only production rules which define the behavior of the cognitive processor, but also some perceptual-motor parameters which define the time courses of (simulated) perceptual information processing and (simulated) motor action [4]. Importantly, perceptual processing is modeled as a computation of ‘additional waiting time’ before the production rules can be applied. By defining certain parameters in the model, this waiting time can, for instance, vary for different modalities. Similarly, EPIC does not simulate actual motor movement, but computes the time it would take for a particular motor output to be produced after the cognitive processor has sent the action instruction. This time course depends on specified motor features and the current state of the motor processor (i.e.., the last movement it has prepared). ACT-R/PM is a recent extension of the ACT-R cognitive architecture. It allows a modeler to include time estimates of perceptual and motor processes in a similar fashion as EPIC [1]. In sum, cognitive architectures typically maintain a perception-cognition-action flow of information, where the focus of the modeling effort is primarily on cognitive (i.e., problem-solving) aspects. New extensions of some of the leading architectures allow modelers to include perceptual and motor aspects, but this is typically limited to approximations of the time needed to perceive certain features and to produce certain movements.

3 Cognitive Psychology Existing cognitive architectures are generally based on findings in cognitive psychology. They are mainly inspired by studies on problem solving and decision making. However, (recent) findings in the perception-action domain of cognitive psychology may shed some new light on the assumptions of existing cognitive architectures. In the following

250


we discuss a number of these effects. Subsequently, we describe the Theory of Event Coding that aims at integrating these effects into a single meta-theoretical framework. This is the main theoretical basis of the HiTEC architecture that will be described in the next section. Stimulus Response Compatibility. When studying the design of computer interfaces, Simon [11] accidentally discovered that spatial responses (e.g. pressing a left or right key) to non-spatial stimulus features (e.g., color or shape) are faster if the stimulus location corresponds to the response location. This effect has come to be known as the Simon effect. It suggests that while the only specified task ‘rules’ are “IF red THEN left” and “IF green THEN right”, the non-specified ‘rules’ “IF left THEN left” and “IF right THEN right” are apparently active as well. It is clear that a cognitive architecture that incorporates perceptual and action related processes needs to explain this type of automatic stimulus-response interaction in a natural way. Action influences Perception. In recent experiments, it has been shown [12] that if people prepare a manual grasping or reaching action, they detect and discriminate target stimuli in an unrelated interleaved task faster if these targets are defined on feature dimensions that are relevant for the planned action (e.g., shape and size for grasping, color and contrast for reaching). This finding suggests that action planning can influence object perception. It challenges the traditional view of a strictly sequential flow of information from perceptual stages to stages of action planning and execution. Learning Action Alternatives. Various studies, including research on infants, show that people are capable of learning the perceptual effects of actions and subsequently use this knowledge to select an action in order to achieve these effects [13]. In this way, initially arbitrary actions may become the very building blocks of goal directed action. This principle could introduce a more grounded notion of goal-directedness in a cognitive architecture than merely responding with a set of reactions. 3.1 Theory of Event Coding To account for various types of interaction between perception and action, Hommel, Müsseler, Aschersleben, and Prinz [14] formulated the Theory of Event Coding (TEC). In this meta-theoretical framework they proposed a level of common representations, where stimulus features and action features are coded by means of the same representational structures: ‘feature codes’. Feature codes refer to distal features of objects and events in the environment, such as distance, size and location, but on a remote, descriptive level, as opposed to the proximal features that are registered by the senses. Second, at this common codes level, stimulus perception and action planning are considered to be similar processes; both involve activating and integrating feature codes into complex structures called ‘event files’. Third, action features refer to the perceptual consequences of a motor action; when an action is executed, its perceptual effects are integrated into an event file, an action concept. Following the Ideomotor theory [15], one can plan an action by anticipating the features belonging to this action concept. As a result, actions can be planned voluntarily by intending their perceptual effects. Finally, TEC stresses the role of task


251

context in stimulus and response coding. In particular, feature codes are “intentionally weighted” according to the action goal at hand. In order to computationally specify the mechanisms proposed in TEC and validate its principles and assumptions by means of simulations, we are developing the HiTEC architecture [7]. HiTEC is a generic architecture that can be used to define more specific computational models of human perception and action control and that can serve as a starting point for a cognitive architecture in digital human modeling for HCI. In the following, we describe the HiTEC architecture in terms of its structures and processes and discuss how the architecture incorporates the above mentioned psychological effects.

4 HiTEC The Theory of Event Coding provides a number of constraints on the structure and processes of the HiTEC cognitive architecture. First, we describe the general structure of HiTEC and its representations. Next, we describe the processes operating on these representations, following the two-stage model for the acquisition of voluntary action control [16]. 4.1 HiTEC’s Structure and Representations HiTEC is architected as a connectionist network model that uses the basic building blocks of parallel distributed processing (PDP, [17]). In a PDP network model processing occurs through the interactions of a large number of interconnected elements called units or nodes. During each update cycle, activation propagates gradually through the nodes. In addition, connections between nodes may be strengthened or weakened reflecting long term associations between nodes. In HiTEC, the elementary nodes are codes which can become associated. As illustrated in Fig. 1, codes are organized into three main systems: the sensory system, the motor system and the common coding system. Each system will now be discussed in more detail. Sensory System. The human brain encodes perceived objects in a distributed fashion: different features are processed and represented by different brain areas. In HiTEC, different perceptual modalities (e.g., visual, auditory, tactile, proprioceptive) and different dimensions within each modality (e.g., visual color and shape, auditory location and pitch) are processed and represented in different sensory maps. Each sensory map is a module containing a number of sensory codes that are responsive to specific sensory features (e.g., a specific color or a specific pitch). Note that Fig. 1 shows only a subset of sensory maps. Models based on the HiTEC architecture may include other sensory maps as well. Motor System. The motor system contains motor codes, referring to proximal aspects of specific movements (e.g., right index finger press, left hand power grasp et cetera). Although motor codes could also be organized in multiple maps, in the present version of HiTEC we consider only one basic motor map with a rudimentary set of motor codes.

252


Fig. 1. HiTEC Architecture

Common Coding System. According to TEC, both perceived events and events that are generated by action are coded in one common representational domain [14]. In HiTEC, this is implemented as a common coding system that contains common feature codes. Feature codes refer to distal features of objects (e.g., global location in scene, overall object color, size, et cetera) as opposed to the proximal features coded by the sensory codes and motor codes. Feature codes may be associated to both sensory codes and motor. They can combine information from different modalities and are in principle unlimited in number. TEC assumes that feature codes are not fixed, but that they emerge by extracting regularities from sensorimotor experiences. For example, as a result of frequently using the left hand to grasp an object perceived in the left visual field, the distal feature code ‘left’ may emerge which both codes for left-hand actions and for objects perceived in the left visual field. As a result, feature codes gradually evolve and change over time. Associations. In HiTEC, codes can become associated, both for short term and for long term. Short term associations between feature codes reflect that these codes 'belong together in the current task or context’ and that their binding is actively maintained in working memory. In Fig. 1, these temporary bindings are depicted as dashed lines. Long term associations can be interpreted as learned connections reflecting prior experience. These associations are depicted as solid lines in Fig. 1. Event Files. Another central concept of TEC is the event file [18]. In HiTEC, the event file is modeled as a structure that temporarily associates to feature codes that 'belong together in the current context’ in working memory. An event file serves both


253

the perception of a stimulus as well as the planning of an action. When multiple events are present in working memory, choosing between these events (e.g., deciding between different action alternatives) is reflected by competition between the associated event files. This competition is computationally modeled by means of negative associations between event files, depicted as solid lines with filled disk ends in Fig. 1. 4.2 HiTEC’s Processes Following Elsner and Hommel [16] two-stage model of acquisition of voluntary action we now describe the HiTEC processes that enable the learning of action alternatives. Next, we discuss how HiTEC allows for action and task mediated perception as well as stimulus response compatibility. Stage 1: Acquiring Action – Effect Associations. Feature codes are perceptually grounded representations since they are derived by abstracting regularities in activations of sensory codes. Associations between feature codes and motor codes reflect acquired knowledge of action-effect contingencies: motor codes mi are activated, either because of some already existing action-effect associations or simply randomly (e.g., an infant trying out some buttons on an interactive toy). This leads to a change in the environment (e.g., pressing a button produces a sound) which is registered by sensory codes si. Activation propagates from sensory codes towards feature codes fi. Eventually, these feature codes are integrated into an event file ei which acts as an action concept. Subsequently, the cognitive system learns associations between the feature codes fi belonging to this action concept and the motor code mi that just led to the executed motor action. The weights of these associations depend on activation of the motor code and the feature code. Crucially, this allows the task context to influence the learning of action effects, by moderating the activation of certain feature codes. Due to this top-down moderation, task-relevant features (e.g., button look and feel) are weighted more strongly than task-irrelevant features (e.g., lighting conditions in the room). Nonetheless, this does not exclude task-irrelevant but very salient action effects to become involved in strong associations as well. Stage 2: Using Action – Effect Associations. Once associations between motor codes and feature codes exist, they can be used to select and plan voluntary actions. By anticipating desired action effects, feature codes become active. Now, by integrating the feature codes into an action concept, the system can treat the features as constituting a desired state and propagate their activation towards associated motor codes. Initially, multiple motor codes mi may become active as feature codes typically connect to multiple motor codes. However, some motor codes will have more associated features that are also part of the active action concept and some of the mi – fi associations may be stronger than others. Therefore, in time, the network will converge towards a state where only one code mi is strongly activated, which will lead to the selection of that motor action. In addition to the mere selection of a motor action, feature codes also form the actual action plan that specifies (in distal terms) how the action should be executed (e.g., global button location). This action plan is kept active in working memory, allowing

254


the system to monitor, evaluate and adjust the actual motor action. Crucially, action alternatives can be learned and selected in terms of their perceptual effects. Task Preparation. In human computer interaction, users may have tendencies to respond differently to different stimulus elements. To model this, different event files are created and maintained for the various options (e.g., choosing among buttons that produce different sounds). Due to the negative links between these event files, they will compete with each other during the task. Perception and Action. When the environment is perceived, sensory features will activate a set of feature codes. Activation propagates towards one or more event files (that were formed during task preparation). Competition takes place between these event files. Simultaneously, activation propagates from event files to action effect features and motor codes, resulting in the execution and control of motor action. Note that task preparation already sensitizes feature codes both for the to-be-perceived stimuli and for the to-be-planned responses. Therefore, the cognitive system is biased in perceiving elements in the environments and anticipating responses in terms of these feature codes. As the common feature codes are used for both perception and action, perceptual coding can influence action coding and vice versa. Stimulus Response Compatibility. When feature codes for perceived elements and anticipated responses overlap, stimulus-response compatibility effects can arise: if a stimulus element activates a feature code (e.g., picture of an animal) that is also part of the event file representing/of the correct response (e.g., the sound of that animal), this response is activated more quickly, yielding faster reactions. If, on the other hand, the feature code activated by the stimulus element is part of the incorrect response, this increases the competition between the event files representing the correct and incorrect response, resulting in slower reactions.

5 Discussion We discussed existing cognitive architectures and highlighted their limitations with respect to a number of psychological findings highly relevant to HCI. We subsequently described our HiTEC architecture and discussed how these findings could be explained from HiTEC’s basic structures and processes. Like other cognitive models, HiTEC also consists of perception, motor and cognitive modules. However, in contrast to the sequential architecture of existing models, the modules in HiTEC are highly interactive and operate in parallel. Perception of a stimulus does not need to be completed before an action plan is formed (as suggested by linear stage models). Furthermore, the cognitive module contains common codes that are used for encoding perceived stimulus as well as for anticipated actions. Actions are represented as motor programs in the motor module, but they are connected to their (learned) perceptual action effects (e.g., a resulting visual effect or a haptic sensation of a key press) as proposed by Ideomotor theory [15]. The way in which tasks are encoded in HiTEC (by using competing event files) shows similarities to a system where multiple production rules compete for ‘firing’.


255

However, it is important to note that action alternatives are selected on the basis of their distal feature effects, rather than on the basis of their proximal, motor characteristics. We acknowledge that HiTEC, in its current incarnation, is not yet capable of the rich body of simulations that other cognitive architectures have demonstrated. Indeed, we do not exclude a production rule component in future versions, but we emphasize that the core of HiTEC consists of perception-action bindings. The strengths of HiTEC lay primarily in its ability to learn perceptual action effects in a principled way and using these effects for action selection and control. This naturally results in a mechanism that enables stimulus response translation on an abstract level, thereby allowing the system to generalize over different but similar perceptual features, both in object perception and in action planning. This leniency may avoid creating a ‘production rule’ for each and every minute variant that the system may encounter. This may increase the system’s robustness against variability in perception and action. Of course, expending our simulations to environments and tasks that currently can be simulated by other architectures, requires more intricate techniques to actually learn the sensorimotor contingencies (i.e., feature codes) that we now assume. Also, we now discussed a situation where a simple set of task rules was predefined. Further research is necessary to assess the role of long term memory and motivational influences in this respect. With the rise of new HCI environments, such as various mobile devices, virtual reality and augmented reality, HCI within these virtual environments increasingly resembles interaction in the physical world. This trend stresses the importance of studying the implications of findings in the perception-action domain for the field of HCI. Acknowledgments. Support for this research by the European Commission (PACOPLUS, IST-FP6-IP-027657) is gratefully acknowledged.

References 1. Byrne, M.D.: Cognitive architecture. In: Jacko, J.A., Sears, A. (eds.) Human-Computer Interaction Handbook, pp. 97–117. Erlbaum, Mahwah (2008) 2. Anderson, J.R.: Rules of the mind. Erlbaum, Hillsdale (1993) 3. Newell, A.: Unified theories of cognition. Harvard University Press, Cambridge (1990) 4. Kieras, D.E., Meyer, D.E.: An overview of the EPIC architecture for cognition and performance with application to human-computer interaction. Human-Computer Interaction 12, 391–438 (1997) 5. Welsh, T.N., Chua, R., Weeks, D.J., Goodman, D.: Perceptual-Motor Interaction: Some Implications for HCI. In: Jacko, J.A., Sears, A. (eds.) The Human-Computer interaction Handbook: Fundamentals, Evolving Techniques, and Emerging Applications, pp. 27–41. Erlbaum, Mahwah (2008) 6. Byrne, M.D., Anderson, J.R.: Perception and action. In: Anderson, J.R., Lebiere, C. (eds.) The atomic components of thought, pp. 167–200. Erlbaum, Hillsdale (1998) 7. Haazebroek, P., Hommel, B.: HiTEC: A computational model of the interaction between perception and action (submitted)

256


8. Card, S.K., Moran, T.P., Newell, A.: The psychology of human-computer interaction. Erlbaum, Hillsdale (1983) 9. Norman, D.A.: The Design of Everyday Things. Basic Book, New York (1988) 10. Kirlik, A.: Conceptual and Technical Issues in Extending Computational Cognitive Modeling to Aviation. In: Proceedings of Human-Computer Interaction International, pp. 872–881 (2007) 11. Simon, J., Rudell, A.: Auditory s-r compatibility: The effect of an irrelevant cue on information processing. Journal of Applied Psychology 51, 300–304 (1967) 12. Fagioli, S., Hommel, B., Schubotz, R.I.: Intentional control of attention: Action planning primes action-related stimulus dimensions. Psychological Research 71, 22–29 (2007) 13. Hommel, B., Elsner, B.: Acquisition, representation, and control of action. In: Morsella, E., Bargh, J.A., Gollwitzer, P.M. (eds.) Oxford handbook of human action, pp. 371–398. Oxford University Press, New York (2009) 14. Hommel, B., Muesseler, J., Aschersleben, G., Prinz, W.: The theory of event coding (TEC): A framework for perception and action planning. Behavioral and Brain Sciences 24, 849–937 (2001) 15. James, W.: The principles of psychology. Dover Publications, New York (1890) 16. Elsner, B., Hommel, B.: Effect anticipation and action control. Journal of Experimental Psychology: Human Perception and Performance 27, 229–240 (2001) 17. Rumelhart, D.E., Hinton, G.E., McClelland, J.L.: A general framework for parallel distributed processing. In: Rumelhart, D.E., McClelland, J.L., The PDP Research Group (eds.) Parallel distributed processing: Explorations in the microstructure of cognition. MIT Press, Cambridge (1986) 18. Hommel, B.: Event files: Feature binding in and across perception and action. Trends in Cognitive Sciences 8, 494–500 (2004)

The Five Commandments of Activity-Aware Ubiquitous Computing Applications Nasim Mahmud, Jo Vermeulen, Kris Luyten, and Karin Coninx Hasselt University – tUL – IBBT, Expertise Centre for Digital Media, Wetenschapspark 2, B-3590 Diepenbeek, Belgium {nasim.mahmud,jo.vermeulen,kris.luyten, karin.coninx}@uhasselt.be

Abstract. Recent work demonstrates the potential for extracting patterns from users’ behavior as detected by sensors. Since there is currently no generalized framework for reasoning about activity-aware applications, designers can only rely on the existing systems for guidance. However, these systems often use a custom, domain-specific definition of activity pattern. Consequently the guidelines designers can extract from individual systems are limited to the specific application domains of those applications. In this paper, we introduce five high-level guidelines or commandments for designing activity-aware applications. By considering the issues we outlined in this paper, designers will be able to avoid common mistakes inherent in designing activity-aware applications.

1 Introduction In recent years, researchers have demonstrated the potential for extracting patterns from users’ behavior by employing sensors [1-3]. There are various applications for detecting the user's activities. Systems such as FolderPredictor [4] and Magitti [5] offer suggestions to users to assist them in their current activity. Other work has used activity recognition to provide awareness of people's activities to improve collaboration [6, 7] or the feeling of connectedness within groups [8]. Furthermore, a recent trend is to employ sensors to predict the user's interruptibility, allowing computers to be more polite and interact with users in an unobtrusive way [9-11]. As there is currently no generalized framework for reasoning about activity-aware applications, designers can only rely on the existing systems for guidance. However, these systems often use a custom, domain-specific definition of activity pattern. For example, the Whereabouts clock focuses on the user's location to determine their general activity [8], while FolderPredictor only takes the user's desktop activity into account [4]. Consequently, the guidelines designers can extract from individual systems are limited to that system's specific application domain. Although there are existing, focused design frameworks that deal with background and foreground interaction [12]; employing sensors [13-15]; or allowing users to intervene when a system acts on their behalf [16, 17], it is hard for designers to come to a generalized body of design knowledge for activity-aware systems by integrating each of these frameworks. V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 257–264, 2009. © Springer-Verlag Berlin Heidelberg 2009

258

N. Mahmud et al.

In this paper, we introduce five high-level guidelines or commandments that need to be addressed when designing activity-aware applications. The main contribution of this framework is that it allows designers to avoid common mistakes inherent in designing activity-aware applications, regardless of the targeted application domain and activity recognition technologies. We combine and generalize existing models in a convenient way for designers. Just as for designing desktop applications, designers need general guidelines that they can rely on. The availability of such a set of guidelines is a critical factor in moving activity-aware applications beyond research prototypes and into practical applications. Our hope is that this work is another step towards a generalized body of design knowledge for activity-aware systems. Our own work on applications for detecting user’s activities with sensors and a broad study of existing activity-aware systems and design frameworks led us to develop the following five commandments for activity-aware systems: 1. 2. 3. 4. 5.

View activity patterns in context; Don’t view a user’s activities in isolation, but in their social context; Deal with hierarchical reuse of patterns; Take uncertainty into account at different levels of abstraction; Allow users to intervene and correct the system.

In the following, each commandment is explained into detail together with a motivating example.

2 View Activity Patterns in Context Human activity does not consist of isolated actions, it is rooted in context. As discussed by Suchman [18], people's behavior is contextualized, i.e. the situation is a very important factor in determining what people will do. It is not possible to generalize and predict people's behavior without considering the situation they are in at that time. It is important that designers consider activities in context. Context consists of many aspects, including but not limited to: the time of day, location or the presence of other people (see commandment 2). In activity-aware systems, an important aspect is how the elements inside a pattern are temporally related to each other. The time of occurrence of an element in a pattern, its duration and the time interval between two such elements are important issues to consider. The elements might form a sequence, occur concurrently, or have a more complex temporal relationship with each other such as the ones described by Allen [19]. It is important to take these temporal relationships into account in order to correctly identify different activity patterns. Another use for time as context information is in supporting continuous interaction, as described by Abowd et al. [20]. Although desktop computers allow for multitasking, computer systems still largely expect users to do their tasks on a single machine, and to do one task after another (although they can switch quickly by switching from one window to another). Abowd et al. state that this assumption will not be valid for ubiquitous computing. In real life, people also regularly start new activities and stop doing others, and do activities concurrently (e.g. watching television while ironing clothes).

The Five Commandments of Activity-Aware Ubiquitous Computing Applications

259

For a practical method of implementing these features, we refer to Patterson et al. [21] who evaluated several techniques to detect activities, and explain how to support detection of concurrent and interruptible activities.

3 Don’t View a User’s Activities in Isolation, But in Their Social Context As we discussed before in commandment 1, human activity is highly dependent on the context. An important aspect of context is social context. A user’s social context consists of the people who are in his surroundings. A user’s activity patterns might be different when he is alone than when he is in a group. What’s more, they might differ according to who he is with at that time (e.g. his colleagues, his friends or his family). Suppose Jim has a pattern which consists of playing his favorite music when he gets home from work. This pattern might not be applicable to the situation where he comes home and there are guests in the house. Besides the fact that social context is important to consider for classifying patterns, analyzing the patterns of groups of people might help to create common patterns across people. Someone might only once enter a seminar room and turn off the lights when a speaker is going to give a talk, making it impossible to detect this as a pattern. However, several people might perform this activity in a common pattern. On the other hand, some patterns are specific to certain people (e.g. taking extra strong black coffee from the vending machine). This pattern might not necessarily hold for this person’s colleagues.

4 Allow Hierarchical Reuse of Patterns Activity Theory (AT) defines an operation as an atomic procedure. A set of operations forms an action while a set of actions forms an activity [22]. The level of granularity can be determined according to what the sensors allow the system to detect. For example, in a GUI environment operations could be keyboard and mouse events, an action could be “selection”, while the activity could be “select a file”. A set of actions may be reoccurring and can form an activity pattern. Existing patterns might themselves be used again in another higher level pattern (e.g. making coffee could be a sub pattern in the morning routine pattern). While the terminology is not clearly defined (making coffee could become an action while it was previously an activity), the point we want to make here is that hierarchical patterns are a natural way of describing real-life activities. According to AT [22], interaction between human beings and the world is organized into subordinated hierarchical levels. Designers should make sure that activity patterns can be organized in a hierarchy.

5 Take Uncertainty into Account at Different Levels of Abstraction An activity-aware system should take uncertainty into account at different levels of abstraction. Activities are in general detected by aggregating the values of one or

260

N. Mahmud et al.

more sensors and reasoning about these values to detect patterns. Sensors can be pieces of hardware (e.g. a temperature sensor), software (e.g. the user’s schedule obtained from their online calendar) or a combination of the two (e.g. a GPS beacon). At the lowest level, uncertainty can occur due to imprecision in the data generated by sensors. A sensor could be faulty, or might need to be smoothed over time to get correct results. For example, the user’s distance to a display could be detected with a Bluetooth beacon attached to the display. The sensor would retrieve the Received Signal Strength Indication (RSSI) value from the user’s Bluetooth-enabled phone to the beacon. When the RSSI value is low (around zero) the user is standing close to the display, otherwise he will be further away. However, this value will fluctuate a lot, as shown in Figure 1 where the phone is just laying on a desk. The system should deal with this uncertainty by smoothening the sensor reads. At an intermediate level, there can be uncertainty in pattern recognition. This type of uncertainty can have several causes. It could be caused by inadequate sensors that prohibit certain parts of the user’s state from being detected (e.g. the user’s emotional state). Another cause could be insufficient training data for the recognition algorithm. For example, systems that are designed to be used in public spaces might not be able to learn a lot about their users, since most people will only use the system once. Uncertainty at an intermediate level can furthermore be caused by user actions that the designers of the system didn’t take into account. As an example of this kind of uncertainty, consider an anti-carjacking device that will automatically be triggered when the driver exits the car with the engine still running. The system will then after a certain time automatically disable the car’s engine, close the doors and sound an alarm. The motivation for these actions is to make the car unusable when the driver has been unwillingly forced to exit the vehicle. Now suppose Jim is using his car to deliver a local community magazine in the neighborhood. At each house, he parks his car next to the road with the engine running, steps out to drop the magazine in the mailbox, and gets back in the car. When he gets to his friend Tom’s house, he parks

Fig. 1. Uncertainty at the lowest level: imprecision in data generated by a Bluetooth RSSI distance sensor


261

his car on Tom’s driveway. After dropping the magazine in the mailbox of his friend Tom, he sees Tom working in the garden and goes over to chat with him. The engine of Jim’s car is still running, but he feels his car is safe on Tom’s driveway. Besides, they are both nearby. About a minute later however, the engine of Jim’s car suddenly shuts down, the doors close and a loud alarm starts blaring. Jim is unable to enter his car and has to explain the situation to the authorities who arrive soon after. This embarrassing situation occurred because the designers of the anti-carjacking device did not take into account that a driver might ever step out of the car and leave his engine running for more than two minutes. Finally, at the highest level, there might be uncertainty in the user’s mental model of the system. This contributes to the discrepancy between what the user expects (their mental model of the system), what the system can sense through its sensors (e.g. a positioning system might not be as accurate as the user expects, leaving him wondering why his movements are left undetected), and what is desired (what is needed for the application), as discussed by Benford et al. [14]. They argue that by explicitly analyzing mismatches between each of expected, sensed, and desired, designers can detect problems resulting from these mismatches early on as well as find opportunities to exploit these mismatches. An interesting practical, activity-aware application that deals with uncertainty is described by Patterson et al. [21]. Their system allows fine-grained activity recognition by analyzing which objects users are manipulating by means of RadioFrequency Identification (RFID) tags attached to kitchen objects, and a glove equipped with an RFID reader that is worn by the user. Their system is resilient against intermediate level uncertainty: it can recognize activities even when different objects are used for it (e.g. using a table spoon instead of a cooking spoon for making oatmeal). An important way to deal with uncertainty is to involve the user. The system could allow users to override its actions at any time, thereby offering a safety net for when the system’s actions would be inappropriate. When a system is uncertain, it might also ask the user for confirmation. In an ideal case, computation would be split between humans and computers, letting each one perform the tasks they do best [23]. The next commandment discusses user intervention in activity-aware systems.

6 Allow Users to Intervene and Correct the System Users should be able to correct the activity-aware system when it makes a mistake. The system is bound to make mistakes, as it is impossible to detect the user’s activity correctly in every possible situation. General activity recognition might be classified as an AI-complete problem [24], which means that a solution to this problem would presuppose the availability of a solution to the general problem of intelligence. Since it is inevitable that the system will make mistakes, there should be a way for users to correct the system, so they can stay in control. As discussed by Bellotti et al. [13], the lack of communication between a system that employs sensors and its users is an important problem. Because these systems often have both less means of providing feedback and will take more actions on behalf of the user than desktop systems, the user will quickly feel out of control.

262

N. Mahmud et al.

Dey et al. [17] introduce the term mediation to refer to the dialogue that takes place between user and system to handle ambiguities in context. They describe four guidelines for mediation of context-aware systems: 1. Applications should provide redundant mediation techniques to support more natural and smooth interactions; 2. Applications should facilitate providing input and output that are distributed both in space and time to support input and feedback for mobile users; 3. Interpretations of ambiguous context should have carefully chosen defaults to minimize user mediation, particularly when users are not directly interacting with a system; 4. Ambiguity should be retained until mediation is necessary for an application to proceed. Guideline 1 deals with providing several redundant levels of interaction for user input and system feedback. This could for example range from most implicit to most explicit, depending on the user’s attention and level of engagement in the task. Guideline 2 points out the fact that communication between system and user should take into account the space through which the user is moving, and have a timeout period after which a user might not have the chance to interact with a mediator anymore. Guidelines 3 and 4 refer to the fact that mediation should be used with care, so as not to distract the user unnecessarily. To allow corrections to be made in a timely fashion, systems should make clear what they perceive of the user, what (automated) actions they are performing or going to perform and of course provide a way to undo or correct actions. Ju et al. [16] discuss three interaction techniques in their implicit interaction framework that cover these requirements: user reflection (making clear what the system perceives of the user); system demonstration (showing what actions the system is performing or going to perform); and override (providing a handle for the user to correct the system). To illustrate the necessity of mediation, we discuss an experience of incorrect system behavior that one of the authors had when visiting a house of the future. This exhibit demonstrated how recent technology trends such as context-awareness could influence our life in the future, and might make our homes smart. The author missed his train and arrived a bit too late. He had to enter the seminar room when the first talk had already started, which was very annoying since the only entrance to the room was in front of the room, next to the lecturer. As if this wasn’t enough, the smart room automatically turned on the lights when he entered the room, leaving no escape to an unremarkable entrance. This experience clearly illustrates that activity-ware systems need mediation to ensure that users remain in control and will not get frustrated. If we would apply the three interaction techniques of Ju et al. [16] to this scenario, the system might indicate that it senses that the user is entering the seminar room (user reflection), announce that it is going to turn on the lights (system demonstration), and – most importantly – provide the user with an opportunity to cancel this action (override). The authors believe that designers of activity-aware systems should always keep user intervention in mind. Both the guidelines for mediation [17] and the implicit interaction techniques [16] are useful to take into account for this purpose.


263

7 Conclusion In this work we have presented five commandments of activity-aware ubiquitous computing applications which is a design guideline for application designers. The commandments presented in this paper is high level guideline intended to help designers in designing activity-aware systems. The guideline will help designers to avoid common mistakes in designing activity-aware systems. Existing activity-aware applications often use a domain-specific definition of activity pattern. Thus existing works do not provide a generalized body of knowledge that application designers can rely on. This work is a step towards generalized guidelines for the application designer. The commandments in this paper suggest the designer to consider the users’ activity in their social context; to consider a pattern of activity in context and to allow for hierarchical reuse of patterns. The commandments also suggest the designer to consider uncertainty at different levels of abstraction and to allow users to intervene and correct the system when the system makes mistake. For designing desktop applications, there are guidelines allowing the designers to use off-the-shelf knowledge. This is not the case for activity-aware ubiquitous computing applications. Hence, generalized guidelines for designing these applications are necessary to move them from lab prototypes into commercial development. In our ongoing work, we are developing an activity-aware ubiquitous computing system that takes the five commandments into account. In our future work, we want to ensure the validity of our guideline in practical settings.

Acknowledgements Part of the research at EDM is funded by ERDF (European Regional Development Fund) and the Flemish Government. Funding for this research was also provided by the Research Foundation -- Flanders (F.W.O. Vlaanderen, project CoLaSUE, number G.0439.08N).

References 1. Begole, J.B., Tang, J.C., Hill, R.: Rhythm modeling, visualizations and applications. In: Proceedings of the 16th annual ACM symposium on User interface software and technology. ACM, Vancouver (2003) 2. Eagle, N., Pentland, A.: Reality mining: sensing complex social systems. Personal Ubiquitous Comput. 10, 255–268 (2006) 3. Philipose, M., Fishkin, K.P., Perkowitz, M., Patterson, D.J., Fox, D., Kautz, H., Hahnel, D.: Inferring Activities from Interactions with Objects. IEEE Pervasive Computing 3, 50– 57 (2004) 4. Bao, X., Herlocker, J.L., Dietterich, T.G.: Fewer clicks and less frustration: reducing the cost of reaching the right folder. In: Proceedings of the 11th international conference on Intelligent user interfaces. ACM, Sydney (2006) 5. Bellotti, V., Begole, B., Chi, E.H., Ducheneaut, N., Fang, J., Isaacs, E., King, T., Newman, M.W., Partridge, K., Price, B., Rasmussen, P., Roberts, M., Schiano, D.J., Walendowski, A.: Activity-based serendipitous recommendations with the Magitti mobile leisure guide. In: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems. ACM, Florence (2008)

264

N. Mahmud et al.

6. Tullio, J., Goecks, J., Mynatt, E.D., Nguyen, D.H.: Augmenting shared personal calendars. In: Proceedings of the 15th annual ACM symposium on User interface software and technology. ACM, Paris (2002) 7. Isaacs, E.A., Tang, J.C., Morris, T.: Piazza: a desktop environment supporting impromptu and planned interactions. In: Proceedings of the 1996 ACM conference on Computer supported cooperative work. ACM, Boston (1996) 8. Brown, B., Taylor, A.S., Izadi, S., Sellen, A., Kaye, J.J., Eardley, R.: Locating Family Values: A Field Trial of the Whereabouts Clock. In: Krumm, J., Abowd, G.D., Seneviratne, A., Strang, T. (eds.) UbiComp 2007. LNCS, vol. 4717, p. 354. Springer, Heidelberg (2007) 9. Gibbs, W.W.: Considerate Computing. Scientific American 292, 54–61 (2005) 10. Fogarty, J., Hudson, S.E., Atkeson, C.G., Avrahami, D., Forlizzi, J., Kiesler, S., Lee, J.C., Yang, J.: Predicting human interruptibility with sensors. ACM Trans. Comput.-Hum. Interact. 12, 119–146 (2005) 11. Horvitz, E., Koch, P., Apacible, J.: BusyBody: creating and fielding personalized models of the cost of interruption. In: Proceedings of the 2004 ACM conference on Computer supported cooperative work. ACM, Chicago (2004) 12. Buxton, B.: Integrating the Periphery and Context: A New Taxonomy of Telematics. In: Proceedings of Graphics Interface 1995 (1995) 13. Bellotti, V., Back, M., Edwards, W.K., Grinter, R.E., Henderson, A., Lopes, C.: Making sense of sensing systems: five questions for designers and researchers. In: Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves. ACM, New York (2002) 14. Benford, S., Schnädelbach, H., Koleva, B., Anastasi, R., Greenhalgh, C., Rodden, T., Green, J., Ghali, A., Pridmore, T., Gaver, B., Boucher, A., Walker, B., Pennington, S., Schmidt, A., Gellersen, H., Steed, A.: Expected, sensed, and desired: A framework for designing sensing-based interaction. ACM Trans. Comput.-Hum. Interact. 12, 3–30 (2005) 15. Hinckley, K., Pierce, J., Horvitz, E., Sinclair, M.: Foreground and background interaction with sensor-enhanced mobile devices. ACM Trans. Comput.-Hum. Interact. 12, 31–52 (2005) 16. Ju, W., Lee, B.A., Klemmer, S.R.: Range: exploring implicit interaction through electronic whiteboard design. In: Proceedings of the ACM 2008 conference on Computer supported cooperative work. ACM Press, San Diego (2008) 17. Dey, A.K., Mankoff, J.: Designing mediation for context-aware applications. ACM Trans. Comput.-Hum. Interact. 12, 53–80 (2005) 18. Suchman, L.: Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge University Press, Cambridge (1987) 19. Allen, J.F.: Maintaining knowledge about temporal intervals. Commun. ACM 26, 832–843 (1983) 20. Abowd, G.D., Mynatt, E.D.: Charting past, present, and future research in ubiquitous computing. ACM Trans. Comput.-Hum. Interact. 7, 29–58 (2000) 21. Patterson, D.J., Fox, D., Kautz, H., Philipose, M.: Fine-grained activity recognition by aggregating abstract object usage. Wearable Computers. In: Proceedings of Ninth IEEE International Symposium on Wearable Computers, 2005, pp. 44–51 (2005) 22. Leontiev, A.N.: Activity, Consciousness and Personality. Prentice-Hall, Englewood Cliffs 23. Horvitz, E.: Reflections on Challenges and Promises of Mixed-Initiative Interaction. AI Magazine 28, 19 (2007) 24. Mallery, J.C.: Thinking about foreign policy: Finding an appropriate role for artificial intelligence computers. In: The 1988 Annual Meeting of the International Studies Association, St. Louis, MO (1988)

What the Eyes Reveal: Measuring the Cognitive Workload of Teams Sandra P. Marshall Department of Psychology, San Diego State University, San Diego, CA 92182 and EyeTracking, Inc., 6475 Alvarado Rd., Suite 132, San Diego, CA 92120 [email protected]

Abstract. This paper describes the measurement of cognitive workload using the Networked Evaluation System (NES). NES is a unique network of coordinated eye-tracking systems that allows monitoring of groups of decision makers working together in a single environment. Two implementations are described. The first is a military application with teams of officers working together on a simulated joint relief mission, and the second is a fatigue study with teams of individuals working together in a simulated lunar search and recovery mission. Keywords: eye tracking, pupil dilation, cognitive workload, team assessment.

1 Introduction Many activities require teams of individuals to work together productively over a sustained period of time. Sports teams exemplify this, with players relying on each other to maintain vigilance and alertness to changing circumstances of the game. Other types of teams also require vigilance and alertness to detail and often do so under life-threatening circumstances, such as medical teams, SWAT Teams, or First Responder Teams. Each team depends upon the good performance of all its members, and weaknesses in any one of them will change the way the team performs. For instance, sometimes one team member is overloaded and cannot perform his or her duties quickly enough so the entire team slows down; sometimes a team member loses sight of the situation and makes an error so the entire team needs to compensate; and sometimes the team member is fatigued and cannot function effectively so the other members need to assume more responsibility. It is not always immediately evident when a team member is experiencing difficulty. All too often, the first indication is a major error that occurs when the team member reaches the critical point of being seriously impaired (either overloaded or fatigued). Early indication of such problems is clearly desirable but difficult to achieve. This paper describes a networked system for evaluating cognitive workload and/or fatigue in team members as they perform their tasks. The system uses eyetracking data to create a non-intrusive method of workload evaluation. The paper has three parts: the first describes the system itself and how data are collected, the second describes assessing cognitive workload in teams of military officers as they determine V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 265–274, 2009. © Springer-Verlag Berlin Heidelberg 2009

266

S.P. Marshall

how to share resources, and the third describes evaluation of performance and fatigue in a NASA study.

2 The Networked Evaluation System (NES) The networked evaluation system, hereafter called NES, is a unique network of coordinated eye-tracking systems that allows monitoring of groups of decision makers working together in a single environment. Two versions have been developed and tested. One uses lightweight head-mounted optics and the other uses unobtrusive, remote eye-tracking cameras to monitor each individual’s eyes. Each system then synthesizes data from all subjects in real time to enable the comparison of attention level and cognitive workload of all team members. The end product is a functional state-of-the-art eye-tracking network that can produce information in real time about all the team members collectively as well as individually. The head-mounted NES utilizes the SR Research EyeLink II, which is a binocular eye tracking device that samples at 250 Hz. The remote NES utilizes the Tobii X120, which also is a binocular eye tracking device with a sampling rate of 120 Hz. Both eye trackers provide excellent data for eye position (horizontal and vertical pixels) and pupil size. In both configurations, each eyetracker is controlled by GazeTrace™ software from EyeTracking, Inc. which in turn produces the workload measure before feeding it to the central CWAD server (also produced by EyeTracking, Inc.) software for data synchronization and integration [4]. Both NES systems capture the same data: the location of each eye in terms of horizontal and vertical location on the display and the size of each pupil. A primary difference between the two systems is that the head-mounted system records data every 4 msec while the remote system records data every 8.33 msec. The data are transformed by the central processing unit of the NES into more conventional eyetracking metrics such as blinks, fixations, and measures of vergence. The pupil data also are transformed uniquely in the Index of Cognitive Activity (ICA), a patented metric which assess the level of cognitive workload experienced by an individual [2, 3]. Altogether, these metrics may then be combined to provide estimates of cognitive state [1, 4]. In particular, they are useful for examining whether an individual is overloaded, fatigued, or functioning normally. All eyetracking systems in either NES are interconnected by a private computer network. GazeTrace software controls the eyetrackers, instructing them first to calibrate and then to start collecting data. In real time, the GazeTrace software computes the ICA workload measure and sends it to the CWAD server where it is synchronized into a database with eye and workload data from the other eyetrackers in the session.

3 Assessing Cognitive Workload Level The research reported here was conducted under the Adaptive Architectures for Command and Control (A2C2) Research Program sponsored by the Office of Naval

What the Eyes Reveal: Measuring the Cognitive Workload of Teams

267

Research. It was conducted at the Naval Postgraduate School in Monterey, CA. Researchers from San Diego State University and the Naval Postgraduate School collaborated to carry out the study. The primary purpose of the study was to examine how team members work together to overcome limitations, changes, or problems that arise during a mission. Three-person teams worked together in scenarios created within the Distributed Dynamic Decision-Making Simulation (DDD), a simulation system that allows multiple computers to interface and display coordinated screens for a defined environment. The general focus of the study was the Expeditionary Strike Group (ESG), a relatively new military concept of organization that unites several different commands into a single unit that can move rapidly in response to problem situations. In the ESG simulation, the decision makers are given a mission, a set of predefined mission requirements, and information about assets that they control individually. Working together, they formulate plans of action and execute those plans to accomplish the overall mission objective. Examples of simulations in the DDD environment involve humanitarian assistance, disaster relief and maritime interdiction. The simulation was designed to foster interactions among three specific positions in the ESG: Sea Combat Commander (SCC), Marine Expeditionary Unit (MEU), and Intelligence, Surveillance and Reconnaissance Commander or Coordinator (ISR). Seven three-person teams of officers participated in the study. Each team member was assigned a position (SCC, MEU, or ISR) which he or she maintained throughout the entire study. Each team participated in four two-hour sessions. The first two sessions were training sessions designed to familiarize the officers with the simulation software, the general outline of the mission, and the specifics of their own roles as decision makers. The third session, while primarily designed as a training session, was also a valuable source of data. During this session, the teams worked through two scenarios in the DDD simulation. The first scenario was a training scenario. The second was a new scenario designed to test the team’s understanding of the situation. This same second scenario was then repeated during the fourth and final session under different test conditions. Thus, we had direct comparisons between the third and fourth sessions. The fourth session was designed to be the major source of experimental data. When team members arrived for this session, they were told that many of their assets (e.g., helos, UAVs, etc.) used in the previous sessions were no longer available to them. Consequently, they faced the necessity to decide among themselves how to cover the tasks required in the mission under these reduced conditions. The most obvious way to do this was to combine their individual resources and to share tasks. Teams reached consensus about how to work together by discussing the previous scenarios they had seen and describing how they had utilized their individual assets. They then created a written plan to detail how they expected to work together and to share responsibilities. Finally, they repeated the same scenario that was used at the end of the third session and implemented their new plan of cooperation. Thus, the research design allowed direct comparison of team behavior under two conditions: autonomous task performance and coordinated task performance. Under the first condition, each team member was free to select a task objective and to pursue it without undue deliberation or constraint by the actions of other team members. Under the second condition, team members were forced to communicate their plans to

268

S.P. Marshall

the other team members so that they could allocate the necessary resources in a timely fashion. Many mission objectives required actions to be taken by two or sometimes three team members simultaneously. If one team member did not deploy a specific asset in a timely fashion, the mission objective would not be achieved. This design proved to be extremely valuable in examining how cognitive workload changed from one condition to the other. The underlying simulations were identical, thus the same events occurred at the same time and we could monitor how the teams responded to them. For each run during the third and fourth experimental sessions, all team members were monitored using the unobtrusive networked eyetracking system. Data consisted of all eye movements of each team member, pupil size for both left and right eye measured at 120 Hz, and a video overlay of eye movements on the simulation screen. Each simulation run lasted 20-30 minutes. Several unexpected problems were encountered during data collection. First, some team members assumed extreme positions to the left or right of the computer display, leaning heavily on one elbow as they looked at the screen. They were not viewable by the eyetracking cameras while they were doing so, and data were lost temporarily while they maintained this position. Second, a few of the officers were unable to read the very small print on the display and had to lean forward to within a few inches of the screen to read messages. The eyetracking cameras could not keep them in focus during these times and these data were also lost. Examples of workload results are shown in the following figures. Workload was measured by the Index of Cognitive Activity (ICA), a metric based on changes in pupil dilation [2, 5]. The ICA is computed every 30 seconds to show how workload changed over time during the scenarios. Figure 1 illustrates the difference in cumulative workload for two positions, SCC and ISR, on two different simulations, session 3 and session 4. Each graph shows the two scenarios for the SCC member of a team as well as the same two scenarios for the ISR member of the same team. Teams are identified by letter. For Teams B and D, ISR experienced higher workload than SCC throughout most of the scenario. The cumulative plots shown here begin to rise more steeply for ISR than SCC by the end of 5 minutes (10 observation points). It is interesting to note that the ISR Coordinators in Teams B and D experienced higher workload than the ISR Commander in Team G. A key objective of the study was to understand how the workload of the various team members changed when they had reduced assets and were forced to coordinate their activities. It was expected that workload would rise as assets were reduced. Figure 2 shows the results for three teams under the two conditions. Surprisingly, some SCCs had lower workload under the reduced-asset condition. This unexpected result was explained during the team’s follow-up discussion in which these officers volunteered that they had had difficulty keeping all the assets moving around efficiently under the full-asset condition. Thus, by reducing the number of assets they had to manage, they experienced lower workload even though they had to interact more with their team members.


269

Fig. 1. Cumulative workload for three teams

Fig. 2. Original Scenario versus Reduced Assets Scenario: SCC (original is solid line and reduced is dotted line)

270

S.P. Marshall

4 Assessing Cognitive State This study examined several different psychophysiological measures of task difficulty and subject fatigue. Only the eyetracking data are described here. The study was led by Dr. Judith Orasanu, NASA Ames Research Center and involved collaboration between several research groups at NASA Ames and EyeTracking, Inc. The task required 5 team members to work together to solve a series of lunar search and recovery problems. It also allowed each individual to score points, so that the individual was working not only for the good of the team but was also trying to maximize his or her own points. Multiple versions of the task were employed and were presented in the following order: Run1 (Moderate), Run2 (Difficult), Run3 (Difficult), Run4 (Moderate), Run5 (Difficult), Run6 (Moderate). Eye data were recorded for three participants during six experimental runs, with each run lasting 75 minutes. The three participants were part of a larger 5-person team who were jointly tasked with manning 4 lunar vehicles plus the base station. We eyetracked two operators of lunar vehicles (code named RED and PURPLE) as well as the base operator (BLACK). They were tested six times over the course of a 24hour period during which time they were sleep deprived. Each participant worked in a separate small room and communicated with other team members through a common view of the lunar landscape on the computer display and through headsets. The head-mounted Networked Evaluation System (NES) was used in this study, with each participant undergoing a brief calibration prior to each run. The eye data and workload were then sent in real-time to the central processing CWAD server where all data were time stamped and synchronized for subsequent analysis. A large quantity of data was collected. For each participant on the experimental task of interest, we have a total of 450 minutes of data (6 runs x 75 minutes), which is 27,000 seconds or 13,500,000 individual time points (taken every 4 msec). The data were subsequently reduced to 1-minute intervals by averaging the variables across successive 60 seconds. Seven eye-data metrics were created: Index of Cognitive Activity (ICA) for both eyes, blink rates for both eyes, fixation rates for both eyes, and vergence. All variables were transformed by the hyperbolic tangent function to produce values ranging from -1 to +1. These seven metrics have been employed successfully in the past to examine the cognitive states of individuals in diverse situations including solving math problems, driving a car (simulator), and performing laparoscopic surgery. The six runs were performed by the subjects as three sets of two runs, with each set containing a moderate run and a difficult run. The first set occurred in the first few hours of the study when the subjects were not fatigued; the second set occurred under moderate levels of fatigue; and the third set occurred during the last few hours of the study when the subjects experienced severe levels of fatigue. A patented process based on linear discriminant function analysis was carried out for each subject in each of the three sets to determine whether the eye data were sufficient for predicting task difficulty. The first analysis compared Run 1 with Run 2 to determine if the eye metrics are sufficient for distinguishing between the two levels of task difficulty. The linear


271

discriminant function analysis (LDFA) determined the linear functions that best separated the 1-minute time intervals (75 per run) into two distinct categories for each participant. Classification rates were very high, with 85%, 96%, and 100% success rates for BLACK, RED, and PURPLE respectively. The eye metrics clearly distinguish between the initial moderate and difficult scenario. It is possible to estimate from a single minute of performance whether the individual was carrying out the easier task or the more difficult one. The analysis of the middle set of runs (runs 3 and 4, made under moderate fatigue) also shows successful discrimination between the two levels of task difficulty, with success rates of 99%, 92%, and 90%. And, the analysis of the third set of runs (runs 5 and 6, made under extreme fatigue) shows similar but slightly lower success rates of 85%, 95%, and 86%. Looking across all three sets, it is evident that the eye metrics distinguish between the two levels of the scenario whether participants are alert (first set), moderately fatigued (second set), or very fatigued (third set). The lowest classification rate was 85%, meaning that the eye metrics correctly identified at least 85% of all minutes according to the scenario in which it occurred. It should be noted that all minutes of each scenario were included in these analyses, including initial minutes during which the scenarios presumably looked very similar to participants. The first set of analyses described above looked at task difficulty while holding fatigue constant. Similar analyses look at whether we can distinguish between little fatigue and extreme fatigue while holding task difficulty constant. Two analyses parallel those described above. The first fatigue analysis looked at levels of fatigue during two moderate runs. It compares the initial moderate run (Run1) with the final moderate run (Run6). The former was the run with least fatigue because it occurred first in the experimental study. The latter was presumably the run with the most fatigue because it occurred after participants had been sleep deprived for approximately 24 hours. LDFA classification rates for this analysis were 85%, 95%, and 86% for BLACK, RED, and PURPLE respectively. The second fatigue analysis looked at levels of fatigue during two difficult runs. Once again, the first difficult run (Run2) was contrasted with the final difficult run (Run5). Classifications rates here were 100%, 95%, and 100%. The eye metrics were extremely effective in detecting the difference between low and high fatigue states with near perfect classification across the all 1-minute intervals for all three participants on the challenging difficult runs. A final view of the data illustrates the importance of the Networked Evaluation System. The objective was to determine whether the participants experienced similar levels of workload during the tasks. For this analysis, it is critical that the data be synchronized so that we are comparing precisely the same time interval for every participant. Figure 3 shows the left and right ICA for the three participants across all six runs. These figures show that the ICA varies considerably within each run, peaking at various times and dropping at other times. These figures also show a dramatic impact

272

S.P. Marshall

Fig. 3. Left and right eye ICA across the entire six runs

of fatigue on the ICA (see for examples the fourth panel for BLACK, the last panel for RED and the last two panels for PURPLE). And, there are sizable differences between left and right eyes for all three participants. Each of the panels of Figure 3 could be expanded and mapped against the task details to determine what the participant was doing during periods of high and low workload. Figure 4 contains an annotated graph of the Right ICA for RED during Run1 (first panel of middle graph in Figure 3). This graph has a number of peaks and


273

valleys. Eight peaks were selected for annotation using the screen video from the eyetracking session (audio was not available). For the most part, it is possible to determine from the video what the participant was doing, i.e., working with other team members to process a seismic monitor sensor, working alone to process other sensors, or navigating across the terrain. We assumed that the many steps required to process a seismic monitor required considerable cognitive processing and that moving in a straight line across the grid required very little cognitive processing. And that is what we observed here. As Figure 4 shows, most of the spikes correspond to times that RED was processing sensors, either alone or in tandem with other team members. Most of the time when she was simply moving from one location to the other the Right ICA was descending. (Some spikes are not labeled because the video did not provide sufficient evidence alone to be sure of the task she was attempting.) Thus, we are confident that the ICA can locate time periods that are more cognitively effortful for any participant. It should be kept in mind, however, that participants could have been processing information that is neither on the screen nor spoken by the team. In such instances, we might see active processing but not be able to trace its source.

Fig. 4. Annotated History of Run1 for RED

5 Summary The Networked Evaluation System worked very well in both environments described here. During both studies, it was possible to monitor the workload of the team members in real time as they performed their tasks. An obvious extension to NES

274

S.P. Marshall

would be to create some sort of alert that can inform either the team member directly or a supervisor when levels of workload are unacceptably high or low. Another option would be to have a direct link between NES and the operating system for the task. If the team member’s workload exceeded a defined threshold, the system could reduce the demands on the team member directly without supervisor intervention. Additional studies are now planned or underway in both environments and will provide more data about how NES can be implemented in real settings. Future studies will focus on how to time stamp automatically critical events for post hoc analyses and how to better capture and display task elements that correspond to high and low workload.

References 1. Marshall, S.: Identifying cognitive state from eye metrics. Aviation, Space, & Environmental Medicine 78(5), 165–175 (2007) 2. Marshall, S.: Measures of Attention and Cognitive Effort in Tactical Decision Making. In: Cook, M., Noyes, J., Masakowski, V. (eds.) Decision Making in Complex Environments, pp. 321–332. Ashgate Publishing, Aldershot (2007) 3. Marshall, S.P.: U.S. Patent No. 6,090,051. U.S. Patent & Trademark Office, Washington, DC (2000) 4. Marshall, S.P.: U.S. Patent No. 7,344,251. U.S. Patent & Trademark Office, Washington, DC (2008) 5. Weatherhead, J., Marshall, S.: From Disparate Sensors to a Unified Gauge: Bringing Them All Together. In: Proceedings of the 1st International Conference on Augmented Cognition, Las Vegas, NV, CD-ROM (2005)

User Behavior Mining for On-Line GUI Adaptation Wei Pan1,2 , Yiqiang Chen1 , and Junfa Liu1,2 1

Institute of Computing Technology, Chinese Academy of Sciences {panwei,yqchen,liujunfa}@ict.ac.cn 2 Graduate University of Chinese Academy of Sciences

Abstract. On-Line Graphics User Interface (GUI) Adaptation technology, which can predict and highlight user’s next operation in menu based graphics interface, is the key problem in next generation pervasive human computer interaction, especially for remote control device like Wiimote assisting TV interaction. In this paper, a hierarchical Markov model is proposed for mining and predicting user’s behavior from Wiimote control sequence. The modal can be on-line updated and highlight the next possible operation and then improve the system’s usability. We setup our experiments on asking several volunteers to manipulate one real education web site and its embedded media player. The results shows our modal can make their interaction with GUI more convenient when using Wii for remote control.

1

Introduction

Pervasive Computing is a popular concept and attracting more and more computer and communication experts. [Xu Guang-You:2007] defines it as an attempt to break the pattern paradigm of the traditional relationship between users and computational services by extending the computational interface into the user’s environment, namely the 3D physical space. Graphics User Interface(GUI) is a basic interaction interface in this environment. Lots device used for simplifying and improving the interaction style has been invented recently. Take Wii Remote([nitendo], or wiimote) as an instance, which was originally designed as a TV game controller, has a great potential of becoming a popular HCI device, especially for GUI([Lee, Kim:2008]). The advantages of wiimote assisting GUI should be attributed to its appealing characteristics —wireless connection, simple operation, and human engineering shape. Moreover, the acceleration sensors equipped inside provide more useful information about motions of the user, thus it is possible to develop more intelligent functionalities based on it compared to traditional mouses, like recognizing the user’s gestures [Kela:2006] and so on. Although wiimote greatly improves the user experience, there are several limitations when it is used as a remote mouse with a simple keyboard (just six keys). Let think about the process using the controller. We first choose the input focus and press “OK”to trigger the operation. For example, we can use the V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 275–284, 2009. c Springer-Verlag Berlin Heidelberg 2009

276

W. Pan, Y. Chen, and J. Liu

device to run a media player, pause, or change the video, etc. This kind of interaction would be very useful and attractive when demonstrating in public or in class. However, the survey from volunteers indicates that some operations are not as convenient as expected. For instance, tedious operations of “tab” must be pressed until we reach the right menu we want. In this paper, we propose a solution to improve the usability of the interaction modal. We address the problem of decreasing redundant operations by predicting the user’s next possible operation in advance. While everyone has some general action habits, we aim to mine them out from history data to assist wiimote operation. We employ a hierarchy Markov to model wiimote control sequence. The predicted result using the model serve as a recommendation of next operation to the user to augment the usability and reduce unnecessary operations. We also setup our experiments on an educational web site. The rest of this paper is organized as follows. Section 2 gives out some related work. The system architecture, as well as our behavior mining model is introduced in Section 3. In section 4, we describe the experimental results and give the evaluation. We conclude the work and propose the future work in the last section.

2

Related Work

Our wiimote controlling system is a kind of intelligent user interface, which has been discussed for years and some demo systems are already in use. [Hook:2000] discussed systemically four problems to overcome on the road to intelligent user interface: – usability principle for intelligent interface – reliable and cost-efficient intelligent user interface development method – a better understanding of how and when intelligence can substantially improve the interaction – authoring tools to maintenance the intelligent part of the system There are also several works proposed to handle these challenges. [Lieberman:1995] explored a suggestion tool in web browsing, where some links that may be potentially interesting to users were provided to realize a quicker search. [Insung:2007] introduced an intelligent agent in a health-care system used for self-governing and self-customized by making wise decisions. These are early tries on intelligence interface, but not suitable for new interaction model.[Kela:2006] introducing a new convenient input style, which may be very useful in the pervasive computing environment. In this paper, We sufficiently study the character of wiimote assisting GUI and propose a solution to improve its intelligence. [Etzioni, Weld:1994] is a good example of modeling based on pre-programmed rules. Similar works can be found in [Lester:2000] and [Virvor, Kabassi:2002]. Rules are very useful, however its defect is also obvious—rules are hard to be update online. Especially, powerful rules are hard to be built for wiimote operation sequence. Considering habits may change, on-line updating is indispensable.

User Behavior Mining for On-Line GUI Adaptation

277

In this paper, we setup a hierarchical Markov model to simulate the user habits, and introduce several updating algorithm for model adaptation.

3 3.1

Proposed System Architecture System Architecture

Fig. 1 illustrates the architecture of our recommendation system, which is consisted of four main modules. The user interacts with the system through a wiimote, and then the system responds by making a recommendation about next operation, which is predicted by the behavior mining model. If the user finds the recommendation useful, he/she need to press “OK” to the next step, or choose the one needed. The system will gather the wrong recommendations automatically for further study. There is also a module called Updating Strategy, which can update our model based on the latest collected data. Fig. 2 is the operation interface of an educational web site for primer school study, and our experiments are carried out on it. The white device on the right is a picture of wiimote. 3.2

Model User Behavior

Model module is the most important part of the system in Fig. 1. Since the user’s behaviors on the web site is a sequence of click actions, taking the web

Fig. 1. Architecture of Recommendation System

278


Fig. 2. Web Site where and the device We do the Experiment

Fig. 3. Hierarchical Markov Chain

site hierarchy into account, we model it using a hierarchical Markov chain. [Meyn and Tweedie:2005] discussed Markov Model theory in detail, and [Fine:98] proposed hierarchical hidden markov model similar to the model we will discuss in this paper. Fig. 3 explain the structure of a hierarchical Markov model. Gray lines shows vertical transitions. The horizontal transitions are shown as black lines. The light gray circles are the internal states and the dark gray circles are the terminal states that returns control to the activating state. Let T represent the transition matrix. Suppose we have N statuses (operations) in the system, the size of T will be N ×N . Each element in T gives the probability of transition from one status to another. For example, the ith row gives the transition


279

probability from status i to all the other statuses. T [i][j] = k, (0 ≤ k ≤ 1) means the status i can be followed by status j with the probability k. Apply it to the wiimote operation sequence. The top node indicates the entrance to the web site, the second level gathers three nodes representing three main modules of the web site. The third level contains more detail functional nodes provided by the web site. Lines among nodes represent possible operation ways. The nodes in the same level has the lines connected between each other. Each node has some child nodes in the next level with lines connected. 3.3

On-Line Adaptation

It is apparent that new user behavior data should be utilized to improve the accuracy of our algorithm. A model with online updating will change itself with the real-world environment to be adaptive and intelligent. In our experiment we aim to update one or more transition matrixes. In this paper, we propose two alternative ways based on transition matrix: no history data, and constant number history data. The first one, we just use the latest data to update the transition matrixes. Suppose we get a new instance user behavior data, say, from node i to node j. If we successfully predict node j, we can remain T unchanged. But if next node we predict is node k, then we should update T by the formula below. T [i][k] = T [i][k] + δ T [i][j] = T [i][j] − δ

(1)

The value of δ is the critical task, which can be determined by some experiments. The second alternative method, we can preserve a constant number of history data in the system, say N , as a database. When we receive new K instances, just replace them with the oldest K instances. Then we can rebuild the model by the new data with endurable computing time.

4 4.1

Experiments Implement System

We setup our experiment on an education web site, which is a typical GUI based system, see Fig. 2. We will assign each node in the system with a unique indicator. Nodes in one level can be classified and share the same transition matrix. We make the prediction by two steps. Firstly, retrieve the user position in the system, say l. Then the system will predict the next most possible operation based on the transition matrix Tl . Updating of the model can be done based the new gathered wiimote control sequence. 4.2

Data Collection

First, we analyze the the web site and pick out all the possible operations, resulting in 72 nodes. Then we classify them into 5 classes, which means we will create 5

280


transition matrix in total. In our experiment, we employ 100 volunteers to use the education web site, each one is asked to visit the web site 20 times. And we get about 100 × 20 = 2000 groups of behavior series. We randomly choose 80% of the these data as a training data set, and the rest as the testing data set. 4.3

Experimental Results

First of all, we train the transition matrix on the training data set. Then test the transition matrix on training data set and testing data set respectively. The result of the Hierarchical Markov Model is shown in Fig.4. The blue line represents the prediction accuracy on the training data set, while the red line represents the prediction accuracy on the testing data set. It is very easy to find out that prediction accuracy on the original training data set is better than that on the testing data set. We give the mean and variance statistics in Table. 1. According to the table, the mean prediction of the testing data is nearly 70%, while the we have about 75% on the original data. That both of their variances are very small implies that the prediction result is stable. Following, we exam the first updating algorithm —updating the model with constant history data set. In our experiment, we choose 50 user’s behavior series stored in the system. Once we receive new instance of user’s behavior, we replace the oldest data with new gathered data. Model is updated based on the new history data set. Fig.5 gives the testing result. We can find that the prediction accuracy is a little better than the model without any updating. We

Predict User Behavior Using Hierarachical Markov Model Without Updating 1 Prediction Accuracy on Training Data Prediction Accuracy on Testing Data 0.9

Prediction Accurency

0.8

0.7

0.6

0.5

0.4

0

10

20

30 UserBehavior Data Series

40

50

60

Fig. 4. Test Accuracy of the Hierarchical Markov Model Without out Updating Table 1. Some Statistics of the Prediction Result over all the models Model TrainingMean TrainingVariance TestingMean TestingVariance ModelWithoutUpdating 0.7466 0.0137 0.6903 0.0225 ModelUpdatedWithConstData 0.7227 0.0197 0.7134 0.0181 ModelUpdtedWithDelta(0.2) 0.7373 0.0164 0.7129 0.0203


281

Predict User Behavior Using Hierarachical Markov Model Without Updating 1 Prediction Accuracy on Training Data Prediction Accuracy on Testing Data

0.9


0.8

0.7

0.6

0.5

0.4

0

10

20

30 UserBehavior Data Series

40

50

60

Fig. 5. Test Accuracy of the Hierarchical Markov Model With Const History Data

Mean Prediction Accurency over Delta 0.716 Mean Prediction Accurency 0.714

Mean Prediction Accurency

0.712 0.71 0.708 0.706 0.704 0.702 0.7 0.698 0.696

0

5

10

15

20

25

30

35

40

45

Delta

Fig. 6. Test Accuracy over δ

Predict User Behavior Using Hierarachical Markov Model With Fixed Delta: 0.2 1 Prediction Accuracy on Training Data Prediction Accuracy on Testing Data 0.9


0.8

0.7

0.6

0.5

0.4

0

10

20 30 40 UserBehavior Data Series

50

60

Fig. 7. Prediction Accuracy when δ is 0.2

282

W. Pan, Y. Chen, and J. Liu Testing and Training Accuracy of ELM 0.65 Prediction Accuracy on Testing Data Prediction Accuracy on Training Data 0.6


0.55

0.5

0.45

0.4

0.35

0

5

10

15

20 25 30 UserBehavior Data Series

35

40

45

50

(a) Experiment Result Using ELM Testing and Training Accuracy of Markov Model 0.78 Prediction Accuracy on Testing Data Prediction Accuracy on Training Data 0.76


0.74

0.72

0.7

0.68

0.66

0.64

0

10

20 30 UserBehavior Data Series

40

50

(b) Experiment Result Using Hierarchical Markov Model

Fig. 8. Comparison Between ELM([G.-B. Huang:2006]) and Hierarchical Markov Model

also notice that prediction on some instance is rather low, even below 50%. It will be discussed in Section 5. Another updating algorithm seems to be quite simple, for it need not recompute all the history data set and just update the model by the newest received data. In this method, one of the core problem is how to decide δ. Fig.6 shows the result of testing δ from 0.1 to 0.3 with a step of 0.055. According to the figure, there will be a higher prediction accuracy when δ ∈ 0.1 ∼ 0.2. Larger or smaller value will cause over-computing and decrease the prediction accuracy. In Fig.7, we choose δ = 0.2 to make a model according to the third solution. Table. 1 gives all results of these models. All of them have the mean accuracy about 70%, and the variances are also quite small(smaller than 0.03). In this viewpoint, we may come to a conclusion that the original hierarchical Markov


283

model is good enough, and other improvement tries make little contribution to the prediction result. Compared to other models, such as neuron network, hierarchical Markov model is more suitable for the user behavior model. Fig.8 gives a typical comparison. It is obvious that our model (72%) is much better than neuron network (35%). This result should be ascribed to the similarity between the structure of wiimote operation sequence and hierarchical markov model, which is the weak point of neuron network.

5

Conclusion and Future Work

In our experiment, we ask the volunteers some questions, such as wether it improve the usability, whether we bring in new inconvenience in operation. Most of them think the recommendation provided by the system is helpful. However, some argue that, if they do not operate normally, the prediction is often wrong. It is because the system is trained for normal user behaviors, and once it encounters some abnormal operation series, the reaction would not meet the user’s requirement. One of the solutions is provide two or more recommendation options. Another question is that which recommendation style is appreciate. Here we skip the input focus into the recommendation operation menu item. Most of the volunteers think this is helpful, and some of them suggest using pop-up dialogs to provide two or more recommendation. These advices are useful for improving our system in the future. Devices like wiimote empower GUI. The model with on-line adaptation we setup provides a solution of intelligence interface, which is the main character of next generation pervasive computing. It greatly enhances the user operation experience. The future of pervasive computing HCI should absorb advances from such exciting tries.

Acknowledgements We would like to thank Juan Liu for data collection and all of volunteers. This work is support of the National HighTechnology Research and Development Program(“863”Program) of China(Grant No.2007AA01Z305) and National Natural Science Funds(Grant No.60775027).

References [Kallio:2006]

[Crampton:2007]

Kallio, S., Kela, J., et al.: User independent gesture interaction for small handheld devices. International Journal of Pattern Recognition and Artificial Intelligence 20(4), 505–524 (2006) Crampton, N., Fox, K., et al.: Dance, Dance Evolution.:Accelerometer Sensor Networks as Input to Video Games. Haptic, Audio and Visual Environments and Games (2007)

284


[Kela:2006]

[Hook:2000]

[Insung:2007] [Lester:2000]

[Lieberman:1995] [Virvor, Kabassi:2002]

[Etzioni, Weld:1994] [nitendo] [Lee, Kim:2008]

[Xu Guang-You:2007] [Meyn and Tweedie:2005] [G.-B. Huang:2006]

[Fine:98]

Kela, J., Korpipaa, P., et al.: Accelerometer-based gesture control for a design environment. Personal and Ubiquitous Computing 10(5), 285–299 (2006) Hook, K.: Steps to take before intelligent user interfaces become real. Interacting with Computers 12(4), 409–426 (2000) Insung Jung, D.T., Wang, G.-N.: Intelligent Agent Based Graphic User Interface (GUI) for e-Physician (2007) Lester, W.L., Johnson, J.W., Rickel, J.C.: Animated Pedagogical Agents: Face-to-Face Interaction in Interactive Learning Environments (2000) Lieberman, H.: Letizia: An Agent That Assists Web Browsing (1995) Virvou, M., Kabassi, K.: Rendering the interaction more human-like in an intelligent GUI. Information Technology Interfaces (2002) Etzioni, O., Weld, D.: A Softbot-Based Interface to the Internet. Communications of the ACM (1994) http://www.nintendo.com Lee, H.-J., Kim, H., et al.: WiiArts:Creating collaborative art experience with WiiRemote interaction. In: Proceeding of the Second International Conference on Tangible and Embedded Interaction (2008) Guang-You, X., Li-Mi, T., et al.: Human Computer Interaction for Ubiquitous/Pervasive Computing Mode (2007) Meyn, S.P., Tweedie, R.L.: Markov Chains and Stochastic Stability. Cambridge University Press, Cambridge (2005) Huang, G.-B., Zhu, Q.-Y., Siew, C.-K.: Extreme Learning Machine: Theory and Applications. Neurocomputing 70, 489–501 (2006) Fine, S., Singer, Y., Tishby, N.: The hierarchical hidden markov model:Analysis and applications, 41–62 (1998)

Modeling Human Actors in an Intelligent Automated Warehouse Davy Preuveneers and Yolande Berbers Department of Computer Science, K.U. Leuven Celestijnenlaan 200A, B-3001 Leuven, Belgium {Davy.Preuveneers,Yolande.Berbers}@cs.kuleuven.be

Abstract. Warehouse automation has progressed at a rapid pace over the last decade. While the tendency has been to implement fully automated solutions, most warehouses today exist as a mixture of manually operated and fully automated material handling sections. In such a hybrid warehouse, men and machines move around goods in between sections in order to retrieve, transport and stack goods according to their nature and quantity. The biggest challenge in hybrid warehouses is to optimize the alignment of manual and automatic processes in order to improve the flow of materials between storage areas and distribution centers. Integrating individuals as human actors in an automation solution is not straightforward due to unpredictable human behavior. In this paper, we will investigate how we can model the characteristics of human actors within an automation solution and how software systems can unify human actors with automated business processes to coordinate both as first class entities for logistics activities within a hybrid warehouse.

1 Introduction Warehouses come in different sizes and shapes, but they are all used for the receipt, storage, retrieval and timely dispatch of a variety of goods. To ensure that productivity targets are met and to maximize the manufacturing floor space, warehouse managers often rely for repetitive material handling processes on automatic guided vehicles (AGVs) [1], automated storage and retrieval systems, and on conveyor and sorting systems. For other tasks that require certain creativity, human actors are indispensable. In hybrid warehouses, manual and automated material handling processes are intertwined. Within a hybrid warehouse, it is possible that for a single purchase order, goods from different storage areas need to be collected and consolidated. Fig. 1 shows a manual task that is assigned to a human actor, in this case an individual driving a fork lift to transport goods to their designated area. Fig. 2 shows an example of an automated storage and retrieval system. To ensure a smooth product flow within the warehouse, manual and automated material handling processes need to be properly aligned. However, for warehouse managers it is not evident to integrate human actors into an automation process, because human behavior is far from being predictable and people make mistakes more easily. To circumvent problems that may arise during the harmonization or integration of human actors within automated warehouse systems, V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 285–294, 2009. © Springer-Verlag Berlin Heidelberg 2009

286

D. Preuveneers and Y. Berbers

Fig. 1. Human-driven material handling

Fig. 2. Automated storage and retrieval

warehouse managers often decouple the warehouse into a manually operated and a fully automated storage area. Because automated and manual material handling processes are fundamentally different, their supporting software systems - the Warehouse Management System (WMS) - are often developed with a different background. In many cases both types of warehouse management systems independently do location allocation and transport planning [2] for the goods within their area. Integration is often limited to a high-level coupling between both systems at the Enterprise Resource Planning (ERP) level. Because synchronizing manual and automated systems is hard, hybrid warehouses suffer from significant inefficiencies and suboptimal throughput of the hybrid warehouse due to the extra buffers that are often introduced as a workaround to deal with this performance loss. A global approach to an intelligent hybrid warehouse where manual and automated processes are considered and optimized as a whole could lead to an improvement of the Total Cost of Ownership (TCO) of a hybrid warehouse. The fundamental problem that needs to be addressed to achieve this goal is the lack of modeling and software support to incorporate human actors as first class entities within a warehouse automation process. In this paper we will first investigate how we can model human behavior by identifying the characteristics of a human actor within logistics systems, which tasks human actors fulfill and which properties are of importance. Secondly, we will investigate how we can explicitly model expectation patterns for more complex jobs in order to address the possibility of and the response to unexpected human behavior. A last aspect we address in this paper is a mapping of this human behavior model on a software architecture to fully support human actors within a hybrid warehouse management system. In section 2, we discuss the role and characteristics of a human actor in a hybrid warehouse. We present our modeling support for human actors in a material handling

Modeling Human Actors in an Intelligent Automated Warehouse

287

process in section 3. We discuss our initial mapping of this model onto a software system in section 4. An overview of our contributions, the conclusion and opportunities for future work are presented in section 5.

2 Human Actors in a Hybrid Warehouse In a hybrid warehouse, some of the material handling processes can be automated while others have to be carried out manually. For both types of activities a Warehouse Management System will assign a specific logic [3] to the various combinations of order, items, quantities and locations. Such a logic either optimizes space utilization (e.g. pick-to-clear logic), number of movements (e.g. fewest locations logic), travel times (e.g. nearest location logic), etc. For any of these transportation logics, the consolidation of a customer order would result in a sequence of transport operations of an amount of products at a given location. For larger orders, multiple human actors and automated systems can work on the same task list, which may require some synchronization between the different transport activities. But while automated storage and retrieval systems are capable of storing and consolidating goods at a given and fixed rate, the rate at which human actors can transfer goods is less predictable. In brief, integrating human actors in a hybrid warehouse raises a few concerns:

Choice versus Time Constraints: Human actors have some autonomy to handle more complex jobs, because some decisions are better made by people due to their flexibility, intuition and wisdom. To reduce the duration variability of such jobs, we need to balance the number of options given to human actors and the processing time to complete the job. Indeterminism: Human actors can behave in unforeseen ways, such as performing tasks out of order according to the sequence of tasks they were assigned. The challenge here is to monitor the overall effect of human tasks and to take decisions whenever human actors do not behave as expected. Roles and Responsibilities: The role and responsibilities of human actors participating in a process help to define interchangeable human resources. We must also describe how the different human actors can collaborate and synchronize with each other while the material handling process progresses.

2.1 Choice versus Time Constraints Human actors can introduce unexpected delays in the material handling process. The main reason is that human actors usually perform their tasks slower than automated facilities, but also because they make mistakes more easily. For example, a fork lift driver may waste time if he needs to go and find goods mistakenly placed elsewhere if a given pellet turns out to have fewer items than expected. However, one cannot automate every single task, either because they are too complex to automate, or they require human expertise or intuition to handle a particular product. The main concern here is to find a way to minimize the human impact on the overall processing time. A first alienating approach transforms human actors into robots: (1) reduce the number of possible actions to a minimum; (2) make sure that human actors do not have to take

288


any decision and that they always know what to do next. Unfortunately, this would result in an inflexible sequence of tasks to be executed within a fixed time frame without the ability for an operator to solve problems (e.g. broken vehicles, missing items or other unexpected delays) without jeopardizing the order of the task list. Instead, to maintain a certain level of job satisfaction, we want to give human actors some autonomy to make choices and take decisions, and investigate how dynamic decision models [4] to help estimate delays in order to take appropriate actions when deadlines expire, such as exception handling, sending reminders. Helping human actors to prioritize their task is another approach to reduce these delays. In brief, we have to find a balance between the autonomy of human actors making choices and the delays this autonomy brings. 2.2 Indeterminism Autonomy does not only bring delay, it also brings indeterminism. Indeterminism means that given the same initial conditions a human actor does not always behave in the same way. He may perform unexpected actions or carry out actions in an unexpected order. For example, some operators may cancel a picking job if a pellet contains fewer items than advertised, while others may suspend the job, replenish the good from a reserve storage location, and then resume the original job with picking the number of items needed. Since we want to give autonomy to the human actors, we cannot prevent this indeterminism at all times. However, we can try to prevent it as much as possible, and try to compensate its effects when we cannot avoid it. Preventing indeterminism means that we must describe the allowed degrees of freedom a human operator has to handle a batch of tasks. As a result, we must monitor such tasks assigned to human actors and detect approaching deadlines and expired deadlines. Reminders can be sent when a deadline is approaching. Escalation [5] can be triggered when a deadline expires or when an error or exception occurs. Escalation means that a person with a higher level of responsibility is notified that a deadline expired. Escalation may also transfer the responsibility for a task to another operator. 2.3 Roles and Responsibilities If we want to integrate human actors in automated business processes we must be able to define the roles and the responsibilities of each actor that participates in a process. A high-level overview of material handling activities in a warehouse business process is shown in Fig. 3. Some of these activities can be carried out by both automated systems and by individuals. Roles and responsibilities are assigned to groups for each of these material handling activities. A business process usually involves several participants, some of which may be human actors, others can be automated systems. For example, in a warehouse, there may be groups for truck drivers, automatic guided vehicles, fork lift drivers, packers, conveyors, order managers, automated storage and retrieval systems, etc. Additionally to the roles of their respective groups, human actors can be granted other roles to help define interchangeability of human and systems resources within a warehouse business process.


289

Fig. 3. Simplified schematic overview of material handling activities

3 Modeling Human Behavior and Expectation Patterns In recent years many researchers have proposed various task models [6-8] to describe human activities and related aspects for various domains. Our focus is more oriented to business process modeling. Although there are many business modeling methods, no well established modeling standard is available in the area of hybrid warehouses. Our aim is to design a model that can be easily mapped onto business process standardization efforts in the area of integrating people in service oriented architectures. The basic concepts of our model are depicted in a schematic way in Fig. 4. 3.1 Human Transportation Tasks and Activities Human transportation tasks (picking, replenishing, putting away, etc.) within a hybrid warehouse have a life cycle with states that are independent of the logic that the WMS uses to decide exactly which location to pick from, replenish from/to, and putaway to, and in what sequence these tasks should occur. The life-cycle can be described with the following states:

Unclaimed: The transportation task is available for designation Claimed: The transportation task is assigned to an individual Started: The transportation task is in progress Finished: The transportation task has finished Error: The individual provided fault data and failed the task

In fact, these states are typical for transportation tasks for both humans and automated systems. However, humans are often also involved in other activities that cannot be classified as transportation tasks which have been fully planned in advance: • Wait or Delay tasks: This task represents a lack of activity. One has to wait until a certain condition with respect to the product flow is met. This task can be unplanned or planned. For example, a truck driver may have to wait for a confirmation of an order manager. Finding a realistic distribution of the time of delay of this task is fundamental. • Off-tasks: Off-tasks are typically human and are not related to the product flow or the material handling processes. Such tasks may include having a coffee, responding to a telephone call, going to the bath room, etc. It is hard to estimate if and

290


when these tasks take place, because their occurrence is often unknown in advance, but they can be modeled as delay tasks with possible zero delay. • Escalation tasks: This task does relate to the product flow. If a start or a completion deadline of an ordinary transportation is missed or an error occurs, this may trigger one or more escalation actions that, for example, reassign the transportation task to another participant or handle the exception. • Compensation tasks: This task undoes the effects of partially completed activities to ensure that an order is either fully completed or not carried out at all.

Fig. 4. UML class diagram of the basic concepts of the model

Being able to accurately represent them in a human behavior model is fundamental when manual and automated processes are considered and optimized as a whole. For transportation tasks, we will focus on order picking because a warehouse generally has more outbound transactions than inbound transactions, so being able to quickly and accurately process customer orders is essential to increase customer satisfaction. However, conceptually there is not much difference with the other transport tasks (putting, away, replenishing, cross-docking, etc.).


291

3.2 Defining and Modeling Expectation Patterns People are capable of juggling many tasks at once. This flexible behavior of humans is an advantage for hybrid warehouses, but the disadvantage of multitasking is that people are not interchangeable resources the way automatic guided vehicles and automated storage and retrieval systems are. When humans are in control of certain material handling processes, a single individual can be assigned a set of tasks, or multiple individuals can work in parallel on a single task. The order in which these tasks are executed can matter if we want to reduce delays in the material handling process. In order to model how a collection of transportation tasks are expected to be executed and synchronized, we need to formalize how one task can relate to another. We will define these structured tasks with expectation patterns that describe how a Warehouse Management System would expect a collection of tasks to be executed (see Fig. 5): • Sequence: A Sequence pattern expresses a collection of tasks that is to be performed sequentially and with a specific order. • Spawn: All transportation tasks are executed concurrently. The Spawn pattern completes as soon as all the tasks have been scheduled for execution. • Spawn-Merge: All tasks are executed concurrently with barrier synchronization to ensure that tasks are not executed out of order by different participants. I.e. the Spawn-Merge pattern completes if all tasks have completed. • Any-Order: The Any-Order pattern is used when the order of the tasks is of no importance as long as they do not overlap. The Any-Order pattern completes when all tasks have completed.

Fig. 5. Modeling expectation patterns for picking tasks

292


Obviously these patterns can be combined. For example, the Spawn and SpawnMerge patterns can be combined to define tasks with partial synchronization. In a following section, we will provide an example how this could be used to align manually operated and fully automated transportation tasks. However, the same patterns can be used to model constraints between different customer orders. For example, if different orders require the same type of product, and relevant pellets can only be accessed by one human fork lift operator at a time, then it is best to describe these picking tasks with the Any Order pattern, so that these tasks are not executed in parallel. Explicitly modeling these constraints helps to identify delays in the workflow more easily. Other control flow constructs, such as If-Then-Else and Iteration (not shown in Fig. 5) are used to support conditional execution of tasks and repeated execution of a task in a structured loop.

4 Example Scenario and Implementation To incorporate human actors as first class entities in an automation solution, it is important that already during the modeling of the automation process the role of the human actor can be correctly described. See Fig. 6 for an example scenario of aligning human and automated transportation tasks. Each customer order is translated into a set of transportation tasks (A, B, C, D, E and F) that are either carried out by either human or automated operators. Each human task is carried out by someone with a certain role or responsibility. Human tasks B and C could be collecting in parallel smaller items that are combined into a Spawn-Merge pattern, which is synchronized at task F that could be delivering these items at a designated drop zone for shipping to the client. This pattern is combined with tasks A and F into a Sequence pattern. Because this sequence of tasks is aligned with a sequence of automated transportation tasks D and E (of which the completion time can be accurately estimated), the last human task F in the first sequence has a start/end deadline attached to it with an Escalation task that is activated when the deadline is not met. One of the results could be the activation of a compensating task. For example, if task C would be the picking of hazardous products or goods that can perish, the original transportation task may have to be undone to store the goods again at a place where they can be preserved safely. The proposed model is kept simple in order to keep it intuitive for both technical users and business users, but also to simplify the mapping onto software systems that monitor these tasks. For the implementation of the different types of tasks and the expectation patterns in model, we map our constructs to similar representations within the Business Process Modeling Notation (BPMN) [9]. BPMN already proposes a generic graphical solution to model tasks, events and workflows with diagrams, but is however more complex and less intuitive than the model we proposed and lacks a few concepts to easily model warehouse related aspects of a task. The reason for this approach is that we can leverage software tools that can map BPMN to software systems that assist with the monitoring and the coordination of these tasks. We use techniques similar to those described in [10] in order to transform BPMN process models to Business Process Execution Language (BPEL) web services. The main advantage of mapping a workflow of business processes to an equivalent workflow of software service is, whenever a warehouse manager changes its business process, he just needs


293

Fig. 6. A simple scenario of aligned human and automated transportation tasks

to adapt parameters within our model and the necessary translations to BPMN and BPEL will happen accordingly. BPEL has earned it merits in service oriented architectures which try to uncouple software services from one to the other. For warehouses this would mean that it would be easier to change the process and product flows, but this still needs to be investigated.

5 Conclusions and Future Work The model presented in this paper arose from the real need to be able to integrate individuals as human actors into the product workflow of a warehouse and being able to deal with unpredictable human behavior. Therefore, the focus of our model was more oriented to mapping human actors within established business process practices, rather than focusing on theoretical aspects of task models in general. As a result, we have proposed a simple but intuitive dedicated model that captures several characteristics of transportation tasks, and that addresses the key concerns on choice vs. time constraints, indeterminism in the task flow, and role and responsibilities of each participant in the material handling process. Modeling concepts are included to describe the state and other properties of a transport task and how such a task can relate to nontransportation tasks. In order to better align human tasks with tasks carried out by automated systems, we included concepts to express how a batch of tasks is expected to be executed. This is important whenever multiple individuals and automated systems need to synchronize their activities while completing the consolidation of a single customer order. Nonetheless, some of the aspects need to be further investigated. For example, it is currently unclear how to best model and coordinate escalation tasks when both human actors and automated systems are involved (to make sure that escalation activities will not fail on their own that easily as well). We intend to continue our efforts on leveraging results achieved in the web services community, especially for integrating human tasks in web services orchestrations where two complementary standards BPEL4People [11] and WS-Human Task [12] have been

294


proposed. The specifications are evaluated in [13-14]. Some of the observations were that both proposals provide a broad range of ways in which human resources can be represented and grouped, that there are a number of distinct ways in which manual tasks undertaken by human resources can be implemented, but that shortcomings do exist, for example, to enforce separation of duty constraints in BPEL4People processes. We will investigate how these specifications can be used or augmented specifically for the coordination of activities within hybrid warehouses.

References 1. Burkard, R.E., Fruhwirth, B., Rote, G.: Vehicle Routing in an Automated Warehouse Analysis and Optimization. Technical Report 174, Graz (1991) 2. Lashine, S.H., Fattouh, M., Issa, A.: Location/allocation and routing decisions in supply chain network design. Journal of Modelling in Management 1, 173–183 (2006) 3. Piasecki, D.: Warehouse Management Systems (WMS) (2006) 4. Diederich, A.: Dynamic Stochastic Models for Decision Making under Time Constraints. Journal of Mathematical Psychology 41, 260–274 (1997) 5. Panagos, E., Rabinovich, M.: Escalations in workflow management systems (1996) 6. Verpoorten, K., Luyten, K., Coninx, K.: Task-Based Prediction of Interaction Patterns for Ambient Intelligence Environments. In: [15], pp. 1216–1225 7. Giersich, M., Forbrig, P., Fuchs, G., Kirste, T., Reichart, D., Schumann, H.: Towards an Integrated Approach for Task Modeling and Human Behavior Recognition. In: [15], pp. 1109–1118. 8. Winckler, M., Johnson, H., Palanque, P.A. (eds.): TAMODIA 2007. LNCS, vol. 4849. Springer, Heidelberg (2007) 9. Wohed, P., van der Aalst, W., Dumas, M., Hofstede, A.T., Russell, N.: On the Suitability of BPMN for Business Process Modelling. In: Dustdar, S., Fiadeiro, J.L., Sheth, A.P. (eds.) BPM 2006. LNCS, vol. 4102, pp. 161–176. Springer, Heidelberg (2006) 10. Ouyang, C., Dumas, M., van der Aalst Ter, W.M.P.: From BPMN Process Models to BPEL Web Services, pp. 285–292 (2006) 11. Agrawal, et al.: WS-BPEL Extension for People (BPEL4People), Version 1.0 (2007) 12. Agrawal, et al.: Web Services Human Task (WS-HumanTask), Version 1.0 (2007) 13. Russell, N., van der Aals, W.M.: Evaluation of the BPEL4People and WS-HumanTask Extensions to WS-BPEL 2.0 using the Workflow Resource Patterns. Technical Report BPM07-10 (2007) 14. Mendling, J., Ploesser, K., Strembeck, M.: Specifying Separation of Duty Constraints in BPEL4People Processes. In: Business Information Systems, 11th International Conference, BIS 2008, Innsbruck, Austria, pp. 273–284. Springer, Heidelberg (2008) 15. Jacko, J.A. (ed.): HCI 2007. LNCS, vol. 4550. Springer, Heidelberg (2007)

Bridging the Gap between HCI and DHM: The Modeling of Spatial Awareness within a Cognitive Architecture Bryan Robbins1,2, Daniel Carruth1, and Alexander Morais1,2 1

Human Factors and Ergonomics Group, Center for Advanced Vehicular Systems, Box 5405, Mississippi State, MS 39762-5405 2 Department of Computer Science and Engineering, Mississippi State University, Box 9637, Mississippi State, MS 39762-9637 {bryanr,dwc2,amorais}@cavs.msstate.edu

Abstract. In multiple investigations of human performance on natural tasks in three-dimensional (3D) environments, we have found that a sense of space is necessary for accurate modeling of human perception and motor planning. In previous work, we developed ACT-R/DHM, a modification of the ACT-R cognitive architecture with specific extensions for integration with 3D environments. ACT-R/DHM could leverage existing extensions from the ACTR community that implement the spatial sense, but current research seems to indicate that an “egocentric-first” approach is most appropriate. We describe the implementation of a custom spatial module in ACT-R/DHM, which allows for the consideration of spatial locations by adding a single ACT-R module that performs a very small set of operations on existing location information. We demonstrate the use of the 3D, egocentric-first spatial module to simulate a machine interaction task. Keywords: Digital Human Modeling, Human Performance Modeling, Spatial Cognition, Cognitive Modeling, Cognio simulatetive Architecture, ACTR/DHM, ACT-R.

1 Introduction The interdisciplinary field of Digital Human Modeling (DHM) has much to gain from integration efforts. As DHM research continues to realize the need for the simulation of human cognition, cognitive architectures, as first defined by Newell [1] and now implemented by many [2, 3, 4] seem to be a logical next step in integration efforts. However, many cognitive architectures, because of their heritage in Human-Computer Interaction (HCI) research, provide only marginal support for the consideration of the three-dimensional (3D) virtual environments common in DHM applications. The consideration of the human sense of space (or “spatial sense”) is critical in DHM applications, but does not play a vital role in HCI, and thus is not a strong component of existing cognitive modeling architectures. In previous work [5, 6], the ACT-R cognitive architecture [2] has been extended for use with DHM research as ACT-R/DHM. The goal of ACT-R/DHM is to leverage V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 295–304, 2009. © Springer-Verlag Berlin Heidelberg 2009

296

B. Robbins, D. Carruth, and A. Morais

the ACT-R modeling architecture’s theory of cognition and its decades of development, improvement, and validation for the purposes of DHM research by adding theory-based and architecturally consistent extensions. To-date, ACT-R/DHM has extended the existing visual and motor modules of ACT-R and added a kinesthetic and proprioceptive (KP) module in addition to the spatial module described herein. 1.1 The ACT-R Theory and Implementation Before elaborating on the implementation of the spatial sense and other extensions of ACT-R/DHM, we describe the original ACT-R theory [2] and its implementation in the ACT-R 6 software. ACT-R’s model of knowledge is based on a separation between declarative memory (facts) and procedural memory (actions). Models of human sensory and perception systems convert features of the external environment to an internal representation suitable for processing. Strictly typed “chunks” of information serve as the basic building block for ACT-R’s internal representation. Chunks of declarative memory are manipulated by action elements of rules in procedural memory. The central core of the ACT-R software implements a small number of critical cognitive functions. Additional modules supplement the core with memory, perceptual, and motor capabilities. ACT-R is implemented as a production system, with procedural memory elements constructed by the modeler as If-Then rules called productions. Execution of productions is accomplished by matching the “If”, or LeftHand Side (LHS), of the production against the current state of the modules and, if a match is found, executing the “then”, or Right-Hand Side (RHS) of the production. The modular construction of the ACT-R architecture allows for the extension of existing capabilities and the creation of new modules. The theoretical concepts underpinning the ACT-R architecture are enforced by the interface between the core and the modules. Important architectural constructs include the modules, the module buffers, the module requests and the chunk. The module itself, implemented as a LISP object, encapsulates the model of the system being represented. For example, the vision module simulates human vision, sensory memory, feature integration, attention and any other aspects associated with human vision, within a single module. A module’s buffer(s) makes available the current state of its module providing a window to the module environment. Module requests provide mechanisms for updating the module’s state via productions. Finally, chunks, as the basic building block of the architecture, hold declarative information in a strictly defined type, known as the chunk type. As mentioned, the constructs of ACT-R are more than implementation considerations – they enforce the underlying ACT-R theory. Any new capability added to ACT-R, including the extensions in ACT-R/DHM, must follow the required structure. If an extension deviates from the architectural standards, it gains little from ACT-R’s well established psychological validity. For this reason, we describe the extensions of ACT-R/DHM in terms of the modules, buffers, requests, and chunks affected.

Bridging the Gap between HCI and DHM: The Modeling of Spatial Awareness

297

1.2 ACT-R/DHM - Current Implementation ACT-R/DHM, prior to the development of a spatial module, extended ACT-R in a number of ways. First and most importantly, the vision module of ACT-R, which has primarily been used for stimuli presented on a two-dimensional (2D) computer screen, was expanded to consider 3D space. No additional module requests or buffers were necessary, but instead of storing flat (X, Y) coordinates, new visual chunks were derived from the original structures that encode spherical coordinates: pitch, azimuth and distance. The pitch, azimuth, and distance, or “PAD”, encoding is also reflected in the spatial module and elsewhere in ACT-R/DHM as a consistent representation of space. In addition, ACT-R/DHM includes a kinesthetic and proprioceptive (KP) module that allows for the consideration of avatar movements and body part locations in a cognitive model. The KP module adds a single buffer, “kp”, that holds the current state, (position and movement), for a single, attended body part. The kp position representation is consistent with the PAD from the vision module, spatial module, and other modules, as mentioned above. The movement state simply indicates whether or not the body part is currently in motion. Elaboration on the details of the KP module’s implementation is outside the scope of this paper. 1.3 The Need for Spatial Functions Relatively simple DHM task models clearly illustrate the need for spatial functionality in the majority of DHM scenarios. The following example is based on previous efforts [6, 7] using ACT-R/DHM with a wrapper interfacing to the Virtools™ 3D Virtual Environment and the Santos™ avatar. Santos™ is a digital human model developed at the Center for Computer Aided Design at the University of Iowa for the Virtual Soldier Research Project [8]. Santos includes a full-body avatar, with a skeleton and a posture prediction algorithm that serves as the basis for KP body part and movement information. The virtual environment provides the remaining environmental feature information (i.e. visual feature descriptions) to ACT-R/DHM. Figure 1 shows the virtual environment setup for a vending machine interaction task [6]. Participants were given 10 coins to purchase a beverage of their choice. This task involved a series of human-machine interactions in a large visual field. Participants must learn the layout of the interface, deposit their coins, choose from opions visually presented via labels, select their drink, and retrieve their drink from the machine. As the model performs the physical motions necessary to accomplish the task, the head moves and key features of the interface drop in and out of the visual field. In Figure 1a, many of the key features are not visible when the avatar is standing before the machine (initial position). In this position, the model must have some mechanism to access the current egocentric spatial position of the target feature in order to shift the view towards the target. Figure 1b shows one view encountered during interaction with the machine, and is filled with a number of critical machineinteraction features, such as buttons, the coin deposit, and other features. To accomplish tasks without repeatedly scanning for visual features any time they drop out of the current visual field, spatial memory must be available during task performance and is critical in the most basic of real-world maneuvers. To

298


Fig. 1. The availability of visual features changes as body movements are made in a simple machine interaction task. The initial view (a) contains no objects, but reliable motor planning is possible via spatial cognition. Another view (b) may have a completely different set of features available.

appropriately model human-machine interaction, DHM cognitive architectures must include models of spatial cognition. A significant obstacle to successful modeling of spatial cognition is accounting for storage, processing and recall of spatial knowledge derived from egocentric sensory data. Below, we consider the psychological evidence for spatial awareness and existing ACT-R modeling approaches before arriving at the egocentric-first spatial module now implemented in ACT-R/DHM.

2 Modeling Spatial Cognition We propose two limitations on the implementation of the spatial module of ACTR/DHM. First, the new implementation should be based on current theory of human spatial cognition. Second, the implementation should conform to current ACT-R theory and implementation framework. 2.1 Spatial Theory There exists significant debate in the spatial cognition literature regarding the nature of spatial representations and processes. Our current implementation draws primarily from the theory of McNamara [8], as supported by Rock [9] and Sholl and Nolin [10]. The following theorems integrate concepts from all three authors’ theories. Theorem 1, expressed in the work of McNamara [8] and Sholl and Nolin [10], is that human spatial competence depends on both egocentric (first-person) and allocentric (third-person) representations. Sholl and Nolin define the egocentric representations as “self-to-object” relationships and the allocentric representations as “object-to-object” relationships. Theorem 2, elaborated by Sholl and Nolin, states that human spatial competence requires the interaction of the egocentric and allocentric systems. Fundamentally, the egocentric system is based on instantaneous experiences. Theorem 3 states that all spatial relationships must begin as egocentric relationships because they are derived from egocentric percepts. We also find, however, that


299

egocentric representations are generally inadequate for reuse. Once the egocentric origin moves in the environment (e.g. head or other body movement in the world), previously encoded egocentric relationships are no longer valid for recall. Rather than postulate a continuous update of spatial memory, we opt to implement Theorem 4: spatial relationships are stored as allocentric object-to-object relationships. McNamara considers the storage of spatial relationships in memory [8]. He outlines a theory that emphasizes the role of orientation and frames of reference in spatial encoding. He appeals to the work of Rock [9] regarding orientation detection of new objects, which states that a variety of cues lead to the ultimate determination of object orientation, and that an object’s encoded orientation depends on many factors, including its environment, intended use, shape, and others. McNamara argues that some persistent cues, which he calls environmental cues, are present across multiple egocentric views, and provide a critical link between egocentric views. We extend this idea as Theorem 5, which states that all spatial relationships require a frame of reference and an orientation, and the orientation must be expressed relative to the frame of reference. Note that Theorem 5 applies to egocentric relationships which use the origin of the spatial system as a frame of reference and a native orientation for each object in the egocentric view. Any object may provide a frame of reference for any other object in an object-to-object relationship. However, certain environmental objects are more likely to be used as referents due to location, size, saliency, permanence, etc. No claim is made as to how actual detection and/or recognition of an object’s native or intrinsic orientation should be modeled. Theorems 1-5 summarize fundamental building blocks of human spatial cognition. Thus, any implementation of the spatial sense in ACT-R/DHM should hold to the theoretical claims of these theorems. 2.2 Other ACT-R Implementations of Spatial Cognition Previous efforts have modeled the spatial sense within ACT-R implementations. Before developing a custom module for the spatial sense in ACT-R/DHM, we considered existing extensions from the ACT-R community. Gunzelmann and Lyon offer a spatial implementation that covers many of the elements of spatial theory identified in section 2.1 [11]. Specifically, they propose adding three buffers to the visual system: an egocentric buffer which holds 3D, egocentric location information, an environmental frame of reference buffer that tracks a frame of reference, as suggested by Theorem 5, and an episodic buffer integrates the egocentric and frame of reference information with existing visual information as needed. Gunzelmann and Lyon also propose a spatial module that makes spatial information accessible across multiple ACT-R modules and provides spatial processing for mental transformations, magnitude estimations, and magnitude calculations [11]. Additional ACT-R implementations have been proposed by Best and Lebiere [12] and Harrison and Schunn [13]. Best and Lebiere’s implementation, designed for the development of intelligent agents, preprocesses the environment to directly provide an allocentric representation to the agent. Harrison and Schunn implement a “configural” buffer that associates a visual object with its orientation, then a system to update up to three “behaviorally significant” egocentric relationships based on the direction of body motion (e.g. walking). No persistent allocentric representation is used.

300


To summarize, our implementation of spatial cognition in ACT-R differs from other current efforts either in underlying theoretical claims (i.e. Harrison and Schunn and Best and Lebiere) or architectural implementation (i.e. Gunzelmann and Lyon).

3 The ACT-R/DHM Spatial Module The following section describes ACT-R/DHM’s spatial module, and how this module is used to support modeling of spatial cognition. 3.1 Module Implementation Previous ACT-R/DHM work extended the vision module of ACT-R to support PAD encoding. The spatial implementation requires a single new module, simply named the spatial module. The module provides only one buffer to the environment, also named the spatial buffer. The spatial buffer should only hold chunks of the type spatial-relationship, and only three module requests are provided as operations on spatial-relationship chunks: ego-relate, chain-relate, and mid-relate. The spatial module has no member data of its own and derives from ACT-R’s generic module class. The spatial-relationship chunk type and the module requests of the ACTR/DHM spatial module capture many of the The spatial-relationship chunk type is detailed in Table 1. ACT-R chunks hold smaller pieces of information in slots. The slots of the spatial-relationship chunk hold a frame of reference in the reference slot, an object in the object slot, the position of the object in the pitch, azimuth, and distance slots, and finally the orientation of the object as three axis vectors, <xdir, ydir, zdir>, <xup, yup, zup>, and <xright, yright, zright>. The position and orientation are relative to the frame of reference. The three spatial module requests operate on spatial-relationship chunks to allow for the modeling of spatial competence. Specifications for each module request are included in Table 2. For the ACT-R/DHM spatial module, all operations occur in the spatial buffer. The ego-relate request takes an object as encoded by a sensory/perception module, produces an egocentric spatial relationship to the object, Table 1. The spatial relationship chunk includes information about both the object and its frame of reference

SPATIAL RELATIONSHIP CHUNK Slot Description referent The referent object The encoded object from a object sensory/perception module Object position in egocentric spherical pitch, azimuth, distance coordinates xdir, dir, zdir xup, yup, zup Object orientation axes xright, yright, zright


301

Table 2. The spatial module provides three requests that support spatial reasoning. Requests are used by the cognitive modeler to simulate human performance. The outcome of spatial requests is always placed in the spatial buffer.

SPATIAL MODULE REQUESTS Request Input Arguments ego-relate

visual percept obj

mid-relate

spatial relationship ref-obj spatial-relationship ref-tar

chainrelate

spatial-relationship ref-obj spatial-relationship obj-tar

Output to Spatial Buffer spatial relationship self-obj: referent = SELF object = obj spatial relationship obj-tar: referent = object of ref-obj object = object of ref-tar spatial relationship ref-tar: referent = referent of ref-obj object = object of obj-tar

and places the new spatial-relationship chunk in the spatial buffer. The object chunks themselves must provide some native orientation information. In the case of the vision system, we have extended visual object chunks to include orientation information. When a visual-object chunk is passed to ego-relate, orientation information is passed to the new spatial-relationship. The mid-relate and chain-relate functions build on egocentric spatial-relationship chunks to produce object-to-object relationships. Mathematically, mid-relate and chain-relate are simply vector addition functions, while psychologically they correspond to a “single-step” mental rotation (see Gunzelmann and Lyon [11] and also Kosslyn [14]). The mid-relate request takes two spatial-relationships as arguments, e.g. Self->A and Self->B, and creates an object-to-object spatialrelationship chunk A->B, where A serves as the frame of reference. Similarly, chainrelate takes as input any two spatial-relationship chunks of the form A->B and B->C and creates a spatial-relationship chunk A->C, where A serves as the frame of reference. 3.2 Enforcing Constraints We now review the implementation relative to the previously described theorems and the ACT-R architecture. Theorem 1 states that both egocentric and allocentric encoding is necessary for human spatial competence. The ACT-R/DHM implementation provides a generic representation, the spatial-relationship chunk type, that allows both types of encoding. Theorem 2 states that the interaction of egocentric and allocentric representations is essential to human spatial cognition. The mid-relate function provides a direct transformation from egocentric to allocentric representations. The chain-relate function can also be used for the inverse operation, if the first spatial-relationship in the “chain” is an egocentric relationship, e.g. Self->A + A->B = Self->B. Theorem 3 states that all spatial relationships must begin as egocentric. Sensory percepts are encoded into objects that include egocentric pitch, azimuth, and distance.

302


This egocentric information can be used to generate egocentric spatial-relationships using ego-relate. Theorem 4 states that all relationships are stored in an allocentric form. The mid-relate mechanism converts egocentric spatial-relationships with two objects into a single object-to-object relationship. In our implementation object-toobject relationships are only useful when tied to an egocentric relation. This requirement is based on Theorems 2 and 3. The spatial-relationship chunk also allows for the encoding of a frame of reference and of object orientation, as required by Theorem 5. While the current implementation relies on Rock [9] and McNamara [8] to support the requirement for orientation encoding, it makes no claim to model the determination of orientation by visual or declarative memory methods. This is an interesting question for our implementation that deserves future work as the assumption that orientation can be encoded underpins the spatial module’s encoding of allocentric, object-to-object relationships. 3.3 Spatial Modeling with ACT-R/DHM With the ACT-R/DHM implementation of the spatial sense now specified, the application of the new spatial modeling capability is perhaps most interesting to the DHM community. We now describe the use of the new spatial module to improve the performance of the previously introduced model –the vending machine interaction task. As mentioned previously, ACT-R/DHM to-date has been used to drive the Santos™ digital human model, and Santos™ exists in a high-fidelity 3D virtual environment. The vending machine interaction task now uses the module requests of the spatial module. The model assumes that the human subject is familiar with the parts of the vending machine (e.g. buttons, labels, coin slot, etc.) but has not seen or used this specific machine before. Thus, as the avatar approaches the machine, he encodes the layout of the machine relative to the machine’s background (a large environmental cue) using a series of ego-relate and mid-relate requests. The machine’s background, known in the model as the “CokeBox”, is visible at all times during the machine interaction task, and is therefore an ideal object for the construction of object-toobject relationships. For example, the model encodes egocentric relationships Self>Button1 and Self->CokeBox then uses Mid-Relate with these two egocentric relationships as arguments to create and store the object-to-object relationship CokeBox->Button1. To utilize the stored spatial relationship, the model must relocate CokeBox in the visual field and use chain-relate to program egocentric movements for machine interaction. After encoding the machine layout, the model programs numerous motor movements from the avatar’s hand to the coin slot of the machine, simulating the deposit of coins. Note that as the avatar looks at his hand, the coin slot drops from the visual field. The position of the coin slot relative to the CokeBox must then be recalled from declarative memory and an egocentric spatial-relationship chunk constructed via chain-relate in order to relocate the coin slot and continue depositing.


303

4 Conclusions and Future Enhancements The implementation of a spatial module in ACT-R/DHM resolves significant issues related to knowledge of object locations in 3D environments and provides the capability to model human performance for many dynamic tasks. To make the spatial modeling capability more accessible and accurate, a number of additional enhancements are necessary. For example, the link between the visual and spatial systems should be explored with regard to attention and autonomous behavior. It seems feasible, as Harrison and Schunn have suggested [13], that at least one or more currently attended spatial relationships may be updated automatically as the body moves. In fact, ACT-R/DHM’s KP system provides some functionality that could be used to update spatial knowledge based on kp movement information. The enforcing of limitations on spatial cognition is also an area that needs additional research and implementation. If, as Kosslyn suggests [14], spatial reasoning must occur egocentrically, then only egocentric spatial relationships should be available to the spatial buffer. This could be accomplished by implementing mid-relate and chainrelate functionality within the spatial module and exposing only egocentric spatial relationships via the spatial buffer. While much future work remains to extend this implementation, compare our implementation with alternative implementations, and validate against human spatial cognition data, the ACT-R/DHM spatial module provides significant functionality based on spatial cognition theory and within the existing ACT-R framework.

References 1. Newell, A.: Unified Theories of Cognition. Harvard University Press, Cambridge (1999) 2. Anderson, J.R., Bothell, D., Byrne, M.D., Douglass, S., Lebiere, C., Qin, Y.: An Integrated Theory of the Mind. Psychological Review 111(4), 1036–1060 (2004) 3. Laird, J.E., Newell, A., Rosenbloom, P.S.: SOAR: An Architecture for General Intelligence. Artificial Intelligence 33(1), 1–64 (1987) 4. Kieras, D.E., Meyer, D.E.: An Overview of the EPIC Architecture for Cognition and Performance with Application to Human-Computer Interaction. Human-Computer Interaction 12(4), 391–438 (1997) 5. Carruth, D., Robbins, B., Thomas, M., Letherwood, M., Nebel, K.: Symbolic Model of Perception in Dynamic 3D Environments. In: Proceedings of the 25th Army Science Conference, Orlando, Florida (September 2006) 6. Carruth, D., Thomas, M., Robbins, B.: Integrating Perception, Cognition and Action for Digital Human Modeling. In: Duffy, V.G. (ed.) HCII 2007 and DHM 2007. LNCS, vol. 4561, pp. 333–342. Springer, Heidelberg (2007) 7. Abdel-Malek, K., Yang, J., Marler, R.T., Beck, S., Mathai, A., Zhou, X., Patrick, A., Arora, J.: Towards a New Generation of Virtual Humans: Santos. International Journal of Human Factors Modeling and Simulation 1(1), 2–39 (2006) 8. McNamara, T.: How Are the Locations of Objects in the Environment Represented in Memory? In: Freksa, C., Brauer, W., Habel, C., Wender, K.F. (eds.) Spatial Cognition III. LNCS, vol. 2685, pp. 174–191. Springer, Heidelberg (2003) 9. Rock, I.: Orientation and Form. Academic Press, New York (1973)

304


10. Sholl, M.J., Nolin, T.L.: Orientation Specificity in Representations of Place. Journal of Experimental Psychology: Learning, Memory, and Cognition 23(6), 1494–1507 (1997) 11. Gunzelmann, G., Lyon, D.R.: Mechanisms for Human Spatial Competence. In: Barkowsky, T., Knauff, M., Ligozat, G., Montello, D.R. (eds.) Spatial Cognition 2007. LNCS (LNAI), vol. 4387, pp. 288–307. Springer, Heidelberg (2007) 12. Best, B.J., Lebiere, C.: Spatial Plans, Communication, and Teamwork in Synthetic MOUT Agents. In: Proceedings of the 12th Conference on Behavior Representation In Modeling and Simulation (2003) 13. Harrison, A.M., Schunn, C.D.: ACT-R/S: Look Ma, no “cognitive map"! In: Detje, F., Doerner, D., Schaub, H. (eds.) Proceedings of the Fifth International Conference on Cognitive Modeling, pp. 129–134. Universitats-Verlag, Bamberg (2003) 14. Kosslyn, S.M.: Image and Brain: The Resolution of the Imagery Debate. MIT Press, Cambridge (1994)

Behavior-Sensitive User Interfaces for Smart Environments Veit Schwartze, Sebastian Feuerstack, and Sahin Albayrak DAI-Labor, TU-Berlin Ernst-Reuter-Platz 7, D-10587 Berlin {Veit.Schwartze,Sebastian.Feuerstack, Sahin.Albayrak}@DAI-Labor.de

Abstract. In smart environments interactive assistants can support the user’s daily life by being ubiquitously available through any interaction device that is connected to the network. Focusing on graphical interaction, user interfaces are required to be flexible enough to be adapted to the actual context of the user. In this paper we describe an approach, which enables flexible user interface layout adaptations based on the current context of use (e.g. by changing the size of elements to visually highlight the important elements used in a specific situation). In a case study of the “4-star Cooking assistant” application we prove the capability of our system to dynamically adapt a graphical user interface to the current context of use. Keywords: Layouting, model-based user interface development, adaptation, constraint generation, context-of-use, smart environments, human-computer interaction.

1 Introduction Interactive applications, which are deployed to smart environments, are often targeted to support the users in their every-day life by being ubiquitous available and continuously offering support and information based on the users’ requirements. Such applications must be able to adapt to different context-of-use scenarios to remain usable for each user’s situation. Scenarios include e.g. adapting the user interface seamlessly to various interaction devices or distributing the user interface to a set of devices that the user feels comfortable with in a specific situation. The broad range of possible user interface distributions and the diversity of available interaction devices make a complete specification of each potential context-of-use scenario difficult during the application design. Necessary adaptations require flexible and robust (re-) layouting mechanisms of the user interface and need to consider the underlying tasks and concepts of the application to generate a consistent layout presentation for all states and distributions of the user interface. Based on previous work [12], we propose a constraint-based GUI layout generation that considers the user’s behavior and her location in a smart environment. Therefore we concentrate on the user’s context and identify several types of possible layout adaptations: V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 305–314, 2009. © Springer-Verlag Berlin Heidelberg 2009

306

V. Schwartze, S. Feuerstack, and S. Albayrak

1. Spot-based adaptation: In a smart environment, such as our SerCHo Living Lab, different places identify various situations. Applications can consider these spots to adapt their user interface layout to focus on those parts of the UI that are identified as most important for a certain spot. 2. Distance-based adaptation: The distance of the user to a certain interaction device, such as a wall-mounted display or a mobile phone, can be used to adapt the layout. 3. Orientation-based adaptation: The orientation of the user to an interaction device can influence the presentation of the user interface. Thus, for instance the angle of view in which the user looks at a display can be used to enlarge the visual weight of elements on one side of the user interface presentation. These adaptations can be done either by discretely or continuously modifying the user interface layout and can be combined for a more comfortable interaction experience. Different to the spot-based adaptation that requires the application developer to explicitly specify, which user tasks are most relevant for a certain user location, the distance- and orientation-based adaptations can be performed without any effort of the designer. In the following, we illustrate the definition and usage of layouting statements to create constraint systems that evaluate runtime context information to adapt the user interface layout accordingly.

2 User Interface Layouting Different to other layout generation approaches [11], we create the constraint system at runtime. In our layout model a user interface is described using four basic characteristics: the containment, the orientation, the order and the size of user interface elements (UI elements). The containment characteristic describes the relation of elements as a nested hierarchy by abstract containers that can contain other abstract containers or UI elements. All UI elements are in an order that can be defined by relations like “before” or “after”. The orientation distinguishes between elements that are oriented horizontally or vertically to each other. Finally the size specifies the width and height of containers and UI elements relative to other UI elements or abstract containers. To create a constraint system from these characteristics, we use a set of statements to express the building process. A statement has conditions combined with conjunctions and disjunctions to define the scope of the statement. Conditions can also use additional information about the UI elements to define application independent statements. The formal description of a statement is shown in figure 1 top. If the conditions are fulfilled, the statement is used and the effect modifies the constraint system. At runtime this set of statements is evaluated and creates a constraint system solved by a Cassowary constraint [1] solver. This constraint solver supports linear problems and solves cycles. To generate a flexible constraint system, it also supports a constraint hierarchy using weak, medium, strong and required priorities. The Effect is split into dynamic and static, static statements use only a static value for adaptations; in opposite dynamic statements use a function depending on dynamic information. Dynamic functions are divided into logical- and mathematical functions. Mathematical functions describe the behavior of their value

Behavior-Sensitive User Interfaces for Smart Environments

307

Fig. 1. Statement format and example

in dependency to external information1 like the distance to the screen. Logical functions use external information2 to create a logical value to come to a decision. This kind of function for instance is used to generate the initial orientation for the elements of the user interface. The example shown in figure 1 describes a “Prioritize Statement” changing the space allocation for a specific node, in this case for the element “GiveHelp”. The effect contains a mathematic function with the variable “distance”. If the distance between the user and the screen changes, the function recalculates the prioritized value that describes how much space the element “GiveHelp” additionally gets from other UI elements. 2.1 Statement Evaluation The result of a successful layout calculation is a set of elements, each consisting of the location (an absolute x, y coordinate) and a width and height value. The layout generation is performed in three phases: 1. First an initial layout is automatically generated by a set of predefined algorithms that interpret the design models like the task- and abstract user interface model to generate an initial layout that is consistent for all platforms. The result of the containment statement is a tree structure representing the graphical user interface organization. The orientation statement at first allocates the vertical space and after a designer modifiable threshold value is reached, it uses an alternating orientation. After the definition of the orientation the size statement defines the initial space usage for the user interface elements. Basic constraints assure that all additional constraints added do not corrupt the constraint system.

1

2

Numerical data from different information sources like the context model. Comparable data like numerical and textual information.

308


2. A designer can manipulate the pre-generated layout to match his aesthetical requirements by adding statements that relate information of the design models with a layout characteristic of a UI element. 3. Finally, the user behavior in a smart environment can by considered by adding generic statements that can weight individual UI elements based on the actual context of the user at system runtime. 2.2 Context Related-Layout Adaptations To adapt the interface to specific situations the designer can define context sensitive statements to prioritize specific nodes described in the next section. These statements are only active for specific situations described by context information. Even though our layout model describes the size, order, orientation and containment structure separately, for realizing layout adaptations regarding the user behaviour, we focus on size adaptations, as modifying the other layout characteristics can destroy the user interface consistency, which affects the usability [7]. As we described in the introduction, there are three different statement types: Spot-based adaptation, Orientation-based adaptation, Distance-based adaptation. The basic idea for all adaptations is to highlight the context relevant parts of the user interface for the moment. This is described by a prioritize value characterizing how much additional space an element can use compared to the rest of the interface. The figure below shows an example. The algorithm allocates the space according to the weight (contained elements) so the increase depends on the amount of other elements. In this example we prioritize the red node, the prioritize value of ½ ensures that the node gets additional space of the other nodes. As a result, this statement adds a new constraint with the weight “strong” to the constraint system sizerednode ≥ 2/3 * sizeparentnode. The context based adaptations use static and dynamic statements to recalculate the space allocation for the graphical user interface. The statements, defined by the designer, the prioritize value (static statements) and prioritize function (dynamic statements), used in the next section, are examples and adjustable by the designer. The Spot-based adaptation uses a static prioritize statement for a specific set of nodes and an assigned position of the user. If the user reaches the specified position, the statement is used and adds for the affected nodes the amount of space given by the prioritize value.

Fig. 2. Result of the prioritizing process


309

The orientation-based adaptation uses the Spots “D” and “A2” shown in figure 3 bottom. If the user enters the specified position, this statement is activated and prioritizes a specific node. If the user stands left or right from the screen, this statement prioritizes all nodes with the upper left corner on the opposite side. The Distance-based adaptation uses the distance from the user to the screen to calculate the prioritize value relative from the distance. If the user moves away from the display, the relevant parts of the interface are enlarged. In the following case study these adaptations are described and discussed.

3 Cooking Assistant Case Study To test the adaptations we deployed the cooking assistant into a real kitchen environment of our SerCHo living lab like depicted by the photo in figure 3 top-left. This multimodal application assists the user during the cooking process. The main screen, shown in figure 3, top-right, guides you through the cooking steps and provides help if needed. The figure 3, bottom, illustrates several spots corresponding to the different working positions and user tasks in the kitchen. Since the touch screen supports a view angle of 160 degrees, the user cannot observe the screen from all spots. For the spot-based layouting, we therefore focus on the spots listed in table 1. Figure 4 depicts the box-based preview of our layout editor from which the main screen of the cooking assistant has been derived. By a preceding task analysis, we identified the most relevant interaction tasks. Deriving an initial layout model from a task hierarchy structure has the advantage hat related tasks end up in the same boxes and will be layouted close to each other since they share more parent containers the closer they are related.

Fig. 3. The kitchen with the cooking assistant running on a touch screen (top-left), the main screen of the cooking assistant (top-right), and the location spots defined by the context model (bottom)

310


Table 1. An excerpt of the user contexts that are supported by the application. The second column lists the most relevant application tasks for each user tasks.

Spot

User context

A2 C1.2 B2

Looking for ingredients.

D

E

Preparing ingredients while following the cooking advices and controlling the kitchen appliances. Learning about next steps while cleaning dishes after a step has been done. Concentrating on the video or getting an overview about the recipe steps.

Relevant tasks ordered by Priority. 1. listRequiredIngredients 2. listNextStepIngredients 1. stepDetailedDescription 2. listRequiredIngredients 3. selectAppliance 4. giveHelp 1. presentNextStepSummary 2. listNextStepIngredients 3. stepSelection All tasks same priority

Fig. 4. Changes from automatic generated layout to designer adapted layout

The starting point for all adaptations is the constraint system generated by the automatic statements shown in figure 4 Phase I and adapted by the designer to adjust the space allocation to his wishes. The result of this process is shown in figure 4 Phase II. To adapt the constraint system to a specific situation, we describe three examples below. 3.1 Statements for Spot-Based Adaptation: (B2) While using the cooking assistant (CA), the user is preparing ingredients, following the cooking advices and controlling the kitchen appliances. Because it is difficult to look at the screen from this position, shown in figure 3 bottom, the statement highlights the important information (Task: stepDetailedDescription, listRequired Ingredients, selectAppliance, giveHelp). The condition of the Spot statement is characterized by an environment condition, the position of the user and relevant interactions tasks, as the interface structure is derived from the task model. Because


311

Fig. 5. B2 prioritize “ShowCurrentStepDetail” with elements stepDetailedDescription, list RequiredIngredients, selectAppliance and giveHelp

the container “showCurrentStepDetails” contains the most relevant elements it is prioritized. Additionally, the statement use a static prioritize value, defined by the designer. For this study we use a fraction of 4/5(80%) because the prioritization is high enough to support the user, but low enough to follow the changes in the user interface and not confuse the user. The effect of this statement for the case B2 is shown in figure 5. 3.2 Statements for Distance-Based Adaptation While cleaning dishes after a step has been done, the user wants to learn more about the next step. A video helps to understand what has to be done. Because the focused task is specified in the AUI model, the layout algorithm can prioritize the task containing the specific element. The distance statement is characterized by a function calculating the prioritize value depending on the distance to the screen. This function is expressed by prioritize value = ax2 + bx +c. The constants a,b,c can adapted by the designer to match the function to the maximum distance. For our case study we use this linear function: prioritize value = 4/30003* distance. The user interface prioritizing “giveHelp” depending to the distance is shown in figure 6.

Fig. 6. Distance based adaptation, shown for 100, 200 and 400cm

3.3 Statements for Orientation-Based Adaptation: (A2), (D) If the user has something to do at the spots A2 and D shown in figure 3 bottom, the view angle to the screen is inappropriate.

3

This fraction is calculated by the assumption that the interaction space maximum of 600cm, the prioritization for this distance is 4/5(80%).

312


Fig. 7. Orientation based adaptation for left- and right side

Depending from the angle of view to the screen shown in figure 7, elements with the upper left corner at the affected side rendered broader than half width of the screen. If the user enters Spot D (left) and leaves the normal angle of view (shown in figure 3 bottom) the width of the elements “giveHelp” and “controlAppliance” is growing to half of the screen width. The same happens if the user enters Spot A2 (right) with the elements “listRequiredIngredients”, “listNextStepIngredients”.

4 Related Work Nichols et al. list a set of requirements that need to be addressed in order to generate high-quality user interfaces in PUC [5]. As for layout information they propose to not include specific layout information into the models as this first tempts the designers to include too many details into the specification for each considered platform, second delimits the user interface consistency and third might lower the chance of compatibility to future platforms. Different to PUC we are not focusing on control user interfaces, but end up in a domain independent layout model that specifies the containment, the size, the orientation and the order relationships of all individual user interface elements. Therefore we do not want to specify the layout manually for each targeted platform and do not rely on a set of standard elements (like a set of widgets for instance) that has been predefined for each platform. The SUPPLE system [3] treats interface adaptation as an optimization problem. Therefore SUPPLE focuses on minimizing the user’s effort when controlling the interface by relying on user traces to estimate the effort and to position widgets on the interface. Although in SUPPLE an efficient algorithm to adapt the user interface is presented, it remains questionable if reliable user traces can be generated or estimated. While SUPPLE also uses constraints to describe device and interactor capabilities they present no details about the expressiveness of the constraints and the designers effort in specifying these constraints. The layout of user interfaces can be described as a linear problem, which can be solved using a constraint solver. The basic idea is shown in [12], this approach uses a grid layout to organize the interface and create a constraint system. Our approach instead uses a tree structure and supports more constraint strengths. Recent research has been done also by Vermeulen [8] implementing the Cassowary algorithm [1], a weak constraint satisfaction algorithm to support user interface adaptation at run-time to different devices. While he demonstrates that constraint satisfaction can be done at run-time, to our knowledge he did not focus on automatic constraint generation.


313

Other approaches describe the user interface layout as a space usage optimization problem [4], and use geometric constraint solvers, which try to minimize the unused space. Compared to linear constraint solving, geometric constraint solvers require plenty of iterations to solve such a space optimization problem. Beneath performance issues an efficient area usage optimization requires a flexible orientation of the user interface elements, which critically affects the user interface consistency. Richter [6] has proposed several criteria that need to be maintained when relayouting a user interface. Machine learning mechanisms can be used to further optimize the layout by eliciting the user’s preferences [5]. The Interface Designer and Evaluator (AIDE) [7] and Gadget [2] are incorporating metrics in the user interface design process to evaluate a user interface design. Both projects focus on criticizing already existing user interface layouts by advising and interactively supporting the designer during the layout optimization process. They follow a descriptive approach by re-evaluating already existing systems with the help of metrics. This is different to our approach that can be directly embedded into a model-based design process (forward engineering). To adapt user interfaces to a specific situation, in [9] an XSL transformation is used to adapt the abstract description of the interface to the different devices. Our approach follows a model-based user interface design [8]. Following a model-based user interface development involves a developer specifying several models using a model editor. Each abstract model is reified to a more concrete model until the final user interface has been derived. The result is a fine structured user interface, which could be easily adapted to different situations. An akin approach to create a user interface is presented in [10], the interface structure is derived from the task model and fleshed out by the AUI- and CUI Model. To adapt the interface to mobile devices, different containing pattern are used to organize the information on the screen. Our approach doesn’t break the interface structure into small pieces because all information has to be displayed.

5 Conclusion and Further Work In this paper we presented an approach to adapt the user interface of applications to specific situations. Furthermore our case study “4-Star Cooking Assistant” has shown the relevance to support the user. In the future we have to enlarge the case study to other applications and check more context information about the relevance for GUI adaptations. User-interaction-related adaptation: Based on the user’s experiences and his interaction history (tasks completion and referred objects), the most important areas of control can be visually weighted higher to prevent unprofitable interaction cycles or helping the user in cases where he is thinking (too) long about how to interact or to go any further. User-abilities-related adaptation: The layout adapts to the user’s stress factor by visually highlighting the most relevant tasks, and takes into account if the user is left or right handed by arranging the most relevant parts of the user interface. Finally his eye-sight capabilities can be used to highlight the most important areas of control.

314


References 1. Badros, G.J., Borning, A.: The Cassowary linear arithmetic constraint solving algorithm. In: ACM Transactions on Computer-Human Interaction (2001) 2. Fogarty, J., Hudson, S.: GADGET: A toolkit for optimization-based approaches to interface and display generation (2003) 3. Gajos, K., Weld, D.: SUPPLE: Automatically Generating User interfaces; In: Proceedings of Conference on Intelligent User Interfaces 2004, Maderia, Funchal, Portugal (2004) 4. Gajos, K., Weld, D.S.: Preference elicitation for interface optimization. In: UIST 2005: Proceedings of the 18th annual ACM symposium on User interface software and technology, New York, NY, USA (2005) 5. Nichols, J., Myers, B.A., Harris, T.K., Rosenfeld, R., Shriver, S., Higgins, M., Hughes, J.: Requirements for Automatically Generating Multi-Modal Interfaces for Complex Appliances. In: IEEE Fourth International Conference on Multimodal Interfaces, Pittsburgh 6. Richter, K.: Transformational Consistency. In: CADUI 2006 Computer-AIDED Design of User Interface V (2006) 7. Sears, A.: Aide: a step toward metric-based interface development tools, pp. 101–110 (1995) 8. Vermeulen, J.: Widget set independent layout management for uiml, Master’s thesis, School voor Informatie Technologie Transnationale Universiteit Limburg (2000) 9. Chiu, D.K.W., Hong, D., Cheung, S.C., Kafeza, E.: Adapting Ubiquitous Enterprise Services with Context and Views. In: Dickson, K.W. (ed.) EDOC 2006: Proceedings of the 10th IEEE International Enterprise Distributed Object Computing Conference, Washington, DC, USA, pp. 391–394 (2006) 10. Martinez-Ruiz, F.J., Vanderdonckt, J., Martinez-Ruiz, J.: Context-Aware Generation of User Interface Containers for Mobile Devices. In: ENC 2008: Proceedings of the 2008 Mexican International Conference on Computer Science, 2008, Washington, DC, USA, pp. 63–72 (2008) 11. Lutteroth, C., Strandh, R., Weber, G.: Domain Specific High-Level Constraints for User Interface Layout, Hingham, USA, pp. 307–342 (2008) 12. Feuerstack, S., Blumendorf, M., Schwartze, V., Albayrak, S.: Model-based layout generation. In: Proceedings of the working conference on Advanced visual interfaces, Napoli, Italy (2008)

Non-intrusive Personalized Mental Workload Evaluation for Exercise Intensity Measure N. Luke Thomas1, Yingzi Du1, Tron Artavatkun1, and Jin-hua She2 1

Purdue School of Engineering and Technology at Indianapolis, Electrical and Computer Engineering 723 W. Michigan St. SL160, Indianapolis, IN, USA 2 Tokyo University of Technology 1404-1 Katakuracho, Hachioji-shi, Tokyo 192-0982 Japan {nlulthom,yidu,tartavat}@iupui.edu, [email protected]

Abstract. Non-intrusive measures of mental workload signals are desirable, because they minimize artificially introduced noise, and can be more accurate. A new approach for non-intrusive personalized mental workload evaluation is presented. Our research results show that human mental workload is unique to each person, non-stationary, and not zero-state. Keywords: Personalized mental workload evaluation, exercise intensity measurement, biometrics.

1 Introduction Prediction of a user’s level of mental workload can help detect the physical and psychological status of the human users [1-6]. It is important to perform these measurements without producing further stress, workload, or interference with the user’s normal function in the job [7-11]. In this paper, we propose a biometric-based eye-movement mental workload evaluation system that can automatically identify a user, set system parameters based on the specific user’s needs and previous usage, and detect when a user’s expected workload exceeds some threshold for optimal performance. Biometrics is the process by which one can automatically and uniquely identify humans using their intrinsic qualities, traits, or identifying features. Some examples of biometric identification systems include iris, face, fingerprint, voice, vein, keystroke, and gait recognition systems or algorithms [12-14]. In particular, iris recognition is an ideal biometric recognition technology for accurate, non-intrusive recognition of large numbers of users. Iris recognition is the most accurate biometric recognition technology, with reported results of false match rates of 1 in 200 billion [15]. Additionally, images that are adequate for iris recognition can be acquired at a distance of up to 10 feet from the user, using near-infrared illumination that is unobtrusive. These features make iris recognition an ideal biometric system to identify users of the workload evaluation system, and then tailor the system to their needs and past requests. Eye-tracking and eye movement based mental workload evaluation is a solution, from a system design standpoint, because the information, images of the eye, can be acquired rapidly (in excess of 30 frames per second), could be processed in real-time, V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 315–322, 2009. © Springer-Verlag Berlin Heidelberg 2009

316

N. Luke Thomas et al.

and are highly correlated with a user’s mental workload [1, 7, 8, 16-18]. The information can also be acquired without requiring any special training by the user and without interfering with their normal activity. It has been a focus of workload researchers to produce an average personal model of human mental workload and fatigue [1-4, 11, 19-23]. However, we believe our research shows that mental workload is, instead, a more individual quality of each person. That is, a human being is an inherently non-stationary and non-zero-state system—even in a constrained experimental set-up, it is impossible to replicate the exact same mental and physical state of a person at multiple times. Therefore, since one cannot guarantee an exact, or perhaps even similar, initial state for a workload experiment, it is not appropriate to attempt to apply a single workload or exhaustion model to all people. Instead, we believe that mental workload and fatigue should be modeled on an individual basis, and that while detection is possible, prediction of a user’s workload or fatigue is inherently a flawed approach.

2 Method 2.1 System Setup For our experiment, we acquired videos of user’s eyes using an internally developed helmet mounted cMOS camera. The camera captures 30 frames per second at 640 by 480 pixels. For each user, there were 3 videos taken—5 minutes while driving a

Fig. 1. The head–mounted camera system (right, front, left)

Fig. 2. Example frames from the acquired videos

Non-intrusive Personalized Mental Workload Evaluation

317

motorized cart prior to any physical activity, 10-20 minutes while on a stationary bike, and 5 minutes while driving a motorized cart after the stationary bike exercise. Each video is between 7,000 and 30,000 frames. Figure 1 shows the camera system, and Figure 2 shows frames acquired by the camera system. 2.2 Video-Based Pupil Detection and Classification The videos were processed using internally developed pupil segmentation and measurement software running in Matlab (Fig. 3-4) [24]. The system takes advantage of the motion information in the video to quickly detect the region of interest of the pupil; detects and measures the pupil location, radius, and other eye parameters; and uses a pattern-recognition based system to classify the frame as blink, non-blink, or questionable. N-1 Video Frame

Edge Detected Image

Nth Video Frame

Video-based rough pupil detection

Pattern recognition blink/non-blink classification

Greedy angular pupil detection

Detected Pupil

Fig. 3. Proposed Processing Method

2.3 Data Processing Using the measured data from the videos, the overall workload/blink pattern was extracted from the sequence. Normally, it is easier to detect and classify non-blink frames. Therefore, the analysis was primarily based on the non-blink results. Figure 5 shows several images that are difficult to be conclusively classified as either blink or non-blink. The sequence of frame states and measurements was averaged over the length of the video in 30 second increments. The Cart 1 video was used as a baseline for the other videos, because it was taken of the user prior to physical exercise and, thus,

318


Fig. 4. Final Frame Classification Process

Fig. 5. Frames that cannot be conclusively classified as blink or non-blink

should be indicative of the user’s level at normal workload and exhaustion. Using the parameters from this video, the ratio of non-blink to blink frames was calculated (Fig 7). In this plot, lower values indicate a period in which the user was blinking more often. Additionally, plots of the detected pupil radius were generated. To increase the accuracy of the results, the detected pupil was first measured in the down sampled image used in most processing. However, if a long series of pupils were detected with the same measured radius, the measurement was repeated on the originally sized image for more discerning results. It is important to note that the original-sized pupil radius was only determined when it was expected the down sampled measurement might not have been accurate, and therefore is not necessarily determined over all time intervals. Additionally, a periodic measure of the outer iris boundary was determined for normalization purposes. Since all the pupil and eye parameters are measured in pixels and the distance from the eye to the camera is not necessarily the same from person to person or video to video, we use the iris boundary radius to normalize the pupil radius measurements to be invariant to image acquisition differences from video to video.


319

2.4 Biometrics-Based Data Processing For each user, prior to beginning the workload evaluation, the system would identify the user using iris recognition. If the user had not ever used the system before, they would be enrolled in the system for future use and identification. After identifying the user, the system could use their previously recorded data to set appropriate thresholds for their workload level, set up the system to their ergonomic needs, or provide specialized instructions for the current situation based on past evaluations. Some good quality images acquired have noticeable iris pattern (Fig. 6)—the resolution and image quality is adequate for iris recognition. However, the segmentation of the pupil and iris areas can be quite difficult—many images have significant occlusion from the eyelids and eyelashes, eye gaze can be non-frontal, and the illumination can change throughout the video.

Fig. 6. Example images with adequate quality for iris recognition

3 Experimental Results Our results showed that there is significant variability in physiological changes from person to person. Some individuals blinked less over the course of their exercise, while others blinked more, and others ratio changed periodically. Some individual’s pupil radius increased during the exercise, others stayed constant, and still others decreased. Additionally, for some individuals, their radius and ratio’s were similar from the initial video, cart 1, to the video acquired during exercise, bike. However, for other users, the ‘initial’ state from the cart 1 video were significantly different compared to the results from the bike video, both above and below depending on the user. Figure 7-(a-d) shows the results for 4 representative users—for each user the first plot is the normalized pupil radius and the second is the blink to non-blink ratio. Higher values in the pupil radius plot are indicative of larger pupils, after having been normalized by the measured iris radius. Higher values in the blink to non-blink ratio plot indicate that there were more frames classified as non-blink compared to blink during that time period. On the basis of these results, we believe that attempting to similarly model all people’s physiological reaction to a workload change is inadequate. Instead, it shows that each user’s physiological reactions should be used to develop an individualized workload model, which can then be used and adapted in future evaluations.

320


Fig. 7-a. Subject A

Fig. 7-b. Subject B

Fig. 7-c. Subject C

Fig. 7-d. Subject D


321

4 Conclusion We have developed a system for personalized mental workload evaluation using nonintrusively acquired eye information. The research results show that human mental workload is unique to each person, non-stationary, and not zero-state. Because of this, each user’s mental workload should be modeled individually and adaptively.

Acknowledgement The authors would like to acknowledge the volunteers for data collection in this research in Japan. Part of the research is sponsored by the International Development Fund (IDF) at Indiana University-Purdue University Indianapolis.

References [1] Cain, B.: A review of the mental workload literature (2007) [2] Brookhuis, K.A., de Waard, D.: On the assessment of (mental) workload and other subjective qualifications. In: Ergonomics, November 15, 2002, pp. 1026–1030 (2002); discussion 1042-6 [3] Hancock, P.A., Caird, J.K.: Experimental evaluation of a model of mental workload. In: Hum. Factors, vol. 35, pp. 413–429 (September 1993) [4] Hancock, P.A., Meshkati, N.: Human mental workload. North-Holland, Amsterdam, Sole distributors for the U.S.A. and Canada. Elsevier Science Pub. Co. (1988) [5] Hancock, P.A., Meshkati, N., Robertson, M.M.: Physiological reflections of mental workload. Aviat. Space Environ. Med. 56, 1110–1114 (1985) [6] Hankins, T.C., Wilson, G.F.: A comparison of heart rate, eye activity, EEG and subjective measures of pilot mental workload during flight. Aviat. Space Environ. Med. 69, 360–367 (1998) [7] Itoh, Y., Hayashi, Y., Tsukui, I., Saito, S.: The ergonomic evaluation of eye movement and mental workload in aircraft pilots. Ergonomics 33, 719–733 (1990) [8] Murata, A., Iwase, H.: Evaluation of mental workload by fluctuation analysis of pupil area. In: Proceedings of the 20th Annual International Conference of the IEEE, Engineering in Medicine and Biology Society, vol. 6, pp. 3094–3097 (1998) [9] Sekiguchi, C., Handa, Y., Gotoh, M., Kurihara, Y., Nagasawa, A., Kuroda, I.: Evaluation method of mental workload under flight conditions. In: Aviat. Space Environ. Med., vol. 49, pp. 920–925 (July 1978) [10] Wierwille, W.W.: Physiological measures of aircrew mental workload. Hum. Factors 21, 575–593 (1979) [11] Wilson, G.F., Russell, C.A.: Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Hum. Factors 45, 635–643 (Winter 2003) [12] Daugman, J.: New Methods in Iris Recognition. IEEE Transactions on Systems, Man, and Cybernetics, Part B 37, 1167–1175 (2007) [13] Du, Y.: Review of Iris Recognition: Cameras, Systems, and Their Applications. Sensor Review 26, 66–69 (2006)

322


[14] Proenca, H., Alexandre, L.A.: Toward Noncooperative Iris Recognition: A Classification Approach Using Multiple Signatures. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 607–612 (2007) [15] Daugman, J.: Probing the Uniqueness and Randomness of IrisCodes: Results From 200 Billion Iris Pair Comparisons. Proceedings of the IEEE 94, 1927–1935 (2006) [16] Neumann, D.L.: Effect of varying levels of mental workload on startle eyeblink modulation. Ergonomics 45, 583–602 (2002) [17] Recarte, M.A., Perez, E., Conchillo, A., Nunes, L.M.: Mental workload and visual impairment: differences between pupil, blink, and subjective rating. Span J. Psychol. 11, 374–385 (2008) [18] Backs, R.W., Walrath, L.C.: Eye movement and pupillary response indices of mental workload during visual search of symbolic displays. Appl. Ergon. 23, 243–254 (1992) [19] Cao, A., Chintamani, K.K., Pandya, A.K., Ellis, R.D.: NASA TLX: Software for assessing subjective mental workload. Behav. Res. Methods 41, 113–117 (2009) [20] Moray, N., NATO Special Program Panel on Human Factors: Published in coordination with NATO Scientific Affairs, Plenum Press, New York (1979) [21] Rouse, W.B., Edwards, S.L., Hammer, J.M.: Modeling the dynamics of mental workload and human performance in complex systems. IEEE Transactions on Systems, Man and Cybernetics 23, 1662–1671 (1993) [22] Satava, R.M.: Mental workload: a new parameter for objective assessment? Surg. Innov. 12, 79 (2005) [23] Young, M.S., Stanton, N.A.: It’s all relative: defining mental workload in the light of Annett’s paper. Ergonomics 45, 1018–1020 (2002); discussion 1042-6 [24] Thomas, N.L., Du, Y., Artavatkun, T., She, J.: A New Approach for Low-Cost Eye Tracking and Pupil Measurement for Workload Evaluation. In: 13th International Conference on Human-Computer Interaction (HCI) (2009)

Incorporating Cognitive Aspects in Digital Human Modeling Peter Thorvald1,2, Dan Högberg1, and Keith Case1,2 1 University of Skövde, Skövde, Sweden Loughborough University, Loughborough, UK {peter.thorvald,dan.hogberg}@his.se [email protected] 2

Abstract. To build software which, at the press of a button, can tell you what cognition related hazards there are within an environment or a task, is probably well into the future if it is possible at all. However, incorporating existing tools such as task analysis tools, interface design guidelines and information about general cognitive limitations in humans, could allow for greater evaluative options for cognitive ergonomics. The paper will discuss previous approaches on the subject and suggest adding design and evaluative guiding in DHM that will help a user with little to no knowledge of cognitive science, design and evaluate a human- product interaction scenario. Keywords: Digital human modelling, cognition, context, situatedness, ecological interface design, system ergonomics.

1 Introduction In Digital Human Modeling (DHM), the term ergonomics usually refers to modeling physical aspects of humans with the main focus being on anthropometry and physical strain on the body. This is also reflected in the DHM tools that exist, e.g. RAMSIS, JACK, SAMMIE, V5 Human, etc. [1, 2], tools that mainly, if not exclusively, model physical ergonomics. This paper will suggest and discuss possible ways of bringing cognition into the equation and provide users of DHM tools with an aid in evaluating cognitive as well as physical ergonomics. Computer modeling of human cognition was originally mainly done off-line in the sense that the cognitive system is viewed as a hardware independent program, effectively disregarding the surrounding environment and even the importance of a human body. However, in later years, there has been an increasing interest for viewing the human as part of a complex system, incorporating the environment and the human body in cognitive modeling. This has led to new theories regarding how humans cognize within the world and has allowed us to regard the body and the context as part of the cognitive system. Human cognition is not an isolated island where we can view our surrounding context as merely a problem space. We are very much dependant on our body and our surroundings to successfully survive in the world. Previous suggestions on integrating cognition in DHM tools have largely taken its basis in symbol processing architectures such as ACT-R, Soar and such [3-5], V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 323–332, 2009. © Springer-Verlag Berlin Heidelberg 2009

324

P. Thorvald, D. Högberg, and K. Case

architectures that disregard embodiment and situatedness of cognition. This paper will aim to place the computer manikins used in DHM tools within a context, a context where cognitive offloading and scaffolding onto the environment is supported. The main advantage of using DHM and incorporating the suggested functionality is that it can be used very early in the system development process. It also allows the designer to consider the spatial information that the physical array incorporates. In traditional usability methods, this is seldom the case as design iterations are often done offline in the sense that it only incorporates some (if any) physical properties of the domain where the system is to be implemented.

2 Cognitive Modeling in DHM During the last decade, there have been several attempts at incorporating cognitive modeling in DHM. A research group at Sandia National Laboratories in New Mexico have created a framework based on a modular and symbol processing view of human cognition and others have focused on a rule based system built on architectures such as ACT-R and Soar [3, 4]. Though not built on the same exact architecture, several others have gone about the problem in similar ways, ultimately trying to reach a state where the system can, at the press of a button, perform a cognitive evaluation for us [5]. However, the methodology upon which these architectures are built is challenged by researchers that recommend a more situated view on cognition as a whole. This view, originating in the 1920s from the Russian psychologist Lev Vygotsky, argues that human cognition cannot be viewed apart from its context and body [6]. There is no clear-cut line between what happens in the world and what happens in the head; the mind “leaks” into the world. A view already seen in the DHM society is to stop dividing human factors into “neck up” and “neck down” and instead view the human as a whole [7]. This view finds much support in the work on social embodiment by Lawrence Barsalou and colleagues. They discuss how the embodiment of the self or others can elicit embodied mimicry in the self or others [8], ultimately arguing for a holistic view of the human where the body and mind are both necessary for cognition. Whereas the discussion on embodiment and situatedness is beyond our scope in this paper, it shows us how earlier approaches to modelling cognition in DHM are at best insufficient and that a new approach is needed. The method that this paper is aimed at resulting in will not require any kind of “strong AI” and will have a much lower technological level than many others. However, it will try to consider the human as a system with a physical body, acting within an environment.

3 Cognition in System Ergonomics For a system design to become successful, the incorporation of human factors is essential. To a large part, physical ergonomics is very well accounted for in today’s system design practices, but the cognizing human is often neglected. However, as technology increasingly demands more human processing abilities, the modeling of human cognition becomes more important. The range of human behaviors need to be known to design for human-related control systems [9].

Incorporating Cognitive Aspects in Digital Human Modeling

325

System ergonomics can be used to describe a more or less complex task’s mental demands on a human. It does so by three points [9]. 1. Function The main consideration of function is what the operator has in view and to what extent the task is supported by the system. It is largely defined by the temporal and spatial properties of the activities to be performed. When and where should the task be performed? 2. Feedback The feedback allows the user to identify what state the system is in. If a performed task has resulted in anything, what task was performed etc. It is very important to allow the operator to recognize if an action had any effect on the system and also what the result of it was [10]. For example, even if a computing task on a PC takes some time to calculate, the operator is informed that the computer is working by a flashing led or an hourglass on the screen.

Fig. 1. A seat adjustment control which exhibits excellent natural mapping or matching between the system and the user’s mental model

3. Compatibility Compatibility is largely about the match between systems or between the system and the user’s mental model of the system. The operator should not be required to put too much effort into translating system signals. It relates information sources to each other. A very simple and obvious example from the automotive industry is described by Norman [10] with a seat adjustment control from a car. A similar seat adjustment control can be viewed in figure 1. It is obvious in the figure that the system (the adjustment control) corresponds well to the result of the task of manoeuvring the controls. The control maps very well to the response of the seat and to the user’s probable mental model. However, the compatibility is not exclusively relevant to the psychological issues but a designer also needs to consider the physical compatibility of the user and the system. Controls might for example be spatially located away from the physical reach of the human. Though these three points are hardly sufficient for a comprehensive design tool, they are of great help in an initial state of system design and will prove helpful to us in developing a more detailed design aid.

326


4 Methods for Interface Design and Evaluation In Human-Computer Interaction (HCI) there are several evaluation methods with great use for certain situations. As the aim of this paper is to present a draft of a design tool, we shall take a closer look at a few of these methods along with a task analysis tool. 4.1 Task Analysis All good design processes include some sort of task analysis. To be able to design a system that fits both task and human, we need to know as much as possible about the task. A fairly quick and dirty task analysis which provides a good basis for further development is the hierarchical task analysis (HTA) [11]. A HTA is a tree diagram of the task structure and serves several purposes. It gives us a good overview of the system or the task and subtasks that need to be performed, and provides aid in achieving common ground within a design group. It can also even serve as a task evaluation tool, allowing a designer to find global problems that can be missed while using usability inspection methods such as cognitive walkthrough [12], heuristic evaluation [13, 14] etc. Global issues are mainly related to the structure of the task and the relation between the subtasks whereas local issues are within a subtask with a very limited scope. Make Coffee Do in any order 1-2 Do 3 1. Add Water

2. Add Coffee

Do 1.1-1.2 1.1.Fill Pot with Water

1.2. Pour water into Coffee Maker

3. Press Button

Do 2.1-2.2 2.1. Place Filter

2.2. Add Coffee

Fig. 2. A very simple HTA of the process of making a pot of coffee

The creation of a HTA is fairly simple. First, identify the overlying task to be performed, which in our very simple example, illustrated in figure 2, is making a pot of coffee. The HTA in figure 2 shows this process and also shown are the plans within which each subtask should be performed. In this example it is limited to doing the tasks in order or doing two subtasks first in any order and then continuing with the third. However, these plans can be pretty much anything you want them to be such as selections (do one but not the other), linear or nonlinear, or even based on a specific condition (if X then do Y, else Z). The finished task analysis is then used as a basis for further inspections and design iterations. 4.2 Ecological Interface Design Ecological interface design (EID) is spawned from cognitive work analysis (CWA), which was developed as an analytical approach to cognitive engineering by the Risø


327

group in Denmark [15]. CWA was developed to aid in the design of very critical human-machine systems such as nuclear power plant control rooms to make them safer and more reliable. It is an approach that allows the operator to handle situations that the system designers had not anticipated. CWA is made up of five phases to analyze within a system. These phases are work domain analysis, control task analysis, strategies analysis, social-organizational analysis and worker competencies analysis [16]. Having these analyses allows the designer and the operator a better understanding of the system and already this enables the operator to better respond to unforeseen events. The idea behind EID is to create interfaces based on certain principles of CWA. It is very closely related to the principles of ecological psychology and direct perception, concepts developed by J.J Gibson in the 70’s [17]. Gibson argued that there is enough information in the visual array to directly perceive information and that mental processing of visual information is not necessary. Though this claim is highly challenged, EID is largely built up around these principles in that its goal is to create interfaces containing objects that visually reveal information on their function. A related goal of EID is to make affordances visible in interface design. Affordances, another concept created by Gibson, are the action possibilities of a specific object [17, 18]. The ideas surrounding affordances and EID can also be found in other areas of the scientific literature. In product design, one tends to discuss similar issues in terms of semantics [19]. 4.3 Usability Inspections Usability inspection methods are predictive evaluation methods, usually performed without end user participation (although this is not a prerequisite). Usability experts simulate the users and inspect the interface resulting in problem lists with varying degree of severity [20]. 4.3.1 Cognitive Walkthrough A cognitive walkthrough is usually performed by usability experts considering, in sequence, all actions incorporated in a predefined task. Its focus is almost exclusively on ease of learning, not taking into account first time problems, problems that might not be a problem for the experienced user. The method contains two phases. First the preparations phase where the analyst defines the users, their experience and knowledge; defines the task to be analyzed; identifies the correct sequence of actions to achieve the goal of the task. In the second phase, the analysis phase, the analyst answers and motivates a set of questions for each action within the task [12]. 1. Will the user try to achieve the right effect? For example, if the task is to fill up the car with petrol and a button first has to be pressed from inside the car to open the gas cap, does the user know that this has to be done? 2. Will the user notice that the correct action is available? Simply pressing the button for the gas cap would not be a problem but if the button has to be slid or twisted in some way the user may not think of this. 3. Will the user associate the correct action with the desired effect? Is it clear that this is what the specific control is for? Unambiguous icons and names of controls are important to this aspect.

328


4. If the correct action is performed, will the user see that progress is being made? The importance of feedback, discussed earlier, comes into play here. These questions, though applicable to many tasks, are merely guidelines towards conducting a successful cognitive walkthrough. The method’s advantages are its focus on detail, it identifies local problems within the task and considers the users’ previous knowledge and experiences. However, it rarely catches global problems related to the overlying structure of the task and can be viewed as fairly subjective. It also requires a detailed prototype for evaluation although this would probably not be a problem if it is complementing a DHM-tool where a virtual prototype is likely to already exist. 4.3.2 Heuristic Evaluation Just as in the case of cognitive walkthrough, heuristic evaluations are usually performed by usability experts sequentially going through each action within a main task with a basis in a set of heuristics [14]. The method is developed by usability expert Jacob Nielsen and a set of his heuristics can be found through his publications [13, 14, 21]. Examples of Nielsen’s heuristics are • Match between system and the real world o Similar to the matching and mapping concept discussed in system ergonomics, the system should speak the users’ language, matching the real world in terms of terminology and semiotics. • Consistency and standards o Also related to the matching concept is using accepted conventions to avoid making users wonder whether different words, icons or actions mean the same thing in different contexts. • Recognition rather than recall o Options should be made visible to avoid making the user have to remember how or where specific actions should be performed. • Aesthetic and minimalist design o Dialogues and controls should not be littered with irrelevant or rarely used information. Heuristics can be added and subtracted to fit certain tasks before the evaluation commences. The method results in problem lists with motivations and rankings of the severity of the problems found.

5 A Design Guide for DHM The evaluation and design tools discussed in previous sections are developed for interface design in different settings than DHM. However, the design guide suggested in this section is a hybrid of these, adapted for use under the specific conditions that DHM provides. The method will strive to take into account global as well as local issues through the use of action based interface inspections and a task analysis focusing on the structure of the task. As stated earlier in this paper and by others [22], every good design process starts with a task analysis. For our purposes, a hierarchical task analysis is very suitable as it


329

complements the inspection methods incorporated in this design guide. The HTA will serve several purposes; it will give the designer a better understanding of the task and it will provide a common understanding of the task within a development group. The task analysis can also be used as an evaluation tool of the task itself. It allows the designer to identify problems in the task structure that could result in problems with automatism [23], it can identify recurring tasks and give them a higher priority in the interface etc. Complementary to the task analysis, the designer should consider who the users are and what a priori knowledge they have. This resembles the guiding system for utilizing traditional DHM tools in development processes suggested by Hanson et al. [24], where the users’ anthropometry and tasks are defined before the actual analyses or simulations are performed. The sequence-based walkthrough will take its basis in the task analysis performed. For each subtask (box) of the HTA, a set of questions, based on Bubb’s points regarding system ergonomics [9], will act as guidelines for the design. • Function – When and where should the action be performed? o Will the user identify the action space where the correct action should be performed? What do the physical and geographical properties of each control convey to the user? o Frequency of actions – a frequently recurring action should take precedence in taking up place and intrusiveness in the physical and cognitive envelope. o Importance of action – Safety critical systems should also take precedence in the available information space. o Minimalism of design – avoid taking up space with irrelevant or rarely needed information. Hick’s law: Reaction time is a function of the number of choices in a decision [25]. • Feedback o Will the user understand that a correct or faulty move has been made? o Is the system status visible? • Compatibility o Does the system match other, similar systems in terms of semantics, semiotics etc.? o Does the system match the real world and the plausible mental model of the user? o Are demands on consistency and standards of the domain met? o Action-effect discrepancies – is it obvious beforehand that a certain action will have a certain effect? 5.1 Function In figure 3, there is an example of what a virtual interface, modelled in a DHM-tool can look like. In this case the picture shows a fighter jet cockpit used for evaluation where the pilot needed to locate a “panic button” to bring the aircraft back into control under extreme physical and mental conditions.

330


Fig. 3. Two views of a cockpit modeled in SAMMIE

The action spaces that the user has to identify when performing an action are the controls in front of, and to the right and left of the steering control stick. Preferably, a frequently performed action control should be placed on the control stick or directly in front of it as these are the spaces that best correspond to the physical and cognitive reach of the pilot. Also safety systems, as in the case of the evaluation in figure 3, should be placed so that they are easily accessible for the user. Knowing that certain controls are rarely used, they can be placed to the right and left to avoid having too many options in terms of pushable buttons at the same place. The intrusiveness and affordances of such “high priority controls” should also be accentuated in terms of their design. 5.2 Feedback Understanding what has been done and what is in progress of happening with the system can prove vital in many cases. Surely we can all relate to a situation where we have pressed the print button more than once only to find out that we have printed several more copies than needed. While this may be a minor problem, one can easily imagine the problems that can arise in more critical domains. What if there were no indications for what gear the car’s gearbox was in? The driver would have to test each time to see if the car is in reverse or drive. In an incident at a hospital, a patient died as a result of being exposed to a massive overdose of radiation during a radiotherapy session. The problem could easily have been avoided, had the system provided the treating radiology technician with information of the machines settings [26]. 5.3 Compatibility Accurate mapping between systems and mental models is a key concept in the compatibility section. This includes trying to adhere to consistencies and standards of the organization and the specific field. There should also be clear connection between action and effect. Neglecting these consistencies can lead to serious problems as in the case with an aircraft’s rudder settings. The sensitivity of the rudder could be set through a lever placed to the side of the pilot’s seat. However, between the simulator for the aircraft and the actual aircraft, the lever was reversed, moving in the opposite direction for maximum and minimum sensitivity almost resulting in a crash [27].


331

6 Conclusions and Future Work In ergonomics, it seems to be common practice to separate human factors into “neck up” and “neck down”. Though this approach may make it easier to study ergonomics, it does not portray an entirely accurate picture of the human. The evidence for a tight coupling between mind and body is so overwhelming that instead of talking about mind and body, perhaps we should be talking about the human system. The aim of this paper has been to consider past and current approaches towards integrating cognition into DHM tools and outline a new design guide to help designer achieve this integration in a better way. The guide is not complete and would need extensive further development and testing. However, it is a good start towards including something more into DHM than traditionally has been found there.

References 1. Bubb, H.: Future Applications of DHM in Ergonomic Design. In: Duffy, V.G. (ed.) HCII 2007 and DHM 2007. LNCS, vol. 4561, pp. 779–793. Springer, Heidelberg (2007) 2. Case, K., Porter, J.M.: SAMMIE - A Computer Aided Ergonomics Design System. Engineering 220, 21–25 (1980) 3. Bernard, M.L., Xavier, P., Wolfenbarger, P., Hart, D., Waymire, R., Glickman, M., Gardner, M.: Psychologically Plausible Cognitive Models for Simulating Interactive Human Behaviors. In: Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting, pp. 1205–1210 (2005) 4. Carruth, D.W., Thomas, M.D., Robbins, B., Morais, A.: Integrating Perception, Cognition and Action for Digital Human Modelling. In: Duffy, V.G. (ed.) HCII 2007 and DHM 2007. LNCS, vol. 4561, pp. 333–342. Springer, Heidelberg (2007) 5. Gore, B.F.: Human Performance: Evaluating the Cognitive Aspects. In: Duffy, V.G. (ed.) Handbook of digital human modelling, Mahwah, New Jersey (2006) 6. Clark, A.: Being There: Putting Brain, Body, and World Together Again. MIT Press, Cambridge (1997) 7. Feyen, R.: Bridging the Gap: Exploring Interactions Between Digital Human Models and Cognitive Models. In: Duffy, V.G. (ed.) HCII 2007 and DHM 2007. LNCS, vol. 4561, pp. 382–391. Springer, Heidelberg (2007) 8. Barsalou, L.W., Niedenthal, P.M., Barbey, A.K., Ruppert, J.A.: Social Embodiment. In: Ross, B.H. (ed.) The Psychology of Learning and Motivation, pp. 43–92. Academic Press, San Diego (2003) 9. Bubb, H.: Computer Aided Tools of Ergonomics and System Design. Human Factors and Ergonomics in Manufacturing 12, 249–265 (2002) 10. Norman, D.A.: The design of everyday things. Basic Books, New York (2002) 11. Annett, J.: Hierarchichal Task Analysis. In: Diaper, D., Stanton, N. (eds.) The Handbook of Task Analysis for Human-Computer Interaction, pp. 67–82. Lawrence Erlbaum Associates, Mahwah (2003) 12. Polson, P.G., Lewis, C., Rieman, J., Wharton, C.: Cognitive Walkthroughs: A Method for Theory-Based Evaluation of User Interfaces. International Journal of Man-Machine Studies 36, 741–773 (1992) 13. Nielsen, J.: Usability Engineering. Morgan Kaufmann, San Francisco (1993)

332


14. Nielsen, J.: Heuristic evaluation. In: Nielsen, J., Mack, R.L. (eds.) Usability inspection methods, pp. 25–62. John Wiley & Sons, Inc., New York (1994) 15. Vicente, K.J.: Cognitive Work Analysis: Toward Safe, Productive, and Healthy ComputerBased Work. Lawrence Erlbaum Assoc. Inc., Mahwah (1999) 16. Sanderson, P.M.: Cognitive work analysis. In: Carroll, J.M. (ed.) HCI models, theories, and frameworks: Toward an interdisciplinary science, pp. 225–264. Morgan Kaufmann Publishers, San Francisco (2003) 17. Gibson, J.J.: The ecological approach to visual perception. Lawrence Erlbaum Associates, Hillsdale (1986) 18. McGrenere, J., Ho, W.: Affordances: Clarifying and Evolving a Concept. In: Proceedings of Graphics Interface 2000, pp. 179–186 (2000) 19. Monö, R.: Design for Product Understanding. Skogs Boktryckeri AB (1997) 20. Nielsen, J., Mack, R.L.: Usability inspection methods. Wiley, Chichester (1994) 21. Nielsen, J.: Finding usability problems through heuristic evaluation. In: Proceedings of ACM, Monterey, CA, pp. 373–380 (1992) 22. Pheasant, S., Haslegrave, C.M.: Bodyspace: Anthropometry, Ergonomics and the Design of Work. CRC Press, Boca Raton (2006) 23. Thorvald, P., Bäckstrand, G., Högberg, D., de Vin, L.J., Case, K.: Demands on Technology from a Human Automatism Perspective in Manual Assembly. In: Proceedings of FAIM 2008, Skövde, Sweden, vol. 1, pp. 632–638 (2008) 24. Hanson, L., Blomé, M., Dukic, T., Högberg, D.: Guide and documentation system to support digital human modelling applications. International Journal of Industrial Ergonomics 36, 17–24 (2006) 25. Hick, W.E.: On the rate of gain of information. The Quarterly Journal of Experimental Psychology 4, 11–26 (1952) 26. Casey, S.M.: Set phasers on stun and other true tales of design, technology, and human error. Aegean, Santa Barbara (1998) 27. Casey, S.M.: The atomic chef: and other true tales of design, technology, and human error. Aegean Pub. Co., Santa Barbara (2006)

Workload-Based Assessment of a User Interface Design Patrice D. Tremoulet1, Patrick L. Craven1, Susan Harkness Regli1, Saki Wilcox1, Joyce Barton1, Kathleeen Stibler1, Adam Gifford1, and Marianne Clark2 1

Lockheed Martin Advanced Technology Laboratories 3 Executive Campus, Suite 600, Cherry Hill, NJ, USA {polly.d.tremoulet,patrick.craven,susan.regli, sakirenecia.h.wilcox,joyce.h.barton,kathleen.m.stibler, adam.gifford}@lmco.com 2 2001 South Mopac Expressway, Suite 824, Austin, TX, USA [email protected]

Abstract. Lockheed Martin Advanced Technology Laboratories (LM ATL) has designed and developed a tool called Sensor-based Mental Assessment in Real Time (SMART), which uses physiological data to help evaluate humancomputer interfaces (HCI). SMART non-intrusively collects and displays objective measures of cognitive workload, visual engagement, distraction and drowsiness while participants interact with HCIs or HCI prototypes. This paper describes a concept validation experiment (CVE) conducted to 1) demonstrate the feasibility of using SMART during user interface evaluations and 2) validate the EEG-based cognitive workload values derived from the SMART system by comparing them to three other measures of cognitive workload (NASA TLX, expert ratings, and expected workload values generated with Design Interactive’s Multimodal Information Decision Support tool). Results from the CVE indicate that SMART represents a valuable tool that provides human factors engineers with a non-invasive, non-interrupting, objective method of evaluating cognitive workload. Keywords: Cognitive workload, human computer interaction, human factors, usability, evaluation, user interface design.

1 Introduction In 2005 and 2006, the Office of Naval Research (ONR) Disruptive Technologies Opportunity Fund supported Lockheed Martin Advanced Technology Laboratories’ (LM ATL) research effort exploring the use of neuro-physiological data to measure cognitive workload during a human-computer interface (HCI) evaluation. As a part of this effort, LM ATL designed and developed a tool called Sensor-based Mental Assessment in Real Time (SMART), which non-intrusively collects physiological data (electroencephalographs (EEG), heart rate variability (HRV), galvanic skin response and pupil size) from subjects while they interact with HCIs or HCI prototypes. SMART uses the data to derive objective measures of cognitive workload, visual engagement, distraction and drowsiness, which may be used to evaluate the efficacy of design alternatives, e.g., by helping to identify events of interest during V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 333 – 342, 2009. © Springer-Verlag Berlin Heidelberg 2009

334

P.D. Tremoulet et al.

usability tests, thus reducing data load and providing timely evaluation results to suggest design changes or to validate design. SMART’s sensor-based measure of cognitive workload offers several advantages over existing workload measures, including: 1) increased precision in measuring the subject’s cognitive state via a moment-by-moment data collection, 2) obtaining objective measurement of cognitive workload while the test is being performed, and 3) not distracting subjects from their primary task (e.g., by interrupting to collect subjective ratings or requiring them to attend to a secondary task). However, SMART’s cognitive workload measure needed to be validated, so LM ATL conducted a concept validation experiment (CVE) to demonstrate the feasibility of using SMART during user interface evaluations as well as to collect data necessary to validate the sensor-based cognitive workload values derived from the SMART system by comparing them to NASA TLX, expert ratings, and Multimodal Information Decision Support (MIDS) expected values. In most respects, the CVE was similar to a traditional usability study. Twelve sailors interacted with a high-fidelity prototype of a future release of the Tactical Tomahawk Weapons Control System (TTWCS), performing tasks required to execute missile strike scenarios. However, there were two major differences. First, the CVE scenarios were designed not only to be operationally valid but also to include discrete phases that require specific levels of workload. Moreover, while interacting with the prototype, participants in the CVE wore a set of neurological and physiological sensors including a wireless continuous EEG and Electrocardiogram (EKG) sensor, wired EKG (the wired EKG was used as a backup for the newer wireless configuration) and galvanic skin response (GSR) sensors, and an off-head, binocular eye tracker that logs point of gaze and pupil diameters. 1.1 Sensor-Based Mental Assessment in Real-Time (SMART) SMART provides critical support for interpreting physiological data collected during usability evaluations. SMART logs and displays system events, physiological responses, and user actions while study participants continue to interact with system of interest, in this case a prototype of a future TTWCS system. Advances in neurotechnology and physiological measurements have enabled information to be captured that helps identify and indicate psychological states of interest (e.g., boredom, overloaded, and engaged), which aid and human factors engineers in the evaluation of an HCI. SMART provides four critical types of information during usability evaluations: • Real-Time Logging: Experimenters have the ability to enter events into the log to represent significant events that are not automatically logged. Prior to testing, the experimenter can set up events of interest with a quick key identifier so that expected events can be manually logged during testing. • Real-Time Monitoring: During testing, experimenters can log events and monitor physiological sensors data (Fig. 1). • Time Synchronization: Time synchronization between the physiological sensors logs and the test system logs is crucial in accurately matching sensor derived events

Workload-Based Assessment of a User Interface Design

335

to test platform and participant driven events. After testing, the experimenter can view data via the Timeline summary (Fig. 2). • Data Extraction: Data is also extracted and presented in a CSV format, suitable for uploading into standard statistical analysis applications.

Fig. 1. Real-time monitoring

Fig. 2. Timeline Summary

1.2 SMART’s Cognitive Workload Measure Lockheed Martin Advanced Technology Laboratories (LM ATL) worked with Advanced Brain Monitoring (ABM) to develop an Electroencephalogram (EEG)based gauge of cognitive workload [2] [4]. LM ATL employees wore ABM’s EEG acquisition system while performing a variety of classic experimental psychology tasks in which working memory load was varied (e.g., by varying the number of items that needed to be remembered in an N-back task). The EEG acquisition system measured the electrical activity of the participants’ brains with sensors placed on the scalp, allowing data to be captured unobtrusively. The EEG signals reflect the summated potentials of neurons in the brain that are firing at a rate of milliseconds. Discriminant function analyses determined appropriate coefficients for linearly combining measures derived from continuous EEG into a cognitive workload index, which ranges from 0 to 1.0 (representing the probability of being classified as “high workload”). The index values were validated against an objective appraisal of task difficulty and subjective estimates of workload and task focus. The cognitive workload index derived from this research effort increases with increasing working memory load and during problem solving, mental arithmetic,

336


integration of information, and analytical reasoning and may reflect a sub-set of executive functions. The cognitive workload levels from this index are significantly correlated with objective performance and subjective workload ratings in tasks with varying levels of difficulty including forward and backward digit span, mental arithmetic and N-back working memory tests [4] [3].

2 Method A Concept Validation Experiment (CVE), patterned after an HCI usability study in which active-duty personnel run operationally valid scenarios designed such that different phases elicit different levels of cognitive workload, was conducted in Norfolk, VA, from August 22-31, 2006. . 2.1 Participants Average age of the twelve active-duty participants was 29, and their years of service ranged from 1-18 years. Six participants had experience with one of the previous Tomahawk systems. All but one had some experience at Tomahawk workstations (averaging over six years) and had participated in an average of 52 Sea-Launched Attack Missile Exercises (SLAMEX) and 28 theatre exercises. Two participants had experience participating in an operational launch. All participants were male; one was left-handed, and two reported corrected 20/20 vision. 2.2 Equipment and Materials Testing was conducted at offices at the Naval Station in Norfolk, VA. The TTWCS prototype was simulated on a Dell Inspiron 8500 laptop connected to two, 19-inch flat panel LCD monitors displaying the prototype software. The monitors were positioned vertically to re-create the configuration of the actual Tomahawk workstation. The prototype system recorded timestamps for system and user events to a log file. One video camera was located behind the participant, and video was captured on digital video tape. The output from the console monitors was sent through splitters to two 19-inch displays to allow observers to view the screen contents from a less intrusive location. Continuous EEG data was acquired from a wireless sensor headset developed by ABM using five channels with the following bi-polar montage: C3-C4, Cz-POz, F3Cz, Fz-C3, Fz-POz. Bi-polar differential recordings were selected to reduce the potential for movement artifacts that can be problematic for applications that require ambulatory conditions in operational environments. Additionally, EKG information was collected on a sixth channel. Limiting the sensors (six) and channels (five) ensured the sensor headset could be applied within 10 minutes. The EEG data were transmitted via a radio frequency (RF) transmitter on the headset to an RF receiver on a laptop computer containing EEG collection and processing software developed by ABM. This computer was then linked to an Ethernet router that sent the EEG data to SMART. Wired galvanic skin response (GSR) and EKG sensors were connected to a ProComp™ Infiniti conversion box, which transmitted the signal via optical cable to


337

two destinations. The first destination was a laptop containing Cardio Pro™ software, and to the SMART laptop where software logged the GSR data and used the EKG signal to calculate heart rate variability (HRV). Eye gaze and pupil dilation data were collected using the SmartEye™ system in which two cameras were positioned on either side of the monitor. An infrared emitter near each camera allowed the SmartEye™ software to calculate the associated eye movement and dilation data. Paper questionnaires included a background questionnaire and a user satisfaction questionnaire. An electronic version of the NASA TLX was administered on the TTWCS task machine in between task phases. The test administrator recorded notes concerning the testing session and completed rating of the effort observed by the participant during the test session. Subsequent to the test session, aspects of the scenarios and individual user behavior were coded and entered into the MIDS tool. This tool produces a second-by-second total workload value for the entire scenario. 2.3 Experimental Design The SMART workload validation was a correlational design in which the validity of a novel measure (sensor-based cognitive workload) would be compared with other measures that typify accepted practices for measuring workload within the HCI and Human Factors engineering research communities. 2.4 Procedure Each participant’s test session lasted approximately eight hours. The participants were briefed on the purpose and procedures for the study and then completed a background questionnaire with information regarding demographics, rank, billet, and Tomahawk experience. After the questionnaire was completed, a cap was placed on the participant’s head and six EEG sensors were positioned in it (at F3, Fz, C3, Cz, C4, and Pz). Two EKG sensors were placed on the right shoulder and just below the left rib. The participant was instructed to tell the testing staff if they were uncomfortable before or during the experimental session. An impedance check was done to ensure that interference to the signals was at 50 ohms or below; once this was verified, the RF transmitter and receiver were turned on. The participant was then asked to perform three tasks over 30 minutes to establish an individual EEG baseline and calibrate the sensors. The tasks required reading instructions and performing basic visual and auditory monitoring tasks. Next, individual profiles were created for the SmartEye™ eye tracking system by taking pictures while the participant looked at five predetermined points on the displays. Additionally, during hands-on training, specified facial features were marked on the facial images to further refine the participant’s profile. Finally, three EKG sensors were attached, one on the soft spot below each shoulder and one on the center of the abdomen, and two GSR sensors were placed on the second and fourth toes such that the sensor was on the “pad” of the toe. A towel was wrapped around the participant’s foot to keep it from getting cold.

338

P.D. Tremoulet et al. Table 1. Sensor devices used in CVE Sensor Vendor

Description

ABM

Collects EEG data from five channels and EKG on a sixth channel

SmartEye™

Collects point-of-gaze and pupilometry data

Thought Technology

Collects EKG and GSR data HRV calculated through third-party software

Hardware Six EEG electrodes Two EKG electrodes ABM sensor cap RF transmitter and receiver Two cameras Two infrared-flashes ProComp Infiniti converter Two GSR electrodes Three EKG electrodes

Once all sensors were applied and were collecting data, SMART software was started and began collecting data. Software from ABM, SmartEye™, and CardioPro™ were used throughout the experiment to monitor the signals from the sensors (Table 1). Network time protocol (NTP) was used to ensure that all machines were synchronized to within a fraction of a millisecond. Once the sensor equipment was calibrated and system software was running, participants were trained through lecture and given hands-on practice using the prototype. The lecture portion of training consisted of PowerPoint slides, which gave an overview of the system, as well as detailed pictures and descriptions of the windows and interactions that would be part of the participant’s experimental task. During the hands-on practice, a member of the testing staff guided the participant through sample tasks. The participant performed one practice trial using the system, during which various scenario tasks were accomplished. The testing staff answered any questions that the participant had during training. Training took one and a half hours (one hour of lecture and thirty minutes of hands-on practice). After training, participants were given a thirty-minute break. During the experimental test session, the participants were presented with a scenario that included various tasks associated with preparing and launching Tomahawk missiles, similar to those presented during training. The participant was asked to respond to system events and prompts. An experimenter observed and recorded errors and additional observations using SMART and supplemental written notes. SMART logged the sensor data and the objective cognitive measures derived from it, and the TTWCS prototype logged user and system events throughout the test session. The sessions were also video and audio recorded. The video showed only the back of the participant’s head and the two computer screens. In the test scenario, participants were asked to execute two, 10-missile salvos from receipt of the strike package until the missiles launched for first strike package. The main criterion for successful human-machine performance in this cognitive task environment was the degree to which missiles were successfully launched at their designated launch time (T=0). The scenario took approximately one hour and 15 minutes to complete. The scenario was divided into four task phases (each desired to have a different level of workload) that occurred in a specific sequence for a strike:


339

• Phase 1: Initial validation with no error and engagement planning with errors • Phase 2: Validation due to receipt of execute strike package, preparation for execution, and emergent targets • Phase 3: Monitor Missiles • Phase 4: Launch missiles with emergent targets and receive and prepare second strike package At the end of each phase, the prototype was paused and the participant asked to fill out a questionnaire on perceived workload (NASA TLX). Then the scenario was resumed. After the fourth phase, the sensors were removed and the participant was asked to fill out a questionnaire on the perceived satisfaction with the system and then was debriefed to discuss any questions that they had. 2.5 Data Preparation Four measures of cognitive workload were collected during the test sessions: • Sensor cognitive workload (EEG-based cognitive workload value): Scores from the neurophysiologically-based gauges (CW-EEG) were generated by taking logs of second-by-second gauge output and averaging the workload values during the four different task phases. Data points associated with noisy sensor readings were removed before analysis ([1] provides a description of this procedure.) • NASA TLX (generated by participant survey): Total workload scores from the NASA TLX were used in the analyses. The total workload score is comprised of weighted scores of the six subscales (mental demand, physical demand, temporal demand, performance, effort, and frustration). One participant’s data was removed because he rated all four task phases in both scenarios the same. • MIDS (generated by expert observation and cognitive conflict algorithm): Scores for the MIDS measure were generated for approximately half the participants. Close examination of the task domain and videotape of the study allowed for generation of estimates of workload for individual sensory channels (i.e., visual, auditory, and haptic), cognitive channels (i.e., verbal, spatial) and response channels (i.e., motor, speech). Workload was calculated based on a task timeline by summing (1) the purely additive workload level across attentional channels, (2) a penalty due to demand conflicts within channels, and (3) a penalty due to demand conflicts between channels. The amount of attention the operator must pay to each channel in the performance of each task used a 5-point subjective rating scale (1: very low attentional demand; 5: very high attentional demand). These estimates were combined to create an overall cognitive workload estimate for each second that the participant was completing the scenarios ([5] provides details of this technique.) Values from the six participants were averaged and correlations were made with average values of the other measure. • Expert Rating (generated by expert observation): The expert ratings were generated by three HCI experts using a seven-point Likert scale. Experts rated the participants on six dimensions of interaction with the task. The mental effort rating was then extracted and scaled by rater to help eliminate individual rater tendencies.

340


3 Results As described above, four measurements were collected during each of the four phases of the test scenario. The measures estimated workload using either an external observer who monitored the actions of the participant, by the participant himself, or through the use of physiological-sensor-based gauges. The results are presented below first in terms of descriptive statistics and then in terms of correlations of the measures. 3.1 Descriptive Statistics Table 2 shows the mean values for the four measures. For all four measures, Phase 4 had the highest mean value. During this phase, participants were required to launch missiles while simultaneously preparing a second strike package. They also were given emerging targets, requiring quick response. For CW-EEG, Expert Rating and MIDS, Phase 3 was the lowest mean value. During this phase, the participants were primarily performing a vigilance task as the missiles were being prepared for launch, which required no specific interaction with the TTWCS HCI prototype. Table 2. Overall descriptive statistics Metrics CW-EEG (0 to 1.0)

NASA TLX (0 to 100)

Phase

Mean

Phase 1 Phase 2 Phase 3 Phase 4 Phase 1 Phase 2 Phase 3 Phase 4

0.703 0.702 0.659 0.708 22.56 28.62 24.64 41.56

Metrics Expert Rating (1 to 7) MIDS (1 to 5)

Phase

Mean

Phase 1 Phase 2 Phase 3 Phase 4 Phase 1 Phase 2 Phase 3 Phase 4

2.75 3.46 2.04 4.78 11.31 11.11 3.54 13.31

3.2 Validation of EEG-Based Cognitive Workload Validation of the EEG-based cognitive workload index (CW-EEG) was performed by computing correlations among the scores (NASA TLX, expert rating) or average scores (CW-EEG, MIDS) across subjects of the four measures collected during the CVE. Table 3 lists the Pearson’s product-moment correlation coefficient (r) of four measures of cognitive workload. Note that MIDS data was only coded on half the participants’ data). Table 3. Workload measures correlations table Metrics

CW-EEG

NASA-TLLS

Expert

MIDS

CW-EEG NASATLX Expert Ratings MIDS

-0.19^ 0.38**

0.19^ -0.34**

0.38** 0.34** --

0.51** 0.40* 0.56**

0.56**

--

0.51** 0.40* **p 0, for 1 ≤ i ≤ c, and 1 ≤ k ≤ n, then get the new uik. c

Otherwise if dik > 0, and uik = [0, 1] with

∑u

( new ) ik

= 1 , then uik(new) = 0.

i =1

Step 12: Until || U(new) − U(old) || < ε where ε is the termination tolerance ε > 0. If this condition is not satisfied, then go back to step 9.

5 Analysis and Results Before the proposed data reduction algorithms are applied into the image segment data, the image segment data need to be examined whether the data can be diminished by the redundancy among its original variables with the highly correlated interrelationship. To examine the redundancy, the correlations between the variables of the

714

D.H. Nam and E. Asikele

image segment data set are calculated. As shown in the Table 2, the correlations of the “Brickface” image segment data are presented. The bolded numbers are showing the relatively higher correlation so that there is a possibility to be extracted as a new factor between those measurements. Table 2. Pearson’s correlation values for the “Brickface” image segment data

Region centroid column Region centroid row Vedge-mean

Region centroid column 1

Region centroid row

0.333

1

Vedgemean

Hedgemean

Raw red mean

Raw blue mean

-0.165

-0.266

1

Hedge-mean

-0.015

-0.194

0.351

1

Raw red mean

0.008

-0.729

0.33

0.412

Raw blue mean

0.004

-0.691

0.334

0.408

0.993

1

Raw green mean

-0.017

0.675

-0.248

-0.388

-0.808

-0.747

Raw green mean

1 1

In addition, there are different criterions to select the reduced dimension for the new reduced variables after extracting new variables from the original data. For this example, two combined criterions are applied. One is the eigenvalues-greater-thanone rule by Cliff [10] and the second criterion is the accumulated variance that is more than 0.9 from the reduced system. Using two categories, 4 newly extracted variables are selected among 7 different measurement variables.

Scree Plot of C1, ..., C7 4

Eigenvalue

3

2

1

0 1

2

3

4 Factor Number

5

6

7

Fig. 1. Scree plot for the newly extracted components for “Brickface” image segment data

Data Mining of Image Segments Data with Reduced Neurofuzzy System

715

From Table 3, the evaluated analyses of the performance using the proposed algorithms through the neurofuzzy systems [4][5][7] are shown. From the results of Table 3, the results from the methods using factor analysis [1] and FCM clustering show relatively better results than other methods including the combinations of principal component analysis and FCM clustering. Table 3. Analyses of Performance using proposed algorithm and conventional factor analysis and principal component analysis

fa pca fc pc

CORR 0.3984 0.109 0.3652 -0.2541

TRMS 0.8901 1.5185 0.8807 1.2676

STD 0.6664 1.1062 1.3249 1.2878

MAD 0.8826 1.5059 0.8697 1.2517

EWI 3.0407 5.0215 3.7101 4.5529

6 Conclusion The pattern recognition of image segment data has been implemented through the neurofuzzy systems using the reduced dimensional data in variables and observations. For the implementation, four newly extracted embedded new variables from 7 original measurements variables are used. The proposed algorithm performs the relatively better results than using the conventional multivariable techniques by themselves. As described in Table 3, using the combination of factor analysis and FCM clustering analysis, the prediction of the patterns for the image segment data shows the relatively better results. The prediction results using the conventional principal component analysis shows relatively worse than using the proposed algorithm. This result may lead to the conclusion that for a limited number of input-output training data, the proposed algorithm can offer the better performance in comparison with the performances of the other techniques for image segment data.

Acknowledgments This material is based upon work supported by Clarkson Aerospace Corporation.

References 1. Gorsuch, R.L.: Factor Analysis, 2nd edn. Lawrence Erlbaum Associates Inc., Hillsdale (1983) 2. Kendall, M.: Multivariate Analysis. MacMillan Publishing Co. INC., New York (1980) 3. Yager, R., Filev, D.P.: Essentials of Fuzzy Modeling and Control. John Wiley & Sons, New York (1994) 4. Lin, C., Lee, C.: Neural Fuzzy Systems. Prentice Hall, Englewood Cliffs (1996) 5. Jang, J.S.: ANFIS: Adaptive Network Based Fuzzy Inference System. IEEE Trans. Systems, Man and Cybernetics 23(3), 665–684 (1993)

716

D.H. Nam and E. Asikele

6. Duda, R., Hart, P., Stork, D.: Pattern Classification, 2nd edn. John Wiley & Sons, Chichester (2001) 7. Fuzzy Logic Toolbox, for use with the MATLAB. The Math Works Inc. (2003) 8. Image segment data provided by Vision Group, University of Massachusetts, http://www.cs.toronto.edu/~delve/data/datasets.html 9. Nam, D., Singh, H.: Material processing for ADI data using multivariate analysis with neuro fuzzy systems. In: Proceedings of the ISCA 19th International Conference on Computer Applications in Industry and Engineering, Las Vegas, Nevada, November 13–15, pp. 151–156 (2006) 10. Cliff, N.: The Eigenvalues-Greater-Than-One Rule and the Reliability of Components. Psychological Bulletin 103(2), 276–279 (1988)

Appendix: Abbreviations CORR: Correlation n

TRMS: Total Root Mean Square

TRMS =

∑

( xi − y i ) 2

i =1

n −1

where xi is the estimated value and yi is the original output value. STD: Standard Deviation MAD: Mean of the absolute EWI [9]: Equally Weighted Index, the index value from the summation of the values with multiplying the statistical estimation value by its equally weighted potential value for each field fa: Factor Analysis pca: Principal Component Analysis FCM: Fuzzy C-means Clustering Analysis fc: preprocessing FA and SUBCLUST pc: preprocessing PCA and SUBCLUST.

The Impact of Change in Software on Satisfaction: Evaluation Using Critical Incident Technique (CIT) Akshatha Pandith1, Mark Lehto1, and Vincent G. Duffy1,2,3 1

Industrial Engineering, Purdue University, 315 N. Grant Street, West Lafayette, IN 47907, USA 2 Agricultural and Biological Engineering, Purdue University 3 Regenstrief Center for Health Engineering, Purdue University {apandith,lehto,duffy}@purdue.edu

Abstract. This paper describes an exploratory study that analyzes the impact of change in software on users by utilizing the Critical Incident Technique (CIT). A total of 102 critical incidents were collected from the survey. 77 participants reported both satisfactory and unsatisfactory experiences; 22 reported only satisfactory experiences; and 3 reported only unsatisfactory experiences. Analysis of satisfactory or unsatisfactory experiences revealed several factors such as expectations of users and mismatch in the behavior between the actual and anticipated system by the users, which can be attributed to automation surprise. The important findings of this study are the agglomeration of user feedback such as, avoiding the changes themselves in the first place, focusing on the factors of change viz. amount of change, speed of change, and finally, to provide better help support, which can be used in the design process when there is a change in software. Keywords: Critical Incident Technique, Change in software, Impact of change, Information overload, Automation surprise.

1 Introduction This paper aims to understand the impact of change in software. It is observed that change in any area is inevitable and so are its effects. Whether change is viewed as positive or negative depends not only on the outcome of the change, but also on the degree of influence it exerts on the situation [1]. A change is viewed as negative when people are unable to foresee it, when people dislike its implications, and when people feel unprepared for its effects. In other words, unrealistic expectations, willingness or commitment to change and approach to change are factors that may lead to satisfactory or unsatisfactory experiences. Change is perceived to be positive when people feel in control, are able to accurately anticipate the events and influence the immediate environment or at least prepare for the consequences [1, 3]. Willingness to change is influenced by ability and desire. Deficiencies in ability to adapt to change resulting from inadequate skills should be addressed by training. Information overload has been another problem that’s caused impact on pilots of fighter aircrafts and attack helicopters due to change, as they have to process large amounts of information and make V.G. Duffy (Ed.): Digital Human Modeling, HCII 2009, LNCS 5620, pp. 717–726, 2009. © Springer-Verlag Berlin Heidelberg 2009

718

A. Pandith, M. Lehto, and V.G. Duffy

decisions within split seconds [2, 3]. The same concept of information overload can be applied to understand the burden experienced by the users during change in software Automation surprise can be one of the reactions to a system that undergoes a change. A reliable indicator of how people respond to change is the degree of surprise they exhibit when they encounter the change [4]. In other words, surprises occur when people anticipate one thing and instead experience something drastically different. It is sometimes difficult for the human operator to track the activities of their automated partners [5]. The result can lead to situations where the operator is surprised by the behavior of the automation asking questions like, “what is it doing now?”, “why did it do that?”, or “what is it going to do next?” [6]. Thus, automation has created surprises for practitioners, who are confronted with unpredictable and complicated system behavior, in the context of ongoing operations [7]. Information overload is another important factor that should be considered in software change settings. As more and more icons and features are added, it can create a sense of data overload to the user. The psychologist David Lewis proposed the term "Information Fatigue Syndrome" to describe the resulting symptoms. Other effects of too much information include anxiety, poor decision-making, difficulties in memorizing and remembering, and reduced attention span [8]. These effects merely add to the stress caused by the need to constantly adapt to a changing situation. In this research, the Critical Incident Technique (CIT) is used in an exploratory study to capture the impact of change on users and the way users perceive changes. The captured incidents are then classified based on satisfactory or unsatisfactory experiences [9]. The data has been further classified based on the impact of change on performance, acceptance, and satisfaction. The causes for such effects or surprises have been analyzed in order to reduce negative experiences. Finally, several suggestions are proposed to make the change process smoother and easier, to enhance the overall experience, and also to make use of new software in its intended way.

2 Method 2.1 Critical Incident Technique Critical Incident Technique (CIT) has been used in this study to identify the problems caused by automation surprises and other change related events. CIT is a research method developed by John C. Flanagan to capture human behavior and can be defined as a set of procedures for systematically identifying behaviors that contribute to success or failure of individuals or organizations in specific situations. CIT relies on the idea that the critical incidents will not only be memorable but also have either a positive or negative effect [10]. These incidents can be acquired through interviews, observations, or self-reporting. Some of the advantages of this method are: it is performed by real users in the normal working environment, users self-report the critical incident, there is no direct interaction needed between user and the evaluator during an evaluation session. Also, data capture is cost-effective, high quality and therefore relatively easy to apply to study usability [11].

The Impact of Change in Software on Satisfaction: Evaluation Using CIT

719

2.2 Questionnaire Design The interview questionnaire was derived from samples of other studies that were used in the CIT [10, 12, 13]. Several parameters were considered for developing the questionnaire in an attempt to capture the effect of change. The content of the questionnaire consisted of multiple choice questions, as well as open-ended and subjective rating scale questions. Open-ended questions were developed in order to understand the expectations of the users in depth. The responses were first classified into satisfactory and unsatisfactory experiences and further were classified based on the impact of change on key parameters, such as performance, satisfaction and acceptance. 2.3 Data Collection A pilot study involving 7 subjects was conducted to refine the interview questions and the data-collection methodology. In the standardized semi-structured telephone or face-to-face interview, participants were asked to describe satisfactory and unsatisfactory critical incidents and the circumstances leading to such incidents. Details about additional incidents and possible remedies to reduce such incidents were also collected, if provided. A total of 433 people were contacted for interviews. Approximately 340 telephone calls were made, out of which 58 people accepted to participate in the survey. This led to a 17% success rate with regards to this interviewing method. Ninety three people were contacted for face-to-face interviews, of which 56 agreed to participate in the survey subsequently leading to 60% success rate of face-to-face interviews. Out of a total 114 survey responses, a total of 102 completed surveys were further used in the analysis. The criterion for screening the required participants was individuals having more than a minimum of 2 years experience in using the software.

3 Results and Discussion The results obtained from the open-ended question segments of the questionnaire are presented in detail below for unsatisfactory and satisfactory experiences. Pareto analysis was conducted in order to better understand the factors causing satisfactory and unsatisfactory experiences. A Chi-square analysis was conducted to determine the statistical differences in the types of change between satisfactory and unsatisfactory experiences. Additionally, logit analysis provided an estimate of the log likelihood ratio of satisfactory to unsatisfactory experiences for various categories of changes. The final section presents the subjective rating responses along with descriptive statistics, t-tests as well as correlation and regression analyses of the subjective responses. 3.1 Unsatisfactory and Satisfactory Experiences The most significant unsatisfactory and satisfactory experiences that the users cited to have experienced during change are shown in Table 1. It can be gathered from the interviews that people complained the most about wasting time along with a waste of effort and cause for frustration upon switching to a different version of software. These are

720

A. Pandith, M. Lehto, and V.G. Duffy Table 1. Frequency of Reasons Cited by Subjects

Unsatisfactory Experiences Problem area Waste of time Waste of effort Frustration Annoying Had to re-learn No feedback from system Discomfort to use No additional benefits Too much data Waste of money Requires additional attention

Freq 50 39 24 9 8 7 6 6 3 2 2

Satisfactory Experiences Reasons for Satisfaction Saves time Saves effort Increases feel good factor Reduces error Easy to use Easy to learn Better quality output Easy to understand More reliability Increases comfort Easy to navigate

Freq 80 37 27 10 7 6 6 4 4 3 2

Table 2. Comparisons between Satisfactory and Unsatisfactory Experiences Based on ChiSquared Test Type Of Change

Freq

Negative Experience

Positive Experience

Chi Square

Critical Value

Added

61

22

39

5.23

0.0222

Enhanced

46

31

15

1.20

0.2734

Replaced

17

13

4

6.65

0.0099

Compacted

36

28

8

7.86

0.0051

Removed

11

3

8

.58

0.4455

the top three crucial factors that the users felt were contributors to their dissatisfaction. The most frequently cited cause for satisfaction amongst users was that the change saved their time. The users also expressed satisfaction when the features in the new version saved effort and when it improved satisfaction or increased the feel good factor. The reasons for satisfaction also included reduced errors in software, ease of understanding and learning, as well as the ease to navigate when dealing with change. When comparing the satisfactory and unsatisfactory experiences based on the type of change, we can observe that features that were added or enhanced contributed more to a satisfactory experience whereas modification of any kind to the existing feature contributed more to an unsatisfactory experience. A Chi-square analysis of the statistical significance of differences in the change between satisfactory and unsatisfactory experiences was conducted, the results of which are shown in Table 2. One can observe that types of change such as added, replaced, and compacted features showed statistical significance. Logit analysis was used to develop a regression equation between the categorical variables (type of change) and the type of experience. The equation provides an estimate of the log likelihood ratio of satisfactory to unsatisfactory experiences for various categories of changes. In this way, logistic regression estimates the probability of a certain event occurring. The CATMOD procedure of SAS was used to calculate the maximum likelihood for each model. The majority of these individual estimates are highly significant. As shown in Table 3 the likelihood ratio increases significantly when the features were added, replaced or compacted during a change.

The Impact of Change in Software on Satisfaction: Evaluation Using CIT

721

n

(1)

logit ( Es) = log (ES / 1-E s) = intercept+∑ Bi Xi = g . i=1 Therefore Es = eg / (1+eg) .

(2)

Note that the relationship between type of experience (satisfactory and unsatisfactory) and likelihood estimates (type of change) is expressed by equations (1) and (2). Table 3. Maximum Likelihood Estimates Obtained During Logit Analysis Parameter (Xi) Intercept Added Replaced Compacted

Estimate (Bi) -3.657 1.259 1.562 1.599

Standard Error 1.687 0.55 0.605 0.57

3.2 Subjective Ratings Subjective rating questions were asked during the survey interviews in order to understand various aspects of change and its impact on key parameters such as performance, degree of change, and satisfaction. The following subsections describe the descriptive statistics, correlation and regression analysis of the subjective rating responses. Descriptive Statistics. The frequency of responses, means and standard deviations for each subjective question in satisfactory and unsatisfactory experiences are tabulated in Table 4. The descriptive analysis is used in this study to produce a situation analysis of a problem addressing the impact of change on performance, degree of evidence of changed features and overall satisfaction. This data provides a snap shot of the situation. Table 4 below displays several differences observed in the subjective measures such as performance, evidence of change and overall satisfaction depending on whether the user had a satisfactory or unsatisfactory experience with a change in software. Table 4. Descriptive Statistics of Subjective Questionnaire Items Item Impact on performance Degree of evidence Overall satisfaction Impact on performance Degree of evidence Overall satisfaction

Total Response(N)

Std dev 1 2 Unsatisfactory Experience

Mean

Frequency response 3 4 5 6

7

80

2.65

1.22

13

13

28

16

1

2

0

80

5.06

2.13

12

3

2

10

9

14

29

80

2.93 1.45 16 16 Satisfactory Experience

20

15

8

4

0

99

5.51

0.95

0

0

1

12

39

30

17

99

4.28

1.94

10

11

10

25

14

9

20

99

5.81

0.89

0

0

0

6

31

36

26

722

A. Pandith, M. Lehto, and V.G. Duffy

T-test Comparison between Satisfactory and Unsatisfactory Experiences. A contrast using T-tests was conducted to further explore the relationship between the user’s experiences with change reported in the subjective responses. The main purpose of this analysis was to examine the impact or the criticality of the critical incidents of user’s during change. As shown in Table 5, several differences were observed in the subjective measures of user’s satisfaction, impact on performance and evidence of change depending on whether the user had a satisfactory or unsatisfactory experience during the change in software. As might be expected, users were significantly less satisfied after an unsatisfactory experience ((µ satisfactory=5.81 vs. µ unsatisfactory=2.93; t=-16.94< t176, 0.05=1.99) and more satisfied after a satisfactory experience. The impact of performance when there was a change in software had a significant negative effect during an unsatisfactory experience ((µ satisfactory=5.51vs. µ unsatisfactory=2.67; t=1.72< t176, 0.05=1.99) and a positive effect when the user had a satisfactory experience. Another finding of this study was that the degree of evidence of change also showed statistical significance ((µ satisfactory=4.28vs. µ unsatisfactory=5.06; t=-2.35< t 176, 0.05=1.99). But this time the satisfactory experiences occurred when the changes were less evident and unsatisfactory experiences occurred when the changes were more evident. Table 5. T-test Comparison between Satisfactory and Unsatisfactory Experiences Variables Impact On Performance Degree Of Evidence Overall Satisfaction

DF 176 176 176

Satisfactory Mean 5.51 4.28 5.81

Unsatisfactory Mean 2.67 5.06 2.93

T Value

P> I T I

1.72 -2.35 -16.94

satisfying certain relationships in X × Y, accordingly, the truth grade of conditional statement IF X1 THEN Y1 ELSE IF X2 THEN Y2 ... ELSE IF Xn THEN Yn equals to

∑k=nk=1μA' ⇒B' (xi, yj) =∑k=nk=1min (μA (xi), μB(yj)) , where as < xi, yj >∈Xk × Yk To simplify the situation, we consider the simplest conditional statement IF X1 THEN Y1. In [12], Zadeh takes this as a special form of IF X1 THEN Y1 ELSE Z1 with unspecified Z1. Thus he suggests defining IF X1 THEN Y1 as: IF X1 THEN Y1 ≡ IF X1 THEN Y1 ELSE Z1 ≡ X1 × Y1 + ¬ X × U

Fuzzy Logic in Exploring Data Effects: A Way to Unveil Uncertainty in EEG Feedback

757

whereas U is the universe of discourse. However, in practical application, because of the massive universe of discourse and uncertainties to be explored, it is hardly applicable to determine the part of ¬ X × U. Moreover, in many cases especially in healthcare, data sets of various perspectives do not explicitly excludes each other to divide up the whole search space but rather have complex correlations. Thus, in our method, we simply ignore the latter part and define IF X1 THEN Y1 ≡ X1 × Y1

Thus, the truth grade value of the statement equals to ∑i,j=ni,j=1min (μA (xi), μB(yj)), whereas < xi, yj >∈X1 × Y1. If we take all elements in Y1 perfectly satisfying our needs, which, in other words, ∀ y∈Y1, μB(y)=1, then the truth grade value of the statement is decided uniquely by μA (xi). If an evaluation experiment designed specifically to verify the effectiveness of such single if – then statement, then the contribution of data set X on the output can be revealed via comparison between truth grade value μA (xi) and actual veracity of the statement. The way to achieve this goal is to assign equal μ(x) values to all of the variables in propositional form X1. After reasonable evaluation of the effectiveness of the rule, initial μ(x) values shall be adjusted accordingly. After iterative recycle of such process, preferable assignments shall be achieved. A better way to apply this method is to permit presence of only one uncertain data set to be operated in the process, while other variables have at least certain membership function ranges. Our method developed from fuzzy logic enables multiple user cases. One of the scenarios is building antidepressant multimedia therapy. 2.2 User Case Study in an Antidepressant Multimedia Therapy System Architecture. The system architecture of antidepressant multimedia therapy consists of the following five layers: 1> Context Sensing This layer fetches data from various sources and provides needed processing or translation of data so that information can be integrated for context modeling and representation. Since the information needed in antidepressant multimedia therapy is relevant users' information in the process of context modeling and determines services received by users, it will be referred to as user context in general. Specifically, user context in this system includes user's name, gender, age, patient history, EEG signal and clinical diagnosis. Personal and medical user context can be defined initially while acquisition of EEG signal requires a fetching front end working constantly for the sake of status monitoring and therapeutic effectiveness evaluation. 2> Context Modeling and Representation This layer is responsible for representing user context in a unified model in convenience of latter processing and human understanding of the information. In such cases, coherence between knowledge and relationships expressed in the model and practical situation, precise definition and interoperability are among the things to be considered. Thus a suitable representation method shall be chosen carefully.

758

F. Zheng et al.

3> Rule-Based Context Processing In the layer of rule-based context processing, user context described in the underlying layer is processed to assess users' current status and refined requirements. Correspondingly, certain service will be selected according to the predefined rules either to satisfy users' needs or alleviate their unpleasant disease experience. It is in this layer that effects or weights of various data is actually determined by the processing rules and take effect in deciding corresponding services. Virtually, the application of our method in this case is to aid convenient adjustment of weight or effect assignment of data used in this layer, especially EEG signal. 4> Multimedia Service After services in need have been decided by the rule-based context processing, the multimedia service layer provides various applicable multimedia service catering for users' demands in antidepressant treatment, for example, music playing, online gaming and supervised self-regulatory training. Objective effects and fitting user group of various services shall be carefully evaluated before set of services is determined, so that benchmark of service mediation remains reasonable. 5> Feedback Processing As users receive multimedia services, their neurobiological signal, as EEG signal in this case, is being monitored and personal evaluation is acquired in order to generate preferable feedback of the treatment. The results can aid in adjustment of data weight assignment in the context processing layer as well as validation of rules which may be obtained from empirical data or experiments. If alterations of either combinations of data weights or rules can not improve service performance, then the design of context modeling and even types of user contextual information in the system shall be reviewed and revised. Figure 1 demonstrates the system architecture presented above.

Fig. 1. System architecture of antidepressant multimedia therapy. It consists of five layers: context sensing, context modeling and representation, rule-based context processing, multimedia service and feedback processing.

System Composition. The system is composed of five modules: 1> EEG signal acquisition and processing front end. Electroencephalographic (EEG) data consist of changes in neuroelectrical activity measured over time (on a millisecond timescale), across two or more locations, using noninvasive sensors


759

(“electrodes”) that are placed on the scalp surface. A standard technique for analysis of EEG data involves averaging across segments of data (“trials”), time-locking to stimulus “events,” to create event-related brain potentials (ERPs). The resulting measures are characterized by a sequence of positive and negative deflections across time, at each sensor. Initially, EEG data (frequency and amplitude) in our system is fetched through NeXus-4 at the sampling rate of 256/s for brain waves. Then, the data is stored in rational databases. If amplitude and frequency of the brainwave exceeds threshold value, context processing rules will be triggered for neurofeedback data update and corresponding actions such as music selection and track playing. 2> Context modeling. Context modeling represents user context in a preferable model and knowledge describing manner. It includes patients' personal, medical and EEG data to be concerned in clinical mental disorder diagnosis. Problem to be considered in this module is the weight of each context processing rules and various concepts of the system. Initial fuzzy membership function values are assigned to each data set with uncertainty. 3> Rule-based Inference. This module contains context processing rules related with context profiles which are built with original data and the inference rules determining multimedia services' selection strategy. Most of the rules are expressed in if-then forms. 4> Multimedia service organization and selection strategy. This module includes service evaluation (the indexes deciding to what extent does a certain service satisfy the patient's demand) and service organization. The problems to be covered in this section is how to describe essential service characteristics related with mental disorder treatment, how to effectively organize services and determine the most possible sets and how to control the way of providing the services, like adding some reasonable variations due to patients' preference. 5> Feedback of the effectiveness. Feedback generated by either manual evaluation or patients' EEG data is crucial to application in this domain, as effects of EEG in combined user context and assessment of therapeutic effectiveness remains unclear. Working Procedure. The working procedure of the system is stated as follows: Firstly, EEG acquisition and processing front end fetches signal from multiple channels and store it in relational database after essential signal processing. Signals are initially characterized by frequency and amplitude. If any numeric value ranges above threshold, actions are triggered for context processing, which is actually inference according to patients' context to deduce whether service selection shall be activated. Secondly, after service selection is activated, inference of the service matching begins. Basically, multimedia services are organized by features of fitting user groups. Take music therapy for example, music tracks are divided into several sets specifically for melancholics in different mood or status, namely patients in mania, depression, etc. Classification and labeling of the tracks can be finished in reference to psychological music prescription. Since preferable sound tracks in treating patients in different mood are acquired from psychologists' empirical experience, their effectiveness seems to be proven and attention shall be focused on assessing patients' status and requirements accurately. Furthermore, validation of inference rules is also committed in this process.

760

F. Zheng et al.

Thirdly, evaluation of the effectiveness of the system can be achieved by acquiring manual evaluation or EEG feedback. It can in return aids adjustment of the data weight assignment as well as inference rules to improve accuracy. Figure 2 illustrates the exact system composition and working procedure of antidepressant music therapy, which is a specific case in multimedia treatment against depression.

Fig. 2. Rule-based Inference module takes EEG data and user context as inputs to decide whether music shall be played for patient and if is needed, what kind of music shall be chosen. If music is played, manual evaluation will list satisfactory options for patient to choose, which, combined with user EEG signal, in return aiding in determination of weights of various data.

Validation. To validate the applicability of our method, we designed an experiment to test coherence between truth grade value computed from preclaimed if – then rules and actual improvement in EEG signal. The user context in the validation test is simply self-assessment of personal mood (whether one is tired, nervous, depressed, irritable or insomnic) and EEG signal, whereas the inference rules are defined according to psychological music prescription. Music sets corresponding to each mood listed above are stated as follows: Tired: Set A={The Four Seasons by Vivaldi, La mer by Debussy, Water Music by Handel} Nervous: Set B={Danse Macabre by Saint-Saens, Firebird Suite by Stravinsky} Depressed: Set C={Symphonies 40 "Jupiter" by Mozart, Rhapsody In Blue by George Gershwin, Symphony NO.5 by Beethoven} Irritable: Set D={Royal Fireworks by Handel, William Tell-overture by Gioachino Rossini, The Blue Danube by Johann Strauss} Insomnic: Set E={Lullaby by Mozart, A Midsummer Nights Dream Overture by Mendelssohn, Hungarian Rhapsody by Liszt} We test five peoples for three separate times in a continuous time section and select the most stable group. Each group contains two measures: EEG signal measured in calm sit without listening to any music and the EEG measured just after listening to chosen music. Each measure lasts for five minutes. We apply the dual channel,


761

bi-polar placement of NeXus-4 manual, which, in accordance with 10-20 electrode system, takes C3 as channel 1 with F3 as negative, and C4 as channel 2 with F4 as negative. We simply take the assumption that EEG signal responsible for deciding users' mood is the amplitude of Alpha wave of both channels and the level of abnormality is indicated by the excess of 20 micro volts. The EEG data measured is listed as follows: Table 1. Alpha amplitude of dual channel EEG measured in the validation test for five users Unit: Microvolts

User 1 Channel 1 User 1 Channel 2 User 2 Channel 1 User 2 Channel 2 User 3 Channel 1 User 3 Channel 2 User 4 Channel 1 User 4 Channel 2 User 5 Channel 1 User 5 Channel 2

Before music and After music and Abnormality Abnormality 27.204/+0.36021 20.148/+0.0074 20.737/+0.0001 17.772/-0.1114 23.209/+0.1604 23.670/+0.1835 19.537/-0.0231 22.397/+0.1199 18.657/-0.0672 22.108/+0.1054

13.284/- 0.3358 20.858/+0.0429 11.536/-0.4232 23.153/+0.1577 22.818/+0.1409 21.068/+0.0534 14.988/-0.2506 22.594/+0.1297 14.662/-0.2669 17.682/-0.1159

Improvement of Brainwave (times of onefold) +0.5117 -0.0352 +0.4437 -0.3028 +0.0168 +0.1099 +0.2328 -0.0079 +0.2141 +0.2002

First of all, each user’s mood will be decided in application of our method based on fuzzy logic. Assume that function Ei(x) implies that the Alpha amplitude of user x's EEG signal in channel i is in excess of threshold value (20 macrovolts), then the mood of x can be decided as follows: μx is in the mood he/she assessed(x) = (μE1(x) + μE2(x))* EEG credibility + μthe extent of self-mood assessing assurance(x) * (1- EEG credibility) If substitute x with user 3 and assign 0.3 to EEG credibility, then according to the test μx is in the mood he/she assessed (user 3) = (μE1(user 3) + μE2(user 3))* 0.3 + μthe extent of self-mood assessing assurance(x) * 0.7= (|20.0-23.209| / 20.0 + |20.0-23.670| / 20.0)* 0.3 + 0.9*0.7= (0.1604 + 0.1835) * 0.3+0.63= 0.73317 The result is a rather high truth grade value, indicating that user 3 is probably suffering from tiredness and needs music that can help him relax. Consequently, according to the psychological music prescription, sound tracks in set A shall be played for user 3. Figure 3 shows visible improvement of his brainwave, which supports the conclusion. Comparison of coherence between computed truth grade values obtained by our methods and monitored brainwave data improvements are presented in Table 2. The table illustrates a rather coherent relationship between computed truth grade value and brainwave improvement illustrated in Alpha amplitude, which manifests applicability of this method. However, further exploration of the definition of the function in truth grade value computing is needed and coherence shall be examined in larger sample data test sets.

762

F. Zheng et al.

(a) Before user 3 listening to music

(b) After user 3 listening to music Fig. 3. Comparison of user No. 3's Alpha amplitude in dual channel before and after listening to music in set A. Virtually mean Alpha amplitude is taken into account Table 2. Alpha amplitude of dual channel EEG measured in the validation test for five users Unit: Microvolts User 1 User 2 User 3 User 4 User 5

Computed truth Improvement of Alpha amplitude (dual grade value channel, left-channel 1, right-channel 2) 0.8133 +0.5117 -0.0352 0.6667 +0.4437 -0.3028 0.7332 +0.0168 +0.1099 0.7291 +0.2328 -0.0079 0.7114 +0.2141 +0.2002

3 Conclusions and Future Work In this paper, we introduced a method applying fuzzy logic to explore effects of data sets with uncertainties. It works by comparing practical neurobiological feedback or manual evaluation with rules' truth grade value computed from fuzzy membership function. To illustrate uses of this method, we study a user case of developing an antidepressant multimedia therapy and explain possible use of the method in exploring effects of EEG data representing depressive patients' mood. To validate the applicability of the method, we commit a test with five user samples comparing brainwave improvement and computed truth grade value of users' mood. The result indicates that there lies a rather coherent relationship between these two measures. However, to apply it in practical implementation, further examination of the coherence in larger user data test sets shall be committed. We aim to explore further in the method both for performance features as applicability, accuracy and coherence, etc and completeness of fuzzy logic theory. Besides, relationship between general user context and biological signal in the process of modeling, context processing and fuzzy inference will be studied from various prospects in our future work.


763

References 1. World Health Organization, http://www.who.int 2. Corchado, J.M., Bajo, J., de Paz, Y., Tapia, D.I.: Intelligent Environment for monitoring Alzheimer patients, agent technology for healthcare, to be published in Decision Support Systems, http://www.sciencedirect.com 3. Choudhri, A., Kagal, L., Joshi, A., Finin, T., Yesha, Y.: Patient Service: Electronic Patient Record Redaction and Delivery in Pervasive Environments. In: Fifth International Workshop on Enterprise Networking and Computing in Healthcare Industry (2003) 4. Barger, T.S., Brown, D.E., Alwan, M.: Health-Status Monitoring Through Analysis of Behavioral Patterns. IEEE Transactions on Systems, Man and Cybernetics 35(1), 22–27 (2005) 5. Hatzinger, M., Hemmeter, U.M., Brand, S., Ising, M., Holsboer-Trachsler, E.: Electroencephalographic Sleep Profiles in Treatment Course and Long-term Outcome of Major Depression: Association with DEX/CRH-test Response. Journal of Psychiatric Research 38, 453–465 (2004) 6. Urretavizcaya, M., Moreno, I., Benlloch, L., Cardoner, N., Serrallonga, J., Menchón, J.M., Vallejo, J.: Auditory Event-Related Potentials in 50 Melancholic Patients: Increased N100, N200 and P300 Latencies and Diminished P300 Amplitude. Journal of Affective Disorders 74, 293–297 (2003) 7. Jantzen, J.: Tutorial On Fuzzy Logic. Grid Information Services for Distributed Resource Sharing. Kongens Lyngby, DENMARK. Tech. report no 98-E 868 (logic) 8. Zadeh, L.A.: Outline of a New Approach to the Analysis of Complex Systems and Decision Process. IEEE Transactions on Systems, Man, and Cybernetics SMC-3(1) (January 1973) 9. Mamdani, E.H.: Application of Fuzzy Logic to Approximate Reasoning using Linguistic Synthesis. IEEE Transactions on Computers C-26(12), 1182–1191 (1977)

Author Index

Abdel-Malek, Karim 140 Abel, Steve R. 560 Aksenov, Petr 183 Albayrak, Sahin 305 Aloisio, Giovanni 13 Amantini, Aladino 345 Anderson, Paul 550 Andreoni, Giuseppe 591 Armstrong, Thomas J. 85 Artavatkun, Tron 315 Asikele, Edward 710 Augustin, Thomas 433 Bae, Sungchan 85 Barton, Joyce 333 Baumann, Martin R.K. 192 Benedict, Ashley J. 475 Benson, Elizabeth 599 Benson, Stacey 578 Berbers, Yolande 285 Best, Christopher 85 Bian, Yueqing 512 Birk, Carol 560 Bocchi, Leonardo 132 Br¨ uggemann, Ulrike 355 Bubb, Heiner 95 Burghardt, Christoph 202 Butler, Kathryn M. 483 Cacciabue, Pietro Carlo 345 Cai, Dengchuan 365 Carrier, Serge 19 Carruth, Daniel 295 Case, Keith 323, 673, 700, 727 Chadwick, Liam 502 Chanock, David 550 Chao, Chuzhi 46 Charissis, Vassilis 550 Chen, Yiqiang 275 Cheng, Zhiqing 3 Choi, Jaewon 85 Chouvarda, Ioanna 492 Ciani, Oriana 591 Clark, Marianne 333

Clavel, Céline 211 Coninx, Karin 183, 257 Costa, Fiammetta 591 Craven, Patrick L. 333 Crosson, Jesse C. 475 Dai, Jichang 661 Davis, Peter 700, 727 Demirel, H. Onan 608 Deml, Barbara 433 De Paolis, Lucio T. 13 Doebbeling, Bradley 569 Dong, Dayong 624 Dong, Tingting 46 Du, Yingzi 315 Duffy, Vincent G. 475, 560, 608, 717, 744 Durach, Stephan 443 Dzaack, Jeronimo 375 Eckstein, Lutz 443 Eilers, Mark 413, 423 Ellegast, Rolf 221 Endo, Yui 642 Engstler, Florian 95 Fallon, Enda F. 502 Fan, Xiumin 115 Faust, Marie-Eve 19 Feng, Xuemei 72 Feuerstack, Sebastian 305 Filla, Reno 614 Fu, Yan 512 F¨ urstenau, Norbert 227 Garbe, Hilke 423 Ge, Bao-zhen 691 Gifford, Adam 333 Godil, Afzal 29 Goonetilleke, Ravindra S. 681 Gore, Brian F. 237 Grieshaber, D. Christian 85 Guo, Fenfei 624 Gyi, Diane 673, 700, 727

766

Author Index

Haazebroek, Pascal 247 Hannemann, Robert 475 Hanson, Lars 521 Harbers, Maaike 463 Hashagen, Anja 105 He, Qichang 115 Hermanns, Ingo 221 Hermawati, Setia 632 Heuvelink, Annerieke 463 Hoege, Bo 744 H¨ ogberg, Dan 323, 521, 673 Hommel, Bernhard 247 Hong, Kwang-Seok 36 Hooey, Becky L. 237 Hu, Bin 754 Hu, Yong 115 Huang, Lan-Ling 365 Hultgren, Kyle 560

Loudon, David 540 Lundstr¨ om, Daniel 521 L¨ udtke, Andreas 403 Luyten, Kris 183, 257

Inagaki, Yoshikazu 123 Inoue, Takenobu 384 Ito, Takuma 384

Nam, Deok Hee 710 Niedermaier, Bernhard 443 Niu, Jianwei 55, 64, 737 Nuti, Lynn A. 475

Jeon, Jong-Bae 36 Jin, Sang-Hyeon 36 Jun, Esther 531 Kamata, Minoru 384 Kanai, Satoshi 642 Kawaguchi, Keisuke 642 Keinath, Andreas 443 Kim, Dong-Ju 36 Kim, Joo H. 72 Kirste, Thomas 202 Krems, Josef F. 192 Kuramoto, Itaru 123 Lagu, Amit V. 394 Landry, Steven J. 394 Landsittel, Douglas 578 Lee, Jonathan 531 Lehto, Mark 569, 717 Li, Shiqi 512 Li, Xiaojie 691 Li, Yongchang 754 Li, Zhizhong 55, 64, 737 Liu, Junfa 275 Liu, Li 754 Liu, Taijie 46 Liu, Tesheng 365

Macdonald, Alastair S. 540 Maglaveras, Nicos 492 Mahmud, Nasim 257 Marshall, Russell 632, 673, 700, 727 Marshall, Sandra P. 265 Martin, Jean-Claude 211 Mazzola, Marco 591 McInnes, Brian 653 Mihalyi, Andreas 433 Milanova, Mariofanna 132 M¨ obus, Claus 413, 423 Morais, Alexander 295

Osterloh, Jan-Patrick

403

Pan, Wei 275 Pandith, Akshatha 475, 717 Park, Daewoo 85 Pauzié, Annie 453 Pe˜ na-Pitarch, Esteban 140 Potvin, Jim 653 Preatoni, Ezio 591 Preuveneers, Davy 285 Pulimeno, Marco 13 Qi, Yanbin

754

Rajulu, Sudhakar 72, 599 Ran, Linghua 46 Regli, Susan Harkness 333 Robbins, Bryan 295 Robinette, Kathleen 3 Roetting, Matthias 744 Romero, Maximiliano 591 Sakellariou, Sophia 550 Saleem, Jason J. 569 Schelhowe, Heidi 105 Schiefer, Christoph 221 Schwartze, Veit 305

Author Index Scott-Nash, Shelly 237 She, Jin-hua 315 Shi, Xiaobo 531 Shibuya, Yu 123 Shino, Motoki 384 Sims, Ruth 673, 700, 727 Slice, Dennis 578 Stephens, Allison 653 Stibler, Kathleeen 333 Strohschneider, Stefan 355 Sugiyama, Shigeki 150 Summerskill, Steve 673, 700, 727 Thomas, N. Luke 315 Thorvald, Peter 323 Tian, Qing-guo 691 Tian, Renran 560 Tremoulet, Patrice D. 333 Tsujino, Yoshihiro 123 Urbas, Leon

Weber, Lars 403 Weber, Matthias 170 Wickens, Christopher D. 237 Wilcox, Saki 333 Witana, Channa P. 681 Woolley, Charles 85 Wortelen, Bertram 403 Wu, Jiang 737 Wu, Sze-jung 569 Xiang, Yujiang 72 Xiong, Shuping 681 Xu, Song 55, 64 Yang, Jingzhou (James) Yih, Yuehwern 569 Yin, Mingqiang 512 You, Manlai 365 Young, K. David 691 Yucel, Gulcin 744

375

van den Bosch, Karel 463 Vanderhulst, Geert 183 van der Putten, Wil 502 van Doesburg, Willem 463 Vermeulen, Jo 257 Viscusi, Dennis 578 Wang, Lijing 624 Wang, Xuguang 160 Ward, Ben M. 550 W˚ arell, Maria 521

767

Zabel, Christian 105 Zambetti, Marta 591 Zare, Saeed 105 Zhang, Lifeng 115 Zhang, Xin 46 Zhao, Dan 691 Zhao, Jianhui 681 Zheng, Fang 754 Zhou, Wei 85 Zhu, Tingshao 754 Zhuang, Ziqing 578, 661 Zilinski, Malte 423

72, 140, 661

Digital Human Modeling: Second International Conference, ICDHM 2009, Held as Part of HCI International 2009 San Diego, CA, USA, July 19-24, 2009. (Proceedings. Lecture Notes in Computer Science)

Digital Human Modeling: Second International Conference, ICDHM 2009, Held as Part of HCI International 2009 San Diego, CA, USA, July 19-24, 2009. (Proceedings. Lecture Notes in Computer Science)

Human-Computer Interaction. New Trends: 13th International Conference, HCI International 2009, San Diego, CA, USA, July 19-24, 2009, Proceedings, Part ... Applications, incl. Internet Web, and HCI)

Digital Human Modeling: Second International Conference, ICDHM 2009, Held as Part of HCI International 2009 San Diego, CA, USA, July 19-24, 2009 ... Applications, incl. Internet Web, and HCI)

Online Communities and Social Computing: Third International Conference, OCSC 2009, Held as Part of HCI International 2009, San Diego, CA, USA, July ... Applications, incl. Internet Web, and HCI)

Ergonomics and Health Aspects of Work with Computers: International Conference, EHAWC 2009, Held as Part of HCI International 2009, San Diego, CA, ... Applications, incl. Internet Web, and HCI)

Internationalization, Design and Global Development: Third International Conference, IDGD 2009, Held as Part of HCI International 2009, San Diego, CA, ... Applications, incl. Internet Web, and HCI)

Proceedings of International Science Education Conference 2009

Virtual and Mixed Reality: Third International Conference, VMR 2009, Held as Part of HCI International 2009, San Diego, CA USA, July, 19-24, 2009, ... Applications, incl. Internet Web, and HCI)

Human-Computer Interaction. Novel Interaction Methods and Techniques: 13th International Conference, HCI International 2009, San Diego, CA, USA, July ... Applications, incl. Internet Web, and HCI)

Human-Computer Interaction. Interacting in Various Application Domains: 13th International Conference, HCI International 2009, San Diego, CA, USA, ... Applications, incl. Internet Web, and HCI)

Control and Automation: International Conference, CA 2009, Held as Part of the Future Generation Information Technology Conference, CA 2009, Jeju Island, ... in Computer and Information Science)

Web Engineering: 9th International Conference, ICWE 2009 San Sebastián, Spain, June 24-26 2009 Proceedings (Lecture Notes in Computer Science ... Applications, incl. Internet Web, and HCI)

Pairing-Based Cryptography - Pairing 2009: Third International Conference Palo Alto, CA, USA, August 12-14, 2009 Proceedings (Lecture Notes in Computer Science Security and Cryptology)

GeoSensor Networks: Third International Conference, GSN 2009, Oxford, UK, July 13-14, 2009, Proceedings (Lecture Notes in Computer Science ... Applications, incl. Internet Web, and HCI)

Human-Computer Interaction. Ambient, Ubiquitous and Intelligent Interaction: 13th International Conference, HCI International 2009, San Diego, CA, ... Applications, incl. Internet Web, and HCI)

Adaptive Learning Agents: Second Workshop, ALA 2009, Held as Part of the AAMAS 2009 Conference in Budapest, Hungary, May 12, 2009. Revised Selected ... Lecture Notes in Artificial Intelligence)

Hybrid Learning and Education: Second International Conference, ICHL 2009, Macau, China, August 25-27, 2009, Proceedings (Lecture Notes in Computer Science ... Computer Science and General Issues)

Data Integration in the Life Sciences: 6th International Workshop, DILS 2009, Manchester, UK, July 20-22, 2009, Proceedings (Lecture Notes in Computer Science Lecture Notes in Bioinformatics)

Scale Space and Variational Methods in Computer Vision: Second International Conference, SSVM 2009, Voss, Norway, June 1-5, 2009. Proceedings (Lecture Notes in Computer Science)

Scale Space and Variational Methods in Computer Vision: Second International Conference, SSVM 2009, Voss, Norway, June 1-5, 2009. Proceedings (Lecture Notes in Computer Science)

Online Communities and Social Computing: Second International Conference, OCSC 2007, Held as Part of HCI International 2007, Beijing, China, July 22-27, ... (Lecture Notes in Computer Science)

Online Communities and Social Computing: Second International Conference, OCSC 2007, Held as Part of HCI International 2007, Beijing, China, July 22-27, ... (Lecture Notes in Computer Science)

Contemporary Computing: Second International Conference, IC3 2009, Noida, India, August 17-19, 2009. Proceedings (Communications in Computer and Information Science)

Advances in Web Based Learning - ICWL 2009: 8th International Conference, Aachen, Germany, August 19-21, 2009, Proceedings (Lecture Notes in Computer Science ... Applications, incl. Internet Web, and HCI)

CONCUR 2009 - Concurrency Theory: 20th International Conference, CONCUR 2009, Bologna, Italy, September 1-4, 2009, Proceedings (Lecture Notes in ... Computer Science and General Issues)

Coding and Cryptology: Second International Workshop, IWCC 2009 (Lecture Notes in Computer Science Security and Cryptology)

Frommer's San Diego 2009 (Frommer's Complete)

Algebra and Coalgebra in Computer Science: Third International Conference, CALCO 2009, Udine, Italy, September 7-10, 2009, Proceedings (Lecture Notes in ... Computer Science and General Issues)

Algebra and Coalgebra in Computer Science: Third International Conference, CALCO 2009, Udine, Italy, September 7-10, 2009, Proceedings (Lecture Notes in ... Computer Science and General Issues)

Digital Human Modeling: Second International Conference, ICDHM 2009, Held as Part of HCI International 2009 San Diego, CA, USA, July 19-24, 2009. (Proceedings. Lecture Notes in Computer Science)