Human Factors of Remotely Operated Vehicles, Volume 7 (Advances in Human Performance and Cognitive Engineering Research)

HUMAN FACTORS OF REMOTELY OPERATED VEHICLES i ADVANCES IN HUMAN PERFORMANCE AND COGNITIVE ENGINEERING RESEARCH Serie...

Author: Nancy J. Cooke | Maj. Heather Pringle | Harry Pedersen | Olena Connor

188 downloads 1198 Views 5MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

HUMAN FACTORS OF REMOTELY OPERATED VEHICLES

i

ADVANCES IN HUMAN PERFORMANCE AND COGNITIVE ENGINEERING RESEARCH Series Editor: Eduardo Salas Associate Editors: NANCY COOKE Department of Psychology, New Mexico State University, USA

JAMES E. DRISKELL Florida Maxima, USA

ANDERS ERICSSON Florida State University, USA

Volume 1: Advances in Human Performance and Cognitive Engineering Research, Edited by Eduardo Salas Volume 2: Advances in Human Performance and Cognitive Engineering Research: Automation, Edited by Eduardo Salas Volume 3: Advances in Human Performance and Cognitive Engineering Research, Edited by Eduardo Salas and Dianna Stone Volume 4: Advances in Human Performance and Cognitive Engineering Research, Edited by Michael Kaplan Volume 5: The Science and Simulation of Human Performance, Edited by James W. Ness, Victoria Tepe and Darren Ritzer Volume 6: Understanding Adaptability: A Prerequisite for Effective Performance within Complex Environments, Edited by C. Shawn Burke, Linda G. Pierce and Eduardo Salas ii

ADVANCES IN HUMAN PERFORMANCE AND COGNITIVE ENGINEERING RESEARCH VOLUME 7

HUMAN FACTORS OF REMOTELY OPERATED VEHICLES EDITED BY

NANCY J. COOKE Arizona State University and Cognitive Engineering Research Institute, Mesa, Arizona, USA

HEATHER L. PRINGLE Air Force Research Laboratory, Mesa, Arizona, USA

HARRY K. PEDERSEN Cognitive Engineering Research Institute, Mesa, Arizona and New Mexico State University, USA

OLENA CONNOR Cognitive Engineering Research Institute, Mesa, Arizona and New Mexico State University, USA

Amsterdam – Boston – Heidelberg – London – New York – Oxford Paris – San Diego – San Francisco – Singapore – Sydney – Tokyo JAI Press is an imprint of Elsevier

iii

JAI Press is an imprint of Elsevier The Boulevard, Langford Lane, Kidlington, Oxford OX5 1GB, UK Radarweg 29, PO Box 211, 1000 AE Amsterdam, The Netherlands 525 B Street, Suite 1900, San Diego, CA 92101-4495, USA First edition 2006 Copyright r 2006 Elsevier Ltd. All rights reserved No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means electronic, mechanical, photocopying, recording or otherwise without the prior written permission of the publisher Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone (+44) (0) 1865 843830; fax (+44) (0) 1865 853333; email: [email protected]. Alternatively you can submit your request online by visiting the Elsevier web site at http://elsevier.com/locate/permissions, and selecting Obtaining permission to use Elsevier material Notice No responsibility is assumed by the publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. Because of rapid advances in the medical sciences, in particular, independent verification of diagnoses and drug dosages should be made British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN-13: 978-0-7623-1247-4 ISBN-10: 0-7623-1247-5 ISSN: 1479-3601 (Series) For information on all JAI Press publications visit our website at books.elsevier.com Printed and bound in the Netherlands. 06 07 08 09 10 10 9 8 7 6 5 4 3 2 1

iv

CONTENTS LIST OF CONTRIBUTORS

xi

ACKNOWLEDGEMENTS

xv

PREFACE: WHY HUMAN FACTORS OF "UNMANNED" SYSTEMS?

xvii

HUMAN FACTORS OF UAVs WORKSHOPS 1. CERI HUMAN FACTORS OF UAVs: 2004 AND 2005 WORKSHOP OVERVIEWS Olena Connor, Harry K. Pedersen, Nancy J. Cooke and Heather L. Pringle 2. UAV HUMAN FACTORS: OPERATOR PERSPECTIVES Harry K. Pedersen, Nancy J. Cooke, Heather L. Pringle and Olena Connor

3

21

HUMAN FACTORS ISSUES 3. REMOTELY OPERATED VEHICLES (ROVs) FROM THE TOP-DOWN AND THE BOTTOM-UP T. Oron-Gilad, J. Y. C. Chen and P. A. Hancock

v

37

vi

CONTENTS

4. SUPERVISORY CONTROL OF UNINHABITED COMBAT AIR VEHICLES FROM AN AIRBORNE BATTLE MANAGEMENT COMMAND AND CONTROL PLATFORM: HUMAN FACTORS ISSUES W. Todd Nelson and Robert S. Bolia 5. MODELING AND OPERATOR SIMULATIONS FOR EARLY DEVELOPMENT OF ARMY UNMANNED VEHICLES: METHODS AND RESULTS Michael J. Barnes, Bruce P. Hunn and Regina A. Pomranky 6. UAV OPERATORS, OTHER AIRSPACE USERS, AND REGULATORS: CRITICAL COMPONENTS OF AN UNINHABITED SYSTEM Stephen B. Hottman and Kari Sortland

49

59

71

ERRORS, MISHAPS, AND ACCIDENTS 7. HUMAN ERRORS IN UAV TAKEOFF AND LANDING: THEORETICAL ACCOUNT AND PRACTICAL IMPLICATIONS Avi Parush 8. HUMAN FACTORS IMPLICATIONS OF UNMANNED AIRCRAFT ACCIDENTS: FLIGHT-CONTROL PROBLEMS Kevin W. Williams 9. HUMAN FACTORS IN U.S. MILITARY UNMANNED AERIAL VEHICLE ACCIDENTS Clarence E. Rash, Patricia A. LeDuc and Sharon D. Manning

91

105

117

vii

Contents

10. SPATIAL DISORIENTATION IN UNINHABITED AERIAL VEHICLES Brian P. Self, William R. Ercoline, Wesley A. Olson and Anthony P. Tvaryanas

133

THE ROV INTERFACE 11. MULTI-SENSORY INTERFACES FOR REMOTELY OPERATED VEHICLES Gloria L. Calhoun and Mark H. Draper 12. EVALUATION OF A TOUCH SCREEN-BASED OPERATOR CONTROL INTERFACE FOR TRAINING AND REMOTE OPERATION OF A SIMULATED MICRO-UNINHABITED AERIAL VEHICLE Paula J. Durlach, John L. Neumann and Laticia D. Bowens 13. VIDEO IMAGERY’S ROLE IN NETWORK CENTRIC, MULTIPLE UNMANNED AERIAL VEHICLE (UAV) OPERATIONS Bruce P. Hunn 14. SPATIAL DIALOG AND UNMANNED AERIAL VEHICLES Wendell H. Chun, Thomas Spura, Frank C. Alvidrez and Randy J. Stiles

149

165

179

193

CONTROL OF MULTIPLE ROVs 15. WORKLOAD AND AUTOMATION RELIABILITY IN UNMANNED AIR VEHICLES Christopher D. Wickens, Stephen R. Dixon and Michael S. Ambinder

209

viii

CONTENTS

16. DESIGN OF A MULTI-VEHICLE CONTROL SYSTEM: SYSTEM DESIGN AND USER INTERACTION Shawn A. Weil, Jared Freeman, Jean MacMillan, Cullen D. Jackson, Elizabeth Mauer, Michael J. Patterson and Michael P. Linegang 17. SCALING-UP HUMAN CONTROL FOR LARGE UAV TEAMS Michael Lewis, Jumpol Polvichai, Katia Sycara and Paul Scerri

223

237

18. DELEGATION INTERFACES FOR HUMAN SUPERVISION OF MULTIPLE UNMANNED VEHICLES: THEORY, EXPERIMENTS, AND PRACTICAL APPLICATIONS Raja Parasuraman and Christopher Miller

251

19. OPERATIONAL ANALYSIS AND PERFORMANCE MODELING FOR THE CONTROL OF MULTIPLE UNINHABITED AERIAL VEHICLES FROM AN AIRBORNE PLATFORM Ming Hou and Robert D. Kobierski

267

TEAM CONTROL OF ROVs 20. ACQUIRING TEAM-LEVEL COMMAND AND CONTROL SKILL FOR UAV OPERATION Nancy J. Cooke, Harry K. Pedersen, Olena Connor, Jamie C. Gorman and Dee Andrews 21. A THEORETICAL PERSPECTIVE ON ENHANCING COORDINATION AND COLLABORATION IN ROV TEAMS Ernest S. Park, Verlin B. Hinsz and Jared L. Ladbury

285

299

ix

Contents

22. GUIDING THE DESIGN OF A DEPLOYABLE UAV OPERATIONS CELL Janie A. DeJoode, Nancy J. Cooke, Steven M. Shope and Harry K. Pedersen 23. COGNITION AND COLLABORATION IN HYBRID HUMAN–ROBOT TEAMS: VIEWING WORKLOAD AND PERFORMANCE THROUGH THE LENS OF MULTIMEDIA COGNITIVE LOAD Sandro Scielzo, Stephen M. Fiore, Florian Jentsch and Neal M. Finkelstein

311

329

ROVs ON THE GROUND 24. EXPLORING HUMAN-ROBOT INTERACTION: EMERGING METHODOLOGIES AND ENVIRONMENTS A. William Evans, III, Raegan M. Hoeft, Florian Jentsch, Sherri A. Rehfeld and Michael T. Curtis 25. SITUATION AWARENESS IN THE CONTROL OF UNMANNED GROUND VEHICLES Jennifer M. Riley, Robin R. Murphy and Mica R. Endsley 26. WHAT THE ROBOT’S CAMERA TELLS THE OPERATOR’S BRAIN Roger A. Chadwick, Skye L. Pazuchanics and Douglas J. Gillan SUBJECT INDEX

345

359

373 385

This page intentionally left blank

x

LIST OF CONTRIBUTORS Frank C. Alvidrez

Lockheed Martin Co., CA, USA

Michael S. Ambinder

University of Illinois, IL, USA

Dee Andrews

Air Force Research Laboratory, AZ, USA

Michael J. Barnes

U.S. Army Research Laboratory, AZ, USA

Robert S. Bolia

Air Force Research Laboratory, Wright-Patterson Air Force Base, OH, USA

Laticia D. Bowens

University of Central Florida, FL, USA

Gloria L. Calhoun


Roger A. Chadwick

Department of Psychology, New Mexico State University, NM, USA

Jessie Y. C. Chen

U.S. Army Research Laboratory, FL, USA

Wendell H. Chun

Lockheed Martin Co., CO, USA

Olena Connor

Cognitive Engineering Research Institute, AZ, USA

Nancy J. Cooke

Cognitive Engineering Research Institute, AZ and Arizona State University, AZ, USA

Michael T. Curtis


Janie A. DeJoode

U.S. Positioning Group, LLC, AZ, USA

Stephen R. Dixon


Mark H. Draper

AFRL/HECI, Wright-Patterson Air Force Base, OH, USA

xi

xii

LIST OF CONTRIBUTORS

Paula J. Durlach

U.S. Army Research Institute for the Behavioral and Social Sciences, FL, USA

Mica R. Endsley

SA Technologies, Inc., GA, USA

William R. Ercoline

Brooks Air Force Base, San Antonio, TX, USA

A. William Evans, III


Neal M. Finkelstein

U.S. Army Simulation & Training Technology Center, FL, USA

Stephen M. Fiore


Jared Freeman

Aptima, Inc., Washington, DC, USA

Douglas J. Gillan


Jamie C. Gorman

Cognitive Engineering Research Institute, AZ, and New Mexico State University, NM, USA

P. A. Hancock


Verlin B. Hinsz

North Dakota State University, ND, USA

Raegan M. Hoeft


Stephen B. Hottman

Physical Science Laboratory, New Mexico State University, NM, USA

Ming Hou

Defence Research and Development Canada, Toronto, Canada

Bruce P. Hunn

U.S. Army Research Laboratory, AZ, USA

Cullen D. Jackson

Aptima, Inc., MA, USA

Florian Jentsch


Robert D. Kobierski

CMC Electronics Inc., Ottawa, Canada

Jared L. Ladbury


Patricia A. LeDuc

U.S. Army Aeromedical Research Laboratory, AL, USA

xiii

List of Contributors

Michael Lewis

School of Information Science, University of Pittsburgh, PA, USA

Michael P. Linegang

Aptima, Inc., Washington, DC, USA

Jean MacMillan


Sharon D. Manning

Aviation Branch Safety Office, AL, USA

Elizabeth Mauer


Christopher Miller

Smart Information Flow Technologies, St. Paul, MN, USA

Robin R. Murphy

Center for Robot-Assisted Search and Rescue, University of South Florida, FL, USA

W. Todd Nelson


John L. Neumann


Wesley A. Olson

United States Air Force Academy, CO, USA

Tal Oron-Gilad


Raja Parasuraman

George Mason University, VA, USA

Ernest S. Park


Avi Parush

Carleton University, Ottawa, Canada

Michael J. Patterson


Skye L. Pazuchanics


Harry K. Pedersen

Cognitive Engineering Research Institute, AZ and New Mexico State University, NM, USA

Jumpol Polvichai

School of Information Science, University of Pittsburgh, PA, USA

Regina A. Pomranky

U.S. Army Research Laboratory, TX, USA

Heather L. Pringle

Air Force Research Laboratory, AZ, USA

xiv

LIST OF CONTRIBUTORS

Clarence E. Rash

U.S. Army Aeromedical Research Laboratory, AL, USA

Sherri A. Rehfeld


Jennifer M. Riley

SA Technologies, Inc., GA, USA

Paul Scerri

Robotics Institute, Carnegie Mellon University, PA, USA

Sandro Scielzo


Brian P. Self

United States Air Force Academy, CO, USA

Steven M. Shope

U.S. Positioning Group, LLC, AZ, USA

Kari Sortland

Physical Science Institute, New Mexico State University, NM, USA

Thomas Spura

Lockheed Martin Co., NY, USA

Randy J. Stiles

Lockheed Martin Co., CA, USA

Katia Sycara

Robotics Institute, Carnegie Mellon University, PA, USA

Anthony P. Tvaryanas

Brooks Air Force Base, TX, USA

Shawn A. Weil

Aptima, Inc., Woburn, MA, USA

Christopher D. Wickens


Kevin W. Williams

Civil Aerospace Medical Institute, OK, USA

ACKNOWLEDGEMENTS The editors of this volume would like to thank the authors whose contributions to this area have broken new ground for human considerations in a system that is often mistaken as unmanned. We would also like to thank the attendees of our two workshops on human factors of UAVs who shared their insights and scientific accomplishments with us as well as for those from the development community who conveyed to us the constraints and needs of their community. Thanks also to the sponsors of these workshops who include the Air Force Research Laboratory, the Air Force Office of Scientific Research, NASA, US Positioning, FAA, and Microanalysis and Design. We also thank the many individuals including Leah Rowe, Jennifer Winner, Jamie Gorman, Preston Kiekel, Amanda Taylor, Dee Andrews, Pat Fitzgerald, Ben Schaub, Steve Shope, and Wink Bennett who provided their valuable time and energy to assist with the workshops and this book. Last but not least, we wish to thank ROV operators, those who have attended our workshops, those who we have come to know only through anecdotes, and those who we will never know. It is this group that truly inspired the workshops and the book. We dedicate this effort to them. The conclusions and opinions expressed in this book are those of the authors. They do not reflect the official position of the US Government, Department of Defense, or the United States Air Force.

xv


xvi

PREFACE: WHY HUMAN FACTORS OF ‘‘UNMANNED’’ SYSTEMS?

UAVS TAKE CENTER STAGE UAVs or unmanned (or the more politically correct, ‘‘unpiloted’’ or ‘‘uninhabited’’) Aerial Vehicles and the broader class of remotely operated vehicles (ROVs) have attracted much attention lately from the military, as well as the general public. Generally, ROVs are vehicles that do not carry human pilots or operators, but instead are controlled remotely with different degrees of autonomy on the part of the vehicle. The role of UAVs in the military has rapidly expanded over the years such that every branch of the U.S. military deploys some form of UAV in their intelligence, surveillance, and reconnaissance operations. Recent U.S. military successes include a USAF Predator UAV operating in Iraq, but piloted by a team at Nellis AFB (now Creech AFB) in Las Vegas, Nevada, which successfully aided in finding Saddam Hussein (Rogers, 2004). Another more recent example took place in August 2004 when a Predator UAV armed with Hellfire missiles, also controlled from Nellis AFB, rescued a group of U.S. Marines pinned down by sniper fire in Najaf, Iraq (CNN, 2005). The value of UAVs is recognized by other nations as well who have active UAV programs including, but not limited to, Germany, England, China, France, Canada, South Africa, and Israel. The benefits of UAVs have become so apparent that many civilian applications have arisen, from border and wildfire surveillance, to agricultural uses such as crop dusting and crop health monitoring. For example, the NASA ERAST Pathfinder has been successful in monitoring coffee fields in Hawaii for ripe beans, which has lowered operating costs and increased revenue for the company (Roeder, 2003). UAVs have been so successful, that future planned missions to Mars will see the use of UAVs to explore the Martian surface (UVonline, 2005), in conjunction with one of the most famous ground-based ROVs, the Mars Rover. Other uses for UAVs will eventually include communication relay and weather monitoring by xvii

xviii

PREFACE

high-altitude long-endurance (HALE) platforms as well as surveillance and reconnaissance in the service of Homeland Defense (Hart, 2005). Though UAVs have taken center stage, remotely operated ground, space, and underwater vehicles also have similar military applications that exploit the particular benefits of keeping humans out of harm’s way. The civilian applications of ROVs are also extensive and include search and recovery in areas impossible or too dangerous for humans (e.g., collapsed buildings or mines, surveillance after hurricanes, deep underwater exploration, recovery of nuclear bombs, and plane wreckage underwater).

THE DOWNSIDE Despite the glowing record of successes and utility, the operational record of UAVs has been marred by high mishap rates, frequently cited as a deterrent to the widespread use of UAVs. Mishaps as defined by the U.S. Navy, are unplanned events that directly involve naval aircraft, that result in $10,000 or greater cumulative damage to aircraft or personal injury. Under this classification, a ‘‘Class A’’ mishap is that in which the total amount of damage exceeds $1,000,000 or results in the destruction of the aircraft. The high mishap rate, which by some counts (Department of Defense, 2001) has been 100 times higher than that of manned aircraft, is cited as a deterrent to the increased use of UAVs. For example, the Pioneer UAV has an unacceptable Class A mishap rate of 385 mishaps per 100,000 flight hours since 1986. In contrast, manned Naval aviation has a rate of two mishaps per 100,000 flight hours (Jackson, 2003). Many of these mishaps occur as a result of early deployment of immature technology, but human factors related issues are frequently cited as a cause. For example, Schmidt and Parker (1995), examined 107 mishaps that occurred between 1986 and 1993 and found that 59% were attributable to electromechanical failure and 33% were due to human errors associated with crew selection and training, pilot proficiency, personnel shortages, operational tempo, and errors in teamwork and aircraft control. Seagle (1997) also examined 203 mishaps from 1986 through 1997 and found that 43% of those were attributable to human error. One example of a mishap occurred when a Predator UAV encountered a fuel problem during a descent and upon entering instrument meteorological conditions, icing occurred and the engine lost power. The UAV crashed in an unpopulated wooded area, resulting in no casualties. It was determined that the operators’ attention became too focused on flying the UAV in conditions

xix

Preface

they had rarely encountered. Ultimately, there was a lack of communication between the two operators during the emergency, which resulted in the mishap. The increasing frequency and varied applications in which UAVs are being, and will be used, coupled with the high mishap rate, speak to the need for more human factors research. Moreover, the recorded mishaps are those that result in extensive damage and do not include errors of the larger UAV system such as targeting errors (e.g., taking a photo of the wrong target; getting a poor photo). Even more troubling are the repercussions of such targeting errors that might occur when the payload consists of weapons, as opposed to cameras. In sum, despite the importance and criticality of ROV technologies, there has been surprisingly minimal attention paid to human factors. Part of this may be endemic to the all too common neglect of human factors issues until problems become evident and fixes become necessary. However, in this particular domain, the neglect seems to be more serious than usual and at times tied to misconceptions about the role of the human in this highly automated system.

MYTHS AND FALLACIES The ‘‘unmanned’’ means no humans fallacy. ROV may in many ways be a better label than ‘‘unmanned’’ or ‘‘unpiloted’’ vehicle, for the latter suggests, as some believe, that there are no humans involved. Of course, this view overlooks the scores of humans who maintain, launch, control, operate, monitor, and coordinate these systems from the ground. It also neglects humans who are co-located in the same air, water, or ground space as operators of manned vehicles. Yet, this is just one basic misconception that arises in the context of ROV technology. There are other related myths and fallacies as well that we have identified and that we believe often underlie the neglect of human factors of ROVs. The human has been automated ‘‘out of the loop’’ fallacy. UAVs and ROVs, more generally, are highly automated. Platforms such as the Global Hawk are capable of taking off, flying missions, and landing, all fully autonomously. The belief is that more automation is better and if there is a problem, a person can simply step in and deal with it. The fallacy is that the automation replaces the human; no humans – no need for human factors. However, over 30 years of research has shown that automation indeed changes the human’s task, but not always in a positive manner (Parasuraman & Riley, 1997; Sheridan, 1987,

xx

PREFACE

2002). The human’s task simply changes from one of control to one of oversight. Many mishaps are attributed to the human being ‘‘out-of-the-loop.’’ Further, there are aspects of the task that are best to left to human controllers (i.e., interpretation of target imagery, dynamic replanning in the face of change). The just like air traffic control or manned flight fallacy. This particular misconception is related to the push to relegate control of multiple vehicles to a single human. Many believe that the state of the art is one operator per vehicle and that a one-to-four operator-to-vehicle ratio is the targeted extension. However, current practice requires two to three operators per vehicle, depending on the particular platform. The misconception seems to stem from inappropriate analogies to manned flight or air traffic control. The idea is that a single pilot can control a plane and UAV operation is not different than manned flight. The truth is that a UAV is not simply a vehicle, but a system that includes multiple functions in addition from maneuvering from point A to B. It is also a sensor or weapons platform with functions that go beyond flight. The ‘‘piloting analogy’’ also ignores years of research demonstrating difficulties with remote operations such as time lag, loss of visual cues, and depth perception limits. A related view is that since air traffic controllers monitor dozens of vehicles, UAV operators should also be able to operate multiple vehicles simultaneously. Again, UAV operation involves much more than monitoring and control of aircraft position. There is navigation, maneuvering, sensor operation, and coordination of these activities within the larger system. Further, cognitive workload associated with single vehicle operation can become quite intense when target areas are reached or when dynamic replanning is required.

STEPS TOWARD HUMAN-CENTERED ROV TECHNOLOGY This volume provides a look at some of the human factors challenges associated with ROVs and the research and development work that is being done to address them. The volume starts off with a look at the two Human Factors of UAVs Workshops that brought a critical mass of researchers, developers, and operators together and provided the impetus for this book. Chapter 2 in this section is especially devoted to conveying the operator’s perspective on this topic. There are four chapters that fall generally into the section on Human Factors Issues. In this section you will see an account of the issues from a

xxi

Preface

user’s perspective (Chapter 3), from the perspective of scientists working for the US Air Force (Chapter 4), scientists working for the US Army (Chapter 5), and from the perspective of the national airspace (Chapter 6). In the section on Errors, Mishaps, and Accidents three chapters cover the human factors issues related to ROV accidents and the last one (Chapter 10) focuses on spatial disorientation as a specific cause of some errors and an interesting cause given that the typical pilot-in-the-plane scenario does not hold. The four chapters in The ROV Interface section focus on some specific display and control considerations for the remote control of these systems ranging from tactile displays (Chapter 11) and touch screens (Chapter 12) to video imagery (Chapter 13) and spatial dialog (Chapter 14). Then in the Control of Multiple ROVs section the highly controversial issue of how many ROVs can a single operator control is addressed in five chapters through modeling (Chapters 15 and 19), design (Chapter 16), intelligent automation (Chapters 17 and 18). Four chapters focus on the fact that there are multiple individuals on the ground operating, as a command-and-control team further complicating matters in the section on Team Control of ROVs. The volume concludes with a section that covers ROVs on the Ground, otherwise known as UGVs Uninhabited Ground Vehicles (UGVs). Taken together this work represents the state-of-the-art in our understanding of the human considerations associated with operation of ROVs. When viewed as systems, these human considerations go beyond the interface to vehicle control and extend to the tasks of sensor operation, command-and-control, navigation, communication, time sensitive targeting, and mission planning. Further they extend to applications for training ROV operators, operator selection, integration into the national airspace, and design of technologies to improve remote operation. We have achieved our goal if it is clear from this book that there are human factors of ‘‘unmanned’’ vehicles.

REFERENCES CNN (2005). New videos show Predators at work in Iraq, February 9, 2005.http:// www.cnn.com/2005/WORLD/meast/02/08/predator.video/ Department of Defense. (2001). Unmanned aerial vehicles roadmap, 2000–2025. Washington, DC: Office of the Secretary of Defense. Hart, J. (2005). Northrop Grumman demonstrates autonomous system that delivers real-time surveillance information to military forces in urban battle zones (October 11). El Segundo, CA: Northrop Grumman News Release. Jackson, P. (2003). All the world’s aircraft (pp. 721–722). Alexandria, VA: Janes Information Group.

xxii

PREFACE

Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39, 230–253. Roeder, L. (2003). January–February, hope from on high. Unmanned Vehicles, 8(1), 14–15. Rogers, K. (2004). Nellis crew helped nab Saddam. Las Vegas Review-Journal, (March 2), 1A. Schmidt, J., & Parker, R. (1995). Development of a UAV mishap factors database. Presented at Association of Unmanned Vehicle Systems 1995 conference, Washington, DC, July. Seagle, J., Jr. (1997). Unmanned aerial vehicle mishaps: A human factors approach. Masters thesis, Embry-Riddle Aeronautical University, Norfolk, VA. Sheridan, T. (1987). Supervisory control. In: G. Salvendy (Ed.), Handbook of human factors (pp. 1244–1268). New York: Wiley. Sheridan, T. (2002). Humans and automation. Santa Monica and NY: Human Factors and Ergonomics Society & Wiley. UV online. (2005). NASA experiment explores using thermals to extend UAV endurance. www.shephard.co.uk/UVOnline/default.aspx?Action=-187126550&ID=96c28f5f398d-43a5-9700-3cb00a861edb

Nancy J. Cooke Editor

HUMAN FACTORS OF UAVS WORKSHOPS The human factors of Unmanned Aerial Vehicle (UAV) workshops united UAV developers, operators, and researchers in an effort to highlight human factors issues associated with UAVs. Two workshops addressed a wide range of human factors issues including individual training and selection criteria, display design approaches, and team coordination. Olena Connor’s chapter briefly summarizes the outcome of the two workshops, emphasizing human factors issues that need further investigation in the literature. Harry Pedersen’s chapter focuses on the operator perspectives presented in the workshops. These perspectives were instrumental in drawing attention to real-world constraints that need to be considered in research and development processes. Together, these workshops provided a unique look at the human side of the unmanned aircraft technology.

1


2

1. CERI HUMAN FACTORS OF UAVs: 2004 AND 2005 WORKSHOP OVERVIEWS Olena Connor, Harry K. Pedersen, Nancy J. Cooke and Heather L. Pringle MOTIVATION FOR THE WORKSHOPS The great success of unmanned aerial vehicles (UAVs) in performing nearreal time tactical, reconnaissance, intelligence, surveillance and other various missions has attracted broad attention from military and civilian communities. A critical contribution to the increase and extension of UAV applications, resides in the separation of pilot and vehicle allowing the operator to avoid dangerous and harmful situations. However, this apparent benefit has the potential to lead to problems when the role of humans in remotely operating ‘‘unmanned’’ vehicles is not considered. Although, UAVs do not carry humans onboard, they do require human control and maintenance. To control UAVs, skilled and coordinated work of operators on the ground is required. Despite the fact that UAVs are growing rapidly in numbers, the human factors issues of UAV operations are misunderstood and often overlooked. Too often UAV human factor issues are ignored and the main emphasis is put on UAV hardware capabilities. This state of affairs can partially be explained by a predilection toward technological solutions, coupled with the separation of the human from the part of the technology that is salient Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 3–20 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07001-3

3

4

OLENA CONNOR ET AL.

(i.e., the vehicle, rather than the control station). In addition, the fact that human factors scientists tend to have minimal exposure to the developmental and operational communities further divorces important human factors from technological development. The two workshops described in this chapter were held to address the human factors gaps in UAV technology. These two workshops inspired much of the material in this volume and are summarized in this introductory chapter.

1st ANNUAL HUMAN FACTORS OF UAVs WORKSHOP In May of 2004, the Cognitive Engineering Research Institute (CERI) took initiative in uniting the human factors research community with the UAV operational community and UAV developers in order to identify critical human factors associated with UAV operations. To achieve this goal Dr. Nancy Cooke, Ph.D. from CERI, and Major Heather Pringle, Ph.D., then from the Air Force Research Laboratory (AFRL), organized and cochaired the 1st Annual Human Factors of UAVs Workshop, which was hosted by CERI in Chandler, AZ. This workshop was sponsored by the Air Force Office of Scientific Research (AFOSR), the Air Force Research Laboratory (AFRL), National Aeronautics and Space Administration (NASA), and U.S. Positioning (USP). Specific objectives of the workshop included: identifying human factors issues associated with current and envisioned UAV operations; reviewing existing relevant human factors research on UAVs and determining gaps in the literature, and identifying potential venues through which the workshop’s results could be communicated broadly. The first workshop was organized around the following themes that emerged from the presentations that were submitted and accepted: general human factors issues, perceptual and display issues, cognitive issues, emerging UAV technology, selection and trainings of UAV operators, simulation design, team process, and systems safety. The workshop lasted two days (May 24th and 25th) and consisted of oral presentations, a poster/demo session, and a breakout session. During these events the topics outlined above were discussed and new research questions were identified. The workshop’s structure provided breaks for networking and information exchange. In the following section we highlight topics presented during these sessions (see Tables 1 and 2 for a list of poster/demo presentations and oral presentations).

5

CERI Human Factors of UAVs

Table 1.

Oral Presentations Given at 1st Annual Human Factors of UAVs Workshop, Chandler, AZ, 2004.

Operator Panel The U.S. Air Force at the crossroads: dealing with unmanned aerial vehicles human factors Designing humans for unmanned systems. General Human Factors Issues A shift in automation philosophy from manned to unmanned systems Wright State University’s human-centered research involving uninhabited vehicles in military-focused domains Human machine interaction concepts for the unmanned combat armed rotorcraft Joint HSI considerations in a UAV system of systems

Perceptual and Display Issues The spatial standard observer as a tool for design of UAV viewing systems Challenges UAV operators face in maintaining spatial orientation Comparing visual cues necessary for innerand outer-loop control UAV Human Factors Research within AFRL/HEC Cognitive Issues Supervisory control of multiple autonomous airborne vehicles: lessons learned from tactical Tomahawk human-in-the-loop experiments UAV Crew Systems Research at Ft. Huachuca Unmanned cannot be untrained: synthetic agents for UAV operations training Human vs. autonomous control of UAVbased surveillance: optimizing allocation of decision-making responsibilities

Hoffman, James ‘‘Rainman’’ MAJ, USAF

Goldfinger, Jeff, Brandes Associates, Inc Shively, Jay, U.S. Army Hill, Raymond, R., Narayanan, S., and Gallimore, Jennie, Wright State University Spura, Thomas, Lockheed Martin Systems Integration, and Miller, Christopher, A., Smart Information Flow Technologies Risser, Daniel T. Ph.D., Advanced Engineering and Planning Corporation Inc (AEPCO); Drillings, Michael Ph.D., Army Manpower and Personnel Integration (MANPRINT) Director; Dolan Nancy M. S., Navy HIS, and Hover, Glenn M.S. Air Force, Deputy Performance Enhancement Division Watson, Andrew B., and Ahumada Jr., Albert J., NASA Ames Research Center Gugerty, Leo Ph.D., Clemson University Sweet, Barbara T., and Kaiser, Mary K. Ph.D., NASA Ames Research Center Draper, Mark H. Ph.D., Calhoun, Gloria, Patzek, Mike and Feitshans, Greg, Air Force Research Lab (AFRL)/HEC Cummings, Missy L, Massachusetts Institute of Technology (MIT), and Guerlain, Stephanie, University of Virginia Barnes, Michael J., and Warner, John D.,U.S. Army Research Laboratory Ryder, Joan, and Bell, Benjamin, CHI Systems Inc Freed, Michael, NASA Ames/ Institute for Human and Machine Cognition (IHMC); Shafto, Michael G., NASA Ames, and Harris, Robert, NASA Ames/OSS Inc

6

OLENA CONNOR ET AL.

Table 1. (Continued ) Emerging UAV Technology System for Controlling a Hijacked Aircraft Mini-UAV telemetry and imaging visualization for searching tasks Interfaces for controlling teams of wide area search munitions

Delegation approaches to multiple unmanned vehicle control Local Human Factors of UAVs Research Cognitive modeling in human factors (generally) and in predator UAV operations (specifically) Impact of prior flight experience on learning predator UAV operator skills

Team coordination and UAV operation

Dr. Matos, Jeffrey A., and Milde, Jr., Karl F., M and M Technologies Goodrich, Michael A., and Quigley, Morgan L., Brigham Young University (BYU) Lewis, Michael, and Manojlovich, Joeseph, University of Pittsburgh; Murphey, Robert, and O’Neal, Kevin, Air Force Research Lab (AFRL)/NMNG; Sycara, Katia, Carnegie Mellon University Miller, Christopher A., Funk, Harry B. and Goldman, Robert P., Smart Information Flow Technologies (SIFT) Gluck, Kevin A., and Ball, Jerry T., Air Force Research Lab (AFRL); Krusmark, Michael A., L3 Communications; Purtee, Mathew T., and Rodgers, Stuart M., AFRL Schreiber, Brian, Lockheed Martin, Air Force Research Lab (AFRL)/HEA; Lyon, Don, L3COM, Air Force Research Lab (AFRL)/ HEA, and Martin, Elizabeth, Air Force Research Lab (AFRL)/HEA Cooke, N. J., Arizona State University (ASU), Air Force Research Lab (AFRL), Cognitive Engineering Research Institute (CERI)

Proceedings of the 1st Annual Human Factors of UAVs Workshop Nancy Cooke and Heather Pringle opened the proceedings by discussing human factors issues relevant to UAV operations, reviewed the workshop’s objectives, and briefed the attendees on the itinerary of workshop activities. Opening remarks were followed by an operator panel during which two UAV operators, USAF Major James ‘‘Rainman’’ Hoffman (Predator operator), and Jeff ‘‘Goldy’’ Goldfinger (retired Navy Pioneer operator), from Brandes Associates, Inc. (BAI), shared their experiences in operating UAVs, and highlighted the need for human factors in current and future UAV operations. The issue raised by this engaging panel can be found in Chapter 2 of this volume. It was the presence and energetic participation from real UAV operators that made both workshops unique and of immense value to participants.

7


Table 2.

Posters/Demos Presented at 1st Annual Human Factors of UAVs Workshop, Chandler, AZ, 2004. Title

Poster: Airspace coordination and air traffic control interaction requirements for UAV operators Demo: Supporting optimized manning for unmanned vehicles Poster: System for controlling a hijacked aircraft, Demo: Challenges UAV operators face in maintaining spatial orientation Poster: The influence of coordination and collaboration instructions on UAV team performance Poster: Interface for non-pilot UAV control

Presenters Hottman, Steve, Physical Science Lab; Wernle, Kenneth, 46th Test Group, Holloman Air Force Base, and Sortland, Kari, Physical Science Lab Lamoureux, Tab M., Bruyn, Lora E., and Webb, Robert D., Human systems Incorporated; Fraser, Spence, Schreiner Canada Limited Dr. Matos, Jeffrey A., and Milde, Jr., Karl F., M and M Technologies Gugerty, Leo Ph.D., Clemson University Park, Ernest, and Hinsz, Verlin B. Ph.D., North Dakota University (NDSU) Still, David Ph.D., IHMC; Temme, Leonard, Naval Aerospace Medical Research Laboratory, and Eskridge, Tom M.A., IHMC

‘‘General Human Factors Issues’’ session. During this session the human– machine interaction challenges in operating UAVs were discussed and approaches to overcoming those challenges were suggested. Jay Shively opened the ‘‘General Human Factors Issues’’ session by revealing the benefits (i.e., cost-efficiency) and drawbacks (i. e., increasing operator’s workload) of introducing automation into UAV systems design. He emphasized the need to make behavior of autonomous systems predictable to the operator. Further, challenges imposed by automation, such as increasing the operator’s workload with possible decreases in the operator’s situation awareness, were addressed by Thomas Spura from Lockheed Martin Systems Integration (see Chapter 14). He presented the ongoing design of the highly autonomous Unmanned Combat Armed Rotorcraft (UCAR), which is supposed to be manually operated in certain instances (i.e., for decisive weapon release). Mr. Spura emphasized the question of how to facilitate UCAR’s reliability in order to perform missions effectively. At the end of the session Dr. Daniel Risser (Advanced Engineering and Planning Corporation Inc. (AEPCO)) brought up the highly important topic of standardization. He raised concerns about lack of coordination between human-systems integration communities and suggested that the application of standard-system components (i.e., standard human-control interface) across many platforms is critical for improving human performance.

8

OLENA CONNOR ET AL.

‘‘Perceptual and Display Issues’’ session. During this session perceptual challenges associated with remote control of UAVs were examined. Dr. Leo Gugerty discussed the spatial-orientation problem and its effects on an operator’s performance. He overviewed his research efforts that helped to identify some of the strategies used by operators in solving the spatialorientation problem. He suggested that these strategies (i.e., ‘‘heading referencing’’ or when the operator refers to a map heading while orienting within 3D display) should be considered in interface design and in the development of training programs. Barbara Sweet compared characteristics of inner-loop control image (i.e., altitude description) and outer-loop control image (i.e., velocity description). She claimed that these characteristics necessitate conflicting design requirements, which lead to degradation of visual information displays. Barbara Sweet claimed that design guidance is needed to allow selecting image characteristics that are appropriate for a specific flight task. The ‘‘Perceptual and Display Issues’’ session was concluded by Mark Draper from AFRL who provided a brief overview of research aimed to improve UAV operator interfaces (see Chapter 11). He highlighted research on multi-modal interfaces, control of multiple UCAVs, and decision–support interfaces for a semi-autonomous vehicle. Again, the need for standard software was emphasized. The ‘‘Cognitive Issues’’ session allowed presenters to explore challenges associated with UAV operations and suggest some potential solutions. Missy Cummings opened this session by introducing the ongoing dispute on the number of UAVs (in her research – Global Positioning System (GPS) equipped missiles) that can be effectively controlled by a single operator. She concluded that there is no single answer to these questions, because the ability to control more than one vehicle varies depending on different factors, one of which is task requirements. She stated that future research is needed in this area, specifically focusing on tasks manipulation and associated states of the operator’s workload. The importance of acquiring and facilitating UAV crew skills, such as coordination, dynamic decision-making, and situation awareness, was stressed by Joan Ryder. To advance this topic Joan Ryder offered a potential solution for improving operator training and decision-making by implementing synthetic agents and visual tutors in UAV crew training programs. ‘‘Emerging UAV Technology’’ session. This session was dedicated to ongoing human factors research of innovative approaches and technologies in UAV design and operations. Michael Goodrich discussed the benefits of using mini UAVs for search missions. He presented a novel ‘‘physical icon’’ interface that was designed for mini UAVs applying telemetry data. By


9

merging video processing and interface designs this inventive interface provides an operator with ‘‘simplified reality,’’ a factor that significantly minimizes the operator’s cognitive workload. The ‘‘Emerging UAV Technology’’ session continued with Christopher Miller, who presented delegation approaches to Multiple Unmanned Vehicle Control and discussed their implementation and testing. One of these approaches, a ‘‘policy’’ interface, might be beneficial by allowing efficient use of battlefieldcommunications resources. Miller claimed that implementation of delegation approaches would benefit human intervention with autonomous systems (see Chapter 18). ‘‘Local Human Factors of UAVs Research’’ session. The final workshop session focused on human factors of UAVs research taking place in the Phoenix area. Dr. Kevin Gluck highlighted the advantages of cognitive modeling and discussed its application in AFRL research. Brian Schreiber, from Lockheed Martin, at AFRL/HEA, presented findings of a Predator UAV-simulator study that investigated whether operators’ performance is influenced by the amount and type of previous flying experience. The session was concluded by Nancy Cooke’s presentation on team research. She discussed studies on team performance, team process, and team coordination that were conducted in the context of the Cognitive Engineering Research on Team Tasks (CERTT) Laboratory UAV synthetic-task environment (see Chapter 20).

Posters, Demos, Lunch Talks, and Breakout Sessions The workshop included a poster and demonstration session in which projects relevant to human factors issues of UAVs were presented. The topics of some posters and demonstrations were introduced and discussed during the general sessions as well (see Table 2). All exhibited works are listed in Table 2. Workshop activities also included lunch discussions. For example, Dr. Steven Shope from U.S. Positioning, and Janie DeJoode from CERI, led a lunch discussion on the first day on ‘‘A Look into the Future of UAV Operations.’’ Shope and DeJoode highlighted interesting aspects of a project that they worked on in conjunction with the USAF UAV Battlelab to provide input into next generation design of command-and-control (see Chapter 22). The audience was extremely interested in the accounts from interviews with operators describing the command and control challenges of current UAV systems. For instance, it was mentioned that the UAV operators were

10

OLENA CONNOR ET AL.

Table 3.

Breakout Session Summaries. 1st HF of UAVs Workshop, Chandler, AZ, 2004.

Breakout Session Topics

Breakout Session Summaries

Cognition and perception

There are many issues regarding system automation and its relation to pilot decision-making. What should be automated, when should it be automated, and how can automation be overridden? Taxonomy of mission types can aid in determining when automation should take over for a pilot. Other more specific cognitive and perceptual issues related to automation include disorientation, fatigue, situation awareness, and training. A task analysis to identify core knowledge, skills, and abilities (KSAs) that UAV operators should possess and that therefore should factor into selection and training must be conducted. This analysis could be specific to a type of UAV or more generic. Those KSAs would then lead to a core set of skills that operators would be taught regardless of the UAV they were to operate y. After this ground school, UAV operators would then move onto actual training in light aircraft, and then progress to a specific UAV system. More research is needed on selection and training. The ability to take UAVs and integrate them into existing training and simulations is a major goal. Additional tools such as programming tools, operator training tools, and supervisory level tools need to be developed, tested, and validated. Displays should be standardized across platforms where it can be applied based on the mission without being overly restrictive. Research on haptic and tactile displays must also be done and will rely on definitions of UAV tasks. Taxonomy of classes of teams is needed, including factors such as team size, distribution, heterogeneity, stability, and distinguishing teams from groups. We need an understanding of the knowledge sharing that occurs during missions, at hand-offs, and for team situation awareness. We need to understand group processes such as trust, expectations, planning and workload and applying such measures to dynamic teams must be researched. Methods to assess team performance and to mitigate to improve teams are needed.

Selection and training

Simulation displays and design

Team process

11


Table 3. (Continued ) Breakout Session Topics Systems safety

Breakout Session Summaries The most important issue in integration of UAVs in the NAS is the ‘see and avoid’ problem Certification, standards, and ratings depend on the type of UAV. There are numerous types, sizes, and functions. It is a system not a vehicle. What are the medical qualifications for operators? A system that deals with safety and UAVs must be established

often overwhelmed by orders ‘‘from the top’’ that seemed to be exacerbated due to the compelling visual feed coming back from the UAV. Valuable and effective discussion occurred during the breakout sessions that focused on specific topics which included: cognition and perception; selection and training; simulation, displays and design; team process; and systems safety. The goals of the breakout sessions were to identify top requirements for human factors of UAVs research, which would help to set an agenda for studies needed in this area, and to determine key questions pertaining to human factors that are unanswered and require further examination. Five parallel 112 h breakout sessions were led by facilitators and were organized to encourage effective discussion among attendees who were from different affiliations, diverse backgrounds, expertise, viewpoints, terminologies, and interests. A brief description of each breakout session output is provided in Table 3. Summary of the 1st Annual Human Factors of UAVs Workshop: Results The 1st Annual Human Factors of UAVs Workshop was highly successful in achieving its stated objectives. Surveys completed by attendees indicated high satisfaction with the workshop organization, presented materials, and achievement of personal goals. Useful suggestions for workshop improvements were also provided. The two days of the workshop were filled with interesting presentations, exchanges of fresh ideas, research questions and approaches, and opportunities for future collaboration. The workshop was concluded with a tour of AFRL, during which Heather Pringle demonstrated the Distributed Mission Operation Testbed. In addition Drs. Jerry Ball and Kevin Gluck demonstrated the Predator UAV Synthetic Task environment. Lastly, Dr. Nancy Cooke gave a tour of the CERTT Laboratory.

12

OLENA CONNOR ET AL.

2nd ANNUAL HUMAN FACTORS OF UAVs WORKSHOP Following the successful 1st Annual Human Factors of UAVs Workshop, CERI organized and hosted the 2nd Annual Human Factors of UAVs Workshop on May 25–26, 2005, in Mesa, AZ. The outcomes of the first workshop were examined and considered in stating the goals for second workshop. In particular, human factors issues of UAVs identified during the first workshop were targeted along with other human factors challenges associated with envisioned UAV operations. Moreover, an important objective of the 2nd Annual Human Factors of UAVs Workshop was to attract more individuals from two communities who were underrepresented at the 1st Annual Human Factors of UAVs Workshop – UAV operators and UAV developers. Dr. Nancy Cooke, Ph.D. (CERI) and Major Heather Pringle, Ph.D. (AFRL) were again the main organizers and co-chairs of 2nd Annual Human Factors of UAVs Workshop. The workshop was sponsored by AFOSR, AFRL, Micro Analysis and Design (MA&D) and the Federal Aviation Administration (FAA). Proceedings of the 2nd Annual Human Factors of UAVs Workshop Dr. Cooke and Major Pringle chaired the opening session. After welcoming the attendees they summarized the achievements and outcomes of the 1st Annual Human Factors of UAVs Workshop, and updated the attendees on research progress relevant to human factors of UAVs. They further highlighted the present and upcoming problems that need to be addressed in this area. Dr. Cooke and Major Pringle outlined the objectives of 2nd Annual Human Factors of UAVs Workshop, briefed the attendees with the workshop structure, venue, and available activities. Representatives from each of the workshop sponsors spoke during the opening session. Kip Krebs from the FAA, briefed the audience with FAA perspectives on the operation of UAVs. Another talk was given by Patricia McDermott, from MA&D. Dr. Hendrick Ruck, Ph.D., Director of the Human Effectiveness Directorate from AFRL, presented an overview on the ‘‘Human Effectiveness Directorate.’’ The opening session was highlighted by a presentation by Colonel Michael Colangelo, of the Arizona Air National Guard who had been recently tasked with setting up a Predator squadron in Arizona. Colonel Colangelo described the challenges associated


13

with his new project and highlighted, among other things, the new issues that arise when UAV operators do battle by day and go home to ‘‘play soccer with the kids’’ at night. The 2nd Annual Human Factors of UAVs Workshop consisted of oral presentations, breakout sessions, and a poster/demo session. In the following sections we highlight issues discussed during the paper sessions. See Table 4 for a complete list of paper presentations and Table 5 for the list of poster/ demo presentations. Among the main themes of this workshop were issues of controlling multiple UAVs; training and selection of the operators; UAV integration with other air, ground, space, and sea operations (military and civilian); issues pertaining to other types of remotely operated vehicles (ROVs); perceptual and cognitive challenges in remote control of UAVs and sensor operation; and team coordination and information management. ‘‘Research Testbeds’’ session. Human factor researchers have the ability to conduct effective and cost efficient research in context of Synthetic Task Environments (STEs). STEs are designed with aim to extract and examine certain aspects of studied phenomena. In comparison to expensive and elaborative task simulators, which tend to replicate the physical aspects of the studied environment, STEs replicate the task, allowing the researcher to create and manipulate task scenarios that evoke particular behavioral and cognitive phenomena of interest. The workshops’ presentations revealed that STEs, or research testbeds, enabled human factors researchers of UAVs to conduct successful experiments in studying UAV-relevant cognition and behaviors. Dr. Cooke presented findings of several studies conducted in the context of a UAV ground control simulation in the CERTT Laboratory. The importance of team coordination skill, acquisition and retention of such skill and factors influencing these phenomena were discussed (see Chapter 20). Dr. Stephanie Guerlain spoke about the Multi-Operator Simulation Testbed developed at the University of Virginia, and demonstrated the system’s planning and mission components. Dr. Verlin Hinsz from North Dakota State University, discussed his study of UAV team behavior, interaction and performance in the context of the BRUTE (Basic Research UAV Task Environment) synthetic task environment (see Chapter 21). Techniques to facilitate operator performance, such as improving operators training and selection, were highlighted. ‘‘Interface Issues’’ session. Efficient display of information is a main focus of user-interface studies. This factor is one of critical importance in designing interfaces for UAV systems, because UAVs operators are deprived of sensory and certain perceptual information (i.e., information which is available to manned aircraft pilots). The studies addressing these concerns, and

14

OLENA CONNOR ET AL.

Table 4.

Oral Presentations given at 2nd Annual Human Factors of UAVs Workshop, Mesa, AZ, 2005.

Opening Remarks Welcome and logistics FAA perspective Micro Analysis and Design Perspective Human Effectiveness Directorate: Overview

Arizona ANG Predator Squadron: information briefing Research Testbeds Acquisition and retention of team UAV skills

A multi-operator simulation testbed for studying the human factors of UAV mission planning and execution

BRUTE: A versatile research platform for studying UAV operator behavior Interface Issues Synthetic vision system for improving UAV operator situation awareness Portable, mini-UAV control interfaces for the non-pilot Intelligent attention management tools for tactical situation awareness and supervisory control Evaluation of a touch-screen based operator control interface for training and remote piloting of a simulated micro-unmanned aerial vehicle Automation & Control of Multiple UAVs Command and control of tactical UAVs: optimizing probability of detection Development of prototype operator interfaces for the control of multi-UAVs from an airborne platform Integrating cognitive decision aiding and level IV control of an unmanned air vehicle in the AH-64D Apache: enhancing aircrew tactical decisions

Cooke, N.J.; Cognitive Engineering Research Institute (CERI) Adams, R., and Smith, W.; FAA McDermott, P.; MAAD Ruck, H., Ph.D.; SES Director of Human Effectiveness Directorate, Air Force Research Laboratory Colonel Michael Colangelo; AZANG

Cooke, N. J., Connor, O.O., and Pedersen, H.; Cognitive Engineering Research Institute (CERI) Guerlain, S. (University of Virginia), Allen, S. (Emiror Inc), O’Hargan, K. (University of Virginia), Jones, C. (University of Virginia), and Van Eron, K. (University of Virginia) Hinsz, V.B.; NDSU

Calhoun, G., and Draper, M.; AFRL/HECI Goodrich, M.A., Quigley, M.L., Cooper, L., and Barber, B.D.; BYU St. John, M.; Pacific Science and Engineering Group Durlach, P. J., and Neumann, J.L.; U.S. Army Research Institute for the Behavioral Sciences, Simulator Systems Research Unit

Urioste, M., and Archbold, S.; Lockheed Martin Hou, M. (Defense R&D Canada), and Kobierski, R. (CMC Electronics Inc) Faerber, R. A., and Cloud, T. M.; Boeing Company

15


Table 4. (Continued ) Opening Remarks Coordination and control of cooperative swarms of unmanned combat aerial vehicles via a virtual testbed environmentupdate Analogies Coordination of network centric multiple UAV operations through an intuitive visual interface Human factors challenges for unmanned undersea systems. Evaluation of robot-operator performance. Applying advanced aviation concepts to the integration of unmanned aircraft into the National Airspace System Safety and Research Programs U.S. military unmanned aerial vehicle mishaps: assessment of the role of human factors using HFACS Federal aviation administration unmanned aircraft human factors research program Air Force RPA Roadmap for Human Systems Integration Developing Consensus Standards for FAA Certification of UAV Flight Crew

Findler, M., and Hill, R.; Wright State University

Hunn, B.P.; US ARL

Robertson, D.J.; Lockheed Martin Maritime Sensors & Systems Moses, F.L., and Brooks, P. S.; Institute for Defense Analyses Hitt, J.M.; Booz Allen Hamilton

Tvaryanas, A.P. (311th Performance Enhancement Directorate), Thompson, B.T. (USAFSAM/FEC), and Constable, S. H. (HSW/PER) Williams, K., and Krebs, W.; FAA, CAMI Hover, G., and Tvaryanas, A. P.; Air Force Goldfinger J.; BAI

specific interface design suggestions aimed to facilitate operator’s performance, situation awareness and decrease workload, were presented in the ‘‘Interface Issues’’ session. Gloria Calhoun brought up subject of UAV operators’ situation awareness and how it can be affected by interface design. To aid UAV operators’ situation awareness, a synthetic vision overlay technology was introduced and its strengths were discussed. Joseph Cooper discussed the benefits of miniature UAVs in search and rescue missions. He presented and demonstrated a portable Mini-UAV control interface that was developed to facilitate the search and rescue process by allowing operators to concentrate on video images and not on UAV system monitoring. Mark St. John introduced the following intelligent attention management tools: Assisted Focus, and Change History Explicit (CHEX). The benefits of these tools in maintaining and recovering user’s situational awareness, while performing monitoring tasks in dynamical situations, were discussed.

16

OLENA CONNOR ET AL.

Table 5.

Posters/Demos Presented at 2nd Annual Human Factors of UAVs Workshop, Mesa, AZ, 2005. Title

Poster: Relaxed Constraint Planning and Constraint Negotiation: an unmanned vehicle playbook that goes beyond ‘‘I can’t’’ Poster: controlling teams of UAVs

Poster: the communication of UAV-provided information Poster: using verbal protocol analysis and cognitive modeling to understand strategies used for cardinal direction judgments Poster: general aviation views on UAVs Poster: determining the aviation knowledge and skills in which unmanned aerial vehicle operators must be proficient to achieve certification. Poster: spatial disorientation in uninhabited aerial vehicles. Poster: virtual teleoperation for UAVs. Demo: Coordination of Network Centric Multiple UAV Operations through an intuitive visual interface. Demo: demonstration of human-UAV control link. Demo: demonstration of handheld UAV controller. Demo: demonstration of portable, mini-UAV control interfaces for the non-pilot. Demo: demonstration of BRUTE: a versatile research platform for studying UAV operator behavior. Demo: intelligent attention management tools for tactical situation awareness and supervisory control. Demo: command and control of tactical UAVs: optimizing probability of detection. Demo: a multi-operator simulation testbed for studying the human factors of UAV strike planning and execution.

Presenters Miller, C.A., Goldman R.P., Funk, H.B., and Wu, P.; SIFT

Scerri, P. (Carnegie Mellon University), Sycara, K. (Carnegie Mellon University), and Lewis, M. (University of Pittsburgh) McDermott, P.; MA&D Rodes, W.M., Brooks, J., and Gugerty, L.; Clemson University Wischmeyer, E.; Embry Riddle Aeronautical University Hottman, S.; PSL/NMSU

Olson, W.; U.S. Air Force Academy Oliver J. & Sannier, A;. Iowa State University Hunn, B.P.; US ARL

Pye, J.; Exponent Limbaugh, D. V.; Kutta Consulting Goodrich, M., Quigley, M., Cooper, J., and Barber. B.; BYU Hinsz, V.; NDSU

St. John, M.; Pacific Science & Engineering Group Urioste, M.; Lockheed Martin Guerlain, S., Allen, S., O’Hargan, K., Jones, C., Van Eron, K.; University of Virginia


17

‘‘Automation & Control of Multiple UAVs’’ session. Last year’s workshop revealed that issues regarding the control of multiple UAVs, their autonomy, and operator workload appeared to be of critical importance to UAV operations. These issues were emphasized in the ‘‘Automation & Control of Multiple UAVs’’ session. Marcus Urioste from Lockheed Martin, highlighted several concepts that allow multiple UAVs to be controlled by a single operator. He suggested that such control could be achieved by optimizing the tactical sensing mission, which increases the probability of detection and reduces the operator’s workload. The 2nd Annual Human Factors of UAVs Workshop also attracted international attention. A guest from Canada, Robert Kobierski (CMC Electronics Inc) introduced a multiyear program that was initiated by Defense Research and Development Canada (DRDC). Efforts and results of developing and designing an Intelligent Adaptive Interface (IAI) for efficient control of Multiple UAVs were discussed. Additional details on this topic can be found in Chapter 19 of this volume. During this session the presenters did not dispute whether the control of UAVs should be fully automated of fully manual. The main consideration was the allocation of automation and factors determining when and how the automation should be introduced. Benefits (i.e., cost efficiency) and drawbacks (i.e., cognitive workload demands) of automation were discussed and approaches to decrease workload and facilitate situation awareness and performance of operator during multiple UAVs control were emphasized. ‘‘Analogies’’ session. The topics of the second workshop were expanded and enriched by including presentations on a wider range of ROVs. The workshop’s ‘‘Analogies’’ session was dedicated to identifying human factors challenges that are similar between UAVs and other types of ROVs (i.e., unmanned ground vehicles (UGVs) and unmanned underwater vehicles (UUVs)), and to highlight those human factors issues that are unique to specific types of ROVs. The main application of all ROVs is to keep the human operator out of harm’s way while performing real-time activities such as target acquisition, demining, or search and rescue. However, the nature of remote operations poses challenges that are similar across all types of ROVs, such as lack of sensory feedback, and spatial disorientation, etc. In addition, specific problems are associated with each type of ROV, because each type of vehicle performs distinctive tasks defined by the environments in which ROV operates. For instance, UGVs have significant mobility in rough terrain and in some instances the positioning of the vehicle itself (e.g., upside down, right side up) is unknown. Donald Robertson, from Lockheed Martin Maritime Sensors and Systems, acquainted the audience with the design process of the Remote Minehunting System (RMS) that

18

OLENA CONNOR ET AL.

affords the deployed Commander a remote evaluation of maritime battle space. RMS is the U.S. Navy’s first unmanned mine reconnaissance system and thus there are many human factors challenges associated with its development. Although some human factors issues are similar between UAVs and UUVs (i.e., lack of sensory input), Robertson highlighted specific aspects of controlling a UUV, such as ‘‘controlling a 14,000 pound vehicle in close proximity to large, moving ship’’, and described research addressed identified aspects. Further, Dr. Frank Moses introduced UGVs research with a goal to make his research outcomes applicable to all ROV types, i.e., non-task and non-platform dependent. Based on the results of small ground robot studies, Dr. Frank Moses proposed generalized quantitative measures of operator performance that can be applied to small UGVs and UAVs (i.e., frequency and time between control actions). The process of air traffic controller operations represents an analogy to intuitive visual–display interface that allows the management of a network of multiple UAV operations from a command and control center. Bruce Hunn, from the U.S. AFRL, presented this innovative visual display technology, which merges real time video imagery with virtual imagery in order to effectively present a large amount of information to a UAV operator. Bruce Hunn’s research is described in Chapter 13 of this volume. ‘‘Safety and Research Programs’’ session. In this session the following topics were discussed: the process of integrating UAVs in the National Air Space (NAS) and associated problems, lack of standards and regulations, and human errors resulting in UAV mishaps. Major Anthony Tvaryanas from Performance Enhancement Directorate, briefed attendees with a study on UAV mishaps that occurred within the U.S. military services during the last 10 years (see Chapter 10). Applying the Human Factors Analysis and Classification System (HFACS) human error-related mishaps were identified. Anthony Tvaryanas revealed that latent errors are significantly different between the services, and that mishaps associated particularly with ‘‘human factors were more frequent in the Air Force followed by the Navy and the Army.’’ Dr. Kevin Williams summarized FAA activities aimed at understanding the role of UAVs in the NAS, and introduced FAA plans on developing standards and guidelines for integrating UAVs in the NAS. Jeff Goldfinger wrapped up the ‘‘Safety and Research Programs’’ session. He discussed the problems of UAV integration into the NAS, and suggested that creating certification standards would mitigate this problem. In this regard, progress of ASTM International (formerly known as the American Society for Testing and Materials), organization that is working on this issue, was discussed.


19

Posters, Demos, Lunch Talks, and Breakout Sessions Valuable insight into the human factors of UAVs was gained during the poster and demo session, which was expanded this year to include even more exhibits. Projects presented during this session are listed in Table 5. The workshop format included six 112 h parallel breakout sessions, which were led by facilitators and were organized around the following topics: human performance; selection and training; multiple UAVs control; mishaps, near misses, and error documentation; integration into the NAS. These topics were identified based on issues that were raised during the first workshop. At the beginning of each breakout session a facilitator acquainted the participants with questions that still need to be addressed. A brief description of each breakout session output and summaries of some newly identified gaps in human factors research are given in Table 6. In addition to research papers focusing on human factors issues pertinent to UAVs, participants in the second workshop heard the perspectives from SSG Christopher Gunter and SGT Denton Lytle (Fort Huachuca, AZ) and some other former operators present, during an informal question and answer period over lunch. Summary of the 2nd Annual Human Factors of UAVs Workshop: Results Compared to the 1st Annual Human Factors of UAVs Workshop, the second workshop focused more intensely on the variety and diversity of research approaches and perspectives and on demonstrations of new UAV technology aimed at designing more useful and usable UAV systems. The second workshop was of great value to attendees. They were able to share their experiences, thoughts, and suggestions in a workshop survey that was included in their registration packets. The opinion of one participant characterizes workshop experience succinctly: ‘‘I learned about the diversity of on-going study in the field. It was also helpful to know the interests and activities of the various organizations (military, civil, and academic).’’ About 122 participants attended this workshop and we had an amazing diversity represented by academia (26%), industry (43%), government (31%), and in addition this workshop attracted international attention. The 2nd Annual Human Factors of UAVs allowed the research agenda to continue to be responsive to the constraints and needs of these communities. The topics presented during the workshop targeted previously identified gaps, and included a broad variety of issues concerning ROVs, scientific, and commercial applications of UAVs. In addition, the workshop was

20

OLENA CONNOR ET AL.

Table 6.

Breakout Session Summaries. 2nd HF of UAVs Workshop, Mesa, AZ, 2005.

Breakout Session Topics Human performance

Selection and training of operators

Break Out Session Summaries The UAV industry is searching for solutions to latency problems and lags in control inputs for sensor operators (SO). Predictive capabilities and feature tracking by computers are possible solutions – not saving milliseconds in control for tracking targets. Soda straw views for operators make distances hard to judge. Video overlays and scaling maps will help operators to judge distances. Many mishaps are due to maintenance issues. Possible solutions include simplifying maintenance jobs with ‘‘Pit Stop Engineering,’’ and the consideration of the human factors of maintenance by system designers. Fatigue is a major issue in the military and therefore, systems should be designed for degraded states of mind in consideration. Giving the fatigued operator feedback in different modalities (i.e. tactile and auditory vs. visual only) is key in mitigating workload and keeping the operator engaged. The FAA is in search of guidelines for UAV operations in the NAS which need to consider factors such as system weight, speed, control (i.e. LOS), airspace class, interface type, etc. Proposed track for commercial operator certification: (1) All operators undergo FAA approved ground school training; (2) Training and qualification in specific platform interface (point-and-click vs. stick-and-rudder); (3) Training and qualification in IFR or VFR (LOS control vs. Internal GCS control) Operators will also need to be trained in communication and coordination with ATC and other aircraft to safely operate in the NAS.

structured to increase the opportunities for discussion and networking. Appropriate venues for reporting projects were presented and the results of the workshop were identified. Along these lines, this volume represents an effort to ‘‘spread the word’’ that there are human factors in UAVs.

2. UAV HUMAN FACTORS: OPERATOR PERSPECTIVES Harry K. Pedersen, Nancy J. Cooke, Heather L. Pringle and Olena Connor INTRODUCTION The Cognitive Engineering Research Institute’s First Annual Human Factors of unmanned aerial vehicles (UAVs) Workshop, held on May 24–25, 2004 in Chandler Arizona, and Second Annual Human Factors of UAVs Workshop, held on May 25–26, 2005 in Mesa Arizona, brought to light many human factors issues regarding the technology and operation of UAVs. An integral part of the event was the involvement of military UAV operators from the U.S. Air Force (USAF), U.S. Navy, and U.S. Army. The involvement of UAV operators in the workshops was valuable in linking developers and human factors researchers in the improvement of UAV systems and operations – a practice that is too often implemented only after a system is deployed and the problems are found. The experience of operators serves as a ‘‘user’s account’’ of the issues and problems concerning the operation of UAVs. The fact that operators have had first hand experience in operating UAVs provides a unique perspective to the problem of identifying the most pressing human factors issues. The purpose of this chapter is to highlight the perspectives of two UAV operators that helped to set the tone for the entire First Annual Human Factors of UAVs Workshop.

Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 21–33 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07002-5

21

22

HARRY K. PEDERSEN ET AL.

THE OPERATORS ‘‘Rainman,’’ a major in the USAF, was the first pilot selected to fly the Predator RQ-1A UAV in 1995. At Creech Air Force Base Nevada (Formerly Indian Springs Auxiliary Airfield), he flew the Predator RQ-1A in numerous training and test missions and was also deployed to Bosnia and Herzegovina where he flew over 100 combat missions in support of Operation Joint Endeavor and Operation Joint Guard. After flying the Predator, he proceeded to Langley AFB, VA where he was Chief of the UAV Combat Applications Branch for the Air Superiority Mission Area Team, ACC Headquarters. During this assignment, Major Hoffman became the lead agent for weaponizing the Predator with the Hellfire missile capability within the HQ Directorate of Requirements. The second operator, ‘‘Goldy,’’ was a Pioneer UAV mission commander and internal pilot and served as the officer-in-charge of the sea-going Pioneer detachment that deployed to the Arabian Gulf aboard the USS DENVER. In addition to his Pioneer qualifications, Goldy completed a 20year career as a Naval Flight Officer and accumulated over 1,500 flight hours in various tactical aircraft including the E-2C Hawkeye, FA-18B Hornet, and the A-6E Intruder. During his various shore duty assignments he completed three tours as an advanced tactics instructor and one tour as an Aviation Mishap Investigation and Reporting Instructor. He is currently employed by Brandes Associates Inc. as the Director of Unmanned Aircraft System (UAS) Solutions.

HUMAN FACTORS ISSUES IN THE GROUND CONTROL STATION Rainman’s concerns with human factors in UAVs were driven by a congressional mandate, which stated that by 2010, one-third of the U.S. Military’s deep strike aircraft are to be remotely piloted. If this mandate is to be realized, many technological improvements will be necessary in both current and future systems to allow for the safe and effective operation of UAVs. The current system in which Rainman concentrated his presentation was the Predator RQ-1 in which he has had much experience. The operation of the Predator is a very complex undertaking and there are many issues within the ground control station (GCS) and the aircraft itself, which require attention from the human factors community. One issue

UAV Human Factors: Operator Perspectives

23

in Predator operations discussed by Rainman was the landing phase, which is extremely difficult due to a mere 30-degree field of vision from the flight camera. During landing, there is no sense of the ground in the periphery as in manned planes. Thus, landing requires the operator to look down the runway and ‘‘hope for the best’’ upon touchdown. This problem, coupled with the fact that Predators must fly a very steep glide slope during landings (again, due to the forward-mounted flight camera) contributes to the mishap rates seen during landings. Another issue Rainman discussed which could possibly contribute to mishaps is the mapping of various functions to function keys on the operator’s keyboard. For instance, the keys that turn on/off the lights and the keys that cut the engine are located adjacent to each other. In a dynamic, high workload environment with heavy coordination, an operator may make a mistake leading to the destruction of the UAV. Another issue with the Predator and other UAVs such as the Global Hawk is the lack of auditory and haptic cues and feedback normally found in manned craft. For example, pilots in manned aircraft can hear when an engine is failing and can act immediately whereas a Predator operator will only be able to see the failure by accessing menus on the interface. When an engine stalls, pilots in manned aircraft can ‘‘feel’’ the controls become unresponsive. UAV operators however, have no haptic feedback and must rely on instrumentation and the flight camera to diagnose any problems in flight. Potential solutions presented by Rainman included haptic feedback in throttle controls to provide a sense of flight, vibrating seats to provide a sense of stalls, and seats that rotate and bank in concert with the UAV to further give the operator a sense of motion. Goldy’s presentation also addressed various topics related to issues concerning the interface and control schemes in UAV systems. He outlined a series of goals for human factors researchers and developers to consider, with the overarching theme of designing safe, efficient, and effective controls: Ergonomic goal: Minimize physical fatigue Cognitive goal: Minimize mental fatigue Response goal: Minimize UAV response time. Ergonomics of the GCS should fit the form of the human body and should provide the operator with a comfortable environment with even temperature and lighting. Mental fatigue can be minimized by the appropriate placement of font and text as well as the appropriate use of shapes and symbols. The response goal can be met by studying the advantages and disadvantages of traditional stick-and-rudder controls vs. point-and-click

24


interfaces vs. hybrid control schemes. The response goal was of particular interest to Goldy as he pointed out that little research (if any) as to which interface, point-and-click vs. stick-and-rudder, is better. He also pointed out that what constitutes ‘‘better’’ must also be defined. Fig. 1 below shows different interfaces of two current UAV systems. Goldy also pointed out several cockpit design principles for manned aircraft that should be carried over to the design of UAV GCSs (Ciavarelli, 2003). The following principles were taught by Dr. Anthony Ciavarelli at the Naval Postgraduate School of Aviation Safety as a means by which aircraft mishap investigators use to determine whether or not cockpit design played a role in the mishap: Displays and controls should operate in accordance with provided documentation. Positive transfer of previous training habits should be encouraged by standardization of displays and controls. Confusion between similarly arranged controls and displays should be minimized by implementing formats that allow them to be distinguished from one another.

(a)

(b)

Fig. 1. Predator UAV Stick-and-Rudder Controls (Left Panel) Vs. Point-and-Click Interface Found in UAVs Such as the Global Hawk (Right Panel) (from J. Goldfinger, 2004) Photo Courtesy of Northrop Grumman. Reprinted with Permission. Note How the Operator is Holding the Joystick in the Picture on the Left and how the Buttons are Located on Top of the Stick.


25

Instrument scan time and operator workload can be minimized by the logical arrangement of displays and controls. The goal of human factors researchers and practitioners, as well as engineers and system designers, should be to apply the design principles above and collaborate to insure that such mishaps do not occur. For example, ergonomic issues in UAV GCSs still require much improvement as evidenced by the operator in Fig. 1 who is holding the GCS joysticks with only her forefingers. Each joystick in Fig. 1 is clearly meant to be gripped with the entire hand. However, to do so would be fatiguing for the operator because they are placed so high. One simple change would be to include an armrest to combat fatigue. Also, any changes must be empirically tested to ascertain whether the change is an improvement. Goldy pointed out several issues with the current Predator UAV operator interface that require the attention of engineers, designers, and human factors practitioners. One problem concerns the phenomena of chromostereopsis and eye fatigue, which results from the fact that the Predator headsup-display (HUD) uses red graphics on a blue (sky) background. The HUD also employs sliding bars that do not resemble those in manned aircraft such that the numbers representing data (i.e., altitude) change physical locations where pilots are originally trained with sliding bars that do not move. The use of symbology in the HUD is also not intuitive. For example, in Fig. 2, three triangles are displayed at the bottom of the screen indicating that the landing gear are up. Also, the letters ‘‘AED’’ at the top of the HUD are a mystery – not even Rainman knew what they indicated. There are many issues with point-and-click interfaces as well. For example, in the Global Hawk interface shown in Fig. 1 above, the horizon rotates which is the complete opposite to what pilots of manned aircraft are trained to use. In manned aircraft, the gyro rotates while the horizon is fixed. Also, the tick marks indicating 30 degrees on the altitude indicator are not actually 30 degrees, but are closer to 70 degrees, which is extremely inaccurate. In the case of the Predator UAV, the Air Force Cockpit Working Group at the Air Force Research Lab has identified these design deficiencies and has initiated a program to correct them. The improved HUD (I-HUD) shown in Fig. 3 has been developed and is being retrofitted into the software loads on existing Predator GCSs. The examples provided by Rainman and Goldy have shown that the current doctrine for operators (in the U.S. Navy and U.S. Air Force) call for training pilots in the operation of manned aircraft, and then training them in UAV operations with large amounts of negative transfer in interfaces that

26


Fig. 2.

Fig. 3.

Predator UAV Heads-up-Display (from J. Goldfinger, 2004).

Improved Predator UAV Heads-up-Display. Photo Courtesy of General Atomics. Reprinted with Permission.


27

are flawed and largely untested. While the interfaces are in need of human factors improvement, the examples above also point to a need for examining the criteria to which the operators are selected and trained, and the standards to which they and the UAV community must adhere.

OPERATOR TRAINING AND SELECTION Even though technology is rapidly approaching the point where real-time control may no longer be necessary, human operators are, and will be, an integral part of system control. Rainman raised several important points on this subject. Current UAVs that are capable of autonomous flight still require humans for situation awareness, occasional airspace deconfliction (i.e., when landing), communication with air traffic control, and filing of flight plans. Even ‘‘fully autonomous’’ UAVs such as the X-45 may need real-time intervention (i.e., deconfliction in combat environments when working in tandem with other aircraft) and thus, will require humans to remain ‘‘in the loop’’ or ‘‘on the loop’’ depending on the system design. Therefore, it is apparent that regardless of a particular UAV system’s control scheme, operators will always be needed and thus, selection and training remains an integral part of any UAV system. The current method of operator selection and training the USAF employs is to take pilot training graduates who have flown airplanes such as the B-52, T-38, T-37, and T-1, and put them through UAV training. As UAV production and use grows, more pilots will be taken away from manned aircraft duties. Rainman estimated that the USAF still requires 1,200 pilots a year to maintain current manning for piloted aircraft and may not be able to afford losing those pilots for UAVs. One possible solution may be to install a new training pipeline where UAV operators are trained for instrument flight rules (IFR) in an aircraft such as a Cessna 172 for 80 hours (in order to gain an instrument rating as well as flight experience) after which potential UAV operators go on to Predator or Global Hawk simulator training. From there, operators will move on to initial qualification training, and finally mission qualification training. The proposed benefits of this program are: increased volunteers and morale, retention of experienced, dedicated UAV personnel (since the turnover rate will be lessened), and alleviation of manned aircraft pilot demands. Finally the cost savings is, according to Rainman, the biggest advantage. The total cost to train a B-52 pilot is an estimated $685,051 vs. only $17,800 for a dedicated UAV operator that does not require retraining, survival training, etc (Hoffman, 2004).

28


Goldy began by posing the question: ‘‘what ‘kind’ of humans are needed to operate UAVs?’’ Everyone in the UAV community agrees that humans are needed, but what kind of knowledge, skills, and abilities they should possess has not been decided. For example, do we require officers or enlisted men and women? The USAF utilizes fully trained officers whereas in contrast, the U.S. Army utilizes enlisted personnel who are essentially selected at boot camp to fly the Shadow UAV. The U.S. Navy and Marines take a ‘‘middle-of-the-road’’ approach and utilize enlisted personnel who have aviation maintenance experience and train them to operate the Pioneer UAV. There is a definite difference in philosophy between the branches and this will need to be standardized in the future if and when UAVs should ever take to the skies in the national airspace (NAS) under the military for uses such as border patrol and search and rescue. In addition, any UAV operations conducted in the NAS will also have to be coordinated with the FAA, which also has its own emerging views regarding UAVs. For example, the FAA previously called UAVs ‘‘Remotely Operated Aircraft’’ or ROAs because they did not want to give the public the impression that the aircraft are unmanned. However, the FAA’s views are evolving as they have now adopted the terms ‘‘Unmanned Aircraft System’’ when referring to UAVs (Williams, 2006). On a related note, even the USAF has changed their nomenclature, now calling UAVs ‘‘Remotely Piloted Aircraft’’ (RPA) to acknowledge the fact that there is a human in the loop. Goldy also discussed the importance of selection criteria for operators. For manned aircraft, there are pre-selection criteria such as intelligence tests and medical qualifications that must be passed before training is even considered. UAV operators have no such pre-selection criteria whatsoever. For example, in the U.S. Marines, all that is required to become a UAV operator is to volunteer and have an interest in radio controlled (RC) model airplanes. So why are the branches of the military doing things so differently? More importantly, which branch is correct? These are questions that as of yet, have not been answered. What is known, however, is that the branches agree that operators require some minimal amount of flight training (the amount of which notably must be determined through research and testing) and that dedicated UAV training and personnel holds merit. On a promising note, changes are indeed being made in the operator-selection arena. According to Goldy, the U.S. Army is in the process of changing its doctrine. In essence, the U.S. Army is realizing that when it comes to UAV operation in the NAS (i.e., for homeland security), acquiring operators from bootcamp is not going to be enough. In fact, the Army is beginning to require Hunter and Shadow UAV operators to go through FAA sanctioned ground school training.


29

Why should we concern ourselves with human operators in such systems? An important reason that was discussed by Goldy is the occasional need for dynamic re-tasking of UAVs. For example, he recounted a story in which a Helios UAV was used to monitor the ripeness of beans in a coffee plantation in Hawaii. The plan was for the UAV to autonomously perform a standard ladder search pattern. However, the day turned out to be cloudy and necessitated the re-tasking of the flight such that the UAV was directed to take snapshots where there was no cloud cover. The mission was accomplished in a random, scattered fashion and not the pre-planned, autonomous way that was originally conceived. This re-tasking could not have been accomplished without human intervention demonstrating the need for humans to remain in the loop in even highly automated systems. Goldy also brought up an interesting point regarding the operation of UAVs – the lack of medical standards for operators. In manned aviation, there are strict standards to which pilots must adhere. For example, if you are an alcoholic, you will be disqualified regardless of whether you are in the military or you are a civilian. There are also universally accepted standards for temporary groundings (i.e., temporary decreased visual acuity due to surgery, or decrement in equilibrium due to sinus infection). However, the lack of medical standards for UAV operators, coupled with the lack of operator selection criteria, opens the possibility of serious mishaps and poses a danger to other aircraft (manned or unmanned) operating in the area as well as to people and structures on the ground. For example, up until several years ago, Naval Pioneer aircrew were still able to perform their jobs even if they had just imbibed an alcoholic beverage before reporting to duty because the ‘‘12-hour bottle to throttle’’ rule did not apply to the Pioneer program. This is especially alarming considering the fact that the Naval Pioneer UAV weighs 450 pounds and flies at 70 knots. Even the lightest UAV colliding with a passenger airliner can have dire consequences – not only for those passengers in the airliner, but also for the people on the ground. Related to the lack of medical standards and qualifications for UAV operators is the lack of physicals as well. Hunter UAV operators in the U.S. Army receive the same physical examinations that non-pilot helicopter crewmen receive. In the U.S. Navy, Pioneer operators receive physical examinations comparable to those that air traffic control (ATC) personnel receive. Goldy’s concern with this is that while the two systems are essentially identical in technology and interface, there are nevertheless, two different standards for physicals. Therefore, the problem which must be overcome lies not only in developing selection criteria and medical standards

30


for UAV operators, but also standardizing them based on the same criteria used for manned aircraft. Which criteria to use is also another question that must be answered by human factors practitioners, designers, operators, and entities such as the FAA. Such developments must also not be exclusive to the military because increased UAV operations in the NAS will greatly benefit from the development of such standards as well.

THE FUTURE OF UAV OPERATIONS Rainman was especially concerned with the deconfliction of airspace being a major issue in the coming years. Fully autonomous UAVs such as the X-45 Unmanned Combat Air Vehicle (UCAV), which are projected to work in ‘‘swarms’’ alongside manned aircraft will add to the danger and mishap rates which are already unacceptably high. The problem is compounded by the fact that the USAF is exploring the feasibility of one operator monitoring four UCAVs which brings about questions on how to provide deconfliction not only between the UCAVs and manned aircraft, but also between the UCAVs themselves. In the foreseeable future, Rainman believes that humans must remain in the loop. Even for automated systems such as the X-45 and Global Hawk, where humans are reduced to a supervisory role, there must be feedback that confirms actions, and controls which allow a sense of flight should operators need to manually control the craft. Tools and displays must also be developed, which will allow operators of these craft to maintain situation awareness of their surroundings. The problem of airspace deconfliction is even more important when the operation of UAVs in the NAS is considered where operators and their UAVs must fly in airspace occupied by general aviation craft and passenger airliners, as well as land and takeoff in the same airports. Humans will also be required to remain ‘‘in the loop’’ for dynamic re-tasking and communication with other aircraft and ATC, and to file flight plans in a manner similar to what pilots of manned aircraft do. This will especially be relevant for homeland security, border patrol monitoring, and search-and-rescue in which UAV flights might be conducted over highly populated areas. That UAVs will be used in the NAS also speaks to the different types of missions of the future. Rainman reported that by the year 2025, 80 percent of the third world population will live in urban city settings. As fights become increasingly urban in nature (i.e., the war against terrorism in Iraq) and troops are vulnerable to an enemy that is more familiar with the surroundings, the use of UAVs in urban close-air support will be an invaluable


31

asset to the warfighter. Miniaturized rotorcraft will be able to conduct reconnaissance for ground troops, searching for enemy emplacements. However, questions such as how these craft will be controlled, and who will control them – especially in a busy, dynamic environment – remain to be answered. Another area in which UAVs will be useful is in homeland defense. Rainman discussed a common scenario – the hijacked airliner. The current procedure when a passenger airliner is hijacked calls for sending a manned fighter for confirmation and attempts to talk the plane and hijackers down. The alternative is to obtain permission from the President or Secretary of Defense to destroy the airliner. However, the same data link technology that allows UAVs to fly can be adapted to take control of the hijacked craft and bring it safely down. However, yet another question is raised – how do you convince people to fly in a plane that does not require a pilot even when the technology is sound? The number of different mission types that UAVs will undertake is likely to grow in the coming years and there will be a number of human factors roadblocks that will need to be overcome including: Mission management – coordination with large entities such as the U.S. Air Force Air Operations Center (AOC), deconfliction of airspace where UAVs must share the skies with manned aircraft, and development of usable interfaces for the operators. Development of UAV operating doctrine – centralizing control while decentralizing the execution such that lower echelons are still involved in the decision-making process. Present culture – UAVs are still not widely accepted and therefore work must be done to minimize mishaps, maximize efficiency and safety to promote acceptance of UAVs. Manning problem – determining who will fly UAVs is the biggest problem thus far. Goldy agreed with the view that humans will always be needed in some capacity in the operation of UAVs. In order for humans to effectively and safely operate UAVs now, and in the future, the UAV community must ‘‘design humans for UAVs’’ by defining:

How operators will be selected? What standards will be imposed on operators? What manner of training must be provided to operators? What types of interfaces will exist?

32


Formal studies will be needed in order to derive answers for the questions listed above. However, the UAV community is already heading on the right track. The U.S. Army has already adopted flight regulations for UAVs with Army Regulation 95–23 (Department of the Army, 2004), which outlines such matters as UAV management (i.e., personnel who are authorized to fly Army UAVs, use of airspace), operational safety, training, and flight procedures and rules. ASTM International (formerly known as the American Society for Testing and Materials) also has a subcommittee on UAV operator standards (F38.03). The goal of this subcommittee is to use existing Federal Aviation Regulations (notably part 61.5) and augment them for possible use in awarding UAV pilot certificates. Such regulations and future studies will hopefully prevent mishaps due to the lack of operator qualifications standards. For example, Goldy recounted a story in which a Pioneer UAV trainee experienced problems controlling his training (half-scale Pioneer) UAV. The instructor noticed that the trainee had many scars on his left hand. As it turns out, when the trainee was a child, his fingers were severed in a farming accident and reattached. The fact that no one caught this earlier, that there were no medical pre-selection criteria, that money was wasted on training, and that this could have caused a more serious mishap later, warrants the development of regulations and standards for UAVs.

DISCUSSION The purpose of this chapter was to describe the current state and the future of UAV human factors from the experiences and opinions of two different UAV operators. The discussion ranged from the issues surrounding human factors and ergonomics in the GCSs in use today to the future of UAV operations. However, the most poignant topic for both operators was the selection and training of operators. Rainman made a special case for standardized training for career path UAV operators, instead of pulling them out of their current weapons systems and having to retrain them. Goldy also made a special case for the development of pre-selection criteria and medical standards for UAV operators with the notion that humans will remain in the UAV control loop now, and in the foreseeable future. One thing is certain – UAV operations in the future will become increasingly more complex as designers, mission planners, and operators test the limits of their UAV systems, finding new uses and mission types both in military airspace, and the NAS. Human factors personnel will be required to conduct studies to determine the viability of interfaces and controls as well


33

as define the criteria regarding who should and should not operate UAVs. Throughout this process, the experiences, observations, and opinions of the operators who use these systems and technologies will be invaluable in conveying potential usability issues firsthand.

ACKNOWLEDGMENTS This work and the CERI First Annual Human Factors of UAVs Workshop was supported by the Air Force Office of Scientific Research, the Air Force Research Laboratory, the National Aeronautics and Space Administration, the Federal Aviation Administration, Micro Analysis & Design, and the U.S. Positioning Group LLC. The CERI Second Annual Human Factors of UAVs Workshop was supported by the Air Force Office of Scientific Research, the Air Force Research Laboratory, the Federal Aviation Administration, and Micro Analysis & Design. The authors would like to acknowledge and thank Maj. James ‘‘Rainman’’ Hoffman and Jeff ‘‘Goldy’’ Goldfinger for their valuable insights and participation in the workshop.

REFERENCES Ciavarelli, A. (2003). Cockpit control and display design analysis. Retrieved October 1, 2005, from http://www.nps.navy.mil/AVSAFETY/gouge/COCKPIT%20CONTROL%20AN %20DISPLAY%20Hazard%20Checklist.pdf#search=‘anthony%20ciavarelli%20cockpit%20design%20principles’. Department of the Army. (2004). Unmanned aerial vehicle flight regulations, Army Regulation 95-23, Washington, DC. Goldfinger, J. (2004). Designing humans for UAVs: An operator’s perspective. Paper presented at the CERI First Annual Human Factors of UAVs Workshop, May 24–25, Chandler, AZ. Hoffman, J. (2004). The U.S. Air Force at the crossroads: Dealing with UAVs and human factors. Paper presented at the CERI First Annual Human Factors of UAVs Workshop, May 24–25, Chandler, AZ. Williams, K. W. (2006). Human factors implications of unmanned aircraft accidents: Flightcontrol problems. In: N. J. Cooke, H. Pringle, H. Pedersen & O. Connor (Eds), Human factors of remotely operated vehicles. Volume in Advances in Human Performance and Cognitive Engineering Research Series. London: Elsevier.


34

HUMAN FACTORS ISSUES

The purpose of this section is to broadly focus on human factors issues from remotely operated vehicle (ROV) development to remote operations. The first chapter by Tal Oron-Gilad presents a functional taxonomy to guide UAV technology development. By simultaneously considering a top-down and bottom-up approach, the chapter captures the present and future scope of UAV operations. In the next chapter, Todd Nelson explores relevant human factors issues in the concept of operations for unmanned combat air vehicles. The chapter addresses the individual operator and the airborne controller on a range of human factors issues from fatigue to situation awareness and decision making. Next, Michael Barnes describes the Army’s approach to UAV development. Modeling and simulation are cornerstones to this approach and are useful in revealing human factors issues early in the design process. Finally, Stephen Hottman examines the interaction between controllers and operators. He provides a starting point for the safe integration of UAVs as a system in the national airspace. In sum, these chapters clearly establish that human factors issues are as important to remote aircraft development and as they are to remote aircraft operations.

35


36

3. REMOTELY OPERATED VEHICLES (ROVs) FROM THE TOP-DOWN AND THE BOTTOMUP$ T. Oron-Gilad, J. Y. C. Chen and P. A. Hancock With the continuing evolution of technology, our capacity to sense environments at a distance has grown exponentially over the last two to three decades (Kurzweil, 2003). The natural outcome of this extension of sensory capabilities is the understandable wish to exert comparative action at a distance. But, imagine a world in which we could each effect action simultaneously at diverse locations! It would be like trying to have a phone conversation with every phone user in the world, all at the same time (after all, speech does effect action at a distance). Evidently, this would represent a chaotic cacophony in which, not only would you not be able to interact with everyone effectively, you would not be able to interact with anyone effectively. While extending our capacities for perception and action are

$

This work was supported in part by the Army Research Laboratory through the MicroAnalysis and Design CTA Grant DAAD19-01C0065, Michael Barnes, Technical Monitor, P.A. Hancock, Principal Investigator. This work was also facilitated by the Department of Defense Multidisciplinary University Research Initiative (MURI) program administered by the Army Research Office under Grant DAAD19-01-1-0621, Dr. Elmar Schmeisser, Technical Monitor, P.A. Hancock, Principal Investigator. The views expressed in this work are those of the authors and do not necessarily reflect official Army or Department of Defense policy. Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 37–47 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07003-7

37

38

T. ORON-GILAD ET AL.

evidently important, it is the selective and effective extension of such capacities we really desire. ROVs are one obvious, contemporary technical instantiation of this evolutionary vector in extending human perception-action capabilities. As evident from our initial observations above, the central problem of ROV development is not the feasibility and creation of the hardware per se, as difficult an engineering challenge as this may be. Rather, the crucial issue is the assimilation of the relevant sensory inputs, the processing of information pertinent to specified user goals, and the translation of the user’s subsequent decisions into effective action. Thus, the fundamental barrier to success in this realm is not a technological one but a use-centered one (see Flach & Dominguez, 1995). In this chapter, we therefore adopt this use-centered perspective and present steps toward an integrated, functional architecture for ROV operations and eventually a much-elaborated spectrum of advanced technologies. We do this by considering the issue from both a topdown and a bottom-up perspective.

THE TOP-DOWN APPROACH When we take a top-down approach to understanding issues surrounding ROV implementation, we can employ the metaphor either literally or as a form of abstraction hierarchy (Rasmussen, 1986). Literally, the military’s necessity for moment-to-moment information mandates a suite of contextspecific technological capabilities for sensor and effector systems. This suite includes but is not limited to systems in outer space (such as geo-synchronized orbiting platforms), high altitude atmospheric systems (such as Global Hawk) and other craft which operate less than hundreds of feet from earth down to almost ground level itself. In the abstract sense, a top-down perspective requires us to be very explicit about the goals, the intentions, the requirements, the aspirations, and the limitations of the technology at hand. It is central to any abstraction hierarchy that we make explicit what we really require these burgeoning systems to accomplish. The military’s necessity for moment-to-moment information drives the development of technologies that will increase information-gathering capacity. The development of manned flight is a good example for such development. Flying is a continuous locomotion through a medium that does not permit unaided human presence and thus conquering powered flight is viewed primarily as a technological achievement. Aircraft over the years have grown in their capacities as ever greater functionality

ROVs from the Top-Down and the Bottom-Up

39

was strapped to ever faster vehicles. The rationale for continuing pilot inclusion has to do with the flexibility that the human brings to the human– machine combination. The human aspiration for flight (see Hancock & Chignell, 1995; Magee, 1941) is often tainted by the pragmatic mandate which funds such emerging capacities. The intrinsic enjoyment of flight is now progressively ever more in conflict with the extrinsic reasons for military aircraft. It is into this milieu of dissonance that the advent of the military ROV is being thrust. In ROVs, the operators are included in the loop but the roles of the operators and their motivations are now very different. This brings us to the question of exactly what human capabilities are extended by ROVs. In reconnaissance, for example, the ROV acts predominately as an extension of the human eye. Whether that eye is a camera in near-earth orbit or a video on micro-light, the fundamental goal is the expansion of the range of the end user’s perception. ROVs provide this ‘eye in sky’ capacity but the act of sensing is essentially divorced from the act of perception. This dissociation is highly problematic since the act of perception is itself an active one. Divorcing the process of sensory assimilation from perceptual interpretation and action engagement promises to fracture a fundamental human ability (Gibson, 1966, 1979; Powers, 1973). A frozen field of view provides only impoverished information. Normal vision is not compounded of snap shot-like units but is essentially dynamic and is guided by previous actions taken. That is, perception is predicated on action just as much as action depends on perception. Little wonder then that ‘‘situation awareness’’ is seen as representing the crucial bottleneck in current systems (Endsley, 1995; Hancock & Diaz, 2001; Smith & Hancock, 1995). As ROVs increase in number, capacity, and form, we are rapidly overwhelming the ability of any one user, or even group of users, to extract useful and pertinent information. In a signal to noise sense, our noise level is becoming so high that no possible signal will be observable. In the dynamic conditions of combat, the problem is that signal can very rapidly turn to noise and vice versa. Although the primary problem of today is one of information extraction, in the very near-term we will wish to effect action based on this derived information. Solving the perception issue is fundamental to the perception-action at distance progression. Humans have always sought to extend their range of action commensurate with the range of perception. Remotely operated combat vehicles (ROCVs) represent an obvious contemporary example of this technological magnification. Central limitations of user information assimilation, decision-making and response selection are thus the current barrier to achieving

40


this goal. The top-down perspective emphasizes that the ROV is simply one of a suite of ‘‘vehicles’’ or technologies to achieve desired aims. In contrast, a bottom-up approach must take a much harder look at the limits and constraints in present-day and near-term operational systems. Humans have highly constrained sensory and effector bandwidths. We have evolved to tune these limited perception-action systems to the needs of survival and procreation. Now these peripheral capacities are magnified by technology but the comparable central capacities of perception, decisionmaking, and response-selection remain largely unaltered. Technologies, which seek to support such central capacities, are fundamentally different from those that support peripheral capacities. In facing the harder challenge of cognition, they remain at present much less effective and so the former capacities remain the bottleneck of concern. It is anticipated that the future fighting personnel will have to function under very compressed planning and operating cycles and at very high tempos (Scales, 2001). Under these stresses, the importance of decision-support expressed in the human–system interface will grow ever greater, making it even more crucial to frame information exchange to best facilitate human decision and action response (and see Hancock, Flach, Caird, & Vicente, 1995; Mouloua, Gilson, & Hancock, 2003). Recently, we have argued that the ROV control ratio (i.e., how many ROVs can a single operator control) is not the right question (Hancock, Mouloua, Gilson, Szalma, & Oron-Gilad, 2005). We now wish to expand upon this concern. We argue that under extreme stress, any single operator can and should gain effective situational awareness of only one relevant operational environment at a time. Under stressful conditions, individuals report a phenomenological ‘‘narrowing’’ of the range of cues they extract from the surrounding environment both spatially (Easterbrook, 1959) and temporally (Hancock & Weaver, 2005). The vast amounts of data that ROVs provide render these operators immediately vulnerable to the effects of information overload. Thus, when we see technology as the answer, the end point of that general strategy is an evident situation in which technology becomes the central problem. The tasks associated with use of ROVs include inherent spatial and temporal uncertainty in regard to targets, which result in extended periods of underload (i.e., a vigilance imperative), as well as the sudden overload associated with the onset of conflict and as well as the transitions between these two states (Huey & Wickens, 1993). Furthermore, these effects vary with the context of operation. The key to successful implementation of this technology for use in stressful environments is to ensure that the information presented to operators is structured in a manner


41

amenable to direct interpretation with a minimum of cognitive processing demands (Hancock & Szalma, 2003). We now turn to the bottom-up perspective.

THE BOTTOM-UP APPROACH The space–time taxonomy, initially introduced by Ellis for network categorization (Ellis, Gibbs, & Rein, 1991) captures the basic categorization of the physical and temporal relations between the machines and the humans. This categorization has already demonstrated the capacity to impact on automation implementation (see Yanco & Drury, 2002). This space–time taxonomy divides user–machine interaction into four categories based on whether the humans or the machines are using their computing systems at the same time or at different times and while in the same place or in different places. However, this framework has not considered the magnitude of difference in scale between user and machine (i.e., to what extent are the time scale or the physical dimension of operation different). We therefore suggest a broader space and time taxonomy where space is defined in two dimensions: physical size (nano to macro) and physical proximity, and time is also defined in two dimensions: time scale (momentary and extended) and synchronization. This enhancement of the space– time taxonomy is critical for identifying potential situation awareness limitations and important for the implementation of future ROV’s capacities such as adaptive automation. Action frequently provides real-time response feedback, letting the humans know the outcome of their efforts in order to match outcome to intention. User performance is vastly disrupted if this feedback loop is distorted either spatially or temporally (Smith, 1993; Smith & Smith, 1993). Each spatial and temporal dimension affects the operator’s ability to perceive information and can have an immediate influence on the effectivity of the feedback loop, particularly when action at distance is involved. Spatial context is critical for situation awareness and feedback as it defines the ability of the human operator to understand environmental elements related to the ROV and enhances task engagement. Teleoperation is often prone to poor spatial awareness of the remote environment (see for example, Darken & Peterson, 2002; Tittle, Roesler, & Woods, 2002). Also, the use of cameras to capture remote environments frequently creates a ‘‘keyhole’’ effect or ‘‘soda straw’’ view (Woods, Tittle, Feil, & Roesler, 2004; Murphy, 2004). Furthermore, if the spatial scale is very small (e.g., nano-scale)

42


or very large (e.g., thousands of kilometers) it becomes difficult for the operator to relate empathetically to the perceived entities in terms of personal experience (see Hancock & Chignell, 1995). Physical proximity determines how close to each other the human and the machine are; high proximity means that the human and the machine share the same local environment (i.e., the same ‘workplace’). It implies that the operator is likely to have an immediate empathy with local conditions. In high proximity situations the physical laws that apply to the system are similar to those that apply for the operator. Progressively lower values of proximity indicate that there is a physical distance (or separation) between the operator and the machine. Hence, a nano-scale may be similar in proximity distance-value to a macroscale situation because each is equivalently different from common human experience. This proximity is not a simple spatial referent per se. Lower proximity increases the need to provide more situational information of that remote environment. For instance, Thomas and Wickens (2000) found ‘‘cognitive tunneling’’ when viewing a remote environment using an immersive displays (such as the ones typically used for ground robots) compared to displays with exocentric frame of reference (similar to views from a UAV), which had a greater field of view. Furthermore, humans tend to underestimate distances more in photographic quality imagery than in the real world (Lampton, Singer, McDonald, & Bliss, 1995; Witmer & Kline, 1998; Thompson et al., 2004). In laboratory settings, time has been associated with reaction time, latency of cognitive responses (e.g., Card, Moran, & Newell, 1983), and motor movement (e.g., Fitts, 1954). Recent studies (e.g., Lane et al., 2002; MacKenzie & Ware, 1993; Malcolm & Lim, 2003) reinforce the problematic operator–machine coupling when time delays occur. Sheridan (2002) recommended that supervisory control and predictor displays be used to ameliorate the negative impact of time delays on teleoperation. Furthermore, for the implementation of adaptive automation, the understanding of human perception of time, particularly under stress, becomes a critical consideration (see Hancock & Scallen, 1996). What remains yet to be determined is how to frame information exchange to best facilitate human decision and action response within this remote perception-action system. Fig. 1 provides a descriptive three-dimensional model, which incorporates the operational perspective with the space–time taxonomy. The operational perspective can be either goal-driven or datadriven. There are commonalities between a goal-driven and a top-down abstraction and vice versa; there are commonalities between the data-driven approach and a bottom-up viewpoint. The data-driven perspective is a more


Fig. 1.

43

A Descriptive Three-Dimensional Model, which Incorporates the Operational Perspective with the Space-Time Taxonomy.

passive situation where the operator merely responds to events in the environment if they occur (see also Fong et al., 2004). From a data-driven perspective our expectation is that, the ‘level’ at which the individual (inter) actor engages with the system, represents the crucial arbiter on performance. Those in greatest proximity to the theater of operations are most liable to respond under conditions of stress and fatigue. They are therefore more likely to adopt a data-driven (bottom-up) approach, as immediate changes in threat or identification of unknown objects may have a direct effect on their safety and the safety of their unit of action. In such circumstances the immediate goal changes from the higher-level tactical operational goals to a more self-centered safety goal. More distal operators are liable to have access to facilities, which will mitigate such effects (although in times of great demand such as combat they will be omni-present in the theater of operations). The goal-driven (top-down) strategies recommendations will depend upon the hierarchy of operational contexts. Close to the scene of action, we have previously distilled that simple, graphical representation with unambiguous cues for response need to be employed. As the individual becomes more distal to the locus of action, the imperative to provide these simplifications decreases. Table 1 addresses issues of concern in regard to each dimension of the model. This table is only an exemplar and additional aspects such as communication and coordination will probably be added based on the context of operation. Adaptive automation helps to manage momentary changes and thereby efficiently schedule tasks and regulate demands. However, understanding

44

Table 1.


Issues of Concern in Regard to Each Dimension of the Model.

Space: High–Low Proximity Limited view (‘‘keyhole’’ effect) Restricted field of view: affects target detection, distance estimation, and identification of self–location Degraded depth perception: affects estimates of distance and size Camera viewpoint (Context): contextual change when switching among robots Degraded video image: affects distance, object identification, and size estimation Reduced Bandwidth: affects spatial orientation, object identification, and speed and motion perception Time: Delay and Magnitude Compensatory tracking performance degrades with a latency of about 300 ms When latency is over 1 s, operator switches to a ‘‘move and wait’’ control strategy Placement task performance degrades when standard deviation of latency is above 82 ms Over actuation is common when system delay is unpredictable Perspective: Bottom-up (data-driven)–top-down (goal-driven) Bottomup: Passive perception (interpretation of sensor data) Effectiveness and efficiency of sensor data retrieval Cognitive tunneling Topdown: Active perception (seeking sensor data) Effective ways to present information and provide decision support aids Span of control Intent or directedness toward future state or goal

the reciprocal relationship between adaptive automation and operator performance is limited by the sparse level of systematic research and a still under-developed theoretical framework (although see Hancock & Chignell, 1987; Parasuraman, Sheridan, & Wickens, 2000). Though we recommend and advocate adaptive automation at all levels of the system, interface appearance will change throughout the system, according to the operation, operator, and ROV characteristics. Although inter-operability remains an important issue, and the minimization of training time a real concern, our expectation at present is that multiple levels of operational sophistication will mandate hierarchic and selectable interface modes.

DISCUSSION AND CONCLUSIONS There is a very real danger that the system will not be able to accomplish the missions it was designed for, when we equate progress directly with increasing technical capacity (Perrow, 1984). In actuality, progress is less and less accurately indexed by technical capacities as it is by the ability to achieve the human goals for which the technology was designed. ROVs are a


45

cusp technology in this transition of understanding. Almost inevitably, we focus on flight envelopes, operational capacities, physical limitations, and the like. The overwhelming majority of resources are thrown at these problems in the desperate search for solutions. However, if we start from what it is the user needs then we need not seek optimal engineering solutions and indeed may circumvent some problematic technical issues altogether. The fundamental problem here is information not technology. Our view on system architecture issues follows this view. However, we believe that we are laboring against the perception of the problem as opposed to the true requirements. On the programmatic level, the issues that need immediate attention include information sharing, fostering situation awareness, multiple control problems, display design, and hardware barriers to achieve effective early prototypes. Several of those issues, especially that of human-in-theloop and semi-automated control, are the subject of progressing research (Sheridan, 2002). The issue of control is a vital one since it is evident that we still design as though the machine system is to react conceptually and cognitively as ‘one of us’. For example, we still seek explicit extensions to human sensory capabilities predominately through the eye while the integration of other sensory information still lags behind. Like most such situations (e.g., the evolution of manned flight), we will have to rely on the rebound from failures, some of which have already occurred. In the end, technologies that fail to achieve successful perception-action in distance will fade away. How and when this evolution will occur depends directly on the foresight of ROV designers.

REFERENCES Card, S. K., Moran, T. P., & Newell, A. (1983). The psychology of human–computer interaction. LEA: Hillsdale, NJ. Darken, R. P., & Peterson, B. (2002). Spatial orientation, wayfinding, and representation. In: K. Stanney (Ed.), Handbook of virtual environment technology (pp. 493–518). Mahwah, NJ: Erlbaum. Easterbrook, J. A. (1959). The effect of emotion on cue utilization and the organization of behavior. Psychological Review, 66, 183–201. Ellis, C. A., Gibbs, S. J., & Rein, G. L. (1991). Groupware: Some issues and experiences. Communications of the ACM, 34(1), 39–58. Endsley, M. R. (1995). Toward a theory of situation awareness. Human Factors, 37(1), 32–64. Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47(6), 381–391. Flach, J. M., & Dominguez, C. O. (1995). Use-centered design: Integrating the user, instrument, and goal. Ergonomics in Design, 3(3), 19–24.

46


Fong, T., Kaber, D., Lewis, M., Scholtz, J., Shultz, A., & Steinfeld, A. (2004). Common metrics for human–robot interaction. In: Proceedings of IEEE International Conference on Intelligent Robots and Systems, Sendai, Japan. Retrieved February 23, 2004. from http:// vrai-group.epfl.ch/papers/IROS04-TF.pdf Gibson, J. J. (1966). The senses considered as perceptual systems. Boston, MA: Hougton Mifflin. Gibson, J. J. (1979). The ecological approach to visual perception. Boston, MA: Houghton Mifflin. Hancock, P. A., & Chignell, M. H. (1987). Adaptive control in human-machine systems. In: P. A. Hancock (Ed.), Human factors psychology (pp. 305–345). North-Holland: Elsevier. Hancock, P. A., & Chignell, M. H. (1995). On human factors. In: J. Flach, P. A. Hancock, J. K. Caird & K. Vicente (Eds), Global perspectives on the ecology of human–machine systems. Erlbaum: Mahwah, NJ. Hancock, P. A., & Diaz, D. (2001). Ergonomics as a science of purpose. Theoretical Issues in Ergonomic Science, 3(2), 115–123. Hancock, P. A., Flach, J., Caird, J. K., & Vicente, K. (Eds) (1995). Local applications in the ecology of human–machine systems. Mahwah, NJ: Erlbaum. Hancock, P. A., Mouloua, M., Gilson, R. D., Szalma, J. L., & Oron-Gilad, T. (2005). Is the UAV control ratio the right question? Manuscript(submitted). Hancock, P. A., & Scallen, S. F. (1996). The future of function allocation. Ergonomics in Design, 4(4), 24–29. Hancock, P. A., & Szalma, J. L. (2003). Operator stress and display design. Ergonomics in Design, 11(2), 13–18. Hancock, P. A., & Weaver, J. L. (2005). Temporal distortions under extreme stress. Theoretical Issues in Ergonomic Science, 6(2), 193–211. Huey, M. B., & Wickens, C. D. (Eds) (1993). Workload transition: Implications for individual and team performance. Washington, DC: National Academy Press. Kurzweil, R. (2003). The law of accelerating returns. Retrieved July 7, 2005 from http:// www.KurzweilAI.net Lampton, D. R., Singer, M. J., McDonald, D. P., & Bliss, J. P. (1995). Distance estimation in virtual environments. In: Proceedings of the Human Factors and Ergonomics Society 39th annual meeting (pp. 1268–1272). Santa Monica, CA: Human Factors and Ergonomics Society. Lane, J. C., Carignan, C. R., Sullivan, B. R., Akin, D. L., Hunt, T., & Cohen, R. (2002). Effects of time delay on telerobotic control of neutral buoyancy vehicles. In: Proceedings of IEEE international conference on robotics and automation. Retrieved February 13, 2004, from http://www.ssl.umd.edu/publications/SSL02–003.pdf MacKenzie, S. & Ware, C. (1993). Lag as a determinant of human performance in interactive systems. In: proceedings of ACM conference on human factors in computing systemsINTERCHI’93 (pp. 488–493). New York, NY: ACM SIGCHI. Retrieved May 27, 2004, from www.yorku.ca/mack/p488–mackenzie.pdf Magee, J. G. (1941). High flight. Retrieved from www.wpafb.af.mil/museum/history/prewwii/ jgm.htm Malcolm, A. A., & Lim, J. S. G. (2003). Teleoperation control for a heavy-duty tracked vehicle. Retrieved June 21, 2004, from www.simtech.a–star.edu.sg/Research/TechnicalReports/ TR0369.pdf. Mouloua, M., Gilson, R., & Hancock, P. A. (2003). Designing controls for future unmanned aerial vehicles. Ergonomics in Design, 11(4), 6–11.


47

Murphy, R. R. (2004). Human–robot interaction in rescue robotics. IEEE Systems, Man and Cybernetics Part C: Applications and Reviews, special issue on Human-Robot Interaction, 34(2). Retrieved Apr. 16, 2004, from http://www.crasar.csee.usf.edu/research/Publications/ CRASAR–TR2004–2.pdf Parasuraman, R., Sheridan, B. T., & Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics: Part A, 30(3), 286–297. Perrow, C. (1984). Normal accidents: Living with high-risk technologies. NY: Basic Books. Powers, W. T. (1973). Behavior: The control of perception. Chicago: Aldine. Rasmussen, J. (1986). Information processing and human–machine interaction: An approach to cognitive engineering. New York: North-Holland. Scales, R. H. (2001). Future warfare anthology. US Army War College, ISBN 1-58487-026-5. Sheridan, T. B. (2002). Humans and automation: System design and research issues. Hoboken, NJ: Wiley. Smith, K., & Hancock, P. A. (1995). Situation awareness is adaptive, externally-directed consciousness. Human Factors, 37, 137–148. Smith, K. U., & Smith, T. J. (1993). Human–computer interaction and the automation of work. In: M. J. Smith. & G. Salvendy (Eds), Human-computer interaction: Applications and case Studies, Proceedings of the fifth International conference on human–computer interaction, 2 (pp. 837–842). Amsterdam, NA: Elsevier. Smith, T. J. (1993). Automation of work in dangerous environments. In: M. J. Smith & G. Salvendy (Eds), Human–computer interaction: Applications and case studies, Proceedings of the fifth international conference on human-computer interaction, (Vol. 1, pp. 273–277). Amsterdam, NA: Elsevier. Thomas, L. C., & Wickens, C. D. (2000). Effects of display frames of reference on spatial judgments and change detection. (Tech. Report: ARL-00-14/FED-LAB-00-4). UrbanaChampaign, IL: University of Illinois, Institute of Aviation, Aviation Research Laboratory. Retrieved April 4, 2005, from www.humanfactors.uiuc.edu/Reports& PapersPDFs/TechReport/00-14.pdf Thompson, W. B., Willemsen, P., Gooch, A. A., Creem-Regehr, S. H., Loomis, J. M., & Beall, A. C. (2004). Does the quality of the computer graphics matter when judging distances in visually immersive environments? Presence, 13, 560–571. Tittle, J. S., Roesler, A., & Woods, D. D. (2002). The remote perception problem. In: Proceedings of the Human Factors and Ergonomics Society 46th annual meeting, (pp. 260– 264). Santa Monica, CA: Human Factors and Ergonomics Society. Witmer, B. G., & Kline, P. B. (1998). Judging perceived and traversed distance in virtual environments. Presence, 7(2), 144–167. Woods, D. D., Tittle, J., Feil, M., & Roesler, A. (2004). Envisioning human–robot coordination in future operations. IEEE Transactions on Systems, Man & Cybernetics Part C: Special Issue on Human–Robot Interaction. Retrieved April 21, 2004 from http:// www.csel.eng.ohiostate.edu/woods/papers/woods_hri_IEEE.pdf Yanco, H. A., & Drury, J. L. (2002). A taxonomy for human-robot interaction. AAAI Fall Symposium on Human–Robot Interaction, AAAI Technical Report FS-02-03, pp. 111– 119, November.


48

4. SUPERVISORY CONTROL OF UNINHABITED COMBAT AIR VEHICLES FROM AN AIRBORNE BATTLE MANAGEMENT COMMAND AND CONTROL PLATFORM: HUMAN FACTORS ISSUES W. Todd Nelson and Robert S. Bolia Within the framework of current and near-term concepts of operations (CONOPS), squadrons of uninhabited combat air vehicles (UCAVs) will be controlled or monitored by operators located in ground-based control stations (Wilson, 2002), connected by radios or other communication links to command and control (C2) elements in air operation centers or battle management command and control (BMC2) platforms such as the Airborne Warning and Control System (AWACS) or the Joint Surveillance Target Attack Radar System (JSTARS). The rationale for this view seems to have been provided by analogy with current uninhabited air vehicles (UAVs), and fails to take into account that these UAVs have a primary surveillance mission, whereas UCAV is being designed – at least initially – for the task of suppression of enemy air defenses (SEAD; for a description of the SEAD mission, see Brungess, 1994). Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 49–58 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07004-9

49

50

W. TODD NELSON AND ROBERT S. BOLIA

Taking up a different analogy, some military analysts have suggested that the control of UCAVs tasked with the SEAD mission could be accomplished by air weapons officers on a BMC2 platform, since they currently provide tactical C2 for SEAD missions (Tiron, 2002; Wilson, 2002). Specifically, the weapons section on an AWACS or JSTARS is responsible for the direction and coordination of strike assets. Were the SEAD component of these strike assets augmented or replaced by UCAVs, the air weapons officer would be required to coordinate their participation through a liaison, the UCAV controller. Allowing the weapons officer to control the UCAVs directly may occasion several immediate benefits, including reduced deployment costs, reduced sensor-to-shooter time, and more seamless integration of manned and unmanned strike assets. The latter may be particularly important, given the temporal and spatial complexities of the SEAD mission. While there have been rudimentary treatments of the operational concerns associated with control of UCAVs by BMC2 platforms, little if anything has been written with respect to the relevant human factors questions. Specifically, issues involving operator and team fatigue, mental workload, attention, situation awareness, decision effectiveness, and the challenges associated with human interaction with automated systems may mediate the efficacy of this CONOPS in ways not previously considered. This chapter will introduce the application domain and develop the proposed CONOPS, which will lay the foundation for a critical review of the human factors challenges in the context of theory and research from complex supervisory work environments.

THE CONCEPT OF OPERATIONS UAVs have been used by military forces since at least the War of Attrition – fought between Egypt and Israel between 1967 and 1970 – when the Israeli Army modified radio-controlled model aircraft to fly over the Suez Canal and take aerial photographs behind Egyptian lines (Bolia, 2004). Although the Israelis ill advisedly abandoned the concept before the Yom Kippur War, it was taken up by several nations in the ensuing decades, and today UAVs are regarded as a routine component of surveillance operations, having played a significant role in both Afghanistan and Iraq. While the operating characteristics of current UAVs make them ideal for surveillance missions, the idea of armed UAVs able to engage targets directly has long captured the imagination of air power visionaries. Although

Supervisory Control of Uninhabited Combat Air Vehicles

51

efforts have been made to augment fielded UAV platforms with munitions that can be employed for time-sensitive targeting, such makeshift solutions are unsuitable for the classes of missions for which UCAVs would be most desirable. One of these is the SEAD mission, in which air assets attempt to suppress or destroy enemy air defenses, usually by compelling the antiaircraft of surface-to-air missile (SAM) site to activate their radars, and firing anti-radiation missiles at the activated sites (Li et al., 2002; Clark, 2000; Brungess, 1994). SEAD missions are among the most demanding that a pilot can undertake, making them an ideal candidate for execution by an unmanned platform. In addition to the potential savings in human life and vehicle cost, UCAVs may actually outperform manned aircraft in SEAD operations, given the increase in maneuverability occasioned by the luxury of not having to consider, in designing the air vehicle, the effects of high levels of acceleration on the human operator (Clark, 2000). Due to the fact that the current UAVs have been acquired over time by different services and manufactured by different contractors, control mechanisms vary from platform to platform. As a result, some UAVs are ‘‘flown’’ by a ‘‘pilot’’ sitting at a console using traditional throttle and stick controls, while others have a degree of autonomy. The CONOPS for UCAV is more suggestive of monitoring, or supervisory control, than it is of ‘‘flying’’ an aircraft. Within the context of this paper, UCAV is used to refer to a class of platforms, one of which is also called the UCAV (Boeing X-45). The primary displays for UCAV controllers, who will each be responsible for a number of autonomous vehicles, consists of a two-dimensional geospatial representation of the battlespace, augmented with overlays and imagery relevant to the mission at hand. To this extent, the task – at least its physical manifestation in the display – resembles air traffic control or the type of tactical C2 performed by the mission crew of an AWACS or JSTARS more than it does piloting an air vehicle. It is not much of a leap to suggest that UCAV control might profitably be performed from a console on such an airborne platform. Moreover, it might be argued that an air weapons officer, or weapons director (WD), controlling a strike package could also control the SEAD component of that package directly, without the requirement for an intermediary to supervise the UCAVs. Currently, teams of WDs exercise a level of tactical control over all air activities in their area of responsibility, to include air refueling, early warning, targeting and retargeting for both interdiction and close air support, air intercept, SEAD, and combat search and rescue (Gordon, Coovert, Riddle, & Miles, 2001; Hess, MacMillan, Serfaty, & Elliot, 1999). When UCAVs are introduced into the mix, there is little doubt that WDs

52


will need to integrate control of the assets they monitor with the assets monitored by the UCAV controller on the ground. One solution is to allow the WD to perform both functions. This suggestion is fraught with possibilities, such as shortening the ‘‘kill chain,’’ reducing personnel requirements, and enhanced operational integration, as well as reducing the communications required by the WD to build and maintain situation awareness (SA). The latter is particularly important. Despite the promise offered by ubiquitous data links and a common operating picture, the development, sharing, and maintenance of SA is typically accomplished by means of voice communications. WDs may increase SA by communicating with assets outside of their platform, but only at the expense of increased communications workload, which, because the WDs are already talking to the strike package lead, the senior director, and other off-board assets, is generally high (Bolia, Nelson, Vidulich, Simpson, & Brungart, 2005). Thus, depending on the situation, controlling a SEAD mission as an integral part of the WD’s strike package may allow for the preservation of SA at lower workload levels. On the other hand, care needs to be taken before recommending such a functional re-allocation. It is conceivable, for example, that the difference in level of engagement between the WD’s strike package control task – implemented largely by voice communication with the package lead – and the SEAD monitoring component will lead to disengagement in the latter task, engendering a loss of SA. In addition, there is the issue of over-reliance on the automation, which may result in a failure to detect a vehicular or system malfunction (Parasuraman & Byrne, 2003). The ability of an operator to avoid or manage such errors is an open question. If research should suggest that a single operator cannot feasibly control both the SEAD and strike components of a mission, it might make sense to include a SEAD controller aboard the BMC2 platform. This would presumably support enhanced SA, distributed workload, and, hopefully, an increase in mission effectiveness that might not be attainable via off-platform collaboration. Without additional research, however, this is mere speculation.

OPERATOR INTERACTION WITH AUTOMATED SYSTEMS Supervisory control of UCAVs by WDs, like all future UAV systems, will require highly automated systems (Parasuraman & Miller, 2006). As described


53

by Parasuraman and Byrne (2003), automation refers to the execution by a machine agent (e.g., computer) of a function previously carried out by a human operator. Automation in the aviation domain includes everything from complex, multi-step Flight Management Systems to less complicated, but equally important, automated alerting and warning systems such as the Ground Proximity Warning System and the Traffic Alert and Collision Avoidance System. Regardless of the specific type of automation in use, Parasuraman and Byrne (2003) argue that the rapid growth in the adoption of advanced automation has resulted in significant ‘‘distancing’’ of operators from the systems that they are required to monitor. For the supervisory control of UCAVs, it makes sense to ask whether increased reliance on highly automated supervisory control systems would serve to decouple a WD from the task at hand? More importantly, how may such a decoupling reveal itself? One possibility, which has been proposed by Sarter and Woods (1995), involves the challenge of maintaining mode awareness in highly automated mode-rich systems. Highly autonomous systems, such as those required of UCAV supervisory control, will likely subject operators to extraordinary attentional and cognitive demands as they try to understand the current state and future actions of the automated system. This relatively new form of complex supervisory control is particularly susceptible to inadequate mental models, and is often exacerbated by insufficient feedback about system and automation status (Sarter & Woods, 1995). Accordingly, systems that employ ‘‘clumsy automation’’ (Wiener, 1989) may be highly vulnerable to mode-errors, leading to increased likelihood of mishaps and accidents. This may be particularly troubling for a WD who is using a highly automated system to control the UCAV component of a mission while endeavoring to control manned assets using traditional voice communications or data links. In addition to the challenges associated with maintaining mode awareness, WDs might also be susceptible to automation bias, which may play a major role in the WDs’ complex and rapidly evolving tactical decisionmaking environments. Automation bias is related to the topic of trust in automation, and may reveal itself in the form of undertrust or overtrust. The latter is particularly important in highly reliable automated systems and may give rise to a condition known as automation complacency. Laboratory findings (e.g., Parasuraman, Molloy, & Singh, 1993) have shown that observers engaged in concurrent tasks are quite poor at monitoring highly reliable automated systems for occasional malfunctions. Of course, the danger in the case of supervisory control of multiple UCAVs is that automation failures or malfunctions that go unnoticed – possibly because the

54


WD is busily engaged in the task of controlling his or her fighter assets – may have grave consequences. Undertrust in automation occurs when an automated alerting system is either ignored or deactivated by operators (Parasuraman & Byrne, 2003). Such a situation, which can be caused by the issuance of excessive false alarms, is clearly non-optimal; however, it is often motivated by decision criteria that seek to minimize missed signals. This type of automation failure is of particular concern for WDs, since their application domain is often inundated by a cacophony of concurrent, multi-modal alarms and alerts. Accordingly, the design of automated alerts and alarms for this operational environment will need to be carefully considered so as to maximize operator trust.

FUNCTION ALLOCATION As described by Sheridan (1998), function allocation refers to the assignment of required functions (tasks) to resources, instruments, or agents; or, more simply, the assignment of people or machines to tasks. An early treatment of this topic was offered by Fitts (1951) and resulted in the so-called Fitts list, the now-famous and often controversial inventory comprising the relative strengths and weaknesses of both humans and machines, from which one may decide how to assign functions in human–machine systems. In the case of UCAVs, issues involving real-time re-routing, flight and mission management, weapons selection and delivery, threat avoidance, and time-sensitive targets will all involve design decisions regarding automation and allocation of functions. It is of interest to note that a 1996 Scientific Advisory Board (SAB; Worch, 1996) concluded that, thus far, the application of human factors principles to issues involving automation, allocation of functions, and human–machine interface design for UAVs has been deficient. Consistent with the view that allocation of functions between human operators and machines needs to be dynamic rather than fixed, numerous researchers (see Scerbo, 1996, 2001 for reviews) have embraced the notion of adaptive automation – a scheme by which the authority to initiate changes in automation and task allocation is shared between human operators and the system. While the empirical research corroborating this perspective has grown in the past several years, several foundational questions remain: What are the methods by which systems adapt? What triggers adaptive task re-allocation? How should they be adapted? Such notions are far from trivial, and much work will be required before the function allocation issue can


55

be considered ‘‘resolved’’ (see Sheridan, 1998) for domains as complex and demanding as air battle management and the supervisory control of UCAVs.

MENTAL WORKLOAD AND SITUATION AWARENESS In considering which factors may contribute to the overall effectiveness of an adaptive automation scheme for the supervisory control of UCAVs, one cannot overlook the concepts of mental workload and SA. Vidulich (2003) has suggested that despite the controversial nature of these two concepts, their combined influence on interface design and system advances will continue to be substantial. Concrete agreed-upon definitions of workload and SA are elusive, and have been the focus of much debate. It is useful, however, to follow the advice of Vidulich (2003) and put forward the following working definitions: workload – a general term used to describe the cost of accomplishing task requirements for the human element of a man–machine system (Hart & Wickens, 1990, p. 258); situation awareness – the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of the status in the near future (Endsley, 1995, p. 36). Within the context of the proposed CONOPS, an interesting relationship between workload and SA is implied. Namely, by shifting the task of UCAV supervisory control from ground-based operators to airborne WDs, one might expect improved individual SA due to the reduction, if not elimination, of verbally communicating the current state of the UCAV SEAD mission. Conversely, the additional supervisory responsibilities required of the WDs may result in elevated and unsafe workload levels. This circumstance may be particularly unfavorable in the case of severe task-saturation due to unforeseen situations and emergencies, or severe limitations in the WD’s attentional resources. This may merit further consideration of adaptive automation, but as noted previously, adaptively automated systems may introduce their own set of performance, adoption, and safety issues. Clearly, an understanding of the relationship between supervisory control, mental workload, and SA is warranted, and standardized measurement procedures for characterizing these effects should be pursued. While the previous section may be viewed as useful for generating testable hypotheses involving the proposed task re-allocation and its effects on

56


mental workload and SA, it is important to note that it fails to address the notion of workload and SA as team attributes. Vidulich, Bolia, and Nelson (2004) have recently explored the concept of SA within the context of air battle management, and concluded that current methodological approaches used to assess individual SA might also be extended to assess team SA. Other researchers (Salas, Prince, Baker, & Shrestha, 1995; Stout, CannonBowers, Salas, & Milanovich, 1999) have suggested that the measurement of team SA should consider team coordination and information-sharing capabilities, and the ability to create and maintain shared mental models, especially in high-workload environments.

OPERATOR SELECTION One issue that remains to be raised is operator selection. Specifically, there are those who espouse the notion that UCAV mission managers should be trained military aviators, despite the fact that ‘‘control’’ of the vehicles will be purely supervisory and hence will not require the skills typically associated with piloting an air vehicle. Advocates of this approach suggest that while flying skills per se may not be necessary, fighter pilots will have a better sense of the nuances of tactical air combat, including, according to Air Force Chief of Staff General John Jumper, an appreciation of the momentous responsibility associated with the release of kinetic weapons (Erwin, 2004). In addition, current Federal Aviation Administration regulations require that anyone operating aircraft in controlled airspace have at least a commercial pilot rating (Erwin, 2004). This question requires consideration for two reasons. First, the implementation of a pilot-only policy would render largely irrelevant the subject of the present chapter. Second, the failure to adopt such a policy would still require overlap between the skill set required for WDs and those required for UCAV operators. Although some researchers have begun to examine issues associated with the selection of UCAV mission managers (Tobin, 1999; Dolgin, Kay, Wasel, Langelier, & Hoffmann, 2001), a definitive policy has yet to be embraced.

CONCLUSION This chapter was written to reflect on the host of human factors challenges associated with assigning tactical control of UCAVs to weapons officers


57

aboard an airborne BMC2 platform. It has sought neither to elevate nor demean the suggested CONOPS, but rather to enumerate a list of relevant research questions that might profitably be addressed via experimentation, including issues of human interaction with automated systems, dynamic function allocation, the relationship between workload and SA, and the type of operators required to work effectively in this complex domain. It is only after such issues have been thoroughly considered that the tasking of WDs to control UCAVs can be fairly espoused.

REFERENCES Bolia, R. S. (2004). La batalla olvidada: Israel y la Guerra de Desgaste. Military Review Hispano-American, 84(2), 60–64. Bolia, R. S., Nelson, W. T., Vidulich, M. A., Simpson, B. D., & Brungart, D. S. (2005). Communications research for command & control: Human-machine interface technologies supporting effective air battle management. Proceedings of the 11th international command and control research and technology symposium. Washington: Command and Control Research Program. Brungess, J. R. (1994). Setting the context: Suppression of enemy air defenses and joint war fighting in an uncertain world. Maxwell Air Force Base, AL: Air University Press. Clark, R. M. (2000). Uninhabited combat aerial vehicles: Airpower by the people, for the people, but not with the people. CADRE Paper No. 8, Maxwell Air Force Base: Air University Press. Dolgin, D., Kay, G., Wasel, B., Langelier, M., & Hoffmann, C. (2001). Identification of the cognitive, psychomotor, and psychosocial skill demands of uninhabited combat aerial vehicle (UCAV) operators. SAFE Journal, 30, 219–225. Endsley, M. (1995). Toward a theory of situation awareness in dynamic systems. Human Factors, 37, 32–64. Erwin, S. I. (2004). Should unmanned combat aircraft be piloted only by fighter pilots? National Defense, 89(November), 30. Fitts, P. M. (Ed.) (1951). Human engineering for an effective air-navigation and traffic-control system. Columbus, OH: Ohio State University Research Foundation. Gordon, T. R., Coovert, M. D., Riddle, D. L., & Miles, D. E. (2001). Classifying C2 decision making jobs using cognitive task analysis and verbal protocol analysis. Proceedings of the 6th international command and control research technology symposium. Washington: Command and Control Research Program. Hart, S. G., & Wickens, C. A. (1990). Workload assessment and prediction. In: H. R. Booher (Ed.), Manprint: An integrated approach to systems integration (pp. 257–296). New York: Van Nostrand. Hess, S. M., MacMillan, J., Serfaty, D., & Elliot, L. (1999). From cognitive task analysis to simulation: Developing a synthetic team task for AWACS weapons directors. Proceedings of the 1999 command and control research and technology symposium. Washington: Command and Control Research Program.

58


Li, S. M., Boskovic, J. D., Seereeram, S., Prasanth, R., Amin, J., Mehra, R. K., Beard, R. W., & McLain, T. W. (2002). Autonomous hierarchical control of multiple unmanned combat air vehicles (UCAVs). Proceedings of the American control conference. Anchorage, Alaska. Parasuraman, R., & Byrne, E. A. (2003). Automation and human performance in aviation. In: M. A. Vidulich & P. S. Tsang (Eds), Principles and practices of aviation psychology (pp. 311–356). Mahwah, NJ: Lawrence Erlbaum Associates. Parasuraman, R., & Miller, C. (2006). Delegation interfaces for human supervision of multiple unmanned vehicles: Theory, experiments, and practical applications. This volume. Parasuraman, R., Molloy, R., & Singh, I. L. (1993). Performance consequences of automationinduced complacency. International Journal of Aviation Psychology, 3, 1–23. Salas, E., Prince, C., Baker, D. P., & Shrestha, L. (1995). Situation awareness in team performance: Implications for measurement and training. Human Factors, 37, 123–136. Sarter, N., & Woods, D. D. (1995). How in the world did we ever get into that mode? Mode error and awareness in supervisory control. Human Factors, 37, 5–19. Scerbo, M. (2001). Adaptive automation. In: W. Karwowski (Ed.), International encyclopedia of ergonomics and human factors (pp. 1007–1009). London: Taylor & Francis, Inc. Scerbo, M. W. (1996). Theoretical perspectives on adaptive automation. In: R. Parasuraman & M. Mouloua (Eds), Automation and human performance: Theory and application (pp. 37– 63). Mahwah, NJ: Lawrence Erlbaum Associates. Sheridan, T. B. (1998). Allocating functions rationally between human and machines. Ergonomics in Design, 6(3), 20–25. Stout, R. J., Cannon-Bowers, J. A., Salas, E., & Milanovich, D. M. (1999). Planning, shared mental models, and coordinated performance: An empirical link is established. Human Factors, 41, 61–71. Tiron, R. (2002). Unmanned bomber prepares for crucial tests. National Defense, 86(May), 20–21. Tobin, K. E. (1999). Piloting the USAF’s UAV fleet: Pilots, non-rated officers, enlisted, or contractors? Maxwell Air Force Base, AL: School of Advanced Air Power Studies. Vidulich, M. A. (2003). Mental workload and situation awareness: Essential concepts for aviation psychology practice. In: M. A. Vidulich & P. S. Tsang (Eds), Principles and practices of aviation psychology (pp. 115–146). Mahwah, NJ: Lawrence Erlbaum Associates. Vidulich, M. A., Bolia, R. S., & Nelson, W. T. (2004). Technology, organization, and collaborative situation awareness in air battle management: Historical and theoretical perspectives. In: S. Banbury & S. Tremblay (Eds), A cognitive approach to situation awareness: Theory, measures, and application (pp. 233–253). Aldershot, UK: Ashgate Publishing Ltd. Wiener, E. L. (1989). Human factors of advanced technology (‘‘glass cockpit’’) transport aircraft. NASA Contractor Tech. Report 177528. Moffett Field, CA: NASA Ames Research Center. Wilson, J. R. (2002). UAVs and the human factor. Aerospace America, 40(July), 54–57. Worch, P. R. (1996). United States Air Force Scientific Advisory Board Study on UAV: Technologies and Combat Operations. Washington, DC: General Printing Office (Report No. SAF/PA 96–1204).

5. MODELING AND OPERATOR SIMULATIONS FOR EARLY DEVELOPMENT OF ARMY UNMANNED VEHICLES: METHODS AND RESULTS Michael J. Barnes, Bruce P. Hunn and Regina A. Pomranky In order to impact system design early in the development process, human factors professionals must be able to anticipate how the equipment will be operated and maintained even before prototypes exist or when testing is still in its early stages (Barnes & Beevis, 2003). Perhaps the most challenging task that human factors professionals encounter is predicting crew performance for new systems within its concept of operations during these early stages. Data regarding the operators’ interface requirements, performance levels, crew size, skills, and training requisites are essential to the design process. Fortunately, a variety of modeling and simulation tools have been developed that make meeting this challenge more tractable. The purpose of this chapter is to elucidate a generic design philosophy and discuss specific examples of methodologies and tools used to support actual unmanned aerial vehicles (UAVs) development programs.


59

60

MICHAEL J. BARNES ET AL.

EARLY DESIGN METHODS The most important advance in system design is the development of modeling and simulation methods to predict complex performance before prototypes are developed. New systems are developed in a spiraling approach; as more is learned about the system, design changes are proposed and evaluated. This approach allows the engineering team to ‘‘spin out’’ early versions of the system for preliminary evaluation, permitting changes to be made to the system design without incurring unacceptable cost. Because of the complexity of human performance, current modeling techniques provide only a first approximation. However, it has been demonstrated that even simple, inexpensive modeling approaches are useful in uncovering workload and performance problems related to developing systems (Barnes & Beevis, 2003). More important, these models can serve as the basis for operator simulation experiments that verify and also calibrate the original models. Furthermore, early field tests and system of systems demonstrations that can validate these results under actual conditions are becoming an increasingly significant part of the early design process. Fig. 1 illustrates this interdependence indicating a spiraling process throughout the design starting with simple predictive methods and progressing to more expensive validation methods. These iterations should continue until most of the soldier’s variance is accounted for, and before any formal soldier testing is conducted. Fig. 1 presents the ideal combination of techniques; not all systems can be evaluated this thoroughly but more cost-effective modeling and simulation tools combined with realistic field exercises should make

Performance Models

Iterate to Next Spiral

Design Changes

Soldier Simulations

Field Exercise

Fig. 1.

Early Design Approach.

Modeling and Operator Simulations

61

this approach more the norm as future unmanned systems are developed (Barnes & Beevis, 2003). In the remainder of this chapter, several case studies are presented to illustrate how the techniques in Fig. 1 have been applied in UAV programs.

Case Study 1 – Hunter: Soldier-in-the-Loop Simulations The initial case study illustrates how we used soldier in-the-loop simulation experiments based on preliminary workload modeling to investigate design options for future versions of the Hunter UAV, which is a moderate-sized system, designed to fly 200 km for approximately a 9-hour mission. The Hunter requires a fairly long runway and an external pilot to control the UAV during take-off and landing. Two operators are in each shelter and the shelters must coordinate with the mission commander and the intelligence officer (S-2) both to develop as well as to change their mission plans as events unfold. To support the design effort for the system, task-level workload data were collected during an early user test using the U.S. Army Research Laboratory’s (ARL) Improved Performance Research Integration Tool (IMPRINT). The results identified high-workload mission segments and suggested possible problems with fatigue because of the prolonged mission duration. Based on these early indications of potential performance issues, the Army did a series of 72-hour live simulations at Redstone Arsenal, Alabama. The experiments were conducted in the Simulation Integration Laboratory (SIL) with trained operators ‘‘flying’’ in actual ground stations using software that simulated the Hunter’s flight characteristics. The initial experiment involved eight operators flying 12-hour shifts between preplanned waypoints using video imagery to detect targets during both day and night missions over the 72-hour period. Results indicated that 9–10hour mission (time operator was actually in shelter) length was acceptable but found indications of fatigue problems during night missions. The experiment also resulted in a number of interface design changes (including a better mission planning system). A second experiment with 16 operators was designed to investigate the possibility of a tactical UAV system flying two 3hour missions during their 12-hour shift using a suitcase interface instead of the more extensive Hunter interface. Because workload data suggested that communication was a high workload activity, the mission payload and flight functions were performed by one operator with communication being performed by a second operator in order to test the feasibility of a one person UAV crew (with auxiliary support). The shorter missions were more

62


workload intensive than the previous exercise because the S-2 (intelligence staff officer) redirected the operator’s mission dynamically during the more realistic vignettes. Because the crew rotated every 90 min, there were no fatigue-related problems in the second experiment, but there were unsafe crew decisions resulting in crashes of the simulated UAVs for seven of the missions. Human factors analysis of the crashes indicated: more training was needed for the new interfaces, the 2-in. display size resulted in poor situation awareness, and single crew members had attentional difficulties when aviation and payload demands occurred at the same time (Barnes & Matz, 1998). The results of this effort aided the program office plan for a tactical UAV that was developed years later. The simulations were cost effective, allowed new design options to be evaluated in a safe environment, and resulted in better experimental control than field testing but still captured the 72-hour realism of an actual mission (Barnes & Beevis, 2003).

Case Study 2: Evaluating the Cognitive Requirements of Generic UAV Crews Using Software Tools Traditionally, the acquisition community conducts only cursory crew analyses before an actual system is proposed. In this effort, the human factors research team at Fort Huachuca investigated crew issues that would be generic to a possible family of future UAVs. The team’s objectives were related to understanding the operator’s cognitive skills and workload requirements for three generic problems: flight certification requirements for UAV operators, additional skills for shelter operations, and automation of operator tasks (Barnes, Knapp, Tillman, Walters, & Velicki, 2000). The Job Assessment Software System (JASS) was used to understand crew skill and cognitive requirements necessary to conduct UAV flight operations. JASS is a computer-administered inventory based on operator ratings of 50 cognitive skills (subsumed into seven major categories; see Fig. 2) for specific military functions and has been validated in other military and civilian contexts (Knapp & Tillman, 1998). JASS ratings from 30 UAV operators with a 96U military occupation specialty (MOS) as well as 16 fixed wing and rotor wing aviators were used to compare cognitive profiles related to flight operations in order to gain insight into cognitive commonalities and differences among these groups. The 96U operators were certified as both mission payload and air vehicle operators for the Hunter UAV upon successful completion of their six-month course given at Ft. Huachuca. We decomposed the data further by experience level because

63

Modeling and Operator Simulations 50

Percentage of Important Skills

45 40 35

96U (EP)-Lo

30

96U (EP)-High

25

Aviator-Rotor

20

Aviator-Fixed

15 10 5 0

Commo

Conceptual Reasoning

Speed loaded

Vision

Audition Psychomotor

JASS Categories Related to Air Emergencies

Fig. 2. Comparison of Percentage of Important Skills Used During Emergency Situations for EP having High (More than a Year) and Low UAV Experience Levels (Less than a Year) Contrasted with Rotor and Fixed Wing Flight Certified Aviators.

the accident data suggested that most of the accidents occurred with inexperienced crews during take off and landing. Fig. 2 shows the results of one of the analyses indicating that flight crew had a somewhat different profile than the UAV external pilot (EP) whose job it is to fly the UAV on take-off and landing using a radio-controlled hand-held device. Further analysis suggested that the differences could be explained by the much greater complexity of the aviator’s interface requirements. However, the most important differences were found in comparing the skill profiles between experienced and inexperienced EPs. The experienced operators seemed to depend more on visualization skills related to the JASS conceptualization category, whereas the inexperienced operators depended more on perceptual motor skills related to both perceptual and psychomotor categories in Fig. 2. To better understand these results, a panel containing five experienced UAV operators (E-6 and E-7s) as well as experienced human factors professionals from all three services (four PhD and two MA level) convened in order to help us interpret the significance of the findings. Based on their experience, the UAV operators on the panel suggested that learning how to land a UAV was initially a focused perceptual motor task. However, as

64


operators gained experience, they performed the radio control functions automatically and visualized (i.e., one of the conceptual skills in the JASS inventory) the path of the UAV. This allowed the experienced operator to ‘‘stay ahead’’ of the descent and anticipate rather than respond to landing problems. The hypothesis that EPs used perceptual-motor skills for initial learning and visualization skills for expert performance were reinforced by performance data collected from 29 candidates for the EP school at Ft. Huachuca. We conducted a study to correlate tracking performance to training success. Specifically, the results indicated that five of the six operators who failed to finish the EP course ranked at the very bottom of the 29 tracking scores (Barnes et al., 2000) implying that perceptual-motor skills were essential for training success. In general, because of the differences in manned and unmanned aircraft, the panel did not suggest flight certification requirements for military applications, but they did suggest having a flight certified officer as part of the UAV platoon especially for airspace coordination. Also, based on results from the Israeli Air Force showing the efficacy of computer games for flight training (Gopher, Weil, & Bareket, 1994) as well as our own analysis, we concluded that additional training with simulators (or gaming environments) that emphasized visualization and cognitive skills was a promising and cost-efficient adjunct to the current training regime (i.e., to develop in the novice some of the cognitive skills of the expert). In the same series of studies, we examined the possibility of adding additional MOS skills to the UAV crew complement. The JASS skill profiles of 96D, 96B, and 96U were correlated using the Kendall rank order test. The 96D imagery analyst is a specialist in identifying militarily-relevant objects in great detail, whereas the UAV operator is more of a generalist who classifies UAV images at a low level of resolution making tactical reports rather than detailed imagery distinctions. In contrast, the 96B is more of intelligence generalists whose initial training overlaps many of the topics the 96U must master. The results indicated that the intelligence analyst (96B) and the UAV operator’s skill sets were redundant, whereas the 96D (imagery analyst) had unique skill requirements that were potentially valuable additions to the 96U skill sets. The researchers suggested adding 96D skill sets to the UAV unit as either adjunct operators or through additional 96U training. A general finding was the importance of an expert panel consisting of mixed operational and technical members in interpreting JASS data. The data must be interpreted in context and very few human engineers understand all the variables that constitute future UAV mission environments.


65

Conversely, the results from 70 military personnel collected during the course of the study added additional constraints to the expert panel process by focusing discussion on empirical data. The JASS instrument is still under development for military use and one of its drawbacks is a lack of statistical analyses being used to evaluate JASS results. However, as JASS is developed further, more rigorous analyses based on non-parametric tests (such as reported above) could be added easily to the current software package. As now configured, JASS is easy to administer on a laptop, takes less than an hour per participant, and is inexpensive. The results have proved extremely useful as a first step in matching personnel attributes and important cognitive abilities to developing systems (Knapp & Tillman, 1998).

Case Study 3: Shadow-200 Crew Size Performance Evaluations Using Network Models Crew size is a crucial determinant of system performance and life cycle cost. Analysis of appropriate crew size begins in system preplanning and typically continues through formal testing. Throughout these phases, scant information is available to predict crew-sizing effects on the broad range of mission operations. In a joint project between ARL and Micro Analysis & Design (MA&D), we modeled crew performance for the Shadow-200 tactical UAV using Micro Saintr to predict fatigue and workload effects on crew performance as a function crew size for the proposed 72-hour operational tempo. Specifically, we examined crew requirements for the two shelters (launch and recovery station (LRS) and tactical operations center (TOC)) controlling the Shadow-200 (Walters, Huber, French, & Barnes, 2002). MA&D developed a Micro Saintr crew network model simulating both workload and performance data for soldier taskings for the LRS and TOC. The model simulated a mission commander (MC) supervising two operators in each shelter during multiple mission profiles over the 72-hour operational tempo. These profiles included missions with one ‘‘jump’’ (movement of assets to a different location) per 24-hour period and also missions where two Shadows were in the air simultaneously. The Micro Saintr task network model performance results were modulated by vigilance and fatigue functions to evaluate performance effects as a function of the time operator was in the shelter. The fatigue functions were based on equations developed by the Air Force Research Lab (AFRL) to predict the effects of both sleep deprivation and circadian rhythm (Walters et al., 2002). Fig. 3 shows the modulation effects

66

MICHAEL J. BARNES ET AL. Fatigue Reaction Time Model

Estimate Reaction Time

145.00 135.00 125.00

COS

115.00

Linear

105.00

Average Mod1

95.00 85.00 75.00 6:00

12:00 18:00

0:01

6:00 Time

12:00 18:00

0:01

Fig. 3. The Manikin Data Used to make the Algorithm (Model 1), the Cosine Function Data (COS), the Linear Function Data (Linear), and the Predicted Response Time Capability (Average) Expressed as Percent Baseline Response Time (Walters et al., 2002).

of Circadian rhythm (cosine function) and fatigue (linear effects) on an attention-switching task referred to as the Manikin task. An algorithm based on these two functions was used to modulate the performance data in the Micro Saintr model. In terms of degrading operator performance, the vigilance function had the largest effect followed by time of day, whereas sleep deprivation had the smallest effect. The relatively slight effect of sleep deprivation was due to the legal requirement that operators have a 12-hour duty day. The vigilance variable was sensitive to 2-, 3-, 4-, or 6-hour in-shelter rotation schedules, which combined with the ‘‘surge’’ flight schedules and crew rest requirements, determined the crew size availability for each shelter during the 72-hour operation. The model was used to evaluate the various crew size options based on statistical analyses of the performance data using the Monte Carlo subroutines in the Micro Saintr software package. Fig. 4 is an example of one of the analyses indicating TOC crew size effects on average target detection performance during the 72-hour operational tempo simulation. A crew size of eight was used as a baseline. These results were compared to a crew complement of six showing a slight 4% non-significant decrease from baseline. However, further decreasing the crew size to four had a pronounced effect (20% target detection drop, p o0.05).

67


Mean Targets Detected

120.0 100.0 80.0 60.0 40.0 20.0 0.0

Fig. 4.

8

6 Total # of UAV Operators in the TOC

4

Predicted Target Detection as a Function of Crew Size for 4, 6, and 8 UAV Operators in the TOC Shelter during a 72-Hour Operational Tempo.

Similar analyses were performed for the LRS. The modeling team used workload data to assess the possibility of time-sharing a single MC for the two shelters. The results indicated that the MC would be in overload nearly 50% of the time when on duty. Subsequently, interviews with experts verified that this would result in unacceptable safety problems. The final conclusion was that the data supported a crew complement of 12 for the two shelters (TOC: two MCs and six shelter operators; LRS: two MCs and two shelter operators). These numbers did not include supervisory, administrative, or maintainer personnel, which were not in the scope of the analyses. However, the analyses allowed the system manager to argue for a larger crew size during formal testing. The actual crew size for the Shadow is dependent on many factors including the results of empirical testing during the 72-hour operational tempo user tests. The advantage to the modeling effort was that it provided an initial crew size estimate based on both operator interviews and quantitative analysis that could be used as a baseline crew for formal testing. This approach was both cost effective and at the same time more rigorous than the back of envelope estimates used for previous UAV systems (Walters et al., 2002).

Case Study 4: Field Test The rapid pace of development of UAVs often necessitates evaluating new designs in a field test environment before design decisions are finalized. In

68


contrast to traditional long developmental programs, field tests are being introduced early in the acquisition process to demonstrate the military utility of UAV-related systems. The following example involves assessment of a program to field dissimilar UAVs controlled by multiple ground stations using network centric communications and visualization modules to obtain a common operational picture of the battlefield. ARL personnel participated in field exercises testing the human factors elements of collaborating but dissimilar UAVs. In a series of operations conducted through the U.S. Joint Forces Command, coordinated operations of three to four different UAVs were managed through a distributed network in various geographic locations. From a central command station, these four UAVs collected electro-optical (EO) and infrared (IR) imagery of a variety of targets, and communicated that information via a distributed network to U.S. Army, U.S. Navy, U.S. Marines, and Royal Navy field exercise participants. Information was exchanged in a large-scale exercise involving thousands of troops, vehicles, aircraft, and ships (Hunn, 2004). Human factors concerns included assessments of communication effectiveness, crew coordination, visual display compatibility, system design, ease-ofuse, and information transmission effectiveness. In the field exercises, which took place in Arizona and North Carolina, two to three ground control stations with two to three personnel each were communicating to a central UAV control station manned by three to five personnel, which in turn was communicating to a Combat Air Operations Center (CAOC). Ensuring that the teams could dynamically communicate target and mission objectives through many levels of command was a critical human factors task. This project also provided a chance to examine how well the individual crews communicated with each other and their own chain of command as well the CAOC. In addition, there were numerous new visual and information control systems that were being used in the ground control stations for each UAV as well as the central UAV control station. The ease-of-use and effectiveness of these systems were crucial human factors concerns. During this work, the team demonstrated a visual display style that allowed a crew member to perceive multiple UAVs in their proximate airspace, while also showing the images they were collecting real time, as well as historical database imagery, artificial land boundary lines, and target markers, all displayed on the same screen. These exercises also included assessments of digital chat lines for information distribution, mosaic presentation of individual photo images, and comparisons of verbal communication versus typed, written information. The complex coordination of all

69


these information was conducted in a scenario using aircraft, ships, troops, and live munitions. These demonstrations showed that situation awareness, performance, and effectiveness have been greatly enhanced by the use of sophisticated visual display software that merges historical database imagery with live UAV video imagery, as well as programmable overlay symbols, all managed by a Windowss-based interface technology. Results from this testing showed chat line input was slower than normal voice communication, but that it was more precise, particularly if the chat information was in the form of target coordinates. In addition, subjective ratings of workload and situation awareness were much higher for several new systems than previous approaches (Bowman, Hunn, & Warner, 2005).

SUMMARY The purpose of this chapter was to describe the application of modeling and simulation methodologies and supporting technologies that enhance the ability of human factors professionals to use human-centered design techniques early in the UAV acquisition process. Four examples of early human centered UAV design methods were discussed as case studies: 72-hour live simulations with certified Hunter UAV crews, software modeling for generic systems (JASS), Shadow-200 crew sizing using Micro Saintr, and field exercises with dissimilar but collaborating UAVs. Table 1 contrasts the various methods discussed. Each of the method has a different functional Table 1.

Comparison of Techniques Used in Early Design Phases of UAVs.

Method

Cost

Real World Validity

Experimental Control

Statistical Analyses

JASS IMPRINT/

Low Moderate

High Depends on input

NA Good

Limited Moderate

Micro Saintr Expert panel Soldier-in-theloop simulation Field experimentsmultiple systems

Moderate Moderate to high High to very high

High Moderate to high Highest

NA Good

NA Extensive

Moderate to poor

Difficult because of variable conditions

70


utility. JASS, for example, is the easiest tool to use and its results impart insight as to the cognitive requirements of the proposed system. At the other end of the spectrum, field tests are the most operationally valid but they are extremely expensive and quite often lack experimental control because of the variability associated with real world exercises. Simulation experiments and task modeling are better in terms of precision and cost, but lack the real world validity of observing soldiers operating in actual field environments with multiple systems. As argued elsewhere (Barnes & Beevis, 2003), it is the combination of these tools that allow the design team to make corrections and develop new human-centered design concepts early in the development process. Also, the tools themselves are becoming increasingly sophisticated allowing for more rigorous quantitative analysis. IMPRINT tools based on Micro Saintr modeling techniques recently introduced utilities such as fatigue corrections, advanced workload models, and improved analysis of variance software (Walters et al., 2002). The burgeoning development of new tools and methods to be used early in the design process is gradually allowing human engineers to be a proactive rather than a reactive part of the design process.

REFERENCES Barnes, M. J., & Beevis, D. (2003). Chapter 8: Human system measurements and trade-offs in system design. In: H. R. Booher (Ed.), Handbook of human systems integration (pp. 233– 259). New York: Wiley. Barnes, M. J., Knapp, B. G., Tillman, B. W., Walters, B. A., & Velicki, D. (2000). Crew systems analysis of unmanned aerial vehicle (UAV) future job and tasking environments (ARLTR-2081). Aberdeen Proving Ground, MD: U.S. Army Research Laboratory. Barnes, M. J., & Matz, M. (1998). Crew simulation for unmanned aerial vehicles applications: Shift factors, interface issues and crew size. Proceedings of the human factors and ergonomics society 42nd annual meeting, pp. 143–148. Bowman, E. K., Hunn, B. P., & Warner, J. D. (2005). Human factors assessment of the effectiveness of network centric coordination of dissimilar UAVs. Human systems integration symposium, Society of Naval Engineers, Arlington, VA. Gopher, D., Weil, M., & Bareket, T. (1994). Transfer of skill from computer game to flight. Human factors, 36, 387–406. Hunn, B. P. (2004). Forward look III human factors report. Norfolk, VA: United States Joint Forces Command, Intelligence Directorate 32. Knapp, B., & Tillman, B. (1998). Job assessment software system (JASS). Proceedings of the human factors and ergonomics society 42nd annual meeting, pp. 1319–1322. Walters, B. A., Huber, S., French, J., & Barnes, M. J. (2002). Using simulation models to analyze the effects of crew size and fatigue (ARL-CR-0483). Aberdeen Proving Ground, MD: U.S. Army Research Laboratory.

6. UAV OPERATORS, OTHER AIRSPACE USERS, AND REGULATORS: CRITICAL COMPONENTS OF AN UNINHABITED SYSTEM Stephen B. Hottman and Kari Sortland As the demand for both military and commercial applications of unmanned aerial vehicles (UAVs) rises, it becomes increasingly important to consider methods for their safe integration into the United States (U.S.) national airspace system (NAS). The pursuit of safe and routine flight for all UAVs, both in the U.S. and internationally, creates an entirely new set of human factors issues relating to UAV operations. Not only do human factors of UAVs involve operators using various types of ground control stations, whether for single or multiple UAVs, they also are concerned with the amount and type of qualifications UAV operators (pilots) need. Of additional interest are other airspace users and individuals who provide air traffic service in the NAS, in other words, people whose tasks are impacted by UAV operations in some way. Other issues relevant to UAV integration in the NAS include the certification of UAV ground and air components, validation of sense-and-avoid technologies, and electronic communication integrity, to name a few. In this chapter, research focuses on two human factors issues relating to safe integration of UAVs into the NAS – UAV operator selection and training and the effect of UAVs on other airspace Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 71–88 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07006-2

71

72

STEPHEN B. HOTTMAN AND KARI SORTLAND

users and air traffic controllers. Significant synergism exists between the operator of an aircraft and the Federal Aviation Administration (FAA) personnel who control the airspace. UAVs have been used primarily in military applications, but increasing civilian applications are expected (Hottman, 2000). Prior to UAVs being used routinely in the NAS, a method for certifying the operators and aerospace platforms needs to be developed. The FAA has extensive testing protocols developed to determine whether an individual is qualified as a pilot for an FAA license. The understanding of required attributes or skills of the UAV operator operating in the NAS needs to be developed. In addition, operating procedures that enable UAVs to be integrated in the NAS so flights can be performed simultaneously with manned aircraft safely are necessary.

EXPANDED CIVIL/COMMERCIAL USES UAVs have become a critical component of U.S. military operations, reducing the need to risk the life of a pilot, while performing tasks considered dull, dirty, and dangerous. UAVs currently are serving an important intelligence, surveillance, search, and destroy function in Operation Enduring Freedom. Since September 11, 2001, the public has increasingly been made aware of the role that military UAVs, such as the Predator and Global Hawk, play. As federal agencies plan for the potential increased use of UAVs in Homeland Defense and as improvements of UAV reliability and performance occur, civil and commercial applications are emerging. Applications have been demonstrated or planned in such diverse areas as drug interdiction, border monitoring, law enforcement, agriculture, communications relays, aerial photography and mapping, emergency management, and scientific and environmental research (Hottman, Gutman, & Witt, 2000; Nakagawa, Witt, & Hottman, 2001). Certification procedures, regulatory processes, and operating requirements that meet the FAA’s safety-of-flight approval have not been developed for any class of UAVs (Hottman, 2000). In addition, special subsystems that allow the aircraft to operate in the NAS without a human pilot onboard must be certified for use by the FAA. Also, the interaction of those systems, the human operator, and those regulating airspace, such as air traffic controllers, needs to be defined.

UAV Operators, Other Airspace Users, and Regulators

73

Regulatory Environment Currently, few UAV flights are performed in the NAS, outside restricted airspace, each year. Most of these UAV flights are performed under visual flight rules (VFR), and therefore are not under the control of an ATC facility. However, within the next few decades, UAV flight operations in the NAS could be as routine as other aircraft flights, and a significant number of these flight operations may likely be under instrument flight rules (IFR) and therefore under the control of the en route centers (Hottman, Jackson, Sortland, Witt, & Cooke, 2001). Also, to date, no separate regulatory criteria have been established for UAVs, similar to the regulatory criteria created for other UAVs, such as balloons. Regulations for UAVs will have to be developed in the future. The regulations must address the airworthiness of the UAV, qualifications of the operator (pilot), and any special operating rules when the UAV cannot meet the same requirements as manned aircraft. Even when regulatory criteria have been created specifically for UAVs, their operations may be unique and somewhat different from aircraft with a pilot on board. Requirement to Certify Operators:Equivalent Level of Safety Pilots who operate aircraft (with people on board) are certified by the FAA to perform this function. This certification process is based upon a set of knowledge and skills that the student pilot learns, masters, and then is tested. The testing (written and flight) and resulting pilot certificate represent a level of aviation skill, knowledge, ability, and safety that the pilot has attained. The pilot certificate, the airspace regulations (and FAA controllers), and the vehicle certitude levels (e.g., experimental, special, and standard airworthiness) are means to attempt to have an aviation system maximized to protect passengers, other aircraft and people in the air, and the population and infrastructure on the ground. The UAV, ground control station, and the operator are all components that make up the eventual equivalent level of safety needed by the UAV system in comparison to conventional aircraft. Required Attributes and Skills of Operators The attributes and skills of a UAV operator need to be determined empirically. The basic question is does a UAV operator need to be a rated pilot? If

74

Fig. 1.


UAVs Operate at a Variety of Altitudes and in a Variety of FAA-Controlled Airspace.

so, what other unique skills, if any, does this person need in addition? If the UAV operators need not be rated pilots, then what skills do they need to be proficient on to operate a UAV? Do they need to know how to use a radio and understand unique communication lexicons with the FAA controllers? Do they need 20/20 vision? Can a UAV operator be physically handicapped? Can an individual – ‘‘good’’ at Nintendo and other modern electronic games – also be a ‘‘good’’ UAV operator? Fig. 1 represents a small subset of the approximately 200 types of UAVs flying today. Note that these UAVs do not all have the same operational limits.

UAV OPERATOR SELECTION AND TRAINING Just as the qualifications for pilots of manned aircraft vary (O’Hare & Roscoe, 1990), so could the qualifications of a civil UAV operator differ from those required of a pilot flying a manned aircraft. The skills and aviation knowledge necessary to safely operate most UAVs differ, in large part to the level of automation, from those that are necessary to safely fly a manned aircraft. UAVs are capable of being operated in one or more ways by an operator. These ways or mechanisms have a direct relationship to the amount of autonomy present in a specific UAV system. For instance, many small to


75

medium-sized UAVs are controlled by an external pilot (a pilot who is visually observing and controlling a UAV) within a line-of-sight range (Fig. 2). This pilot, conceptually, controls the UAV in a manner not unlike the radio-controlled hobbyist (except the FAA limits the altitude at which a hobbyist can operate). The pilot utilizes a control box with joysticks, or other flight control devices, to send signals to the UAV, which can change throttle or control surface settings. An air traffic controller, who provides traffic management direction to a pilot, would expect to see, for instance, a UAV’s heading and altitude change if so directed by radio communication. The joystick provides synonymous input to the same joystick/controller or yoke in an aircraft with a human-on-board. A significant human factors issue is the potential for reversal (left, right) errors when the UAV is flying toward versus away from the operator. For UAVs with greater autonomy or complexity, a number of pilots may be involved. In some cases, an external pilot may perform the takeoff of the UAV and then pass control to an internal pilot who may be in continuous control of the UAV, perform primarily a monitoring function while the UAV is in autonomous mode, or perform as a monitor but is prepared to

Fig. 2.

An Example of an External Pilot who is Directly Visually Coupled with the UAV.

76


take direct control for a few moments or longer (Fig. 3). In this case, the external pilot would interface with air traffic controllers dedicated to operations near an airport, while a different controller responsible for airspace management away from airports are more likely to interact with an internal pilot. Most automation today is represented by the Global Hawk, which can taxi, takeoff, perform its en route activities, and then land with human pilots functioning as monitors, albeit, ready to take command at any time. While the method of piloting some UAVs (e.g., Predator) is quite similar to those of a manned aircraft, most UAVs are flown by methods that differ considerably. For example, some UAVs (Pointer, Javelin, etc.) are flown like model aircraft. Other UAVs (Global Hawk) are flown through the use of a Flight Management Control System (FMCS) that uses computer-generated input from the operator or an onboard autonomous control unit to effect movement of the control surfaces on the UAV. Therefore, the skill requirements may not need to be identical. Similarly, the aviation knowledge required of the operators of UAVs may not always have to be at the same level as that required for pilots of aircraft with people on board. In essence, this is no different than the aviation knowledge requirements that exist between certificated pilots of manned aircraft. Some UAV operators, those piloting UAVs that operate in terminal airspace (Classes D and C); other classes of airspace (Classes A and B) where positive control is provided by air traffic

Fig. 3.

Internal Pilot (and Payload Operator) can Make Direct UAV Control Inputs or Serve as a Monitor.


77

control (ATC); or, within Class E Airspace, when operating under VFR or IFR, should basically have an equivalent level of aviation knowledge as the pilot of manned aircraft flying in the same airspace under the same criteria. Yet, many potential UAV operations (e.g., chemical spraying of vegetation; science and data collection activity; when conducted at very low altitudes and within confined airspace) may not require the operator to have much aviation knowledge. Other considerations that must be addressed are the medical requirements. Certain UAV flights may not require the operator to meet the same medical standards necessary for the pilot of a manned aircraft performing the same flight activity as the UAV. Therefore, because of the disparity between the requirements for pilots of manned aircraft and those for most UAV operators, a separate certification (licensing) system could be established for UAV operators. From the FAA’s perspective, a UAV is an aircraft. The operator of a UAV is some type of ‘‘pilot’’ who will need to be certified as having the knowledge base that is determined to be necessary and/or appropriate and who also is proficient and skilled. Research has been on-going on some aspects of UAV operators (Flach, 1998; Tirre, 1998). The attributes of UAV operators have not been addressed in a consistent manner within the Department of Defense (DoD). One service requires rated pilots, another that the operator be an officer, yet another that the operator be an enlisted individual. The United States Coast Guard recently has performed a study to address the officer/nonofficer and pilot/nonpilot issue. Research is needed to determine training requirements for a potential UAV operator, how to determine the operator’s proficiency, and, more importantly, how UAV operator certification will be defined for non-DoD flights. Tobin (1999) examined the criteria for UAV operators focusing primarily on the Predator but with the intent to have applicability to future unidentified UAV systems. Tobin concluded that the best staffing alternative was to utilize rated officers. Further, his recommendation was that these rated officers also be pilots to facilitate the interaction with the FAA when the UAVs were flown in the NAS. Additionally, Tobin stated the Predator squadron relied heavily on the operational experience of the pilots that was obtained prior to being assigned as a UAV operator. The Predator training was based upon prior pilot certification as the entry skill level. Weeks (2000) examined UAV operator qualifications by interviewing military, government, and subject matter experts. However, since UAV requirements were not established at that time, he documented input from flight engineers and UAV operators. Weeks (2000) noted that there are large

78


differences in the qualifications of the operators of UAVs, but that more research was needed to identify the essential skills of the UAV operator. Also, for the Predator UAV, the U.S. Air Force conducted research to reach consensus on what qualifies an operator to fly a UAV. The sevengroup experiment team consisted of a variety of piloting experience ranging from actual Predator operators to a group with no flight experience. Results from this study were used to formulate qualification requirements to operate the Predator (Schreiber, Lyon, Martin, & Confer, 2002). UAV operator error has been attributed to a number of UAV accidents. Particular flight phases (e.g., takeoff, en route, and landings) also have different levels of accidents or errors associated with them too (Table 1; as reported by Adams, 2005). Although Manning, Rash, LeDuc, Noback, and McKeon (2004) stated that the U.S. Army’s current accident reporting system does not allow an easy means to identify human error data, a review of the detailed accident analysis agrees with the information in Table 1. Specifically, the frequency of accidents or incidents by the external pilot at landing and takeoff is high. Training implications are suggested by these error data. Research is underway at New Mexico State University (NMSU) to begin to understand UAV pilot requirements. An ongoing study is utilizing FAArated pilots and nonpilots performing a series of tasks, including a video game-type flight simulator, a Personal Computer Air Training Device [(PCATD) and FAA-certified training device], an austere landing task, and the Multi Unified Simulation Environment (MUSE). MUSE is a DoDdeveloped desktop UAV simulation environment with the additional potential for use as a training device for helping to train UAV operators. MUSE represents some of the types of tasks that an internal pilot would perform in the en route part of a UAV flight or mission.

Table 1. UAV

Pioneer Hunter Shadow Predator Global Hawk

UAV Accident Statistics Associated with Critical Flight Phases. Years – Covered

Total Number of Accidents/Incidents

Landing

Takeoff

1968–2002 1995–2003 1980–2004 1994–2004 1998–2004

239 32 24 15 3

68% 47% N/A 13 —

10% 20% N/A N/A —


79

The landing task was selected due to the number of accidents that occur during this part of the flight regime for UAVs. A takeoff task also is under development at NMSU. Takeoff and landing both have high frequencies of accidents. Other, more empirical, research has been conducted by different agencies. The Coast Guard, for instance, utilized nonpilots, provided training, and then had the participants operate UAVs from a vessel on the water. Mixed success was found during the test. The NMSU empirical research needs to be contrasted with a number of activities that are attempting to identify UAV operator requirements through standards organization (Bishop & Picariello, 2004) or industry working groups. These approaches are utilizing individuals with a variety of expertise to develop, generally through a consensus process, a nonresearchbased definition or description for UAV operators related to civil aviation certification. Similar consensus activities are ongoing in Europe and other locations.

THE EFFECT OF UAVs ON AIRSPACE USERS AND REGULATORS In addition to UAV operator selection and training research, a broader look at the human factors of UAVs also needs to focus on those people who are impacted by UAV operations, such as other airspace users and individuals who regulate civilian airspace both domestically and internationally. All domestic U.S. airspace is regulated (to some degree) by the FAA, which imposes its own set of rules on how various categories and classes of airspace are defined, how they are divided up and controlled, as well as defining standards for flight safety and certification for pilots, aircraft, and equipment used on aircraft. UAVs must abide by the same regulations as manned aircraft if they are going to routinely fly in the NAS. One area where UAV integration into civilian airspace has potential impact is ATC. The number of aircraft operations affects air traffic controller tasks and workload. The addition of UAVs to the NAS not only adds to this workload, but also contains the added complexity that most UAVs have different performance characteristics than manned aircraft, requiring controllers to know and use more information to maintain aircraft separation. UAVs typically have slower climb, descend, and turn rates than manned aircraft, for example. A simple difference such as this may necessitate a change in how a controller manages a UAV in relation to other aircraft within a sector of airspace.

80


Studies that have begun to examine human factors issues relating to the integration of UAVs into the NAS have evaluated methods for symbolizing or representing UAVs in ATC (Hottman et al., 2001; Hottman, Sortland, Witt, & Cooke, 2002; Sortland & Hottman, 2003; Sortland, 2003). Earlier studies included task analyses of controller duties and evaluations of the frequency and importance of controller tasks and subtasks (Hottman et al., 2001). More recent studies have focused on evaluating the symbology used in ATC tasks, including that found in radar data blocks and aircraft call signs (Hottman et al., 2002; Sortland & Hottman, 2003; Sortland, 2003). Human factors issues relating to UAV integration into international airspace also have been considered. Of concern here is how UAVs currently operate in civilian airspace, potential areas of impact on airspace users or regulators, and the development of various studies on UAV symbologies for the ATC environment, both domestically and internationally. To date, only a limited amount of research has been conducted on how other people – those not using or operating UAVs – are affected by UAV operations. These people fall into two categories: airspace users and airspace regulators. Although no known studies have been conducted on the impact of UAV operations on other airspace users, the importance of this topic cannot be ignored. UAVs range (as shown in Fig. 1) in dimension from the size of a portable CD player (or even smaller) with the Black Widow to that of the Global Hawk, about 114 feet in wingspan (Masey, 2004). There are different purposes for UAVs, whether they are used for tactical operations, surveillance, reconnaissance, or telecommunications. Perhaps most importantly, UAVs fly at different speeds and possess different turn ratios – not just from one another, but also from manned aircraft. Each of these characteristics implies a unique potential impact on other aircraft within a section of airspace.

The Impact of UAVs on Regulated Airspace As UAV flight gradually becomes more frequent in civilian airspace, airspace regulators, such as the FAA, show an ever-increasing interest in all aspects of UAV operation. Research has been conducted to respond to this interest and to the impact of UAV operations on airspace regulators, both within the U.S. and on a global scale. In an ideal situation, UAVs look and act similarly to manned aircraft and can be treated as such. Because UAVs currently operate so differently from manned aircraft (individual on-board and sense-and-avoid function), the FAA has placed restrictions on their


81

operation within FAA-regulated airspace (FAA Order 7610.4, 2004). One difference that stands out is how UAVs are to accomplish see-and-avoid functions without a pilot on board the aircraft. Following the line of reasoning that at least in the short term UAVs should be treated differently than manned aircraft while flying in civilian airspace, researchers at NMSU/PSL conducted several studies on the representation and impact of UAVs on the air traffic management (ATM) or ATC system (Hottman et al., 2001; Hottman et al., 2002; Sortland & Hottman, 2003; Sortland, 2003). At the time these studies began, no separate regulatory criteria similar to that for other UAVs, such as moored balloons, kites, unmanned free balloons, and unmanned rockets (FAA Regulation Part 101, 2001), had been established for UAVs. This motivated a line of research that was intended to explore if or what regulatory criteria might be created for UAVs and primarily what information about UAVs might need to be presented to airspace regulators. Initial focus was placed on how information regarding UAVs could be presented to air traffic controllers at their operating positions (Hottman et al., 2001). It was felt that the need for controllers to be able to distinguish a UAV from other aircraft would become more important as the number and frequency of UAV flights increased in the future. The integration of UAV operator research and controller research is considered vital to obtain a greater understanding of aviation systems overall. Air traffic controller tasks are complex and have the potential to create a great deal of workload. Therefore, careful consideration needs to be put into the amount and type of information presented to controllers about UAVs. Air traffic controllers interact with an entire network of aircraft that is constantly moving. It is essential to understand the small subtasks in a control environment and their relation to the greater structure of ATC when considering changes to the information controllers use to perform their daily tasks. ATC is made up of three separate components and facilities – terminal, en route, and flight advisory service (Wickens, Mavor, Parasuraman, & McGee, 1998; Smolensky & Stein, 1998). The terminal and en route facilities are responsible primarily for providing separation between aircraft operating in the NAS; whereas, the flight advisory component is responsible for weather and other information that might affect flight safety. In addition, the flight advisory service function is the initial point at which flight plans enter the ATC system. In the terminal environment, there are two categories of operations. The first involves air traffic controllers working in the Air Traffic Control Tower (ATCT) who are responsible for the movement of aircraft on the airport

82


surface and in the airspace that generally can be seen from the tower. The second involves air traffic controllers working in a facility known as a Terminal Radar Approach Control (TRACON). Controllers working at a TRACON are assigned to radarscopes and are responsible for the safety of aircraft primarily operating under IFR during the departure and arrival phases of flight. Typically, the horizontal volume of airspace that a TRACON encompasses is contained within a 30–60 mile radius of the primary airport. The vertical volume of airspace varies for TRACONs and is dictated by the level of air traffic volume. Beyond and above the TRACON airspace is the airspace under the control of the Air Route Traffic Control Centers (ARTCCs), commonly referred to as en route Centers or just Centers. Controllers in the ARTCCs provide air traffic service to aircraft operating under IFR between the departure and arrival at TRACONs’ airspace. When workloads permit, these same ARTCC controllers provide traffic advisory service to aircraft operating under VFR that have established radio communication with the ARTCC and specifically request this service. Pilots flying within Class A airspace (18,000 through 60,000 feet MSL) are required to operate under IFR and therefore to interact with ATC. Once UAV usage becomes routine, interaction with air traffic controllers will occur on a regular basis. However, for a very low-altitude UAV (operating altitude less than several hundred feet), ATC interaction may occur infrequently. ATC interfaces need to be developed to accommodate this new air traffic by potentially including a special representation for UAVs. Initial research conducted by NMSU/PSL evaluated all categories of ATC facilities, but focused on ARTCCs. A lengthy structured questionnaire and a task inventory based on six controller subtasks were conducted at an ARTCC (Hottman et al., 2001). The subtasks were distilled from less structured interviews conducted previously at an ATCT and a TRACON. They included separation, the task of maintaining a defined amount of separation between aircraft; sector handoff, the transferring of planes between controllers; inquiry, a general point-and-click on the aircraft target symbol on the radarscope for more information; point-out, when a controller needs to briefly identify an aircraft to another controller, position relief briefing, when a controller is replaced by another controller who then assumes responsibility for providing air traffic service; and no aircraft, when a controller does not respond to an airplane entering his sector of airspace. It was found that the subtask, aircraft separation, was the most important, the most difficult subtask to perform, and happens continually – at


83

least once an hour (Hottman et al., 2001). To successfully perform aircraft separation, controllers use a piece of information that appears on a radarscope called the data block. It portrays information for each aircraft, including its identification (the call sign), assigned and current altitude, computer identification number, and airspeed. Although this information varies slightly for TRACON versus ARTCC controllers, the data block appears to be the most important piece of information for identifying an aircraft as a UAV. During initial investigations by NMSU/PSL, a Data Block Usage Questionnaire was administered to controllers at an ARTCC (Hottman et al., 2001). They were asked to rate each area of a standard data block for both frequency and importance during the various subtasks. The aircraft call sign was rated with high importance and also was verbally identified by controllers as the area they would modify for UAV information. Based on these findings, the data block was determined to be ideal for the display of UAV information, in addition to two other areas – the flight progress strip (a strip of paper containing information about an aircraft, including its flight plan) and a pull-up screen on the radarscope called the full route information screen. All three of these areas could be modified to portray information about a UAV to controllers without being obtrusive to the controllers’ tasks. Subsequent research investigated the use of different call signs for the identification of UAVs (Hottman et al, 2002; Sortland & Hottman, 2003; Sortland, 2003). Because an aircraft call sign appears both in the radar data block and in flight progress strips, the various call signs were evaluated within these two contexts. For civil aircraft, the call sign, or aircraft identification, is the aircraft registration number. For commercial airlines the call sign normally is assigned two to three letters by the FAA to identify that airline (e.g., AA for American Airlines) followed by the flight number. For military flights, the identification is comprised of one to two letters identifying the service (e.g., AF for Air Force, N for Navy, etc.) followed by the aircraft’s military registration number. The military also is permitted to use special tactical identifications (e.g., COWBOY, ROVER, etc.). Early evaluations of various call signs for the identification of UAVs were patterned after commercial airline call signs (Hottman et al, 2001, 2002). Later studies expanded on this to include call signs that were patterned after identification for civil aircraft (Sortland & Hottman, 2003). To date, the following aircraft call signs have been evaluated at five TRACONs and five ARTCCs, all located throughout the U.S.: UAV173, UIN237, UN4237, and UM9417.

84


A unique methodology was created to evaluate the effectiveness of the call signs (Sortland & Hottman, 2003). Two testing tools were created – one provided a performance measurement and the other subjective ratings. The first tool was a computer simulation of a radarscope, which ran on a laptop (see Fig. 4). The simulation served as a low-fidelity prototype that included 10 different fictional scenarios that were designed to imitate typical air traffic that a controller might see on a radarscope. The scenarios presented a mixture of regular air traffic, military and civilian, with either one UAV or no UAVs present at all. Controllers were asked to select the data block that they thought represented a UAV and their responses were timed. The second testing tool was a questionnaire that collected subjective ratings from the controllers about the four call signs within the context of a data block and a flight progress strip (see Fig. 5). This allowed the controllers to carefully evaluate each of the call sign designs and to provide feedback on them. Additional questions in the questionnaire were designed to obtain some basic demographic information about the controllers and their knowledge of and attitudes toward UAVs. Because different information and layouts are used within ARTCCs and TRACONs, two versions of both the radarscope computer simulation and

Fig. 4.

An Air Traffic Scenario from the Radarscope Computer Simulation for ARTCC Controllers.


Fig. 5.

85

One of the TRACON Data Blocks and One of the TRACON Flight Progress Strips Used in the Questionnaire.

the questionnaire were used in order to cater to both types of facilities. In all, 53 ARTCC controllers and 46 TRACON controllers had both the radarscope computer simulation and the questionnaire administered to them. They were all ‘‘current’’ air traffic controllers (none was retired) and had been employed as controllers for an average of 17.1 years. UAV173 and UM9417 were the aircraft call signs that controllers selected the fastest using the computer simulation program, and they were the call signs that received the most favorable ratings from controllers at both types of facilities (Sortland & Hottman, 2003). It would be interesting to evaluate additional call signs, such as tactical call signs similar to those used by the military in subsequent studies. Furthermore, no studies to date have directly addressed the use of international call signs for UAVs or the use of automatic dependent surveillance (ADS) systems, such as ADS-B (ADS-Broadcast) for ATM, and how UAV identification methods might be integrated into these systems. The International Civil Aviation Organization (ICAO) has endorsed a global air transport system concept called communication, navigation, and surveillance/ATM (CNS/ATM) (Sudarshan, 2003). Communication, navigation, and surveillance form the basic services of ATM. ADS was developed as part of the surveillance component, where aircraft automatically transmit their position and other relevant information to ATC facilities and other aircraft that possess ADS equipment. ADS-B is a broadcast surveillance application of ADS systems that provides both air and ground situational awareness for both pilots and air traffic controllers. It will enable pilot-to-pilot as well as pilot-to-controller interaction and was designed to make air traffic operations safer and in some instances more efficient. New human factors issues may arise as UAV operations are integrated with the ADS-B systems along with civil aircraft. As the use of ADS systems becomes widespread, the information relevant to air traffic controllers about UAVs may change. Air traffic controllers present a unique subject pool because they are very highly trained and are

86


very thorough in their approach to work. Because so much training is required of controllers, it may not be necessary to create call signs that are unique to UAVs. It may be more useful to focus instead on providing information about the type of aircraft. Knowing the type of aircraft is important to a controller because it determines the operational characteristics, such as cruise speed, climb/descent ratios, maximum altitude, maneuverability, and other factors of an aircraft, which are all relevant parameters related to ATM and situational awareness. In other words, some controllers are content simply to memorize additional call signs and to treat all aircraft the same. This willingness may extend to memorizing the performance characteristics of additional aircraft types, whether they are manned or unmanned. Since these studies evaluating how to represent UAVs in ATC began, two additional testing tools have surfaced. One is a program developed by the FAA called Sector Design Analysis Tool (SDAT). SDAT allows users to enter recorded data of actual air traffic from an ARTCC into a program that runs on a laptop computer. The amount and nature of air traffic can be modified easily. SDAT is very flexible and can be used to create a much higher fidelity prototype of a radarscope than the radarscope computer simulation created at NMSU, in addition to other things. The second tool is the dynamic simulator (Dysim), a full-fledged ATC radarscope and accompanying equipment located at all ARTCCs. The Dysim is used to train controllers before they control actual aircraft. The only limitation of the Dysim is the difficulty in gaining access to use it. Both of these tools can be used to create synthetic tasks that allow a lot of flexibility in evaluating what type and amount of information should be provided to controllers about UAVs.

CONCLUSION A UAV system can be viewed as a system of systems. The human operator in control of the UAV platform is a critical component as is the individual or team of air traffic controllers that provide aircraft separation (as well as other important tasks) for aircraft in the NAS. To date, little empirical research has been performed on the UAV operator to determine training and certification requirements. Phenomenological and group consensus approaches have been advanced. The authors propose that an empirical approach be continued with consideration to examining operator requirements based in part on the operational UAV system capability and the corresponding NAS component


87

where the UAV platform will be operated. Those data could then be generalized to address the operator as a component of the general UAV system of systems. A similar approach may be taken with respect to air traffic controllers. Additionally, the integration of the UAV operator and controller research would be a robust paradigm and is of interest to the authors. Discussed in this chapter is original research performed by NMSU that may be considered as setting the stage for air traffic controllers working with UAVs in the NAS. In order for a UAV to operate in civil airspace on a routine basis, data about the UAV must be seamless and easily accessible for controllers. From the systems perspective, it also means that controllers interfacing with UAV operators (during routine or emergency events) should have confidence in communications, whether they take place over radio, internet, or telephone. Further, research with controllers needs to answer questions on workload, interface issues, team impact, symbology, and procedures. The NMSU ATC research is an excellent basis to be expanded to address further ATC issues. The proposed approach is first laboratory simulations, to be followed by validation in operational centers.

REFERENCES Adams, R. (2005). Personal Communication. Bishop, S., & Picariello, P. (2004). ASTM’s F38.03 UAV operator standards update. Proceedings of the TAAC conference, Santa Ana Pueblo, NM. FAA Order 7610.4 (2004). Special military operations. Section 9. Remotely operated aircraft (ROA). Retrieved March 3, 2005 from http://www.faa.gov/ATpubs/MIL/Ch12/ mil1209.2.html. FAA Regulation Part 101. (2001). Title 14, Chapter I, Subchapter F, Part 101. Retrieved March 3, 2005 from http://www.faa.gov/regulations_policies/faa_regulations/ Flach, J. (1998). Uninhabited combat aerial vehicles: Who’s driving? In: Proceedings of the human factors and ergonomics society 42nd annual meeting, (pp. 113–117). Chicago, IL. Hottman, S. (2000). The technical analysis and application center: Progress toward certification and operation in the NAS. Proceedings of the unmanned systems 2000. Association for Unmanned Vehicle Systems International, Arlington, VA. Hottman, S., Gutman, W., & Witt, G. (2000). Research and validation of requirements for UAV operations in the USA. Paris, France: UAV 2000. Hottman, S., Jackson, S., Sortland, K., Witt, G., & Cooke, N. (2001). UAVs and air traffic controllers: Interface considerations. In: Proceedings for the association for unmanned vehicle systems international. Baltimore, MD. Hottman, S., Sortland, K., Witt, G., & Cooke, N. (2002). Air traffic controllers at en route centers and UAVs. In: Proceedings for the association for unmanned vehicle systems international. Orlando, FL.

88


Manning, S. D., Rash, C. E., LeDuc, P. A., Noback, R. K., & McKeon, J. (2004). The role of human causal factors in U.S. army unmanned aerial vehicle accidents. USAARL Report No. 2004–11. U.S. Army Aeromedical Research Laboratory. Masey, J. (Ed.) (2004). Unmanned vehicles handbook 2005. Burnham, England: The Shephard Press. Nakagawa, G., Witt, G., & Hottman, S. B. (2001). HALE UAV certification and regulatory roadmap. TAAC Document C020-01. O’Hare, D., & Roscoe, S. (1990). Flightdeck performance: The human factor. Ames, IA: Iowa State University Press. Schreiber, B. T., Lyon, D. R., Martin, E. L., & Confer, H. A. (2002). Impact of prior flight experience on learning Predator UAV operator skills. United States Air Force Research Laboratory, Warfighter Training Research Division, AFRL-HE-AZ-TR-2002–0026. Smolensky, M. W., & Stein, S. (Eds) (1998). Human factors in air traffic control. San Diego, CA: Academic Press. Sortland, K. (2003). A positive routine: Testing an air traffic control radar scope simulation prototype. Unmanned Vehicles, 8(5), 4–5. Sortland, K., & Hottman, S. (2003). UAV representation in air traffic control. In: Proceedings for the association for unmanned vehicle systems international. Baltimore, MD. Sudarshan, H. V. (2003). Seamless sky. Hampshire, England: Ashgate Publishing Limited. Tirre, W. (1998). Crew selection for uninhabited air vehicles: preliminary investigation of the air vehicle operator (AVO). Proceedings of the human factors and ergonomics society 42nd annual meeting (pp. 118–122). Tobin, K. E. (1999). Piloting the USAF’s UAV fleet: Pilots, non-rated officers, enlisted, or contractors? School of Advance Airpower Studies, Air University, Maxwell Air Force Base, AL. Weeks, J. L. (2000). Unmanned aerial vehicle operator qualifications. United States Air Force Research Laboratory, Warfighter Training Research Division, AFRL-HE-AZ-TR2000–0002. Wickens, C. D., Mavor, A. S., Parasuraman, R., & McGee, J. P. (Eds) (1998). The future of air traffic control: Human operators and automation. Washington, DC: National Academy Press.

ERRORS, MISHAPS, AND ACCIDENTS

An unfortunate aspect of remotely operated vehicle (ROV) operation is that things can and do go wrong. A closer examination of errors, mishaps, and accidents provides converging evidence on important human factors issues across remotely operated systems. The first chapter in this section, by Avi Parush, examines unmanned aerial vehicle (UAV) accidents and classifies the errors according to well-known frameworks in human factors, Rasmussen’s Skills, Rules, and Knowledge framework and Reason’s Generic Error Modeling System. Importantly, this chapter also addresses the implications for ROV design and training. The next chapter by Kevin Williams spans a variety of UAV systems and focuses on flight control errors (i.e., errors during flight, during transfer of control, and automated vehicle control). Some display solutions are provided. Patricia LeDuc’s chapter looks specifically at military UAV accidents. She extends important lessons learned to the battlefield. The last chapter examines spatial disorientation causes of UAV accidents and provides a framework for considering spatial disorientation accidents in UAVs. To that end, Brian Self argues we need to better understand misperceptions of air vehicle orientation, whether the human is onboard the vehicle or not. Together, the analyses of errors, mishaps, and accidents can lead to better designs and safer, more successful operations.

89


90

7. HUMAN ERRORS IN UAV TAKEOFF AND LANDING: THEORETICAL ACCOUNT AND PRACTICAL IMPLICATIONS Avi Parush THE HUMAN OPERATOR AND UAV FLYING The successful culmination of missions based on Unmanned Aerial Vehicles (UAV) can be measured with two main parameters: (1) successful mission completion: all objectives of the mission (e.g., maneuvering and navigation, reconnaissance and targeting or search and rescue, and return) were accomplished and (2) safety: no damage to the vehicle and no fatalities or injuries to any human were sustained throughout the mission. Automation of the UAV’s control and operations increasingly becomes a determining factor in successful mission completion and increased safety. However, in this day and age of automatically launched and retrieved swarms of UAVs, the human operator still has a critical role. Human-controlled UAVs will persist for a long time and human error is a factor that still needs addressing in the age of automation. Even a single person, who has flown radio-controlled model aircraft as a hobby since childhood, can still cause the crash of an expensive UAV in a matter of seconds. Moreover, there are aspects of human error in UAV control that can have important implications to the implementation of automation and to keeping the human operator in the control loop. Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 91–103 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07007-4

91

92

AVI PARUSH

This chapter presents the empirical study of human errors during the critical phases of human-controlled UAV land-based takeoff and landing and proposes a theoretical account to understand those errors. This is aimed at deriving practical implications for the implementation of automation and for training. The chapter includes three main parts: (1) a theoretical framework for understanding human errors, (2) an analysis of human errors data in UAV takeoff and landing, and (3) a discussion of the theoretical and practical implications.

A FRAMEWORK FOR UNDERSTANDING HUMAN ERROR The framework of human error utilized in this chapter is based on Norman’s (1983) and Reason’s (1987, 1990) definitions, and on Rasmussen’s (1987) Skills–Rules–Knowledge theoretical framework. Human error is a term describing a planned sequence of mental or physical actions that fail to achieve the intended objective, and that this failure cannot be attributed to chance. Norman (1983) made a distinction among the variety of possible failures by suggesting that a failure can be related to a problem either with the intention and/or planning of the actions or to the actual execution of the actions. There can be inappropriate intentions and planned actions and such failures are referred to as mistakes or planning failures. For example, a UAV pilot may have decided to initiate a ‘‘Return Home’’ procedure based on perceiving and understanding a certain state of the aircraft. However, the understanding turned out to be wrong or the decision turned out to be the wrong choice for that situation. Even if the appropriate sequence of actions was intended and planned properly, there can still be a failure in achieving the intended outcome. Such failures are referred to as slips and lapses or execution failures. The distinction between the planning stage and the execution stage can be combined with the cognitive information processing approach accounting for human operator performance. Rasmussen (1987) proposed an account for cognitive control that is based on three levels of behavior, Skill, Rule, and Knowledge (SRK) that express operators’ expertise and familiarity with their task and thus enables the identification of error sources for each behavior level. Skill-based behavior refers to automatic, sensory-motor behavior performed usually in highly routine situations. The typical errors associated with this level of behavior are execution errors (slips and lapses),

Human Errors in UAV Takeoff and Landing

93

such as omission of an action or wrong action commission (wrong action, wrong timing, inaccurate actions, etc.). Rule-based behavior is typical of familiar situations where the operator has acquired an inventory of rules of the ‘‘if–then’’ kind (‘‘if’’ the symptoms are X, ‘‘then’’ it means the situation is Y, or ‘‘if’’ situation is Y, ‘‘then’’ action should be Z). Consequently, errors in this behavior level are typically planning or intention errors of misunderstanding or misclassifying a given situation and thus implementing the wrong diagnostic rule and/or implementing the wrong action. Knowledgebased behavior is typical of new, non-familiar situations that require basic understanding and knowledge of the system and the situation. Handling such situations requires conscious cognitive processes such as retrieval of information from long-term memory, search for new information, analytical problem solving, and decision-making. Planning and intention errors are typical of knowledge-based behavior and are associated with inappropriate or incomplete search of information, and making the wrong diagnosis or decision based on incomplete knowledge. When attempting to understand and analyze the possible causes of planning and execution errors, external and internal Performance Influencing Factors (PIFs) should be considered. PIFs are usually associated with the behavioral aspects of human error and the underlying psychological mechanism. External factors are various circumstances of any mission ranging from organizational and task characteristics to human-system interface aspects such as controls, displays, and ergonomics. Internal factors are training and experience, personality, and motivation and attitudes (Swain & Guttmann, 1983; Sanders & Shaw, 1988). This framework is used to 1. classify the errors in the empirical study of UAV-related human errors, 2. theoretically account for and understand the possible causes of those errors, and 3. derive practical implications.

EMPIRICAL STUDY OF UAV-RELATED HUMAN ERRORS The study of human errors associated with UAV takeoff and landings was based on UAV operations of a large para-military organization deploying UAVs for reconnaissance, surveillance, and search missions. The data were extracted from a corpus of reports and accounts of various UAV accidents

94

AVI PARUSH

and critical events spanning three consecutive years. A UAV accident was defined as an event where the UAV was severely damaged and grounded until repaired or was declared as a total loss (Class A and B mishaps). A critical event included near misses or any deviation from a routine and safe flight.

DATA ANALYSIS APPROACH An accident or critical incident report typically included the following parts: a general description of the event, the method used by the inquiry personnel, the main findings in terms of technical and human factors, a conclusion of the cause or causes of the event, and practical recommendations. Analysis in this study included review of the report, identification of the possible causes associated with the event, and determining a distinction between technical problem and human error. Once an event was defined as having human errors associated with it, the errors were classified based on the descriptions and details of the interviews with the operators that were included in the report.

Findings The data included only accidents and critical events and findings here are reported relative to this data set. Reported failure rates were always computed only relative to the total number of accidents and critical events and not the total number of flights.

FAILURES BY FLIGHT TYPE AND CAUSE Rate of failures was examined as a function of two main variables: flight type and plausible cause. Three main types of flights were considered: (1) training flights – usually taking place during the training period of the UAV pilots and payload operators, (2) operational flights – actual reconnaissance and surveillance missions, and (3) test flights – flights testing new payloads, new features installed in the UAV, or acceptance test after repairs and maintenance. Two primary potential causes were considered: (1) technical or mechanical cause – such causes included technical failures to the aircraft itself such as loss of up or down link, engine mechanical failure, payload failures, etc. and (2) human factors – causes or factors associated with the

95


accident or event that were linked to the human operator. The findings are presented in Fig. 1. It can be seen that the highest rate of accidents and critical events was associated with training flights. The second highest rate was found with operational flights, and the lowest rate was found with test flights. The other aspect of the findings is the differences in proportion of accidents and critical events associated with a technical failure as compared with human failure. The data indicate that in the case of training flights, the rate of human failures was slightly higher than technical failures. This trend is reversed for both operational and test flights, with a higher rate of technical failures. This finding is reasonable since training flights usually involve less experienced pilots and operators, whereas operational and test flights involve experienced pilots. It should be noted that many of the events were classified as having both technical and human factors associated with them. With the exception of coordination and violation errors (to be discussed below), all events and accidents started with a technical/mechanical problem.

35

30

Failure Rate

25

20 Human Technical

15

10 5

0 Operational

Training

Testing

Flight Type

Fig. 1.

Overall Failure Rate as a Function of Flight Type and Potential Cause.

96

AVI PARUSH

FAILURE RATE OF EXTERNAL AND INTERNAL PILOTS Another critical variable that was examined in the analysis was the relationship between failures and operator type: (1) the external pilot – the operator responsible for manual and visually based takeoff and landing and (2) the internal pilot – the operator responsible for controlling the vehicle after takeoff, en-route to the mission and inbound for landing after the mission. Failure rate as a function of flight type and operator type is presented in Fig. 2. The proportions reported in this analysis are relative only to the number of accidents and critical events associated with human error. Overall it can be seen that the highest rate of failures was found for the training flights, a finding that was also reflected in Fig. 1. The striking finding is the large difference between the two operators. In all three flight types, there was a higher rate of failures associated with the external pilot as compared to the internal pilot. This difference was particularly evident in training flights, where over 60% of human errors in training flights were associated with the external pilot as compared to less than 5% associated 70

60

Failure Rate

50

40

External Pilot Internal Pilot

30

20 10

0 Operational

Training

Testing

Flight Type

Fig. 2.

Failure Rate as a Function of Flight and Pilot Type.


97

with the internal pilot. This is an indication that the weaker component in the team was the external pilot; the implications of which will be discussed later.

ERROR CLASSIFICATION The original accident inquiries and reports did not address the type or possible origins of the human error. During the re-analysis of the data, error classification based on the framework described in the introduction was employed. In analyzing human errors associated with UAV takeoffs and landings, overt behavioral manifestations of slips, lapses, and mistakes were divided into four major categories: 1. Individual physical actions, which are the actual responses, and actions that the human operator performs, defined as slips and lapses. 2. Individual perceptual processes included sensory detection, recognition, and identification of stimuli and situations. 3. Individual cognitive processing included comparisons, computing, planning, making a choice, etc. This category was sub-divided into decisionmaking and violations. 4. Team communication and coordination processes included receiving or relaying information, replying etc., and was associated with the intention/ planning phase (mistakes). The findings are presented in Fig. 3. In terms of individual-related errors, it was found that most error types committed by the external pilot were associated with either a perceptual process or action execution process. Perceptual errors included not detecting a change in the aircraft situation or not understanding its current situation. The action errors included primarily performing the wrong action. These can be considered as slips and lapses or skill-based errors. Failures to understand perceived information was also viewed as rule-based errors. The next highest proportion of errors were violation errors. These errors were distinct from the decision-making errors because they were identified as cases in which the pilot knowingly decided to employ a different procedure than the one prescribed for a given situation. The least proportion of external pilot errors were associated with poor planning or decision-making. This latter class of errors occurred when there was no explicit mandatory procedure that the pilot had to employ. These can be considered as rulebased bordering knowledge-based errors.

98

AVI PARUSH 30

Failure Rate

25 20 15 10 5 0 Action

Decision

Perception

Violations "Grabbing" Coordination

Individual Error Classification External Pilot

Fig. 3.

Team

Internal Pilot

Rates of Various Human Error Classes Divided by Pilot Type.

Two other types of failures were related to teamwork. One failure type, termed ‘‘Grabbing’’ refers to a situation when an instructor or a senior operator, upon perceiving an emergency situation that was not handled properly by the trainee or junior operator, took control forcefully without any coordination with the other operator. Grabbing occurred primarily in training flights and rarely in test or operational missions. The proportion of ‘‘Grabbing’’ was similar to the proportion of failures associated with lack of or poor coordination between the internal and external pilot. This poor coordination was primarily evident in phases of the flight when control over the UAV had to be transferred manually between the two pilots. Coordination was the only error type that involved the internal pilot with almost the same proportion as the external pilot.

SUMMARY OF THE FINDINGS The human operator seems to be a weak component in the systems of flying UAVs. While the highest proportion of failures associated with accidents and critical events were due to technical failures, there was a relatively high proportion of human failures. The external pilot was a primary weak component. Errors classified as actions-related and perception-related were


99

found to be the most frequent. In addition, there was relatively high proportion of errors related to teamwork.

DISCUSSION Theoretical Account Rasmussen’s SRK framework and Reason’s Generic Error Modeling System (GEMS) (1987, 1990) can be used in accounting for the pattern of findings in this study. UAV pilots are usually highly trained, highly experienced, and highly skilled operators. The organizational and operational framework of flying UAVs for a variety of missions can be characterized as having a clear definition of jobs: the mission manager, the pilots, the payload operator, maintenance personnel, etc. All of these jobs, their boundaries, and the overlaps between them were usually well defined. In addition, there were many elaborate procedures for each of the jobs, tasks, and missions. A major portion of training is devoted to drilling and practicing those procedures. By the end of the training period, most pilots and operators have probably developed skill-based and rule-based behaviors. This is related primarily to the intensive practice of all the procedures to a point when the operator posses an inventory of ‘‘if–then’’ rules for a variety of situations. In terms of Reason’s GEMS (1987, 1990), the control over performance is done in a Schematic Mode where information is processed rapidly and in parallel (somewhat similar to the ‘‘automaticity’’ in the behavior of highly trained experts). However, the schematic mode is ineffective in new, unfamiliar, and surprising situations. In such situations, the operator needs to actively explore and acquire information, process the information, and make diagnostic decisions, all in order to handle the new situation. This is typical of knowledge-based behavior, and in terms of performance control it is done in Attentional Mode, requiring active utilization of working and long-term memory and attention (Reason, 1987, 1990). As was found in the study reported here, a high proportion of errors were reflected in actions (slips and lapses or skill-based behavior), rule-based perceptions, and rule-based decision-making. Since routine performance of UAV operators is primarily skill- and rule-based behavior, it is reasonable to suggest that most of their errors would be associated with failure to apply the acquired skill or rules. A possible account for the occurrences of skill and rule-based errors is the existence of PIFs causing latent failures. Latent failures are events or actions that have potential negative impact that

100

AVI PARUSH

materialize at a later point in time with the combination of other failures or specific operational contexts. Active failures have their negative impact almost immediately and are usually associated directly with the actual accident or critical event (Reason, 1990). The dynamics of most UAV accidents associated with human error were characterized in this study by an initial technical problem, which can be viewed as the latent failure. When the skill- and rule-based errors are considered within the context of the preceding technical/mechanical failure (i.e., latent failure), it is reasonable to assume that the errors occurred in a sudden, surprising, and unfamiliar situation. Moreover, some of the errors occurred within the context of poor team coordination (between instructor– student or between external–internal pilots), which may introduce a new situation to the individual operator. Such situations would require the transition to knowledge-based behavior, or in Reason’s terms, switch from schematic mode to attentional mode of performance control, in order to prevent skill- or rule-based errors or at least reduce their likelihood. Thus, it can be concluded that a possible source or mechanism for the active failure to recover from the latent failures (mechanical or human) reflected a failure to make the appropriate transition from skill- or rule-based behavior to knowledge-based behavior. This possible account is the basis for the following implications for implementing more automation in UAV flying and for training.

Implications for Automation Failure to make a successful transition to knowledge-based behavior as is implied from this study of human-controlled UAV takeoff and landing has implications for the implementation of automation in UAV operations. Some studies have demonstrated a reduction of operator workload with automation implemented in UAV flying (e.g., Dixon, Wickens, & Chang, 2003). However, problems with automation that have been discussed extensively in the literature are the risk of the individual and team being out of the control loop, maintaining poor situation awareness, developing poor mental model of the system, and having difficulties regaining control of the system if automation failed (e.g., Kaber & Endsley, 1997). In terms of Rasmussen’s SRK framework, the potential problem of the timely and effective transition from skill- and rule-based behavior to knowledge-based behavior can occur with human interacting with automation.


101

Conclusions of studies analyzing human errors and critical incidents with UAV control frequently address issues such as user interfaces and procedures (e.g., Williams, 2004). The practical implications are the redesign of user interfaces, the refinement of existing procedures, development of new procedures, or additional practice of operational procedures. However, the potential occurrence of failure to make a transition to knowledge-based behavior cannot be solved only by user interface redesign or new operational procedures.

Implications for Training A successful and effective transition from skill and rule based behaviors to knowledge-based behavior should be considered as a meta-skill and should be trained and practiced as such. The conclusions and theoretical account proposed here suggest that practice and training of operational procedures are essential but may not be sufficient in improving procedural skills and expertise. If a severe problem exists in the transition between skill and rulebased behaviors to knowledge-based, then training and practice should directly address this problem. Specifically, it is proposed that individual and team training should address primarily: 1. The early detection and identification of latent failures such as technical problems or team-related problems. This will make operators more aware of the pre-conditions appropriate for a transition from skill and rule based behaviors to knowledge-based behavior. 2. The effective performance of operators in the resulting new and stressful context. The training strategy should not simply consist of practicing more emergency and failure situations. The training strategy should address practicing specifically the transition between schematic to attentional mode of performance control in a variety of emergency and failure scenarios. Specific training strategies can include the early development and continuous update of mental models of the UAV and the control task and the practice of knowledge-based behaviors (e.g., information seeking, new rule generation, inference making, etc.) under stressful conditions. With the advent of UAV simulators, training scenarios can include the strategy of freezing a situation and giving the trainee ‘‘pop quizzes’’ addressing aspects such as existence of potential latent failures, system

102

AVI PARUSH

status, team status, and situation extrapolation. Such a strategy was shown to be effective in training people to acquire spatial knowledge of an area although they used automatic navigation aids such as a GPS (Parush, Ahuvia-Pick, & Erev, 2005). This approach to training should be distinguished from the Situation Awareness Global Assessment Technique, SAGAT, (Endsley, 2000), which is also based on freezing a simulation run and ‘‘popping’’ quizzes in order to measure users’ situation awareness, as opposed to train situation awareness. In SAGAT, the ‘‘pop quizzes’’ are about a variety of parameters in a variety of tasks, and are randomly presented so the user cannot expect what will be the next question. This is done in order to reduce the potential influence of this measurement technique on situation awareness itself and consequently contaminate the measurement. A similar training approach can be less invasive, that is, not require freezing the simulation, but rather embedding implicit ‘‘pop quizzes’’ in the simulation scenario. Schwaninger (2004) suggested a procedure to train and increase correct detection and rejection rates of airport security screeners. In this procedure, suspected objects were once in awhile ‘‘inserted’’ into the x-ray screening displays and operators detection rates were measured and feedback was given to them. It was found that using this technique, correct detection and rejection rates increased significantly. Similarly, relevant ‘‘test situations’’ can be implicitly embedded in UAV simulation scenarios, thus increasing operators’ awareness to potential latent failures, system status, team status, and situation extrapolation, without actually freezing the simulation. As was noted above with regards to SAGAT, this non-disruptive approach should be distinguished from the Situation Present Assessment Method, SPAM (Durso et al., 1995). The SPAM is a query technique aimed at measuring, not training, users’ situation awareness without interrupting the simulation by embedding ‘‘test’’ tasks within the simulation scenarios. Training is by no means the one and only magic solution to reducing human error in UAV flying. However, effective training strategies along with automation, effective user interface, and appropriate procedures, can together play a critical role in improving safety.

REFERENCES Dixon, S. R., Wickens, C. D., & Chang, D. (2003). Comparing quantitative model predictions to experimental data in multiple-UAV flight control. Proceedings of the 47th human factors and ergonomics society. Durso, F. T., Truitt, T. R., Hackworth, C. A., Crutchfield, J. M., Ohrt, D. D., Nikolic, D., Moertl, P. M., & Manning, C. A. (1995). Expertise and chess: A pilot study


103

comparing situation awareness methodologies. In: D. J. Garland & M. R. Endsley (Eds), Experimental analysis and measurement of situation awareness (pp. 295–303). Daytona Beach, FL: Embry-Riddle Aeronautical Press. Endsley, M. R. (2000). Direct measurement of situation awareness: Validity and use of SAGAT. In: M. R. Endsley & D. J. Garland (Eds), Situation awareness analysis and measurement. Mahwah, NJ: Lawrence Erlbaum Associates. Kaber, D. B., & Endsley, M. R. (1997). The combined effect of level of automation and adaptive automation on human performance with complex, dynamic control systems. In: Proceedings of the 41st annual meeting of the Human Factors and Ergonomics Society (pp. 205–209). Santa Monica, CA: Human Factors and Ergonomics Society. Norman, D. (1983). Position paper on human error. NATO Advanced Research Workshop on Human Error, Bellagio, Italy. Parush, A., Ahuvia-Pick, S., & Erev, I. (2005). Acquiring spatial knowledge with automatic navigation systems. Unpublished manuscript. Rasmussen, J. (1987). Cognitive control and human error mechanisms. In: J. Rasmussen, K. Duncan & J. Leplat (Eds), New technology and human error (pp. 53–61). New York: Wiley. Reason, J. T. (1987). The Chernobyl errors. Bulletin of British Psychological Society, 40, 201–206. Reason, J. T. (1990). Human error. USA: Cambridge University Press. Sanders, M., & Shaw, B. (1988). Research to determine the contribution of system factors in the occurrence of underground injury accidents. Pittsburgh, PA: Bureau of Mines. Schwaninger, A. (2004). Increasing efficiency in airport security screening. In the Proceeding of AVSEC World, Vancouver, B.C., Canada, November. Swain, A., & Guttmann, H. (1983). Handbook of human reliability analysis with emphasis on nuclear power plant applications. NUREG/CR-1278. Washington, DC: Nuclear Regulatory Commission. Williams, K. W. (2004). A summary of unmanned aircraft accident/incident data: Human factors implications. DOT/FAA/AM-04/24, Office of Aerospace Medicine, Washington DC 20591.


104

8. HUMAN FACTORS IMPLICATIONS OF UNMANNED AIRCRAFT ACCIDENTS: FLIGHT-CONTROL PROBLEMS Kevin W. Williams Unmanned aircraft (UA)1 have suffered a disproportionately large number of mishaps relative to manned aircraft (Williams, 2004). In 1996, the Air Force Scientific Advisory Board (AFSAB) identified the human/system interface as the greatest deficiency in current UA designs (Worch et al., 1996). A recent review of accident statistics for military UA lends support to this statement. However, the particular interface deficiencies responsible for most accidents differed across the various systems (Williams, 2004). In reviewing accident data for UA, two approaches can be used. The first is to focus on a particular aircraft system, noting the deficiencies in the interface design for that system that led to individual accidents. The second is to look at accidents across systems, focusing on categories of mishaps that occur across a variety of systems. The first approach can result in specific design changes for a particular system. The second approach can reveal basic human factors issues that apply across a variety of systems. In this chapter, we will look at UA accidents across a variety of systems and will focus on categories of accidents involving flight control. The success or failure of UA will at least in part be determined by how easily they can be flown. Once the pilot has been separated from the aircraft, designers are faced with the basic problem of how to control the aircraft during the flight. Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 105–116 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07008-6

105

106

KEVIN W. WILLIAMS

Three flight-control categories have been selected for review. The first category involves the use of an external pilot (EP) to control the flight of the aircraft. The second flight-control category concerns the transfer of control during flight. The third flight-control category is the automation of flight control. Rather than just a statistical summary of accident data related to UA, which can be found elsewhere (Manning, Rash, LeDuc, Nobak, & McKeon, 2004; Williams, 2004), this chapter will include reviews of a number of specific accidents related to the categories listed above. The intent is to suggest possible interventions to prevent such accidents in the future.

EXTERNAL PILOTING The most basic solution for monitoring position and attitude of an UA is through direct line-of-sight. Because they are usually standing outside, a pilot that maintains direct line-of-sight with the aircraft is usually referred to as the EP, as opposed to an internal pilot (IP) who obtains position and attitude information electronically while inside of a ground control station (GCS). Flight using an EP represents the most basic solution to the problem of separating the pilot from the aircraft while still enabling the pilot to monitor the location and attitude of the aircraft. Pilot perspective is changed from an egocentric to an exocentric point of view. Maintaining visual contact with the UA, the EP can control the aircraft using a hand-held radio control box. Many of these control boxes are similar to those used by radiocontrolled aircraft hobbyists and provide direct control of the flight surfaces of the aircraft through the use of joysticks on the box. Very little automation is involved in the use of such boxes, which control the flight surfaces of the aircraft. For those systems that require an EP, statistics show that the control of the UA during landing is a difficult problem (Gawron, 1998; Williams, 2004). For example, with the Hunter system, flown by the U.S. Army (see Fig. 1), 47% of the human-factors-related accidents occurred while landing the aircraft. An additional 20% of the accidents involved an error by the EP during takeoff (Williams, 2004). Likewise the Pioneer (see Fig. 2), which also uses an EP, experiences a proportionately large number of accidents during landing. A recent analysis of accident data for the Pioneer found that the largest percentage of human factors accidents (68%) was associated with the difficulty experienced by the EP while landing the aircraft (Williams, 2004).

Human Factors Implications of Unmanned Aircraft Accidents

Fig. 1.

Hunter Unmanned Aircraft.

Fig. 2.

Pioneer Unmanned Aircraft.

107

108

KEVIN W. WILLIAMS

Probably the main reason for EP control difficulties, especially during landing, is the fact that there is an inconsistent mapping between the movement of the joystick and the response of the aircraft (McCarley & Wickens, 2005). The failure of a flight control to perform consistently (from the perspective of the pilot) is a violation of the human factors principle of motion compatibility (McCarley & Wickens, 2005; Wickens & Hollands, 2000). For example, when the aircraft is approaching the EP, the control inputs to maneuver the aircraft left and right are opposite to what they would be when the aircraft is moving away from the EP. This inconsistent mapping problem is present for any UA operated using a traditional control box via visual contact. Many current systems have eliminated the need for an EP either by automating the takeoff and landing process or by providing adequate visual, positional, and attitudinal information and control to the IP to accomplish these tasks. Another solution is to make improvements to the control interface for the EP. Quigley, Goodrich, and Beard (2004) designed and tested a variety of control interfaces for improving the performance of the EP. These interfaces included a direct manipulation interface that presents a fixed-horizon wing-view representation from the viewpoint of an observer behind the aircraft, a voice-controlled interface, and a physical-icon interface that is basically a hand-held model of the aircraft that, when manipulated, sends control signals to the aircraft that mimic the attitude of the model. Each of the interfaces provided some benefit to the EP but each also had drawbacks. The physical-icon interface had the fastest response time but did not resolve the issue of inconsistent mapping. The direct manipulation interface provided consistent mapping of the controls but required frequent visual accommodation adjustments between the interface and the aircraft. The voice interface resolved the mapping issue, as long as the commands were world-centered (e.g., ‘‘go north’’), but was not as reliable or responsive as the other interfaces.

TRANSFER OF CONTROL One of the activities unique to remotely piloted aircraft is the transfer of control from one controlling system to another. While transfer of control can occur within a manned cockpit, it does not entail the difficulty or variety of methods encountered with UA systems. Transfer of control is usually required in UA at some point during the flight because of the limited range


109

of the control station and/or stationary pilot. This transfer can occur in one of several ways and the protocol for transferring control can differ radically from one system to another. Control can be transferred from an EP to an IP, from an IP in one GCS to a pilot in a second GCS, and from one side of a GCS to a set of duplicate controls within the same station. In addition, transfer of control during a flight can occur from one pilot to another, analogous to a crew change in a manned aircraft. However, even this control transfer is different in a remotely piloted aircraft because the replacement crew does not have to have been present in the control area during the flight, nor does the replaced crew have to remain close to the control area after the crew has been relieved of flight responsibility. The difficulty in the transfer of control is demonstrated by the fact that problems with control transfer occur in almost every UA system. This section will review several mishaps involving the transfer of control in different systems. Two mishaps involving the Army’s Hunter system are related to the transfer of control of the aircraft (U.S. Army, 2004). In the first mishap, a maintenance crewman turned off the autopilot capability of the aircraft during routine maintenance of the aircraft but inadvertently failed to restore the autopilot functionality prior to returning the aircraft to flight status. The aircraft took off under control of the EP who did not use the autopilot and was thus unaware that it was nonfunctional. The mishap occurred after control of the aircraft was handed off from the EP to the IP. The IP is usually given control of the aircraft with the autopilot functioning since the primary means of control by the IP is through the use of knobs and dials for setting the aircraft heading, airspeed, and altitude. Because the IP was not expecting a nonfunctional autopilot, he failed to notice that the aircraft was descending and was unable to recover the aircraft before it crashed. In the second Hunter mishap, control of the aircraft was being transferred from one EP to another during training. The EP receiving control neglected to complete all control box checks and failed to notice that one of the switches on the box was in the wrong position (Williams, 2004). In the case of a U.S. Army Shadow system (see Fig. 3), two aircraft were damaged during a single mission. The first was damaged due to a failure of the automated landing system. After the accident, the GCS crew issued a command to the damaged aircraft to kill its engine, but because of damage to the antenna the command was not received. That same GCS was then tasked with controlling a second Shadow on an approach. Unfortunately, after taking control of the second Shadow, the aircraft received the ‘‘engine kill’’ command that was still waiting for an acknowledgment from the GCS

110

KEVIN W. WILLIAMS

Fig. 3.

Shadow Unmanned Aircraft.

software, causing the second Shadow to also crash. This accident was classified as both a procedural error (because the crew failed to follow all checklist items prior to the transfer of control of the second aircraft) and a display design problem (because there was not a clear indication to the crew of the status of the ‘‘engine kill’’ command that had been issued) (Williams, 2004). More recent was the crash of a Helios UA in June 2003. The Helios is intended to fly for extended lengths of time at very high altitudes. During a flight test of the Helios in June 2003, the aircraft flew into turbulence that exceeded the maximum capability of the aircraft and broke apart. According to the accident report, at least one contributing factor to the accident was the high workload associated with the control handoff procedure. ‘‘Approximately 10 minutes prior to the event, the pilot was progressively becoming task-saturated with multiple demands on his attention. This included concurrent concerns with the flight hand-off procedure’’ (Noll et al., 2004, p. 84). Yet another instance of a problem with the control handoff occurred during the test of an Altair UA in July 2004. The Altair is a derivative of the Predator B UA. The control station for the Altair normally consists of two pilot stations side by side, which allows control of the aircraft to be transferred from one side to the other during flight. During a flight test, the left


111

pilot console malfunctioned. The crew proceeded to switch control to the right seat pilot console. The switchover took time to accomplish and during that period the aircraft went into lost link mode. Uplink was re-established but as the right seat pilot console came online the engine shut down. Accounts of the event from observers suggested that the cause of the shutdown was a fuel control switch that was not in the correct position on the right seat console. The pilot used his checklist to perform an engine restart and managed to successfully restart the engine and return to the original altitude (Randy Sundberg, personal communication). A common theme across most of the above reported mishaps is a lack of awareness of system settings on the part of the receiving crew. Sometimes this is because checklists are not properly followed. Other times the displays do not present system status information adequately to the pilot. Strategies for mitigating control handoff difficulties will be discussed in the conclusions.

AUTOMATION As UA continue to proliferate, the technology involved with the flightcontrol system continues to become more and more sophisticated. Perhaps because the pilot has been removed from the aircraft, there seems to be an unspoken goal of UA designers to remove the pilot altogether from the system. This involves automating all of the flight-control responsibilities. One of the latest military UA, the Global Hawk, is totally automated from initial taxi and takeoff to landing. The ultimate solution seen by many to the problem of flight control of UA is to automate the control. One reason for the tendency to automate is the difficulties experienced by pilots in controlling the aircraft. EPs have the problems of limited range and inconsistent control mapping as was discussed earlier. IPs have the problems of delayed control feedback, poor visual imagery, a small field of view, and a general lack of sensory cues (McCarley & Wickens, 2005). The degree of automation varies a great deal from system to system, with some systems having the capability to be ‘‘hand-flown’’(e.g., Predator) while other systems have flight control totally automated from takeoff to landing (e.g., Global Hawk). Accidents involving flight-control automation suggest that it is difficult to anticipate all possible contingencies that can occur during a flight and that even if the automation functions as intended, unintended consequences can occur because of events that were not anticipated. A few examples of unintended consequences will illustrate this point.

112

KEVIN W. WILLIAMS

Fig. 4.

Firescout Unmanned Aircraft.

The first accident, which occurred on November 4, 2000, involved the crash of a Navy-owned Vertical Take-off and Landing Tactical Unmanned Aerial Vehicle (VTUAV) called the Fire Scout (see Fig. 4). The investigation of the accident revealed that human error associated with damage to onboard antennas during ground handling led to the accident. Because of the damage to the antennas, an incorrect signal was emitted, causing the radar altimeter system to incorrectly track the altitude. The antennas gave a false reading that indicated that the Fire Scout was at an altitude of 2 ft above the ground when, in fact, it was hovering at an altitude of 500 ft (Strikenet, 2001). After the ‘‘land’’ command was given, the aircraft descended 2 ft to 498 ft above the ground. The guidance and control system interpreted the incorrect altitude signal as an indication that the Fire Scout had already landed and, performing as designed, shut down the engine. A mishap of a Global Hawk (see Fig. 5) occurred when the aircraft suffered an inflight problem with temperature regulation of the avionics compartment and landed at a preprogrammed alternate airport for servicing. After landing, the aircraft was commanded to begin taxiing. Unknown to the crew, a taxi speed of 155 knots had been entered into the mission plan at that particular waypoint by the automated mission planning software in use at the time. The mission planning software had been programmed to set the descent speed of the Global Hawk to 155 knots. Any time that two consecutive waypoints varied in altitude by more than a specified amount, the software set the speed to be established between those


Fig. 5.

113

Global Hawk Unmanned Aircraft.

waypoints at 155 knots. What was not anticipated by the software developers however was two consecutive taxi waypoints on the ground differing by more than the specified amount in altitude. The software, performing as designed, had inserted an airspeed of 155 knots between the waypoints. The aircraft accelerated to the point where it was unable to negotiate a turn and ran off of the runway, collapsing the nose gear and causing extensive damage to the aircraft. The final example is in regard to the previously mentioned mishap involving the Helios system. After entering turbulent conditions, the pilot was unable to provide adequate control inputs to avoid the breakup of the aircraft. According to the mishap report, ‘‘The pilot’s control panel was designed to provide only standard ‘autopilot type’ mode and navigation inputs; it was not designed to provide for direct pilot-in-the-loop control of attitude nor was it designed to provide the pilot [with the] capability to recognize an impending departure from controlled flight or to stabilize the aircraft’’ (Noll et al., 2004, p. 81). In all of these examples, we see evidence that the developers of the automation were unable to predict all possible contingencies. This led to situations in which the automation performed as designed, but not as anticipated.

114

KEVIN W. WILLIAMS

SUMMARY AND CONCLUSION In this chapter, we have seen examples of three types of UA accident categories. All three categories focus on a type of control problem. EP control problems are related to the inconsistent mapping of the control box to the movement of the aircraft. At least two solutions present themselves. The first solution is to design the control box in such a way as to achieve a consistent mapping. Quigley et al. (2004) looked at the use of several options for controlling the aircraft, but all had drawbacks. A second solution is to eliminate the need for an EP by automating those portions of flight that currently require an EP or by moving the flight control to an internal pilot. Such a solution has been accomplished with several current UA systems. But the use of an IP is affected by factors such as a limited field-of-view, delayed control response and feedback, and a lack of sensory cues, and the implementation of automation presents its own problems. It cannot be expected to provide a perfect solution. The problem of transfer of control centers on the fact that the receiver of control is not always fully aware of the status of the system. The problem can be confronted by designing the displays in such a way that all critical system parameters are available to the pilot during the transfer. Most research on UA display design has focused on the task of navigation (e.g., Henry, 2001). Additional research is needed to look at the types of information required during the transfer of control and useful ways to depict that information. A second method for alleviating the problems encountered during control transfer would be to improve training. Pilots must be trained to follow checklists during certain procedures. They must also be trained in regard to potential problems that can arise during the transfer of control. Another possible method for reducing problems related to transfer of control, though not supported empirically at present, is a yoked interface between control stations performing a handoff. Basically, the idea consists of establishing a protocol between two control stations (or within stations if the goal is to transfer control from one side to the other) that ensures that all system parameters of the receiving station match those of the sending station. Transfer of the data could be accomplished either through the aircraft data link or directly between the stations as technology permitted. Problems with automation focus on the fact that not all circumstances can be predicted beforehand. The inability to anticipate all possible contingencies leads to situations in which the system behaves as it was designed, but not in a manner that was expected. There are at least two solutions to this


115

problem. The first solution is to design the system in a way that keeps the pilot more aware of what the aircraft is going to do during the flight. This requirement of course assumes that the pilot will also have the ability to intercede in the automated task that is being performed. In implementing this solution, we have to deal with what is usually called the ‘‘out-of-theloop’’ syndrome (Endsley & Kiris, 1995; Moray, 1986; Wickens & Hollands, 2000). The out-of-the-loop syndrome refers to the finding that humans working with automation have a diminished ability to detect system errors and respond to them through the performance of a manual task. The second solution to the automation problem is to design the automation to be more flexible so that, even when a particular contingency has not been anticipated, the system is still able to generate an appropriate response (Parasuraman & Miller, 2006). This is a problem for those developing intelligent systems, and this field is still in its infancy. One conclusion that can be derived from the mishaps presented in this chapter is that flight control of UA is problematic. Understanding the specific issues discussed here should help reduce those accidents. However, as the interfaces of these aircraft continue to evolve, other issues will appear. Much work remains to improve the user interface for UA. New displays, control-interface concepts, and improved automation remain to be developed.

NOTES 1. The use of the term ‘‘unmanned aircraft’’ in this chapter is synonymous with several other terms including unmanned/uninhabited aerial vehicle (UAV), remotely operated vehicle (ROV), remotely operated aircraft (ROA), and others. Unmanned aircraft (UA), and unmanned aircraft system (UAS) are the currently accepted terms adopted by the U.S. Federal Aviation Administration and the Department of Defense.

REFERENCES Endsley, M. R., & Kiris, E. O. (1995). The out-of-the-loop performance problem and level of control in automation. Human Factors, 37(2), 381–394. Gawron, V. J. (1998). Human factors issues in the development, evaluation, and operation of uninhabited aerial vehicles. AUVSI ’98: Proceedings of the Association for Unmanned Vehicle Systems International (pp. 431–438) Huntsville, AL. Henry, M. (2001). GUI techniques for assessing autonomous vehicle behavior. Air Force Research Laboratory/HECI Technical Report CSC-01-333-F.

116

KEVIN W. WILLIAMS

Manning, S. D., Rash, C. E., LeDuc, P. A., Noback, R. K., & McKeon, J. (2004). The role of human causal factors in U.S. Army unmanned aerial vehicle accidents. U.S. Army Aeromedical Research Laboratory Report # 2004-11. McCarley, J. S., & Wickens, C. D. (2005). Human factors implications of UAVs in the national airspace. University of Illinois Institute of Aviation Technical Report (AHFD-05-5/ FAA-05-1). Savoy, IL: Aviation Human Factors Division. Moray, N. (1986). Monitoring behavior and supervisory control. In: K. Boff (Ed.), Handbook of perception and human performance (pp. 40/1–40/51). New York: Wiley. Noll, T. E., Brown, J. M., Perez-Davis, M. E., Ishmael, S. D., Tiffany, G. C., & Gaier, M. (2004). Investigation of the Helios Prototype Aircraft mishap: Volume I, mishap report. Retrieved 9/2004 from http://www.nasa.gov/pdf/64317main_helios.pdf Parasuraman, R., & Miller, C. (2006). Delegation interfaces for human supervision of multiple unmanned vehicles: Theory, experiments, and practical applications. In: N. J. Cooke, H. Pringle, H. Pedersen & O. Connor (Eds), Human factors of remotely operated vehicles. Amsterdam: Elsevier Publishing Co. Quigley, M., Goodrich, M. A., & Beard, R. W. (2004). Semi-autonomous human-UAV interfaces for fixed-wing mini-UAVs. Proceedings of IROS 2004, Sendai, Japan, September. Strikenet. (2001). VTUAV P1 accident investigation board report. Retrieved April 15, 2004 from http://www.strikenet.js.mil/pao/vtuav_crash_invest_news_release.html U.S. Army. (2004). Risk management information system website. Downloaded from https:// safety.army.mil/home.html Wickens, C. D., & Hollands, J. G. (2000). Engineering psychology and human performance (3rd ed.). New Jersey: Prentice-Hall, Inc. Williams, K. W. (2004). A summary of unmanned aircraft accident/incident data: Human factors implications. Technical Report Publication No. DOT/FAA/AM-04/24. U.S. Department of Transportation, Federal Aviation Administration, Office of Aerospace Medicine, Washington, DC. Worch, P. R., Borky, J., Gabriel, R., Heiser, W., Swalm, T., & Wong, T. (1996). United States Air Force Scientific Advisory Board report on UAV technologies and combat operations. SAB-TR-96-01. Washington, DC: General Printing Office.

9. HUMAN FACTORS IN U.S. MILITARY UNMANNED AERIAL VEHICLE ACCIDENTS Clarence E. Rash, Patricia A. LeDuc and Sharon D. Manning The Department of Defense (DoD) defines an unmanned aerial vehicle (UAV) as a ‘‘powered aerial vehicle that does not carry a human operator, uses aerodynamic forces to provide lift, can fly autonomously or be operated remotely, can be expendable or recoverable, and can carry lethal or nonlethal payloads’’ (Department of Defense, 2001a). UAVs can range in size from just a few feet (1 m) to hundreds of feet (50 m) in length with wingspans up to several hundred feet. Payload capability currently approaches one ton with flight times as high as 48 h and a maximum altitude capability in excess of 60,000 ft (18,000 m) (Larm, 1996; Department of Defense, 2005). As UAV development is a very active field, these operating parameters are constantly changing. UAVs can be categorized as either autonomous or remotely operated. In autonomous designs, the UAVs fly and execute mission profiles under computer software control. With autonomous UAVs, specialists program an onboard computer that controls the aircraft flight from point-to-point. The UAV generally takes off and lands itself. While a human may develop the software routines that make up the control program defining where the UAV should go and what it should do, it is the onboard computer that actually controls the UAV in flight (Garamone, 2002). A basic remotely operated UAV typically has a crew of three personnel designated as the air Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 117–131 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07009-8

117

118

CLARENCE E. RASH ET AL.

vehicle operator, external pilot, and mission payload operator (Barnes, Knapp, Tillman, Walters, & Velicki, 2000). The external pilot ‘‘flies’’ the UAV during takeoffs and landings. The air vehicle operator monitors the UAV’s progress between way points, making any necessary adjustments as it flies to and from the target area. The mission payload operator executes the search pattern within the target area. Today, most UAVs are a blend of both the autonomous and remotely operated systems. While having seen limited use for several decades, UAVs are now rapidly coming into their own as major tactical and strategic systems on the modern battlefield. A myriad of UAV types are currently being operated in the military environment. Many more are in various stages of planning, development, testing, and fielding. As expected, the increase in UAV use has been accompanied by an increased frequency of accidents. Both past and recent studies have implicated human error in 60–80% of manned aircraft accidents for civil and military aviation. It is not unreasonable to believe that with the rapid technological advances and future projected uses for UAVs, human error rates in UAV accidents will soon parallel those of manned aircraft. The U.S. military branches have attempted to investigate UAV accidents, analyzing them for identification of causal factors. In spite of the implication of a reduced human role in UAV systems, all branches of the military have found human error to be a significant causal factor in UAV mishaps and accidents, ranging from 28% to 50% across branches and 21% to 68% across UAV type. This chapter compares and contrasts two different approaches used to examine the role of human factors in U.S. Army UAV accidents and summarizes data from the other branches of the military. In demonstrating that human error plays as significant a role in UAV accidents, this work shows the need for development and implementation of countermeasures that target these errors. Within the U.S. Army, UAV accidents were considered ground accidents and investigated accordingly until 1 October 2003 when Department of Army Safety message R041331Z reclassified these accidents as aviation accidents, requiring expanded data recording of accident details (Director of Army Safety, 2003). However, even with such improved emphasis, the ability to gain insight into UAV accident causal factors within the Army and across the other military branches is hindered due to the nature of the details surrounding many accidents. In military conflicts, knowledge of UAV losses, either due to accidents or enemy fire, is naturally a sensitive nature. Very few statistics are available in the open literature. However, the available data currently do point to a higher accident rate for UAVs over manned aircraft (Department of Defense, 2001b; Jackson, 2003).

Human Factors in U.S. Military UAV Accidents

119

Since 1994, the U.S. Air Force has reported losing half of its R-1 Predator fleet due to crashes or enemy fire (Cable News Network, 2002). According to Tvaryanas (2004), most of the crashes (8 of 15 or 67%) were due to human error. In 1999, the U.S. Navy and U.S. Marine Corps, acknowledging an ‘‘unacceptable’’ UAV accident rate, attributed half of the accidents in part to human factor causes (Ferguson, 1999). As of 2001, UAVs represented a miniscule 0.6% of the DoD military aircraft fleet. The ratio of manned to unmanned flight hours was 300 to 1. However, UAVs suffered an accident rate of 10–100 times that of manned aircraft (Department of Defense, 2001b). A study based upon an accumulated 50,000 flight hours attributed 75% of accidents to propulsion, flight control systems, and operator error. Operator error, as a causal factor, was present in approximately 20% of all accidents (Department of Defense, 2001b). These accident data, however limited, have attracted the attention of and raised concerns within the military. Even the Department of Homeland Security, while developing a UAV program for border patrol duty, has expressed concerns for the high accident rate of UAVs, citing worldwide accident rates 100 times greater than for manned vehicles (Jackson, 2003). According to Hansman (2005), the mishap rates of UAVs range from 31 to 363 per 100,000 flight hours compared with 0.012 to 6.7 per 100,000 flight hours for commercial and civil aviation. These rates are estimates based on past trends, as few UAVs have accumulated 100,000 flight hours. However, there does appear to be a significant enough number of UAV accidents to warrant investigation of causal factors, for the purpose of reducing equipment losses and increasing mission success. Recent research has shown that while the manned aviation accident rate attributable entirely to mechanical failure has decreased noticeably over the past 40 years, the rate attributable at least in part to human error has declined at a lower rate (Shappell & Wiegmann, 2000). As mechanical failures decrease with the maturation of UAV technology, human error will naturally account for a higher percent of accidents. Knowledge of these human-related causal factors is necessary for the successful formulation of countermeasures that prevent these types of accidents. Such an understanding can be achieved by the application of accident analysis techniques to existing accident databases.

MILITARY UAV ACCIDENT DATA DoD accidents are classified according to the severity of injury, occupational illness, and vehicle and/or property damage costs (Department of

120


Table 1. Class A An accident in which the resulting total cost of property damage is $1,000,000 or more; an Army aircraft or missile is destroyed, missing, or abandoned; or an injury and/or occupational illness results in a fatality or permanent total disability

Army Accident Classes.

Class B

Class C

Class D

An accident in which the resulting total cost of property damage is $200,000 or more but less than $1,000,000; an injury and/or occupational illness results in permanent partial disability, or when three or more personnel are hospitalized as inpatients as the result of a single occurrence

An accident in which the resulting total cost of property damage is $20,000 or more but less than $200,000; a nonfatal injury that causes any loss of time from work beyond the day or shift on which it occurred; or a nonfatal occupational illness that causes loss of time from work or disability at any time

An accident in which the resulting total cost of property damage is $2,000 or more but less than $20,000

Source: Department of the Army (1994)

Defense, 2000). All branches of the military have similar accident classification schemes, with Class A being the most severe. Table 1 shows the accident classes for the Army. The Air Force and Navy definitions of Class A–C accidents are very similar to the Army’s definition. However, they do not have a Class D. As the total costs of some Army UAVs are below the Class A criteria ($325,000 per Shadow aircraft; Schaefer, 2003), reviewers have begun to add Class D data into their analyses (Manning, Rash, LeDuc, Noback, & McKeon, 2004; Williams, 2004). U.S. Army An in-depth examination of U.S. Army UAV accidents (Class A–D) was conducted by Manning et al. (2004) using data obtained from the U.S. Army Risk Management Information System (RMIS) accident database maintained by the U.S. Army Combat Readiness Center (formerly the U.S. Army Safety Center), Fort Rucker, AL. A search for the period FY95– FY03 found a total of 56 UAV accidents (see Table 2). Each accident was reviewed, and human factor accidents classified, using two approaches. The

121


first was a variant on a methodology referred to as the Human Factors Analysis and Classification System (HFACS). The HFACS is a broad human error approach for investigating and analyzing the human causes of aviation accidents (Shappell & Wiegmann, 2000, 2001). It was developed and tested as a tool for use within the U.S. military and has gained wide acceptance within both the aviation accident investigation and human factors communities. Based upon Reason’s (1990) model of latent and active failures, the HFACS addresses human error at all levels of the system, including aircrew and organizational factors. The second approach used accident methodology defined in Pamphlet 385-40, ‘‘Army accident investigation and reporting.’’ Human causal factors used by this analysis approach are broken down into five types of failure: individual failure, leader failure, training failure, support failure, and standards failure (Department of the Army, 1994). The following sections expand on these two analysis approaches, describing and providing examples of the human error categories used by each approach.

The HFACS The HFACS captures data for four levels of human-related failure (Shappell & Wiegmann, 2000): unsafe acts, preconditions for unsafe acts, unsafe supervision, and organizational influences. These four levels of human-related failure are expanded into 17 causal categories (Shappell & Wiegmann, 2001). The levels, categories, and subcategories are briefly described below with selective examples from Shappell and Wiegmann (2000), which provide an expanded list of examples for each category. Table 2.

Summary of U.S. Army UAV Accidents by Causal Factors.

Causal factor(s)

Frequency

Percent

Material failure Environment Human error Material failure & human error Environment & human error Material failure & environment Material failure & environment & human error Undetermined (or left blank)

18 3 6 6 5 0 1 17

32 5 11 11 9 0 2 30

Total

56

100

122


The unsafe acts level is divided into two categories: errors and violations. These two categories differ in ‘‘intent.’’ Errors are unintended mistakes and are further categorized into skill-based errors, decision errors, and perceptual errors. Examples of skill-based errors include inadvertently leaving out an item on a checklist, failure to prioritize actions, and omitting a procedural step. Examples of decision errors include using the wrong procedure, misdiagnosing an emergency, and performing an incorrect action. Perceptual errors are those made due to the presence of visual illusions and spatial disorientation. Violations are willful errors. Examples include violating training rules, performing an overaggressive maneuver, and intentionally exceeding mission constraints. The unsafe preconditions level is divided into two major categories: substandard conditions of operators and substandard practices of operators. The substandard conditions of operators category is subdivided into three subcategories: adverse mental states, adverse physiological states, and physical/mental limitations. Examples of adverse mental states include complacency, ‘‘get-home-itis,’’ and misplaced motivation. Examples of adverse physiological states include medical illness and physical fatigue. Examples of physical/mental limitations include insufficient reaction time and incompatible intelligence/aptitude. The substandard practices of operators category is subdivided into two subcategories: crew resource management and personal readiness. Examples of crew resource management include failure to use all available resources and failure to coordinate. Examples of personal readiness are self-medication and violation of crew rest requirements. The unsafe supervision level is divided into four categories: inadequate supervision, planned inappropriate operations, failure to correct a known problem, and supervisory violations. Examples of inadequate supervision include failure to provide training, failure to provide operational doctrine, and failure to provide oversight. Examples of planned inappropriate operations include failure to provide correct data, failure to provide sufficient personnel, and failure to provide the opportunity for adequate crew rest. Examples of failure to correct a known problem include failure to initiate corrective action and failure to report unsafe tendencies. Examples of supervisory violations include authorizing an unnecessary hazard and failure to enforce rules and regulations. The organizational influences level has three categories: resource/acquisition management, organizational climate, and organizational process. Examples of resource/ acquisition management include lack of funding, poor equipment design, and insufficient manpower. Examples of organizational climate include policies on drugs and alcohol, value and belief culture, and chain-of-command structure. Examples of organizational process include quality of


123

safety programs, influence of time pressure, and the presence or absence of clearly defined objectives. As described above, the HFACS is used to classify human factors in accidents at both a broad level and with further refinement of the data into narrower categories that can be used for analyses. Department of the Army Pamphlet 385-40 The second analysis approach was based on the accident methodology defined in Department of the Army Pamphlet 385-40, ‘‘Army accident investigation and reporting.’’ The Army uses a ‘‘4-W’’ approach to accident analysis that addresses the sequence of events leading to the accident. The ‘‘4-Ws’’ are: (1) When did error/failure/environment factor/injury occur? (2) What happened? (3) Why did it happen? (4) What should be done about it? Human causal factors are classified into five types of failure: individual failure, leader failure, training failure, support failure, and standards failure (Department of the Army, 1994). These failure types are described as follows: Individual failure – when the soldier/individual knows and is trained to a standard but elects not to follow the standard (i.e., lack of self-discipline – mistake due to own personal factors such as attitude, haste, overconfidence, self-induced fatigue, etc.). Leader failure – when the leader fails to enforce known standards, make required corrections, or take appropriate action. Training failure – when the soldier/individual is not trained to a known standard (i.e., insufficient, incorrect or no training on the task – insufficient in content or amount). Support failure – when there is inadequate equipment/facilities/services in type, design, availability, condition, or when there is an insufficient number/type of personnel, and these deficiencies contribute to human error. Standards failure – when standards/procedures are not clear or practical, or do not exist. Comparison of Analyses Findings Application of both the HFACS and DA PAM 385-40 analyses identified the same 18 accidents involving human error. While there was no one-to-one correspondence between the categories defined by the two analyses, the authors proposed an association depicted by the relationships in Fig. 1. The proposed relationships loosely correlate the HFACS categories of ‘‘Unsafe acts’’ and ‘‘Unsafe preconditions’’ with ‘‘Individual failure;’’ ‘‘Unsafe supervision’’ correlates with ‘‘Leader failure’’ and ‘‘Training failure;’’ and

124


Individual failure Unsafe acts Leader failure Unsafe preconditions Training failure Unsafe supervision Support failure Organizational influences Standards failure

Fig. 1.

Suggested Relationships between the HFACS and DA PAM 385-40 Categories of Causal Factors.

‘‘Organizational influences’’ correlates with ‘‘Support failure’’ and ‘‘Standards failure.’’ When accident proportions were grouped according to the proposed relationships of causal factor categories, they were approximately equal (Table 3). Considering the small number of accidents involved, this finding would seem to strongly support the proposed association of causal categories in Fig. 1. In the investigation of human error in accidents, it is useful to be able to describe the user population. Unfortunately, Manning et al. (2004) found considerable omissions in the accident reports regarding demographic data for the UAV operators associated with the accidents. Age data were available for only 35 of the 56 operators; gender data for only 39 of the 56; and personnel classification data only 40 of the 56. However, for the data present, the operator age ranged from 20 to 55 years, with a mean and standard deviation of 30.3 and 10.7 years, respectively. Operators were mostly male (64%) and U.S. Army personnel (54%). Two blocks on the accident reporting form (Department of the Army, 1994) address the number of hours the operator had been on duty prior to

125


Table 3.

Accident Proportions by Grouped Relationships. Sum of HFACSa (%)

Sum of PAM 385-40a (%)

PAM 385-40 categories

Unsafe acts Preconditions for unsafe acts

61

55

Individual failure

Unsafe supervision

50

44

Leader failure Training failure

Organizational influences

44

44

Support failure Standards failure

HFACS categories

a

Percentages exceed 100% because any given accident may have multiple causal factors.

the accident and the number of cumulative hours of sleep the operator reported having within the previous 24 h. For the hours on duty block, there were valid responses for only 35 of the 56 accidents. Based on these valid responses only, the range was 0–12 h, with a mean and standard deviation of 5.2 and 2.6 h, respectively. A similar problem with recorded data existed for the hours of sleep block. Based on 33 valid responses, the range was 0–10 hours, with a mean and standard deviation of 7.9 and 0.8 h, respectively. Eleven accidents reported zero hours for both blocks. These were considered invalid responses, since all but four accidents were in a training environment, and a report of zero hours sleep seemed unlikely. These gaps in the data compromise the ability to obtain a complete picture of the full role of human error in U.S. Army UAV accidents. In a follow-up analysis, Giffin (2004) categorized these same UAV accidents generally from the perspective of location of the accident with some expansion on causal factor. The largest group of accidents involved crashes in the training area of operation (AO), with losses attributed to weather, maintenance, mechanical failure, and operator error. The next largest group of accidents occurred in and around the recovery launch site (RLS) and involved two personal injuries. This second perspective on the data suggests that human error in UAV operations occurs frequently during training and supports the idea of automation for the most difficult phases of flight such as take off and landing. The most recently available statistics on U.S. Army UAV accidents was provided by Williams (2004). Repeating the RMIS database query of Manning et al. (2004) for the extended period of January 1980 to June 2004, 74 accidents were identified. The earliest occurred in March 1989, the latest

126


in April 2004. By UAV type, 32 of the 74 accidents involved the Hunter; 24 involved the Shadow; and the remaining 18 accidents involved drones, trainers, or unidentified types. A classification of only the 56 Hunter and Shadow accidents showed that 47% of Hunter accidents, 21% of Shadow accents, and 36% of combined Hunter and Shadow accidents have human factors as a causal factor. This tracks closely with the 32% statistic found by Manning et al. (2004).

U.S. Air Force U.S. Air Force UAV accidents have had the greatest media exposure. Nearly half the U.S. Air Force’s fleet of RQ-1A Predator UAVs have crashed or been shot down in the years since they have been deployed. The 2002 accident rate for the Predator was estimated at 32.8 per 100,000 flying hours. However, it must be considered that no single Predator has actually amassed 100,000 flight hours. The accident rate was extrapolated from early developmental data (Hebert, 2003). More current data suggests that the Predator accident rate is beginning to level off (Hansman, 2005). An accident analysis performed by Williams (2004) reported a total of 12 U.S. Air Force Predator accidents. By far, the most common causal factor was human factors, which was identified in 67% of the 12 accidents. Rogers et al. (2004) analyzed 48 combined U.S. Army and U.S. Air Force accidents over the 10-year period of January 1993 to June 2003. These researchers found that 33 of the 48 (69%) involved human factors causes.

U.S. Navy Ferguson (1999) surveyed U.S. Navy and U.S. Marine Corps Pioneer UAV accidents for the period FY86–FY98. His summary relied on data from a 1995 report (Schmidt & Parker, 1995) that examined 107 accidents occurring between the years 1986–1993 and a later 1997 report (Seagle, 1997) that analyzed 203 accidents from the expanded period of 1986–1997. Schmidt and Parker (1995) provided a causal factor breakdown and concluded that approximately one-third of the accidents involved human error. Seagle’s (1997) revisiting of the Pioneer accidents produced a comparable 31% human error percentage (88 of 203). Williams (2004) performed a third analysis of 239 Pioneer accidents from 1986 to 2002. He reported human factors-related causal factors in approximately 28% of the accidents.

127


SUMMARY The investigation of UAV accidents by the military branches has been limited in nature. This can be attributed to three factors: the restricted nature of military operations, the initial limited approach to accident data collection for UAV accidents, and the relatively small number of UAVs in service during early fielding. Studies investigating the role of human error in UAV accidents and the associated human factors issues have been equally sparse. Over almost three decades of UAV operation by the military, the major causal factor for accidents has been aircraft (mechanical) failure. However, the percentages of UAV accidents involving human error, as reported by numerous studies, have been significant and have remained relatively consistent. A summary of these percentages by U.S. military branch is presented in Table 4. Manning et al. (2004) evaluated U.S. Army UAV accidents over the period of FY95–FY03. Their study was confined mostly to training accidents but was focused on the human error aspects of the accidents. Giffin (2004) provided a second perspective on these data that suggested that human error in UAV operations occurs frequently during training and most often in the difficult phases of flight such as take off and landing. Further examination of this data showed that of the Hunter accidents, 47% were due to problems with the external pilot during aircraft landing; while only 5 of the 24 Shadow accidents (21%) were attributed to human factors issues (Williams, 2004). The three main reports, which examined the role of human causal factors in U.S. Navy and U.S. Marine Corps UAV accidents, showed similar rates. Schmidt and Parker (1995) concluded that approximately one-third of the accidents involved human error, Seagle (1997) found a comparable 31%, and Williams (2004) reported human factors-related causes in approximately 28% of the accidents. As seen with the U.S. Army data, most of the Table 4.

Summary of UAV human error accidents rates across military branches. U.S. Air Force (%)

Manning et al. (2004) Schmidt and Parker (1995) Rogers et al. (2004)

U.S. Army (%) 32

33 69

Seagle (1997) Williams (2004)

U.S. Navy (%)

31 67

36

28

128


U.S. Navy and U.S. Marine Corps UAV accidents occurred in a training environment during the take off and landing phases of flight. Rogers et al. (2004) summarized a total of 48 U.S. Air Force accidents. Of the 48 mishaps, approximately two-third of these mishaps involved some form of human-systems issues. Williams (2004) summarized much of the same accident data cited above and found human factors errors in 67% of the Predator accidents. This finding for U.S. Air Force data is different than for the UAVs used by the other branches of the service. Analyses of the U.S. Army, Navy, and Marine UAV accident data consistently find human factors casual rates in one-third of all accidents (approximately half of that found in U.S. Air Force UAV accidents). These services have reported material failure to be the largest causal factor in UAV accidents. Two explanations for these disparate results seem plausible. First, reports from the U.S. Army, Navy, and Marine Corps accidents were primarily based upon UAVs that require an external pilot to land the aircraft. The most common human factor error reported was a misjudgment on the part of the external pilot during this phase of flight. The U.S. Air Force accident rate is derived from UAVs with automatic landing capabilities. At first this seems counterintuitive. However, in spite of the word ‘‘unmanned’’ in the phrase UAV, virtually all UAVs are still manned, in the sense that there are humans involved in some manner with all UAV flights (Mouloua, Gilson, & Hancock, 2003). In some cases, the aircraft is guided manually using stick and rudder controls, with the operator receiving visual imagery from a forward looking camera mounted on the vehicle. Even in autonomous UAVs like the Global Hawk, with automated take off and landing capabilities, humans conduct mission planning, mission software development, and mission oversight with the capability of hands-on override. Automation often is touted as a means to reduce personnel requirements and accident rates. However, such automation does not ensure reduced human error. In fact, mismatched or misapplied automation can induce human error due to inadequate feedback, under reliance on the automation, or over reliance on the automation (Parasuraman, 1997; Norman, 1990; Sorkin, 1988). A second possible explanation for the differing results among the service’s UAV accidents is the location where the accidents occurred. Most of the accidents reviewed for the U.S. Army, Navy, and Marine Corps reports took place in a training environment. This does not appear to be the case with data from the U.S. Air Force. It is not unreasonable to assume that students in a training environment are supervised much more closely than UAV operators conducting real-world missions. Additionally, it would seem possible that workload, stress, and fatigue are much higher among UAV


129

operators in actual field environments than in training environments. McCauley (2004) discussed several of these issues in a white paper on ‘‘Human Systems Integration and Automation Issues in Small Unmanned Aerial Vehicles.’’ The purpose of the white paper was to investigate methods to improve UAV effectiveness and to make recommendations for the development of future semi-automated UAV systems. The paper identified that personnel selection and training, sleep, circadian rhythm, and fatigue as important human systems integration issues.

CONCLUSION The predominant means of investigating the causal role of human error in all accidents remains the analysis of postaccident data (Wiegmann & Shappell, 1999). Manning et al. (2004) reported that, regardless of the approach used (HFACS or DA PAM 385-40) the same individual UAV accidents were determined to have human factors causes. Either method appeared to be adequate for the gross purpose of classification. It was clear, however, from the various reports that pinpointing causality for these accidents was not quite as simple. Manning et al. (2004) found that when the accident reviews were thorough in data collection, the HFACS provided more useful and detailed information to help examine individual human errors. However, many of the studies mentioned in this chapter did note a lack of information contained within the individual accident reports (e.g., fatigue, workload, and operator experience). With the rising costs of UAVs and the high rates of human error consistently reported across the airframes, all branches of the military have begun to recognize that insufficient emphasis has been placed on human factors issues in unmanned operations. In a 1996 U.S. Air Force report on ‘‘UAV Technologies and Combat Operations,’’ the Air Force Scientific Advisory Board reviewed UAV technology maturity with respect to Air Force mission tasks (U.S. Air Force Scientific Advisory Board, 1996). A major finding was that human factors considerations were being overlooked. It was reported that the understanding and application of systematic approaches to allocating functions between humans and automation and the application of human factors principles in system design have been particularly deficient. One important recommendation was to improve ‘‘(the integration) of human controllers into (the) automated systems’’ of UAVs. Having established the role of human error as a significant and consistent cause of UAV accidents, the next desirable step to take would be for all

130


branches of the service to adopt the same type of accident reporting and classification schema. However, the implementation of any augmented accident data collection method is useful only if investigators conscientiously collect the necessary data. As has been seen throughout this chapter, a significant number of data parameters, many of which are believed by the authors to be readily available, are not being recorded. Therefore, it is further recommended that the military, down to the unit level, place command emphasis on the training of investigators, emphasizing the need for, and the importance of, conscientious recording during accident investigations. This will be necessary in order to achieve the goal of using UAV accident data to help develop accident prevention strategies.

REFERENCES Barnes, M. J., Knapp, B. G., Tillman, B. W., Walters, B. A., & Velicki, D. (2000). Crew systems analysis of unmanned aerial vehicle (UAV) future job and tasking environments. Report No. ARL-TR-2081. Aberdeen Proving Ground, MD: Army Research Laboratory. Cable News Network. (2002). Nearly half air force’s UAV predators lost since deployment. Retrieved January 16, 2003, from http://www.rense.com/general33/half.html Department of Defense. (2000). Accident investigation, reporting and record keeping. Instruction No. 6055.7. Office of the Secretary of Defense, Washington, DC. Department of Defense. (2001a). Dictionary of military and associated terms, Joint Publication 1-02 (p. 557). Washington, DC: Office of the Secretary of Defense. Department of Defense. (2001b). Unmanned aerial vehicles roadmap, 2000–2025. Washington, DC: Office of the Secretary of Defense. Department of Defense. (2005). Unmanned aircraft systems (UAS) roadmap, 2005–2030. Washington, DC: Office of the Secretary of Defense. Department of the Army. (1994). Army accident investigation and reporting. Army Pamphlet 385-40. Washington, DC. Director of Army Safety. (2003, October). Classification of unmanned aerial vehicle (UAV). Message, Director of Army Safety, Washington, DC, R041331Z. Ferguson, M. G. (1999). Stochastic modeling of naval unmanned aerial vehicle mishaps: Assessment of potential intervention strategies. Unpublished master’s thesis, Naval Postgraduate School, Monterey, CA. Garamone, J. (2002). Unmanned aerial vehicles proving their worth over Afghanistan. Retrieved January 17, 2003 from www.gordon.army.mil/ac/sumr02/uav1 Giffin, B. (2004). UAV Risk Management. Flightfax, 32(8), 10–11. Hansman, J. R. (2005). Human factors of UAVs: ‘‘Manning the unmanned’’. Retrieved from http://www.cerici.org/workshop/presentation/opening.pdf Hebert, A. (2003). New horizons for combat UAVs. Air Force Magazine Online. Retrieved March 2, 2005 from http://www.afa.org/magazine/dec2003/1203uav.asp Jackson, P. (2003). Jane’s all the world’s aircraft 2003–2004 (pp. 721–722). Alexandria, VA: Jane’s Information Group.


131

Larm, D. (1996). Expendable remotely piloted for strategic offensive air power roles. Maxwell Air Force Base, AL: Air University Press. Manning, S. D., Rash, C. E., LeDuc, P. A., Noback, R. K., & McKeon, J. (2004). The role of human casual factors in US army unmanned aerial vehicle accidents. USAARL Report No. 2004-11. U.S. Army Aeromedical Research Laboratory, Fort Rucker, AL. McCauley, M. E. (2004). Human systems integration and automation issues in small unmanned aerial vehicles. Monterey, CA: US Naval Postgraduate School NPS-OR-04-008. Mouloua, M., Gilson, R., & Hancock, P. (2003). Human-centered design of unmanned aerial vehicles. Ergonomics in Design, 11, 6–11. Norman, D. (1990). The problem with automation: Inappropriate feedback and interaction, not over-automation. Proceedings of the Royal Society of London, B237, 585–593. Parasuraman, R. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253. Reason, J. (1990). The contribution of latent human failures to the breakdown of complex systems. Philosophical Transactions of Royal Society London Series B: Biological Sciences, 327(1241), 475–484. Rogers, B., Palmer, B., Chitwood, J., & Hover, G. (2004). Human-systems issues in UAV design and operation. Wright-Patterson AFB, OH: Human Systems Information Analysis Center HSIAC-RA-2004-001. Schaefer, R. (2003, February). Unmanned aerial vehicle reliability study. Washington, DC: Office of the Secretary of Defense. Schmidt, J., & Parker, R. (1995). Development of a UAV mishap factors database. Presented at association of unmanned vehicle systems 1995 conference, Washington, DC, July. Seagle, J. Jr. (1997). Unmanned aerial vehicle mishaps: A human factors approach. Masters thesis, Embry-Riddle Aeronautical University, Norfolk, VA. Shappell, S., & Wiegmann, D. (2000). Human factors analysis and classification system – HFACS. Washington, DC: Department of Transportation, Federal Aviation Administration DOT/FAA/AM-00/7. Shappell, S., & Wiegmann, D. (2001). Unraveling the mystery of general aviation controlled flight into terrain accidents using HFACS. Presented at the 11th international symposium on aviation psychology, Ohio State University, Columbus, OH. Sorkin, R. D. (1988). Why are people turning off their alarms? Journal of the Acoustical Society of America, 84, 1107–1108. Tvaryanas, A. P. (2004). USAF UAV mishap epidemiology, 1997–2003. Presented at the human factors of uninhabited aerial vehicles first annual workshop, Scottsdale, AZ, May. U.S. Air Force Scientific Advisory Board. (1996). UAV technologies and combat operations. SAF/PA96-1204 UAV. Wiegmann, D. A., & Shappell, S. A. (1999). Human factors analysis of aviation accident data: Developing a needs-based, data-driven safety program. Paper presented at the 3rd workshop on human error, safety, and system development, June 7–8, Liege, Belgium. Williams, K. W. (2004). A summary of unmanned aircraft accident/incident date: Human factors implications. Civil Aerospace Medical Institute, Federal Aviation Administration. DOT/FAA/AM-04/24.


132

10. SPATIAL DISORIENTATION IN UNINHABITED AERIAL VEHICLES Brian P. Self, William R. Ercoline, Wesley A. Olson and Anthony P. Tvaryanas The mishap UAV operator (MUAVO) flared at an altitude that was higher than normal. Although the exact flare height could not be determined, the tower controller estimated it to be about 10 feet. The MUAVO stated he ‘‘expected the runway to be there’’ at the completion of his flare, and was surprised when ‘‘the bottom fell out.’’ Until the sink rate became visible, the MUAVO was unaware of his true height.1

While the threat of spatial disorientation (SD) is well documented and accepted in traditional manned aviation, it is sometimes regarded with more skepticism when discussed in reference to operations involving unmanned aerial vehicles (UAVs) or uninhabited aerial combat vehicles (UCAVs). However, as the mishap report above illustrates, UAV operators may be just as susceptible to SD as those who fly in the cockpit. The example above cannot be dismissed simply as an isolated case. A recent 10-year cross sectional review of human factors in UAV mishaps within the U.S. Department of Defense (DoD) found misperception error was present in 5% of mishaps. Additionally, misperception error accounted for 10% of UAV mishaps in which operator error was a causal factor irrespective of branch of military service (Tvaryanas, Thompson, & Constable, 2005). It is worth noting that current human error rates are overshadowed by the higher unreliability of


133

134

BRIAN P. SELF ET AL.

other UAV subsystems. As was the case in manned aviation, the proportion of mishaps involving misperception errors can be expected to increase as subsystem reliabilities improve and electromechanical causes of mishaps decline. Although there are mishap reports indicating that SD may be a problem in UAV operations, there is very little data analyzing the exact SD mechanisms or even primary illusions that occur. One case study of SD in an Air Force MQ-1 Predator operator (Landsman, 2005) discusses the non-standard presentation of flight data provided to the operator. The UAV mishap study (Tvaryanas et al., 2005) provides an assessment of the frequency of SD in DoD UAV mishaps, but does not discuss exact SD mechanisms. Finally, some studies have also examined navigational ‘‘cardinal-direction judgment problems’’ during UAV flight (Gugerty & Brooks, 2004). Due to the dearth of research in this area, we will provide a theoretical framework for examining SD problems in UAVs. We first present some background information and definitions of SD. This is followed by a suggested taxonomy for describing UAV operations, and then a discussion of when SD might occur during UAV flight. Finally, some suggested countermeasures to the UAV SD problem are provided.

BACKGROUND Definition of SD SD is defined as a failure to sense correctly the attitude, motion, and/or position of the aircraft with respect to the surface of the earth (Benson, 2003). The types of SD are generally thought to be ‘‘unrecognized’’ and ‘‘recognized’’ (Previc & Ercoline, 2004). Although a third type has been reported (incapacitating), this type seems irrelevant to UAV operations. Unrecognized SD occurs when the person at the controls is unaware that a change in the motion/attitude of the aircraft has taken place. The cause is often the result of a combination of sub-threshold motion and inattention. This type of SD is known to be the single most serious human factors reason for aircraft accidents today, accounting for roughly 90% of all known SD-related mishaps (Davenport, 2000). Recognized SD occurs when a noticeable conflict is created between the actual motion/attitude of the aircraft and any one of the common physiological sensory mechanisms (e.g., visual,

Spatial Disorientation in UAVs

135

vestibular, auditory, and tactile). Recognized SD is the most common type of SD, accounting for the remaining SD-related accidents.

An SD Focused UAV Taxonomy The occurrence of SD types will be dependent on several characteristics of UAV flight. UAV is a very broad term and covers a wide range of vehicles and capabilities. Therefore, some method of classification is necessary in order to study, generalize, predict, and prevent SD phenomena in a logical and organized manner. Existing UAV taxonomies are based on attributes such as weight, performance, and mission, and do not consider variables relevant to the study of SD. SD mishaps occur when inaccurate operator perceptions result in inappropriate control inputs. The potential for SD and the specific types of visual and/or proprioceptive misperceptions experienced by UAV operators are largely dependent on UAV visual reference, control methodology, and operator platform; our taxonomy is based on these three factors.

Visual Reference (VR) Since the UAV operator is not physically located in the vehicle, false perceptions leading to SD arise primarily from the inaccurate visualization of the UAV’s relationship with the surface of the earth. The first major distinction is between those UAVs which require the operator to rely on an external view (VR–EV) vs. those which provide some type of operator display. The operator in an external view UAV often uses a portable control station analogous to the controls used by traditional radio-controlled aircraft hobbyists to guide the air vehicle using only direct visual contact. Operator displays may be further divided into those that incorporate an egocentric (VR–EG) vs. an exocentric (VR–EX) view. Egocentric displays depict aircraft position and attitude from the perspective of the vehicle, as if the operator were actually inside the UAV. These displays can range from a simple attitude indicator to those incorporating a sensor image of the outside world. For example, the Predator provides an egocentric display, which superimposes a Head Up Display (HUD) over a camera scene of the outside world. Exocentric displays show the position of the aircraft based on an external view, whether it is a ‘‘God’s eye’’ or a ‘‘tethered’’ view. Each of these display types can lead to various perceptual errors. These errors will be influenced by competing or

136


interfering perceptual cues associated with the operator platform (e.g., ground vehicle, ship, or aircraft). Operator Platform (OP) Since the physical location of the UAV operator is not constrained to the aircraft, operations may occur in a variety of environments. With respect to SD, the most relevant environmental factor is presence or absence of operator motion. Thus, we distinguish between stationary (OP-S) and mobile (OP-M) platforms. Mobile UAV control stations could be located on a ground vehicle, ship, or even another aircraft. SD in stationary platforms will be limited to visual perceptual errors. In contrast, the visual, vestibular, and proprioceptive cues in a mobile platform may exacerbate existing perceptual errors or create opportunities for others. In order for these errors to be expressed as inappropriate control inputs, the operator must be able to affect the flight profile of the UAV. Control Method (CM) There are a variety of existing taxonomies describing levels of automation (Sheridan, 1997), all of which describe a continuum from manual to completely autonomous control. From an SD perspective we are primarily concerned with describing how the operator expresses incorrect perceptions through control inputs. In this light we will use Sheridan’s (1997) distinction between manual control (CM–MC), supervisory control (CM–SC), and fully automatic (CM–FA) systems. We define a manual control system as one in which the operator’s inputs provide inner loop control of the UAV. For example, the operator may have inputs to control ailerons, rudder and elevators. In a supervisory control system the operator controls the UAV by providing higher level goals and targets (e.g., altitude, airspeed, headings, and waypoints), which the automated systems aboard the UAV translate into flight control inputs. A fully automatic system is defined as one in which the UAV is programed prior to takeoff. Summary Fig. 1 provides a graphical depiction of our taxonomy. Each axis depicts the levels of the three factors – visual reference, operator platform and control

137


Fig. 1.

An SD Focused Taxonomy.

method. Within a factor, the potential for SD increases with distance from the origin. For example, SD is judged more likely under manual control than for supervisory or automated control since many automated systems preclude the operator from providing inappropriate control inputs arising from SD. Note that a given UAV cannot be strictly categorized by a fixed level of each factor. Instead, a given UAV may vary across levels within all three factors depending on mission considerations and operator preference (e.g., switching autopilot on and off). A discussion of the USN/Marine Corps RQ-2 Pioneer provides an example of the multiple and migratory nature of classification resident in a single UAV system. The External Pilot (EP) uses a portable control station analogous to a traditional radio-controlled aircraft system to guide the air vehicle during takeoff from beside the runway (VR–EV; OP–S; CM–MC) or on the deck of a ship (OP–M). After launch, control of the air vehicle is passed to the internal pilot located inside the ground control station. The air vehicle can be manually controlled using video (VR–EG; CM–MC), operated via commands to the autopilot (CM–SC), or using preprogramed GPS coordinates (CM–FA). For recovery, control is passed back to the (EP) who

138


may either land the air vehicle on a runway (VR–EV; CM–MC; OP–S) or onboard a ship (OP–M).

MECHANISMS OF SD IN UAVS Now that we have established the definition of SD and outlined an SD taxonomy for describing UAV operations, we can begin to discuss the mechanisms of SD in UAVs. Visual illusions will certainly be a factor in UAV SD, given this is the primary sensory input for operators. Visualvestibular mismatch may also be a factor in SD, especially for the OP–M case. Even the OP–S UAV operator is subject to conflicting non-motion ambient and proprioceptive cues from the ground control station environment and motion visual cues from the sensors and displays. Finally, psychological factors such as task saturation and channelized attention may be contributing factors in UAV SD events.

Visual Orientation in Space Humans derive the majority of their spatial orientation from visual stimuli, known as visual dominance. Our visual system is capable of full, precise mapping in three dimensions and will typically override auditory and somatosensory spatial information. Most of our orientation cues actually come from ambient vision as opposed to focal vision, which provides object recognition and identification. This dependence on ambient vision is demonstrated by a pilot landing his aircraft. Because the pilot’s direct central view of the runway is blocked by the nose of the aircraft and his central attention is on the instrument panel, his ambient vision toward the sides of the aircraft is critical in maintaining orientation. When landing, the pilot also uses optical flow to help judge the speed of the aircraft during approach. Optical flow is but one example of the nature of visual information as it applies to aircraft control. There are numerous other examples that have been known to cause misperceptions in attitude, altitude, and airspeed. Many different cues exist to help us orient ourselves in space, including motion flow, linear perspective, texture size, density gradients, motion parallax, brightness gradients, and aerial perspective. If any of these cues are degraded due to a poor visual environment, SD can easily result due to visual dominance. A comprehensive review of visual orientation mechanisms is provided by Previc (2004).


139

Visual Illusions during UAV Flight Traditional SD mishaps have been attributed to inherent inadequacies in a pilot’s sensory systems to provide reliable information in the flight environment (Previc, 2004). In the case of UAVs, added to this are limitations in the sensor and display systems which further restrict or alter cues presented to the operator coupled with incongruous or absent vestibular and kinesthetic feedback. Because most UAV operators rely on direct visual, sensor, and/or visual symbology input, many of the visual illusions described by Previc (2004) are applicable to the UAV operator (for more information refer to the USAF Spatial Disorientation website, 2005). A perfect example of situations that may lead to operator SD can be found in a mishap report: y pilot was not trained in flight physiology or night illusions. Pilot was conducting a night time recovery and misjudged mishap UAV airspeed and distance to the recovery zone. Pilot failed to maintain sufficient airspeed and mishap UAV stalled, impacting the water.1

Many UAV operations take place in conditions similar to that described above: at night, over water, and during high workload conditions. Manned aircraft pilots have long struggled with spatial orientation when flying with night vision goggles due to lack of textural and perspective cues, as well as a limited field of view (Crowley, 1991). It can be very difficult to discern differences between various ground lights and runway lights; in unusual attitudes, it can even be difficult to tell the difference between point lights on the ground and stars (especially if bodies of water show reflections of stars). Land that slopes gradually towards mountains or sloping cloud decks can also disorient pilots. Because pilots typically try to orient themselves with a level horizon, these cues can cause UAV operators to inappropriately control the aircraft. These problems are most likely to occur for operators under manual control with egocentric visuals, although external view operators might also experience false horizons or night illusions. For example, external view pilots visually tracking a UAV during a night landing based on vehicle position lights could be susceptible to autokinesis. Manual control/egocentric UAV operators may also find themselves prone to SD because of illusions resulting from shape and size constancy. Typically, operators are accustomed to flying and landing at a specific location. If they then attempt to land at an unfamiliar airport, the pilot may unknowingly try to ‘‘fit’’ the new visual scene to one previously experienced. If the actual runway is sloped or a different width than the pilot typically experiences, the pilot may misinterpret their altitude or distance from the

140


runway. If an external view pilot is flying at an unfamiliar location, he may also misjudge his altitude or runway approach. Another mishap demonstrates how important the operator’s field of view is: Mishap pilot (MP) was conducting a night visual approach and landing. Thinking he was on the runway, MP released back pressure on the control stick while still 12 feet above the runway. The MUAV touched down nose gear first. The MUAV began a series of four nose gear-first bounces of increasing magnitude which resulted in the failure of the nose gear shock strut assembly. MP lost the external visual picture when the IR camera was damaged on the fourth bounce.1

This mishap description shows how important ambient vision is for egocentric display pilots who have manual control over their UAV. In UAV direct video, sensor technology (e.g., IR camera), or different display symbology, only the focal field of view is stimulated. To compensate, the pilots must apply new strategies for orienting themselves. If other tasks also require focal attention, the orientation of the aircraft may suffer. This can become increasingly dangerous if the focal visual scene (which can be compared to looking through a soda straw) has poor resolution, a slow refresh rate, inadequate detail, or a time lag due to transmission difficulties. Although the egocentric, manual control case is the most analogous to manned flight, SD can also occur in other control methods and display types. Movement of the UAV pilot relative to the air-vehicle and the advent of external pilots have created new visual illusions: The mishap student external pilot (MSEP) failed to correct for left drift of the mishap UAV because of the perspective of the air vehicle as it passed the external pilot station and his inexperience in detecting this visual illusion.1

Perceptual problems of visually judging whether the UAV is flying towards or away from the external pilot has significant consequences because of the issue of control reversal: MSEP attempted to correct to centerline but made incorrect control inputs. MSEP was not familiar with the phenomenon where control inputs will appear to be opposite as the UAV is approaching and normal as it passes and continues away (e.g., control reversal).1

A recent study (Gugerty & Brooks, 2004) reported on ‘‘cardinal-directionjudgment problems’’ when referencing UAV sensor imagery. The paper summarized interview data from USAF Predator operators and data from controlled experiments and simulation tasks using Air Force recruits. It was found that cardinal-direction-judgment problems were common in Predator operations and air-vehicle cardinal orientation significantly impacted accuracy and response time to spatial orientation problems. Pilots or operators with erroneous spatial orientation may provide incorrect directions to


141

weapons officers or sensor operators, seriously degrading mission performance. These examples are by no means comprehensive and it is likely new UAV-specific illusions will be discovered as UAVs continue to evolve.

Non-Visual Orientation in Space While visual illusions can occur on any operator platform with any visual reference, true vestibular illusions can only occur on a moving platform. The reader is referred to several excellent texts on aviation physiology for a full review of non-visual orientation (Dehart & Davis, 2002; Previc & Ercoline, 2004), but a brief overview will be supplied here. As mentioned previously, humans exhibit visual dominance in obtaining orientation cues. The vestibular system, consisting of the semicircular canals and the otoliths, also provides critical information. The semicircular canals act as angular acceleration sensors, but can become fooled during prolonged rotational maneuvers. The otoliths act as tilt sensors and provide linear acceleration cues; SD can occur because the otoliths cannot always differentiate between the two. Finally, the somatosensory system consists of different pressure sensors and proprioceptors throughout the body. During forward acceleration, a pilot feels ‘‘pushed back’’ in the seat. Pilots describe using the combination of these different non-visual cues as ‘‘flying by the seat of the pants.’’

Mismatch Illusions during UAV Flight Because they are not physically located on the aircraft, UAV operators will by definition have a visual-vestibular mismatch. A stationary platform UAV operator is bombarded with ambient visual and nonvestibular proprioceptive cues from the control station environment, which convey no motion. At the same time, operators receive focal visual cues from the sensors and displays that convey motion. In the event that the operators control the UAV from a moving platform, they may even receive motion inputs that are totally independent and even contradictory of the visual inputs they receive from the UAV. The most comprehensive study that addresses these issues was performed by Reed (1977). During this experiment, motion cues (from a hydraulically actuated platform) and visual cues (from a video screen representing flight of a remotely piloted vehicle (RPV)) that represented turbulence were presented to subjects with varying flight experience. The tests included

142


conditions where only the RPV experienced turbulence, a compatible condition where the motion platform moved in the same direction as the RPV (e.g., the visual rolled right, the platform rolled right), and two incompatible conditions where the platform moved opposite the visual (e.g., the RPV rolled right, the platform rolled left). Visual-proprioceptive conflicts increased the number of control errors by all subjects although pilots tended to make more errors than nonpilots. This may suggest potential negative transfer from manned flying experience. Interestingly, the experimental condition involving only motion of the visual image produced more control errors than the compatibility condition. As might be expected, the incompatibility conditions resulted in the most control errors. Results also showed that presence of motion (both compatible and incompatible) shortened response times, serving as an alerting stimulus. Reed (1977) reinforced the idea that even on a stationary platform, egocentric operators can experience SD and may make control errors during flight. He also made the argument for worrying about SD during the more complex case of placing the operator on a moving platform whose motion is incompatible with that of the UAV. Several examples of this exist in the operational world, and strategists have envisioned additional scenarios to extend UAV range and operability. Early in the Pioneer Program, the Navy launched UAVs from battleships and Landing Platform Dock (LPD) class ships, and even experimented with controlling a Firescout from a P-3C Orion aircraft. An Airborne Manned Unmanned System Technology demonstration has experimented with having the aircrew of an Apache Longbow operate the sensors of a Hunter UAV (Fayaud, 2001). The French have plans to fly a UCAV from the back seat of their fighter aircraft (Barry & Zimet, 2001). All of these examples can easily result in sensory miscues, causing both SD and possibly motion sickness. Only a few studies beyond Reed have examined flying performance while aboard a moving platform. A recent study found that placing flight simulators near the ship’s center of gravity during low-levels of ship motion did not elicit any signs of motion adaptation syndrome (Muth & Lawson, 2003). In a study using a head mounted virtual display during sustained vertical oscillations, Dizio (2000) found the two effects to be somewhat additive. He concluded that using a virtual environment aboard a ship that is uncoupled to its motion is likely to cause some motion sickness. Another study (DeVries & Jansen, 2002) has shown that visually induced motion can have deleterious effects on situational awareness and the absolute localization of objects. Several of these studies have included motion sickness and simulator sickness, which both involve sensory conflict. These issues may become


143

important as UAV operator platforms evolve, and readers are referred to Lawson, Graeber, Mead, and Muth (2002) for more detail. The laboratory based studies mentioned here only scratch the surface of problems that may be experienced by pilots trying to fly and land UAVs from a moving platform. The Reed (1977) report shows that control errors can result from visual-only cases, which applies directly to the manual control, stationary platform, egocentric view UAV operator. Operators on moving platforms will have a variety of random input cues that may involve turbulence, unexpected flight maneuvers of their own aircraft, low frequency rocking from their ship, or accelerations and bumpiness in a tank or truck. It will be extremely challenging for UAV operators to ignore their vestibular and kinesthetic input and concentrate on the task of flying their UAV.

Psychological Factors Certain psychological factors may certainly exacerbate the occurrence of SD described above. The instrument crosscheck is a selective attention task essential for both preventing and detecting SD. Selective attention failures such as task misprioritization are thus at the core of SD. In fact, nearly all ‘‘Unrecognized’’ SD mishaps occur as a result of task misprioritization (DeHart, & Davis, 2002). Under conditions of high workload or time pressure, attention management may falter resulting in channelized attention and a breakdown in the essential instrument crosscheck. UAV controllervehicle interface design features such as multifunction controls, complicated multi-menu computer screens, and a lack of standardization contribute to failures of attention management. Additionally, conditions affecting a UAV operator’s physical or mental health, such as medications, alcohol or illicit substances, fatigue, and stress can also have detrimental effects.

COUNTERMEASURES The physiological chain of events leading to SD has been well described in traditional manned aviation (DeHart & Davis, 2002). This chain starts with confusing sense organ stimulation or perceptual processing which leads to orientation illusions, eventually either recognized or unrecognized SD, and in some rare cases incapacitating SD. The same interventions employed in manned aviation can also be utilized in UAV operations since the underlying mechanics of SD are unchanged. Many of the known SD illusions can

144


be prevented by procedural changes, improved instrumentation, and increased instrument proficiency. For example, flight procedures can be modified to avoid visual or vestibular stimuli, which tend to create SD illusions. Additional countermeasures can be tied to the UAV SD taxonomy discussed in the ‘‘An SD Focused UAV Taxonomy’’ section. For those UAVs with egocentric displays, vehicle symbology can be improved to better translate vehicle position and motion information into more intuitive orientation cues. Current attitude indicators and HUD symbology primarily stimulate focal vision and are inefficient in presenting visual spatial orientation information (DeHart & Davis, 2002). Increased ambient visual cuing should be provided by increasing sensor field-of-views or employing enhanced synthetic vision (UAV Reliability Study, 2003). Preliminary work with multimodal displays (e.g., haptic or aural feedback) has had mixed to promising results but still needs further study. Finally, repeated exposure of UAV operators to SD provoking environments will assist in their development of perceptual processes, which result in accurate rather than inaccurate orientation percepts. In addition to symbology, display design must consider attention management. This is largely dependent on the UAV operator appropriately prioritizing tasks and continuously performing an instrument crosscheck, both of which should be instilled during their initial UAV training. Even earlier, the UAV control interfaces should be initially designed to avoid situations which are prone to channelized attention such as complex multilayer computer menus. UAV control methods also provide opportunities to implement SD countermeasures. Shifting from manual control to a higher level of automation should reduce the likelihood of SD related accidents. This shift can be either temporary or permanent. For example, judicious use of automation such as engaging the autopilot or selecting a higher level of autonomy can help give the UAV operator time to recover once orientation illusions are encountered (DeHart & Davis, 2002). Additionally, increased automation can also be proactively employed in environments prone to orientation illusions such as auto takeoff and recovery (UAV Reliability Study, 2003). Finally, additional research is needed in the two areas that do not have a correlate in the literature on manned aircraft – UAVs controlled from a moving platform and those where the operator relies on an external view. Research is needed to better understand the SD implications of conflicting and interfering cues that may arise from a moving platform. If control will be from a moving platform, simulators should include conflicting motion cues (Reed, 1977). Also, further work will be required to formulate SD countermeasures for external pilots since they have no correlate in manned

145


aviation and currently do not utilize any attitude symbology. The next step in addressing SD in UAV operations will likely involve increased automation of UAV operations and enhanced synthetic visual and auditory spatial environments which incorporate sensor imagery, thereby allowing operators to spatially orient in a natural fashion.

CONCLUSION Numerous mishaps have been reported where UAV operators have misjudged their attitude, motion, and/or position with respect to the ground, particularly during landings. We have provided a taxonomy based on control method, visual reference, and operator platform to help classify SD occurrences in UAVs. Visual illusions may be especially prevalent in manual control, egocentric display operations due to the absence of visual ambient cues. Direct visual external operators have experienced a number of new illusions, particularly a control reversal as the UAV flies towards and then away from the operator. If the operator is placed on a motion platform, such as a plane, ship, or tank, vestibular illusions and more serious visualvestibular-somatosensory mismatches can occur. In order to help counteract these illusions, it is imperative that effective simulators and training regiments be developed, new multi-modal display symbologies be examined, and research on UAV SD mechanisms be conducted.

NOTES 1. Excerpt from Air Force, Army, or Navy UAV mishap reports. These mishap reports are maintained at the respective military service safety centers: Air Force Safety Center (AFSC/SEFL), Army Safety Center (CSSC-O), and Navy Safety Center (NAVSAFCEN Code 144). The views expressed are those of the authors and do not reflect the official policy or position of the US Air Force, Department of Defense or the US Government.

REFERENCES Barry, C. L., & Zimet, E. (2001). UCAVs- technology, policy, and Operational challenges. Defense Horizons, 3(October), 1–8. Benson, A. (2003). Spatial disorientation – general aspects. In: J. Ernsting, A. Nicholson & D. Rainford (Eds), Aviation medicine (pp. 419–436). Oxford, UK: Butterworth Heinemann.

146


Crowley, J. (1991). Human factors of night vision devices: Anecdotes from the field concerning visual illusions and other effects. USAARL Report No. 91–15 Davenport, C. (2000). USAF SD experience: Air force safety center statistical Review. Proceedings of the Recent Trends in Spatial Disorientation Research Symposium. San Antonio, Texas. Dehart, R., & Davis, J. (2002). Fundamentals of aerospace medicine (3rd Ed.). Lippincott: Williams & Wilkins. DeVries, S. C., & Jansen, C. (2002). Situational awareness of UAV operators onboard of moving platforms. Proceedings of the international conference on human-computer interaction in aeronautics, (pp. 144–147) HCI-Aero. Dizio, P. (2000). Motion sickness and postural side effects of combined VE and ship motion exposure. 3rd Annual motion science consortium meeting, Arlington, VA. Fayaud, G. R. (2001). The airborne manned–unmanned system. Unmanned Systems, 19(4), 16–21. Gugerty, L., & Brooks, J. (2004). Reference-frame misalignment and cardinal direction judgments: Group differences and strategies. Journal of Experimental Psychology Applied, 10(2), 75–88. Landsman, G. (2005). Spatial disorientation in remotely piloted aircraft [Abstract]. Aviation, Space, and Environmental Medicine, 76(3), 240. Lawson, B., Graeber, D., Mead, A., & Muth, E. (2002). Signs and symptoms of human syndromes associated with synthetic experiences. In: K. Stanney (Ed.), The handbook of virtual environment technology (pp. 681–699). Mahwah, NJ: Lawrence Erlbaum Associates. Muth, E., & Lawson, B. (2003). Using flight simulators aboard ships: Human side effects of an optimal scenario with smooth seas. Aviation, Space, and Environmental Medicine, 74(5), 497–505. Previc, F. (2004). Visual orientation mechanisms. In: F. Previc, & W. Ercoline (Eds), Spatial disorientation in aviation. Progress in astronautics and aeronautics. (Vol 203). New York: AIAA. Previc, F., & Ercoline, W. (Ed.), (2004). Spatial disorientation in aviation. Progress in astronautics and aeronautics (Vol 203). New York: AIAA. Reed, L. (1977). Visual-proprioceptive cue conflicts in the control of remotely piloted vehicles. Brooks AFB, TX: Air Force Human Resources Laboratory AFHRL-TR-77-57. Sheridan, T. (1997). Supervisory control. In: G. Salvendy (Ed.), Handbook of human factors and ergonomics (pp. 1295–1327). New York: Wiley. Tvaryanas, A. P, Thompson, B. T., & Constable, S. H. (2005). U.S. Military unmanned aerial vehicle mishaps: Assessment of the role of human factors using Human Factor Analysis and Classification System. HSW-PE-BR-TR-2005–0001. Unmanned aerial vehicle reliability study. (2003). Office of the Secretary of Defense. Washington: Department of Defense; 2003. Retrieved January 16, 2005 from http:// www.acq.osd.mil/uav/ USAF Spatial Disorientation website. (2005). Retrieved 30 October 2005 from http://www. spatiald.wpafb.af.mil

THE ROV INTERFACE

All remotely operated vehicle (ROV) operators control and communicate with their remote vehicles in the same way: via the interface. This section examines fundamental control and design issues associated with ROV interfaces. The first chapter extends beyond the visual display issues found in many ROV interfaces. In her chapter, Gloria Calhoun also addresses multisensory interfaces such as tactile, spatial aural displays, and speech-based input. Moreover, she showed the limitations of simply adapting visual interfaces from direct control systems to remote control ones. The next chapter quantifies the difficulty of controlling a remote vehicle, particularly for operators with little to no experience. Paula Durlach highlights important interface issues for these novices in order to facilitate manual control of ROVs. Next, Bruce Hunn’s chapter brings the latest video display technologies into focus. He shows how synthesized information is more than adding extra information to the interface. Rather, it simplifies complex data into an integrated picture. This capability may be particularly well suited as more complex ROV systems are developed. Finally, Wendell Chun’s chapter takes a new approach with verbal ROV interfaces. His spatial dialog approach incorporates relative spatial information in the ROV dialog so that the communication is in terms of the operator’s position. Thus, through this dialog, a greater spatial understanding is shared. In sum, the interface is as indispensable to the ROV system as the human. These chapters pave the way ahead for improving the match between them.

147


148

11. MULTI-SENSORY INTERFACES FOR REMOTELY OPERATED VEHICLES Gloria L. Calhoun and Mark H. Draper In remotely operated vehicles (ROVs), one or more humans remotely manipulate a machine to perform a task. Examples include operating unmanned ground and air vehicles, land rovers on Mars, waste clean-up robots in nuclear facilities, etc. Typically, controllers of ROVs are limited to a reduced stream of sensory feedback delivered almost exclusively through the visual channel. It is hypothesized that operator situation awareness and performance may be improved through increased multi-sensory stimulation akin to that experienced in non-remote control (i.e., operator directly controlling a machine within the task environment). These improvements might stem from increasing the operator’s sense of ‘presence’ in the remote environment, and/or from increasing information throughput afforded by effective use of simultaneous multi-channel (e.g., audio and tactile feedback) stimulation. For instance, multi-sensory interfaces may enhance time-sharing (pertinent information from different sources comes through different sensory channels simultaneously) and improve attention capture (two or more senses transmit identical or supporting information simultaneously). Additionally, multi-sensory display techniques may result in a more intuitive presentation of information. This chapter will introduce multi-sensory interfaces that research has suggested as potential candidates for ROVs. Due to space limitations, the focus will be on technologies examined during the past few years at the Air Force Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 149–163 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07011-6

149

150

GLORIA L. CALHOUN AND MARK H. DRAPER

Research Laboratory (AFRL) at Wright-Patterson Air Force Base. These multi-sensory technologies were evaluated to assess their utility in improving control operations performed by the pilot and the sensor operator (SO) for a teleoperated ROV application. Specifically, AFRL’s research on visual, tactile, and spatial aural displays, as well as speech-based input will be reviewed.

RESEARCH SIMULATOR The ROV ground control simulator (Fig. 1) used in this multi-sensory research consists of two workstations: pilot and SO. At the left workstation, the pilot controls ROV flight (via stick-and-throttle inputs as well as invoking auto-holds), manages subsystems, and handles external communications. From the right workstation, the SO is responsible for locating and identifying points of interest on the ground by controlling cameras mounted on the ROV. Each station has an upper and a head-level 1700 color CRT display, as well as two 1000 head-down color displays. The upper CRT of both stations displays a ‘God’s Eye’ area map (fixed, north up) with overlaid symbology identifying current ROV location, flight waypoints, and current

Fig. 1.

ROV Ground Control Simulator.

Multi-Sensory Interfaces for ROVs

151

sensor footprint. The head-level CRT (i.e., ‘‘camera display’’) displays simulated video imagery from cameras mounted on the ROV. Head-up display (HUD) symbology is overlaid on the pilot’s camera display and sensor specific data are overlaid on the SO’s camera display. The head-down displays present subsystem and communication information as well as command menus. The simulation is hosted on four dual-Pentium PCs. The control sticks are from Measurement Systems Inc. and the throttle assemblies were manufactured in-house.

VISUAL DISPLAYS Since operators in teleoperated systems receive the majority of information through the visual channel and since the visual channel offers tremendous bandwidth potential, improvements in the visual displays of information were considered. Specifically, two areas have been addressed at AFRL: (1) use of head-coupled head-mounted displays (HMD), and (2) use of a synthetic visual overlay that augments the real-time video imagery to increase situation awareness. Helmet Mounted Information Display Research Head-coupled helmet mounted displays have been found to enhance widearea search tasks (Geiselman & Osgood, 1994). One potential advantage of head-coupled control versus manual control over one’s viewpoint is the addition of ecologically relevant proprioceptive cues. Some studies have suggested that head-coupled configurations facilitate awareness of areas already searched, thereby potentially reducing the re-scanning of those same areas (Pausch, Proffitt, & Williams, 1997). It was hypothesized that a HMD would enhance the SO’s wide-area searches and overall spatial orientation while reserving the higher resolution fixed console display for fine inspection/discrimination tasks. Two studies were conducted to evaluate the usefulness of HMDs for SO tasks. The overall approach was to compare the utility of a manual joystick and fixed console display to that of various head-coupled HMD configurations for search tasks. A Kaiser Electro-Optics ProViewTM50 ST fullcolor LCD HMD and an Ascension Flock of Birdss head tracking system were used for the first evaluation. Besides the Baseline Condition (manual rate-controlled joystick/stationary CRT console display), three different

152


HMD configurations were evaluated (the latter two designed to ease neck strain during rearward searches): Position HMD: 1.0 gain zero-order head-coupled HMD control. +Gain HMD: 1.5 gain, zero-order head-coupled HMD control. Camera moved 1.5 times the operator’s yaw angular head movement. Hybrid: Position HMD configuration with simultaneous manual control available via the joystick. Subjects controlled the gimbaled camera position with either the right-hand joystick or head-coupled HMD (depending on trial configuration) and the camera zoom factor with the left-hand throttle stick. SO tasks included a front-field road search task and a rear-field off-road ground search. Both tasks involved a large area search task (conducted by the HMD when available), followed by a target discrimination task (which always required the fixed console display). Thus, the HMD conditions necessitated a switch to the fixed display console. Search time was significantly shorter with the Baseline than any of the HMD conditions for front-field search and a similar trend was revealed for rear-field search (Fig. 2). Lower workload and fewer simulation sickness symptoms were reported with the Baseline Condition. Thus, the results failed to show a benefit for head-coupled HMD-based configurations. Another study (Draper, Ruff, Fontejon, & Napier, 2002) was conducted to specifically evaluate the utility of a head-coupled HMD for the SO’s conduct of a continual 3601 search around the current ROV position for points of interest (without the additional switch to the console display that was required in the first evaluation). Six camera control/display

Fig. 2.

Mean Search Time with Each Camera Configuration for Front-and RearField Searches.


153

configurations were evaluated, two involving a stationary display (manual joystick with 10 or 251/s gain) and four involving a HMD (Sony Glasstron full-color LCD HMD (LDI-100B) and an Ascension Flock of Birdss headtracker). The results again indicated that display method was a critical factor – fewer points of interest were found with the HMD than with the stationary CRT/manual joystick. Performance with the head-coupled control configurations was also generally poorer than manual joystick control in respect to number of duplicate designations, and ratings on workload and situation awareness. These results suggest that the utility of head-coupled HMDs in the performance of large-area teleoperated search tasks is not obvious in the experimental paradigms employed to date and requires further investigation. Synthetic Vision Overlay to Augment Real-time Video Imagery ROV video imagery is used by pilots to verify clear path for runway operations, scan for obstacles, and identify landmarks. Sensor operators use video imagery to conduct intelligence, surveillance, and reconnaissance activities. However, video imagery quality can be compromised by narrow camera field-of-view, datalink degradations, system lag, poor environmental conditions (e.g., dawn/dusk/night, adverse weather, clouds), bandwidth limitations, or a cluttered visual scene (e.g., urban areas or mountainous terrain). If imagery interpretation can be enhanced and made more robust, operator performance is expected to improve. Synthetic vision technology can potentially ameliorate negative video characteristics and enhance operator imagery interpretation. Spatially relevant information is constructed from databases (e.g., terrain elevation, maps, photo-imagery) as well as numerous near real-time updates via networked communication with other information sources and overlaid conformal onto the dynamic camera video image display. AFRL, in collaboration with Rapid Imaging Software, Inc. (RIS), conducted a usability evaluation with ROV operators to identify candidate synthetic vision overlay concepts for teleoperated applications (Draper, Nelson, Abernathy, & Calhoun, 2004). Some concepts are illustrated in Figs. 3 and 4. Evaluation of these concepts through RIS’s SmartCam3DTM technologies is now underway in high-fidelity tests. It is also plausible that a synthetic vision system can include symbology that supports distributive collaborative communication. An operator can designate a point of interest on the display via a cursor/trackball, resulting in a vector line or ‘‘telestrator-type’’ illustration presented on the display of

154

Fig. 3.


Synthetic Vision Symbology Added to Standard ROV Gimbal Video Imagery. (Symbology Marking Areas of Interest, Runway, etc.).

Fig. 4. Picture-in-Picture Concept, with Real Video Imagery Surrounded by Synthetic-Generated Terrain Imagery/Symbology. Affords Virtual Expansion of the Available Sensor Field-of-View Well beyond the Physical Limits of the Camera.


155

other operators or members of the networked environment. For teleoperated camera control, the telestrator symbology could indicate the direction and angular distance the camera’s line-of-sight should traverse in order to view the point of interest. In a study exploring this concept, the results indicated that the telestrator expedited transfer of location information between operators and reduced the need for verbal communication which freed the audio channel for other tasks (Draper, Geiselman, Lu, Roe, & Haas, 2000a).

TACTILE DISPLAYS Tactile displays transmit information through the skin by varying one or more dimensions of vibratory stimulation (locus, amplitude, frequency, duration, and spatial/temporal separation). To date, tactile display applications have been limited and few design guidelines exist (van Erp, 2002). Tactile presentation of information is, however, a promising display technology (van Veen & van Erp, 2003). Tactile messages are silent and not intrusive and tactile sensors are always ready to receive information. Tactile displays are ‘omnidirectional’ in that the operator does not have to be looking at a particular location to be alerted, and they can also be designed to convey spatial information. Additionally, tactile information can be received simultaneously to visual and aural information. AFRL has evaluated two applications of tactile displays: to cue the presence of turbulence and to notify operators of a serious system malfunction. Turbulence Tactile Cue External disturbances such as wind turbulence can be detrimental to safe and effective ROV control. Unfortunately, the physical separation of the crew from the vehicle makes detection of sudden turbulence onset very difficult, often solely indicated by unexpected perturbation of the video imagery transmitted from a camera mounted on the vehicle. A study exploring turbulence cues (Draper, Ruff, Repperger, & Lu, 2000b) employed a tactile display consisting of a 1-s, low-gain, high-frequency perturbation of the control stick in the axis-direction and scaled-ratio magnitude of the turbulence event (using an Immersion Corporation 2000 Force Feedback Joystick). This tactile display, simultaneous with visual cues (perturbation of nose camera imagery) was compared to three other conditions (Visual cues alone [Baseline], Visual with Aural Cue [1-s pure tone], and Visual with both

156


Fig. 5. By Cue Combination, Mean Landing Error (top) and Percent Accuracy in Detection of Turbulence Direction (bottom). V, Visual; VT, Visual & Tactile; VA, Visual & Aural; VTA, Visual, Tactile, & Aural.

Aural and Tactile Cues) under two levels of video image visibility and two levels of turbulence severity. The results showed that the pilots landed with less error in the two conditions that included a tactile cue (Fig. 5, top). When queried regarding the primary rotational displacement (pitch or roll) of the ROV resulting from a randomly driven turbulence event, controller responses were significantly more accurate when tactile feedback had been present (Fig. 5, bottom). It should be noted that the operators commented that only a slight tactile feedback is required (and desired) to alert turbulence onset.


157

System Alert Tactile Cue For a series of evaluations exploring tactile alert cues, small electromechanical vibrating tactors were mounted in elastic bands over one or both inner wrists (Fig. 6). It was envisioned that the activation of tactile cues would be reserved for critical warning events, rather than be frequently activated, to prevent habituation. One study showed that tactile stimulation, when presented in concert with visual and aural alerts, did not aid or degrade performance in a teleoperation simulation (Calhoun, Draper, Ruff, Fontejon, & Guilfoos, 2003). This study also suggested that the non-visual alerts, tactile and aural, were equally compelling and that tactile alerts can substitute for aural alerts as a redundant cue to visual alerts. It was then hypothesized that a tactile redundant cue could result in improved performance as compared to an aural redundant cue in a high auditory-load environment. However, the results of a follow-on experiment indicated no differences between the two non-visual alert cues (Calhoun, Fontejon, Draper, Ruff, & Guilfoos, 2004). Thus, the aural and tactile cues, employed as redundant cues, were equivalent in improving reaction time. Perhaps tactile cues would prove more effective than aural cues in noisy environments when each are employed as sole alerts, rather than redundantly with a noise-robust visual alert. Another postulation was that tactile

Fig. 6.

Tactors Used in Vibration Alerts.

158


alerts may be especially useful in noise when long vigilance durations were required. However, in a follow-on study, the aural and tactile redundant alerts were equally compelling regardless of the auditory load level and temporal placement of the cue within a 30 min trial (Calhoun, Ruff, Draper, & Guilfoos, 2005). In sum, performance results from this series of studies demonstrated that tactile cueing was an effective redundant alert, a suitable substitute for an aural redundant cue, and was accepted by the operators – the tactile stimulation was not uncomfortable and did not distract attention in a multi-task environment. Even though a performance advantage was not demonstrated for tactile alerts over aural alerts in a high noise environment, many of the controllers provided comments indicating a benefit or definite preference for the tactile alerts: ‘‘In high chatter, tactile is even better at drawing attention’’ and ‘‘this is the only thing I was ‘feeling’. I was hearing many things.’’ Thus, tactile displays may have a beneficial role in teleoperated systems that was not revealed by the experimental paradigms employed to date.

DIRECT VOICE INPUT In most teleoperated systems, functions are controlled via keyboard buttons or mouse inputs that designate desired functions and call up menu pages with options. For many tasks, numerous button selections are required simply to access the appropriate menu page. By integrating voice recognition capabilities into the control station, simple voice commands can substitute for many of these button selections, enabling operators of teleoperated systems to maintain their hands on the controllers and keep their heads up, directed toward the camera display and critical symbology. An experiment was conducted to examine the utility of conventional manual input versus speech input for a variety of tasks performed by teleoperated ROV operators (Williamson, Draper, Calhoun, & Barry, 2005). Speech input was achieved with Nuance v6.0 (Nuance Communications, Inc.), a speaker-independent continuous speech recognition system. Due to the inherent advantages of voice control, many voice commands functioned as ‘‘macros’’ and effectively replaced numerous sequential button presses. To activate the speech recognition system, a ‘‘push-to-talk’’ button on the joystick was utilized. Operators used checklist books that detailed the button presses (manual input) and voice commands (speech input) required for data entry completion. Visual feedback of each spoken command was presented on the camera display. Operators performed a continuous flight/


Fig. 7.

159

Mean Task Completion Time with Manual and Speech Input for Each Task Type.

navigation control task while completing eight different data entry task types with each input modality. Results showed that speech input was significantly better than manual input in terms of task completion time, task accuracy, flight/navigation measures, and pilot subjective ratings. Average time savings for data entry tasks ranged from 3.14 s to 21.43 s. Across tasks, data entry time was reduced by approximately 40% with speech input. This performance advantage, coupled with the other advantages of speech input – head-up, handfree control that facilitates flight/navigation, improved data entry efficiency through intuitive voice macros, reduced errors, and natural intuitive control input – warrants additional research to confirm that speech input is still beneficial in operational control station auditory environments and does not conflict with intercom operations and intra-crew communications (Fig. 7).

SPATIAL AURAL DISPLAYS A high level of controller situation awareness is required in teleoperated systems. At present, this awareness is formed and maintained by means of

160


visual displays and monaural auditory warnings which fail to leverage the natural spatial auditory processing capabilities of humans. That is, the ability of operators to determine the location of a sound source and monitor events at multiple locations simultaneously has not been fully exploited. Spatial auditory display technologies take advantage of the properties of the binaural auditory system by recreating and presenting to an operator the spatial information that would naturally be available in a ‘‘real-world’’ listening environment. Research on binaural hearing suggests that humans use the auditory modality for both the development and maintenance of situation awareness in natural environments. It is reasonable to assume, then, that this capacity can be leveraged in teleoperated systems. AFRL has reviewed spatial audio display concepts to identify potential teleoperated applications to explore in future research studies (Simpson, Bolia, & Draper, 2004). These concepts are described below. Much of the research done on monitoring multiple auditory channels has involved the use of spatial audio to improve speech intelligibility and reduce workload for operators listening to multiple radios or intercom channels (Bolia & Nelson, 2003). This could be useful to operators who often engage in verbal communications with a variety of distributed team members, not only for its potential to improve communications effectiveness, but also because spatial awareness of talker location would provide the operators with an additional cue to the identity of the talker based on a predetermined mapping of communications channels. Spatial auditory displays have also been shown to reduce acquisition/ identification times in a search task by a factor of 2–5 in very simple visual scenes; much greater benefits occurred as the complexity of the visual scene increased (Bolia & Nelson, 2003). As a result, spatial audio cueing might be especially useful to assist operators of teleoperated systems in finding items of interest in the spatial environment. A spatial audio display can also support the temporal aspect of situation awareness (Simpson et al., 2004). Auditory motion perception can be used to demonstrate trajectories of elements in the environment. This is especially compelling when used in conjunction with analogous visual displays for predicting future states, allowing the operator to ‘‘control several seconds ahead of the vehicle.’’

SUMMARY Table 1 shows a summary of the multi-sensory interfaces evaluated by AFRL in a ROV task environment. For each interface, the candidate ap-

161


Table 1.

Assessment of Multi-Sensory Interfaces for ROV Utility.

Interface Visual Head mounted displays

Synthetic augmented view

Tactile Force-feedback control stick

Wrist-worn vibrating tactor

Application

Enhance wide-area camera search

Overlay information on camera video

Cue presence of turbulence

Cue system malfunction

Study Results

Potential Utility

Increased search time Increased workload More sickness symptoms Increased situation awareness Reduced search time Reduced workload

To be determined

Improved landing accuracy Increased situation awareness Reduced workload No performance advantage, but subjects preferred Equal to aural redundant cues

Strong

Inputs faster than manual Reduced errors

Strong

Strong

To be determined

Speech-based input Input and call up data

plication is listed as well as the key results. An assessment of the maturity of the technology for application to ROV applications is also provided. This assessment is solely the viewpoint of the authors and based on research conducted at AFRL. It may very well be that each interface has ROV utility that was not revealed by the experimental paradigms employed at AFRL to date. Also, technological advancements in the interfaces may accelerate their ROV utility. In sum, AFRL research has demonstrated the value of several multisensory interfaces for improving ROV operations. It has also shown the limitations of some interfaces and that technology proven useful for other direct control systems may not be useful for teleoperated systems. This

162


illustrates the importance of evaluating candidate interface technologies in an environment representative of the ROV system under consideration.

REFERENCES Bolia, R. S., & Nelson, W. T. (2003). Spatial audio displays for target acquisition and speech communications. In: L. J. Hettinger & M. W. Haas (Eds), Virtual and adaptive environments: Applications, implications, and human performance issues (pp. 187–197). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Calhoun, G., Draper, M., Ruff, H., Fontejon, J., & Guilfoos, B. (2003). Evaluation of tactile alerts for control station operation. In: Proceedings of the human factors and ergonomics society, Human Factors and Ergonomics Society, Santa Monica, CA (pp. 2118–2122). Calhoun, G. L., Fontejon, J. V., Draper, M. H., Ruff, H. A., & Guilfoos, B. (2004). Tactile versus aural redundant alert cues for UAV control applications. In: Proceedings of the human factors and ergonomics society, Human Factors and Ergonomics Society, Santa Monica, CA (pp. 137–141). Calhoun, G. L., Ruff, H. A., Draper, M. H., & Guilfoos, B. (2005). Tactile and Aural Alerts in High-Auditory Load UAV Control Environments. In: Proceedings of the human factors and ergonomics society, Human Factors and Ergonomics Society, Santa Monica, CA (pp. 145–149). Draper, M. H., Geiselman, E. E., Lu, L. G., Roe, M. M., & Haas, M. W. (2000a). Display concepts supporting crew communication of target location in unmanned air vehicles. In: Proceedings of the human factors and ergonomics society, Human Factors and Ergonomics Society, Santa Monica, CA (pp. 385–388). Draper, M. H., Ruff, H. A., Repperger, D. W., & Lu, L. G. (2000b). Multi-sensory interface concepts supporting turbulence detection by RPV controllers. In: Proceedings of the human performance, situation awareness and automation conference, Human Performance, Situation Awareness and Automation, Savannah, GA (pp. 107–112). Draper, M. H., Nelson, W. T., Abernathy, M. F., & Calhoun, G. L. (2004). Synthetic vision overlay for improving RPV operations. In: Proceedings of the association for unmanned vehicle systems international (AUVSI), Association for Unmanned Vehicle Systems International, Anaheim, CA. Draper, M. H., Ruff, H. A., Fontejon, J. V., & Napier, S. (2002). The effects of head-coupled control and head-mounted displays (HMDs) on large-area search tasks. In: Proceedings of the human factors and ergonomics society, Human Factors and Ergonomics Society, Santa Monica, CA (pp. 2139–2143). Geiselman, E. E., & Osgood, R. K. (1994). Utility of off-boresight helmet mounted symbology during a high angle airborne target acquisition task. Proceedings of the SPIE conference helmet & head-mounted displays & symbology design requirements, 2218, 328–338. Pausch, R., Proffitt, D., & Williams, G. (1997). Quantifying immersion in virtual reality. Proceedings of SIGGRAPH 97, computer graphics proceedings, annual conference series, ACM SIGGRAPH, Los Angeles, CA (pp. 13–18) August. Simpson, B. D., Bolia, R. S., & Draper, M. H. (2004). Spatial audio display concepts supporting situation awareness for operators of unmanned aerial vehicles. Proceedings of the human performance, situation awareness and automation conference (pp. 61–65). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.


163

van Erp, J. B. F. (2002). Guideline for the use of vibrotactile displays in human computer interaction. 2002 EuroHaptics. Edinburgh, UK: University of Edinburgh. Manuscript available at http://www.eurohaptics.vision.ee.ethz.ch/2002/vanerp.pdf van Veen, H., & van Erp, J. (2003). Tactile information presentation in the cockpit. Proceedings of the haptic human-computer interaction workshop, University of Glasgow (pp. 50–53). Glasgow, Scotland, UK: University of Glasgow. Williamson, D. T., Draper, M. H., Calhoun, G. L., & Barry, T. P. (2005). Commercial speech recognition technology in the military domain: Results of two recent research efforts. International Journal of Speech Technology, 8, 9–16.


164

12. EVALUATION OF A TOUCH SCREEN-BASED OPERATOR CONTROL INTERFACE FOR TRAINING AND REMOTE OPERATION OF A SIMULATED MICRO-UNINHABITED AERIAL VEHICLE Paula J. Durlach, John L. Neumann and Laticia D. Bowens Uninhabited systems are envisioned to be a key part of the Army’s Future Force. Part of that vision includes a backpackable micro aerial vehicle (MAV), controlled by an operator without any traditional pilot experience. Because of its size and its availability to light infantry, the MAV could provide unprecedented situation awareness at the platoon level. The Defense Advanced Research Projects Agency (DARPA) is currently conducting an Advanced Concept Technology Demonstration (ACTD) program to develop the technologies that will be required for an operational MAV. The program has focused on flight control, power and propulsion, navigation, and communications for a vertical-take off and landing vehicle (see Clarke, 2004) . In the spring of 2004, experimenters from the U.S. Army Research Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 165–177 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07012-8

165

166

PAULA J. DURLACH ET AL.

Institute’s (ARI) Simulator Systems Research Unit began evaluating the trainability of a prototype operator control unit (OCU) in simulation. The prototype OCU, which was developed by Northrop–Grumman for the DARPA MAV ACTD, took a unique approach to vehicle teleoperation (manual control). This prototype OCU allowed manual control by touching a spot on the sensor image itself. Hypothetically, this should be easier than standard flight controls for an operator lacking air-pilot experience. By touching on a spot in the sensor image, the operator should be able to direct the vehicle relative to what is seen in the sensor image. Digital command buttons supplemented this form of manual control. These included up/down buttons to control altitude, left/right buttons to control horizontal movement, and rotate buttons to control orientation. Besides operating in manual mode, the simulated MAV could also operate in autonomous mode. This involved creating and running missions consisting of waypoints that were specified by selecting points on the terrain map. Once the mission was created, it needed to be saved, sent to the simulated MAV, and then started. Transitions between autonomous and manual modes were also possible.

Fig. 1.

Illustration of the OCU in Map Mode. The Sensory Image from the Forward Camera is Shown in the Lower Right Corner.

Evaluation of a Touch Screen-Based Operator Control Interface

167

The OCU could operate either in map mode or full-screen video mode. In map mode, the terrain map – with superimposed mission route (if preloaded), MAV location, and OCU location – dominated the display. However, there was also an inset window displaying video imagery sent by the MAV. Fig. 1 illustrates this. The MAV was equipped with forward-looking and downward-looking cameras (both fixed). Only one of these views could be seen at any given time. The video sensor image could be enlarged by touching it. This would switch the OCU into full-screen video mode. In this mode, the map disappeared, and the sensor image covered most of the screen. Command buttons would also appear around the edges of the image in full-screen video mode, and these allowed for operational control (e.g., up/down), as well as switching back to the map view (Fig. 2). The affiliated simulation was based on Riptide software, developed at the NASA Ames Research Center. The simulation included a 6-degree-of-freedom flight model and a synthetic environment. The synthetic environment modeled the McKenna urban terrain site, Fort Benning, and fed the appropriate sensor imagery back to the OCU. The synthetic environment was static. That is, it could not be altered or display any moving entities besides the MAV.

Fig. 2.

Illustration of the OCU in Video Mode. The Arrow in the Center Indicates North.

168


STUDY 1: PARTICIPANTS AND PROCEDURE The goal of our initial study was to assess the usability of the prototype OCU and establish associated training issues. Seven participants completed self-paced training, guided by a training manual produced by ARI. Training was divided into three modules: (1) introduction and autonomous control, (2) manual control, and (3) creating and editing autonomous missions. Primary focus was on how to use the OCU to fly the MAV. Modules did not include elements such as fueling, setup, or tactics. A facilitator was present at all times to observe user interaction with the system and to manage the software. Data captured included time to complete each training module and related practical exercises, user feedback on questionnaires, and a written test on training content. Participants had either graduate-level experience in human factors psychology, prior military experience, or both. They were, therefore, able to provide valuable insights while they learned to operate the simulated MAV.

STUDY 1: RESULTS Completion of training and questionnaires took between 3 and 5 hours, and was spread over one to three sessions, depending on the participant. Fig. 1 illustrates the time taken by participants to complete each of the three training modules. From the data, not only did we generate recommendations for improving this specific OCU, but also ones applicable to OCUs in general. These lessons included: the need to provide salient status information and timely feedback, the need to include alerts (e.g., for altitude), and the need to support users through a required sequence of steps (Fig. 3). In general, OCU design needs to incorporate salient and timely indicators to maintain the operator’s situation awareness. In the examined system, important status information was either lacking (velocity) or present but lacking salience (altitude). In their comments, four of the seven trainees noted their desire for an air speed indicator. Also, four commented that the altitude indicator was not salient enough. Lack of awareness of altitude was responsible for most of the MAV ‘‘crashes’’ suffered during training. We concluded that if the MAV does not have autonomous crash avoidance software, then the OCU should produce an alert when it is within a certain minimum height above the ground (e.g., 3–6 m). Another particular problem for the OCU was lack of salient feedback from operator inputs. All inputs were made with a stylus on a touch screen;

169


Introduction & Autonomous Control Manual Control Creating & Editing Autonomous Missions Number of Participants

5 4 3 2 1 0 0-15

16-30

3 1-45

46-60

61-75

76-90

91105

106120

Time to Complete Training (Minutes)

Fig. 3.

Distributions of Time Taken by Participants to Complete Each Training Module During the Trainability/Usability Study.

however, it was often unclear to trainees whether their touches had registered. This uncertainty was exacerbated by the delay that occurred between an input and an observable MAV reaction. Moreover, sometimes actual registered inputs failed to be successfully accepted by the MAV. That is, the OCU registered the command, but subsequent OCU-to-MAV communication failed. Operators had difficulty determining the difference between delayed MAV reactions and input or communications failures. They often responded by repeating the input, which produced undesirable effects in the case when input had registered but response was delayed. We recommend that some form of immediate feedback follow all inputs, even if the response of the system is delayed. For example, inputs on a touch screen should produce a noticeable change (e.g., visual or auditory cue), so the operator knows the screen has registered the command. If the command involves communication with the MAV, the operator also requires a salient signal to let him/her know whether the communication was successful. With regard to creating and saving autonomous missions, the simulation we studied modeled the communications links in an actual MAV system. After an operator created and saved a mission, in order to run it, the mission parameters had to be sent to and accepted by the MAV. To actually start the

170


mission, another command had to be sent from the OCU to the MAV. Feedback on these transmissions was provided by the display, but the messages were not very salient. The most salient signal that the mission was beginning was actual movement of the MAV icon on the map screen or changes in the sensor imagery. The trainees were informed explicitly about these steps, yet they often failed to complete them all. We repeatedly observed a trainee waiting for the mission to start after merely creating and saving it, or after saving and sending it. Although all trainees completed the sequence required to start a mission several times during their session, when asked to list the sequence on the skill test subsequent to training, only four of the seven trainees responded accurately. We recommend that the OCU support the operator during the most probable sequences of actions. For example, after saving a mission the operator could be prompted, ‘‘Do you want to send this mission to the MAV?’’ and after sending a mission, ‘‘Do you want to start this mission now?’’ Manual Control The Manual Control training module was the most time consuming part of the training session. Five of the seven participants rated the Manual Control training module as frustrating, whereas only one participant rated the other training modules frustrating. Five of the seven participants rated the Manual Control module as difficult, whereas none of them rated the other training modules difficult. This frustration was partly due to the decoupling of perception and action that occurs when maneuvering a remote vehicle on the basis of video feed from a fixed camera (Peruch & Mestre, 1999; Woods, Tittle, Feil, & Roesler, 2004). A human moving through space has several sources of information besides straight-ahead-vision, and these may be absent or distorted when navigating through video input. This may interfere with judgment of scale, depth, and velocity. In addition, people moving through space make anticipatory head and eye movements toward the position they are approaching; they don’t just look straight ahead (Rybarczyk, Galerne, Hoppenot, Colle, & Mestre, 2001). Another reason for frustration was the complex rules that governed the translation of a single input into a MAV response. Image content itself had no influence on how the MAV responded. Each touch on the sensor image moved the MAV forward into the image and affected its heading. Touching on the left or right side of the sensor image rotated MAV heading by an angle proportional to the distance from the center. Touching below the center reduced altitude whereas touching above the center maintained


171

altitude. The actual distance (and speed) that the MAV moved was a function of its current altitude. The higher the current altitude, the faster and further the MAV moved in response to an image touch. No explicit feedback was provided except for heading information via a north-pointing arrow. Only the second author, who spent substantial time practicing and designing missions, ever gained a good ‘‘feel’’ for how the MAV would respond to sensor-image inputs. We suggest that the rules used to implement manual control via image touches were too complex for trainees to grasp without hours of practice, especially without adequate feedback on speed or heading. Simpler rules must be used to translate inputs into MAV behavior. Instead of one input moving the MAV in three dimensions, it would be better to translate a single touch to up/down and left/right as a function of where the touch is made on the screen. Rotation could be left to a separate command button. In addition, the MAV should either travel a set distance per input, or travel indefinitely at a set speed and altitude until a new command is issued. Finally, the operator needs status information on speed and bearing. Any operator, especially a novice, needs a good understanding of how a robotic entity is going to respond to manual control inputs. It is too demanding to expect trainees to understand how a single input would affect the MAV in three dimensions; especially if in addition, speed is a function of altitude. Three other factors added to the frustration experienced during manual control: (1) the MAV’s physical movement dynamics, (2) the delay between issuing a command and observing a response, and (3) poor feedback and status information. Because of the simulated physical dynamics of the MAV, it behaved in unexpected ways. For example, it had a tendency to sway like a pendulum after completing a lateral movement; this effect was accentuated at higher speeds. Undesired MAV behaviors were especially likely to occur when the operators were uncertain if their inputs had registered. The delay between operator input and observable reaction, combined with poor feedback on whether the input had registered, resulted in operators making repeated rapid inputs. Rapid inputs then resulted in either a system overload – where nothing happened – or over-steering, which generated unwanted and erratic MAV behaviors.

STUDY 2: PARTICIPANTS AND PROCEDURE Study 2 had two main aims. The first was to examine manual control skills in more detail. As such, we designed four flight missions intended to require

172


increasingly precise manual control. The data collected would inform us as to whether operators indeed found these missions increasingly difficult. The missions are outlined in Table 1. The second aim of Study 2 was to explore how the available sensor imagery would affect mission performance. Although not intended for use in this way, the prototype system allowed us to present three sensor windows simultaneously (instead of just one, as in Study 1), and to manipulate their content. The three-window configuration used in Study 2 is illustrated in Fig. 4. The main view (21.6 10.9 cm) displayed the view from a camera angled 151 from horizontal. Two smaller windows were situated below this. One (10.8 3.2 cm) displayed a satellite view, which was a downward view of the environment taken from a point 500 ft above the MAV. The content of the other window (10.8 3.2 cm) varied by condition (manipulated between participants). This window displayed a view from a MAV-mounted camera angled at 30, 60, or 901 from horizontal. This manipulated view always appeared contralateral to the operator’s dominant hand. This made it less likely that the participant would obstruct the image while making touch inputs on the larger window above it. We hypothesized that imagery provided in this third window might assist operators with additional information; and wished to determine which camera angle would provide the most assistance. Table 1.

Synopsis of the Four Missions Used in Study 2.

Mission 1

From a hover altitude of 150 ft. above ground (AG) the pilot was required to fly the MAV to the town and obtain two specified buildings in both camera views. The pilot then flew the MAV to a designated landing zone (LZ), which was large and should have provided little challenge for landing

Mission 2

From a hover altitude of 39 ft (AG) the pilot was required to fly to and over a wooded area. Over the woods, the pilot was required to reach an altitude of 100 ft (or more), and then proceed to the designated LZ. The LZ was marked and was about half the width of the LZ used in Mission 1

Mission 3

From a hover point directly over a road, the pilot was instructed to fly directly above it and land near a ‘T’ intersection located at the end of the road. This mission required accurate MAV maneuvering. The LZ was much narrower than previous missions. There were no obstacles

Mission 4

From a starting altitude of 100 ft (AG) the pilot was required to fly to the town and land on the rooftop of a three-story building. Information was given on the approximate height of the rooftop ledge (45 ft). This mission required the pilot to maintain total situation awareness of obstacles, heading, altitude, speed, and to make a precision landing


173

Fig. 4. Illustration of the Video Mode Screen Layout as it Appeared in Study 2. All Participants Saw Sensory Imagery Taken from the MAV at 151 from Horizontal in the Main (Large) Window. A Downward Satellite View Taken from 500 ft Above Ground was Shown in One of the Smaller Windows (Lower Right). In that View, the MAV Position was Shown by an Icon (White Doughnut). The Remaining Window Displayed the Manipulated View, an Image Shot from the MAV at 30, 60, or 901 from Horizontal. The Imagery Used in this Illustration was Generated by Google Earth (not the Actual Simulation Used in the Study).

Thirty-three participants were recruited from the Orlando community for an approximately 90 min session. Participants were provided with a training guide, which they progressed through interactively with the experimenter. Initial training took approximately 30 min and was followed by a 5 min freeflight period. Only manual control was covered. Participants then completed a written test to confirm at least a basic level of understanding. The experimenter reviewed any missed items. Subsequently, each participant completed four separate manual flight missions. Before performing each mission, the participant was briefed and watched the MAV perform the mission in autonomous mode. The experimenter then placed the MAV in its starting position and gave control to the participant. Table 1 gives a synopsis of each mission and the skills required for each. Data were collected on time to complete each mission, as well as the number of physical interactions with the OCU (screen touches). The experimenter also rated operator performance (SME rating) on a scale of 1 (terrible) to 10 (excellent) at the

174


conclusion of each mission. Workload measures were collected using the NASA–TLX, and after the four missions were finished, the participant completed a demographic survey and a usability questionnaire.

STUDY 2: RESULTS In general, mission requirements had a greater impact than the camera angle on mission performance. We had hypothesized that the camera angle would have a greater effect on landing accuracy than it did. In retrospect, it is possible that the satellite view provided all conditions with a good enough downward image for landing, and for maneuvering above the road (Mission 3). According to a mixed analysis of variance, with Condition as the betweensubject factor and Mission as the within-subject factor, camera angle affected mission completion time significantly only for Mission 1, F(6, 90) ¼ 2.69, po0.05. This mission involved capturing the image of two different buildings in both the main camera view and the manipulated camera view. Typically, participants captured a target in the main window and then the manipulated window. In the 301 condition (Condition 30), the image moved seamlessly from one window to the other, pretty much on any heading. For the other camera angle conditions, the operators had to maneuver the MAV closer to the targets in order to capture them in the manipulated window. In other words, the operators in Condition 30 were able to acquire the targets in both windows from a further distance. Consequently, this condition yielded shorter Mission 1 completion times (mean 4.6 min) compared with the 601 and 901 conditions. Means for the 601 and 901 conditions were 6.5 and 7.3 min, respectively. The mission requirements influenced time to complete mission, touch rate, workload, and SME rating. This was hardly surprising as the missions were designed to be increasingly difficult. Participants rated workload higher for Missions 3 and 4 (57.4 and 59.6, respectively) compared with Missions 1 and 2 (41.2 and 43.2, respectively), F(3, 90) ¼ 20.78, po0.01. Only 1 out of the 33 participants crashed the MAV during Missions 1 and 2, whereas nine crashed during Missions 3 and 4; the difference in these proportions was significant, p o0.01. In fact, all of these nine crashes were during Mission 4. During Mission 4, 5 of the 12 participants in Condition 30 crashed on their first attempt. The analogous numbers for Conditions 60 and 90 were two (out of 12) and three (out of 12), respectively. While these differences failed to be statistically significant, experimenter observations did suggest a


175

clear rationale for this pattern. People in Condition 60 had the best view of the building and the rooftop as they approached it. This gave them adequate time to slow the MAV to a hover directly above the building, and then land. People in the other two conditions appeared to have more difficulty judging depth (or distance from the building), and so tended to either overshoot (90) or undershoot (30) the rooftop. They then had to make further maneuvers to get back above the building, and it was during these further maneuvers that crashes tended to occur. Moreover, people in Condition 30 were at a disadvantage because once they got within a relatively close range of the building the main view (151) and 301 views provided little aid in locating the rooftop. Mission 3 also received a high workload rating relative to Missions 1 or 2. Although it took participants the least amount of time to complete, the requirements were very stringent (fly directly above the road). A single poorly chosen input could take the MAV off course. Participants could see continuously throughout the mission whether they were over the road or not. The experimenter could see this as well; mean SME rating for Mission 3 was the lowest of the four missions. To examine the relationships among the dependent variables, comprehensive scores were computed by averaging the scores for each mission. A comprehensive usability score was computed by averaging all the usability questions (scale was 1–10 with lower scores favorable and higher scores unfavorable). Table 2 shows the Spearman correlations among the variables. Workload, usability rating, and SME rating were significantly intercorrelated in ways that seem sensible. A high workload score tended to predict an unfavorable usability score and a lower SME rating. A low workload was associated with a positive usability score and a higher SME rating. SME Table 2.

Spearman Correlations among the Comprehensive Scores for the Dependent Variables. Workload

Workload Usability SME rating Completion time Touch rate

Usability 0.598

0.598 0.480 0.220 0.338

0.448 0.253 0.123

SME Rating 0.480 0.448 0.51 0.024

Completion Time 0.220 0.253 0.51

Touch Rate 0.338 0.123 0.024 0.022

0.022

Note: Low usability scores were associated with favorable opinions of the system, and high usability scores with unfavorable opinions. Indicates significance at p ¼ 0.05.

176


rating and completion time was also correlated significantly. In general, the shorter the mean completion time, the higher the mean SME rating.

CONCLUSION The findings of Study 1 reinforced many commonly accepted human factors and interface design principles, such as the need for salient feedback. We are somewhat dismayed that these principles still seem to be ignored by design engineers. As has been found before (Woods et al., 2004), manual control was the most challenging aspect of training, as indicated by the time it required and participant comments. The second study focused on manual control and the effects of supplementing the main camera view with additional sensor information from different perspectives. While we did not find dramatic statistical effects of which perspective was added (30, 60, or 901), our observations of participant behavior suggested that different perspectives may be more or less helpful, depending on the nature of the mission. If the mission requires location of targets at a distance, then narrower angles are more helpful (e.g., 301). If the mission requires more precise maneuvers and an awareness of what is below the MAV (e.g., for a precision landing), a wider angle (60–901) is desirable. Our impression was that something short of a straight downward view is optimal (perhaps 60–751), because these angles allow the operator to anticipate what will be below the MAV (if it is moving) and make adjustments before arriving directly above the spot. In fact, we suggest that altering the downward view from 901 to about 701 might be beneficial, if the MAV can only be equipped with forward and downward fixed cameras. We are currently investigating this. We also observed that some participants were ‘‘naturals’’ and mastered manual control very quickly, while others had more difficulty. We were unable to relate this to any of the demographic data we collected (e.g., remote control experience or video-game playing); but, we may have lacked the power to pick up such relations. In order to minimize training times, selection methods for potentially proficient operators should be developed. Determination of predictor variables may also assist in developing training modules, to the extent they are experiential, as opposed to trait characteristics (e.g., trainable spatial skills vs. personality factors). Future investigation of MAV pilot training would benefit from use of a dynamic simulation, in which specific scenarios can be designed to capture the different mission-dependent skills that an operator will need to acquire.


177

Our measures of performance were all observational, and we needed to use the characteristics of the static simulation environment to design test missions. Ideally, the simulation environment should allow automated performance monitoring (e.g., routes, collisions, etc.) and the ability to set up test courses (e.g., fly a slalom) or hide reconnaissance targets. As the MAV transitions from an experimental vehicle to an asset of the future force, specific training routines, and performance criteria will need to be determined before an operator graduates from simulation training to live vehicle operation. In addition, standardized courses and measures would allow for comparison of the ease of handling micro air vehicles with different operating characteristics.

REFERENCES Clarke, P. E. (2004). Vertical UAV eyed for small units. Military aerospace technology online edition, downloaded 06/15/05 from http://www.military-aerospace-technology.com/article.cfm?DocID=520. Peruch, P., & Mestre, D. (1999). Between desktop and head immersion: Functional visual field during vehicle control and navigation in virtual environments. Presence, 8, 54–64. Rybarczyk, Y., Galerne, S., Hoppenot, E., Colle, E., & Mestre, D. (2001). The development of robot human-like behaviour for an efficient human-machine co-operation, Proceedings of the 6th European conference for the advancement of assistive technology, 10, 274–279. Woods, D. D., Tittle, J. S., Feil, M., & Roesler, A. (2004). Envisioning human–robot coordination in future operations. IEEE Transactions on SMC Part C, Special Issue on Human–Robot Interaction, 34.


178

13. VIDEO IMAGERY’S ROLE IN NETWORK CENTRIC, MULTIPLE UNMANNED AERIAL VEHICLE (UAV) OPERATIONS Bruce P. Hunn The successful operation of an unmanned aerial vehicle (UAV) is more highly dependent on interface design than that of a manned aerial vehicle. Removed from the haptic, audio, and visual feedback inherent in manned aircraft, the UAV operator must glean most of the information necessary to successfully fly the aircraft and perform the mission strictly from a few visual displays. Current UAV systems perform their activities using conventional, aircraft style displays, however those types of displays may not be suited to multiple, networked UAV operations because they were designed to represent the system function of only one aircraft, rather than multiple aircraft. Previous and current UAV display technology has followed this aircraft-driven design trend of the aircraft centric approach of displaying one’s own aircraft information, rather than the synthesis of multiple aircraft information. This paper presents a review of conventional UAV video display technology and current state-of-the-art video display concepts, and discusses why newer, integrated-information visual displays can be more effective for the comprehension of larger amounts of information than existing display types. Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 179–191 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07013-X

179

180

BRUCE P. HUNN

The use of UAVs, also called Unmanned Aircraft Systems (UAS) or Remotely Operated Vehicles (ROVs), has its origins at the beginning of aviation where Civil War balloons and World War I (WW I) aircraft both employed cameras to record military information. Partially manned B-17 aircraft of WW II were piloted to a location where the pilots bailed out, and then were guided by television cameras to their targets, using the first ‘‘remote’’ controls, and ‘‘automatic pilots’’ (Goebel, 2004). Along with gliding bombs, these remotely piloted aircraft set the stage for UAVs of the future by combining automated and manual control systems. Since WW II, tremendous advancements in communication links and beyond line-of-sight radio control have enhanced the operational demands of UAV systems. Beyond-line-of-sight radio control, in particular, has allowed Predator UAV missions to be controlled from Air Force Bases in Nevada for Predator flights in Iraq (Hunn, 2005). In addition, some UAV operators have been simultaneously tasked with accomplishing missions over extended periods of time and in some cases with several UAVs at the same time (Hunn, 2005).

UAV VISUAL IMAGERY In the beginning of unmanned systems (such as the WW II vintage glide bombs or remotely guided aircraft), the mission of the UAV was quite simple: take off (or be released), fly to the target, and destroy the target. Therefore, the visual interface required was correspondingly simple. Commonly a television camera was mounted on the front lower portion of the air vehicle or on the glide bomb and that camera provided a narrow field-ofview representing the flight path of the UAV, and the pilot of that weapon merely steered the weapon to the target via feedback from that camera. The numerous wartime failures of such simple systems required more complex control and display systems to be designed, and those systems mirrored manned aircraft control systems. However, in the time that has intervened since the first UAVs were operational, the destruction of a UAV is now no longer an operational consideration; instead, the UAV is seen as a sustainable/re-useable tool much like a conventional aircraft. Reconnaissance, battlefield damage assessment, and weapons delivery are now, like manned aircraft, the missions of the UAV. In many ways the trend in UAV technology is to mimic and eventually replace manned aircraft for a variety of operations. This trend in UAV missions now places two primary tasks at the forefront of the UAV operation:

Video Imagery’s Role in Network Centric, Multiple UAV Operations

181

1. Continuous UAV Command and control 2. Target data collection and processing. If these tasks are broken down by aircrew position, the pilot (often called the operator) is the prime command and control coordinator, while the second crew member (the sensor operator) is responsible for collecting, processing and communicating sensor data. There is often an overlap of duties between these two crew members, but the operation of smaller UAVs is commonly only controlled by these two personnel. An illustration of the ground control shelter interface used by a typical Army UAV pilot (AVO, Air Vehicle Operator) for the US Army Shadow UAV follows (Fig. 1). This screen includes: a moving map navigation display in the lower left, a secondary subsystem’s status window in the upper left, an aircraft location display in the upper right, and the primary UAV subsystem’s display in the

Fig. 1.

Shadow AVO Screen Configured with Four System Status Windows (Zamudio, 2004).

182

BRUCE P. HUNN

lower right (Department of the Army, 2002). The displays represented in the lower right of the screen are digital representations of aircraft analog gauges and indicate common performance values like heading, attitude, altitude, temperatures, pressures, fuel level, etc. The moving map display in the lower left quadrant of the display screen is virtually the same as a printed navigation map, and has the typical look-down field-of-view. On the moving map display the UAV’s flight path is shown as a series of waypoints connected by a solid line, with the overlay of a UAV pictorial icon shown on the upper leg of the flight path, thus symbolizing the current location of the UAV. In contrast, Fig. 2, taken from a Scan Eagle developmental UAV shows a sensor operator’s video image of a typical military (missile) target. In Fig. 2, there is a target locator box superimposed on the object of interest (the missile). Targeting information such as location, distance, and bearing to the missile are also shown as well as similar location data on the UAV collecting the imagery (a Scan Eagle UAV). Coordinates for both UAV and target are exhibited on the top and bottom of the display image. This image also represents a typical sensor operator field-of-view of a target.

Fig. 2.

Video Target Image of Missile with Launcher and Support Equipment (Gallagher & Raymore, 2004).


183

NETWORK CENTRIC UAV VISUALIZATION SYSTEMS While legacy UAV display systems have performed well in operational conditions to date, the additional burden of multiple, dissimilar UAV operations add complexity which often taxes an operator’s or manager’s situational understanding. When UAVs operate as a network, an understanding of how each of those systems are operating, as well as of the information they are collecting, cannot be easily assimilated without an improved style of interface. This is because current UAV interfaces deal almost exclusively with single aircraft information. The following figures show network centric imagery, where a variety of UAVs are displaying their position and information collection capability on a single screen. Displayed in the following figures are various UAVs through a network-based software package called TerraSightTM. For example, Fig. 3 shows an actual operations scenario where two

Fig. 3.

Two UAV Operations (Gallagher & Raymore, 2004).

184

BRUCE P. HUNN

UAVs are working as a coordinated team to observe the same target on the ground. In this scenario, the combination of the two UAVs yields a greater capability than either of them alone because of the specialized observational abilities inherent to each UAV. One UAV can use an electro-optical (EO) sensor while the other is using an infrared (IR) sensor. An advantage in having two UAVs observe the same target with different sensors lies within the capabilities of each sensor type. For example, the IR sensor-equipped UAV can determine if a mechanical object like a tank has recently moved by the tank’s residual heat emissions, while the EO sensor can better resolve visual information (such as a tank’s markings). However, only one image can be viewed with a conventional UAV display. In contrast, the TerraSightTM system allows both images to be displayed along with a threedimensional view of the battle space. The ability to achieve a greater level of system situation awareness from this type of visual display was commented on positively by personnel who have used this system (T. Ferrazano, USJFCOM, personal communication, July 5, 2004). User comments on this display included a greater level of situation awareness, more effective communication of UAV positions and better coordination of target movement information. This system provides a big picture awareness of an operation not available with current UAV displays.

VIDEO DISPLAY SYSTEM CHARACTERISTICS Like manned aircraft, UAVs have the same prioritized, meta-requirements of ‘‘aviate (fly), navigate, communicate, and manage’’ (Tsang & Vidulich, 2003, p.147). All four meta-requirements apply to the pilot of a UAV, while the last two (management and communication of information), also apply to the sensor operator since these are their primary duties. All of these UAV command and control functions must meet the criteria of ‘‘effectively processing information y having legible displays y employing display integration y enhancing proximity and compatibility y providing pictorial realism y considering the principle of the moving part y as well as, aiding and guaranteeing discriminability’’ (Tsang & Vidulich, 2003). Currently, these needs are being met by aircraft-like, digital renditions of conventional knob and gauge displays. The progression of UAVs beyond these basic flight display requirements is now being based on the need to create a networked system where multiple UAVs are operating. An analogy for this situation using existing systems is that of the air traffic controller (ATC),


185

who must manage exocentrically, a variety of air vehicles, which are separated in space and time. To further continue with this ATC analogy, a typical air traffic control screen encompasses several hundred air miles, and is a fixed location, look-down, planar, field-of-view display. Aircraft are designated by call numbers along with airspeed information and their heading track. Within this display of aircraft information are overlaid simplified geographical and airspace boundary lines, along with color-coded weather radar information, and this level of information has provided safe guidance for worldwide air traffic control for many decades. In Fig. 4, there is a corresponding representation of a Shadow UAV mission payload operator (MPO) screen with navigation and target marking fields displayed. On this screen’s upper left is a compass rose display, which shows bearing as well as attitude, and altitude, displayed both digitally and symbolically. Below that, in the lower left, is a moving map display showing the flight path of the UAV (lines), target designators (boxes), and way points (dots). In the right upper and lower quadrants are target coordinates, target marking

Fig. 4.

Shadow UAV MPO Screen (Zamudio, 2004).

186

BRUCE P. HUNN

displays, and sensor controls. Using a screen setup such as this, the MPO has the same flight information as the UAV pilot, but the MPO’s task is to focus on data collection from sensors and marking of targets. However, in this particular case, the MPO is performing an additional role that includes navigation monitoring, route planning, or target designation. While these legacy UAV displays provide system and navigation information, as well as sensor information, they neither provide a good awareness of other UAVs nor do they merge (sensor fuse) real time and historic information. In short, they do not employ many principles of sensor fusion commonly used in advanced display technology. In this example, the term sensor fusion does not relate to MPO-type sensors, but to the idea that all sources of information (from any type of sensing system) are merged to achieve a greater level of effectiveness than isolated sensor displays could achieve. One trend in video display technology is to merge the ATC concept with a three-dimensional battle board model, thus creating a three-dimensional sense of the battlefield, which fuses several sources of information into a cohesive, comprehensive, big picture, and this is what Fig. 3 illustrated and is further reviewed in the following sections.

NETWORK CENTRIC UAV VISUALIZATION The information display system previously referred to as TerraSightTM (Fig. 3) can portray multiple UAVs within a scalable volume of digital space (S. Das, personal communication, July 6, 2004). Relying on a base of geographic digital photos, operators can select a volume of the earth’s surface pre-recorded in a database and zoom in until they have achieved their own, user defined and manageable, field-of-view. The UAVs are displayed real time as pictorial icons superimposed over the digital map database. They are designated using alphanumeric codes that appear adjacent to their pictographic representation. Similar to an ATC view, this allows observing the movement of the UAVs within a volume of space as defined by the system user. In contrast to an ATC approach, this system can also provide a real time overlay of conventional navigation charts or artificial features onto this digital photo database. This can allow the pilot or sensor operator the ability to fade back and forth between the photo-database and a navigational chart, or it can also allow an operator to mix and match items on the screen, where one part of the screen is focused on a navigation chart, a database, or a live action image from the UAV sensors (Gallagher & Raymore, 2004). Fig. 5 shows additional features such as: the aspect angles


Fig. 5.

187

Three-Dimensional Display of Multiple UAVs (Gallagher & Raymore, 2004).

and fields of view of each UAV as well as the actual projected video images each UAV sensor is currently collecting. Objects of interest and boundaries are overlaid on the imagery projected in this screen capture (Gallagher & Raymore, 2004). It is proposed that this synthesis of information types provides a better understanding of a networked system than previous visual display types.

VIDEO DISPLAY COMPLEXITY It is not enough to merely display large amounts of information using a video display system; that information must present a coherent picture that can be understood and acted on by its user. Combining numerous and conventional UAV displays into a single screen would rapidly become very challenging to an operator when that data must be interpreted and allocated

188

BRUCE P. HUNN

to individual aircraft out of many UAVs displayed. The descriptive visual display approach just mentioned supplies a global perspective, while not overloading the operator with extraneous information. Fig. 6 shows two UAVs operating within the defined geographic boundaries as shown by the color-coded lines. The position of each UAV in relation to each other can be determined quickly and the positions of each UAV relative to defined boundaries or targets on the ground can also be quickly determined. Both UAVs are observing two adjacent areas well within the boundary lines and the UAV to the right is observing an area designated by the black hexaform. Orientation of live video imagery is provided by a north-facing arrow near the top right of the screen image. Lines, arrows, and symbology provide guidance to the operator, and establish relationships of objects. Within the image above, the real time optical sensor images of each UAV are superimposed on the digital database (i.e., rhombus figures on the ground). These live video images are shown within the projected lines coming from the UAV to where they intersect with the

Fig. 6.

Three-Dimensional UAV Views (Gallagher & Raymore, 2004).


189

ground. While these live images are very small in this particular screen print, they can be dragged, dropped, enlarged, and examined while the mission is occurring. The ability to move back and forth between digital and real imagery, or to mix those types of images is believed by operational personnel to be very desirable in a military operation (Hunn, 2004). For example, using this system, operators can contrast historical imagery with real time imagery. Changes in topology, structures, and even ground cover vegetation can be readily observed, and compared to live feed video to provide tactical or strategic advantages to personnel on the ground. The configurable Windowss environment of this system provides an intuitive way to display systems output, and to allow operators to manipulate that information in a way that is commonly used in other day-to-day computer applications like word processing programs. Mixing real time and historical digital footage along with artificial target markers, and geographic, or political boundaries creates a rich information environment that is meaningful in terms of mission critical factors such as camouflage, changes in building structures, or movement of vehicles.

VIDEO DISPLAY TECHNOLOGY TRENDS AND FUTURE PROGRESS The increase in video display technology sophistication has come a long way since the silver emulsion photograph or WW II optical bombsight image. The recent merging of digital databases with artificial symbology and real time imagery in a Windowss environment has produced systems where situation awareness can be greatly enhanced not only for single UAVs but also with multiple UAVs. In the future, this same video display technology may even be applied to the direct command and control of UAV systems. Unfortunately, current UAV command and control systems are as complex as the knob and gauge aircraft-style systems that they have copied. In fact, many UAV command and control systems are nothing more than video projections of knob and gauge displays. Earlier control and display systems were complex from the standpoint that they required single point, serial focusing on each element. These types of systems did not typically synthesize information, such as can be accomplished by a pictorial icon. Rather than displaying discrete system functions (multiple gauges approach), a coherent display technology could focus on synthesizing

190

BRUCE P. HUNN

information in much the same way that video three-dimensional displays described above have done. While automation is often touted as a cure for increasing display complexity, it can create a problem by presenting too much information to the operator in the mistaken belief that more information is better information. This UAV design trade-off issue is between what information is available versus what information needs to be displayed. A more intuitive and simpler control and display concept would be in presenting only that information necessary to maintain the cognitive awareness level sufficient to operate the system, but to require no more information processing than what is needed for the tasking at hand, and the use of visual means may be a vehicle for that solution. Human visual and cognitive systems have the ability to process a lot of information, and with proper visual display technology, they can create a simple picture of a complex series of data inputs (like the integrated visual display systems discussed above). It could also be argued that manned flight was once sustained with only four or five singular, mechanized, data displays (instruments) however, technology and automation can now provide more information than is necessary to perform the same functions in a manned or unmanned aircraft. Addressing that information overload, a properly designed visual display system could once again simplify complex information requirements by portraying them as an intuitive, visual, moving picture image.

SUMMARY It could be argued that combined presentations of diverse types of information (such as illustrated in this chapter) could provide an awareness of multiple UAVs and their environments that are superior to isolated sources of data (i.e., paper navigation charts, live video feeds, photos, drawings, analog or digital portrayals of knobs and gauges, etc.). It is also possible to present complex spatial relationships, such as those associated with multiple UAVs using a three-dimensional display representation (such as illustrated in this chapter). By integrating and simplifying the type of data presented using a visual medium, the operator’s cognitive resources may be better capable of grasping the whole picture of a complex scene, and thus be more capable of dealing with complex operational issues. By using combined and synthesized information presentation, a better match can be achieved


191

between the requirements of the task at hand and the capabilities of the human to attend to, and understand, the information being presented.

REFERENCES Department of the Army. (2002). Technical manual operators manual for Shadow 200 TUAV system (TM 9-5895-681-10). Redstone Arsenal, AL: U.S. Army Aviation and Missile Command, Joint Tactical Unmanned Aviation System. Gallagher, J., & Raymore, P. (2004). VICE, video imagery capability enhancement. Hurlburt AFB, FL: USAF Command and Control Battle Laboratory. Goebel, G. (2004). World War Two glide bombs. Retrieved January 5, 2004, from v1.5.0 / chapter 4 of 13/01 nov 04/ http://www.vectorsite.net/twbomb4.html. Hunn, B. P. (2004). Forward look III human factors report. Norfolk, VA: U.S. Joint Forces Command, Intelligence Directorate 32. Hunn, B. P. (2005). ER/MP (extended range multipurpose) manning study. Huntsville, AL: SFAE-AV-UAV-MAE, Program Manager. MAE. US Army. Tsang, P. S., & Vidulich, M. A. (Eds) (2003). Principles and practice of aviation psychology (pp. 147–155). Mahwah, NJ: Lawrence Erlbaum Associates. Zamudio, L. (2004) US Army UAV Training Center, E Co. 305th MI BN. Ft Huachuca, AZ (UAV display images).


192

14. SPATIAL DIALOG AND UNMANNED AERIAL VEHICLES Wendell H. Chun, Thomas Spura, Frank C. Alvidrez and Randy J. Stiles BACKGROUND Lockheed Martin has been a premier builder and developer of manned aircraft and fighter jets since 1909. Since then, aircraft design has drastically evolved in many areas including the evolution of manual linkages to fly-bywire systems, and mechanical gauges to glass cockpits. Lockheed Martin’s knowledge of manned aircraft has produced a variety of Unmanned Aerial Vehicles (UAVs) based on size/wingspan, ranging from a micro-UAV (MicroStar) to a hand-launched UAV (Desert Hawk) and up to larger platforms such as the DarkStar. Their control systems vary anywhere between remotely piloted to fully autonomous systems. Remotely piloted control is equivalent to full human involvement with an operator controlling all the decisions of the aircraft. Similarly, fully autonomous operations describe a situation that has the human having minimal contact with the platform. Flight path control relies on a set of waypoints for the vehicle to fly through. This is the most common mode of UAV navigation, and GPS has made this form of navigation practical. For a fully autonomous UAV, the platform makes its own decisions. However, it should be noted that a human operator is never out of its Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 193–206 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07014-1

193

194

WENDELL H. CHUN ET AL.

command structure; instead, they assume a supervisory role in its overall control construct. Autonomous operation is important for two reasons: the burden of the communication link between the aircraft and its operator is significantly reduced or eliminated, and secondly, the migration to more capable software enables a new paradigm in which a single operator can control more than one vehicle at a time (force multiplication). Nevertheless, state-of-the-art UAV control is still waypoint navigation where the operator picks a sequence of points and each waypoint consists of latitude/longitude information, airspeed/altitude at each waypoint, and other information such as fuel load. After the UAV flies a certain distance and prior to reaching its last points, additional waypoints are transmitted to the UAV. The waypoints are spaced far enough apart to accommodate for vehicle speed and turning radius. While autonomy can be integrated onboard or offboard the vehicle, the inputting of the waypoints and other remote pilot functions occurs at the operator control station. As technology advances, the role of the human will transition from those who designed the vehicle to those who can operate and maintain the vehicles. In addition to logistics, set-up, refueling or recharging, and periodic maintenance, the human can play an important role in flying the vehicle. Even as autonomy for UAVs matures, it cannot operate without a control station for safety reasons. Control stations can be UAV-specific and unique, or be able to control multiple UAVs by having common interfaces and protocols. Unfortunately, these types of control stations do not exist today. For the military, air operations are based on a 24-hour cycle, and crews operate in 8–12 hour cycles. Increased vehicle endurance based on autonomy can reduce the manpower requirements for monitoring consoles leading to reduced costs. Past lessons learned include using qualified test pilots during testing who possess an understanding of aerodynamics, which can save the aircraft when necessary, and can be trained to react to unexpected events. Another way to reduce operator costs is to increase the emphasis placed on the operator–vehicle interface. From a human factors perspective, the man–machine interface must provide the following:

Adequate fault annunciation to the operator Incorporate immediately recognized screens Suggest appropriate responses to situations Factor in operator workload, especially during emergencies Include adequate training for emergency procedures.

In recent years, there has been a movement by the military to have greater inter-operability between different control stations and an assortment of

Spatial Dialog and Unmanned Aerial Vehicles

195

UAVs. Hardware-in-the-loop testing is recommended prior to the first flight of any new platform. It is interesting to note that actual pilots fly to train, but UAV operators rely more on simulations to fly.

UNMANNED COMBAT ARMED ROTORCRAFT In 2002, the Defense Advanced Research Projects Agency (DARPA) initiated the Unmanned Combat Armed Rotorcraft (UCAR) program. The intent of this program was to design, develop, integrate, and demonstrate the enabling technologies and system capabilities required to effectively and affordably perform armed reconnaissance and attack within the Army’s objective force system-of-systems environment. The UCAR program completed two of the three planned phases prior to being cancelled, but before it was cancelled the contractors completed demonstrations of their concepts and technologies. UCAR systems are designed to use both offboard situational awareness and targeting information, and onboard sensors for long-range target identification. Fig. 1 depicts the UCAR system-of-systems operational concept. As part of the Lockheed Martin solution, the UCAR teams become natural companions as future scouts for the Apache, Army Airborne Command and Control System (A2C2S), and other airborne platforms, as well as ground commanders, where the Wolves will see first, act first, and finish decisively with lethal efficiency. Managing workload was a fundamental component of the command and control puzzle to be solved for a UCAR operator. Emerging technologies in Human Interaction were utilized in the Lockheed solution to allow the operator to manage workload as opposed to be overrun by it (Spura, Accettullo, Franke, & Zaychik, 2005). These technologies included hardware as well as software components, including Management by Exception, Negotiated Interruption, alternative-alerting modalities, and augmented reality. Management by Exception provides the operator with the solution recommended by the system but does not require operator approval prior to initiation; this frees the operator from acknowledgment tasks but does provide the operator to reject the solution if it is unacceptable. Negotiate Interruption is a system capability that allows the operator to postpone action on a message from the system, thus offering a powerful mechanism for the operator to manage workload. Alternative alerting modalities were experimented with, including the Naval Aerospace Medical Research Laboratory (NAMRL) developed tactile vest, as a means of alerting the operator to incoming messages from the system. Augmented reality was

196

UCAR Concept of Operations.


Fig. 1.


197

proposed to increase the information available to the operator in the visual field. The utilization of these technologies provided the operator with the ability to manage multiple unmanned systems while continuing to execute a pre-existing role, such as an Apache Copilot–Gunner. One of the biggest obstacles to successful command and control of unmanned vehicles is the workload incumbent upon the warfighter. Current solutions to this problem require dedicated operators at a large control station that are designed to control only a single vehicle. Through the use of alternate interface concepts and workload management techniques, a single operator will be able to manage multiple unmanned systems in the future. Workload presents a challenge to the growth toward control of multiple vehicles by dedicated operators. In addition, there is a move toward relocating control of unmanned vehicles to warfighters in the field, who have their own tasks and survival on which to focus, making it imperative to remove the bottleneck surrounding the human in the loop. There are two main classes of strategies for accomplishing this: (1) moving the human’s workload to modes of operation where the impact is less and (2) decreasing the level of vigilance in interaction between the human and the system by automating the control of that interaction. A key element to decreasing human workload is to parallelize tasks to the greatest extent possible. This is a difficult challenge because cross-task interference can actually increase workload and result in more errors. This is especially true for unmanned system control tasks, which all largely utilize the same visual interface mechanisms, resulting in cognitive interference. However, recent work in the application of Multiple Resource Theory to UAV controls indicates that there is a workload benefit to moving some control tasks to other modalities. Multiple Resource Theory argues that the performance of tasks in different modalities (for example, performing one visually/manually and one auditory/verbally) can result in less cognitive interference between the tasks because they use different sets of resources within the cognitive system. The two best candidates for producing a nonvisual mode of interaction are aural and tactile. The clearest direction for non-visual work offload is into the auditory modality. Humans process computer-generated auditory alerts all the time, navigating the ‘‘boops’’ and ‘‘bleeps’’ emitted by current devices. However, to truly affect workload, the entirety of interaction should be shifted into the non-visual modality. Operators must be able to communicate through sound as well as receive meaningful auditory cues. That is, operators must be able to converse verbally with the man-machine interface system to fully receive the benefits of multi-modal workload reduction.

198


Natural spoken dialog is a mixed-initiative interaction in which each participant may steer the conversation by eliciting information from the other participant at any time. Modern spoken dialog systems model this mode of interaction, supporting systems that can request clarifications from the user and react to tangential or even non-sequitur utterances by the user on subsequent turns of the conversation. This is accomplished by constructing a speech system that keeps track of dialog context and fits utterances from the user within the framework of that context. For example, if the user asks, ‘‘What is the status of the EO/IR?’’ then follows it with ‘‘What about the missile pod?’’ the user is clearly asking for an update of the status of the missile pod, even though status was not mentioned in the second utterance. Instead, the second utterance builds off of the status request context established by the prior utterance. By maintaining the context of the conversation, the spoken dialog system can simulate the filling-in-the-blanks ability that humans take for granted. In addition, the modern spoken dialog system is also constructed with the domain knowledge to insert a degree of intelligence into the system’s interactions. For example, if there are multiple missile pods that could be referred to, the system should reply with a request for clarification: ‘‘Which missile pod do you want the status of?’’ By doing so, the system takes initiative in the dialog, working as a partner to the user in making the conversation reach its successful end. The use of spoken language interaction during UCAR operations was shown to be very useful in minimizing workload and at the same time allowing the operator to interact with multiple teams of UCAR entities.

SPATIAL DIALOG The UCAR case study indicates a benefit from multi-modal interaction between a UAV and operator. Current approaches for crew-autonomy interfaces focus only on spoken dialog and traditional screen displays, effectively ignoring the need for two-way spatial interaction and programming-by-example, especially in remote situations requiring variable autonomy. Furthermore, today’s UAVs are relegated to communicating with commercial telecommunication hardware and protocols. This implies levels of data abstractions that may or may not be compatible with humans or useable across different platforms. Inherent in the levels of abstractions are spatial locations, configurations, plan intent, timing instructions, and information content. There can be communications at multiple levels and in parallel. The issue of crew-UAV


199

interfaces requires research beyond the existing teleoperated and master-slave control configurations previously deployed. The goal for spatial dialog research is to improve remote virtual presence and collaboration in a data-rich environment for mixed teams of human crew, autonomous vehicles, and remote humans in the field. With our spatial dialog approach, we provide virtual interactive presence for collaborating with autonomous systems by incorporating the spatial context along with the spoken context. Spatial Dialog is an interaction and communication technology that uses a deeper understanding of spatial context and a richer spatial vocabulary to realize improved human–computer symbiosis. Spatial context is useful for communicating, understanding, and remembering information. It includes the location of objects and people, and the place in which events are occurring. Using computers that see, we can pick out faces in a scene, track people, and follow where people are pointing and the objects they use. This is illustrated in Fig. 2. Using augmented reality and a real-time understanding of the spatial scene, we can overlay information on the real world or virtual models as a key spatial component of the dialog process, especially for remote virtual tele-presence, tele-supervision, and tele-science cases. The impact for this technology will be improved specification and understanding of robotic tasks. To illustrate

Fig. 2.

Spatial Dialog Approach.

200


this, we provide a narrative ‘‘day-in-the-life’’ describing this technology, and we provide examples for applications to future objectives. In the narrative, bolded italics indicate dialog and actions that use spatial context. It’s the year 2010, and a team of Special Operations Forces has encountered a problem. An unexpected leak in a fuel tanker has affected the distance to be traveled by a ground convoy. Most of the remaining fuel is used for support functions, and the team must conserve resources prior to reaching its objectives. Ammunition is low and the convoy is carrying sick people, and the team wants to extend resources until supplies can be replenished. Mike, the mission specialist back at Headquarters, is communicating with Bob, one of the soldiers in the Arctic, and SAM-1, a semi-autonomous UAV that is normally used for discovery missions, is stored at the far end of the Arctic Camp. Mike and Bob are using the SMSS (spatial mission specification system that uses spatial dialog technology) to look at a shared 3D representation of the base area, with each mission module shown in the layout. Bob points at a remote region in the area, ‘‘Mike, I think this area has an abandoned airfield and supplies.’’ After short message latency, Mike responds by selecting two intermediate sites as potential waypoints. ‘‘Yes, but the convoy can’t get there within a day. The distance is too great and traveling at high speed over bad terrain could endanger the wounded.’’ Mike, with a pause, asks the spatial dialog system to enlarge the scene that he is looking at, and proceeds to give instructions to SAM-1, ‘‘SAM, this is the area. Because there could be potential supplies here,’’ he says while pointing at the area in the upper section of the local map, ‘‘you need to go to this area and look for structures that have shapes like these to find the needed supplies. Before you do that, map these buildings and other pertinent natural features.’’ SAM-1, who really is working with an internal task and spatial model of the region, responds to this request by using everyone’s 3D augmented reality graphics display to show his path to the supplies, ‘‘Understood, confirm that I will visit these three areas in this order,’’ then SAM-1 shows the path it will fly, ‘‘and model these buildings when it gets there. Where would you prefer that we set up camp?’’ Bob points to a bunker near the center of the structures, and say, ‘‘SAM, please map the planned trajectory to get to here.’’ Mike, Bob, and SAM-1 (the UAV) work through this process for mapping the area of interest, and SAM-1 begins to execute the mission as planned. By using the spatial dialog technology, the team was able to put


201

together a specific course of action with a minimum amount of questions, repeated sequences, or false assumptions. The specification of spatial items along with speech was sufficient for them to achieve a shared model of the task at hand. The future the UAV will be equipped with a spatial dialog harness so that an authorized operator can speak directly to your UAV and show them the actions they should take, and they can talk with you the same way to clarify points (multi-modal dialog between human and UAV in situ). With the graphical overlay of information on the team’s spatial surroundings, the exploitation of the team’s surroundings for task specification, explanation, and clarification becomes much easier. Remote team members can highlight physical items when they are talking about them, autonomous systems can visually explain their planned actions to humans before carrying them out, and humans can visually specify or constrain the actions of autonomous UAVs as they work with them, enabling them to engage in a spatial discourse about the actions the team of humans and UAVs will take. This system integrates a number of related efforts from the fields of robotics, unmanned vehicles, multi-modal interaction, and cognitive psychology that have been referenced here. For the Virtual Environments for Training (Johnson, Rickel, Stiles, & Munro, 1998) project for the Office of Naval Research, Lockheed Martin developed one of the first instances of embodied conversational agents (called Soar Training Expert for Virtual Environments) that acted as mentors and missing team members for task training in the virtual environment. The operator could select objects in the world and use voice commands to ask the agent about the object. The operator could ask for his mentor to show him a task, and then follow the mentor’s gaze, hands, and voice as the mentor agent showed the steps of the task. Furthermore, the operator could be given a tour of the spatial layout of his task area, such as the engine room of a ship, and carry out spatial and spoken tasks with his agent teammembers. It is on this project that we first realized the significance of spatial dialog, where the emphasis is not just on fusing multi-modal input, but includes the multi-modal output of agents situated in the world with the operator. This is the foundation for this project. An excellent survey of socially interactive robots, a category of robots where human-robot interaction is important, is provided by Fong, Nourbakhsh, and Dutenhahn (2002). In this survey, Fong notes that creating a robot that communicates at a human peer level remains a grand challenge. However, he does provide examples where task-focused dialog has been achieved. Fong notes that users need explicit conversational feedback from robots before they carry out their instructions, and how some systems can

202


use query-response dialogs as a means for humans to help robots on their tasks. Fong identifies systems that use various types of perception, such as people tracking, speech, gaze, and gesture recognition separately, but does not note any robotic systems which combine these modes for input and output, such as we are working on for task-based spatial dialog. A UAV is an extension of existing mobile robots. Researchers at NASA Ames Research Center (Bradshaw et al., 2002) have worked to develop a Personal Satellite Assistant (PSA), a softball-sized robot that will work in teams with humans and other PSAs onboard spacecraft to support systems monitoring, communications, information updating, and remote operations. The PSA supports a spectrum of adjustable autonomy, where it must reason about its own goals and those of other agents, including humans. To accomplish teamwork with humans, NASA has employed Brahms to model human interaction and tasks with the PSA, identified initial agent conversational policies, and has developed an initial speech interaction prototype. The PSA will operate in a complex spatial environment, and has a lot of potential for the application of spatial dialog principles; showing a crew member a particular system while telling them about it is one case. At this stage, the PSA project does not have a formal notion of spatial dialog between a human and a PSA, where a crew member gesturing or looking at an item will modify the PSA’s understanding of the crew member’s speech. Our use of augmented reality overlays for spatial output in a spatial dialog could also be of benefit for remote operation of the PSA, helping to establish a shared frame of reference, and it could be of benefit for onboard operation, allowing the PSA a means to overlay spatial information on the real scene for the crew member. Spatial Dialog builds upon this previous work of Bradshaw by incorporating the KAoS, a human–machine interaction framework into the system. Trevor Darrel and others at the Massachusetts Institute of Technology Artificial Intelligence (MIT AI) Lab have prototyped face–pose interaction for gaze-mediated agent dialogs using stereo motion camera techniques (Darrell et al., 2002). Using an SRI, International stereo camera, they set up an interactive room prototype to test different modes of interaction with agents. They also established ideal conditions for testing these modes using a Wizard of Oz experiment. In Wizard of Oz (WOZ) experiments, a set of subjects interacts with a software system driven by a human wizard (Bernsen, Dybkjær, & Dybkjær, 1998). The wizard operates the software, and the goal is to emulate the planned system capabilities and identify areas for improvement. The human subject does not know there is a human wizard operating the system. The data collected from WOZ experiments are often


203

used to inform the design and implementation of multi-modal interaction systems and speech dialog systems because of their complexity. The modes were look-to-talk, a gaze-based approach for onset of speech, talk-to-talk, where keywords were used to determine onset of speech, and push-to-talk, a common method where a button is pressed to determine onset of speech. Although the WOZ experiment subjects said that they preferred the talk-totalk method, they actually used the gaze-to-talk method to answer most of the study questions than the other modes of interaction. In our approach, based on augmented reality technology, we can identify what objects the person is looking at, effectively determining its face–pose relative to these objects, and enabling a look-to-talk mode while additionally overlaying information in the spatial context as another part of the dialog process. Teams of agents (human and/or robot) must coordinate their mental states by communication to realize some form of joint intention. The process of coordinating mental states by providing evidence is called grounding. The model for the process of grounding is based on a series of contributions in the conversation, where each contribution has two phases, a presentation phase and an acceptance phase. The presentation phase is carried out to achieve the first agent’s discourse goal of achieving a common understanding, and the acceptance phase, conducted by the second agent, may consist of acknowledgment, further dialog for clarification, or repair statements. Repairs are partial statements where some parts of the statement must be removed to be understood. Repair categories include removing what was said earlier to start fresh, modifications to the earlier statement, or abridged re-statements. For spoken dialog, we shall use Heeman’s algorithm (Heeman & Allen, 1999), which is based on a probabilistic assignment of words to parts of speech, for recovering from repairs. We will be able to incorporate gaze and gesture as additional repair indicators, which we call visual recovery, and could potentially increase our repair rate over that of speech alone. Barbara Tversky and others have performed a number of cognitive psychology studies on what we describe as spatial relationships (Tversky, Lee, & Mainwaring, 1999). The following are several key concepts that we have drawn from this body of research that we use in describing, understanding, and reasoning about spatial references in dialogs. Deictic Center (DC), where the speaker moves the listener’s center of thought from the here and now into the space and time of the description, performing a deictic shift out of reality and into another setting. The DC can be thought of as the context for scenes. We will also investigate perspective taking and descriptive styles. Perspective taking, where the narrator assumes the physical perspective of the listener for describing a scene, or vice versa is an important concept.

204


Deictic also refers to time, and there is perspective taking in time as well. Egocentric perspectives are a body-centered reference, where a person describes space relative to their body, with terms such as left, right, above, below, in front, behind, etc. This is sometimes called a relative frame of reference. Descriptive styles, such as gaze tour based on person’s viewpoint onto smaller environments such as rooms, route perspective based on moving through a larger environment using body-centered references, and survey perspective based on a view from above similar to a map using hierarchical and canonical frames of reference are key attributes. The term cognitive map is used differently by different people (Tversky, 2000). Some regard it as quite accurate and similar to a real map. Tversky’s work has shown that mental representations of environments contain systematic errors that cannot be resolved as a flat map. People create representations on the fly, calling out whatever information about the environment that is relevant. Some of that information will be verbal instructions, some memory for maps, some memory for the environment, etc. This mental structure is often referred to as a cognitive collage. Linguists and psycholinguists typically discern three types of reference systems; egocentric, intrinsic, and extrinsic. The main problem is distinguishing egocentric and intrinsic, because egocentric is with respect to the viewer’s intrinsic sides. Levinson avoids this problem by using relative references, and claiming this must be a three-way relation, so the bike is in front of the house is intrinsic, but the bike is right of the tree is relative; thus the bike is right of the tree relative to me, to my right. So these are three frames of reference. The descriptive styles tend to adopt one or the other of these, so gaze is actually relative for the most part, route is intrinsic for the most part, and survey is extrinsic. However, in actual descriptions, people do something linguists and many psycholinguists presumed they would never do, they mix perspective. A key concept to our spatial dialog work is the use of the DC. The DCs are used in our spatial dialog approach to model how to generate and interpret descriptions of locations for different perspectives (Galbraith, 1995). We shall create a sequence of DCs to describe or modify team plans. We will be experimenting with three types of DCs, used in a variety of combinations; gaze-based DCs for enclosed spaces, route-based DCs for situations such as land rover operations, and survey-based DCs for overviews. Interesting combinations of route and survey DCs are possible. For instance, the overall narrative may be told using a route DC, with transitions into survey DCs where appropriate. Similarities to the ancient


205

method of loci are evident in using a series of route DCs as the overarching structure. The combination of two DCs at once may be possible, where a surrounding DC, similar to the information cockpit concept (Tan, Stefanucci, Proffitt, & Pausch, 2001), is used to display the route; while a sequence of related survey DCs are used as the central focus. This may have some of the same recall effects found by Tan, Stefanucci, Proffitt, and Pausch (2002). With an augmented reality approach such as that we are using, where it is possible to show a map on a table and then fly into the map to view it with a first person view, we can support the transition between egocentric DC views immersed in the landscape, and exocentric DC views surveying the landscape (terrain below). The spatial dialog approach delivers a data-rich virtual presence system where humans and robots communicate spatially for mixed initiative team tasks using both multi-modal output, where augmented reality and speech are used to graphically overlay robotic task plans on the spatial scene, and multi-modal input, where gaze, gesture, and position in the scene are fused with speech. This enables local and remote humans to direct the path of the UAV or teams of UAVs. Our spatial dialog research focuses on providing direct two-way communication between operators and autonomous unmanned systems. Humans and machines represent and store knowledge in multiple ways. However, the efficient transfer of knowledge relies on an understanding of knowledge representation. We use existing knowledge encoding schemes such as the Institute for Human and Machine Cognition’s (IHMC) Concept Mapping and KAoS formalisms to support this representation. We also integrate existing notational methods for the precise semantic theories used. Communication between agents (humans and robots) is dependent on knowledge representation and the semantics of notation (i.e. maps, mathematical diagrams, encoded meanings, etc.). This project is in the early stages of a multi-year effort that is sponsored by NASA.

SUMMARY Lockheed Martin has been a long-time developer of manned aircraft for the military services. Artificial Intelligence techniques (planners, diagnostic, modeling) and increased onboard processing capacity has forced us to plan for future upgrades leading to autonomous operations. Even as the UAV becomes fully autonomous, there will always be a human at a control console, either in the field or at central location. However, the operator’s role

206


will evolve over time with technology insertion from research programs like spatial dialog into development programs such as UCAR. To address the forecasted workload of UAV operators, multi-modal interactions (spoken dialog with interactive presence) may lead to greater effectiveness and efficiency.

REFERENCES Bernsen, N. O., Dybkjær, H., & Dybkjær, L. (1998). Designing interactive speech systems: From first ideas to user testing. New York: Springer. Bradshaw, J. M., Sierhuis, M., Gawdiak, Y., Thomas, H., Greaves, M., & Clancey, W. J. (2002). Human-centered design for the personal satellite assistant. International conference on human-computer interaction in aeronautics 2000. Toulouse, France. Darrell, T., Tollmar, K., Bentley, F., Checka, N., Morency, L., Rahimi, A., & Oh, A. (2002). Face-responsive interfaces: From direct manipulation to perceptive presence. Proceedings of the international conference of ubiquitous computing. Go¨teborg, Sweden. Fong, T., Nourbakhsh, I., & Dutenhahn, K. (2002). A survey of socially interactive robots: Concepts, design, and applications. Technical Report CMU-RI-TR-02-29, Robotics Institute, Carnegie Mellon University. Galbraith, M. (1995). Diectic shift theory and the poetics of involvement in narrative. In: J. Duchan, G. Bruder & L. Hewitt (Eds), Deixis in narratives: A cognitive science perspective (pp. 19–60). Hillsdale, NJ: Lawrence Erlbaum Associates. Heeman, P., & Allen, J. (1999). Speech repairs, intonational phrases and discourse markers: Modeling speakers’ utterances in spoken dialog. Computational Linguistics, 25(4), 527– 571. Johnson, W. L., Rickel, J., Stiles, R., & Munro, A. (1998). Integrating pedagogical agents into virtual environments. Presence: Teleoperators and Virtual Environments, 7(6), 523–546. Spura, T., Accettullo, E., Franke, J., & Zaychik, V. (2005). Operator interfaces for organic command and control of unmanned vehicles. Paper presented at the AHS Forum 61, Grapevine, TX. Tan, D. S., Stefanucci, J. K., Proffitt, D. R., & Pausch, R. (2001). The infocockpit: Providing location and place to aid human memory. Paper presented at the workshop on perceptive user interfaces 2001, Orlando, FL. Tan, D. S., Stefanucci, J. K., Proffitt, D. R., & Pausch, R. (2002). Kinesthesis aids human memory. Paper presented at the CHI 2002 conference on human factors in computing systems, Minneapolis, MN. Tversky, B. (2000). Levels and structure of spatial knowledge. In: R. Kitchin & S. M. Freundschuh (Eds), Cognitive mapping: Past, present and future (pp. 24–43). London: Routledge. Tversky, B., Lee, P. U., & Mainwaring, S. (1999). Why speakers mix perspectives. Journal of Spatial Cognition and Computation, 1, 399–412.

CONTROL OF MULTIPLE ROVS

One vision of future remotely operated vehicle (ROV) operations involves increasing the number of vehicles operated by a single person. The chapters in this section examine how such increases can be achieved while minimizing any adverse impact on the operators. For example, Chris Wickens assesses the impact of offloading tasks to automation. While this may reduce the operator’s workload, the chapter suggests that for some tasks, it may be better to go without any automation than to have it with low reliability. Next, Jared Freeman introduced the Relational Knowledge Framework as a way of examining the complexities of automated systems. This approach optimizes autonomy according to the type of task to be performed by the ROV(s). Consider increasing the size of a UAV team to 100 or more vehicles, then the interaction of operator workload and vehicle automation becomes a critical issue. To address this, Lewis employed an algorithm for human control of small and large teams of advanced munitions and found some support for applying the algorithm to the control of small and large teams of ROVs. Another means of adapting the control of multiple ROVs was examined by Parasuraman. He defines the tradeoffs of a delegation strategy as a function of three factors: the competency of the human–machine system, the workload of the human, and the unpredictability of the machine. Finally, Hou’s chapter presents an interface solution to assist the operator’s decision making in the context of multiple ROVs. The Intelligent Adaptive Interface was developed to manage information dynamically, to consider the operator’s workload and without prompting, communicate the information to the appropriate people. In sum, increasing the size of ROV teams has the potential to multiply the force available to a single operator, but there are many human factors issues to resolve before this becomes a reality. 207


208

15. WORKLOAD AND AUTOMATION RELIABILITY IN UNMANNED AIR VEHICLES Christopher D. Wickens, Stephen R. Dixon and Michael S. Ambinder Unmanned air vehicles (UAVs) such as the Army’s Hunter and Shadow have contributed substantially to supporting mission effectiveness in recent operations with their surveillance capabilities. Future operations will undoubtedly require increased use of these and other UAV assets. However, such needs may encounter the constraints of human personnel to supervise the UAV, given currently the requirement for two soldiers to coordinate the in-flight operation of a single UAV. Our research effort at the University of Illinois, Urbana-Champaign has focused on strategies for reducing the manpower requirements of UAV supervision in a Hunter/Shadow type simulation, from a 2:1 ratio of soldiers to asset, to a 1:1 and 1:2 ratio. Such strategies require the consideration of two important human performance concepts, related to workload and automation dependence. Regarding workload, the effort to assign tasks normally associated with multiple-operators (2:1), to a single operator (1:1), and then to double the number of tasks (1:2), can potentially overload the operator’s limited processing resources, leaving performance on certain tasks vulnerable (Wickens & Hollands, 2000). In other systems, such as the commercial airliner cockpit, such potential workload increases associated with downsizing (from 3 to 2) have been offset by automation to replace the Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 209–222 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07015-3

209

210

CHRISTOPHER D. WICKENS ET AL.

activities of one of the missing participants (in this case, the flight engineer on the Boeing 737). However, a long history of research in humanautomation interaction (e.g., Sheridan, 2002; Parasuraman & Riley, 1997; Parasuraman, Sheridan, & Wickens, 2000) has revealed that automation may not entirely eliminate human cognitive demands, to the extent that such automation will require both set up and supervision. The latter task of automation supervision is particularly relevant to the extent that the automation may be imperfect (e.g., less than 100% reliable). In typical UAV operations, the sources of such imperfections are manifold. As two simple examples, an automatic target recognition (ATR) device to aid surveillance will undoubtedly make some misclassifications, if it is only provided low-resolution imagery to work with; or an autopilot may become challenged to hold a precise course, if icing or severe turbulence disrupts the handling qualities. As we discuss below, the impact on soldier workload of supervising UAV automation depends critically upon the level of reliability of the automation as well as the qualitative kinds of failures that may occur. In four experiments on the UAV Hunter/Shadow simulation at Illinois, we have examined the workload effects of imperfect automation, and have also attempted to develop a computational model of these effects. Such models, if valid, are of considerable importance as they may be used to make predictions of soldier capabilities in the absence of time-consuming humanin-the-loop simulation data.

THE GENERAL UAV SIMULATION Fig. 1 presents the interface used by our pilots to fly the UAV simulation. In the simulation, pilots were responsible for: 1. A primary mission task, in which they tracked the UAV to waypoints and reported on ‘‘command targets’’ (CT) at those known coordinates by reference to the navigational display in the lower left. They could refresh their memory for command target information as required by depressing a ‘‘recall key.’’ Tracking was normally accomplished with a rotational heading control necessary to compensate for periodic disturbances (altitude and airspeed were fixed). 2. A secondary surveillance task in which they searched for ‘‘targets of opportunity’’ (TOO) at unknown locations while en route. Unlike the CTs, these TOOs were camouflaged, and very difficult to see as they passed through the 3D image window (upper left) while the UAV flew overhead.

Workload and Automation Reliability in UAVs

Fig. 1.

211

A UAV Display with Explanations for Different Visual Areas.

3. A secondary systems monitoring task, in which they were required to detect and respond to onboard system failures (SF). This required monitoring of four gauges that slowly oscillated, with one occasionally crossing into a danger zone. When this occurred, the pilot was required to detect it (with a button press), and enter certain critical digital information related to diagnosis and current location. Once either a TOO or a CT was found, the pilots were required to enter a ‘‘loiter mode’’ by depressing a key which brought the UAV to a racetrack pattern around the target. They were then required to zoom in their camera image, and orient it so as to keep the target in view, and report properties of the target (e.g., which side of the bunker contained tanks, how many helicopters were present at the target). The combined operations in these

212


tasks imposed very high workload on the pilots. In the following, we summarize the results of the five experiments and one meta-analysis of the literature addressing three general categories of effects: pilot workload mitigation and modeling and effects of imperfect automation. The majority of participants in our simulation were student pilots nearing completion of their private pilots’ certificate course. Each participant typically flew some practice legs, followed by three scenarios each consisting of 10 legs (each leg defined by a trajectory to a single CT, overflying a TOO, and possibly encountering one or more system failures).

WORKLOAD MITIGATION AND MODELING: EXPERIMENTS 1 AND 2 In Experiment 1 (Wickens & Dixon, 2002), employing the general UAV paradigm described above, we examined two techniques for mitigating the high ‘‘triple task’’ workload of controlling a single UAV. An autopilot mitigation essentially replaced the entire task of flying the UAV (monitoring and controlling the heading trajectory) with a perfectly reliable autopilot, requiring only keyboard entry of the coordinates of the next CT. An auto-alert mitigation provided an auditory warning whenever one of the system gauges passed its critical level. Because we were also particularly interested in the role of the auditory modality in offloading the heavy visual monitoring load of the UAV task, we also switched a second task to auditory presentation: the display of the command target information. Distributing information delivery between auditory and visual channels is a mitigation that is predicted by multiple resource theory (MRT) to improve time-sharing ability (Wickens, 2002). However, it is sometimes asserted that such benefits, while found in the laboratory, may not be apparent in more real-world environments in which people tend to become single channel processors of information. That is, it is a central processing bottleneck, rather than limits of sensory-perceptual mechanisms that constrain multiple task performance (Liao & Moray, 1993). Thus the MRT-predicted benefits of an auditory offload were contrasted with the single channel theory prediction of no benefits. The results of Experiment 1 (Wickens & Dixon, 2002) clearly indicated significant benefits of both types of mitigation. Autopilot automation removed a task and improved time-sharing by reducing overall demands. Auditory offload re-distributed resource demands across modalities, and enabled better time sharing by capitalizing on multiple resources. However, because of the categorical differences between the two (the autopilot


213

involved total elimination of one task, the auto-alert involved elimination of the visual monitoring component of one task, and the visual reading of another), it was difficult to draw any scientifically meaningful conclusion regarding the relative benefits of the two forms. In Experiment 2 (Wickens, Dixon, & Chang, 2003; Dixon, Wickens, & Chang, 2005) we extended the two mitigation techniques to a two UAV simulation, in which the pilot sat in front of two workstations. The two UAVs were independent of each other. In different conditions, pilots could either control one or two UAVs, and in each of these conditions, the UAVs could be in a baseline configuration, or in either of the two workload mitigation configurations (auditory or autopilot). In all dual UAV conditions, the mitigation configuration of the two UAVs was the same (both autopilot or both auditory). The results were evaluated from both an overall performance and a finegrained attentional modeling perspective. Regarding overall performance the results suggested that the two mitigation techniques were somewhat successful in buffering the dual UAV workload costs on primary mission completion and system monitoring, but that they failed to provide any protection for the TOO surveillance task. That is, flying two UAVs, even with mitigation, will allow only mission critical tasks to be performed, but would sacrifice the capability of the single pilot to perform effective en route surveillance. And even this assumes perfect automation, the issue of imperfection to be considered in the following section. Our analysis of attention models focused on the viability of three classic models of multiple task performance to account for variance in performance between the different conditions, defined by the three different UAV configurations (baseline, auditory, and autopilot) invoked in single and dual UAV control, during low- and high-workload periods of flight. The single channel model (Welford, 1968; Liao & Moray, 1993), briefly considered in the context of Experiment 1, assumes the pilot capable of only one task at a time. If a second task is imposed while a first is ongoing, the latter must be fully delayed till the former is completed. The single resource model predicts that concurrent performance (parallel processing) is possible to the extent that the demand level of each time shared task is reduced, as if all are drawing from a single pool of mental resources of limited capacity (Kahneman, 1973; Sarno & Wickens, 1995). The multiple resource model, as described earlier, assumes that demands may also be offloaded by distributing tasks across resources. Each of these models may be invoked as part of more complex performance models such as those embodied in IMPRINT (Laughery & Corker, 1997), so that testing their relative viability in accounting for data in a realistic simulation is important.

214


We examined the viability of the single channel model to account for the data through two tests (see Wickens et al., 2003 for details). In the ‘‘summing test’’ we predicted the time it would take to perform two tasks on different UAV workstations by summing the single task times. If the actual time to complete both, when one task arrives before the other task is completed, is equal to the sum of the single task times, then the single channel assumptions are upheld. If the sum is greater, then not only is single channel theory upheld, but an added penalty for attention switching between workstations is manifest. If, on the other hand, the actual completion time is less than the sum of single task times, it implies some degree of parallel processing, consistent with (single or multiple) resource sharing. Our analysis revealed that dual UAV control in the baseline condition was well modeled by the single channel model plus a substantial cost for switching between tasks. In the autopilot condition, the data were well modeled simply by single channel theory, without switching cost. Note that this is also consistent with a resource sharing plus switching cost model, where the sharing benefits and switching costs offset each other. Most importantly, the auditory mitigation condition provided evidence for resource sharing. The actual completion time was substantially (4–5 s) less than that predicted by a single channel model. In the second test of single channel theory, the arrival time test applied the classic procedures developed from single channel models (Welford, 1968; Keele, 1973), whereby the completion time of a second arriving task is predicted to be a linearly increasing function of how soon it arrives following the initiation of the first arriving task. That is, every second earlier that the second task arrives, adds one second longer that it has to wait before it has access to the pilots’ single channel information processor, and hence adds one second for its total completion. We selected all instances when a task (TOO or CT inspection, system failure report) on one UAV arrived while a task on the other UAV was ongoing; we computed the inter-arrival time, and then plotted this time against the completion time of the second arriving task. Importantly, this analysis revealed no linear relationship between arrival time and completion time, again casting doubts on the viability of a pure single channel theory to account for all aspects of the data. In Experiment 2, we examined the viability of a single resource model to account for the data by the following procedure. First, we obtained independent estimates of the demand level of each of the single task components, in each of their different manifestations (e.g., demand level of system monitoring, or TOO monitoring, demand level of diagnosing system failures, of zooming and reporting TOOs, etc.). Using a time-line analysis, we


215

then identified the different intervals of time during which all possible combinations of tasks occurred. During each of these intervals, we then predicted the amount of workload the pilot would experience under a single resource model by summing the single task demand values (Sarno & Wickens, 1995; Wickens, 2002). These demand values were then correlated with the actual performance level of the various subtasks, measured within the relevant time interval. These correlations revealed that the single resource model did a modest job in predicting performance: for TOO surveillance performance r ¼ 0:25 (detection time) and r ¼ 0:57 (detection miss rate); for system failure monitoring, r ¼ 0:37 (detection time) and r ¼ 0:41 (detection accuracy). However, for detecting target trajectory-tracking performance (in the baseline and auditory conditions when this was not automated) the correlation was actually in the opposite direction (r ¼ 0:44) such that higher predicted workload was associated with reduced error. We then considered the extent to which some assumptions of multiple resource model might improve the fit. Rather than invoking a full fledged multiple resource model (see Sarno & Wickens, 1995; Wickens, 2002; Horrey & Wickens, 2003), we instead provided one very plausible augmentation to the single resource model predictions described in the proceeding paragraph. Whenever the auditory supported system-monitoring task competed with another task in single UAV performance, the workload penalty (computed by the sum of demands) was reduced by 2.0 (relative to visual system monitoring). Whenever tasks competed within a dual UAV combination, the penalty was reduced by 3.0. Recomputing predicted values in this way substantially improved the prediction of system failure monitoring performance, increasing the correlation for detection time from 0.37 to 0.76, and the correlation for SF miss rate from 0.41 to 0.74. The revised computations had little effect on prediction of performance for the other tasks (tracking and TOO surveillance), suggesting that the multiple resource benefit (or cost for shared visual resources in the baseline) was realized only by the task whose modality was changed. Thus collectively the modeling data suggest that single channel theory may be adequate under high-load dual UAV conditions and when only a single perceptual resource is used, but become less sufficient when load is reduced, and when separate perceptual resources are employed (the former reflecting some of the findings by Liao & Moray, 1993). Under these circumstances assumptions of both single resource theory (demand values) and multiple resource theory (reduced penalties for separate resources) should be incorporated into performance models (see also Sarno & Wickens, 1995; Wickens, Goh, Helleberg, Horrey, & Talleur, 2003).

216


IMPERFECT AUTOMATION: EXPERIMENTS 3 AND 4 AND A METAANALYSIS The evaluations of Experiments 1 and 2 were, in a sense, best case scenarios in which automation functioned perfectly. In the reality of UAVs, the assumption of perfect automation is problematic. Interviews with subject matter experts (Hunter/Shadow pilots), revealed the numerous occasions of encounters with aspects of UAV supervision where the automation did not work as expected, or where onboard system failures thwarted smooth mission functioning. Furthermore, UAVs are often called upon to carry out automated surveillance functions that challenge computer vision, just as they might challenge human vision. In Experiment 3, we selected two qualitatively different aspects of automation that might ‘‘fail,’’ each linked with the two automation-based workload mitigations that were examined in Experiments 1 and 2. We failed the autopilot, leading to unpredictable, but subtle ‘‘drifts’’ of the trajectory off the pre-selected course, and we failed the auto-alert system, in a way that parallels a long history of research on human complacency with automated diagnostic systems (e.g., Parasuraman, Molloy, & Singh, 1993). In the two subsequent Experiments 3 and 4 (Dixon & Wickens, 2003, 2004, 2006) several different conditions were created: baseline (no automation), perfect automation of each aspect (with baseline in the other aspect), perfect automation of both aspects, and six conditions of imperfect automation. Of these six, one had a 70% reliable autopilot, the other five varied reliability of the system failure monitoring automation from 80% to 70% (two versions) to 60% (two versions). (The two versions varied in terms of whether the alert threshold was set to generate more misses or more false alarms, a distinction not addressed in the current chapter; see Dixon & Wickens, 2006). Pilots only flew a single UAV, and the different kinds, availabilities and reliabilities of automation were varied between pilots. Prior to the experiment, each pilot was informed generally as to the level of reliability of any automated system component. One important distinction we make here is between low and high workload. Thus system failure monitoring can be discriminated between lowworkload periods, when pilots are simply monitoring the flight path and the TOO image window, and high workload periods, when system failures occur while the pilot is engaged in a TOO or command target inspection (zooming and panning). Correspondingly, TOO monitoring can be discriminated between low-workload periods, and those high-workload periods when a TOO appears in the image window while the pilot is diagnosing and responding to a system failure.


217

The collective results of Experiments 3 and 4 (see details in Dixon & Wickens, 2003, 2006) revealed that both forms of automation were beneficial when perfect (replicating effects in Experiments 1 and 2). The benefits were most realized in high-workload periods. The results also revealed that unreliability of both forms degraded performance, but such degradation was significantly less for the imperfect autopilot than for the imperfect diagnostic automation, at the equivalent 70% reliability value (i.e., both were designed to ‘‘fail’’ on 30% of the encounters). We believe that this difference may be due to the fact that navigation performance of the UAV trajectory was viewed by our pilots as more critical to mission success (reaching the command targets), than health monitoring, and therefore pilots treated path monitoring as the ‘‘more primary’’ of the tasks. Then, examining only imperfect diagnostic automation, we compared the 80%, 70%, and 60% levels with the baseline level of performance, in essence asking the question: ‘‘how poor can this automation be, before it is worse than no automation at all.’’ The results were clear cut: 80% was better, but 70% and 60% was worse (a finding replicated by a fifth experiment in which visual scanning was measured; Wickens, Dixon, Goh, & Hammer, 2005). Generally, false-alarm prone automation was more disruptive than missprone automation. Two additional conclusions were drawn from the data: (1) imperfect automation costs primarily emerged at higher workload, and (2) these costs (and increasing costs with lower reliability) were borne more by the automated system monitoring task than by the two concurrent tasks (TOO monitoring and trajectory guidance). Such effects can be modeled by a resource model in which resources are allocated away from the automated task toward the mission critical ‘‘primary’’ task (trajectory monitoring). The former suffers the decrement of imperfection, since the pilot was not effectively monitoring the raw data (system gauges), and the costs were more amplified when resources were scarcer under high workload. This resource allocation effect was confirmed when visual scanning was measured in four of the experimental conditions (Wickens et al., 2005). The potential implications of an approximate ‘‘reliability threshold’’ below which automated reliability should not fall, was investigated in the final work described here. We conducted a quasi-meta-analysis (Wickens & Dixon, in press), in which the human performance data were extracted from all available studies that we could find of imperfect diagnostic automation. We only considered studies in which the humans also had perceptual access to the raw data upon which the automation made its diagnosis (e.g., in the current UAV simulations, these ‘‘raw data’’ are reflected by the system

218


failure gauges). For each study we assessed the degree to which performance with the diagnostic aid was better than, equivalent to, or worse than the baseline performance, and regressed this trichotomous measure onto the actual reliability of the aid. This regression is shown in Fig. 2, and reveals the relatively stable linear relationship (correlation r ¼ þ0:64) suggesting that, not surprisingly, combined human-automation diagnostic performance increases linearly with aid reliability. Three characteristics of the study however are less intuitively obvious. First, confirming the trend revealed in Experiments 3 and 4, there is a point below which the availability of the aid leads to worse performance than having no aid at all. This point is at reliability r ¼ 0:71 (95% confidence interval ¼ 7 0.07). Below this cutoff, we use the metaphor of the ‘‘concrete life preserver’’ to characterize the diagnostic aid. That is, the user appears to depend upon it even when better performance would be obtained if it were ignored. Second, we performed a separate regression on those studies that were carried out (like ours) within a dual task context, and found that for this subset, the linear model fit even more strongly (correlation r ¼ þ0:78), a finding that is reasonable if we infer that automation dependence would be greater when human processing resources are made scarce by the dual task requirements. Third, when we examined performance on those concurrent tasks (such as the TOO monitoring in the current experiment rather than the automated diagnostic task), as a function of diagnostic automation

Fig. 2. Regression of Benefits/Costs Relative to Baseline (Heavy Line) on Automation Reliability, with a 95% Confidence Interval (Light Lines). A Single Point may Sometimes Represent More than One Study, which may have Identical Data X-Y Values.


219

reliability, the regression line was essentially flat. This finding indicates that people generally tend to treat diagnostic automated tasks as ‘‘secondary,’’ buffering the primary concurrent tasks from whatever resource demands are imposed by decreasing reliability. Importantly, the collective results described above suggest that pilots do have a tolerance for imperfect automation, as long as this reliability does not drop below perhaps an 80% level and, we note, as long as pilots are prewarned of the nature and source of the imperfections. When reliability drops below, problems may occur. As we saw with regard to autopilot guidance, these problems may not be severe.

CONCLUSION: CHALLENGES FOR A MODEL An effective human performance model of UAV supervision must include both effects of automation and of workload. By ‘‘workload’’ we refer here to the capacity to perform multiple tasks. An approach we present below is to begin at a very course level, and then identify necessary refinements and elaborations or qualifications. The simplest workload model is: WL ¼ N where N is the number of assets (i.e., UAVs). This is consistent with single channel theory, which, as we saw from Experiment 2, was a reasonable approximation for all-visual interfaces. However we also note that a more appropriate model associates N with the number of visually rendered tasks which, in the current simulations was either 3 or 2 (autopilot) for single UAV control, or 6 or 4 (autopilot) for dual UAV control. This provides greater resolution. To elaborate this model, we have noted above, the costs of imperfect automation, may also be revealed as workload costs, although the data suggest that these may reflect a drop in performance of the automated task, more than they increase the resources allocated to that task (and degrade concurrent tasks). A simple and computationally elegant representation here is WL ¼ N=r where r is the reliability of the automated component. Thus, for a single automated component, this relationship captures the rough linearity of performance with reliability shown in Fig. 2.

220


As revealed in Experiments 1 and 2, auditory interfaces can substantially decrease the workload via multiple resources. A computational approach to this would be to decrease the total task workload by a ‘‘modality mix’’ factor ranging from 1.0 (no auditory) to 0.75 (maximum reduction), a factor proportional to the number of channels offloaded from vision to audition. Also as revealed by both the modeling efforts of Experiment 2, and by the emergence of automation unreliability effects primarily at high workload, some accommodation should be allowed for differential resource demands of component tasks. To accomplish this, the model would replace N with SUM D, where D ¼ the demand level, ranging from 0 (fully and reliably automated) to 1. The final challenge of such a model, and its greatest practical implications and value, must be to establish (or recommend) a limitation as to how many UAVs are ‘‘too many,’’ exceeding a workload ‘‘red line’’ where critical performance will fail. With the simplest form of the model, our data from Experiment 2, suggest that 1 UAV (or two non-automated tasks) defines this limit. Extrapolation of these predictions with the complete model remains a work in progress.

ACKNOWLEDGMENT The authors wish to acknowledge the research support of contract #ARMY MAD 6021.000-01 from Micro-Analysis and Design, via General Dynamics and the Army Research Laboratory. Dave Dahn and Marc Gacy were the scientific/technical monitors. We also acknowledge the invaluable software support of Ron Carbonari, and contributions to data collection made by Dervon Chang, Juliana Goh, and Ben Hammer. Dr. Michael Barnes and personnel of the E CO 305th Military intelligence battalion at Ft Huacchucca provided subject matter expertise in assisting us to develop the simulation. The opinions expressed in this chapter are those of the authors and do not necessarily reflect those of the US Army.

REFERENCES Dixon, S., & Wickens, C. D. (2003). Imperfect automation in unmanned aerial vehicle flight control (AHFD-03-17/ MAAD-03-2). Savoy, IL: University of Illinois, Aviation Human Factors Division.


221

Dixon, S., & Wickens, C. D. (2006). Automation reliability in unmanned aerial vehicle flight control: Evaluating a reliance-compliance model of automation dependence in high workload. Human Factors, 48. Dixon, S. R., & Wickens, C. D. (2004). Reliability in automated aids for unmanned aerial vehicle flight control: Evaluating a model of automation dependence in high workload (AHFD-04-5/MAAD-04-1). Savoy, IL: University of Illinois, Aviation Human Factors Division. Dixon, S. R., Wickens, C. D., & Chang, D. (2005). Mission control of unmanned air vehicles: A workload analysis. Human Factors, 47, 479–487. Horrey, W. J., & Wickens, C. D. (2003). Multiple resource modeling of task interference in vehicle control, hazard awareness and in-vehicle task performance. Proceedings of driving assessment 2003: 2nd international driving symposium on human factors in driver assessment, training, and vehicle design, Park City, UT. Kahneman, D. (1973). Attention and effort. Englewood Cliffs, NJ: Prentice-Hall. Keele, S. W. (1973). Attention and human performance. Pacific Palisades, CA: Goodyear Publishing Company. Laughery, K. R., & Corker, K. (1997). Computer modeling and simulation. In: G. Salvendy (Ed.), Handbook of human factors and ergonomics, (2nd ed.) (pp. 1375–1408). New York: Wiley. Liao, J., & Moray, N. (1993). A simulation study of human performance deterioration and mental workload. Le Travail humain, 56(4), 321–344. Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253. Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, & Cybernetics, 30(3), 286–297. Parasuraman, R. M., Molloy, R., & Singh, I. L. (1993). Performance consequences of automation induced ‘‘complacency’’. International Journal of Aviation Psychology, 3, 1– 23. Sarno, K. J., & Wickens, C. D. (1995). The role of multiple resources in predicting time-sharing efficiency. International Journal of Aviation Psychology, 5(1), 107–130. Sheridan, T. (2002). Humans and automation: System design and research issues. New York: Wiley Interscience. Welford, A. T. (1968). Fundamentals of skill. London: Methuen. Wickens, C. D. (2002). Multiple resource and performance prediction. Theoretical Issues in Ergonomic Sciences, 3(2), 159–177. Wickens, C. D., & Dixon, S. (2002). Workload demands of remotely piloted vehicle supervision and control: (I) Single vehicle performance (AHFD-02-10/MAAD-02-1). Savoy, IL: University of Illinois, Aviation Human Factors Division. Wickens, C. D., & Dixon, S. (2006, in press). The benefits of imperfect diagnostic automation: A synthesis of the literature. Theoretical Issues in Ergonomics Sciences. Also available as (Technical Report AHFD-05-1/MAAD-05-1). Savoy, IL: University of Illinois, Aviation Human Factors Division. Wickens, C. D., Dixon, S., & Chang, D. (2003). Using interference models to predict performance in a multiple-task UAV environment – 2 UAVs (AHFD-03-9/MAAD-03-1). Savoy, IL: University of Illinois, Aviation Human Factors Division.

222


Wickens, C. D., Dixon, S., Goh, J., & Hammer, B. (2005). Pilot dependence on imperfect diagnostic automation in simulated UAV flights: An attentional visual scanning analysis (AHFD-05-2/MAAD-05-2). Savoy, IL: University of Illinois, Aviation Human Factors Division. Wickens, C. D., Goh, J., Helleberg, J., Horrey, W., & Talleur, D. A. (2003). Attentional models of multitask pilot performance using advanced display technology. Human Factors, 45(3), 360–380. Wickens, C. D., & Hollands, J. (2000). Engineering psychology and human performance (3rd ed.). Upper Saddle River, NJ: Prentice-Hall.

16. DESIGN OF A MULTI-VEHICLE CONTROL SYSTEM: SYSTEM DESIGN AND USER INTERACTION Shawn A. Weil, Jared Freeman, Jean MacMillan, Cullen D. Jackson, Elizabeth Mauer, Michael J. Patterson and Michael P. Linegang Over the past decade, the use of remotely operated vehicles (ROVs) in the armed services has grown markedly. Unmanned Aerial Vehicles (UAVs) are used for surveillance, reconnaissance, and STRIKE missions; unmanned underwater vehicles (UUVs) are used for mine detection and tracking; and unmanned ground vehicles (UGVs) are used to expand sensor range for improved intelligence gathering.1 These vehicles expand the effectiveness of the military while limiting the risk to human beings. As a consequence of the success and utility of ROVs, their use is expected to increase rapidly. However, the current personnel requirement per vehicle does not scale to the desired level of ROV use. Historically, control of any one of these vehicles has required multiple operators at each stage of a mission (e.g., launch, navigation, recovery). If the current control structure were maintained, the number of operators and analysts would grow with the number of vehicles. This conflicts with the stated military goal of personnel reduction (Clark, 2003).


223

224

SHAWN A. WEIL ET AL.

To minimize the growth of ROV operating crews, innovative techniques are being developed to make UV control more efficient. These techniques rely heavily on automating various ROV functions, including vehicle piloting, sensor selection and calibration, asset and task allocation, and initial data analysis. It is anticipated that automating control of these ROV functions will decrease operator workload to a level that will allow a single human operator to control multiple vehicles. This entails a transformation in the role of the human from operator to supervisor, manager, or commander. With increases in automation come new complexities in user–system interaction. Reduction in situational awareness due to inactivity, over- or under-reliance on automation, and misunderstanding of automation modes can all occur if human interaction is not considered in the design of ROV control systems (Parasuraman & Riley, 1997; Sarter, Woods, & Billings, 1997; see Civil Aviation Authority [2004] for a review of automation in civil aviation). This chapter describes a Relational Knowledge Framework (RKF; Freeman & MacMillan, 2002) designed to guide the design of user interfaces (UIs) in a manner that enables effective management of multiple ROVs and mitigates the pitfalls that automation can introduce. Several examples are described that illustrate the application of this framework to the design of UAV control interfaces.

THE PROBLEM As they are currently conducted, missions by single ROVs consist of several sub-tasks. After a vehicle has been launched, a human operator or a small team is responsible for controlling the flight, navigation, status monitoring, flight and mission alteration, problem diagnosis, communication and coordination with other operators, and often data analysis and interpretation. These tasks are similar in terms of their locus of control (e.g., keyboard and mouse input, joystick, trackball, visual display). ROV operation is a cognitively complex task. Many of the tasks (above) have different information requirements and tap different cognitive skills. A given operator is responsible for some or all of these tasks for a given vehicle. The operator must perform the tasks in sequences that are determined by external events in missions characterized by long periods of inactivity followed by intense spikes in workload. The current methods for controlling a single vehicle on a mission will not easily scale up to an environment that engages tens of ROVs in a single

Design of a Multi-Vehicle Control System

225

mission (Johnson, 2003). Operators would be required to attend to multiple task sequences simultaneously, and to coordinate intensively with other control teams. While this might be possible in periods of low-workload conditions (e.g., during a long ingress or egress), current teams and technology would not be able to perform the mission properly when multiple highworkload tasks overlap (e.g., if all ROVs converged on their targets simultaneously). Without an increase in manning, control accuracy would decrease. With an increase in manning, coordination between teams within missions would dissolve. In either case, the probability of error would rise steeply. This problem will be compounded if the operators are asked to control vehicles of various types (e.g., Firescout, Predator, Global Hawk). These vehicles have different capabilities (sensor types, rates of speed, payloads, responsiveness, etc.), and they may operate in different physical environments (i.e., air, ground, underground, water surface, underwater, ocean floor). Thus, the operator(s) must monitor and manage the state of vehicles with respect to vastly different control constraints, and this will boost operator workload.

SOLUTIONS IN TECHNOLOGY AND TRAINING Two types of solution are often proposed to the problem of human control of multiple ROVs: automation and training. The automation strategy is to offload the burden of work from the human operators to (computing) machines. Under this solution, the responsibility for selecting the proper resource/sensor from an available pool, piloting the vehicle, and perhaps some of the sensor analysis and decision making is accomplished in whole or in part by ‘‘intelligent’’ algorithms. The human operator serves as supervisor or manager of the automation. Among the challenges that must be addressed in the envisioned multivehicle control systems is the choice of the degree, or level of autonomy (LOA; Sheridan, 1992), for each of the sub-tasks for ROV control. Some tasks are well suited to automated control (e.g., maintaining altitude) while others may be stronger candidates for full human control (e.g., weapons deployment). Still other tasks may be best controlled by an intermediate level of automation. For example, the automation may present the human operator with a subset of solutions for a problem, and the operator chooses among them. There are many types of automation, each developed to suit particular task characteristics. However, choosing the right LOA for each ROV task is not a trivial matter.

226


There is a strong desire to build supervisory control systems for multiple, heterogeneous unmanned vehicles that are ‘‘intelligently’’ automated (Johnson, 2003) – that is, automated appropriately for the tasks they perform (Sheridan, 1990). Several terms are associated with schemes to allocate control, including variable autonomy, adaptive autonomy, and mixedinitiative autonomy (for a review, see Inagaki, 2003). These differ in their criteria for determining function allocation and LOA, but they all have in common the trading of control between human operators and automated control systems. An alternative approach to improving the efficiency of ROV control is training. Training strategies often specify that humans develop robust mental models of their systems in order to monitor and control them. However, human operators are generally incapable of building complete mental models of complex systems operating in dynamic situations. Cohen, Parasuraman, and Freeman (1998) proposed a somewhat subtler approach. They posited that expert operators develop mental models of a system to help them discern the operational contexts of which the system is ‘‘cognizant’’ (i.e., can be trusted to perform competently), and the level of accuracy to expect from the system in those contexts it recognizes correctly. Specifically, operators need only partial models of the system. Rough models discriminate the system’s abilities in different tactical situations and are used, in essence, to turn the automation on or off. Fine-grained models are used to assess system accuracy in the potentially small set of tactical situations that the system recognizes well, and in which it is turned on. This lowers the criteria for competency among operators to a more realistic level, and specifies the cognitive problem in a way that supports design and training.

RELATIONAL KNOWLEDGE FRAMEWORK Principled approaches to design are a third strategy for implementing future ROV systems. We developed the RKF to further specify the requirements for operator knowledge (and, thus, for training) and to provide more support for designing usable systems. The RKF emphasizes the role of knowledge concerning relations between, for example, mission plan and mission state, system state and system norms, real-world entities and system entities, and the relative influence of various system inputs on system performance. The framework posits several fundamental classes of human–system interactions (HSIs) for planning and executing control of ‘‘intelligent’’ ROVs, and several relations between these classes that should inform design.


227

Fundamental Classes of Human–System Interaction We propose six fundamental classes of HSI with intelligent control systems for ROVs. These six HSI classes are: system configuration; system input; review of system recommendations; monitoring mission execution; refining system performance; and direct vehicle control. We define and illustrate these classes here and present implications of these classes for design in the next section. 1. System configuration: The human operator must configure the system to determine which functions will apply to a mission and at what levels of precision. For example, the operator may specify the controllers (algorithms) to be used in different mission phases and some speed/accuracy tradeoffs. 2. System input: The operator must provide some or all of the data for the system to process in the specified configuration. Examples include specifying current weather, targets, and other data uniquely available to the operator but not directly to the controlled system. 3. Review recommendations: The human must review system recommendations and accept, adjust, or reject them. For example, when the system generates alternative courses-of-action, the human must select and potentially refine the best choice. 4. Monitor mission execution: The human operator must monitor system execution of the mission. For example, the operator should track the actual routes of ROVs relative to planned routes to ensure that unexpected events are managed and do not significantly impede the mission. 5. Refine system performance: The operator must monitor and refine system performance. Examples include monitoring for sluggish system response and degraded information quality. 6. Direct control: The human must be able to take direct control of vehicles and functions otherwise allocated to the system. This may occur when ROVs are orphaned from their C2 units. Relational Knowledge Types Four types of knowledge drive human performance in the interactions described above. This knowledge concerns mainly the relations between information, objects, actions, and events, and it has several interesting implications for system interface design. The four knowledge types are: 1. Situational awareness: The human operator must understand the relationship between the mission and the plan as well as current system state

228


relative to norms. This requires UI displays that emphasize departures from plans and performance norms as well as system diagnostic aids. 2. Mental models of the system: The operator must understand which inputs will significantly influence the system. This knowledge helps the operator to invest effort in interactions that matter. This requires displays that convey the current sensitivity of the system to different inputs. 3. Translation between representations: The human must understand the correspondence between entities and events in the real world and those in the system. This requires designs that simplify this mapping. For example, icons representing real-world entities should serve not only as situation awareness (SA) cues, but also as interfaces to system parameters for controlling their display. 4. System control: The operator must have expert skills in the buttonology of the control interface for data and vehicles; this demands a humancentered approach to the design of interfaces. Table 1 further illustrates the implications for system interface design imposed by this framework by presenting challenges imposed by these cognitive issues in relation to the six interaction classes. An empirical approach to validating this framework would systematically vary the design features listed in this table to test hypotheses concerning the corresponding cognitive states listed there.

LESSONS FROM IMPLEMENTATION The RKF provides guiding principles for the design of effective control systems. UI designers can use RKF knowledge classes and the associated design implications to maintain awareness of the multiple, interdependent challenges of ROV management and control. The examples below illustrate use of the RKF in systems being developed to allow operator control of multiple heterogeneous ROVs. The initial application of the RKF was for the Mixed-Initiative Control of Automa-Teams (MICA) project, which addressed the control of groups of semi-autonomous vehicles (Freeman & MacMillan, 2002; Linegang, Haimson, MacMillan, & Freeman, 2003). The high-level mission goals and concept of operations (CONOPs) of Unmanned Combat Air Vehicle (UCAV) missions formed the basis for the initial UI design. These tasks included automated system configuration based on mission characteristics, the ability to monitor mission performance relative to objectives and constraints, and the need for

Interaction Class System configuration

Interaction Classes, Cognitive Challenges, and Design Implications. Cognitive Challenge

Design Implication

Understand the functions the system can apply

Present well-categorized, mission-specific function menus Present reminders of mission-specific factors to which the system is insensitive, but which are of known importance to domain experts Represent the margin of error or confidence bounds around system estimates

Understand the conditions under which the system can competently perform Understand the system’s reliability in contexts it ‘‘understands’’ System input

Map real-world events to system data requirements

Map real-world event states to current system values Review system recommendations

Think critically about complex recommendations or premises

Understand how and when to query the system for explanations Understand how manual edits may influence plans

Label parameters using meaningful domain terms. Highlight selected objects on all displays simultaneously to clarify relationships between multiple representations of the same object Allow users to input categories (rather than scalar values) when they tend to categorize real-world events or entities

229

Flag predictions and plans based on low certainty estimates, highlight information gaps, and present alternative plans or assessments Display information sources that are relevant to each known information gap and assumption Provide indicators of system sensitivity to various input parameters in the current context


Table 1.

Interaction Class Monitor mission execution

Cognitive Challenge Understand status of execution relative to the plan Recognize events that should trigger human decisions to alter automation Understand which deviations from the plan have serious consequences

Understand the methods and costs of dynamically re-planning to compensate for emergent problems Refine system performance

Know norms of system performance in the given mission type Discriminate degradation levels that significantly endanger the mission

Direct control

Understand when to transfer control

Understand how to transfer control

Understand how to control entities

Design Implication Display planned route, goals (e.g., targets), and constraints (e.g., Surface-to-Air-Missile sites) Display confidence bounds with respect to route, time, and risks Make explicit the decisions the human must make. Where decisions can be scheduled, present reminders to the operator in a timely manner Represent the impact on mission schedule of delays due to re-planning and impact on success Represent current system performance relative to norms and thresholds given the mission type Represent the impact of current system degradation on mission schedule and outcomes Build and maintain a user-extensible ‘‘tip sheet’’ to document user methods of refining system performance for reference and training Present re-routing tools and other controls automatically in situations in which stakes and opportunities change radically Represent who is in control (the operator or the software), the control switch, and progress toward transferring control (if the process is lengthy) Implement sound user interface design principles for vehicle control and feedback


Understand how to diagnose and work around system malfunctions

230

Table 1. (Continued )


231

operators to understand the output of any automated controller. Specifically, the UI needed to aid the human operator in monitoring mission execution in order to identify perturbations from planned activities, to allow operator control if necessary, and to enable the human operator to effectively trade control of the system with automated control mechanisms. Based on these drivers, Linegang et al. (2003) defined a MICA UI (represented in Fig. 1) that addressed the challenges of mission monitoring and supervisory control of multiple UCAVs. In order to ensure enough flexibility in control for the operator to input higher-order control ‘‘intent’’ to the system, input functionality was integrated into the UI suite (task bar in upper-left side). This control flexibility allowed the operator to quickly assign a range of assets and targets, geographic objectives and constraints, and values for tracking, engaging, and assessing these constraints in response to a changing battle space environment. Similarly, the authors attempted to increase mission monitoring effectiveness by improving the operator’s overall SA. This goal was accomplished in the UI by informing the operator of increases in risk; the synchronization matrix in the upper right of the UI (Fig. 1) makes explicit the risk associated with prosecuting individual targets, and the sources of that risk. Additionally, hierarchical filter displays in the lower right of the UI allowed global performance monitoring of the on-going mission with a ‘‘big picture view,’’ and allowed the operator to drill down to the details of the relative effectiveness of mission progress according to a number of parameters (e.g., targets, friendly assets, geographic areas). These displays allowed the operator to view key events (e.g., kill status) as a function of mission progress, and to select a key event (e.g., kill status for a specified target) on demand or in response to an alert function. In sum, this design should help operators to understand mission execution relative to the plan (synchronization and hierarchical filters), and understand how events will trigger the need for decision making or mission re-planning (synchronization matrix). Mauer, Jackson, Patterson, and Bell (2005) described an extension to the preliminary UI concepts defined by the MICA team to support an ROV team supervisor. This work was conducted in a program2 focused on developing the Mission Control System for Naval Autonomous Vehicles (MCS-NAV). The MCS-NAV system optimizes the allocation of assets and tasks based on objectives and constraints provided by a human operator. In order to assure a high degree of congruence between the environment and the system’s representation of that environment, the System Input class of the RKF specifies that system parameters be described in meaningful domain terms. By interviewing domain experts, the IA team defined these

232

The Conceptual UI for MICA.


Fig. 1.


233

terms. Before each mission, a mission commander provides an operator with a prioritized list of tasks to be completed during the mission. Each item on this Mission Essential Task List (METL) is phrased as an action to be completed (e.g., obtain imagery, deploy weapon) and a recipient (or object) of that action (e.g., a target or a region). To accomplish these tasks, the operator must enter each item from the METL into the system, monitor its progress, and troubleshoot any problems. The METL, thus, defines meaningful terms in this domain. To help operators act on the METL, the design team applied the RKF System Configuration class. This RKF class suggests that presenting wellcategorized, mission-specific groups of tasks allows operators to develop accurate conceptual models of the system and an understanding of those functions that the system can apply. Accordingly, operator tasks were grouped into two primary categories: mission planning and mission monitoring. Tasks in the mission planning category were further segregated into those related to managing tasks, time, and communications. The resulting conceptual design for the MCS-NAV is shown in Fig. 2. Additionally, the System Input class of the RKF identifies the challenge an operator faces in trying to map real-world events to system data requirements. To support this task, the MCS-NAV workstation design allows the operator to highlight a selected object across displays simultaneously to clarify the relationships between multiple representations of the same object. In the example shown in Fig. 3, selecting the task ‘‘Image T27’’ from the ‘‘Tasks’’ tab in the ‘‘Mission Manager’’ group box highlights that row in the table, but it also highlights communications about that task in the ‘‘Communications’’ group box and expands the corresponding timeline in

Fig. 2.

The Conceptual Model for the MCS-NAV Workstation Partitions the Display into Major Classes of Tasks.

234 The Mission Planner Display, with Different Representations of the Same Selection Highlighted Simultaneously to Clarify Relationships.


Fig. 3.


235

the ‘‘Mission Timeline’’ group box. On the Mission Monitor display (not shown here), the asset and its associated target (or region of interest) are similarly highlighted. The examples described above are just a few from the MICA and IA programs that provide concrete instantiations of the RKF. Each element of the UI design was based on the careful study of operator conceptualizations of the task requirements and the need for effective presentation of system state and operator responsibilities. While these designs have not been field tested, they provide the foundation for simultaneous control of multiple ROVs in complex environments by a single supervisor or a small team.

CONCLUSION As the requirements for monitoring and controlling multiple heterogeneous ROVs by human operators become more complex, it will become increasingly important to design systems that are sensitive to the abilities and limitations of human operators. In particular, these systems must address human sensitivity to workload, the impact of perceptual salience on attention and prioritization, the primacy and fallibility of long-term memory, and the complex demands of communications. The RKF has been developed to facilitate effective human-centered design, and in particular to improve human performance in the complex task of ROV operation. The illustrations presented in this chapter are concrete examples of the use of the RKF in the ROV domain. UI design choices were based on analyses of operators’ conceptual models of the problem space. This enabled the design team to achieve greater transparency in the cycle relating the real world, system representations of it, system functions, and their impact on the real world. System input is described in domain relevant terms, higherorder intent and state are represented directly in the interfaces, and system tasks are segregated in alignment with the human conceptualization of the tasks to complete. Control of multiple ROV assets will be achieved only with a high degree of automation. Automation hides critical state and process data from human operators, often productively, but sometimes destructively depending on the interaction of LOA, the mission context, and the operator’s state. The RKF helps designers to understand and foresee the cognitive requirements of users, and the design strategies that can support them in complex interactions with multiple, heterogeneous autonomous vehicles.

236


NOTES 1. The vehicle control environment we envision might be termed Remotely Supervised Vehicles (RSV) or Remotely Commanded Vehicles (RCS). Rather than coin a new acronym, however, we use the term found throughout this book – Remotely Operated Vehicle – to denote the situation in which computers operate or control vehicles, while humans manage their allocation to missions and targets. 2. The Intelligent Autonomy (IA) program sponsored by the Office of Naval Research.

REFERENCES Civil Aviation Authority. (2004). Flight crew reliance on automation. London, UK: Safety Regulation Group, Civil Aviation Authority. Clark, V. (2003). CNO guidance for 2003: Achieving Sea Power 21. Washington, DC: Department of the Navy. Retrieved December 16, 2005, from http://www.chinfo.navy.mil/ navpalib/cno/clark-guidance2003.html. Cohen, M. S., Parasuraman, R., & Freeman, J. T. (1998). Trust in decision aids: What is it and how can it be improved? Proceedings of the 1998 command and control research and technology symposium, Monterey, CA. Freeman, J. T., & MacMillan, J. (2002). Mixed-initiative control of robotic systems. Proceedings of the 2002 command and control research and technology symposium, Monterey, CA. Inagaki, T. (2003). Adaptive automation: Sharing and trading of control. In: E. Hollnagel (Ed.), Handbook of cognitive task design. Mahwah, NJ: Lawrence Erlbaum Associates. Johnson, C. L. (2003). Inverting the control ratio: Human control of large, autonomous teams. Proceedings of workshop on humans and multi-agent systems, Melbourne, Australia, 14 July, 2003. Retrieved from www.traclabs.com/~ cmartin/hmas/wkshp _2003/papers/ Johnson.pdf Linegang, M., Haimson, C., MacMillan, J., & Freeman, J. (2003). Human control in mixedinitiative systems: Lessons from the MICA-SHARC program. Proceedings of the 2003 IEEE international conference on systems, man, and cybernetics, Arlington, VA. Mauer, E., Jackson, C., Patterson, M., & Bell, C. (2005). From operator to supervisor: A display for supervising unmanned vehicle teams. Proceedings of the human systems integration symposium 2005. Alexandria,VA: American Society of Naval Engineers. Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human factors, 39, 230–253. Sarter, N. B., Woods, D., & Billings, C. (1997). Automation surprises. In: G. Salvendy (Ed.), Handbook of human factors and ergonomics. New York. Sheridan, T. B. (1990). Function allocation: Algorithm, alchemy or apostasy. International Journal of Human–Computer Studies, 52, 203–216. Sheridan, T. B. (1992). Telerobotics, automation, and human supervisory control. Cambridge, MA: MIT Press.

17. SCALING-UP HUMAN CONTROL FOR LARGE UAV TEAMS Michael Lewis, Jumpol Polvichai, Katia Sycara and Paul Scerri SCALING-UP HUMAN CONTROL FOR LARGE UAV TEAMS Wide area search munitions (WASMs) are a cross between an unmanned aerial vehicle and a munition. With an impressive array of onboard sensors and autonomous flight capabilities WASMs might play a variety of roles on the modern battle field including reconnaissance, search, battle damage assessment, or communications relay. The first of these high concept munitions, the low cost autonomous attack system (LOCAAS), is a miniature, autonomous powered munition capable of broad area search, identification, and destruction of a range of mobile ground targets. The LOCAAS uses a small turbojet engine capable of powering the vehicle for up to 30 min and laser radar (LADAR) with automatic target recognition (ATR) to identify potential targets. While the LOCAASs were originally designed to operate individually, flying preprogrammed search patterns, the WASM concept envisions artificially intelligent munitions that communicate and coordinate to perform their tasks. Were multiple independent LOCAASs to fly in close proximity, a variety of problems including fratricide, strikes against already Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 237–250 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07017-7

237

238

MICHAEL LEWIS ET AL.

dead targets, suboptimal coverage of the search region, and absence of battle damage assessment might arise. These problems could all be resolved by cooperation among the munitions. The next generation of WASMs are posited to have reliable communication with each other and with manned aircraft and ground forces in their area to allow cooperation and control. These communication channels will be required to transmit data, perhaps including video streams, to human controllers, as well as for inter-WASM coordination. We are developing and testing prototype interfaces for interacting with small WASM teams and developing new approaches to allow human control and coordination to be scaled to large (100–1000) WASM teams. Human control of teams of autonomous machines presents a variety of new human factors problems discussed in Roth, Hanson, Hopkins, Mancuso, and Zacharias (2004). Fully autonomous teams must be programmed in detail before their mission begins. This is typically accomplished using a graphical interface on which a sequence of waypoints are specified (Endo, MacKenzie, & Arkin, 2004; Miller & Parasuraman, 2003) and changes in mission phase and reactive behaviors are associated with some of these waypoints. Programmed behaviors may involve either individual robots (Endo et al., 2004) or a cooperating team (Scerri, Sycara, & Tambe, 2004a; Scerri, Xu, Liao, Lai, Lewis, & Sycara, 2004b). Once the mission is started the human operator may have no further input. Interacting with an executing team offers more possibilities for control. These interactions may redirect the team by changing waypoints, search regions, targets (Cummings, 2004), or otherwise manipulating robot goals. Other avenues to control include altering selected behaviors such as the selection of plays in Playbook (Miller & Parasuraman, 2003), or altering behavioral parameters such as changing the value of a robot’s wanderlust (deviations from a direct path between waypoints) in MissionLab (Endo, et al., 2004). Anticipating the effects of actions and exerting effective control becomes progressively more difficult as the locus of control shifts from observables such as targets to algorithmic parameters. We are currently exploring approaches to controlling teams that combine specifying human roles in team plans, selection among plans, and control of algorithmic parameters as well as manipulation of goals (Scerri et al, 2004a,b ). In this chapter we describe a prototype interface for controlling small (4–8) teams of WASMs that has been evaluated for an AC-130 flank patrol task and will be used in an upcoming P-LOCAAS1 flight test. We then present preliminary results for techniques that may allow operators to control very large UAV teams through reconfiguring coordination algorithm parameters and

Scaling-up Human Control for Large UAV Teams

239

developing transfer of control policies that allow UAVs to adjust their level of autonomy to compensate for variations in operator workload.

SMALL TEAM WASM CONTROL INTERFACE Our user interface (shown in Fig. 1) for controlling small WASM teams was constructed by adding a toolbar, taking advantage of drawing and other display functions of the FalconView (FalconView, 2005) personal flight planning system, a popular flight planning system used by military pilots. The user controls individuals or teams of WASMs by sketching ingress paths, search or jettison regions and other spatially meaningful instructions known as tactical areas of interest (TAIs). When a target is detected the user may be alerted and requested to authorize or abort the attack depending on the rules of engagement. Through a series of dialogs and menu selections the user can select individual WASMs for a task or allow the team to make its own allocation. The FalconView interface communicates with a Lockheed-Martin simulation of LOCAASs and the OneSAF testbed baseline (OTB) (OneSAF, 2004) platoon to brigade level simulation to provide a realistic simulation of the

Fig. 1.

FalconView based WASM control interface.

240


interface’s capabilities for controlling teams of WASMs. Because many platform simulators such as the AC-130 used in our test also use the Distributed Interactive Simulation (DIS) protocol, OTB provides a ground truth server for linking our WASM simulations with other platforms on the simulated battlefield. The simulated WASM broadcasts protocol data units (PDUs) defined by DIS to update its position and pose while listening for PDUs with locations within its sensor cone to detect targets. The laptop presenting the user interface uses custom defined supervisor and weapon state PDUs to convey instructions to the WASMs and monitors WASM PDUs for newly found targets to be added to its display.

CONOPS TEST AND DEVELOPMENT An initial evaluation of the FalconView tasking interface was conducted for WASM conops for flank patrol for an AC-130 aircraft supporting special operations forces on the ground. The AC-130 is a large, lumbering aircraft, vulnerable to attack from the ground. While it has an impressive array of sensors, those sensors are focused directly on the small area of ground to be attacked. In the test scenarios the WASMs were launched as the AC-130 entered the battlespace. The WASMs were intended to protect the flight path of the AC-130 into the area of operations, destroying ground-based threats as required. Once the AC-130 entered a circling pattern around its targets, the WASMs were to set up a perimeter defense, destroying targets of opportunity both to protect the AC-130 and to support the soldiers on the ground. Even under ideal conditions there will be only one human operator on board of the AC-130 responsible for monitoring and controlling the group of WASMs. Hence, high levels of autonomous operation and coordination are required of the WASMs themselves. Fig. 2 shows the configuration of the simulators. Instructors at the Hurlburt Field Special Operations Command training facility flew three scenarios, one training and two with active data collection in an AC-130 simulator. Flight paths and ground situations were played back from previous training missions with instructors filling the navigator and fire control officer positions. In each scenario the gunship flew to an engagement area where it circled attacking a ground target. WASMs were launched and tasked using the FalconView interface which showed tracks for the AC-130, WASMs, and targets detected by either the AC-130 or the WASMs. Depending on the task


Fig. 2.

241

Configuration of simulations.

configuration selected by the controller, WASMs either reported, returned a video image with request to authorize attack, or attacked when a target was detected by ATR. Nine targets were present in the first test scenario. The controllers launched eight WASMs in pairs to kill six of these nine targets. Two of the WASMs ran out of fuel before finding and eliminating their targets. The scenario ran for 27 min. The second scenario was similar. Nine targets were again arrayed in the region of attack and eight WASMs were launched. In scenario 2 seven of the targets were killed with one WASM running out of fuel. The second scenario ran for 30 min. Controllers were debriefed after each mission leading to observations we have classified as dealing with interface improvements, heuristic evaluation, and workload.

INTERFACE IMPROVEMENTS Controllers found the sketch-based targeting interface (Fig. 1) easy to understand and use. Although no accidental launch was observed, the controllers felt a confirm dialog was needed to guard against that eventuality. On several occasions the controllers had wanted to redirect sub groups of munitions rather than targeting them individually or as a team. They expressed a desire for some mechanism such as the control-select convention used in Microsoft products to allow them to designate subteams to perform actions. There was general agreement that the 40 meter bounding box for target-centric

242


commands became too small when the map was zoomed out to observe a large area making it very difficult to select targets. One controller observed that the term ‘‘OK’’ was misleading on the attack dialog that gave the alternatives, ‘‘Attack’’ or ‘‘OK’’. We have renamed this option ‘‘Close’’.

HEURISTIC EVALUATION The controllers commented that the WASMs allowed them to search a much wider area than that covered by the standard AC-130 sensors. On several occasions they launched WASMs for general reconnaissance rather than directing them at potential targets. They felt the WASMs could provide badly needed complementary ISR (intelligence, signals, reconnaissance). Because AC-130s are primarily flown on night missions due to their vulnerability during the day they generally lack the range provided by EO (electro-optical) sensors. The forward looking infra red (FLIR) they use at night has a more limited range and could be supplemented by using WASMs as forward eyes. It was pointed out that camera video would be needed if the WASMs were to be used for battle damage assessment. The controllers liked the feature of showing tracks picked up by the AC-130s sensors on the FalconView interface but felt that tracks picked up by WASMs should be shared with other onboard targeting and navigation systems as well.

WORKLOAD A variety of comments pertaining to workload were recorded during the debriefing. WASM control was felt to impose very little additional workload during the scenarios that were run. Despite this low workload, there were occasions in each scenario when the responsibility for controlling the WASMs was handed off between the navigation and fire control officers. All thought that the workload associated with controlling WASMs would be much higher in scenarios involving interaction with friendly forces on the ground. There was a consensus among the instructors that the electronic warfare officer was the least loaded of the crew and the best candidate for handling an additional task such as WASM control. This test demonstrated that up to eight WASMs could be controlled by a single operator provided the vehicles are granted sufficient autonomy and control consisted of supervision and direction rather than momentto-moment operation. Our sketch-based interface was shown to be easy to


243

learn and use at least for those already familiar with FalconView. A more advanced version of this interface will be used to launch and direct a live P-LOCAAS prototype that will fly a mission with three simulated teammates in summer 2005. In this follow on work simple heuristics and standard search patterns have been replaced by Machinetta (Scerri et al., 2004a,b), a multiagent teamwork infrastructure that provides the ability to instantiate and execute team-oriented plans. This will allow WASMs to perform battle damage assessment for one another, stage simultaneous attacks on a target, and perform other coordinated activities that could multiply the effectiveness of such munitions. Although the P-LOCAAS test flight will use the current sketch-based FalconView interface new interface techniques and control concepts will be needed to control the larger teams of WASMs envisioned by military planners (Vick, Moore, Pirnie, & Stillion, 2001).

HUMAN FACTORS FOR LARGE-SCALE UAV TEAMS While it is feasible for a human to direct and monitor a relatively small number of UAVs using an interface such as ours, the operator rapidly becomes saturated as the number of platforms increases. Miller (2004) analyzed the workload involved in target confirmation requests to authorize UCAV weapons release and concluded that under anticipated detection rates an operator would likely become overloaded controlling as few as 13 UCAVs doing nothing but target confirmation. We have been investigating approaches that might help human commanders control much larger UAV teams. We assume that control through mission planning, redirection or redefinition of tactical regions, changes in plan libraries, or changes in rules of engagement as exercised through the FalconView interface do not pose a threat to operator workload because they are independent of number of UAVs. The tasks of monitoring team performance and intervening when trouble is detected, by contrast, are expected to increase rapidly in difficulty with the size of the team. We do not expect human operators to be able to effectively monitor teams of hundreds of UAVs. Instead, we believe some annunciation scheme is needed to allow the UAVs to draw the operator’s attention to potential problems. To do this, the team must identify situations where human input might be needed and explicitly transfer responsibility for making that decision to a human. These decisions will typically require projections into the future or global judgments that are not considered by the reactive teamwork algorithms. We

244


have identified three types of potential coordination problems likely to be susceptible to detection and resolution: 1. Unfilled task allocations. Role allocation can be allowed to continue, be suspended for some time, or its associated plan can be cancelled. If a human is not available to make the decision, the agent will autonomously suspend allocation for a period. 2. Untasked team members may be symptomatic of the team not effectively positioning resources to achieve current and future objectives. There are two things that can be done when a team member does not have a task for an extended period: do nothing or move the agent to some other physical location. If a human is not available to make a decision, the agent will autonomously decide to do nothing. 3. Unusual plan performance. Team plans and sub-plans, executed by team members to achieve goals and sub-goals will typically have logical conditions indicating when the plan has become unachievable or irrelevant. We currently allow a plan designer to specify an expected length of time and bring to the attention of the human plans that exceed this expected time. When a plan execution does not meet normal performance metrics, there are two things that can be done: cancel the plan or allow it to continue. Because a human may or may not be available and meta-reasoning decisions must be made in a timely manner or the value of the decision is lessened, responsibility for this decision is determined through a transfer-ofcontrol strategy, a pre-planned sequence of actions either transferring control of a meta-reasoning decision to some entity or taking an action to buy time. Mathematical models of transfer-of-control strategies are presented in (Scerri, Pynadath, & Tambe, 2002; Scerri et al, 2004a,b) and capture intuitions such as the increasing appropriateness of terminating a long running plan as it continues to run. We have conducted preliminary experiments to evaluate how the underlying algorithms work in finding potential team problems and dealing with the possibility that a human is not available to make these decisions when they arise. The interfaces were augmented for this experiment with code that made decisions at various lags to simulate human performance. These ‘‘human’’ decisions were made between 5 s and 2 min after control was transferred provided the ‘‘human’’ is not occupied with another task. The experiments involved a team of 80 WASMs operating in a large environment. The primary task of the team was to protect a manned aircraft by finding and destroying surface-to-air missile sites spread around the environment. Half the team spread out across the environment searching for targets while the other half stayed near the manned aircraft destroying

245


surface-to-air sites as they were found near the aircraft. Plans were simple, requiring a single WASM to hit each found target. If a target was not hit within 12 min of being found, this was considered abnormal plan execution and meta-reasoning was invoked. Meta-reasoning was also invoked when a WASM was not allocated to hit any target for 20 min. Finally, metareasoning was invoked when no WASM was available to hit a found target. Two simulated commanders were available to make meta-reasoning decisions. Six different scenarios were used, each differing the number of surface-toair missile sites. Each configuration was run ten times. As the number of missile sites increases, the team will have more to do with the same number of WASMs, thus we can expect more meta-reasoning decisions. Fig. 3 shows that the total number of meta-reasoning decisions does increase slightly with the number of targets. Over the course of a simulation, there were around 100 meta-reasoning decisions or about one per agent and slightly less than one per minute. However only about 20% of these were transferred to a simulated human. The large number of decisions that were made autonomously was primarily because simulated humans were busy and not available to make those decisions, precisely the eventuality the transferof-control strategy was designed to address.

Meta-Reasoning Decisions 1300 1250 Number of Decisions

1200 1150 1100 1050 1000 950 900 850 800 0

Fig. 3.

50

100 Number of Targets

150

200

Meta-reasoning decisions and number of targets (total over 10 trials).

246


PARAMETRIC CONTROL FOR LARGE TEAMS Due to the high computational complexity of coordination, critical coordination algorithms typically use heuristics that are parameterized and need to be tuned for specific domains for best performance. For example, different coordination configurations might be required for different rates of change in the world, individual failure rates or communication bandwidth availability. A coordination configuration specifies parameter values for a team’s coordination algorithms such as the time-to-live for tokens being passed between team members or the expected utility threshold that must be exceeded before sending a message. When several coordination algorithms are used together, e.g., algorithms for task allocation, communication and planning, the performance of one algorithm will likely affect the performance of the other algorithms, thus tuning parameters of the individual algorithms must be performed together. Due to the non-determinism of environments and coordination algorithms and the sensitivity of performance to circumstances these relationships are highly non-linear. They are also highly variable even for the same configuration. In order for operators to configure and control teams effectively we are developing methods to create a team performance model to capture the relation between the environment, team configuration parameters, and measures of performance. To create this concise model from data we are using genetic algorithms to learn a dynamic neural network (Polvichai & Khosla, 2002). Searching the team performance model to find the combinations of input parameters that result in the desired output allows operators to specify performance tradeoffs and rapidly find a configuration that best meets those constraints. Since not all parameters are configurable, e.g., the observability of the domain cannot be changed during execution, we cannot simply use back propagation of the neural network to find input parameters that meet our output requirements. Instead we perform a search over the changeable configuration parameters to find a configuration that best meets the required performance tradeoffs. In initial experiments we have demonstrated the ability of an operator to control the global behavior of a large team using a team performance model to guide actions. The user configures the team at the start of the mission. Performance measures from the simulation are graphically displayed on the user interface at every time step. When performance changes are requested the offline features of the team performance model are used to find suitable reconfigurations. The user interface and reconfiguration assistance were evaluated


247

over 10 scenarios. Scenarios were selected to provide situations that would require users to reconfigure their team in order to meet performance targets. For example, in a mission involving a very large team of 300 agents the user might be requested at some point in the mission to reduce the number of messages per agent or increase the number of plans instantiated. Performance measures are recorded throughout the execution. Each scenario was run for 250 time steps, with each step taking 5 s. The data presented here represent 4 h of runtime with a user in the loop. One scenario with a team of 200 agents is shown in Fig. 4. For the first intervention, the user is asked to

Fig. 4. Six performance measures are shown with an initial configuration and three reconfigurations during execution. The light lines show the values predicted by the model. The dark lines show the average values obtained. The jagged lines show the observed values.

248


increase level of rewards (goal linked outcomes such as target identifications) obtained by the team disregarding other performance measures. Using the output-to-input feature of the team performance model the user finds a new coordination configuration that increases reward performance and reconfigures the team. In the second intervention the operator must reduce network communication bandwidth by limiting the time-to-live for information tokens to 2 hops requiring further team reconfiguration to lessen the degradation in performance. For the final intervention, the user must reconfigure the team to increase the accumulation of ‘‘rewards’’. Results for six of the performance measures are shown in Fig. 4. The bold lines show average values for the configured system while the lighter lines indicate the values predicted by the output-to-input mappings from the team performance model. The jagged lines show the moment-to-moment variation in the actual performance measures. Despite the high variability of team performance measures the model accurately predicts the direction of effects of reconfiguration on average performance values across all six measures. By demonstrating the team performance model’s effectiveness for predicting the effects of team configurations these tests demonstrate the potential of our approach for both the initial configuration of UAV teams and supervisory control over executing teams.

CONCLUSION In this chapter we have examined difficulties involved in controlling large teams of WASMs. Some forms of interaction, particularly those that specify goals such as waypoints or tactical areas of interest, appear largely immune to scaling problems. Others, such as changes in the plans to be executed whether directly controlled as in Playbook (Miller & Parasuraman, 2003) or indirectly through mission phases (Endo et al., 2004) are design time problems that if properly engineered should impose minimal workload at runtime. There appears to be a consensus among researchers (Miller, 2004; Nickerson & Skiena, 2005; Scerri et al., 2004a,b; Crandall, Nielsen, & Goodrich, 2003) that monitoring should be directed by annunciation from the platforms and that strategies that require servicing of individual robots are most likely to limit the size of teams. One alternative proposed by Nickerson and Skiena (2005) is to control UAVs through call centers. By sharing service requests among operators rather than assigning them fixed teams, the call center could balance the load as subteams of UAVs move between areas of low and high target densities. This solution, however,


249

could aggravate problems with situation awareness that are likely to arise as operators are forced to shift attention among a large number of platforms. This is especially significant because the advantages human control is supposed to bring the team are largely associated with providing context whether it is considering the potential for collateral damage in authorizing an attack or the decision to abandon a target because there are unlikely to be sufficient forces remaining after current attacks are completed.

NOTES 1. The P-LOCAAS or powered low cost autonomous attack system is the prototypical wide area search munition.

ACKNOWLEDGMENT This work was supported by AFRL/MN and AFOSR grant F49620-01-10542. We would like to acknowledge the important contributions of our collaborators, Rob Murphy and Kevin O’Neal of AFRL/MN, Doug Zimmerer and Rolan Tapia of Lockheed Martin, and University of Pittsburgh research programmers, Joe Manojlovich and Paul Arezina.

REFERENCES Crandall, J., Nielsen, C., and Goodrich, M. (2003). Towards predicting robot team performance. Proceedings of IEEE international conference on systems, man, and cybernetics, 906–911. Cummings, M. (2004). Automation bias in intelligent time critical decision support systems. AIAA intelligent systems conference, AIAA, 2004-6313. Endo, Y., MacKenzie, D., & Arkin, R. (2004). Usability evaluation of high-level user assistance for robot mission specification. IEEE Transactions on Systems, Man, and Cybernetics, 34(2), 168–180. FalconView Website. Retrieved March 14, 2005 from http://www.falconview.org Miller, C. (2004). Modeling human workload limitations on multiple UAV control. Proceedings of the 48th annual meeting of the Human Factors and Ergonomics Society, Santa Monica, CA: HFES, 526. Miller, C., and Parasuraman, R. (2003). Beyond levels of automation: An architecture for more flexible human-automation collaboration. Proceedings of the 47th annual meeting of the Human Factors and Ergonomics Society, Santa Monica, CA: HFES, 182–186. Nickerson, J. V., and Skiena, S. S. (2005, January 3–6). Attention and Communication: Decision scenarios for teleoperating robots. Proceedings of the 38th annual Hawaii international conference on system sciences.

250


OneSAF Website. Retrieved January 18, 2004 from http://www.onesaf.org Polvichai, J., and Khosla, P. (2002). An evolutionary behavior programming system with dynamic networks for mobile robots in dynamic environments. In Proceedings of 2002 IEEE/RSJ international conference on intelligent robots and system, 1, 978–983. Roth, E., Hanson, M., Hopkins, C., Mancuso, V., and Zacharias, G. (2004). Human in the loop evaluation of a mixed-initiative system for planning and control of multiple UAV teams. Proceedings of the Human Factors and Ergonomics Society 48th annual meeting, 280–284. Scerri, P., Xu, Y., Liao, E., Lai, G., Lewis, M., & Sycara, K. (2004). Coordinating large groups of wide area search munitions. In: D. Grundel, R. Murphey & P. Pandalos (Eds), Recent developments in cooperative control and optimization. Singapore: World Scientific. Scerri, P., Sycara, K., and Tambe, M. (2004). Adjustable Autonomy in the Context of Coordination. In AIAA 3rd unmanned unlimited technical conference, workshop and exhibit. Scerri, P., Pynadath, D., & Tambe, M. (2002). Towards adjustable autonomy for the real world. Journal of Artificial Intelligence Research, 17, 171–228. Vick, A., Moore, R., Pirnie, B., & Stillion, J. (2001). Aerospace operations against elusive ground target. Santa Monica, CA: Rand Corp.

18. DELEGATION INTERFACES FOR HUMAN SUPERVISION OF MULTIPLE UNMANNED VEHICLES: THEORY, EXPERIMENTS, AND PRACTICAL APPLICATIONS Raja Parasuraman and Christopher Miller Many different types of ‘‘unmanned’’ or ‘‘remotely operated’’ vehicles (ROVs) are being developed for use in aerial, ground, and underwater environments. Despite the term ‘‘unmanned,’’ controlling such robotic vehicles requires considerable manpower, from those operating the ROVs to users of the information provided by ROVs and to command and control personnel. The Global Hawk unmanned air vehicle (UAV), for example, requires a team of at least three personnel for the control of the vehicle, including the UAV ‘‘pilot,’’ a payload operator, and a mission planner. Earlier generation ROVs such as the Predator required an even larger team. The required team composition for other types of military ROVs, such as unmanned ground vehicles (UGVs), is still the subject of analysis and debate. For example, the U.S. Army’s Future Combat Systems (FCS) will incorporate numerous ROVs, with the type of ROV and team size being reconfigurable components tailored to specific combat missions. The goal of these systems is to extend manned capabilities and act as ‘‘force multipliers’’ Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 251–266 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07018-9

251

252

RAJA PARASURAMAN AND CHRISTOPHER MILLER

(Barnes, Parasuraman, & Cosenzo, 2005). More long-term goals are proposals to create large groups of N ROVs that can operate largely autonomously, while being overseen by M operators, where N4M, as in DARPA’s Mixed Initiative Control of Automata Teams (MICA) program (MICA, 2002). These trends in ROV development have created a need for understanding how a small team of operators can effectively control a large number of ROVs of different types and capabilities. These robotic vehicles can move and navigate with varying degrees of autonomy. They can also engage in goal-directed behaviors and communicate with and provide feedback to human supervisors. Human supervision is necessary, not simply to operate the ROVs, but also to manage unexpected events and to ensure that mission goals are met. The functional design of the interface between human supervisors and robotic agents is therefore a key issue for effective teaming, communication, and mission success. In this chapter, we outline a framework for human supervision of multiple ROVs based on the concept of delegation. Delegation-type interfaces represent a form of adaptive or, more accurately, adaptable automation (Opperman, 1994; Parasuraman, Sheridan, & Wickens, 2000; Scerbo, 2001). We propose that such interfaces can retain the benefits of automation while minimizing some of its costs (Miller & Parasuraman, in press). We describe various applications of the delegation interface concept in the context of unmanned vehicle operations. Finally, we outline the results of laboratory experiments examining the efficacy of delegation interfaces for human supervision of multiple robots.

THEORY OF DELEGATION INTERFACES FOR HUMAN SUPERVISION OF AUTOMATION A fundamental issue driving much of the current research is the design of the interface between humans and ROVs. Autonomous robots are sufficiently different from most computer systems as to require new research and design principles (Adams & Skubic, 2005; Kiesler & Hinds, 2004). Previous work on coordination between humans and automated agents has revealed both benefits and costs of automation for system performance (Parasuraman & Riley, 1997). Automation is clearly essential for the operation of many complex human–machine systems. But in some circumstances automation can also lead to novel problems for operators. Automation can increase

Delegation Interfaces for Human Supervision of Multiple UAVs

253

workload and training requirements, impair situation awareness and, when particular events co-occur in combination with poorly designed interfaces, lead to accidents (e.g., Degani, 2004; Parasuraman & Riley, 1997). Retaining the benefits of automation while minimizing its costs and hazards may require the interface between humans and robotic agents to be adaptive or adaptable, rather than fixed and static (Parasuraman et al., 2000). The performance benefit of adaptive compared to static automation is well documented (e.g., Kaber & Endsley, 2004; Parasuraman, Mouloua, & Molloy, 1996). However, if adaptation is executed without user approval or knowledge, the cost of system unpredictability may outweigh the benefit of automation (Billings & Woods, 1994). If users explicitly task automation at times of their choosing, however, automated behaviors should be more transparent – as long as the automation adheres to their tasking instructions. However, involving the human operator in making decisions and issuing commands about when and what to automate can increase the operator’s workload. Thus, there is a tradeoff between increased unpredictability versus increased workload in systems where automation is invoked by the system or by the user, respectively. The key idea behind delegation is that the supervisor generally has flexibility along this tradeoff dimension – that is, can delegate bigger, coarser-grained tasks or smaller, more precise ones with more or less explicit instruction about their performance and that this can, in turn, result in selecting appropriate ‘‘operating points,’’ depending on context and task demands. Delegation-type interfaces, similarly, allow for automation to be implemented at a flexible and variable point in this tradeoff space. Delegation in this sense is similar to Sheridan’s (1987) concept of real-time supervisory control, but seeks to provide a mechanism for human/machine interactions which is richer and more flexible than has traditionally been the case in human/machine supervisory control systems (but not richer than Sheridan, 1987, initially envisioned). Delegation architectures provide highly flexible methods for the human supervisor to declare goals and provide instructions and thereby choose how much or how little autonomy to give to automation on an instance-by-instance basis. Three parameters of the human–machine system are important in designing delegation interfaces. The first is the competency of the system, or its ability to achieve correct behavior in context. A system is more competent whenever it provides correct behaviors more frequently or in a greater number of contexts. A second important parameter is the workload associated with the human operator’s use of the delegation interface. The third parameter is the unpredictability of the system to the operator, which refers

254


to the inability of the human to know exactly what the automation will do when. Unpredictability is a consequence of the human not personally taking all actions in the system – of not being ‘‘in control’’ directly and immediately. Expending workload (especially mental workload) to monitor and understand a system reduces its unpredictability to the user, hence unpredictability is generally inversely correlated with workload and with those specific aspects of situation awareness which pertain to the understanding of the automation behaviors and the system functions controlled. Good system and interface design, including improved feedback to the user about automation states and behaviors (Parasuraman et al., 2000), as well as other options such as increased reporting requirements and good hiring and training practices, can serve to reduce unpredictability. However, in general, any form of task delegation – whether to automation or other humans – must necessarily result in a degree of unpredictability if it offloads tasks (and does not replace operators’ workload with other tasks – including monitoring and managing the offloaded tasks). These three parameters define a trade off space within which a given level of competency can be achieved through some mix of workload and unpredictability. A user may reduce workload by allocating some functions to automation, but only at the expense of increased unpredictability (at least with regards to those functions); conversely, reducing unpredictability by having the user perform functions increases workload. Alternate designs for a level of competency represent different mixes of workload and unpredictability. It is sometimes possible to reduce both workload and unpredictability for a given level of competency through better design. It is also, ironically, entirely possible to increase both workload and unpredictability without achieving increased competency, which may be a feature of some current UAVs.

IMPLEMENTING DELEGATION INTERFACES FOR ROV SUPERVISION We have developed a series of prototype delegation systems for a variety of ROV applications. The first delegation interface was designed for the purpose of enabling ground-based, pre-flight mission planning for UAVs (Miller & Parasuraman, in press). With this interface (shown in Fig. 1), the user could call a high-level ‘‘play,’’ which corresponded to a single mission type (e.g., Airfield Denial), but also had extensive capability to stipulate the


Fig. 1.

255

Delegation Interface for Ground-Based UAV Mission Planning.

method and procedure for doing Airfield Denial by filling in specific variable values (i.e., which specific airfield was to be attacked, what UAVs were to be used, where they should rendezvous, etc.) and by stipulating which submethods and optional task path branches would be used and which would not (e.g., whether or not this instance of Airfield Denial would include an optional ‘‘Suppress Defenses’’ sub-task). One lesson we learned early on was that a variety of user interaction styles could and should be supported by the delegation interface, and that each of them should reference and be integrated by the central task model. The top portion of this interface consists of a window that allows detailed navigation and visualization of the play’s task structure – conveying both sequential (via left–to–right and parallel) and functional (via hierarchical drill down) relationships of tasks – what we called a Task Decomposition View. Such an interface supports detailed interactions with the task structures of the play and allows the user to visualize and/or constrain those structures. Here, the user directly manipulates tasks by asserting which ones are to be included and which avoided.

256


A second application involved controlling the actions of ground robots whose primary duty was to enter a building in a group and map it out, sending back video images to a console (Goldman, Haigh, Musliner, & Pelican, 2000) (see Fig. 2). This domain had comparatively few alternate plays. Thus, the primary play-relevant components for this interface are located in the upper left hand corner and are a simple, relatively ‘‘flat’’ tree structure of alternate deployment and surveillance plays organized by mission phase. Given the time pressure imposed in these operations, the user did not need to have the ability to ‘‘drill down’’ and impose more specific instructions about exactly how a ‘‘Group Deploy’’ task was to be performed. Instead, users simply activated general behaviors for the entire team of robots on a single-action basis. Hence, the building map and video were more important and were expected to be given substantially more space during typical use. Additional interactions and visualizations were possible – including direct, waypoint instructions to individual robots and navigation through their resulting video images – through additional popup windows.

Fig. 2.

Delegation Interface for Ground Robots Performing a Building Exploration and Mapping Task.


Fig. 3.

257

Play Calling Interface for the RoboFlag Simulation.

A similar design was developed to support interactions with game-playing robots in the RoboFlag simulation (Parasuraman, Galster, Squire, Furukawa, & Miller, 2005), shown in Fig. 3. Again, as for the building exploring robots, there were comparatively few ‘‘plays’’ with simple decomposition structures that could be commanded by the human operator and, as a result, the ‘‘play’’ elements of the interface are simple and occupy very little of the overall space – only the three buttons in the upper right hand corner in this figure. Because plays could be called with regards to any number of the blue team’s robots, however, and because waypoint commanding was available to users in this domain, the overall map display was an interactive feature of this interface. Users could assert waypoint commands for individual robots via the map display and were required to select one or more robots to apply a play command prior to asserting that command. Another important element of this delegation application is that it involves not pre-mission planning but

258


real time control, which increases the importance of providing an integrated monitoring and commanding environment. The final application we describe has two different interfaces (Figs. 4 and 5). We call this interface a Playbook (trademark, Smart Information Flow Technologies, LLC) due to its reliance on the metaphor of a sports team’s playbook of shared plays. The application for this interface involves the control of multiple, heterogeneous UAVs by dismounted infantry or special forces troops – an extremely challenging problem due to the limited interface hardware available (to be carried), the concurrent workload, and the training burden imposed by requiring operators to be able to interact with multiple types of UAVs. Hence, we adopted a less direct control or ‘‘play calling’’ perspective in the design of this interface and, instead, viewed the user as ‘‘requesting services’’ from a pool of available UAVs that are ultimately the responsibility of some other actor. Nevertheless, the operator of this Playbook application may request that a specific service be performed using one or more of the UAVs (which may or may not be specified). The Playbook is used to specify the initial service request (in terms of a play which may contain stipulations on how it is to be conducted) but it is also used, in conjunction with a pre-existing Ground Control Station (GCS) to actively control or manage the UAV(s) during the execution of the service for which they have been remanded to the requestor. For the most part, control is performed by the Playbook and GCS automation, but there are opportunities for human intervention, countermanding, and revision. We developed two alternate interfaces for this Playbook. The first, shown in Fig. 4, was designed for use on a laptop or tablet PC, where there is both sufficient screen space and enough time for the user to develop or inspect a more detailed plan/request for the UAV(s) to perform. The user interacts with this interface to first call a play from a list, via a pull down menu in the upper left hand window of the screen illustrated in the upper left. Once a play is selected, the window in the lower left hand corner provides a set of constraints or options pertinent to that play. Some of these constraints (as indicated by an arrow) must be filled in by the user – for example, for this ‘‘Overwatch Request’’ play (a type of sustained surveillance), the user must stipulate the target and a radius to define the area to be watched. The user may, but is not required to, also stipulate any of the other available parameters including the specific type of UAV platform to be used, the beginning/end times for surveillance, the degree of stealth to be used in performing the play, etc. While the above interface embodies many design manipulations to speed the requesting of a service, it is still too large and too time consuming to be


Laptop or Tablet PC Version for Dismounted Infantry UAV Play Calling.

259

Fig. 4.

260


Select a Play …and call it

…and either accept (command execution)

Fig. 5.

Review/Edit Default Parameters

…review/edit execution

…or interact with map to insert new targets

A Hand-Held Display for Dismounted Infantry UAV Play Calling.

practical for many kinds of dismounted infantry or special forces operations. For example, one scenario we have been considering is the ability for a platform sergeant who is a ‘‘UAV operator’’ in addition to other duties, to call for surveillance of the approaches to a building while hotly pursuing saboteurs. Comprehensive reviewing of plans and provision of constraint details is probably beyond the time available for this user. Furthermore, the user probably does not want to carry the weight (or take the time to unpack and set up) a device large enough to present the screen illustrated in Fig. 4. For this sort of operation, we have begun development of a PDA-based version of the playbook, as illustrated in Fig. 5. Here, the user calls a play (requests a service) from a short pull down menu of available plays – which may have been configured a priori as likely to be necessary in this mission. Once a play has been selected, a reduced set of parameter values is presented to the user (as illustrated in the middle figure). Again, a few of these must be filled in by the user, but most include context-sensitive default values which may, but do not have to be, edited. From this screen, the user can either immediately command execution or can navigate to other screens to review the proposed route or create alternate target points. This interface was designed to minimize display hardware requirements and to maximize the speed with which service requests can be issued. As a result, comparatively


261

little support is provided for inspecting, customizing or negotiating about plans. This particular emphasis on speed over specificity is an artifact of our focus on the design scenario described above and may need to be revised in the future. Nevertheless, the contexts of use dictate a somewhat different form of delegation. In a pre-mission or out-of-the-fight context, delegation may well entail more prolonged discussions between supervisor and subordinates about what is feasible, unfeasible and optimal. On the other hand, in the heat of battle or pursuit, delegation may more frequently take the form of crisp orders that need to either be obeyed or declared infeasible. Here, we have leaned toward the latter extreme and have designed a highly portable, PDA-based interface that enables users to command or request a complex, multi-vehicle surveillance plan for a target of their choosing with as few as three button clicks in well under 15 s.

THE EFFICACY OF DELEGATION INTERFACES: EXPERIMENTS WITH ROBOFLAG The applications discussed previously indicate the diverse range of possibilities for using delegation interfaces for human supervision of robotic agents and ROVs. In this final section, we present the results of recent laboratory experiments that have examined the efficacy of delegation interfaces for flexible and effective supervision of multiple autonomous agents. Empirical studies of the effectiveness of different human–robot interfaces are still relatively rare. Here we briefly discuss the results of experiments that examined the use of a simple delegation interface with the RoboFlag simulation environment (Parasuraman et al., 2005). These studies represent the initial validation of the delegation concept in the context of human supervision of multiple robots. RoboFlag is a simulation of hardware robots that have been designed to move and navigate autonomously, exhibit cooperative behaviors, and act as a team with other robots to pursue goals in games such as capture the flag and soccer (Campbell, et al., 2003). In the experiments briefly described here, we modified the RoboFlag simulation to emulate a typical ROV mission involving a single operator managing a team of robots (Fig. 3). The mission goal was to send the robots from a home area into opponent territory populated with red team robots of equal size, access and obtain a specified target (the flag), and return home as quickly as possible with minimum loss of assets. Individual human operators supervised up to eight robots via the interface illustrated in Fig. 3 and using automated behaviors

262


or plays as well as manual (waypoint) control, in pursuing the goal. Plays directed robots to engage in autonomous, cooperative behaviors in pursuit of a sub-goal; e.g., in ‘‘Patrol Border,’’ a user-selected number of blue team robots went to the border area, spaced themselves appropriately along the length of the border, and attempted to prevent red team robots from crossing. Measures of performance included mission success rate, mission execution time, strategy use (plays vs. manual control), subjective mental workload, and global situation awareness. One of the postulated benefits of delegation-type interfaces is that they allow for flexible use of automation in response to unexpected changes in task demands, while keeping the operator’s mental workload in managing the automation within a reasonable range. We therefore varied two sources of task demand: (1) the adversary ‘‘posture,’’ in which the opponent engagement style was changed unpredictably to be offensive, defensive, or mixed, and (2) environmental observability, by varying the effective visual range of each robotic vehicle under the control of the operator. This experiment showed that the delegation interface allowed for effective user supervision of robots, as evidenced by the number of missions successfully completed and time for mission execution. Only a few of the findings can be highlighted here, though a more complete report can be found in Parasuraman et al. (2005). First, participants had perfect mission success when the enemy robots were in a defensive posture and made no attempt to capture the participant’s flag and, thus, could not win. All such trials were eventually won by the participant, but overcoming the defensive posture required more time (see Fig. 6). Second, significantly fewer missions were successfully completed when the opponent posture was mixed rather than offensive. Nevertheless, users still had moderate success (about 62%) and relatively short mission completion times (about 51 s) in the mixed posture condition, which was the most challenging because of increased opponent uncertainty. These findings suggest that delegation-type interfaces allowed users to respond effectively to unpredictable changes in opponent posture by tasking robots appropriately. A second experiment confirmed these results and also showed that they were associated with the flexibility afforded by the delegation interface as implemented in RoboFlag. Specifically, we compared the flexible delegation approach to fixed delegation approaches, by providing users: (1) only manual control or (2) only automated plays or, (3) both types of control, under the same varying adversary postures (offensive, defensive, mixed) manipulated in Experiment 1. We hypothesized that the use of the delegation interface would afford users maximum flexibility, allowing them to decide when workload was high (and therefore to use automation), or when the


263

Mission Success Rate (%) and Mission Completion Time (s)

140 120 Mission Success Rate 100

Mission Completion Time

80 60 40 20 0 Offensive

Defensive

Mixed

Fig. 6. Effects of Opponent Robot Team Posture (Offensive, Defensive, Mixed) on Mission Success Rate and Mission Completion Time in the RoboFlag Simulation.

automation was ineffective. The mission performance measures supported this hypothesis. Additionally, the flexible delegation interface allowed users to mount a more effective response to variable opponent postures than did the static control conditions (manual or automated). However, there was some cost of such flexibility, in terms of the operator’s workload. As Fig. 7 shows, workload was rated higher when participants had flexible access to both manual control and automated plays, particularly in the defensive posture and to a smaller degree in the mixed condition. However, the increase in workload in the flexible condition was quite small, and the overall workload was in the moderate range (50 out of 100). This suggests that the cost of flexibility was not so large as to jeopardize the user’s ‘‘workload margin’’ in case of any unexpected increase in other task demands. These findings show that flexible automation usage is frequently desirable rather than the extremes of full or no automation. In these cases, systems must be designed for an appropriate relationship between operators and automation allowing both parties to share responsibility, authority and autonomy over many work behaviors in a safe, efficient, and reliable fashion. The results of the experiments by Parasuraman et al. (2005) provide empirical evidence for the efficacy of delegation interfaces in human supervision of multiple autonomous robots. They also provide an initial analysis of when various dimensions of delegation flexibility are useful in supervisory control.

264

RAJA PARASURAMAN AND CHRISTOPHER MILLER 100

80

Manual Automated Both

60

40

20

0

Offensive

Defensive

Mixed

Opponent Posture

Fig. 7. Subjective Workload as a Function of Control Type (Manual Control Only; Automated Plays Only; or Both Manual Control and Automated Plays) and Opponent Posture.

CONCLUSION Effective human supervision of multiple autonomous agents requires approaches to supervisory control that preserve the benefits of high-level automation while providing the user the flexibility to tailor the automation depending on contextual and task demands. The concept of delegation provides a framework for human supervision of multiple ROVs. Delegationtype interfaces allow the user sufficient flexibility in supervision of multiple ROVs while keeping workload within a manageable range. Four applications of delegation interfaces to different tasks in human–robot teaming are described. The results of laboratory experiments examining the effects of delegation interfaces on human–system performance during multiple robot use provide the initial empirical evidence for the efficacy of this interface concept.

ACKNOWLEDGEMENT We thank Harry Funk, Scott Galster, Karen Haigh, Hiroshi Furukawa, Robert Goldman, Dave Musliner, Billy Pate, Michael Pelican, Peter Squire,


265

and Peggy Wu for their contributions and comments. This work was supported by: the DARPA MICA Program (subcontract to Cornell University); an ARL grant under the Adaptive Decision Architectures Program of the Collaborative Technologies Alliance; an Initiatives grant from the Honeywell Technology Center; Honeywell’s subcontract to Carnegie Mellon University’s Tactical Mobile Robots program for DARPA; and a Small Business Innovation Research grant from DARPA/IXO through the U.S. Army’s Redstone Arsenal under contract to Geneva Aerospace.

REFERENCES Adams, J. A., & Skubic, M. (2005). Introduction, special issue on human–robot interaction. IEEE Transactions on Systems, Man, and Cybernetics – Part A: Systems and Humans, 35, 433–437. Barnes, M., Parasuraman, R., & Cosenzo, K. (2005). Adaptive automation for robotic military systems. Technical Report, ARL-HRED. Aberdeen, MD: ARL. Billings, C. E., & Woods, D. (1994). Concerns about adaptive automation in aviation systems. In: R. Parasuraman & M. Mouloua (Eds), Automation an human performance: Current research trends (pp. 264–269). Hillsdale, NJ: Erlbaum. Campbell, M., D’Andrea, R., Schneider, D., Chaudhry, A., Waydo, S., Sullivan, J., Veverka, J., & Klochko, A. (2003). RoboFlag games using systems based hierarchical control. In: Proceedings of the American Controls Conference, (pp. 661–666). Degani, A. (2004). Taming Hal: Designing interfaces beyond 2001. New York: Palgrave. Goldman, R., Haigh, K., Musliner, D., & Pelican, M. (2000). MACBeth; A multi-agent, constraint-based planner. In: AAAI Workshop on Constraints and AI Planning, (pp. 1–7). Kaber, D. B., & Endsley, M. R. (2004). The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task. Theoretical Issues in Ergonomics Science, 5, 113–153. Kiesler, S., & Hinds, S. P. (2004). Special issue on human-robot interaction. Human Computer Interaction, 19, 1–2. MICA. (2002). Mixed-initiative control of automa-teams. Retrieved October 1, 2005 from http:// .dtsn.darpa.mil/ixo/programdetail.asp?progid=7&action=challenges Miller, C., & Parasuraman, R. (in press). Designing for flexible interaction between humans and automation: Delegation interfaces for supervisory control. Human Factors. Opperman, R. (1994). Adaptive user support. Hillsdale, NJ: Erlbaum. Parasuraman, R., Galster, S., Squire, P., Furukawa, H., & Miller, C. (2005). A flexible delegation-type interface enhances system performance in human supervision of multiple robots: Empirical studies with RoboFlag. IEEE Transactions on Systems, Man, and Cybernetics. Part A. Systems and Humans, 35, 481–493. Parasuraman, R., Mouloua, M., & Molloy, R. (1996). Effects of adaptive task allocation on monitoring of automated systems. Human Factors, 38, 665–679. Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39, 230–253.

266


Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics – Part A: Systems and Humans, 30, 286–297. Scerbo, M. (2001). Adaptive automation. In: W. Karwowski (Ed.), International encyclopedia of human factors and ergonomics. London: Taylor and Francis, Inc. Sheridan, T. (1987). Supervisory control. In: G. Salvendy (Ed.), Handbook of human factors (pp. 1244–1268). New York: Wiley.

19. OPERATIONAL ANALYSIS AND PERFORMANCE MODELING FOR THE CONTROL OF MULTIPLE UNINHABITED AERIAL VEHICLES FROM AN AIRBORNE PLATFORM Ming Hou and Robert D. Kobierski The Canadian Forces (CF) is considering more widespread use of uninhabited aerial vehicles (UAVs) to provide a new integrated intelligence, surveillance, and reconnaissance capability. In this role, UAVs would be force multipliers, releasing manned aircraft for other roles (e.g., the Arctic sovereignty patrol). However, at the moment UAV control is operator intensive and can involve high levels of workload. As the quantity and variety of data collected increase, the workload of UAV operators increases significantly. Moreover, the allocated data must be integrated and/or converted into information and then disseminated to those operators who make decisions. In the recent past data collection, data fusion, information management and distribution, intelligence collecting, and data-related decisionmaking have threatened to become a bottleneck. This situation is made even more complex by increasing joint operations, and rapid and flexible warfare. Feedback from the operation of UAVs indicates that improvements in the operator interface aspects of these emerging systems would reap significant Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 267–282 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07019-0

267

268

MING HOU AND ROBERT D. KOBIERSKI

gains in system performance and effectiveness. This applies for both effective control of UAVs as well as management of data and efficient dissemination of the associated information. The level of automation to be applied to the decision-making processes is a key aspect facing both tactical commanders and UAV system managers. As a result, supporting technologies that combine operators and automation to satisfy mission requirements need to be investigated. Intelligent Adaptive Interfaces (IAIs) are technologies for reducing operator workload and improving decision effectiveness in the employment and operation of UAV platforms. IAIs are human–machine interfaces that should improve the efficiency, effectiveness, and naturalness of human–machine interaction by acting adaptively and proactively to external events based on internal mission requirement. Specifically in the context of UAV control, an IAI is driven by software agents that support the decisionmaking and action requirements of operators under different levels of workload and task complexity. The IAI manifests itself by presenting the right information or action sequence proposal or perform actions at the right time. In addition to reducing workload for humans involved in UAV missions, IAIs are seen as opportunities to reduce manning requirements (e.g., moving from ratio of 10 operators controlling one UAV to one operator controlling 10 UAVs). Such technologies are likely to be integrated into many future CF systems. Defence Research & Development Canada has initiated a multi-year program for the development of IAIs for advanced UAV control for the Air environment. The aim of this program is to develop, demonstrate, and prioritize enabling technologies that can be applied to advanced operator interfaces to support reduced manning and enhanced performance in complex military systems, specifically multiple UAV control from an airborne platform. This program will also produce preliminary design guidelines for IAIs. The multi-year program has three phases: IAI concept development, interface prototyping, and experimentation and demonstration. In the concept development phase, a network model has been built to predict the performance of UAV operators controlling multiple UAVs through a simulation. The validity of the network model will be assessed in the third phase by an empirical evaluation of the performance of operators working with prototype interfaces that have been developed. Thus, both the validity of the model and experimental results will guide the use of network modeling methodology and the design of operator interfaces using IAI technology as well. This chapter summarizes the operational analysis and simulation results of the first phase in which the focus was to investigate the efficacy of

Operational Analysis and Performance Modeling

269

IAIs in an operational situation. The selected environment involved CF maritime aircraft operations in support of counter-terrorism activities. The associated operator interfaces within which the IAIs were modeled using network analysis techniques were the tactical workstations of a modernized maritime CP140 aircraft. For UAV operations, the CP140 crew was augmented with a UAV Pilot, UAV Sensor Operator (UAV Senso), and Tactical Navigator (TACNAV) in the tactical compartment of the aircraft. Research on IAIs in the military research and development community was limited when the first phase of the program was done. As the result, this project began with a Hierarchical Goal Analysis (HGA) within which standard mission, function, and goal/task analysis procedures were followed to gain more detailed understanding of implementation issues and opportunities for IAIs. Following the positive result reported herein, a detailed and focused literature review (Miller & Hannen, 1998; Miller, Goldman, Funk, Wu, & Pate, 2004; Taylor et al., 2002) was conducted and will be reported in a separate paper. The structure of this chapter is first to introduce the methodologies used for operational analysis and performance modeling, then the results are provided, and followed by discussions on the implications of the potential IAI functions for the control of multiple UAVs.

NETWORK MODELING METHODOLOGY As a standard procedure of human factors engineering, the design of complex systems (e.g., operator interfaces) starts with analyses of system objectives, missions, functions, and tasks. Perceptual Control Theory (PCT) provides a theoretical framework for guiding this process. PCT is founded on notions from control theory, in which closed-loop, negative-gain, feedback systems can be used to build powerful models of goal-directed behavior and for implementing complex systems (Powers, 1973). One of the strengths of PCT over competing human behavior theories is that it explains how humans can control systems that are subject to a wide variety of external influences. UAV control is through the operators’ interaction with the interfaces in remote control stations. A closed-loop feedback system is crucial for both operators and control systems to understand the states and goals of each other. It is likely that advanced UAV control systems will require operators to interact with automated systems such as IAIs. IAIs are sophisticated and will require knowledge about mission goals, the operators’ goals, and states, as well as the UAV and environmental states. Thus, the methods of analysis used in this research were based on PCT given its

270


engineering origins in control theory and advantages accommodating various external disturbances. Following a PCT approach, Hendy, Beevis, Lichacz, and Edwards (2002) suggested analyzing operations/missions as hierarchies of goals that are nested, sequenced or otherwise linked into logical networks. These networks could then be used to model and predict operators’ performance, an approach called HGA. The PCT-based HGA technique permits two additional analyses to be performed: stability and information flow. The former identifies possible conflicts among multiple human and automation agents acting on the same system. The latter is an analysis of information flow up the hierarchy, which can influence feedback in the system and affect error correction at higher levels. Traditional task network analysis techniques consider only the downward flow of information in the hierarchy. In the case of UAV control systems, it is critical that information in the form of system and operator goals and sub-goals be able to flow freely in both directions. Since stability and information flow analyses could contribute to the generation of a robust and complete task and goal decomposition for the UAV control system, the PCT-based HGA was chosen to team with classic task analysis to conduct operation, task, and goal analyses in this study. The analytical effort commenced with the development of a multiple UAV control scenario that reflected a portion of the CF Atlantic Littoral Intelligence, Surveillance and Reconnaissance Experiment (ALIX) program (Newton & Regush, 2004). From this scenario, individual operator and team goals were generated at successively deeper levels of decomposition. Each goal was associated with a PCT control loop from which multiple tasks (or behaviors) were spawned, each with related lower level goals. At the lowest level, all operator tasks which influence the ‘‘world’’ were ordered in a comprehensive temporal manner and were logically connected in operational sequence diagrams. In turn, they were modeled as a ‘‘flow chart’’ (task network) model. This probabilistic task network model could, through multiple concurrent task workload assessment and computer-controlled task conflict resolution, predict operator performance and effectiveness. The task network model was implemented in an Integrated Performance Modeling Environment (IPME). The model was run in two modes: one assuming the operators used a conventional interface without automation and the second assuming interface automation used an IAI. The frequency of occurrence of operator overload condition due to multiple ongoing tasks (task conflicts) and the simulated time to complete critical task sequences were measured for both modes. The objective of this work was to produce a HGA and a network model based on the mission scenario that simulated


271

controlling multiple UAVs from an airborne platform (CP140 aircraft). The network model was used to evaluate time pressure (workload), task conflict frequency, and time to complete critical task sequence for the three operators in control of the multiple UAVs with and without IAI.

Mission and Scenario Analysis In order to understand mission goals, an operational mission and scenario analysis for the control of multiple UAVs from the CP140 aircraft was conducted using a fictitious counter-terrorists mission, conducted in February 2011 (CMC Electronics Inc, 2004). As a prerequisite in the analysis, a one-hour scenario was developed with extensive Subject Matter Expert (SME) sessions. These SMEs were CF personnel involved with the ALIX program and had extensive CP140 and UAV operational experience. The scenario was a minute-by-minute description of a fisheries patrol southeast of St. John’s, Newfoundland, Canada along with plots of the mission evolutions. In the scenario, approximately 200 vessels were plotted in the vicinity of the nose and tail of the Grand Banks. A CF patrol frigate was onscene with two Vertical Take-Off UAVs (VTUAVs) and a maritime helicopter. A CP140 patrol aircraft equipped with 16 Mini UAVs and its own sensor suite was also included in the scenario. The CP140 was under the control of the Maritime Operations Centre in Halifax, Nova Scotia, Canada, and a Medium Altitude Long Endurance (MALE) UAV was under the control of the Regional Operations Centre, which is also located in Halifax (for the purpose of this scenario). The UAV tactical crew on the CP140 consists of the UAV Pilot, the UAV Senso, and the TACNAV. Positions of the crew in CP140 tactical compartment are illustrated in Fig. 1. The TACNAV was given the most complex role as the UAV team leader. Besides coordinating with the crew on board the CP140, he was responsible for understanding the overall tactical situation, planning appropriate responses to that situation, and delegating work to the members of the crew in order to affect those responses. The UAV pilot was responsible for the safe and appropriate conduct of all UAVs under the crew’s control. He might be required to flight plan individual UAVs, but the TACNAV was ultimately responsible for that task. The UAV Senso was responsible for managing the information being returned by the sensors onboard the UAVs under the crew’s control, and for relating findings based on that sensor data back to the rest of the crew, as appropriate. The result was that there existed an

272

UAV Sensor Operator

Tactical Navigator

UAV Pilot

Navigator Communicator

Fig. 1.

Layout of CP140 Tactical Compartment with UAV Crew Positions.


Non-Acoustic Sensor Operators


273

environment to assess the efficacy of IAI across three levels of job complexity: UAV Pilot, UAV Senso, and TACNAV. The UAV Pilot job was judged to be the least complex (keep the air vehicle safe and positioned accurately), and the TACNAV job was judged most complex (find the terrorist and resolve the situation).

Hierarchical Goal Analysis The implementation of PCT within this project was introduced earlier in this chapter. Each higher-level goal was achieved through a PCT closed-loop feedback system. Although the multiple UAV control is completed by a team of three members, each upper level goal was only assigned to one team member. In the CP140, each workstation is continuously occupied and team duties are predefined and closely adhered to. This is unlike team relationships where team members can ‘‘stand-in’’ for others when the prime individuals are not available. Responsibility for each operator goal, regardless of the level, is assigned to a specific team member. Each operator, at various times in the mission, supports the goal(s) of other operators as well as working toward accomplishing individual goals. The analysis reported herein was conducted to investigate individual IAIs for specific workstations. It was therefore assumed that, although an intelligent agent may support more than one operator at the same time, the manifestations would be tailored to each workstation. The team-level issues associated with conceptually adding a fourth ‘‘electronic’’ team member were not investigated and are left to a subsequent phase of development. In order to accomplish each goal the operator must complete tasks (behaviors) in the ‘‘world’’ and each of these tasks spawns lower-level goals (Hendy et al., 2002). In this research, the HGA consisted of a decomposition of goals from the highest level (e.g., GOAL ¼ counter-terrorism mission is completed) down to lower and lower levels (e.g., GOAL ¼ VTUAV sector search is planned). In PCT, goals are defined in perceptual terms (e.g., the operator must perceive that ‘‘the counter-terrorism mission has been completed’’ to achieve the goal). A hierarchical decomposition of the goals of the envisaged UAV control system for the proposed scenario was conducted for the purpose of developing a model from which performance predictions could be made and the potential IAI functions identified. The goal decomposition for all three UAV operators took place according to a means-end hierarchy, and the project requirements were typically satisfied at the fifth level of the decomposition (CMC Electronics Inc., 2004). The analysis

274


outputs were candidates identified for IAI automation with variables to be controlled as well as the completion times for the modeling process in IPME. Table 1 is a sample of HGA output.

Identification of IAI Candidates In order to compare conventional interfaces with IAI-based interfaces, it is necessary to identify candidate goals for IAIs first. Goals and tasks identified by the HGA that can be implemented automatically (with automated intelligence or expert knowledge) were chosen to be appropriate IAI candidates. Those tasks that have no associated data on the aircraft data bus could not be identified as IAI tasks. For example, an incoming verbal tasking from the Regional Operations Centre could not result in the initiation of an IAI task because the CP140 system intelligent agents would not be aware of the contents of the message. On the other hand, the CP140 IAI system could track keystrokes and cursor movement and would be able to deduce that an operator was attempting to complete multiple concurrent tasks. In this case, the interface could adapt to the situation and provide partial or complete assistance to the busy operator. For example, the interface may include potential UAV search patterns if the operator has airborne UAVs, yet un-tasked due to high workload. Accordingly, route planning was chosen as the first type of such IAI task. The second type of IAI task chosen was a communication task. For this task, information regarding upcoming communication must be available to the IAI sub-system. In general, verbal communication among the crew is preferred, providing that crewmembers are not over-tasked. However, when a commonplace communication is initiated which interrupts the recipient while that person is engaged in a complex cognitive task, the communication is detrimental. As a result, during high-workload periods and when the data are available for the system to facilitate communications, an ‘‘intelligent communication agent’’ should augment or facilitate transfer of directions or information. For example, based on the HGA it was determined that whenever the TACNAV inserted a fly to waypoint for a UAV which was just about to be launched, TACNAV would advise the UAV Pilot that the waypoint was inserted and give the general location of the waypoint. During low-workload periods, this transfer of information would be good to accomplish verbally because the TACNAV could immediately verify that the UAV Pilot understands and is in agreement with the tasking. During highworkload periods, this message might be lost, forgotten, or misinterpreted.

A Sample Output of HGA Results.

Number

Goal/Objective and Subgoals/Subobjectives

Top 7 y 7.3 y 7.3.2 7.3.2.1 7.3.2.2 7.3.2.3 7.3.2.4 7.3.2.5 7.3.2.6 7.3.2.7 7.3.2.8 7.3.2.9 7.3.2.10

I want to perceive the (y) conduct of the terrorist patrol mission y UAVs are employed

IAI Candidate

Influenced Variable

Completion Time (Sec)

TACNAV

y Mini UAV is employed y Mini UAV flight path is controlled y Mini UAV heading has changed to a new heading y Mini UAV altitude has changed to a new altitude y Mini UAV speed has changed to a new speed y Mini UAV altitude change has been initiated y Mini UAV automatic over-flight function is initiated y Mini UAV is set to autonomous operations y Mini UAV initiates a pre-planned route about a contact y Mini UAV initiates a self-destruct manoeuvre y manual control of Mini UAV is initiated y Mini UAV is manoeuvring about contact

Assignment to UAV Crew

TACNAV

Yes Yes

Heading Altitude

Yes Yes Yes Yes

Change initiated Over-flight initiated Setting Route planned Manoeuvre initiated Control initiated

Yes

Pilot Pilot Pilot Pilot Senso Senso Pilot Pilot Pilot Pilot

6 to 8 22


Table 1.

6 to 8 6 to 10 7 to 45 5 to 8 6 4 14 to continuous

275

276


It was assumed that, for the purposes of this study, the IAI would initiate the communication by setting a flag with the information on the TACNAV’s workstation that could be sent with the touch of a function key. This stimulus-response TACNAV task would be non-intrusive and would transfer this internal IAI message to the UAV Pilot to review at the earliest opportunity and without interference.

IPME Task Network Modeling Following the conduct of the HGA, a performance network model was developed in an IPME. IPME is a Linux-based discrete event simulation software application used to model and assess human and system performance (Dahn & Laughery, 1997). It comprises an environment model, a crew model, a task network, a performance-shaping model, and optional external models. With these models, researchers can obtain useful information about a process that might be too expensive or time-consuming to test in the real world. When combined with IPME’s scheduling algorithms, these models can help an analyst predict operator workload and performance before expensive software and infrastructure elements are developed. The IP/PCT mode of IPME uses an algorithm and an information processing scheduler to adjust operator performance based on various factors such as time pressure, task criticality, and task conflicts. The analysis reported herein was completed using the IP/PCT mode of IPME. A sequential type of operational network model was developed in IPME based on the HGA inventory (hierarchy). The inventory of goals created while producing the HGA was allocated to an operator or a system, as shown in Table 1. Although the network model was a UAV operator model, external events (other aircrew activities, UAV activities, and other unit activities) had to be established to allow the network to function as a closedloop feedback system. These were prepared and included in the network of tasks performed by three UAV operators. A task network model consists of networks and tasks. In the model developed here, a task network was created following the HGA structure and validated methodically against operational sequence diagrams (OSDs). OSDs are graphical illustrations of the logical interconnections of operators’ tasks and the flow of information throughout the system during the conduct of the mission scenario. OSDs not only use symbology to indicate actions, inspections, data manipulation (i.e., transmission, reception, and storage), time delays, and decisions, but also provide parameters such as task priority


277

and resource allocation determined by SMEs. Thus, the task network models could be easily built, mapped, and validated against OSDs to ensure that the model exhibits the behavior and task structure defined by OSDs. The network was used to define operator behavior, operator assignment, and interactions between tasks. By linking together various networks or tasks, the graphical network model attempted to replicate the behavior of the real system. Each task had a set of expressions associated with it to control when it executed, the state of the system when the task began or ended, and what, if anything, happened if a task failed to complete. In this study, tasks also included workload-specific information. The definition of the behavior of each individual task was done through the various tabs, the function list and the variable catalogue of IPME. The model was run in IP/PCT mode and data were collected with automation assistance (IAI ON) and without (IAI OFF). Fig. 2 shows one example of the part of the task network model. In IP/PCT mode, additional tabs are available to define critical IP/PCT parameters such as task priority (CMC Electronics Inc., 2004).

Simulation Output and Measurements IPME contains built-in data collection and reporting capabilities that determine and log, for each event of each probabilistic run of the network model, items such as operator time pressure (workload), specific task(s) shed at any instance, number of task ongoing for each operator, etc. The data collected were used to analyze the model’s (operators’) behavior and determine the effects of using IAIs on the operators’ performance. Since the model was developed to compare missions conducted with and without IAIs, measurements of effectiveness and performance had to provide a means of assessing the merits of incorporating these new technologies (IAIs). In addition, since this was the first opportunity to incorporate a full PCT evaluation of two design options involving differing levels of interface complexity, the measurements must allow the merits of the PCT to be fully exercised. The simulation output reported here consisted of two variables. The first variable was Task Conflict Frequency. It was defined as, for each run, the percentage of time during a sliding one-minute window that the operator must interrupt or delay a task with significant cognitive components. The other output was Goal Completion Time for critical tasks (higher-level goals in HGA). It was also chosen as a means to determine not only how an IAI system would affect an operator’s ability to achieve an objective, but also

278

IAI OFF TACNAV Task 278

M

IAI OFF

UAV Pilot Task 290

UAV Pilot Task 280

UAV Operator Task 291

CP140 Crew Task 281

TACNAV Task 292

T

M

IAI ON

M

CP140 Crew Task 293

TACNAV Task 294

M

CP140 Crew Task 300

TACNAV Task 278

CP140 Crew Task 295



UAV Pilot Task 280

TACNAV Task 297

CP140 Crew Task 281

UAV Pilot Task 298

IAI OFF

M

CP140 Crew Task 301

TACNAV Task 302

T

IAI ON

M

CP140 Crew Task 301

TACNAV Task 302

IAI ON Fig. 2.

An Illustration of Activities Conducted with and without IAI in a Task Network Model.

TACNAV Task 303


In


279


how the IAI system improved the operator’s performance. These two measurements were both operationally relevant and technically feasible.

RESULTS AND DISCUSSION Information was presented for each of the three UAV operators of the CP140 crew for each of the two variables introduced above – each with a conventional interface (IAI OFF) and again with IAI-based interface (IAI ON). After running the IPME model 10 times, data were collected and analyzed for the two IAI conditions (ON and OFF). Fig. 3 provides a sample plot of Task Conflict Frequency normalized against the mean mission timeline. The term ‘‘normalized’’ means that the respective plot of each of the 10 runs of the model was temporally adjusted to match the mean mission time. Fig. 3 shows that with IAI mode selected ON, the operators were dealing with fewer task conflicts (17%) comparing to IAI OFF mode (38%) around 14 min after the mission simulation started. Fig. 3 also shows 45 40

IAI OFF

35 30

Mean Task Conflict Frequency (%)

25 20 15 10 5 0 45

40

IAI ON

35 30 25 20 15 10 5 0

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Mission Time (min)

Fig. 3.

A Sample of Task Conflict Frequency Comparison with and without IAI.

280


that the mission evolved much faster with the IAI selected ON (about 17.5 min) comparing to IAI OFF mode (19.7 min). This clearly indicates an advantage with the use of IAI in that reduced goal completion times left more time for planning and problem solving. The reduced goal completion times were also evident in Fig. 4, which provides the results of seven of thirty-eight higher-level goals in the third part of the scenario, which contained the highest workload for the UAV crew. These goals were the ones influenced temporally by tasks related to the two IAI functions modeled (i.e., crew communication and route planning). It indicates that the greatest advantage of using the IAI mode of operations appears to be a reduction in the time required to complete higher-level goals. For example, goal 3 in Fig. 4 was a TACNAV goal of ensuring that rapid control was exercised over three Mini UAVs, which were launched in rapid succession over the terrorist boat. The time to complete this activity was

Mean Completion Time (sec)

Higher-Level Goal Completion Time 160 150 140 130 120 110 100 90 80 70 60 50 40 30 20 10 0

IAI OFF IAI ON

1

2

3

4

5

6

7

Higher-Level Goal Goal 1: Crew is briefed on the response to the loss of VTUAV1 Goal 2: Mini UAVs 2-5 are ready for launch Goal 3: Deployment of Mini UAVs 2 to 4 Goal 4: Control of the Laser UAV is established Goal 5: Reassessment of need to attack terrorists is completed Goal 6: Search for the Lethal UAV is discussed and search plan is initiated Goal 7: All UAVs are set to autonomous mode allowing CP140 to commence search

Fig. 4.

A Sample of Task Completion Time Comparison with and without IAI.


281

modeled to be a reduction by more than 80% due to positive intervention of the IAI agents. This was complimented by modest reductions in the Task Conflict Frequency. The simulation results also indicated that the UAV tactical crew maintained a constant intensity of activities throughout the various levels of workload and role complexity. This was anticipated because the network assumed trained operators would operate at a steady and consistent level. The difference between mission activities with automation (IAI ON) and without (IAI OFF) was reflected in the time to complete critical task sequences. With goals that required planning the UAV flight paths or search patterns, reduced times were found using the IAI to assist the operator. The automation was evidenced through the interface anticipating the users’ activities in high-workload situations and offering pre-calculated routes for the operator to accept or modify. All goals that required intense verbal communications within the crew were also assisted using the IAI mode. Automation agents were modeled to anticipate the communication requirements and facilitate internal tasking and information transfer using an ‘‘intelligent communication agent’’. This agent proposed messages at opportune times and, without auditory distraction, passed the internal communication to the appropriate individual(s). Similarly, non-distracting feedback that the message had been received was modeled.

CONCLUSION The task network analysis conducted using the IP/PCT mode in IPME provided numerical evidence of improved performance. For some of the higher-level goals, time to achieve the goal could be reduced significantly (e.g., one could be more than 80%) when the IAI mode of operation was selected. In addition, the task conflict frequency could also be reduced by more than 50% when the IAI mode was turned on in the control stations. The control of multiple UAVs from an airborne platform is a cognitively complex task with high workload, but using IAIs in these advanced systems will likely provide a real potential for improvement in the effectiveness of a UAV crew. The IAI tasks identified in this simulation study (i.e., route planner and communication agent) could only be considered as initial estimates of augmented cognition aids, further analysis regarding the most beneficial IAI aids is recommended. Additionally, it is important to focus on an experimental design to test the IAI based analysis conducted in this

282


study. Considerable effort may be required to implement IAIs into prototype systems, and strong empirical evidence would be important to substantiate this effort. A human-in-the-loop experiment will also test the validity of the task network modeling method used in this study for other similar research.

REFERENCES CMC Electronics Inc. (2004). Hierarchical goal analysis and performance modeling for the control of multiple UAVs/UCAVs from airborne platform. Toronto, Canada: Defence Research & Development Canada-Toronto (Contract Rep. CR 2004-063). Dahn, D., & Laughery, K. R. (1997). The integrated performance modeling environment – simulating human-system performance. In: S. Andradottir, K. J. Healy, D. H. Withers & B. L. Nelson (Eds), Proceedings of the 1997 Winter Simulation Conference (pp. 1141– 1145). Atlanta, GA, USA: ACM December 7–10. Hendy, K. C., Beevis, D., Lichacz, F. & Edwards, J. L. (2002). Analyzing the cognitive system from a perceptual control theory point of view. In Cognitive systems engineering in military aviation environments: Avoiding cogminutia fragmentosa! A report produced under the auspices of The Technical Cooperation Programme Technical Panel HUM TP-7 Human Factors in Aircraft Environments. Human Systems IAC, Wright-Patterson AFB, OH. Miller, C., Goldman, R., Funk, H., Wu, P., & Pate, B. (2004). A playbook approach to variable autonomy control: Application for control of multiple, heterogeneous Unmanned Air Vehicles. In Proceedings of FORUM 60, the Annual Meeting of the American Helicopter Society. Baltimore, MD. Miller, C., & Hannen, M. (1998). User acceptance of an intelligent user interface: A rotorcraft pilot’s associate example. In: M. T. Maybury (Ed.), Proceedings of the 4th international conference on intelligent user interfaces (pp. 109–116). New York, NY: ACM Press. Newton, S. J. & Regush, M. M. (2004, September 7–9). Atlantic Littoral Intelligence, Surveillance and Reconnaissance Experiment (ALIX) overview. In I. Glen (Chair), Keynote speech given at UVS Canada 2004 conference. Ottawa, Ontario, Canada. Powers, W. T. (1973). Behaviour: The control of perception. New York: Aldine De Gruyter. Taylor, R. M., Bonner, M. C., Dickson, B., Howells, H., Miller, C., Milton, N., Pleydell-Pearce, K., Shadbolt, N., Tennison, & J., Whitecross, S. (2002). Cognitive cockpit engineering: Coupling functional state assessment, task knowledge management, and decision support for context-sensitive aiding. In: M.D. McNeese, & M. Vidulich (Eds), Cognitive systems engineering in military aviation environments: Avoiding cogminutia fragmentosa! (pp. 253–312), Human Systems Information Analysis Center State-of-the-art Report 02– 01. Wright-Pattern Air Force Base, OH: Human Systems Information Analysis Center.

TEAM CONTROL OF ROVS

In contrast to the focus on a single operator controlling multiple remotely operated vehicles (ROVs), this section focuses squarely on a current reality: it takes multiple people to control a single ROV. Nancy Cooke describes the use of a Synthetic Task environment to examine team cognition and skill acquisition. In particular, she reveals multiple factors that affect team learning on unmanned aerial vehicle (UAV) tasks and that need to be considered in training UAV teams successfully for the larger command and control environment. The chapter by Ernest Park highlights the disparity between expectations of UAV team performance and actual team performance. In fact, he suggests this is a result of behavioral and decision errors due to poor coordination and collaboration. In order to raise the level of actual performance to its potential, he recommends that shared interests and shared understanding be incorporated in ROV team training programs. Next, Janie DeJoode introduces the concept of a Deployable UAV Operations Cell (DUOC) to improve the communication and coordination of the human-UAV in the overall command and control system. A cognitive task analysis of the design clarified strengths and areas to improve in the DUOC design. Finally, the chapter by Sandro Scielzo argues that the robot itself should be considered an important member of the overall team. In fact, the chapter suggests that human–robot interactions are becoming so sophisticated and interdependent that measures of team performance would be more robust if they considered information processing on the part of the human and the machine. In conclusion, increasing our understanding of team composition, coordination, and cooperation, is an important step in the way ahead for ROV systems.

283


284

20. ACQUIRING TEAM-LEVEL COMMAND AND CONTROL SKILL FOR UAV OPERATION Nancy J. Cooke, Harry K. Pedersen, Olena Connor, Jamie C. Gorman and Dee Andrews UAVs (unmanned aerial vehicles) are complex systems, typically operated by more than one individual. The human role in UAV operation is often neglected, and consequently the role of multiple interacting humans is seldom acknowledged. The consideration of a single UAV as a complex command-and-control task presents an interesting contrast to proposals of multiple UAV control (four or more) by a single individual. But the reality is that these complex systems often require dozens of personnel from maintenance and weather personnel to the ground crew and intelligence experts, giving credence to the statement, ‘‘There are no unmanned systems.’’ Our work in the CERTT (Cognitive Engineering Research on Team Tasks) Laboratory has focused on the team dynamics associated with UAV operation. For the purposes of our work, we define a team as ‘‘a distinguishable set of two or more people who interact dynamically, interdependently, and adaptively toward a common and valued goal/object/mission, who have each been assigned specific roles or functions to perform, and who have a limited life span of membership’’ (Salas, Dickinson, Converse, & Tannenbaum, 1992, p. 4). We have been particularly interested in cognitive activities such as planning, decision making, and situation assessment that are Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 285–297 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07020-7

285

286

NANCY J. COOKE ET AL.

carried out by teams in the UAV setting. These collaborative cognitive processes which include team coordination, are referred to as team cognition. Team cognition is assumed to be at the center of team performance in highly cognitive settings such as UAV ground control. Therefore, increasing team performance effectiveness and safety require a firm understanding of team cognition. There are, however, more questions than answers. How can team performance be measured? How can we characterize and assess cognitive skill at the team level? Can assessment occur without disruption of operational performance and can it occur in time for intervention? How is team cognition and performance impacted by training, technology, and team composition? Is team cognition different than the sum of the cognition of individual team members? How does individual task competence enhance or inhibit team cognition and situation awareness? What are effective training regimes or decision tools for these team members? To what degree does a history of teammate and task environment interactions contribute to team cognition? How long is team skill retained? Our research program in the CERTT Laboratory is focused on these and other questions pertaining to team performance and cognition. In this chapter, we highlight results obtained in our lab that pertain to skill acquisition for teams in a synthetic UAV environment. The studies described address the course of team skill acquisition and some factors relevant to this developmental course. By way of preview, our results speak to the importance of acquiring skill at coordinating with fellow team members versus acquiring shared knowledge about the task.

THE UAV SYNTHETIC TASK ENVIRONMENT The heart of the CERTT Laboratory, shown in Fig. 1, is a flexible Synthetic Task Environment (STE) that is designed to study many different synthetic tasks for teams working in complex environments. STEs provide an ideal environment for the study of team cognition in complex settings by providing a middle-ground between the highly artificial tasks commonly found in laboratories and the often uncontrollable conditions found in the field or high fidelity simulations. The CERTT Lab contains an STE that has been abstracted from a cognitive task analysis of controlling a Predator UAV (Cooke & Shope, 2004; Gugerty, DeBoom, Walker, & Burns, 1999). We emphasize the psychological fidelity of the UAV-STE in that it emphasizes many team aspects of

Acquiring Team-Level Command and Control Skill for UAV Operation

Fig. 1.

287

CERTT Lab (a) AVO Participant Console and (b) Experimenter Console.

tasks found in actual UAV operations such as planning, replanning, decision making, and coordination. ‘‘Psychological fidelity refers to whether the simulation ‘feels right.’ That is, does the simulator provide the trainee the same subjective cues and feedback the trainees find when they use the real equipment?’’ (Andrews & Bell, 2000, p. 364). Also, does the simulator exercise the same psychological processes (e.g., decision making, planning) that occur in the real task? We are currently studying team cognition with the use of the CERTT UAV-STE controlled by a three-person team whose mission is to take reconnaissance photographs at designated and ad hoc target areas (i.e., targets are not always preplanned). Our general assumption is that our work on team cognition in this environment applies to team cognition involved in real UAV control and even broader Remotely Operated Vehicles (ROV) control. The team members involved in this synthetic version of the Predator task are the Air Vehicle Operator (AVO) who flies the UAV by controlling the heading, altitude, and airspeed; the Payload Operator (PLO) who controls camera settings and takes reconnaissance photos; and the Data Exploitation, Mission Planning and Communications Operator (DEMPC) who plans the mission and acts as the navigator. Individual team members are provided with distinct, though overlapping training; have unique, yet interdependent roles; and are each presented with different and overlapping information during missions. Therefore, to successfully complete a mission, the team members need to share information with one another in a coordinated fashion. More details about the CERTT Laboratory and the UAV STE can be found in other publications (Cooke, Rivera, Shope, & Caukwell, 1999; Cooke & Shope, 2002).

288


TYPICAL CERTT UAV STUDIES We have conducted a series of five studies in the CERTT Lab that together provide some data that emphasize the importance of team coordination. From these pooled results, we have identified specific factors that seem to have the greatest impact on team performance. The factors include experience coordinating with others in a cognitively similar team environment, familiarity with teammates, and retention interval. In this section, we describe the general methodology for our CERTT Lab experiments. Details can be found in Cooke, Shope, and Kiekel (2001b), Cooke, Kiekel, and Helm (2001a), and Cooke et al. (2004).

Participants Individuals primarily from New Mexico State University and some from Arizona State University voluntarily participated in five experiments and were randomly assigned to three-person teams. Individuals or their organizations were compensated for their participation. In addition, the three team members on the team with the highest performance score received a monetary bonus.

Procedure and Equipment The studies took place in the CERTT Lab’s UAV-STE. Each participant was seated at a workstation consisting of three computer monitors. Two experimenters were seated in a separate adjoining room. Experimental sessions began with PowerPoint training and testing customized for each of the three roles (participants received training specific to the assigned role). This was followed by a skills check to verify that each individual could competently perform the tasks required of that role (i.e., AVO, PLO, or DEMPC). Training sessions were aimed at individual tasks only and did not involve any team-level instruction. Each experiment consisted of between five and ten 40-min missions. Team performance (based on a composite of team-level outcome variables), team process behavior, situation awareness, and communication data were collected during the missions and taskwork and teamwork knowledge were assessed in knowledge sessions apart from the missions. Most communication occurred via microphones and headsets, though some involved computer messaging. Team performance is the


289

primary measure considered in this chapter and at its core reflects the effectiveness of the team in terms of number of targets photographed per minute and amount of time spent in warning and alarm states. Across the studies, experimental factors including knowledge sharing, distributed vs. co-located mission environments, and workload have been manipulated.

RESULTS PERTAINING TO SKILL ACQUISITION Here, we highlight results that are relevant to the topic of acquisition of team-level skill (for full results see Cooke et al., 2001a,b, 2004). We discovered that command and control skill in our UAV lab task is acquired over the first four 40-min missions. Fig. 2 displays learning curves for teams of three operators in our first experiment in terms of team performance where performance is a team-level metric that primarily reflects the number of targets that the teams photographed successfully for each mission. Thus in this experiment teams who had no prior command-and-control experience, reliably reached asymptotic levels of performance in about four missions. This pattern of skill acquisition has also been found in previous CERTT Lab experiments using teams of undergraduate college students

Fig. 2.

Acquisition of UAV Task (Team Performance Scores) for 11 Teams from CERTT Experiment 1.

290


with no prior experience (Cooke et al., 2001a, 2004). You might also notice some curious dips in performance at Mission 8. We believe that this can be attributed to retention interval, as some teams took longer to reschedule after spring break (between Missions 7 and 8) than others. What is the team learning during those first four missions? This question is at the heart of team-level skill acquisition. It is important to keep in mind that participants work on a mission as a team, but before they first come together at Mission 1, each of them has been trained individually on his or her role’s specific task to criterion levels of performance. That is, each individual is required to pass written tests pertaining to his or her role and each individual is checked in the simulator for the requisite skills prior to participating in the first mission. Thus, we speculate that individual skill is fairly stable at this time. Because individuals have attained a criterion level of individual task skill prior to the first mission as a team, we postulate that it is team skill that develops over the first four missions. In particular, we assume that team members are learning how to coordinate their unique perspective on the information of the UAV task with the right person at the right time. Are teams learning how to coordinate with teammates in these early missions? There are other possibilities. It could be that teams are acquiring shared mental models in those early missions. That is, their team-level understanding of the task is developing from a collection of individual task experts to a team that is more or less ‘‘on the same page’’ with regard to the task. In a heterogeneous team, like the three CERTT UAV-STE operators, this shared understanding might require additional insight into other task roles and information requirements not acquired in training, but discerned only through interaction. There are several findings described in what follows that lead us to believe that the critical factor in team-level skill acquisition is indeed not team-level knowledge development, but rather the development of team coordination skill. In three CERTT UAV experiments, knowledge measures of teamwork and taskwork knowledge at the team level were taken on multiple (2 to 4) occasions apart from the mission. Further, these measures took team heterogeneity into account in that the metrics included indices of role-specific and interpositional knowledge. Across these three experiments there were minimal changes in taskwork knowledge which was fairly stable after training and the first mission (Cooke et al., 2001b, 2004). This means that teams acquired team-level knowledge of the task either during training (PowerPoint and skills check) and/or during the first mission. Therefore the acquisition of shared mental models of taskwork does not fully account for


291

team-level skill acquisition of this task that occurs over the first four missions. Interestingly, unlike taskwork knowledge, team level knowledge of the team (i.e., teamwork knowledge or the understanding of team roles and associated information capabilities and needs) did show some change over knowledge sessions. Similarly, team process behaviors (e.g., communication, conflict management, leadership behaviors) also improve over the first four missions. Finally, across a number of studies, we also find that various manipulations such as cross training (Cooke et al., 2003), the ability to share information among teammates (Cooke et al., 2001b), and geographic colocation versus distribution (Cooke et al., 2004) that are predicted to affect team knowledge, do impact team knowledge, but not team performance. That is, there appears to be a decoupling between team knowledge and team performance. The fact that performance is measured more frequently (at every mission) compared to knowledge (one to three sessions) suggests that this difference is not due to differential sensitivity of the measures. Together these results suggest that the development of team knowledge or a shared understanding may be a prerequisite for effective team performance, but is not the lone determinant of team-level skill acquisition. Although team knowledge (especially taskwork knowledge) does not seem to be a primary determinant of team skill acquisition, team coordination does. This is not to say that team knowledge is not required, as it does seem important in early missions. However, we are saying that team knowledge alone cannot account for the variance that occurs in team development. Instead, we propose that the acquisition of team coordination skill plays a critical role in team skill acquisition. Support for the importance of coordination in team-level skill acquisition in our UAV task is evident in a CERTT UAV study in which five expert teams were run through the simulation. The expert teams did not consist of expert UAV operators, but instead due to pragmatic issues of participant availability, consisted of three-person teams who were experts at working together as a team in technological environments (i.e., aviation, commandand-control, or military tasks) for an extended period of time. With the exception of a team comprising of CERTT experimenters, these expert teams had no prior experience with our UAV task, although one team had worked together in a UAV development environment. The five teams included (1) Flight Instructors, (2) Internet Video Game Players, (3) UAV Developer Team, (4) Flight Students, and (5) CERTT Experimenters. Interestingly, in this case, we see an exception to the four-mission-asymptote

292

NANCY J. COOKE ET AL. 500 Team Performance

450 400 350

Benchmark

300

Experiment 1

250

Experiment 2 Experiment 3

200

Experiment 4

150 1

2

3

4

Mission

Fig. 3.

Acquisition of UAV Team Performance across Five CERTT UAV Experiments.

rule described previously. Results revealed that expert teams (designated in Fig. 3 as ‘‘Benchmark’’) reached an asymptotic level of performance in a single 40-min mission. Performance of each expert team was compared to the mean mission performance of all relevant teams from the other four experiments. For Missions 1–4, expert teams’ performance was compared to the mean of all 69 previous teams; however, for Mission 5, the expert teams’ performance was only compared to the performance of the 40 teams in Experiments 3 and 4, as these were the only other experiments that included a similar workload manipulation. Table 1 indicates the specific expert teams (designated by a descriptor relevant to their background) that performed 1.5 standard deviations above (+) or below ( ) the mean mission performance of the other teams from the first four experiments. The Experimenter, Flight Student, and Video Game teams excelled relative to the other expert teams and above the mean performance found in previous experiments. We do not believe that this rapid skill acquisition is due to increased task knowledge, as this was already at criterion by Mission 1 for all individuals, but due to intrateam familiarity and coordination experience. However, it is difficult within the confines of this experimental design to separate familiarity with teammates from task experience as a team. The CERTT Experimenter team is an interesting case because the team possessed the team-level knowledge of the task normally acquired in Mission 1. This team also possessed coordination skills achieved through time

293


Table 1. Expert Teams who Achieved Performance Scores Ranging Outside|1.5|Standard Deviations of Non-Expert Teams’ Performance at Each Mission. Flight Inst. Team

Mission Mission Mission Mission Mission

1 2 3 4 5

Video Game Team

UAV Team

+ + +

Flight Student Team

+ +

Experimenter Team

+ + + + +

working together as experimenters. They had coordinated experimenter tasks and communications regarding the operation of the simulation, but prior to this study, they had never flown together as participants in the STE. The CERTT Experimenter team performed well even in the first mission. It is also interesting to note that with the exception of the CERTT Experimenter team, the best performing team was the video game team of three males who had played the internet video game ‘‘Counter-Strike’’ several hours per day. We speculate that this team-level experience transferred to team coordination in the CERTT UAV-STE. Alternatively, it could be that individuals who have a predilection for coordination are drawn to play internet video games and would also excel in other tasks requiring coordination. We might also speculate that the flight students had more coordination experience as a team than instructors who might teach coordination and communication, but who themselves were not likely to practice these skills frequently as a team. Finally, the UAV developers may have experienced negative transfer due to (1) differences in the way that they coordinated (strategic instead of tactical) and (2) differences between their UAV platform and the STE specifics. The results support the proposal that team-level knowledge is developed early (i.e., training and Mission 1) and that coordination skill develops in unfamiliar and unpracticed teams in the first three to four missions.

DISCUSSION The results from the CERTT UAV test bed indicate that acquiring teamlevel skill in this setting goes beyond an individual’s understanding of his or

294


her task and beyond a broader shared understanding of the team task (i.e., team roles, task knowledge associated with other positions). Beyond task knowledge, team skill in this command-and-control environment develops over the first four missions in which team members actively interact. Specifically, team knowledge or shared understanding is relatively stable by the first mission, however, team process behaviors and teamwork knowledge improve over this time period and teams who have worked together before in a coordinated task can accelerate this team-level skill acquisition. Thus, it appears that during the first four missions, the team is not developing a shared mental model, but is learning how to interact; how to push and pull information; how to coordinate. Coordination thus seems to be key to the development of team-level skill in this kind of command-and-control task. Accordingly, our work in the CERTT Laboratory has begun to focus on coordination as a critical component of team performance. We define team coordination as the timely and adaptive passing of information among team members. Advances in technology have increased the cognitive complexity of tasks and therefore, have increased the need for coordinated teamwork. Team coordination is characterized by timely and adaptive information exchange among team members. In this regard, team coordination underlies emergent team cognition, especially in command-and-control environments (Cooke & Gorman, 2006). The information exchanged may be about the task or about the team, but it is the ongoing exchange process concerning dynamic aspects of the environment that seems critical to team performance, not the resulting knowledge structure. The development and maintenance of coordination skill is challenging in domains such as UAV command-and-control for several reasons including the (1) unanticipated nature of the situation (e.g., ad hoc targets, weather changes), (2) possible ad hoc formation of teams and even team structure, and (3) extended intervals with little or no team training. For instance, with regard to the third reason, there can be fairly long periods when commandand-control teams are not able to train and practice together, yet they are expected to be competent as soon as they are deployed. The need for acquisition and retention research on team coordination is apparent (Adams, Webb, Angel, & Bryant, 2003; Schendel & Hagman, 1991). The CERTT UAV-STE provides an excellent setting for research on acquisition and retention of team coordination. This is evident because the amount of exposure that teams have to operational tasks or to each other between laboratory sessions is nearly impossible to control for teams that work together in a natural operational setting (e.g., UAV operator teams).


295

Therefore, current efforts in the CERTT Lab include empirical and modeling work on the acquisition and retention of team coordination in the context of the UAV STE. In our ongoing empirical work we are manipulating the retention interval (3–6 weeks versus 10–13 weeks) and whether teams come back for a second session intact or ad hoc (i.e., unfamiliar with their teammates). This will allow us to empirically separate the roles of familiarity and experience in the acquisition of team coordination, allowing us to address the factorial contribution of each. At the same time we are developing a local optimal model of coordination that we can use to quantify team coordination that occurs at target waypoints in the mission scenarios. When the UAV enters a target area, an experimenter logs the occurrence of specific coordination-relevant events (e.g., the DEMPC informs the AVO of flight restrictions for that target area), which are automatically time stamped. These data are then contrasted with an optimal model of team coordination at each target area. The degree to which there is a deviation from optimal at each waypoint defines a metric of coordination. Through this procedure we can obtain an index of team coordination at multiple target waypoints within a 40-min mission and eventually use this to predict team performance. Parallel research efforts are exploring the use of automated communication analysis techniques as a surrogate for manual logging of coordination data (Kiekel, Gorman, & Cooke, 2004). Techniques from the dynamical systems area used to model dynamics of motor coordination will be used to model the global development of team coordination from local target deviations under the different experimental conditions. We are very excited about this effort and its potential to provide converging evidence regarding the development of team coordination. These data obtained from the ongoing retention study will contribute to the theoretical foundation of team performance through a better understanding of how coordination develops in teams. From a pragmatic perspective, this research will provide useful information and through the modeling efforts, predictive tools (models) for understanding commandand-control training needs and improving retention of coordination skill in these teams. The data and associated models should suggest interventions that can be used in training to prolong the retention of such skills (i.e., video game practice, intelligent agent coordinator). Use of such techniques should increase the effectiveness of teams, while saving time and training resources. Increased training in team coordination retention should also benefit teams that are assembled ad hoc and therefore are unfamiliar with each other.

296


Specifically relevant to UAVs is the perspective that operation of a single UAV is a team command-and-control skill, which is in fact, embedded within a larger command-and-control organization. The results from this research program suggest that team coordination is an important part of team cognition and that through communication analysis there is potential to assess a team’s cognitive state in an unobtrusive and real-time fashion. Results also suggest that team training and technological interventions designed to facilitate coordination are important for effective UAV system performance. Results from ongoing research will speak to the significance of retention intervals and ad hoc teams to UAV command-and-control performance.

ACKNOWLEDGMENTS This work was sponsored by AFOSR grants F49620-01-1-0261, F49620-031-0024, and FA9550-04-1-0234, AFRL grant FA8650-04-6442, and ONR grants N00014-00-1-0818 and N00014-03-1-0580. Janie DeJoode, Preston Kiekel, Leah Rowe, Steven Shope, and Jennifer Winner contributed significantly to the empirical work presented here.

REFERENCES Adams, B. D., Webb, R. D., Angel, H. A., & Bryant, D. J. (2003). Development of theories of collective and cognitive skill retention. Humansystems Report No. CR-2003-078. Andrews, D. H., & Bell, H. H. (2000). Simulation-based training. In: S. Tobias & J. D. Fletcher (Eds), Training and re-training (pp. 357–384). New York: McMillan. Cooke, N. J., DeJoode, J. A., Pedersen, H. K., Gorman, J. C., Connor, O. O., & Kiekel, P. A. (2004). The role of individual and team cognition in Uninhabited Air Vehicle command-andcontrol. Technical Report for AFOSR Grant Nos. F49620-01-1-0261 and F49620-03-1-0024. Cooke, N. J., & Gorman, J. C. (2006). Assessment of team cognition. In: W. Karwowski (Ed.), International encyclopedia of ergonomics and human factors (2nd ed.). Boca Raton, FL: CRC Press, LLC. Cooke, N. J., Kiekel, P. A., & Helm, E. (2001a). Measuring team knowledge during skill acquisition of a complex task. International Journal of Cognitive Ergonomics: Special Section on Knowledge Acquisition, 5, 297–315. Cooke, N. J., Kiekel, P. A., Salas, E., Stout, R. J., Bowers, C., & Cannon-Bowers, J. (2003). Measuring team knowledge: A window to the cognitive underpinnings of team performance. Group Dynamics: Theory, Research and Practice, 7, 179–199. Cooke, N. J., Rivera, K., Shope, S. M., & Caukwell, S. (1999). A synthetic task environment for team cognition research. Proceedings of the Human Factors and Ergonomics Society 43rd Annual Meeting, (pp. 303–307). Santa Monica, CA: Human Factors and Ergonomics Society.


297

Cooke, N. J., & Shope, S. M. (2002). Behind the scenes. UAV Magazine, 7, 6–8. Cooke, N. J., & Shope, S. M. (2004). Designing a synthetic task environment. In: S. G. Schiflett, L. R. Elliott, E. Salas & M. D. Coovert (Eds), Scaled worlds: Development, validation, and application (pp. 263–278). Surrey, England: Ashgate. Cooke, N. J., Shope, S. M., & Kiekel, P. A. (2001b). Shared-knowledge and team performance: A cognitive engineering approach to measurement. Technical Report for AFOSR Grant No. F49620-98-1-0287. Gugerty, L., DeBoom, D., Walker, R., & Burns, J. (1999). Developing a simulated uninhabited aerial vehicle (UAV) task based on cognitive task analysis: Task analysis results and preliminary simulator data. Proceedings of the Human Factors and Ergonomics Society 43rd Annual Meeting (pp. 86–90). Santa Monica, CA: Human Factors and Ergonomics Society. Kiekel, P. A., Gorman, J. C., & Cooke, N. J. (2004). Measuring speech flow of co-located and distributed command and control teams during a communication channel glitch. Proceedings of the Human Factors and Ergonomics Society 48th Annual Meeting. Monica, CA: Human Factors and Ergonomics Society. Salas, E., Dickinson, T. L., Converse, S. A., & Tannenbaum, S. I. (1992). Toward an understanding of team performance and training. In: R. W. Swezey & E. Salas (Eds), Teams: Their training and performance (pp. 3–29). Norwood, NJ: Ablex. Schendel, J. D., & Hagman, J. D. (1991). Long-term retention of motor skills. In: J. E. Morrison (Ed.), Training for Performance: Principles of applied human learning (pp. 53– 92). Chichester, UK: Wiley.


298

21. A THEORETICAL PERSPECTIVE ON ENHANCING COORDINATION AND COLLABORATION IN ROV TEAMS Ernest S. Park, Verlin B. Hinsz and Jared L. Ladbury The use of remotely operated vehicles (ROVs) has increased in recent years, and the market is expected to grow dramatically by 2020 as military, civil, and commercial applications continue to develop (DeGarmo & Nelson, 2004). Numerous explanations underlie this increasing enthusiasm. These vehicles can be deployed in high-risk missions, reducing the potential for pilot and crew casualties. ROVs, such as the Mars Exploration Rover or the U.S. Air Force Predator system, are also popular because their capabilities are unconstrained by the limits of human physiology (e.g., g-force limitations, oxygen requirements). In addition, ROVs are relatively cost efficient due to their less bulky design, and absence of ancillary requirements necessary for humans such as heaters, ejector seats, or power systems for gee suits. In addition to the benefits, ROV use has been accompanied by problems as well. A major deterrent to the more widespread adoption of ROVs, and unmanned air vehicles (UAVs) in particular, are high mishap rates (Defense Science Board, 2004). In the military, UAV mishap rates are 100 times higher than for manned aircrafts (Blazakis, 2004). Investigations conclude Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 299–309 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07021-9

299

300

ERNEST S. PARK ET AL.

that the lack of substantial experience by UAV operators and maintainers is a significant contributor to the occurrence of accidents (Defense Science Board, 2004). Human-error related accidents, which accounted for 32% of all of the error-related accidents in a U.S. Army study, were primarily caused by decision, skill-based, and perceptual errors (Manning, Rash, LeDuc, Nobak, & McKeon, 2004). Since equipment failures are also a major contributor to high mishap rates, our intent is not to blame UAV operators or to undermine their value. Instead, our aim is to present a theoretically based training approach that is expected to improve team performance in an ROV task environment. Considering the important benefits provided by ROV implementation, we suggest that a logical approach is to address and minimize the costs of using ROVs. To that end, one priority should be to reduce the occurrence of mishaps caused by decision, skill-based, and perceptual errors. Since ROVs are frequently operated by a team, one approach of enhancing performance is to maximize the benefits while minimizing the costs associated with working in groups (Hackman, 1992). Assigning teams to operate ROVs can be advantageous for many reasons. In induction-type tasks, in which one must search for descriptive, predictive, and explanatory generalizations or principles to some problem, groups are superior relative to individuals (Laughlin, 1996). Therefore, in information-rich contexts such as a typical ROV mission, it is likely that teams will be better able to uncover solutions to the novel problems they encounter than individuals acting alone. Because they have multiple members, teams also possess a greater capacity to process information (Laughlin, VanderStoep, & Hollingshead, 1991), attend to information (Hinsz, Tindale, & Vollrath, 1997), store information (Hinsz, 1990; Wegner, 1986), and to correct memory and decision errors (Hinsz, 1990). When assessing team performance, it is essential to recognize that the actual productivity the team achieves often fails to equal the team’s potential productivity (Steiner, 1972). Absence of a clear plan or approach for successful action frequently results in improper and inefficient use of time and resources (Weingart, 1992). Thus, teams are not freely granted the benefits associated with them, but instead, members must actively coordinate and collaborate with one another to reap such advantages. In a team context, processes can be conceptualized as consisting of the actual steps taken by the members when performing a task. These includes the intra- and interpersonal actions by which team members transform their resources into a product, as well as the nonproductive behaviors instigated by frustration, competing motivations, and inadequate understanding (Baron, Kerr, & Miller, 1992; Steiner, 1972).

A Theoretical Perspective in ROV Teams

301

Team processes that revolve around the integration and coordination of resources, attitudes, and abilities often enhance performance. For example, processes of a productive team may include (1) the intellective and communicative behaviors by which members evaluate, pool, and assemble their resources, (2) decisions about who will do what and when, and (3) behaviors aimed at motivating members to fully contribute to the task. From this perspective, actual productivity will resemble potential productivity to the degree that faulty processes are eliminated or reduced (Steiner, 1972). We suggest that a team training program designed to increase coordination and decrease process losses be implemented to enhance team performance and reduce human errors among ROV operating teams. This training program should also be constructed to minimize motivation losses, and ideally facilitate motivation gains and helping behaviors. Within the framework of groups as complex systems, coordination involves the ‘‘establishment, enactment, monitoring, and modification over time of a network that connects and coordinates the group’s members, tasks, and tools’’ (McGrath, Arrow, & Berdahl, 1999, p. 1). We define the concept of collaboration, on the other hand, as anticipatory behavior aimed toward helping and assisting one’s teammates.

COORDINATION AND COLLABORATION DEMANDS IN ROV TEAMS To understand the importance of coordination and collaboration for ROV teams, let us examine some of the typical tasks that ROV operators might be required to perform (Cooke & Shope, 2004; Gugerty, DeBoom, Walker, & Burns, 1999). To do so, we will use the members of a U.S. Air Force Predator crew as an example. The team consists of three members: an Air Vehicle Operator (AVO) who pilots the aircraft, a Payload Operator (PLO) who operates the surveillance equipment, and a Data Exploitation, Mission Planning, and Communications Operator (DEMPC) who is responsible for mission planning. In the course of a mission, the AVO is responsible for the take off and landing of the aircraft. Because they fly the aircraft from a remote location, AVOs are generally required to use visual input from a camera mounted on the nose of the aircraft to guide their flight. Once in the air, the PLO can operate cameras and sensors mounted on the belly of the plane to gather information. The DEMPC, who is in contact with the upper

302


echelons of the organization, provides the AVO with the desired heading and the PLO with target coordinates. Typical mission goals for a UAV team might include gathering quality images of multiple targets, while optimizing efficiency and safeguarding the aircraft. While performing its duties, the team must also remain within specified borders and avoid flying into restricted areas. Among the chief concerns of the team are to keep the aircraft within its specified altitude parameters, consider weather constraints which influence the flight and maneuverability of the system, gather quality images, and to satisfy the efficient completion of ad hoc target objectives. To fulfill their goals efficiently, the team members must collaborate with one another and coordinate their actions. For the PLO to capture useful surveillance images, the aircraft must be positioned accordingly. Therefore, it is necessary that the PLO and AVO communicate in a way such that the AVO can maneuver the aircraft to a precise location. Because operators control the aircraft remotely, they do not have access to as much sensory information as they would if they were inside the aircraft (e.g., sound of crosswinds, feeling turbulence). Thus, efficient coordination between the team members is particularly necessary because the AVO must rely heavily on input from the camera operator. The camera operator in turn, must consider some of the constraints that the AVO faces, such as the direction of wind or geographic boundaries (e.g., threat zones or mountains). To collect an image, the camera operator is required to provide input concerning optimal positioning to the AVO, and the AVO needs to maneuver the vehicle so the desired information can be obtained. When the AVO is unable to place the aircraft in the requested position, all of the team members need to coordinate their knowledge, consider the various situational constraints, and work together to determine alternative actions. During the mission, it is conceivable that the DEMPC will receive a request to gather images of an ad hoc target, which may alter the mission plan that was previously generated (Hall & Gugerty, 1997). Therefore, a revised mission plan must be constructed to incorporate the new target objective. To do so efficiently, members of the UAV team needs to contribute information that is available to them. If there are strong winds or threat zones present, the AVO should disclose this knowledge. If the target needs to be approached from a particular angle or direction, information that might only be available to the camera operator upon inspection of the target, then the PLO needs to inform the team prior to the determination of the flight path. Once the DEMPC can incorporate all of this information, the team as a whole can make decisions concerning mission planning. To


303

collect surveillance imagery and to make mission decisions, team members must recognize what information is relevant, what information they personally do and do not have access to, what information their teammates do and do not have access to, and what information is needed by their teammates and when.

MENTAL MODELS IN ROV TEAMS The coordination and collaboration demands that UAV operators must confront highlight the important role that shared interests and understandings have on coordinated actions. This sentiment has been echoed and further developed by others. For example, Cannon-Bowers, Salas, and Converse (1993) suggest that team performance is enhanced when members share an understanding of the task, their teammates’ roles and expertise, as well as the context in which they operate. Team members must have accurate knowledge structures, or mental models, so they can generate predictions and expectations about their teammates, the task demands, and their environment. It is precisely this ability to anticipate and predict teammates’ information and coordination demands that leads a team to coordinate effectively (Tannenbaum, Salas, & Cannon-Bowers, 1996). Thus, the function of shared mental models among team members is to allow them to draw upon their own spring of well-structured knowledge as a source for selecting actions that are consistent and coordinated with those of their teammates (Mathieu, Heffner, Goodwin, Salas, & Cannon-Bowers, 2000). For effective team performance, theorists suggest that multiple mental models need to be shared (Cannon-Bowers et al., 1993). For example, team members should have mental models of their technology and equipment, and importantly, must mutually comprehend how their actions interact with the input of other team members. Teammates should also have shared task models that describe and organize knowledge about how various procedures will lead to task accomplishment (Klimoski & Mohammed, 1994). This task model should include knowledge about their task strategies, and potential contingency plans. A shared understanding of how the team interacts is also crucial to team functioning (Hinsz, 1995). Team interaction models should contain descriptions of the roles and responsibilities of team members, perceived patterns of interaction, knowledge of information flow and communication channels, role interdependencies, and recognition of information sources. This team interaction model facilitates the coordination of action, the synchrony and sequencing of member behaviors in time and place which

304


is necessary for effective performance to achieve the team’s goals (McGrath et al., 1999). And lastly, team members can share a mental model of the team itself. This team mental model holds information that is specific to one’s teammates, such as their knowledge, skills, attitudes, preferences, inclinations, strengths, and weaknesses. This type of knowledge is important because members can then adjust their behavior in accordance to what they expect their teammates to do (Moreland, 1999). The more information that one has about a teammate, the more accurate one’s predictions about their intentions and actions should be (Cannon-Bowers, Tannenbaum, Salas, & Volpe, 1995; Mathieu et al., 2000). The development of a team mental model will lead to the coordination of understanding, which refers to the shared perceptions among members, including an appreciation of the ways in which members interpret situations differently. Coordination requires an understanding of how the group will proceed considering these potentially differing perspectives. This type of coordination is necessary to obtain, interpret, and apply information effectively to guide the team’s activities (McGrath et al., 1999).

TRAINING COORDINATION AND COLLABORATION IN ROV TEAMS Given that shared mental models are important for team coordination and performance, one issue that surfaces concerns the development of such models. Research suggests the development and adjustment of shared mental models is facilitated via intra-team feedback (Blickensderfer, Cannon-Bowers, & Salas, 1997; Rasker, Post, & Schraagen, 2000). Intra-team feedback occurs when teammates provide each other with feedback information. This can take the form of performance monitoring and team selfcorrection. Performance monitoring takes place when members give, seek, and receive task-clarifying feedback during task execution (Cannon-Bowers et al., 1995). This requires accurately monitoring the activities and progress of teammates, providing constructive input concerning errors, and offering advice for improving performance (McIntyre & Salas, 1995). Team selfcorrection is the process that takes place after the task in which team members review events, correct errors, discuss strategies, and plan for subsequent tasks. Through this process, teammates can adjust their team attitudes, behaviors, and cognitions without external intervention (Rasker, Post, & Schraagen, 2000).


305

Though necessary to consider when discussing team performance, we suggest that the sole focus should not be on just coordination. In dynamic and stressful situations, such as those frequently encountered by ROV teams, we argue that collaborative behaviors should also be encouraged because it is in these situations that helping behaviors are most necessary. Collaboration should enhance performance in dynamic, fast-paced, stressful situations because it reduces the need for extended and explicit communication, and allows for behaviors to be performed in a timely manner. For example, if the camera operator on a UAV team scans the target when the aircraft is too far away to gather the information required to fulfill the target objective, he may still be able to ascertain the direction from which the AVO will need to approach it. Relaying this information to the rest of the team in advance would be an example of collaboration because the PLO anticipates what information the AVO and DEMPC need for their tasks and provides it. Explicit communications are reduced because the PLO performs this function without being prompted by direct requests and efficiency is enhanced because the AVO can initially steer the aircraft toward the target from the appropriate direction, eliminating the need for backtracking or rerouting. Another example of collaboration would be when the AVO sees obstructions around a target, such as hills or towers, and maneuvers the aircraft so that the camera operator is better positioned to collect the desired images. This would be collaborative behavior because this process requires the AVO to first identify a situation where the camera operator will need assistance, and then to perform compensatory actions without being asked. In an inefficient team, the AVO could easily neglect the responsibilities and information requirements of the camera operator, and instead wait for the teammate to announce that the target is obscured. This neglect of collaborative behavior would decrease efficiency if the AVO then has to re-route the aircraft in a roundabout fashion relative to if the obstruction was anticipated in advance. Collaboration is related to coordination in that they both require the development of shared mental models. Arguably, one cannot successfully and efficiently assist a teammate in an anticipatory fashion if they are unaware of the teammates’ responsibilities, requirements, and strengths and weaknesses. Sharing task and team interaction mental models should increase both coordination and collaboration, but for different reasons. Sharing knowledge of who does what and when will facilitate behavioral synchrony because teammates will recognize how they should act in concert with their teammates. But additionally, sharing this type of knowledge

306


should facilitate helping behaviors because teammates will be better prepared to recognize when their teammates need assistance, and this knowledge should serve as a basis for understanding how to help them. This is one reason why cross-training interventions, which are geared toward providing relevant information about other teammates’ tasks and responsibilities, can be so beneficial for team effectiveness (Cannon-Bowers, Salas, Blickensderfer, & Bowers, 1998; Tannenbaum et al., 1996). Suggestions for motivating collaborative behavior can also be rooted in early research by Latane´ and Darley (1968). Although their work focused on pro-social behavior in emergency situations, their theoretical framework offers insight into how collaborative behaviors can be increased. First, the situation must be noticed and then interpreted as one in which assistance is needed. Then, one must assume responsibility for providing assistance. If one takes on the perspective of ‘‘that’s not my job,’’ helping behaviors are not likely to occur. Knowing how to correctly help in the situation is a key final ingredient toward instigating compensatory behaviors. This ‘‘decision tree’’ outlined by Latane´ and Darley (1968) again illustrates the importance of having shared task and team interaction mental models. Shared mental models allow team members to develop and maintain accurate and shared assessments of the situation (Cannon-Bowers & Salas, 1998), thus increasing the likelihood that one will notice and interpret a situation where assistance is applicable. Shared mental models will also provide information about what kind of help is needed, as well as knowledge about how to provide such help. Based on all of these theoretical guidelines, we have designed a team training program for ROV operators aimed at enhancing coordination and collaboration. In addition to encouraging the development of shared mental models among teammates, this program is also designed to emphasize a sense of shared fate and to minimize the impact of normative barriers that inhibit helping behaviors and error correction in teams. By making the team members’ shared fate salient, we believe that teammates will feel more responsible and accountable for the overall performance of the team, and thus, more motivated to help one another in times of need. By confronting the barriers that prevent team members from correcting one another when mistakes occur, we believe team members will be more likely to reduce inefficient actions and decision errors. By implementing proper training for coordination and collaboration, we argue that teams will be able to take advantage of their increased capacity to process and remember information. The faulty process losses that might otherwise occur in the team setting will also be reduced, decreasing the discrepancy between potential performance and actual performance.


307

The effectiveness of our training program is currently being tested in a laboratory setting. During training, an emphasis is placed on the outcome interdependence of the team, stressing that members are ‘‘all in it together.’’ Because the team’s actions must be coordinated to achieve its goals, no one operator can be held responsible if the mission fails. During training, teammates are also instructed to explicitly articulate their particular duties, focusing on who does what, when, why, and how. Teammates are also asked to state when and how potential obstacles and problems can be identified, and to develop a plan for handling such situations. By having teammates share this type of knowledge, we feel problems can be minimized and often averted.

CONCLUSION Because of the numerous advantages that result from ROV use, it is likely that implementation of these systems will continue to thrive despite the problems that accompany them. Fortunately, some of these problems can be remedied or reduced by enhancing the capabilities of ROV teams. Our focus is on two sets of processes that can utilize the benefits of working in teams: coordination and collaboration. We argue that coordination and collaboration can be trained in ROV teams to reap the rewards of working in groups. Coordination should improve the performance of ROV crews by increasing efficiency. Collaboration should aid performance by increasing anticipatory helping behaviors leading to the minimization of behavioral and decision errors. Ideally, coordination and collaboration behaviors will help teams reach their potential and reduce mishap rates. We suggest that coordination and collaboration can be infused in ROV operations by adopting a team training protocol geared toward developing and maintaining shared mental models and a sense of interdependence. Shared mental models will facilitate these skills by providing members with knowledge that can be used to form accurate expectations about their task and team. As a consequence of enhanced coordination and collaboration, ROV teams should be more capable of guiding their vehicles in successful missions even under dynamic and constrained conditions.

ACKNOWLEDGEMENT This material is based on research sponsored by the Air Force Research Laboratory, under agreement numbers F49620–02–1–0234 and F49620–

308


03–0353. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory or the U.S. Government. Direct inquiries can be sent to Ernest Park or Verlin Hinsz of the Department of Psychology, North Dakota State University, Fargo, North Dakota 58105, e-mail: [email protected] or [email protected].

REFERENCES Baron, R. S., Kerr, N. L., & Miller, N. (1992). Group process, group decision, and group action. Bristol, PA: Open University Press. Blazakis, J. (2004). Border security and unmanned aerial vehicles. Report RS21698. Congressional Research Service, The Library of Congress, Washington, DC. Blickensderfer, E., Cannon-Bowers, J. A., & Salas, E. (1997). Theoretical bases for team selfcorrection: Fostering shared mental models. In: M. Beyerlein (Ed.), Advances in interdisciplinary studies of work teams (pp. 249–279). Greenwich, CT: JAI. Cannon-Bowers, J. A., & Salas, E. (1998). Individual and team decision making under stress: Theoretical underpinnings. In: J. A. Cannon-Bowers & E. Salas (Eds), Making decisions under stress: Implications for individual and team training (pp. 17–30). Washington, DC: APA. Cannon-Bowers, J. A., Salas, E., Blickensderfer, E., & Bowers, C. A. (1998). The impact of cross-training and workload on team functioning: A replication and extension of initial findings. Human Factors, 40, 92–101. Cannon-Bowers, J. A., Salas, E., & Converse, S. A. (1993). Shared mental models in expert team decision making. In: N. J. Castellan Jr. (Ed.), Current issues in individual and group decision-making (pp. 221–246). Hillsdale, NJ: Erlbaum. Cannon-Bowers, J. A., Tannenbaum, S. I., Salas, E., & Volpe, C. E. (1995). Defining team competencies and establishing team training requirements. In: R. Guzzo & E. Salas (Eds), Team effectiveness and decision making in organizations (pp. 333–380). San Francisco: Jossey-Bass. Cooke, N. J., & Shope, S. M. (2004). Designing a synthetic task environment. In: S. G. Schiflett, L. R. Elliott, E. Salas & M. D. Coovert (Eds), Scaled worlds: Development, validation, and applications (pp. 263–278). Burlington, VT: Ashgate. Defense Science Board. (2004). Unmanned air vehicles and uninhabited combat aerial vehicles. Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics, Washington, DC. DeGarmo, M., & Nelson, G. M. (2004). Prospective unmanned aerial vehicle operations in the future national airspace system. Technical report 04–0936. American Institute of Aeronautics and Astronautics, Washington, DC. Gugerty, L., DeBoom, D., Walker, R., & Burns, J. (1999). Developing a simulated uninhabited aerial vehicle (UAV) task based on cognitive task analysis: Task analysis results and preliminary simulator performance data. Proceedings of the Human Factors and


309

Ergonomic Society 43rd annual meeting, Santa Monica, CA: Human Factors and Ergonomics Society (pp. 86–90). Hackman, R. J. (1992). Group influences on individuals in organization. In: M. D. Dunnette & L. M. Hough (Eds), Handbook of industrial and organizational psychology, (Vol. 3, pp. 47–69). San Diego, CA: Academic Press. Hall, E., & Gugerty, L. (1997). Predator operations cognitive task analysis results. Briefing prepared by the Air Force Research Laboratory, Brooks AFB, TX. Hinsz, V. B. (1990). Cognitive and consensus processes in group recognition memory performance. Journal of Personality and Social Psychology, 59, 705–718. Hinsz, V. B. (1995). Mental models of groups as social systems: Considerations of specifications and assessment. Small Group Research, 26, 200–233. Hinsz, V. B., Tindale, R. S., & Vollrath, D. A. (1997). The emerging conceptualization of groups as information processors. Psychological Bulletin, 121, 43–64. Klimoski, R. J., & Mohammed, S. (1994). Team mental model: Construct or metaphor? Journal of Management, 20, 403–437. Latane´, B., & Darley, J. M. (1968). Group inhibition of bystander intervention in emergencies. Journal of Personality and Social Psychology, 10, 215–221. Laughlin, P. R. (1996). Group decision making and collective induction. In: E. Witte & J. H. Davis (Eds), Understanding group behavior: Consensual action by small groups (pp. 61– 80). Mahwah, NJ: Erlbaum. Laughlin, P. R., VanderStoep, S. W., & Hollingshead, A. B. (1991). Collective versus individual induction: Recognition of truth, rejection of error, and collective information processing. Journal of Personality and Social Psychology, 61, 50–67. Manning, S. D., Rash, C. E., LeDuc, P. A., Noback, R. K., & McKeon, J. (2004). The role of human causal factors in U.S. Army unmanned aerial vehicle accidents. Report 2004–11. U.S. Army Aeromedical Research Laboratory, Fort Rucker, AL. Mathieu, J. E., Heffner, T. S., Goodwin, G. F., Salas, E., & Cannon-Bowers, J. A. (2000). The influence of shared mental models on team process and performance. Journal of Applied Psychology, 85, 273–283. McGrath, J. E., Arrow, H., & Berdahl, J. L. (1999). Cooperation and conflict as manifestations of coordination in small groups. Polish Psychological Bulletin, 30, 1–14. McIntyre, R. M., & Salas, E. (1995). Measuring and managing for team performance: Emerging principles from complex environments. In: R. Guzzo & E. Salas (Eds), Team effectiveness and decision making in organizations (pp. 149–203). San Francisco: Jossey-Bass. Moreland, R. L. (1999). Transactive memory: Learning who know what in work groups and organizations. In: L. L. Thompson, J. M. Levine & D. M. Messick (Eds), Shared cognitions in organizations: The management of knowledge (pp. 3–31). Mahwah, NJ: Erlbaum. Rasker, P. C., Post, W. M., & Schraagen, J. M. C. (2000). Effects of two types of intra-team feedback on developing a shared mental model in command and control teams. Ergonomics, 43, 1167–1189. Steiner, I. D. (1972). Group process and productivity. New York: Academic Press. Tannenbaum, S. I., Salas, E., & Cannon-Bowers, J. A. (1996). Promoting team effectiveness. In: M. A. West (Ed.), Handbook of work group psychology (pp. 503–529). Hoboken, NJ: Wiley. Wegner, D. M. (1986). Transactive memory: A contemporary analysis of the group mind. In: G. Mullen & G. Geothals (Eds), Theories of group behavior (pp. 185–208). New York: Springer. Weingart, L. R. (1992). Impact of group goals, task component complexity, effort, and planning on group performance. Journal of Applied Psychology, 77, 682–693.


310

22. GUIDING THE DESIGN OF A DEPLOYABLE UAV OPERATIONS CELL Janie A. DeJoode, Nancy J. Cooke, Steven M. Shope and Harry K. Pedersen INTRODUCTION Today’s battlespace is a very complex system of humans and technology. It could be thought of as a system of layers – where there might be a layer of ground operations and a layer of air operations. Within the air operations layer exists two additional layers of manned air operations and unmanned air operations. If you peel back all layers of today’s battlespace and just view the ‘‘unmanned air operations’’ layer, you will find another complex system of humans and technology working as just one element of the overall system. This system of uninhabited air operations might consist of different types of uninhabited air vehicles (e.g., Predator, Hunter, etc.) performing different types of missions (e.g., Intelligence, Reconnaissance, SurveillanceIRS; IRS-strike; search and rescue, etc.). It is overwhelming to think that there is one large entity responsible for the command-and-control of the entire air operations layer of today’s battlespace. Air Operations Centers (AOCs) are the nerve cells for the commandand-control of all air operations (both manned and unmanned operations). The footprint of the AOC consists of multiple workstations and hundreds of operators who man the AOC around the clock and perform a myriad of Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 311–327 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07022-0

311

312

JANIE A. DEJOODE ET AL.

command-and-control functions, such as time-sensitive targeting, weather monitoring, battlefield management, and special operations support. The multiple layers of coordination required in operating a large system of humans and technology, such as the AOC, can have negative impacts on team performance. For example, if a decision to strike a time-sensitive target cannot be passed through the decision making path in a timely manner, critical opportunities can be missed, resources can be lost, or in the worst case, lives could be lost. Or, if an officer with insufficient knowledge of UAVs commands a ground control crew to fly a UAV into weather too dangerous for the aircraft (although suitable for other types of manned aircraft), a multi-million dollar resource could be wasted. To some, the sheer magnitude, complexity, and coordination requirements of command-and-control systems make the need for human factors glaringly obvious; to others, it is not until inefficient coordination results that a problem is even realized. Unfortunately, inefficient coordination is often misinterpreted as a need for more humans or technology (Klinger & Klein, 1999). This chapter presents an example in which human factors was incorporated into the design of a command-and-control node early in the design stage. Specifically, a preliminary cognitive task analysis (CTA) was performed on the Deployable Uninhabited Aerial Vehicle Operations Cell (DUOC), a concept for the future designed to provide specialized commandand-control of UAVs. The objective of the CTA was to develop an understanding of the interactions among the variety of people, roles, organizations, and technology involved in UAV command-and-control. It was our goal to use the results of the CTA to lay the foundation for developing a synthetic task environment (STE) of the DUOC. However, results from CTAs in general, can be used as the basis for a multitude of further activities (e.g., designing or re-designing the system, training, etc.; Cooke & Shope, 2004).

DUOC BACKGROUND The DUOC is an initiative of the UAV Battlelab at Creech Air Force Base (formerly Indian Springs Auxiliary Airfield) that is designed to serve as a distributed node of the AOC that specializes in mission management for UAVs. DUOC Concept. The UAV Battlelab’s concept of operations defines the DUOC as a standardized, yet flexible, state-of-the-art tool to manage, monitor, and support multiple numbers and types of UAVs conducting a variety of mission types over the battlefield at the tactical level, as depicted

Guiding the Design of a Deployable UAV Operations Cell

Fig. 1.

313

DUOC Concept.

in Fig. 1. This layer of command-and-control mediates interactions between the AOC and the ground control stations (GCSs), presumably allowing for reductions in AOC personnel and equipment requirements by transferring parts of the UAV mission control to the DUOC and at the same time enabling greater AOC mission management, situation awareness, support, and dynamic decision making. DUOC Shelter. The DUOC would be of a standardized design and housed in a rapidly deployable shelter, as shown in Fig. 2. The physical location of the DUOC can be anywhere, ranging from a theater deployment, in which the DUOC is co-located with the AOC or the GCS, to a stateside facility connected via satellite to operations anywhere in the world. DUOC Internal Functions. The preliminary internal design of the DUOC has seven core functional positions, with supplemental positions depending on UAV payloads/armament and mission objectives. The seven team members will be highly interdependent not only with one another but also with outside entities, such as the AOC, the Distributed Common Ground Station, and the GCSs. The seven core functions, seen in a notional layout in Fig. 3, include a Tactical Coordinator, Administration Officer, Weather Officer, Intelligence Officer, two Time Critical Targeting Officers, and a Target Officer.

314


Fig. 2.

Fig. 3.

DUOC Mobile Shelter.

Notional DUOC Command Table.


315

THE COGNITIVE TASK ANALYSIS A CTA decomposes a task at the level of the cognitive structures and processes required to perform the task. Some tasks may depend heavily on rapid decision making processes, whereas others may rely more on remembering procedures and implementing plans. CTA methods, such as interviews and field observations, serve to highlight the critical cognitive underpinnings of a task and may also reveal potential for cognitive pitfalls, errors, and limitations. Because preliminary designs and interfaces for the DUOC had been proposed by the UAV Battlelab, the goal of this CTA was to evaluate and assess those designs. As the first step in the CTA, we gathered and reviewed existing documents, presentations, and other materials relevant to the DUOC initiative. These documents and briefings contained information such as the DUOC levels of coordination (theater level and tactical level), training plan, timeline of design, functional data flow, etc. Twelve subject matter experts (SMEs) who had field experience in Predator operations were involved in the CTA. Some SMEs were current Predator operators while other SMEs were former operators. The group also included the DUOC architects as well as one SME who had experience working in the AOC. Knowledge Elicitation: Interviews Numerous types of interviews were conducted with SMEs, all focusing on the cognitive aspects of the DUOC tasks. Most interviews were unstructured, maximizing breadth of coverage, while other interviews (e.g., critical incident technique) were more structured to target specific cognitive features (Cooke, 1994). Unstructured and Structured Interviews Our primary method for eliciting knowledge in the CTA was unstructured and structured interviews. The unstructured interview was an open-ended, informal interview not guided by a predefined set of questions, whereas the structured interview was guided by a specific set of questions and was focused on eliciting knowledge on a particular topic. Over 20 h of interviews were conducted with SMEs. Presenting the full interview synopsis is beyond the scope of this chapter; thus, a sample of the information gathered from SMEs is highlighted below, organized by topic. It should be noted that the material below represents SMEs’ views on particular topics. This material

316


was transcribed from audio recordings and minor insertions or deletions were made for grammatical and presentation purposes; thus, it should not be considered direct quotations.Topic: Operators’ views of a formalized operations cell Deployments would be easier with the ops cell. It’s a good filter of trash incoming to the GCS. Strong leadership in the ops cell makes a huge difference and it’s important to know who has operational control y who owns [the GCS operator] right now? Experienced operators can lose situation awareness if many people are pulling at them, making it difficult to know who to listen to.

Topic: Risk of UAV vs. Priority of Target There is a need for the people in the AOC to understand the risk associated with flying in certain conditions. For example, the on-site weather officer at the ground control station (GCS) may advise the GCS to head towards home because the weather is deteriorating. However, a General in the AOC might command the GCS to remain in the field and complete the mission. So the GCS would inform the general that they will follow his orders but they might lose the airplane. The AOC often didn’t realize the situation with the weather and once this was communicated, the AOC would say, ‘ y by all means, head for home.’ GCS operators weren’t always clear on what the priority was (e.g., risk the UAV in poor weather for purposes of the mission?). A new pilot in the GCS probably wouldn’t challenge the order to complete the mission and just assume that the priority is to get the target and risk the UAV survival.

Topic: Why is the DUOC needed? Situation awareness of threats is needed. With a constant flow of intelligence from the DUOC, the GCS operators will feel more in the loop. An intermediary is needed to screen and prioritize information. There is a need to push information more quickly. Time and resources are wasted in waiting for the green light from the AOC. With multiple types of UAVs, there will be only 1 belly button to go to for information Versus going to a different liaison officer depending on the type of UAV.

Topic: Video product of UAVs Everyone wants it. Once a picture is all over the place, people pick up the phone and want to communicate about it.


317

Focusing on the video feed can take attention away from other important displays. If there is the ability in the DUOC to switch to the different video feeds that feed into the AOC, the AOC will be requesting a switch all the time.

Critical Incident Interview A critical incident interview (Hoffman, Crandall, & Shadbolt, 1998) was also conducted with one SME. The critical incident interview, a structured interview organized around an account of a specific incident from the SME’s own experience, involved a case-specific, multi-trial retrospection structured and guided by probe questions. The purpose of using this technique was to capture the kinds of expertise and coordination required to manage unexpected or non-routine UAV operations. The critical incident interview involved a series of steps where the SME selected an incident from memory and recalled the incident, after which the interviewer retold the incident to ensure it has been captured correctly and asks relevant ‘‘what if’’ questions. For a detailed account of these steps, see Hoffman et al. (1998). In a 2-h session, the SME recounted an incident where the Predator UAV was tasked to do route surveillance for the Army, which had been threatened during inspection of Serbian headquarters. A paraphrased excerpt from the interview follows: NATO told the Bosnia Serbs that they would do inspection of military headquarters. Serbs said: ‘No you’re not and we’ll shoot down any of your aircraft’. Predator was tasked to do route surveillance for Army folks going to this headquarters that was isolated in the woods. They heard that there were lots of Apache helicopters standing by. The Predator monitored the route the Army was using. Once there, it provided surveillance of what was happening on ground. There was a standoff at front gate. U.S. Army and Serbian troops with machine guns were staring at each other. Predator was eventually asked to look for escape routes in case it became hostile. The problems were that Predator was being used to report what was going on Versus pushing info such that it could be used to influence what was going on. There was no downlink from Predator to troops on ground to report people hiding in trees, etc. The Predator couldn’t provide situation awareness and protection to troops.

The critical incidents told by the SME highlighted potential problems with managing UAV operations from an ad hoc, non-standardized, and illequipped operations cell. For example, during the ‘‘what if’’ step of the interview, the SME reported that a formally defined and standardized operations cell could have mitigated the problems encountered: If there would have been hostile action, the GCS would have had to call the AOC, who would then have to contact the task force and then some Army unit. By this point, there would likely be casualties. It would have taken at least 10 minutes to get the Predator information to where it needed to go. If the DUOC had managed the same operation, it

318


could have provided situation awareness to the UAV ground control crew, who in turn could have provided the necessary information to the Army troops on the ground via the downlink.

The critical incident interview technique proved to be useful for eliciting information that is difficult to capture in basic structured interviews, which often focus on typical, routine events.

Data Representation A major component of CTA is organizing and representing the data gathered from the interviews. We used a variety of methods for representing and analyzing the information gathered from SMEs. Due to space limitations, we present only one of those methods here, concept mapping. Concept Mapping We used a technique known as concept mapping (Novak, 1995) to understand and depict communications among the DUOC and other external entities. In general, the concept mapping technique provides a way to organize the information and knowledge gleaned from SMEs. Based on the information gathered at the UAV Battlelab, we constructed a concept map of the communication content internal and external to the DUOC (see Fig. 4). This map is a graphical representation of some of the communications that may take place among the AOC, DUOC, and GCSs. Each shaded node represents an entity or a role and each non-shaded node represents a topic of communication in the environment. The label on each link indicates the relationship between the nodes. We also used concept mapping to depict the DUOC as a system acting as an intermediary between the AOC and the GCSs. Fig. 5 illustrates how the DUOC will serve as a filter of information. Each of the lightly shaded boxes represents a topic of communication that would need to occur among the entities. Comparing Fig. 5 to Fig. 6, in which the UAV operations system is depicted without the DUOC as an intermediary, suggests this: If the DUOC were not an intermediate entity, both the AOC and the GCSs would have higher volumes of communication, perhaps more so for the AOC because it must pass information to multiple GCSs. Furthermore, the entities would also be required to communicate about a more diverse set of topics. That is, if the AOC took over duties fulfilled by the DUOC, the AOC would then have to pass more information to the GCSs.


Concept Map of Communication Content among AOC, DUOC, and GCS.

319

Fig. 4.

320


Fig. 5.

Representation of DUOC as Intermediary in the Overall UAV Operations System.

The entire set of communications necessary among the AOC and the GCSs is partitioned into two smaller subsets of communications when the DUOC is involved (one set between the AOC and the DUOC and one set between the DUOC and the GCSs). These two subsets of communications are not necessarily mutually exclusive, as topics of communication may be common to both groups. Therefore, in Fig. 5, the DUOC serves as a single entity that ‘‘knows everything,’’ rather than two separate entities being required to know everything, as in Fig. 6.

COGNITIVE FEATURES OF THE DUOC TASKS Refined Cognitive Tasks This step involved taking all the information gathered from the CTA to abstract the primary cognitive issues underlying the DUOC task. To do this,


Fig. 6.

321

Representation of UAV Operations System without DUOC as Intermediary.

we first developed a comprehensive list of cognitive tasks that each DUOC member would be required to perform in a general operation. A critical objective of CTA is to distinguish cognitive tasks that are inherent in the system from cognitive tasks that result from implementation. A basic cognitive task is not subject to change; it must always be done, such as identification of friendly forces. In contrast, cognitive tasks can also result from implementation of interfaces and equipment. For example, the basic task of identifying friendly forces can be relegated to a machine, or the task may require a DUOC member to do an extensive search of a display. In some cases, cognitive tasks exist that are specifically tied to the interface. We determined that each cognitive task identified in our CTA of the DUOC is a basic or implementation-independent cognitive task. These tasks, organized by type of cognitive task, are outlined below: Decision Making Tactical UAV Coordinator decides what recommendations to make to the AOC based on information gathered concerning weather, threats, etc. Tactical UAV Coordinator decides which UAV to transfer control based on resources available (fuel, mission requirements, etc.) during a handoff of command-and-control.

322


Weather Officer decides on information to be included in formal reports, delivered to the DUOC team at regular intervals. Target Officer decides (based on judgment) appropriate weapons system for target. Target Officer decides whether appropriate information is available to GCS crew (accomplished through 9-Lines). DUOC team recommends whether UAVs carry weapons (restrictions imposed by ATO). Information Management Tactical UAV Coordinator develops pre-briefing prior to the operations to maintain situation awareness in DUOC. Tactical UAV Coordinator develops regular interval briefings for DUOC team (battle damage assessment, mission priorities, strike time, and location). Tactical UAV Coordinator collects important pieces of information and sends to relevant personnel. Weather Officer develops pi-reps and signets to GCS (involves monitoring and filtering data). Knowledge Management Administration maintains knowledge of mission requirements. Administration knows when to distribute reports about UAV mission findings. Target Officer knows rules of engagement. Weather Officer maintains familiarity with weather patterns in the region where the UAV is flying. Weather Officer knows the capabilities and structural limitations of each UAV. Analysis and Assessment Tactical UAV Coordinator assesses sensor information. Tactical UAV Coordinator assesses that UAV is visiting correct waypoints (‘‘knows’’ where UAVs are). Target Officer assesses and analyzes appropriateness of attack system (how to approach targets, combining different aircraft, etc.). Target Officer assesses target nominations. Target Officer analyzes probability of kill success versus collateral damage. DUOC team assesses priority of information flowing between AOC and GCS to distribute information only to appropriate entities.


323

Intelligence Officer analyzes threats to UAV. Intelligence Officer assesses and organizes external intelligence from AOC. Intelligence Officer assesses target decks to analyze effects of mission changes to target deck (e.g., if UAV #1 goes to target A, what happens to the rest of the target deck?). Intelligence Officer assesses Electronic Order of Battle (EOB) updates on what UAVs are finding. Weather Officer analyzes weather along proposed routes. Weather Officer analyzes weather over DUOC. Weather Officer assesses weather from live video feeds from UAVs. Time Critical Targeting Intelligence Imagery Analysis Personnel (TCT 1N1) Combat ID analyzes real-time data (video feed) and imagery (photos, SAR) from UAVs. TCT 1N1 Precise Coordinates analyzes real-time data (video feed) and imagery (photos, SAR) from UAVs. TCT 1N1 Precise Coordinates assesses coordinates for quality/control. Monitoring Tactical UAV Coordinator maintains situation awareness of UAV health status. Weather Officer monitors weather along proposed routes. Weather Officer monitors weather over DUOC. DUOC team monitors and provides situation awareness of threats. DUOC team monitors multiple UAVs of different types. Coordination Tactical UAV Coordinator manages transfer of command-and-control of UAV. TCT 1N1 Combat ID coordinates with GCS to identify sensor and coverage needs (may be affected by weather or threats). TCT 1N1 Combat ID coordinates with other 1N1 in analyzing imagery. TCT 1N1 Precise Coordinates communicates with GCS to identify sensor and coverage needs. DUOC team coordinates multiple UAVs of different types. Prediction Intelligence Officer predicts enemy movements during battle. Intelligence Officer predicts number of threats known based on intelligence analyses.

324


Weather Officer predicts weather along proposed routes. Weather Officer predicts weather over DUOC. Planning Intelligence Officer plans UAV route. DUOC team identifies conflicts between risk to UAV and priority of mission, and challenges AOC orders if necessary. Target Officer identifies friendly forces. TCT 1N1 Combat ID identifies targets of opportunity. TCT 1N1 Precise Coordinates identifies precise coordinates of a target by using Raindrop and comparing imagery with a National Terrain Database. DUOC team prioritizes targets. We have also identified constraints associated with DUOC functions that could potentially interfere with the team’s ability to perform the mission and coordinate effectively. Some constraints are due to the nature of the task. For example, weather is a basic constraint because the DUOC cannot control the weather but it can affect mission success. However, constraints can also result from the way a task is implemented. For example, poor leadership may constrain the DUOC team’s ability to coordinate, but this can be changed by obtaining a new leader or by training the current leader to be more effective. The constraints are listed below, categorized as either basic or implemented constraints. Basic Constraints Differences in terminology exist between joint forces. Workload varies for team members and is irregular across different missions. Mission objectives are dynamic. Multiple UAVs must be monitored simultaneously. Terminology and configuration among different UAVs is not consistent. The format of target locations may differ depending on what other military forces need (if the mission involves other forces). Teams are heterogeneous in terms of expertise. UAV data can be displayed to an endless number of parties, creating a ‘‘Big Brother’’ situation that causes operators to hesitate in making decisions. A heterogeneous mix of customers uses UAV data products. Some customers do not know how to fully exploit the UAV resources.


325

Constraints due to Implementation Imagery Analysts must disengage from one set of imagery to fully engage in other imagery continuously (particularly if multiple missions are tracking Time Sensitive Targets (TSTs)). The DUOC must keep distributed parties (Distributed Common Ground Station, GCS, operators on break, etc.) informed and in the loop. Weather forecasts must be specific to UAV operations. The Weather Officer must be familiar with the weather patterns of the region in which the UAV is flying. The DUOC must maintain real-time coordination with the AOC. Over-reliance on a particular communication mode can cause channel overloading. No formal training (including teamwork training) exists for DUOC personnel. DUOC team members must assess their own performance. DUOC resources must extend to any type of UAV mission. Mission planning is very dynamic. Decision making to match targets to weapons is complex and timeconsuming because many factors are considered (such as refueling capability). Decision making based on judgments varies with operator experience. UAV operators need to know who ‘‘owns’’ them or who they should listen to in order to maintain situation awareness. Non-secure communications lines inhibit the GCS’s ability to communicate with the appropriate forces. Abstracted Cognitive Features of DUOC Task From the list of cognitive tasks, we abstracted the most fundamental aspects of the DUOC task. These are features of the task that center on team cognition (Cooke, Salas, Kiekel, & Bell, 2004). The abstracted cognitive features of the DUOC task are: 1. Decision making requires knowledge and information sharing among team members. Because knowledge and information are distributed among team members, decisions must be based on information integrated from multiple sources. For instance, the Tactical UAV Coordinator makes decisions to recommend a mission change to the AOC based on weather forecasts from the Weather Officer, threat analysis from the Intelligence Officer, mission requirements from the Administration Officer, etc. This

326


information sharing requirement of decision making can be replicated in a possible STE version of the DUOC task. 2. Coordination involves the push and pull of critical information in a timely manner. Effective coordination within the DUOC team and between the DUOC team and other entities relies on successful delivery of information to the appropriate sources and receipt of the necessary feedback. A target nomination, which is based on dynamic information such as enemy location, sent to the AOC by the DUOC must be approved or rejected in a timely manner to fully exploit the UAV resources. Coordination, with the demand for the timely push and pull of information, will be an integral part of the DUOC STE. 3. Information management maintains situation awareness. Information management is a filtering process of reporting only relevant information to the necessary parties. This process requires information sharing among the team members and involves filtering of information at multiple levels. For example, to report battle damage assessment, the Tactical UAV Coordinator must collect the strike location from the TCT 1N1 Precise Coordinates Officer, collateral damage details from the Target Officer, and other details of the strike from the Target Officer. The Tactical UAV Coordinator would refine this information to report the battle damage assessment to the entire DUOC team as well as to the AOC and to the GCSs. Thus, in the STE, we will replicate this dependency between information management and situation awareness. 4. Prediction and planning at the individual and team level are based on dynamic, fleeting data. Time-sensitive targets, weather changes, and UAV resources (such as payload capability) are examples of factors that contribute to the dynamic nature of managing UAV operations. These kinds of dynamic events that occur in the field of practice will be preserved in the STE.

CONCLUSIONS The CTA of the DUOC uncovered a wealth of information about the people, roles, organizations, and technology involved in UAV commandand-control. One primary benefit of this CTA was that it was performed in the design stage of the DUOC, allowing for an increased likelihood that the results could be applied toward improving the DUOC interfaces and system design. However, performing a CTA of a concept, or an envisioned world, can raise questions about the validity of the data collected. We overcame the


327

challenges of studying an ‘‘envisioned world’’ by involving the DUOC architects as the primary subject matter experts in the CTA. We also interviewed former and current UAV operators who have experience with setting up and controlling ad hoc UAV operations cells. For these reasons, we consider the results of this CTA to be a valid decomposition of the DUOC’s tasks at the cognitive level that lay the foundation for future human factors research and design of UAV operation cells.

REFERENCES Cooke, N. J. (1994). Varieties of knowledge elicitation techniques. International Journal of Human-Computer Studies, 41, 801–849. Cooke, N. J., Salas, E., Kiekel, P. A., & Bell, B. (2004). Advances in measuring team cognition. In: E. Salas & S. M. Fiore (Eds), Team cognition: Understanding the factors that drive process and performance (pp. 83–106). Washington, DC: American Psychological Association. Cooke, N. J., & Shope, S. M. (2004). Designing a synthetic task environment. In: S. G. Schiflett, L. R. Elliott, E. Salas & M. D. Coovert (Eds), Scaled worlds: Development, validation, and application (pp. 263–278). Surrey, England: Ashgate. Hoffman, R. R., Crandall, B., & Shadbolt, N. (1998). A case study in cognitive task analysis methodology: The critical decision method for the elicitation of expert knowledge. Human Factors, 40, 254–276. Klinger, D. W., & Klein, G. (1999). Emergency response organizations: An accident waiting to happen. Ergonomics in Design, 7(3), 20–25. Novak, J. D. (1995). Concept mapping: A strategy for organizing knowledge. In: S. M. Glynn & R. E. A. Duit (Eds), Learning Science in the schools: Research reforming practice (pp. 229–245). Mahwah, NJ: Lawrence Erlbaum Associates.


328

23. COGNITION AND COLLABORATION IN HYBRID HUMAN–ROBOT TEAMS: VIEWING WORKLOAD AND PERFORMANCE THROUGH THE LENS OF MULTIMEDIA COGNITIVE LOAD$ Sandro Scielzo, Stephen M. Fiore, Florian Jentsch and Neal M. Finkelstein Robotics technology is becoming increasingly prevalent within military teams, predominantly under the form of operating Unmanned Air Vehicles (UAVs) and Unmanned Ground Vehicles (UGVs), typically referred to as Remotely Operated Vehicles (ROVs). Understanding the cognitive and $

Writing this paper was partially supported by Grant Number SBE0350345 from the National Science Foundation and by contract number N61339-04-C-0034 from the United States Army Research, Development, and Engineering Command, to the University of Central Florida as part of the Collaboration for Advanced Research on Agents and Teams. The opinions and views of the authors are their own and do not necessarily reflect the opinions of the University of Central Florida, the National Science Foundation, the RDECOM-STTC, the U.S. Army, DOD or the U.S. Government. There is no Government express or implied endorsement of any product discussed herein. Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 329–342 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07023-2

329

330

SANDRO SCIELZO ET AL.

coordinative processes emerging from using such technologies is vital in order to sustain complex operational demands. Moreover, these advances have produced hybrid human–robot teams with unique individual and team cognition characteristics that have to be understood in order to broaden theory and practice in such teams (Cooke, Salas, Kiekel, & Bell, 2004; Salas & Fiore, 2004). Furthermore, it is necessary to develop sensitive measures of collaboration and performance, which would accurately diagnose system and operator performance. In turn, these metrics, when used within a contextual theoretical framework, can indicate how to design superior technologies – taking into account both operator and hybrid team demands – for the successful implementation of human–robot teams. This chapter offers an approach based upon the integration of research and theory in both instructional and educational science domains in order to demonstrate how different metrics of performance can successfully measure and assess technology driven human–robot dynamics. Specifically, Cognitive Load Theory (CLT) will be described as it relates to multimedia information processing in order to generate a theoretical framework for use in human– robot teams. In turn, this framework will show how composite measures of individual levels of workload in relation to human–robot team levels of performance can afford useful efficiency indicators about human–robot interactions. Our overarching goal with this chapter is to discuss how such approaches add value to theory and research within human–robot teams.

COGNITIVE LOAD THEORY AND HUMAN–ROBOT TEAMS Cognitive Load Theory (CLT) is the product of over a decade of research in the instructional science domain (Chandler & Sweller, 1991; Sweller & Chandler, 1994), and its applications to other areas of inquiry continues to expand (see Cuevas, Fiore, & Oser, 2002; Paas, Renkl, & Sweller, 2003a; Paas, Tuovinen, Tabbers, & Van Gerven, 2003b; Scielzo, Fiore, Cuevas, & Salas, 2004). The core of CLT is based on two sets of what are termed cognitive load factors that are either endogenous or exogenous from the viewpoint of an operator interacting with the environment. Endogenous (or intrinsic) factors are sources of cognitive load in terms of the general amount and complexity of information with which the operator has to interact. In training environments, intrinsic load is directly proportional to the amount of materials that trainees need to acquire. As such, the more complex the

Cognition and Collaboration in Human–Robot Teams

331

information is in terms of volume and conceptual interactivity, the higher the cognitive load will be. In operational settings, high intrinsic load can occur whenever informational demands that need to be processed are high. Within the context of human–robot team environments, there is likely to be unique intrinsic load factors emerging from this hybrid teamwork interaction (e.g., information produced by synthetic team members). Another source of cognitive load comes from exogenous or extraneous factors. In training and operational settings alike, extraneous cognitive load may occur dependent upon the manner in which information needing attention is presented. Specifically, the more complex the human–robot team interface is in relation to the process by which information is displayed and/or communicated, the more extraneous cognitive load can be present. For example, the technological tools involved in the communication of information, and the associated modalities used to process information may inadvertently result in cognitive load. Simply put, high extraneous cognitive load can be produced as a result of using sub-optimal information presentation and communication. Overall, exogenous factors can stem from the added complexity of human–robot operations in terms of distinct command-and-control systems that emerge from using novel technology. Within such operations, it is particularly important to control sources of extraneous cognitive load that have been shown to produce two distinct negative effects on information processing – redundancy of information and split-attention. These have been shown to attenuate processing capacity thereby minimizing optimal information processing (e.g., Sweller, 1994; Mayer, 1999).

MULTIMODAL INFORMATION PROCESSING In human–robot environments, the operational information requiring attention by team members is mostly multimodal in nature, that is, there is likely to be an interaction between visual and audio informational components on top of team demands necessary for goal accomplishment. Schmorrow and colleges (Schmorrow, Stanney, Reeves, Kingdon, & Samman, 2003) highlighted the importance of investigating factors contributing to these multimodal interactions (e.g., sensory integration, sensory parallelism, and sensory transformation). Operators will likely have to attend to a number of simultaneous or sequential audio sources (e.g., team communication) while paying attention to visual stimuli (e.g., monitoring devices, displays, etc.) resulting in complex interactions between visual and audio stimuli. Understanding how this dynamic multimodal environment unfolds

332


in human–robot team interaction is a necessary precursor to support efficient and accurate information processing in the service of human–robot team operational effectiveness. In sum, within paradigms utilizing robotic technology, it is necessary to understand that the human component will be subject to two sets of stimuli. On the one hand, the technology behind the interface between human and machines will provide a set of visual and audio stimuli to assist the human operator in controlling or interacting with their robotic counterpart. On the other, the operator will continue to receive stimuli from the surrounding environment, in terms of team interactions, affecting all senses. Hence, multiple stimuli similar or dissimilar in nature will have to be attended to and processed simultaneously, which in turn will affect team behaviors. In concomitance with CLT, multimodal cognitive load becomes a complex phenomenon, which needs to be better framed in order to efficiently mitigate how and when multimodal information is dispersed. Such necessity warrants the development of a framework aimed at isolating critical factors involved in human-agent teams.

TWO-STAGE MODEL OF MULTIMODAL COGNITIVE LOAD Individual human operators within human–robot teams will inevitably have to make sense of incoming sensorial information by selecting, at a first stage, all that is necessary, and, at a second stage, to process the attended information. To better understand this process, Scielzo and Fiore (2005) proposed a two-stage model of cognitive load that focuses on information processing bottlenecks while interacting within complex multimodal environments (see Fig. 1). The utility of this model resides in its potential to indicate how to develop optimal multimodal interfaces by illustrating where attentional bottleneck may be prevented. Specifically, this model builds upon Wickens’ (1987) multiple resource theory and Mayer’s (2001) cognitive theory of multimedia learning in order to bring a comprehensive view on mental resource allocation during multimedia learning. Next, the model’s main components are presented, and a number of testable propositions to consider are brought forth in the context of human–robot teams. Sensory Stage This is the stage during which sensorial environmental information (i.e., sounds, text, visual aids) are perceived and attended to. The major


Fig. 1.

333

Two-Stage Model of Multimodal Cognitive Load.

bottleneck in this stage resides in splitting attention needs among the different sources of information, which will cause the operator’s limited attention resources to be depleted. Once information is selected and attended to, it is processed in the second stage.

Working Memory Stage This is the stage during which attention resources are used to attribute meaning to the attended stimuli and to process information in one of two working memory stores – audio and visual stores. These two specialized components are responsible for processing audio or visual information independently, and to combine such information with the help of the central executive element of working memory (see Baddeley & Hitch, 1974; Baddeley, 1992). In this stage, the major bottleneck occurs in the form of redundancy of information; that is, the necessary duplication of information across modalities (e.g., Mayer, 2001). Overall, two types of redundancies of information can arise in working memory: within-mode redundancy and between-mode redundancy (Scielzo & Fiore, 2005). On the one hand, within-mode redundancy refers to the duplication of information originating from narrative information (e.g., voice

334


communications) and visual symbols (e.g., use of textual information). The term within-mode refers to information that is processed within the audio working memory component. Specifically, within-mode redundancy will take place when a human voice or its synthetic equivalent, and text-based information (which, although visual in nature, is phonologically translated when attended to, see Baddeley, 1992) are processed concurrently. On the other, between-mode redundancy arises when simultaneous multimodal information is independently processed within working memory stores. For example, between-mode redundancy will arise when both visual and auditory information are combined in the central executive component of working memory. Within human–robot teams, between-mode redundancy will arise at two levels: (1) at the human–robot level, whenever the robotic interface will simultaneously present visual and audio information pertaining to operational parameters (e.g., UAV or UGV control functions) and (2) at the team level, whenever the operator has to integrate visual information from the robotic interface along with audio information (e.g., communication patterns) from other team members. Scielzo and Fiore (2005) suggest that redundancy can have a differential effect on information processing according to the level of congruency between the duplicated information (i.e., the conceptual overlap between different sources of information). For example, in the context of a human operator receiving both visual and auditory information regarding their robotic counterpart, if the similarity between the content of the two sources of information is high, congruency between the visual and auditory information will also be high (e.g., the operator sees and hears related information about robotic parameters). Conversely, discrepant information between visual and audio sources conveying dissimilar information will produce low congruency. As such, if at the human–robot level, visual information complementing each other is presented – such as in the event where both visual and audio sources simultaneously convey that a goal has been achieved (e.g., displays, Graphical User Interfaces, etc.) and audio (e.g., synthetic voice, cueing sounds or alarms, etc.) – generation of high levels of congruency will likely occur. At the team level, high congruency can be more difficult to achieve if communication patterns (e.g., teammate asking for system status) are generated while the operator is attending to the robotic interface. In sum, a high level of congruency for within- and between-mode redundancy will have a positive affect on the processing of information (e.g., Moreno & Mayer, 2002), whereas a low level of congruency will hinder information processing (e.g., Kalyuga, Chandler, & Sweller, 1999; Mayer & Sims, 1994; Moreno & Mayer, 1999).


335

PROPOSITIONS SUPPORTING HUMAN–ROBOT TEAM INTERACTIONS Below, we present a number of propositions addressing human–robot team interactions in terms of multimodal exchange of information, including minimizing attention depletion, maximizing congruent redundant information, and minimizing intrinsic and extraneous sources of cognitive load.

Minimizing Attention Depletion Human–robot technology should be used to minimize depletion of the operator’s limited attention resources at the sensory and working memory stages of multimodal information processing. Proposition 1. In the sensory stage, human–robot technology should employ salient visual cues guiding the operator’s focus (i.e., what stimuli to attend to) in order to reduce split attention. In terms of displays, contextsensitive deictic visual stimuli could reduce such split attention. Proposition 2. In the working memory stage, human–robot technology should display visual or textual information that helps multimodal information integration in order to aid in the decision-making process while minimizing attention resources depletion. Maximizing Congruent Redundant Information Human–robot technology should provide functional visual redundancies and prevent those that unnecessarily diminish attention resources. Proposition 3. Human–robot technology should verify that textual information does not detract the operator from processing audio information (e.g., voice or other environmental sounds that are sources of information), thereby maximizing within-mode redundancy. Proposition 4. Human–robot technology should maximize high congruency between auditory, visual, and text information in order to maximize between-mode redundancy.

336


Minimizing Intrinsic and Extraneous Cognitive Load Human–robot technology should minimize visual load emerging from the interaction of human–robot teams whenever possible in order to maximize information processing efficiency. Proposition 5. Human–robot technology should prioritize context relevant information and diminish superfluous visual information in order to minimize intrinsic cognitive load. Proposition 6. Human–robot technology should minimize extraneous cognitive load by reducing the spatial distance between different sources of visual information. For example, congruent text and visual information should be in close proximity whenever possible.

HUMAN–ROBOT TEAM TRAINING FRAMEWORK From the organizational sciences, research has shown that expert teams develop a shared mental model in which to coordinate behaviors by anticipating and predicting each team member’s needs and adapting to task demands (Cannon-Bowers & Salas, 1998; Salas & Fiore, 2004). Additionally, in expert teams, implicit and explicit coordination strategies are important in facilitating teamwork processes. As the execution of complex tasks is increasingly relying on human-agent teams, we must understand how expertise in such teams develops and how to diagnose the factors contributing to effective process and performance. In particular, it is important to note that although humans are more traditionally seen as interacting with a system in which the robot component is just an artificial entity, the nature of the interaction between humans and robots is more and more comparable to that of experienced human teams. This notion of human–machine systems is largely based on the salient characteristics robots typically have in relation to the extent they engage in autonomous decision-making activity. In other words, a robot is completely dependent on human input through an interface to accomplish operational tasks. However, this picture is rapidly changing in a large part due to robots’ increased artificial intelligence present in most UGVs and UAVs, which have a level of independence that allows for autonomous decision-making without human supervision (e.g., target recognition, re-plotting routes according to context, etc.). In this respect, human–robot teams are increasingly behaving more like expert human–human teams in terms of team characteristics (e.g., power distribution,

337


team resources, etc.) and processes (e.g., coordination, communication, conflict resolution, etc.). In short, as the level of technological sophistication continues to increase, for the sake of maximizing overall team performance, it is important to look at human–robot interactions as a form of team interactions (e.g., Fiore, Jentsch, Becerra-Fernandez, Salas, & Finkelstein, 2005; Sycara & Lewis, 2004). It is with this perspective in mind that our framework and team measures are developed. The importance of generating testable propositions within the context of human-agent teams is only warranted when valid and reliable cognitively diagnostic tools are used to help isolate team performance variability. As such, this section offers a theoretical framework to aid in diagnosing the effectiveness of technology-based interventions designed to support human– robot teams. This conceptual framework (see Fig. 2), adapted from the work of Fiore and colleagues (see Fiore, Johnston, Paris, & Smith, 2005) emphasizes human–robot dynamics in relation to multimodal processing of information. Furthermore, this framework combines approaches developed in the instructional sciences, with respect to CLT, to illustrate how subjective measures can be used in combination with task performance. This section first reviews the instructional efficiency metric developed by Paas and Van Merrie¨nboer (1993). Then, a description is provided on how this composite measure can be adapted as a diagnostic aid for the purpose of determining how agent-based tools alter workload, metacognition, and performance. Team Decision Efficiency When allowing for subjective assessments of workload with actual performance, CLT provides a unique level of diagnosticity to system assessment. In Human-Robot Processes

Subjective Self Assessments Individual Level Workload

Hybrid Team Process

Multimodal Cognitive Load Individual Level Metacognition

Fig. 2.

Objective Metrics

Combinatory Metrics Team Decision Efficiency

Team Performance

Team Metacognition Bias

Theoretical Framework Illustrating Combinatory Measures.

338


other words, ‘‘measures of cognitive load can reveal important information about the cognitive consequences of instructional conditions that is not necessarily reflected by traditional performance-based measures’’ (Paas & Tuovinen, 2004, p. 134). Within CLT, instructional efficiency is used to illustrate the combinatory effect of operators’ subjective assessment of workload and overall task performance (Paas, Van Merrie¨nboer, & Adam, 1994). Furthermore, CLT-based empirical studies employing instructional efficiency have been successfully used to show the manner in which manipulations in training system design can alter complex cognitive processes in terms of learning outcomes (e.g., Paas et al., 2003b; Tuovinen & Paas, 2004). CLT brings an important perspective to human–robot teams, training, and system design. In particular, it allows us to consider the relative efficiency of the human–robot team interactions. Human–robot team efficiency indicates how much cognitive resources are required for successful operation within hybrid teams as well as system overall performance. More importantly, instructional efficiency provides the possibility to isolate interventions or manipulations that actually maximize performance outcomes while minimizing mental load (Paas & Van Merrie¨nboer, 1993). To analyze such interventions, Fiore et al. (2005) adapted the Kalyuga et al. (1999) variant of the Paas and Van Merrie¨nboer (1993) instructional efficiency score. This score is calculated using standardized scores of workload (subjective assessment of mental effort) and performance. Specifically, this approach is adapted to show how instructional efficiency scores can be represented as the perpendicular distance from a line representing a level of zero efficiency with the formula: E¼

zperf

zwrkl pffiffi 2

(see to Kalyuga et al., 1999 for the specific derivation of this formula). Positive values are points above the zero efficiency line and are viewed as efficient, that is, the combination of high performance associated with lower subjective workload. Conversely, negative values are points below the zero efficiency line and are viewed as inefficient, that is, when low performance is combined with high workload. Overall, team decision efficiency is the result of the observed relation between subjective mental effort and task performance in a particular operational environment (Fiore et al., 2005). In this context, mental effort refers to the amount of resources an operator has to allocate to meet task demands or task induced cognitive load, which is assessed using mental workload ratings (Paas et al., 1994). Johnston, Fiore, Paris, and Smith (in


339

press) used this technique to document sensitivity to process and performance when technological interventions are introduced into the team environment. In particular, they found that the team decision efficiency score was able to illustrate how a decision support system simultaneously decreased workload within teams while facilitating performance.

Team Metacognition Bias Metacognition in hybrid teams (i.e., the ability to subjectively monitor one’s performance in terms of meeting operational and team demands) can be examined similarly to workload assessment through CLT. This involves the systematic investigation of metacognitive processes during task performance in order to determine the degree to which cognitive demands may be differentially burdened while interacting in human–robot environments. Specifically, the subjective appraisal of operator performance in conjunction with objective performance can be assessed in order to yield a measure of metacognitive bias (e.g., Fiore, Cuevas, Scielzo, & Salas, 2002). This new measure determines the degree to which operators are able to correctly monitor their own performance while engaged in complex tasks and team processes. Following the logic used in the previously described measure, by standardizing both performance prediction and actual performance, it is possible to calculate a metacognition bias index. As Fiore et al. (2005) illustrated, rather than using workload as with the prior formula, standardized scores of performance prediction along with standardized scores of actual performance are used. As such, a positive score indicates over confidence, a negative score indicates under confidence, and a score equal to zero indicates a level of hypothetical perfect metacognition (equal prediction and performance). Within the proposed theoretical framework, the Team Metacognitive Bias index offers the possibility of isolating performance decrements due to overestimation of one’s performance. Specific to human–robot team interactions, Team Metacognition Bias can help isolate sources of bias due to both the discrepancy between overall team performance and performance prediction, and the discrepancy between actual robotic performance and estimated robotic performance. It is necessary to mitigate such sources of bias if overall human–robot team performance is to be maximized. For example, the Team Metacognition Bias score can enable teams to better remediate through more targeted training by helping to identify areas of overconfidence to suggest which skills/behaviors may require added training.

340


CONCLUSION The propositions set forth in this chapter along with their contextual theoretical framework are intended to provide a general idea of the complex interactions between a variety of cognitive load factors and multimodal information processing demands. Furthermore, this chapter wishes to inspire future research to investigate how technological manipulations in human–robot environments can facilitate information processing. Specifically, when exploring paradigms that are based on human-agent environments, it is necessary to isolate and comprehend the cognitive mechanisms involved in processing information from robotic interfaces while achieving operational and team goals. Failing to achieve this objective may prevent accurate guidance in the technological components of interface development. Overall, this chapter described how incorporating elements of multimodal information processing within a human-agent theoretical framework can augment models of teamwork and team process by taking into consideration various aspects of workload and by providing measures capable of innovatively diagnosing human–robot team performance. By viewing human– robot teamwork from the framework of easing extrinsic cognitive load to increase efficiency, additional methods for using robot team members to alter workload and performance can be developed. The presented framework provides such an opportunity while offering tangible measures of hybrid human-agent teams. Testing and validation are the logical next steps to verify to what extent hybrid team workload can be modeled around CLT and multimedia load on one side and metacognitive load on the other. Ultimately, the goal is for research to illustrate that team decision efficiency and team metacognition bias can be diagnostic of overall hybrid humanagent team performance. In short, our measurement framework provides the appropriate level of diagnosis to test the propositions put forth in our model. We hope that ongoing research in this field will soon provide the data by which to validate and further refine our theory and measures.

REFERENCES Baddeley, A. D. (1992). Working memory. Science, 255, 556–559. Baddeley, A. D., Hitch, G. J. (1974). Working memory. In: G. A. Bower (Ed.), Recent advances in learning and motivation (Vol. 8, pp. 47–90). New York: Academic Press. Cannon-Bowers, J. A., & Salas, E. (1998). Making decisions under stress: Implications for individual and team training. Washington, DC: American Psychological Association.


341

Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and Instruction, 8(4), 293–332. Cooke, N. J., Salas, E., Kiekel, P. A., & Bell, B. (2004). Advances in measuring team cognition. In: E. Salas & S. M. Fiore (Eds), Team cognition: Understanding the factors that drive process and performance. Washington, DC: American Psychological Association. Cuevas, H. M., Fiore, S. M., & Oser, R. L. (2002). Scaffolding cognitive and metacognitive processes: Use of diagrams in computer-based training environments. Instructional Science, 30, 433–464. Fiore, S. M., Cuevas, H. M., Scielzo, S., & Salas, E. (2002). Training individuals for distributed teams: Problem solving assessment for distributed mission research. Computers in Human Behavior, 18, 125–140. Fiore, S. M., Jentsch, F., Becerra-Fernandez, I., Salas, E., & Finkelstein, N. (2005). Integrating field data with laboratory training research to improve the understanding of expert human-agent teamwork. In the IEEE Proceedings of the 38th Hawaii international conference on system Sciences, Los Alamitos, CA. Fiore, S. M., Johnston, J., Paris, C., & Smith, C. A. P. (2005). Evaluating computerized decision support systems for teams: Using cognitive load and metacognition theory to develop team cognition measures. Proceedings of the 11th international conference on Human– Computer interaction. Las Vegas, NV: HCII. Johnston, J., Fiore, S. M., Paris, C., & Smith, C. A. P. (in press). Application of cognitive load theory to developing a measure of team decision efficiency. Military Psychology. Kalyuga, S., Chandler, P., & Sweller, P. (1999). Managing split-attention and redundancy in multimedia instruction. Applied Cognitive Psychology, 13, 351–371. Mayer, R. E. (1999). Instructional technology. In: F. T. Durso, R. S. Nickerson, R. W. Schvaneveldt, S. T. Dumais, D. S. Lindsay & M. T. H. Chi (Eds), Handbook of applied cognition (pp. 551–569). Chichester, England: Wiley. Mayer, R. E. (2001). Multimedia learning. Cambridge, England: Cambridge University Press. Mayer, R. E., & Sims, V. K. (1994). For whom is a picture worth a thousand words? Extensions of a dual-coding theory of multimedia learning. Journal of Educational Psychology, 86, 389–401. Moreno, R., & Mayer, R. E. (1999). Cognitive principles of multimedia learning: The role of modality and contiguity. Journal of Educational Psychology, 91, 358–368. Moreno, R., & Mayer, R. E. (2002). Verbal redundancy in multimedia learning: When reading helps listening. Journal of Educational Psychology, 94, 156–163. Paas, F., Renkl, A., & Sweller, J. (2003a). Cognitive load theory and instructional design: Recent developments. Educational Psychologist, 38, 1–4. Paas, F., Tuovinen, J., Tabbers, H., & Van Gerven, P. W. M. (2003b). Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist, 38, 63–71. Paas, F., & Tuovinen, J. (2004). Exploring multidimensional approaches to the efficiency of instructional conditions. Instructional Science, 32, 133–152. Paas, F., & Van Merrie¨nboer, J. (1993). The efficiency of instructional conditions: An approach to combine mental effort and performance measures. Human Factors, 35, 737–743. Paas, F., Van Merrie¨nboer, J. J. G., & Adam, J. J. (1994). Measurement of cognitive load in instructional research. Perceptual and Motor Skills, 79, 419–430. Salas, E., & Fiore, S. M. (Eds) (2004). Team cognition: Understanding the factors that drive process and performance. Washington, DC: American Psychological Association.

342


Schmorrow, D., Stanney, K., Reeves, L., Kingdon Hale, K., & Samman, S. (2003). Multimodal interaction design. Tutorial presented at HCI International, Crete, Greece (June). Scielzo, S., & Fiore, S. M. (2005). Augmenting cognition with augmented reality: A multimedia cognitive load approach for extreme environment research. In: Proceedings of the society for human performance in extreme environments 2nd annual meeting (pp. 21–25). Orlando, FL: HPEE. Scielzo, S., Fiore, S. M., Cuevas, H. M., & Salas, E. (2004). Diagnosticity of mental models in cognitive and metacognitive processes: Implications for synthetic task environment training. In: S. G. Schiflett, L. R. Elliott, E. Salas & M. D. Coovert (Eds), Scaled worlds: Development, validation, and applications (pp. 181–199). Aldershot, UK: Ashgate. Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction, 4, 295–312. Sweller, J., & Chandler, P. (1994). Why some material is difficult to learn. Cognition and Instruction, 12, 185–233. Sycara, K., & Lewis, M. (2004). Integrating intelligent agents into human teams. In: E. Salas & S. M. Fiore (Eds), Team cognition: Understanding the factors driving process and performance (pp. 203–231). Washington, DC: American Psychological Association. Tuovinen, J. E., & Paas, F. (2004). Exploring multidimensional approaches to the efficiency of instructional conditions. Instructional Science, 32, 133–152. Wickens, C. D. (1987). Information processing, decision-making, and cognition. In: Salvedy Gavriel (Ed.), Handbook of human factors (pp. 72–107). Oxford, England: Wiley.

ROVS ON THE GROUND

Remote vehicles can operate in a variety of mediums. While previous sections concentrated on vehicles operating in the air, this section turns to vehicles operating on the ground. First, A. William Evans overviews the various means of evaluating human–robot interfaces on the ground: simulations, scale environments, and full size facilities. Importantly, he provides a framework for evaluating the strengths and weaknesses associated with each approach, thereby giving researchers a means to find the most appropriate one for their studies. Next, Jennifer Riley’s chapter takes a look at situation awareness in the urban search and rescue operations. She illustrates problems in situation awareness on the ground, from robot localization to inadequate support for team operations that are similar to problems in situation awareness in the air. Finally, Roger Chadwick explores the interaction of the perceptual and cognitive processes of the operator with the capabilities of the remote vehicle and its displays. The chapter demonstrates important trade offs between the benefit of information and cost of getting it. Overall, these chapters point to parallels between remote operations on the ground and remote operations in the air.

343


344

24. EXPLORING HUMAN–ROBOT INTERACTION: EMERGING METHODOLOGIES AND ENVIRONMENTS A. William Evans III, Raegan M. Hoeft, Florian Jentsch, Sherri A Rehfeld and Michael T. Curtis Congressional mandate has decreed that by the year 2015, 33% of all the United States military ground forces shall be uninhabited or robotic in nature (National Research Council, 2003). While preparations have already begun for this effort, remotely operated vehicles (ROVs), both operating on the ground and in the air, are comparably new tools on the battlefield. Still, they have already been employed in Operations Enduring Freedom (Afghanistan) and Iraqi Freedom (Weinberger, 2004). Robotic agents, being either completely tele-operated or tele-navigated, are in use as aids for reconnaissance tasks and for search and rescue missions, such as in the recent Russian mini-sub rescue (CNN.com, 2005). Additionally, ROVs have been used for a while now in bomb disposal. The need for robotic agents does not end there, but will continue to expand into more dangerous situations. In the types of missions mentioned above, robotic agents currently operate either semi-autonomously (aerial ROVs flying a predetermined route for


345

346

A. WILLIAM EVANS III ET AL.

reconnaissance) or, more often, completely tele-operated (full remote control of bomb disposal unit). However, the future of robotic agents will move them into combat, an area that will require the use of more and more autonomous, robotic teammates to keep human soldiers out of harm’s way. Thus, we can say with certainty that, within the next decade, ROVs will increasingly be utilized on the battlefield to augment airmen, sailors, soldiers, and marines (CNN Technology, 2004). In combat situations, where it is expected that the use of uninhabited units will provide the largest advantages over the use of human units, it is important that robotic agents should be able to adapt to changes in both the environment and mission plans. This push toward the integration of human soldiers and robots has led to the emergence of Human-Robot Interaction (HRI) research in a variety of domains. Supporting the inclusion of ROVs into the military will require research with the purpose to understand and enhance the interaction of a number of soldiers with multiple robotic systems that vary in size and functioning from small ground ROVs (used for bomb disposal), to medium sized aerial reconnaissance ROVs, to six-ton armored robot vehicles (ARVs), used as command stations. Some of the recent efforts in HRI research have investigated the ratio of operators to vehicles (MacMillan & Johnson, 2004), the perceptual and cognitive issues affecting operators of multiple robots (Chadwick, Gillan, Simon, & Pazuchanics, 2004; Rehfeld, Jentsch, Curtis, & Fincannon, 2005; Goodrich & Olsen, 2003), unique team issues in the human-robot context (Hoeft, Jentsch, & Bowers, 2005) and situation awareness of operators in search and rescue missions (Murphy, 2005; Riley & Endsley, 2004). Recently, research has been conducted through a variety of test-bed media to facilitate the assessment of HRI. Three of the most prominent classes of test-beds are (a) computer simulations (including first person shooters [FPS] and squad-based simulations, such as Full Spectrum WarriorTM, as well as non-commercial simulations), (b) scale facilities of realistic environments (such as those found at New Mexico State University and the University of Central Florida), and (c) full-scale test-beds (such as the experimental utility vehicle [XUV] program at Ft. Indiantown Gap [Schipani, 2003]). Examples of these test-beds can be found in Fig. 1. In this chapter, we describe the various environments and methodologies currently being used to study HRI, and we compare and contrast each of their respective strengths and weakness for exploring different HRI issues. We have also included recommendations as to the types of HRI research which could benefit from employing the various methodologies.

Emerging Methodologies

347

Fig. 1. (a) Sample Screen Shot of Computer Simulation Battlefield 1942TM (Obtained from eagames.com), (b) Scale Facility Vehicle Camera View of UCF MOUT, and (c) Full Size XUV Test Vehicle (Photograph Available Through Department of Defense).

348


COMPUTER SIMULATION Background Computer simulation is a test-bed for research that began in the 1970s, grew tremendously in popularity in the 1990s, and has since continued to mature in complexity and realism. When computer simulations were in their infancy, their biggest advantage was the ability to have complete control over the environment in which the simulation took place. Missions could be changed from daylight to twilight with a few keystrokes. Weather conditions could be altered or inserted based on the needs of the experiment. Perhaps most importantly, the landmasses in which the simulations took place were boundless, in their cyber world. Caro (1988) showed that computer simulations used in aviation training could be highly effective tools. In fact, in the aviation industry, computer simulations have all but eliminated the terrain boards (scale models of real environments that were filmed for use in flight simulators), which were used for simulation purposes in the past. Jentsch and Bowers (1998) have shown that computer simulations of lower physical and functional fidelity can also be useful at training critical knowledge and skills, thereby, demonstrating that as far as fidelity is concerned, ‘‘more is not always better.’’

Advantages Limitless Terrain Modern commercial off-the-shelf (COTS) simulations come in a variety of forms. The two most common control platforms for research, specifically in the area of teamwork, are FPS and squad based simulations. COTS simulations can span many eras throughout history: some take place on the backdrop of World War II (Battlefield 1942TM; Electronic Arts Games, 2004), others move into more modern conflicts such as Vietnam and Operation Restore Hope in Mogadishu, Somalia (Delta Force: Black Hawk DownTM; Novalogic, 2002), and more yet take place in the modern era simulating Operation Iraqi Freedom (Full Spectrum WarriorTM, THQ Inc. 2003), and other anti-terrorism action (Rainbow Six 3: Raven ShieldTM; UbiSoft, 2003). Computer simulations can essentially be developed to take place in any location, real or imaginary, during any time.

349


Realistic Viewpoints While only a few of the above-mentioned computer simulations may include models for ROVs, most still allow researchers the capability to measure interaction between a human and a simulated agent. This interaction could take many forms, from interacting with simulated teammates (squad based simulations) to controlling vehicles through the computer interface (simulating tele-operation). It is this simulated control that will provide users with an experience that is similar in nature to control an ROV. Users will have limited viewpoint angles, which will resemble those gained through ROV sensory equipment. Further, control of those viewpoints, through rotating camera views, will also be much the same, in terms of functioning, to those found on actual ROVs, providing a heightened sense of realism for the user.

Drawbacks Limited Physical Fidelity Computer simulations are not without their drawbacks, though. Simulations with fidelity-approaching reality require not only large amount of time to develop, but necessitate highly experienced programmers to utilize all of the abilities of the software, which all translate into increased costs. Moreover, while simulations have grown in their sophistication when modeling military forces, civilian entities are lacking in both quality and quantity. Neutral objects such as buildings and non-military vehicles are not generally interactive in these simulations, meaning, for example, that a user could drive a tank into a building and the tank would either stop or continue right on through the building, with neither the tank nor the building incurring any damage. This type of interaction can lead to false beliefs and inaccurate mental models about the abilities of the equipment, contributing to unsafe and reckless control behaviors.

Practical Uses Fitting Environment to Needs Computer simulations do have a place in the world of HRI research. Rapid availability of COTS simulations makes them ideal for experiments that do not require a great deal of individual augmentation of missions. COTS simulations have shown great increase in artificial intelligence (AI), which

350


can be used to represent autonomous robotic teammates. The real key is finding the simulation that best suits individual research needs. For example, if investigating control of several robotic agents simultaneously, perhaps a squad based simulation dependent on command and control orders provided by the user would be the best choice. Conversely, research relating to emotion and robotic agents might want to consider FPS simulations that allow the user to directly interact with AI agents within a more immersive environment. Realism is Key It should be noted that there are some instances when the use of a computerbased simulation may not be the best choice for conducting research. For example, Salas, Bowers, and Rhodenizer (1998) determined that high levels of physical fidelity are necessary to attain certain training goals. COTS computer simulations have a tendency to be either high fidelity with a low amount of adaptable functions or low physical fidelity with a large amount of adaptable functions. The lack of a high fidelity program that is readily available with a lot of adaptable functions can prove to be a hindrance in those research programs that are dependent on higher levels of realism. Therefore, the details of the task and resulting training that is necessary will dictate the need for the fidelity of the computer simulation.

SCALE ENVIRONMENTS Background Scale environments are not novel to the simulation field. The aviation industry has used terrain boards extensively in the past. This form of simulation, however, has fallen out of favor until recently. Often seen as archaic in an ever increasingly technological world, creating scale models for simulation purposes have often been dismissed as too ‘low tech’. However, researchers at several American universities have found that scale model facilities have proven to be very useful in their research endeavors. Chadwick et al. (2004) used scale models of caves at New Mexico State University to study the control of multiple robotic agents by human personnel. Researchers at the University of Central Florida (UCF; Jentsch et al., 2004) created a 1:35 scale MOUT (Military Operations in Urban Terrain) research facility to study the interactions between humans and robots in team

351


environments. The MOUT facility at UCF allows researchers to manipulate a number of variables, including the number of vehicles, operators and robotic assets, operator-to-robot ratios, and other external factors such as mission difficulty, ROV performance reliability, as well as the observation and measurement of operator, and robot performance within the mission evaluations.

Advantages Limited Materials Costs Through the use of such scale facilities, researchers are able to create realistic environments to their exacting research needs. Using remote controlled vehicles in place of actual robotic agents, researchers are able to create the illusion of autonomous vehicles, with the aid of confederates, without the need for sophisticated and expensive equipment. Overall, scale environments can be relatively inexpensive to create, requiring some vehicle and figure models, materials to create rooms or buildings, and an area in which to house the facility. The MOUT facility at UCF costs approximately $15,000 for materials. High Physical Fidelity Researchers at UCF, using a lab space of approximately 150 200 , created a 1:35 scale environment, which would represent an area, scaled in size, similar to military action in Mogadishu, during Operation Restore Hope (made famous through Mike Bowden’s book Black Hawk Down and the subsequent Hollywood movie) and in An-Nasariyah, during Operation Iraqi Freedom. Using tactics, such as winding roads, effective mission distances can be created that are comparable to 3–5 km in the real world. These distances also are similar in size to those used in the full size XUV testing facility in Ft. Indiantown Gap, PA. Combining these characteristics with the use of miniature cameras allows researchers to create an environment that appears realistic when viewed through a television monitor. In fact, not only does the environment appear real, but stimuli within the environment react in realistic ways. For example, if one of the robotic vehicles hits a tree, that tree will fall over; the same applies to a house, person, or another vehicle. Because of this, users are likely to feel a greater sense of consequence when performing in a scale environment, as compared to a computer simulation.

352


Drawbacks Labor Intensive Use and creation of such scale facilities are not without certain inconveniences. Labor for constructing such facilities can quickly grow to great proportions. Within university settings, this concern is somewhat reduced because of the involvement of many graduate and undergraduate students who are willing to put a great deal of effort into building a scale facility; however, this may prove to be a more daunting task in the private sector. Size itself can also become a factor contributing to difficulties in creating a scale facility. Securing 300 square feet of space (just for the actual environment, not including supporting areas that are needed) is not expected to be a simple task. As well, the use of a scale facility can necessitate a number of personnel to be available to operate it. For example, the scale MOUT facility at UCF requires three experimenters for a trial involving two ROVs: one to interact with participants and two to act as confederates to provide the ROVs’ ‘automation’. Technological Issues Scale facilities also have some innate issues stemming from the use of a large amount of wireless or remote controlled equipment. Interference, and the control of said interference, can play a major role in the success or failure of a scale facility. Proper planning of the equipment to be used is critical as the use of too much equipment on similar frequencies can lead to vehicle and/or camera failure.

Practical Uses Flexibility Scale facilities, still, can provide a great deal of flexibility in research for those with enough personnel to make them work. University research programs generally have a resource for many laborers for minimal costs and can produce and operate a scale facility with very limited expenses. Even for private companies the use of a scale facility is not unreasonable, but simply requires adequate preplanning to ensure that there is a minimum of waste materials and labor. Scale facilities, once in operation, can quickly be modified for new experiments/scenarios. Simply moving target items to new locations within the

353


facility or changing out critical components can provide researchers with a flexible test-bed that can be utilized for an unlimited amount of research. In addition to these potential changes the multitude of COTS remote-controlled vehicles (cars, tanks, planes, etc.) provide researchers with the ability to test several variants of ROVs using the same facility with only minor changes to hardware within it. Low Dynamic Stimuli It is also important to realize some of the limitations associated with scale facilities. Besides being able to provide highly realistic views for participants, scale facilities can be lacking in certain dynamic features. Most targets/ stimuli within scale facilities are static. These static targets are not able to provide the same realism through movement that can be found in computer simulations and full-scale test-beds. This is not necessarily an issue for all studies but research that requires target and/or stimuli movement may want to strongly consider alternatives before turning to the scale facility as the test-bed of choice.

FULL SCALE ENVIRONMENTS Background Full-scale environments for the study of HRI obviously provide the greatest sense of realism, as human users are able to interact with the actual robots (or very similar replications) that would be used in the field by military personnel. Several test-beds for full-scale experimentation exist, and in addition, contests are regularly held to challenge designers to create more usable and reliable robotic units. One such event is the Defense Advance Research Projects Agency (DARPA) Grand Challenge, an annual event that recently has attracted nearly 200 entries (DARPA, 2005). In the Grand Challenge, teams attempt to build vehicles that will travel completely autonomously across 175 miles of desert terrain in 10 h. Thus far, no one has succeeded at that goal but invaluable lessons have been learned from the experience and the two million dollar prize could be awarded very soon. Full size test-beds do not only work with fully autonomous vehicles. At Ft. Indiantown Gap, PA, the XUV has been undergoing a number of

354


tests (Schipani, 2003). The modified Humvees is controlled remotely via a control station mounted in a pursuit vehicle. This vehicle is able to be tele-operated, given a number of predetermined route plans to execute, or operated in tandem as part of a convoy of XUVs.

Advantages Real World Equipment Comparisons Using full size testing facilities has obvious advantages when it comes to realism and transferability. Working with actual prototypes (or in some cases already-proven designs), researchers, using full-scale test-beds, are able to acquire a superior view into the actual relationships that users will have with robotic agents. Since all facets of a full-scale facility will be actual working components and not just ‘man behind the curtain’ simulations, researchers will gain access to data in all areas of HRI. Experiments can focus on user interface design, team design, sensory needs, and any other issue that may arise concerning human control of robotic agents.

Drawbacks Expensive Equipment Full-scale facilities have some inherent problems associated with their use in research. To begin, the technology needed to create the robotic units comes only at a premium price. With very few mass-produced components at this stage of development, virtually every piece of equipment is custom built. Costs do not end there, as large proving grounds are needed to operate ROVs at these sites and, in most cases, many support personnel are required to ensure smooth operation.

Real-World Damage Some dangers exist in the use of full size facilities, as well. Compared to scale facilities and computer simulations the damage incurred when a ROV hits an object is far more severe. Buildings can be destroyed, vehicles crushed, and people injured, not to mention the damage that can be inflicted on ROVs, costing millions of dollars.

355


Practical Uses Money to Spend Full size testing facilities can be of great value to those research projects with enough budget to support such an endeavor. Even the best modeling software can have trouble replicating the real world environments able to be used with a test-bed of this type. Research teams with smaller budgets may even benefit from the expanding availability of such facilities. DARPA, through the Grand Challenge, and other agencies throughout the world, have begun providing such facilities for suitable equipment tests, so that research dollars can be concentrated onto actual equipment and not facilities. Better for Equipment than Relationships When considering HRI in robotics research, full size test-beds may be more than is needed. To test the interactions of potential users with robotic agents, it is not of vital importance that there is full size equipment at users’ disposal. In fact, with the expectations that users will be established in a safe location while robotic units travel into far more dangerous areas, it is only important to provide the illusion of controlling a full size vehicle, which we have already shown can be accomplished through the use of either computer simulations or full-scale facilities, as an alternative.

CONCLUSION The goals of this chapter are (a) to present an overview of the current practices of many HRI researchers and (b) to present enough information about the various test-beds to allow researchers to make informed decisions about which environments might be of most value for their specific research endeavors. There are a number of factors that need to be taken into account when considering which test-bed is appropriate for answering a specific research question; some of these are presented in Table 1. Each of the three test-beds discussed in this chapter has its advantages and disadvantages, and it is up to each individual researcher to weigh the pros and cons in choosing the desired methodology for his or her experiments. None of the previously mentioned methods would be incorrect to use in any specific situation, they may just lead to a more or less efficient means of finding the answers to one’s research questions. The rapidly

Characteristics of the Different Test-Beds.

Computer Simulations Acceptance Cost Fidelity

Flexibility

Personnel

Size/Space

Start up/ preparation

Miscellaneous

High user acceptance Affordable Available COTS versions Limited physical fidelity

– Moderate to high environmental flexibility – Modification programming required – Skilled programmers required

– Reusable software – Changes have moderate costs associated with them – Limitless terrain – Minimal lab space required

– Minimum start up time and cost for COTS software – Longer times and greater costs for custom simulations – Easily accessible

Scale Environments – – – –

Moderate user acceptance Minimal overall materials cost Moderate overall labor costs High physical fidelity

– High environmental flexibility

– Moderate personnel requirements for operation – Labor intensive creation

– Reusable facility – Changes have minimal costs associated with them – Moderate to large lab space requirements – Adjacent space required for operators – Moderate start up time and costs for facility – Moderate training required for operation – Technological issues (radio frequency interference)

Full Size Facilities – High military acceptance – Expensive equipment required – Highest physical fidelity – Provides most generalizable results – Low environmental flexibility – Low external controls

– High personnel requirements for operation, military approved personnel only – High organizational management and logistics requirements – Reusable facilities – Major costs associated with changes to facility – Large space requirements – Location concerns (safety and security) – Intensive start up costs and time – Extensive training needed for operation – Can result in real world damage


Reusability

– – – –

356

Table 1.

357


evolving realm of robotics requires keen evaluation techniques to ensure that the humans are able to keep up with their new teammates.

REFERENCES Caro, P. W. (1988). Flight training and simulation. In: E. L. Wiener & D. C. Nagel (Eds), Human factors in aviation (pp. 229–262). San Diego: Academic Press. Chadwick, R. A., Gillan, D. J., Simon, D., & Pazuchanics, S. (2004). Cognitive analysis methods for control of multiple robots: Robotics on $5 a day. In: Proceedings of the human factors and ergonomics society 48th annual meeting, New Orleans, LA (pp. 688–692). CNN.com. (2005). Crew safe after mini-sub rescue. Retrieved August 8, 2005, from http:// www.cnn.com/2005/WORLD/europe/08/07/russia.sea/index.html. CNN Technology. (2004). Firm cheers loss of robot in Iraq. Retrieved April 13, 2004, from http://www.cnn.com/2004/TECH/04/12/tech.arms.robots.reut/index.html. DARPA. (2005). 195 Teams sign up to compete in the DARPA Grand Challenge. Retrieved March 10, 2005 from http://www.darpa.mil/grandchallenge/GC05FinalApps2-15-05PR.pdf. Electronic Arts Games. (2004). Battlefield: 1942. [computer software]. Redwood City, CA. Goodrich, M. A., & Olsen, D. R. Jr. (2003). Seven principles of efficient interaction. In: Proceedings of IEEE international conference on systems, man, and cybernetics, Denver, CO (pp. 3943–3948). Hoeft, R. M., Jentsch, F., & Bowers, C. (2005). The effects of interdependence on team performance in human-robot teams. In: Proceedings of the 11th annual international conference of human-computer interaction. [CD-ROM]. St. Louis, MO: Mira Digital Publishing. Jentsch, F., & Bowers, C. A. (1998). Evidence for the validity of PC-based simulations in studying aircrew coordination. International Journal of Aviation Psychology, 8, 243–260. Jentsch, F., Evans, A. W. III, Feldman, M., Hoeft, R. M., Rehfeld, S., & Curtis, M. (2004). A scale MOUT facility for studying human-robot interaction and control. Proceedings of the 24th annual army science conference. Orlando, FL. MacMillan, J., & Johnson, C. L. (2004). Human control of teams of unmanned/robotic vehicles: Exploring the limits of the possible. In: Proceedings of the human factors and ergonomics society 48th annual meeting, New Orleans, LA (pp. 523–527). Murphy, R. R. (2005). Humans, robots, rubble, and research. Interactions, 12(2), 37–39. National Research Council. (2003). Building unmanned group vehicles requires more funding, greater focus. Retrieved on July 22, 2004 from http://www4.nationalacademies.org/ news.nsf/isbn/0309086205?OpenDocument. Novalogic, Inc. (2002). Delta Force: Black Hawk Down. [computer software]. Calabasas, CA. Rehfeld, S. A., Jentsch, F. G., Curtis, M., & Fincannon, T. (2005). Collaborative teamwork with unmanned ground vehicles in military missions. In: Proceedings of the 11th annual international conference of human-computer interaction. [CD-ROM]. St. Louis, MO: Mira Digital Publishing. Riley, J. M., & Endsley, M. R. (2004). The hunt for situation awareness: Human-robot interaction in search and rescue. In: Proceedings of the human factors and ergonomics society 48th annual meeting, New Orleans, LA (pp. 693–697).

358


Salas, E., Bowers, C., & Rhodenizer, L. (1988). It is not how much you have but how you use it: Toward a rational use of simulation to support aviation training. International Journal of Aviation Psychology, 8, 197–208. Schipani, S. (2003). An evaluation of operator workload, during partially autonomous vehicle operations. Proceedings of PerMIS_2003, Galthersburg, MD. THQ Inc. (2004). Full spectrum warrior. [computer software]. Calabasas Hills, CA. UbiSoft. (2003). Tom Clancey’s Rainbow Six 3: Raven Shield. [computer software]. San Francisco, CA. Weinberger, S. (2004). Military sending 163 robots to Iraq as ‘stopgap’ measure against IEDs. Defense Daily News, April 30, p. 4.

25. SITUATION AWARENESS IN THE CONTROL OF UNMANNED GROUND VEHICLES$ Jennifer M. Riley, Robin R. Murphy and Mica R. Endsley Robotic systems are proving to be assets in many civilian and military operations, particularly those that are poorly suited to physical insertion of human operators (e.g., exploration of collapsed structures in which areas requiring visual search are too small and/or unstable for human entry) (Murphy, 2004). In these situations, robotic entities become the remotely controlled eyes, ears, and sometimes hands, of the human operator. The robots are used to project the operator’s presence and intent upon objects in the remote space. The effectiveness of human interactions with the robot system drives the overall operational performance. The quality of this human–robotic interaction (HRI) is highly dependent upon the operator’s ability to develop and maintain situation awareness (SA) during performance.

$

Writing by the first and third author was prepared through participation in the Advanced Decision Architectures Collaborative Technology Alliance sponsored by the U.S. Army Research Laboratory (ARL) under Cooperative Agreement DAAD19-01-2-0009. The views and conclusions contained herein, however, are those of the authors and should not be interpreted as representing the official policies, either expressed or implied of the ARL or the U.S. Government.


359

360

JENNIFER M. RILEY ET AL.

An important issue in the control of remotely operated vehicles is the limitation in development of SA due to impoverished sensory information, attentional resource limits, task/environmental stressors, and system design faults. In addition, where autonomy is added to compensate for some of these problems and provide for a lower operator-to-vehicle ratio, problems associated with supervisory control can contribute to SA challenges. This chapter discusses the importance of SA and provision of SA requirements through the interface, effects of task, environmental and system factors on operator SA, and interface design issues that impede SA development during task performance. Examples of SA challenges are provided in the context of robot-assisted urban search and rescue (USAR) operations and are derived from extensive experience in this environment and naturalistic observations of remote control operations.

SITUATION AWARENESS AND CRITICAL INFORMATION REQUIREMENTS SA has been described as ‘‘generating purposeful behavior.’’ that is, behavior that is directed toward a task goal (Smith & Hancock, 1995). It involves being aware of what is happening around you and understanding what occurring events mean with respect to your current and future goals. Endsley (1995) has formally defined SA as the ‘‘perception of elements in the environment within a volume of time and space, the comprehension of their meaning and the projection of their status in the near future’’ (Endsley, 1995, p. 36). SA has been hypothesized as being critical to operator task performance in complex and dynamic operations (Salas, Prince, Barker, & Shrestha, 1995), like tasking and controlling remotely operated systems. Operators in remote control of ground vehicles need to be aware of where the vehicle is, what the vehicle is doing, and how activities as part of the overall task lead to accomplishment of mission goals. They must also consider the health of the overall system and how the environment affects vehicle status and the ability to complete tasks. In studying robot control in simulated USAR operations, Drury, Scholtz, and Yanco (2003) observed that most of the problems encountered when navigating robots resulted from the human’s lack of awareness of these elements. To support SA in HRI it is important to understand what the SA requirements are for tasking and controlling the robot in an environment. In teleoperation, the robotic vehicle is an element of task performance for which the human operator must build and maintain SA. The robot, and its associated

SA in the Control of UGVs

361

displays and interfaces, is also the operator’s primary means for developing SA on elements and events in the task environment. Further the operator needs to be concerned with knowledge of the status of task performance or progress toward meeting operational goals. Thus, to support HRI one must gain perspective on information requirements that are relevant to the task, system, and environment, which can influence overall performance.

SITUATION AWARENESS REQUIREMENTS ANALYSIS Information or SA requirements focus on the dynamic information needs relevant to a particular job or task domain (Endsley, 2000). Goal-directed (cognitive) task analysis (GDTA) can be used to delineate an operator’s SA requirements (Endsley, 1993). This methodology seeks to determine what operators would ideally like to know to meet each of the goals as part of a task. In most operational settings, including control of robotic vehicles, it is difficult to provide all of the SA requirements. In some cases, desired information is not available with current technologies. The focus, however, on defining an ideal set of requirements for SA in the GDTA process allows us to consider much of the information that the operator would really like to know without limiting future designs based upon current technology (Endsley, Bolte, & Jones, 2003). In GDTA, analysts elicit the goals and critical decisions involved as part of operator performance during interviews with subject matter experts (e.g., robot controllers, USAR personnel). There may be several subgoals as part of each goal analysis, and multiple decisions to be met for goals. Specific information needs at three levels of SA, basic perception (Level 1), comprehension (Level 2), and future projections (Level 3), are then mapped to these decisions. For example, in tasking remotely piloted vehicles, an important subgoal is ‘‘avoid collisions.’’ To meet this subgoal, vehicle operators must determine if an obstacle is present and decide if the obstacle can be avoided. These decisions require the operator to have developed Level 1 SA (perceptions) on the location, heading and speed of the remote control vehicle, as well as on characteristics of the terrain or potential obstacle (e.g., size, distance, etc.). For building SA and making decisions, these data items usually must be cognitively integrated for comprehending a situation and projecting what needs to be done about it. To learn more about perceptual issues that lead to SA problems in HRI, GDTA was conducted for teleoperation of ground robots for USAR. The

362


Table 1.

High-Level Robot-Assisted USAR Goals and Subgoals.

1. Navigate in environment 1.1. Find/stay on path, move in correct direction 1.2. Avoid collisions 1.3. Track position (localization) 2. Maintain ‘‘good’’ working status for vehicle 3. Communicate critical information 4. Assess safety of area (structural assessment) 5. Detect/locate victims

Table 2.

5.0

Example Decisions and SA Requirements for Robot-Assisted USAR Goal.

Goal

Critical Decisions

Some SA Requirements

Detect/locate victims

Have I searched this area?

Landmarks Terrain features

Is this object a victim?

Object characteristics Size, shape, form, color, movement, fluids, location

Where should I look next?

Auditory signals

What is the location of the victim?

Projected localization Current location of robot Past locations of robot Current heading

GDTA provided insight into the information critical to USAR success and provided a basis for identifying operational issues leading to poor SA. Table 1 lists the high-level goals relevant to robot-assisted USAR. Table 2 shows an example of the detailed GDTA (decisions and SA items) for a portion of these goals.

OPERATIONAL AND SA ISSUES Environmental Factors and Perceptual Demands Search and rescue teams work in a psychologically and physiologically stressful environment (Casper & Murphy, 2003). The environment in which USAR professionals work can be described as one of clutter, debris, and


363

darkness, making the detection of victims in a collapsed structure a demanding visual task. The environmental conditions and the stress of finding victims fast can be taxing on SA. Low light and insufficient system-based illumination makes search difficult. Micire (2002) reports on the difficulties with inadequate headlights (and camera gain control) on robots used during the World Trade Center response. The surrounding clutter and debris make scenes hard to interpret. This is severely compounded by soda-straw views and grainy images that are often provided with the camera. Operators also contend with the misleading viewpoint from the robot’s perspective. The robots are small and low to the ground, which means the size of objects (and distance to objects) can be greatly misjudged, leading to incorrect or total lack of object identification. Zooming in on objects with on-board cameras (when possible) is often not helpful because of the lack-of-depth perception and inability to recognize partially occluded objects viewed from unnatural angles. Operators typically need to work to get the robotic systems very close to objects to make determinations. In all cases, they make joint decisions (with other operators viewing imagery from the same robot) regarding whether or not an object is a potential victim (Murphy, 2004). Burke, Murphy, Coovert, and Riddle (2004) and unpublished follow-up studies have found that USAR robot operators have great difficulty building and maintaining SA. Operators were found to spend significantly more time gathering data about the environment and robot than on navigation. Indeed, the robots were mobile only 51% of the mission duration, with 49% of the time spent attempting to disambiguate what operators were looking at. Limitations of the robots and the nature of the deconstructed environment make perception of cues very difficult, resulting in little opportunity to develop high levels of SA. This can lead to operator frustration and an increase in perceived workload, which serves to further stifle development of SA. Under stress, operators in search and rescue may become disorganized and fail to employ effective scan strategies and they may pay less attention to peripheral information (Endsley et al., 2003). Stressors and frustration can undermine SA and lead to premature closure during performance; that is, making decisions without taking into account all available information. In USAR operations, this could result in premature decisions to abandon a search effort due to a lack of salient cues indicating a victim. Difficulties in Robot Localization Localization of the robot during navigation is an important goal. The ability to maintain awareness of where the robot is ultimately affects performance

364


in directing rescuers to the position of victims. Localizing the vehicle involves comprehension of where the robot is with respect to other locations (e.g., the insertion point, recognizable landmarks, the robot operator’s position). It also includes perceiving and understanding direction of motion of the robot during navigation. That is, not just knowing the robot is moving forward, backward, etc., but, also knowing that it is moving toward a particular landmark in the environment (e.g., toward an adjacent wall, toward the operator’s position). Frequent disorientation in the USAR task environment hinders SA development regarding the location of the robot system (localization). Robot operators appear to expend considerable cognitive effort tracking control actions in order to figure out where the robot is and where it has been, due to disorientation. In a study to assess the utility of interface designs for supporting SA and performance in robot-assisted USAR activities, Yanco and Drury (2004) observed that in trials where subjects appeared to spend a substantial amount of time in trying to acquire SA (30% of the trial time), operators still expressed confusion about where the robot was located and whether the robot was approaching collision with obstacles. Disorientation may occur immediately upon insertion of the robot into the operational environment (e.g., collapsed structure), or it may occur over time as a result of non-line of sight operations. Robotic vehicles may be inserted into a void from an entry point that is above the space to be searched (from the second floor of a structure, down to the first floor) (see Fig. 1). The void may be dark, dusty, or just filled with clutter, making it difficult to ascertain the position and direction of the vehicle once it is on the terrain. In this type of situation, operators are disoriented before navigation

Fig. 1.

Tethered Robot Poised for Vertical Entry from Upper-Level Floor.


365

begins and they often take several minutes prior to moving a significant distance attempting to determine the direction of motion and orientation of the robot. They work to understand which way the robot is pointed and which way the robot needs to move in order to begin the search. Operators may also become disoriented as a result of their inability to remember the control actions completed through the interface and maintain awareness of the effect of those control actions on the robot position in the space. This can occur often during a USAR operation once the robot moves out of the operator’s line-of-sight. The operator must then attempt to develop and maintain a mental model of where the robot is. This may involve attempting to recall past locations, remembering the direction of motion and how long the robot moved in a particular direction, looking for recognizable landmarks in the camera images, or backtracking to a known location. For tethered system, controllers often rely on the tether cord to guide them back to at start point or some other location. Poor SA on the location of the robot affects the driver’s ability to project where the robot is and can result in errors in knowledge regarding the position/location of victims. In a field observation in which robot controllers were training with USAR professionals in a simulated robot-assisted rescue mission, researchers observed frequent controller disorientation (Riley & Endsley, 2004). The USAR professional leading the search effort was observed frequently questioning the robot controller regarding the whereabouts of the robot. When a victim (mannequin) was detected, the robot controller provided information on where the victim was located and instructions on how to reach the victim for extraction. Upon entry into the void, the rescuer and the controller discovered that the victim was not located on the wall suggested by the controller, but was on the opposite side of the room. In an actual rescue mission, inability to accurately describe where a victim is located could result in critical time lost in need to continue searching for the victim once the rescue team has entered the structure. Understanding How the Remote Vehicle is Situated in the Environment A related SA problem in HRI deals with knowledge of how the robot is situated in the environment. This is sometimes referred to as robot situatedness. The term relates to the orientation of body of the robotic vehicle on the terrain, for example, the pitch and/or roll of the vehicle during navigation. Some of the robots used for USAR are configurable. The bodies or components of the bodies of these vehicles can change position or shape to facilitate better mobility or object-viewing. For example, the Inuktun Micro

366


Variable Geometry Tracked Vehicle (VGTV) is a small robotic vehicle that can be configured to vary the height (viewpoint) of the camera off the ground. Understanding how the robot is situated involves understanding how position/location, body orientation, and robot configuration interact to affect the robot’s mobility, the control actions that are required for traversing a particular type of terrain, the operators’ perception of what they might be seeing, and the ability to project the next step or goal in task completion (e.g., continue search, extract robot from tenuous position on terrain, etc.). It is difficult to acquire SA on situated-ness during non-line-of-site operations. Rescue teams cannot see the robot for visual confirmation, and current interfaces fail to provide adequate and/or salient information regarding how the robot is oriented or configured. Riley and Endsley (2004) discuss an instance where a remote vehicle, deep in the rubble pile during a USAR scenario, is responding to input, but the operator is unsure of its responses. The team expresses concern that there may be a problem with the controls. Ironically, the robot that has been deployed can navigate terrain both right- and wrong-side up. Because the team is out of visual range, the operator is not aware that the robot has flipped over (i.e., is operating wrong-side up). One might wonder why this cannot be ascertained from looking at the camera image, as the image should appear upside down. It does not, however, because in a collapsed building void spaces often do not afford views of a ceiling or sky, and the clutter above to robot inside the wreckage looks just like the clutter below. There is no apparent difference in the camera feed, and the information provided at the interface on robot orientation does not support ‘‘at a glance’’ awareness of the upside-down state. Critical time is spent in trying to determine the state of the robotic system because SA on situated-ness is inadequate. Team Communications and Processes The search process is a team effort that involves four basic activities: searching for signs of victims, reporting critical information to team leaders, making structural assessments, and estimating the volume of space that has been searched (Burke et al., 2004). A search and rescue team would consist of at least two people following the ‘‘buddy’’ system (Murphy, 2004). The team most likely would consist of a robot controller, a USAR professional (expert in searching and extracting victims), and possibly a second USAR professional (dedicated to mapping the environment and/or assessing structural stability). The search effort may require multi-tasking across the team


367

or it may require that one individual take on multiple roles during performance. Each member of the team works to acquire SA to meet task goals and to communicate with each other regarding, for example, where to look, where the robot can/cannot go, robot status, etc. See Burke et al. (2004) for more details on these roles. Originally during operations at the World Trade Center and training exercises immediately before and after, team members often shared a single display. Since 2002, it has been a standard practice for the team to have two displays (presenting the same imagery), one on the main ‘‘operator control unit’’ and one on a secondary viewing device such as a handheld monitor. Although team members are looking at the same image (on separate displays) they may be looking for different SA items or using the same information to answer different critical questions. The robot operators/drivers use the interface to navigate the vehicle. They watch the camera feed to assess the terrain and potential obstacles, to determine where the robot can be driven or identify areas of limited mobility or danger to the robot. Controllers mentally track control input (e.g., turned left, rotated flippers up), over time, in order to continuously localize the robot and comprehend its situated-ness in the space. They also monitor the system status and update other team members on these SA items. The search specialist, dubbed the ‘‘problem holder’’ or ‘‘mission specialist,’’ uses the camera view and follows SAR tactics to instruct the robot operator regarding which direction to point the camera or in which direction to drive. The search professional may assume the roles of the structural and hazmat specialist if none are available. The specialist assesses color, shape/form of objects, size and distance of objects, movement in the area, and environmental cues suggesting office areas, bathrooms, etc. where people may have been located during the collapse. They may take note of environmental and structural hazards that might hinder extraction. The map developer looks for structural and material hazards. They work to identify landmarks and unique features that can be pointed out to the extraction team and to track operator control actions to determine the path to the victim. Riley and Endsley (2004) observed that search teams may experience difficulty sharing a single display. This may hinder SA development, which is a problem because the individual SA of team members must come together to effectively convey information to other stakeholders. For example, the extraction team will want information on how to get to the victim (what path to take), the victim’s location, and the characteristics of the void. A medical team may need information on the status of victims and the type/ urgency of treatment needed upon extraction.

368


Current Interface Designs Beyond the difficult-to-interpret camera imagery that results from environmental conditions and limited field of view, there are low-level data on the robot presented through some robotic interfaces. Some interfaces dedicate almost 50% of the display space to robot status indicators. There is often, but not always, information on tool angle and pitch, vehicle heading and speed, status of batteries, multiple motors, and radio communications. Voltage levels may also be presented. The data are typically provided at a low level, that is, not integrated or presented such that overall robot status can be ascertained relatively quickly. So while navigating, the operator must scan the interface and cognitively integrate or fuse data to comprehend higher-level SA elements such as the distance traveled based on speed and time on task, the time until battery level is too low for operation, and if the overall motor system is within an acceptable operating range. The SA of the controller and the team can be impacted by the lack of integrated or high-level data at the interface. As attention and working memory are limited, displays that provide direct presentations of comprehensions and projections can benefit the operator’s SA. In some cases, lowlevel data are not really needed or desired. When discussing interface design and information needs with subject matter experts, robot controllers strongly indicate a desire for reduced information regarding status of individual components of a robotic system. They suggest a high-level depiction of the health of systems, rather than detailed diagnostics on particular sub-systems, which may reduce the amount of time and attention needed to assess overall system health, free cognitive resources for other task components, and reduce the potential for ignoring status information altogether. (It is important to note that user opinions may not always map to effective interface designs, so empirical testing is required to determine which data on robot status should be included on the interface.)

Multi-tasking in Remote Control of Robots In teleoperation of robotic systems in USAR, multi-tasking may be required. A robot controller may also be the search expert or responsible for developing a layout of the environment. Research has pointed out that a human operator in teleoperation may often focus on information relevant to a particular component of a robotic task (e.g., navigation) in the environment to the exclusion and detriment of other task component (e.g., victim


369

detection and identification) (Draper, Kaber, & Usher, 1998). Allocation of resources to one element or stimulus in the environment may mean loss of SA on certain other relevant elements. In multi-task situations, because an operator’s cognition is distributed across multiple, competing goals at the same time, the operator may develop SA on one goal or task component, singularly, or two or more goals or task components together. An operator’s ability to develop good SA on multiple components will be critically affected by the capability to divide attention across multiple environmental elements and/or activities. Acquisition and maintenance of SA under multi-tasking can also be affected by the design of the interface and its utility in supporting divided attention, in adequately directing attention to goal-specific items, and/or in facilitating appropriate shifts between goal-driven and data-driven information processing.

Application of Autonomy to Robotic Systems Robot-assisted search and rescue, at present, involves strict teleoperation of the robotic system. This degree of human involvement is likely to continue given the operating conditions, however, a future goal is implementation of autonomous capabilities. It is important to consider the SA and control issues that might occur as a result of varying levels and approaches to autonomy. It is likely that many of the same automation issues observed in the application of automation to industrial settings will be observed in HRI (i.e., out-of-theloop syndrome, mode awareness problems, vigilance decrements). Consider autonomous navigation (of ground vehicles) that might be applied to free-up cognitive and attentional resources for other task components, like dedicated target (victim) detection and identification – although given that the robots are moving only 51% of the mission suggests that autonomous navigation may not significantly lessen operator involvement. An operator who is only required to monitor navigation rather than drive the system, might lose SA on where the robot system has been, how the robot got to its current location, or what control actions resulted in the current robot situated-ness. Loss of SA could mean difficulties, and increased time required, in taking over during required interventions (e.g., extracting the robot from perilous situations). (It should be noted that there are differences in the navigation issues inherent to UGVs versus UAVs. For instance, ground vehicles can rely heavily on GPS navigation, and the positioning of a UAV does not involve as many degrees of freedom. These points suggest that utility in application of autonomy, for example in vehicle navigation, may differ across platform types.)

370


SUMMARY Unfortunately, the operational and SA issues discussed here are common to remote control of ground-based robotic vehicles (not just those involving search and rescue missions). SA is critical to effective utilization of robots in all domains. The fact that operators have difficulty in acquiring SA on robots and using robots to build SA on the task environment speaks to the need for empirical research on how to enhance performance by facilitating SA development. Interface designs, such as heads-up type displays and augmented reality, should be investigated for utility in supporting SA of ground vehicle orientation and configuration. Methods for improving team performance and development of Shared and Team SA should be studied because of current and future teamwork requirements in USAR and other domains. Effective ways to integrate object recognition tools (both automated and decision support) and tools that support users in depth and distance judgments should be explored. Additionally for future operational situations, automated behaviors need to be further investigated. Questions regarding what should be automated for effective multi-tasking are important. In addition, there is still a need for research and analysis regarding the SA requirements for robot tasking and controlling. These goals depend on a clear understanding of operator SA needs, in order to develop effective strategies and designs for providing the high levels of SA needed for successful robot operations in complex environments.

REFERENCES Burke, J. L., Murphy, R. R., Coovert, M. D., & Riddle, D. L. (2004). Moonlight in Miami: A field study of human–robot interaction in the context of an urban search and rescue disaster response training exercise. Human–Computer Interaction, 19(1), 85–116 (Special issue on Human–Robot Interaction). Casper, J., & Murphy, R. R. (2003). Human–robot interactions during the robot-assisted urban search and rescue response at the World Trade Center. IEEE Transactions on Systems, Man and Cybernetics Part B, 33(3), 367–385. Draper, J. V., Kaber, D. B., & Usher, J. M. (1998). Telepresence. Human Factors, 4(3), 354–375. Drury, J. L., Scholtz, J., & Yanco, H. A. (2003). Awareness in human–robot interactions. In: Proceedings of the IEEE conference on systems, man and cybernetics, Washington, DC, October. Endsley, M. R. (1993). A survey of situation awareness requirements in air-to-air combat fighters. International Journal of Aviation Psychology, 3(2), 157–168. Endsley, M. R. (1995). Toward a theory of situation awareness in dynamic systems. Human Factors, 37(1), 32–64.


371

Endsley, M. R. (2000). Theoretical underpinnings of situation awareness: A critical review. In: M. R. Endsley & D. J. Garland (Eds), Situation awareness analysis and measurement (pp. 3–32). Mahwah, NJ: LEA. Endsley, M. R., Bolte, B., & Jones, D. G. (2003). Designing for situation awareness: An approach to user-centered design. New York: Taylor & Francis. Micire, M. (2002). Analysis of the robotic-assisted search and rescue response to the World Trade Center disaster. Unpublished master’s thesis, University of South Florida, Tampa, FL, USA. Murphy, R. (2004). Human–robot interaction in rescue robotics. IEEE Systems, Man and Cybernetics Part C: Applications and Reviews, 34(2), 1–16. Riley, J. M., & Endsley, M. R. (2004). The hunt for situation awareness: Human-robot interaction in search and rescue. Proceedings of the Human Factors and Ergonomics Society 48th Annual Meeting (pp. 693–697). New Orleans, LA. Salas, E., Prince, C., Barker, D. P., & Shrestha, L. (1995). Situation awareness in team performance: Implications for measurement and training. Human Factors, 37(1), 123–136. Smith, K., & Hancock, P. A. (1995). Situation awareness is adaptive, external directed consciousness. Human Factors, 37(1), 137–148. Yanco, H. A., & Drury, J. (2004). ‘‘Where am I?’’ Acquiring situation awareness using a remote robot platform. Proceedings of the IEEE conference on systems, man and cybernetics, Washington, DC, October.


372

26. WHAT THE ROBOT’S CAMERA TELLS THE OPERATOR’S BRAIN Roger A. Chadwick, Skye L. Pazuchanics and Douglas J. Gillan The failure of all 15 robotic entries at the 2004 DARPA Grand Challenge race across the Mojave Desert suggests that, for now, the possibility of fully autonomous robots is remote. Among the most natural and easy-to-do activities for mobile, adult humans is moving through space. However, that ease and naturalness belies the complexity of the perceptual, cognitive, and motor processes that underlie visually guided motion. Although the ease of human navigation may deceive robotics developers into believing that automating spatial navigation should be similarly straightforward, reliable solutions for autonomous navigation and localization in dynamic real world environments have not been found (Murphy et al., 1993; Santos, Castro, & Ribeiro, 2000; Tomatis, Nourbakhsh, & Siegwart, 2003). Furthermore, autonomous navigation is the only one component of effective, reasoned autonomous operation. Given that teleoperations are likely to continue to be an important mode of robotic control for the foreseeable future, what perceptual information does the operator of a ground-based remotely operated vehicle (ROV) or of multiple ROVs need in order to navigate the system through space, and how should the information be presented to the operator? This chapter will discuss observational studies and experiments performed in the Robotics Control Laboratory in the Psychology Department at New Mexico State University investigating the perceptual and cognitive processes involved when an operator controls two ROVs simultaneously. The research Human Factors of Remotely Operated Vehicles Advances in Human Performance and Cognitive Engineering Research, Volume 7, 373–384 Copyright r 2006 by Elsevier Ltd. All rights of reproduction in any form reserved ISSN: 1479-3601/doi:10.1016/S1479-3601(05)07026-8

373

374

ROGER A. CHADWICK ET AL.

includes studies of operators controlling virtual ROVs (e.g., using CeeBot, a software-based programmable robot simulation created by Epistec) or controlling minirobots in a miniature environment that, when seen through a video-based user interface, appears to be full sized. The research that we summarize in this chapter includes (1) experiments on navigation using an egocentric or first person viewpoint, including the role of landmarks and the use of maps, (2) studies of the size of the ROV’s field of view and camera viewpoint, and (3) research on how operators integrate interface components that display spatial position information with camera-based views of the robots’ positions to develop an awareness of the location of multiple ROVs. The chapter will also discuss the implications of integrating visual information provided to the operator via multiple ROVs, including both ground and air vehicles.

NAVIGATION AND USE OF MAPS Imagine trying to navigate through your environment while looking straight ahead through a narrow tube. These are essentially the conditions the operator of an ROV in teleoperation mode may have to contend with. We hypothesized that controlling ground-based ROVs would be easier if an operator developed an explicit overview of the space in which the ROV was maneuvering. Accordingly, we conducted a study in which we had naive undergraduate participants explore a maze-like virtual desert environment while drawing a map of the area. After completing a 30-min mapping task, participants re-entered the maze to search for and retrieve a target object. The virtual robots and landscape, which had minimal landmarks and a maze of navigable paths (see Fig. 1), were created using CeeBot (see Chadwick,

Fig. 1. Virtual ROV and Maze Landscape Used in Mapping Study. Landmarks, Such as the Tree Shown in the Left Panel, were Highly Valued by Participants in Orienting Themselves within the Mostly Featureless Terrain.

375

Robot’s Camera and Operator’s Brain

Gillan, Simon, & Pazuchanics, 2004, for a discussion of the CeeBot tool). Maps drawn by participants were rated independently by three raters (graduate psychology students) for the usefulness of the map for navigating the area on a scale from 1 (not at all useful) to 7 (extremely useful). Cronbach’s alpha was computed as a consistency estimate of inter-rater reliability at 0.96. Participants showed large individual differences in both map quality and ability to navigate the area successfully to retrieve a target; as can be seen in Fig. 2, mapping and navigation ability were highly related. Regression analysis revealed that both gender and map-drawing ability were statistically significant predictors of performance in the retrieval task (Response Time ¼ 915 239[Gender] 53[Map Quality], R2 ¼ 0:95; po0:05). Performance was better for men than women, and better for those participants who produced better maps. While neither of these results is particularly surprising, the relation between mapping skill and time to navigate an area may indicate that the participants developed a mental model of the area as they viewed it through the ROV’s camera, then used that mental model to move 1000 Gender Male Female

900

Total Population

Retrieval Time (sec)

800 700 600 500 400 300 200 0

1

2

3

4

5

6

7

Map Quality Rating (1-7)

Fig. 2. The Relation between Map Quality and Target Retrieval Time. Participants that Produced Better Maps were Subsequently Faster in Navigating the Maze to Retrieve the Target.

376

Fig. 3.


Mapping Observations Showing (Far Left) the Actual Terrain Map, and Various Participant-Drawn Maps.

the ROV back to retrieve the target. Alternatively, perhaps mapping ability and navigation ability are both offshoots of a general spatial ability that differs from person to person. Examples of participant-drawn maps are shown in Fig. 3, illustrating the different skill levels of various participants. These maps reveal problems maintaining scale and directional orientation. We observed that participants relied heavily on landmarks (trees, flags, objects, etc.) or distinctive terrain features, often spontaneously verbalizing short noun assignments to each landmark as they passed by, such as ‘‘radar’’ or ‘‘big tree’’, and annotating their maps with an appropriate symbol. Our observations in this and other preliminary studies conducted in our laboratory indicate that many participants operating virtual ROVs in featureless environments tend to lose spatial awareness, even when supplied with an accurate map. The participants in our studies failed to attend consistently to the maps, which resulted in serious navigation errors. They used landmarks for localization when such distinctive features were available (e.g., Lynch, 1960; Sorrows & Hirtle, 1999), but in featureless terrain, participants became extremely disoriented and confused. One solution to an operator’s disorientation might be to provide real time you-are-here maps. However, such displays require timely and accurate ROV localization information, either from an active aerial view, or GPS position telemetry system. These capabilities or resources may not always be available – for example, if an ROV is inside of a building or a cave.

CAMERA VIEWPOINT AND FIELD OF VIEW As mentioned previously, many ROVs provide their operators with only a small field of view (FOV). This view, which has been described as equivalent to looking through a soda straw, greatly limits ROV operators’ awareness of


377

the space around their vehicle. These restricted views make navigation difficult because they (1) cut out valuable visual information about the space surrounding an ROV, (2) induce misperceptions of size, distance, and direction, and (3) foster poor cognitive map formation and greater memory and attention loads (Neale, 1997). We hypothesized that providing operators with an appropriate camera viewpoint of their ROVs might reduce the negative effects of restricted FOV. Currently, many ROV displays use a first person camera viewpoint, an immersed perspective displaying only the environment in front of an ROV, to provide visual information to operators. However, we believe that a third-person camera viewpoint, a perspective that provides an operator with a display of the robot in the environment by tethering a camera behind the unit, could provide a substitute for the contextual information lost in a restricted FOV. To test this hypothesis, we manipulated the ROV camera’s FOV size and viewpoint in a navigation task. FOV size was manipulated by masking the screen that displayed an ROV’s camera view to reduce the FOV from 601 to 481. Camera viewpoint was manipulated by providing a first-person, immersive camera view from the front of the ROV, or a third person, tethered view from behind and above the ROV, which included the ROV in the display. The operators’ task involved controlling a virtual robot (generated with Microsoft and FASA Studio’s MechWarrior 4: Vengeance game level editor) through a course containing obstacles, e.g., a zigzag pathway with indestructible barriers, a pathway with destructible boxes, and a field of destructible cars. Movement controls were all keyboard based. Participants had two primary goals for the task: (1) to get through obstacles as quickly as possible and (2) to avoid collisions with destructible objects in the environment. Based on these goals, we measured two performance factors: (1) the time to complete the obstacle course and (2) the number of objects destroyed due to accidental collision. As we predicted, operators navigated the obstacle courses significantly faster when given a third person (tethered) viewpoint than when given a first person (immersed) viewpoint, F ð1; 23Þ ¼ 23:67; po0:01: In contrast, between the first and tethered third person viewpoint conditions, participants did not have a significantly different number of collisions resulting in object destruction, F ð1; 23Þ ¼ 0:96; p ¼ 0:33: The small reduction in FOV, from 601 to 481 , significantly increased task completion time only in the destructible box portion of the obstacle course, F ð1; 23Þ ¼ 4:68; p ¼ 0:41; and consistently led to more collisions in all portions of the obstacle course, F ð1; 23Þ ¼ 1:41; p ¼ 0:16: Analyses of the interaction between viewpoint and FOV for time performance in the zigzag pathway obstacle course gave some support

378

ROGER A. CHADWICK ET AL. Time to Complete Zigzag Portion of Obstacle Course

Time in Seconds

100

80 FOV Full Restricted

60 1st Person 3rd Person View Perspective

Fig. 4. The Interaction between FOV (Full or Restricted) and Camera Viewpoint Indicates that Using a Third Person (Tethered) Viewpoint Improves Performance and Compensates for the Effects of a Restricted Field of View.

to the hypothesis that the additional information provided by a third-person viewpoint would facilitate performance most under conditions with a limited FOV, F ð1; 23Þ ¼ 3:35; p ¼ 0:08: Time performance under a third-person viewpoint was somewhat less affected by the reduced FOV (see Fig. 4). Although research on navigation has previously suggested that a more egocentric, first-person viewpoint results in better navigation performance than a more external viewpoint (Wickens & Hollands, 2000), our research indicates that for ROV control, seeing the ROV in its environment can be better, especially if the displayed field of view is small. The improvements in performance produced by the viewpoint might be greater in conditions of highly restricted FOV that would more closely resemble currently operational ROVs. How might a camera be used to provide a third-person viewpoint? One possibility would be to have a camera mounted on an extendable manipulator arm that could be extended to show an overview that included both the ROV and its surrounding environment.

MULTIPLE ROVS IN AN URBAN SEARCH AND RESCUE SIMULATION One way to provide an ROV operator with additional camera views is to have more than one ROV available to provide external views of the other


379

ROVs in the task environment. Alternatively, an overhead view could also provide additional visual information to the operator. However, increasing the number of ROVs that operators have to control will necessarily involve management of multiple display and control components. We performed an experiment on single-operator teleoperations control of multiple ROVs in a simulated urban search and rescue (USAR) scenario using miniature radio controlled robotic vehicles in a 1:12 scaled environment (Chadwick, 2005). For the experiment, we manipulated the number of ROVs (one or two ground vehicles) and the availability of a simulated uninhabited aerial vehicle (UAV) view (present vs. absent). The eye in the sky UAV view was captured with the use of three black and white video cameras mounted in the ceiling with a video selector unit that allowed operator selection of one of the three cameras. Each ROV camera provided a view of approximately one third of the test area, with little overlap. After participants were trained in ROV operation and given an evaluative pretest (a timed trial run used to evaluate participant skill), they explored a simulated disaster area, looking for targets to photograph. Participants marked the locations of the targets on a paper map of the area, providing a measure of spatial comprehension. We hypothesized that using two robotic ground vehicles would be beneficial in avoidance of robot faults (e.g., getting stuck, rolling over, etc.). Participants were constrained in their strategy in using two ROVs by instructions and reminders to use the two robots as a team, viewing each other as they proceeded through the search area. This particular multiple ROV strategy was emphasized because we wanted to study the predicted beneficial effect of having the alternate robot’s view available when crossing difficult terrain and avoiding faults. In summary, results of the USAR simulation experiment showed that using two ROVs was less efficient in locating targets and did not reduce robot faults, but the use of the aerial view was beneficial for spatial comprehension without causing any loss of efficiency. Specifically, operators found significantly more targets in the one-ROV conditions on average than in the two-ROV conditions (means of 9.6 and 7.3, respectively), F ð1; 56Þ ¼ 16; po0:01: Operators committed more faults, in general, when they had to control two ROVs than when they had to control only one (means of 3.0 and 2.2, respectively), although this result was not statistically significant. When considering the number of faults per target found, operators had fewer faults in the single ROV conditions than in the two ROV conditions (means of 0.28 and 0.43, respectively), F ð1; 56Þ ¼ 3:7; p ¼ 0:06: Thus, one hypothesized benefit from using two ROVs – that one ROV could provide a

380


helpful auxiliary view of the other in navigating difficult terrain obstacles – was not supported. The cost of operating the second ROV appears to have exceeded any benefit obtained from having the second camera view. However, despite the performance deficits when participants had to control two ROVs, participants expressed a preference for having the second ROV during post-session interviews. The availability of the aerial view in the USAR simulation facilitated target localization, especially in the single robot condition. Operators produced a smaller error between the location of a target that they marked on the map and the actual target location (i.e., the target offset) when the aerial view was available, than when they had no aerial view (means of 8.1 and 13.5 mm, respectively). This difference was statistically significant (po0:05) when operator skill (as measured by pretest scores) was controlled for in a regression analysis. Presence of the aerial view had no effect on the number of targets located or the number of faults. This result is hardly surprising, since the real-time aerial view provides an accurate you-are-here map of the area and a ready solution to the localization problem. When operators used the aerial view to identify their terrestrial ROV location, they typically performed some simple identification maneuver with the ROV (e.g., moving it back and forth) to confirm location and orientation. Accordingly, identification schemes (such as large numbers and orientation indicators on the top of vehicles) would help operators localize specific ROVs from the air. The benefits of the additional visual information from the aerial view appear to have exceeded the costs of interacting with the UAV, given that the operator had only to select one of the three views, not actually control the UAV. That benefit-to-cost ratio contrasts with the addition of a ground ROV in this experiment in that there was a high cost associated with the burden of controlling the second ROV. In addition, we observed that, with two ROVs, participants focused relatively more attention on viewing the other ROV, than on looking for targets. When an operator controls multiple ROVs, greater costs than benefits may not always be the case. Rather, differences in task specifics or operator strategies could produce different outcomes. For example, in the USAR scenario, if the ROVs searched separate sections in the disaster area rather than working as a team on the same area, especially if the layout of the search area contained dead ends that required substantial backtracking, the benefits of using two ROVs could outweigh the costs. There may also be cases when the difficulty of obstacles is such that having the secondary view is essential in avoiding catastrophic faults.


381

DISPLAY POSITIONING AND MULTIPLE CONTROLS When a single operator has to control and monitor multiple ROVs, a number of display and control options are possible. Our studies in this area concerned the number and position of active video displays, and the use of multiple controls. Multiple controls allow simultaneous commanding of multiple vehicles. Display options (for two ROVs) include a single selectable video display or multiple simultaneous displays. If multiple displays are used they could be configured horizontally (side by side) or vertically (one above the other). We examined these options in a series of experiments using CeeBot virtual ROVs. In the first series of experiments, operators were given control of two ROVs (designated R1 and R2), each with an independent task. R1’s task was to locate targets and deliver them to a specific location. R2’s task was simply to follow a randomly moving target vehicle. We replicated this experiment with various combinations of displays, display positions, and control configurations. Participants who were provided with only a single display for both ROVs performed extremely poor on the R2 monitoring task. In this single display condition, operators needed to switch regularly between ROV displays, and failed to do so in a timely manner. With dual, simultaneous displays, movement of the target vehicle was easily detected and task switching was accomplished reasonably well. There was little difference in task performance between using a single switchable control and using two sets of controls. In a follow-on experiment we compared performance between a single, switchable monitor, and multiple monitors, configured either horizontally or vertically. In this study, the operator had to control two ROVs, navigating them on two similar roadways, locating target boxes and delivering them to platforms. In addition to finding and delivering targets with both ROVs, the operator had to avoid collisions with other moving vehicles. Dual controls were provided such that both ROVs could be controlled simultaneously. The results indicated that operators reported lower overall subjective workload and less frustration (as measured with the NASA-TLX) in the single display configuration, but were unable to deal effectively with dynamic external events. Operators in the single display condition could not react to the presence of other vehicles in the environment of the unattended ROV. Consequently, the single display condition resulted in a greater number of collisions.

382


Comparing the conditions with two displays, operators were faster to complete the task in the vertical display configuration relative to the horizontal (mean times of 600 and 700 sec, respectively), tð16Þ ¼ 2:29; po0:05: However, these differences may have been influenced by task specifics involved in both ROVs traveling up vertically symmetrical landscapes. The reduction in completion time may also have been due to problems maintaining simultaneous control of both ROVs that were exacerbated in the horizontal display configuration. Most operators attempted simultaneous control of both vehicles, and observations indicate that they could be somewhat successful with ROVs simply traveling in a straight line, however any degree of complexity of control could not be accomplished well while attempting simultaneous control. Nontrivial control activity on an attended ROV would often result in control drift of the unattended ROV, resulting in unintended direction changes, which could result (especially in the horizontal display condition), in disorientation upon re-attending to this vehicle. In real-world applications, control drift and unintended movement of an unattended unit could be disastrous. For this reason, multiple controls allowing a single operator to control more than one unit simultaneously are not recommended.

CONCLUSIONS Any attempt to fully answer the question, what does the robot’s camera tell the operator’s brain, will require a complex assessment and understanding of the capabilities of the robots, the tasks for which the robots will be used, the physical environments of the robots, the number of robots, and the autonomy of the robots. Likewise, as we try to design interfaces for mobile systems, from vehicles under teleoperation control to highly autonomous robots, the above factors must be taken into account. A straight-forward approach to the role of visual information in ROV control would be that the more visual information given to operators from robots’ cameras and sensors, the better their spatial awareness and performance. However, a more sophisticated approach, and one that the research summarized here points towards, is that operators trade off the benefits having information with the cost of getting information. Thus, we often observe that additional information leads to poorer performance because of the increased cognitive effort of controlling the second entity that provides the information. We have also observed that the goal of controlling a second robot whose task is to provide additional visual information can


383

deflect the operator from the main goal of gathering information (e.g., identifying targets). In addition, extra information (e.g., from a second monitor) can lead to a diffusion of attention that disrupts performance. If our research has taught us anything, it is that there are limitations in attentional resources and situational comprehension that must be seriously considered when assigning multiple ROVs to a single human operator. We cannot simply provide the operator with as much information as is available without consequences. Additional capabilities, in the form of additional visual information in one form or another, can have the unintended result of diverting the operator’s attention away from the most pertinent information. One solution to the problem of multiple vehicle control may lie in the implementation of manageable autonomy. Just as automatic transmission, cruise control, and power steering free the automobile operator from some of the demands of the primary driving task and allow allocation of resources for dialing and conversing over a cell phone, clever autonomy in robots may allow a shift in cognitive focus away from the moment-by-moment demands of control into the realm of problem solving and strategy. However, just as conversing on cell phones can lead to disastrous results when primary driving task demands suddenly shift, a fallback to direct control of robotic units will always need to be considered. The inattention to moment-by-moment detail made possible by autonomous functionality may result in insufficient situational awareness when fallback to manual operations is required. One feasible solution to this dilemma is the implementation of smart attentional cueing and situation awareness enhancing interface features. In our studies we observed repeatedly that operators responsible for multiple tasks with multiple vehicles often became engrossed in interesting details of lower priority tasks, and tended to task switch at convenient, rather than optimal, moments. Critical external events impacting unattended units often failed to receive timely shifts in attentional focus. Task switching often resulted in a need for reorientation to the situational demands of the newly attended unit. To the extent that control system processes can detect and warn the operator of attentional demand, the system can assist in allocation of operator resources to the real priorities. Multiple robots might be interpreted to mean multiple situations, multiple landscapes, multiple goals, and multiple problems. Success in extending the individual operator’s effectiveness by allowing one entity to control many – force multiplication if you will – depends on the ability to map extended functionality onto existing cognitive limitations, rather than requiring the extension of cognitive abilities.

384


REFERENCES Chadwick, R. A. (2005). The impact of multiple robots and display views: An urban search and rescue simulation. In: Proceedings of the human factors and ergonomics society 49th annual meeting, Human Factors and Ergonomics Society, Orlando, FL (pp. 387–391). Chadwick, R. A., Gillan, D. J., Simon, D., & Pazuchanics, S. (2004). Cognitive analysis methods for control of multiple robots: Robotics on $5 a day. In: Proceedings of the human factors and ergonomics society 48th annual meeting, Human Factors and Ergonomics Society, New Orleans, LA (pp. 688–692). Lynch, K. (1960). Image of the city. Cambridge, MA: MIT Press. Murphy, K. N., Juberts, M., Legowik, S. A., Nashman, M., Schneiderman, H., Scott, H. A., & Szabo, S. (1993). Ground vehicle control at NIST: From teleoperation to autonomy. Seventh annual space operations, applications, and research symposium, August 3–5, Houston, TX. Neale, D. C. (1997). Factors influencing special awareness and orientation in desktop virtual environments. In: Proceedings of the human factors and ergonomics society 41st annual meeting, Human Factors and Ergonomics Society, Santa Monica, CA (pp. 1278–1282). Santos, V. M., Castro, J. P., & Ribeiro, M. I. (2000). A nested-loop architecture for mobile robot navigation. The International Journal of Robotics Research, 19, 1218–1235. Sorrows, M. E., Hirtle, S. C. (1999). The nature of landmarks for real and electronic spaces. In: C. Freksa., D. M. Mark (Eds), Spatial information theory. Lecture notes in computer science. (Vol. 1661, pp. 37–50). Berlin: Springer. Tomatis, N., Nourbakhsh, I., & Siegward, R. (2003). Hybrid simultaneous localization and map building: A natural integration of topological and metric. Robotics and Autonomous systems, 44, 3–14. Wickens, C. D., & Hollands, J. G. (2000). Navigation and interaction in real and virtual environments. In: Engineering psychology and human performance (3rd ed., pp. 158–195). Upper Saddle River, NJ: Prentice-Hall.

SUBJECT INDEX accident investigation 121, 123, 130 accidents 105–106, 111, 115 adaptable automation 252 adaptive automation 41–44, 54–55 aerial vehicle 165 Air Operations Cell (AOC) 311–313, 315–318, 320–326 air traffic control (ATC) 73, 77, 79–82, 85–86 Altair 110 artificial intelligence 202, 205 attention 213–214 automation 7, 17, 91–92, 100, 102, 106, 111, 113–115, 209–210, 212–213, 216–220, 252–254, 258, 262–264 automation agent 270, 281 automation reliability 209 autonomous operation 193–194, 205 autonomous systems 53 autonomy 225–226, 384–385 camera angle 172, 174 camera-viewpoint 376, 378–380 Cognitive Engineering Research Institute (CERI) 3–4, 9, 12 Cognitive Engineering Research on Team Tasks Laboratory (CERTT) 285–295 cognitive issues 4–5, 8 cognitive load 329–332, 335–336, 338, 340 cognitive task analysis (CTA) 312, 315, 318, 320–321, 326, 363 collaboration 299, 301, 303–307, 329–330

command and control 49, 181, 184, 189, 285, 289 communication 287–288, 291, 293, 295–296 concept mapping 318 control handoffs 110–111 control method (CM) 136–138, 140, 144–145 controls 108–109 coordination 286–288, 290–296, 299, 301–307 crew size 59, 65–67 critical incident 315, 317–318 data-driven perspective 42–43 delay 169, 171 delegation interface 251–255, 261–264 Deployable UAV Operations Cell (DUOC) 311–313, 315–318, 320–326 display configuration 383–384 displays 8, 10–11, 239–240 emerging technology 4, 8–9 ergonomics 23, 32 error classification 97 external piloting 106 external UAV pilot 75–76, 78 failures 92, 94–102 feedback 168–171, 176 field-of-view (FOV) 378–380 Fire Scout 112 flight control 105–106, 108, 111, 114–115 385

386 full scale environments 353 function allocation 54, 57 Generic Error Modeling System (GEMS) 99 Global Hawk 23, 25, 27, 30, 111–112 ground control station (GCS) 22–25, 32 heads-up-display (HUD) 25 Helios 110, 113 helmet mounted display 151 human error 91–94, 96–97, 100–101, 118–119, 121, 123–129 human factors 3–4, 6–9, 11–13, 17–19, 21–23, 25, 27, 30–32, 49–50, 54, 56, 117–118, 121, 123, 126–129 Human Factors Analysis and Classification System (HFACS) 121, 123, 129 human-machine interaction 268 human-robot 261, 264 human-robot interaction (HRI) 348, 351, 355–357, 361–363, 367, 371 human-robot team 330–332, 335–336, 338 human-system interactions 226–227 Hunter UAV 61–62, 69, 209–210, 216 hybrid team 330, 338–340 imagery 167, 170, 172 intelligent adaptive interface (IAI) 268–271, 273–274, 276–277, 279–281 intelligent control system 227 interfaces 7–9, 13, 15, 17–18, 176, 238–244, 246 internal UAV pilot 75–76, 78 interviews 315, 318 Job Assessment Software System (JASS) 62–65, 69–70

SUBJECT INDEX large-scale teams 243 low cost autonomous attack system (LOCAAS) 237–239, 243 manual control 166, 168, 170–173, 176 mapping 376–378 metacognition 337, 339–340 micro aerial vehicle (MAV) 165 mishaps 94, 105, 109, 111, 115 mismatch illusions 141 mixed-initiative control 228 modeling 212–213, 215, 220 multi UAV control 251, 258 Multi Unified Simulation Environment (MUSE) 78 multi-sensory interface 149, 160–161 multi-vehicle control systems 225 multimedia 329–330, 332, 340 multiple resources 212, 220 multiple robots 385 multiple UAV 184, 186, 189–190 multiple UAV control 213–214, 219, 231, 268, 270, 273 National Airspace System (NAS) 71–73, 77, 79–81, 86–87 network centric 179, 183, 186 network model 268, 270–271, 276–277 operator operator operator operator 74, 79 operator

72–79, 81, 86–87 certification 77 interface 267–269 selection 27, 29, 56, 71, training 27

partial mental models 226, 228 performance influencing factors (PIF) 93, 99 performance modeling 267, 269–270 Playbook 258, 260

387

Subject Index Predator 22–23, 25, 27, 110–111, 299, 301, 311, 315, 317 proximity 41–44 relational knowledge framework 224, 226 remotely operated combat vehicles (ROCVs) 39 research testbed 13 RoboFlag 257, 261–262 robot navigation 375–379 robotics 357, 359 robots 375–376, 381, 384–385 ROV 13, 17–19, 37–41, 44, 149–153, 155–156, 158, 160–161, 180, 223–228, 231, 235, 251–252, 254, 261, 264, 287, 299–301, 303–307, 329, 347–348, 351, 353–356, 362, 375–385 safety 4, 11, 18, 91 scale environments 352–353, 355 SD countermeasures 144 SD taxonomy 138 search munitions 237 Shadow 109–110, 209 shared mental models 303–307 simulation 59–62, 66, 69–70, 166–167, 169, 176–177, 348, 350–353, 355–357 situation awareness 50, 52, 55, 361–363 skills rules knowledge (SRK) 92, 99–100 soldier-in-the-loop simulations 61 space 38, 41–44 spatial aural display 150, 159 spatial dialog 193, 198–202, 204–205 spatial disorientation (SD) 133–144 speech recognition 158 supervisory control 49, 51–53, 55, 253, 263–264 suppression of enemy air defenses (SEAD) 49–52, 55 synthetic vision overlay 153

tactile display 155, 158 taskwork 288, 290–291 team cognition 286–287, 294, 296 team training 301, 306–307 teams 286, 288–295, 329–332, 334, 336–340 teamwork 288, 290–291, 294 tele-operation 166, 375–376, 381, 384 time 37, 40–44 training 92–96, 98–102, 165, 168, 170, 173, 176–177, 350, 352 training requirements 77 uninhabited combat air vehicles (UCAV) 49–56 unmanned aerial vehicle (UAV) 3, 21, 51–52, 59, 61–65, 67–69, 71, 91, 112, 117, 129, 133, 179, 193, 209–217, 219–220, 223, 237, 267–271, 273–274, 276, 279–281, 285, 312–313, 315–319, 321–327 UAV accidents 93, 100 UAV command and control 312 UAV operations 93, 100 UAV synthetic task environment (STE) 286–288, 290, 293–295 unmanned aircraft systems (UAS) 180 unmanned combat armed rotorcraft (UCAR) 195, 198 unmanned ground vehicles (UGV) 361, 371, 381 urban search and rescue (USAR) 362–368, 370, 372, 380–382 usability 168, 174–175 user interface design 224 vehicle certification 73 video displays 179, 184, 186–187, 189 video imagery 179, 188 videogames 291–293, 295 vigilance 65–66 visual displays 151, 160

388 visual illusions 138–141, 145 visual reference (VR) 135–138, 141, 145 wide area search munition (WASM) 237–245, 248

SUBJECT INDEX workload 50, 52, 55–57, 61–62, 65, 67, 69–70, 174–175, 194–195, 197–198, 206, 209–210, 212–213, 215–217, 219–220, 329–330, 337–340

Human Factors of Remotely Operated Vehicles, Volume 7 (Advances in Human Performance and Cognitive Engineering Research)

Advances in Human Performance and Cognitive Engineering Research, Volume 2 (Advances in Human Performance and Cognitive Engineering Research)

Advances in Human Performance and Cognitive Engineering Research, Volume 3 (Advances in Human Performance and Cognitive Engineering Research)

Human Factors of Visual and Cognitive Performance in Driving

Understanding Adaptability, Volume 6: A Prerequisite for Effective Performance within Complex Environments (Advances in Human Performance and Cognitive Engineering Research)

Human Factors in Alarm Design

Handbook of Cognitive Task Design (Human Factors and Ergonomics)

Human Factors In Consumer Products

Human Factors for Engineers

Aviation Psychology and Human Factors

Engineering Physiology: Bases of Human Factors Engineering Ergonomics, 4th Edition

Handbook of Human Factors in Litigation

Human Factors in the Training of Pilots

Human Factors in Simulation and Training

Handbook of human factors in litigation

Advances in Human Palaeopathology

Advances in Human Palaeopathology

Handbook of Human Factors in Web Design

Handbook of human factors and ergonomics methods

Human Factors Methods for Improving Performance in the Process Industries

Advances in Chemical Engineering, Volume 7

The history of human factors and ergonomics

Handbook of Human Factors and Ergonomics Methods

Research in Personnel and Human Resources Management, Volume 25 (Research in Personnel and Human Resources Management)

Advances in Ecological Research, Volume 7

Advances in Botanical Research, Volume 7

Human Reliability, Error, and Human Factors in Engineering Maintenance: with Reference to Aviation and Power Generation

Methods in Cognitive Linguistics (Human Cognitive Processing)

Methods in Cognitive Linguistics (Human Cognitive Processing)

Human Factors Interaction Theories in Consumer Product Design (Handbook of Human Factors in Consumer Product Design)

Ten Questions About Human Error: A New View of Human Factors and System Safety (Human Factors in Transportation)

Human Factors of Remotely Operated Vehicles, Volume 7 (Advances in Human Performance and Cognitive Engineering Research)

Advances in Human Performance and Cognitive Engineering Research, Volume 2 (Advances in Human Performance and Cognitive Engineering Research)

Advances in Human Performance and Cognitive Engineering Research, Volume 3 (Advances in Human Performance and Cognitive Engineering Research)

Human Factors of Visual and Cognitive Performance in Driving

Understanding Adaptability, Volume 6: A Prerequisite for Effective Performance within Complex Environments (Advances in Human Performance and Cognitive Engineering Research)

Human Factors in Alarm Design

Handbook of Cognitive Task Design (Human Factors and Ergonomics)

Human Factors In Consumer Products

Human Factors for Engineers

Aviation Psychology and Human Factors

Engineering Physiology: Bases of Human Factors Engineering Ergonomics, 4th Edition

Handbook of Human Factors in Litigation

Human Factors in the Training of Pilots

Human Factors in Simulation and Training

Handbook of human factors in litigation

Advances in Human Palaeopathology

Advances in Human Palaeopathology

Handbook of Human Factors in Web Design

Handbook of human factors and ergonomics methods

Human Factors Methods for Improving Performance in the Process Industries

Advances in Chemical Engineering, Volume 7

The history of human factors and ergonomics

Handbook of Human Factors and Ergonomics Methods

Research in Personnel and Human Resources Management, Volume 25 (Research in Personnel and Human Resources Management)

Advances in Ecological Research, Volume 7

Advances in Botanical Research, Volume 7

Human Reliability, Error, and Human Factors in Engineering Maintenance: with Reference to Aviation and Power Generation

Methods in Cognitive Linguistics (Human Cognitive Processing)

Methods in Cognitive Linguistics (Human Cognitive Processing)

Human Factors Interaction Theories in Consumer Product Design (Handbook of Human Factors in Consumer Product Design)

Ten Questions About Human Error: A New View of Human Factors and System Safety (Human Factors in Transportation)

Recommend Documents