People and Computers

People and Computers IX Cambridge Programme on Human-Computer Interaction Programme Advisers P. Barnard, Applied Psyc...

Author: Gilbert Cockton | Stephen Draper | George R. S. Weir

104 downloads 1287 Views 6MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

People and Computers IX

Cambridge Programme on Human-Computer Interaction Programme Advisers P. Barnard, Applied Psychology Unit, MRC, Cambridge, UK T. Bosser, University of Miinster W. Buxton, University of Toronto S. Card, Xerox PaRC J. Carroll, IBM Yorktown Heights J. Grudin, University of California, Irvine Thomas Landauer, University of Colorado J. Long, Ergonomics Unit, University of London T. Malone, Massachusetts Institute of Technology H. Thimbleby, University of Stirling T. Winograd, Stanford University

People and Computers IX Proceedings of HCI '94, Glasgow, August 1994

Edited by

G. Cockton University of Glasgow

S.W. Draper University of Glasgow

G.R.S. Weir Strathclyde University

| CAMBRIDGE UNIVERSITY PRESS

Published by the Press Syndicate of the University of Cambridge The Pitt Building, Trumpington Street, Cambridge CB2 1RP 40 West 20th Street, New York, NY 10011, USA 10 Stamford Road, Oakleigh, Victoria, Australia (5) Cambridge University Press 1994 First published 1994 Printed and bound in Great Britain by Biddies Ltd, Guildford and King's Lynn Library of Congress cataloging in publication data available British Library cataloguing in publication data available

ISBN 0 521 48557 6 paperback

Contents Invited Papers 1

The Future of Graphic User Interfaces: Personal Role Managers Ben Shneiderman & Catherine Plaisant

3

2

Beyond the Workstation: Mediaspaces and Augmented Reality Michel Beaudouin-Lafon

9

Methodology of Interactive Systems Development 3

Transferring HCI Modelling and Design Techniques to Practitioners: A Framework and Empirical Work Simon Buckingham Shum & Nick Hammond

21

4

The Use of Visual Indexing as an Interview Support Technique David Fulton

37

5

A Domain Analysis of Air Traffic Management Work can be Used to Rationalise Interface Design Issues John Dowell, Ian Salter & Solaleh Zekrullahi

53

6

Manuals as Structured Programs Mark Addison & Harold Thimbleby

67

7

Improving Education through Computer-Based Alternative Assessment Methods Jody Paul

81

8

Visual Programming in a Visual Domain: A Case Study of Cognitive Dimensions Francesmary Modugno, T R G Green & Brad A Myers

91

9

Evaluating Evaluation Methods A Dutt, H Johnson & P Johnson

109

Crafting Interaction: Styles, Metaphors, Modalities and Agents 10 A Comparison of Placement Strategies for Effective Visual Design Jean Vanderdonckt, Missiri Ouedraogo & Banta Ygueitengar

125

11 Evaluation of Alternative Operations for Browsing Hypertext Maria da Graca Campos Pimentel

145

12 On the Problem of Selecting Interaction Objects Frangois Bodart & Jean Vanderdonckt

162

v/

Contents

13 Minimising Conceptual Baggage: Making Choices about Metaphor Ben Anderson, Michael Smyth, Roger P Knott, Marius Bergan, Julie Bergan & James L Alty

179

14 Keeping an Eye on your Interface: The Potential for Eye-Based Control of Graphical User Interfaces (GUI's) Howell O Istance & Peter A Howarth 15 A Linguistic Approach to Sign Language Synthesis Alan Conway & Tony Veale

195

16 Generalisation and the Adaptive Interface Jonathan Hassell & Michael Harrison 17 Agent-Based Interaction Russell Beale & Andrew Wood

211 223 239

Modelling Humans, Computers and their Interaction 18 Multi-Perspective Modelling of Interface Design Issues: Undo in a Collaborative Editor Richard M Young & Gregory D Abowd

249

19 Qualitative Models of User Interfaces Mark Treglown

261

20

273

21

Modelling Interaction using Template Abstractions Chris Roast The Formal Analysis of Human-Computer Interaction During Accident Investigations Chris Johnson

285

Notations and Tools for Design 22

XUAN: Enhancing UAN to Capture Temporal Relationships among Actions Phil Gray, David England & Steve McGowan

301

23

Structured Notations to Support Human Factors Specification of Interactive Systems Kee Yong him & John Long

313

24

Discount Dialogue Modelling with Action Simulator Andrew F Monk & Martin B Curry

327

25

Executable Task Analysis: Integration Issues Conn V Copas & Ernest A Edmonds

339

26

Beyond Data Models for Automated User Interface Generation Angel R Puerto, Henrik Eriksson, John H Gennari & Mark A Musen

353

Computer-Supported Cooperative Work 27

Designing a User Interface for Folding Editors to Support Collaborative Work Richard J King & Ying K Leung

369

Contents

vii

28

Collaborative Use of X-Windows Applications in Observational Astronomy Darryn Lavery, Alistair Kilgour & Pete Sykes

383

29

Que Sera Sera — The Problem of the Future Perfect in Open and Cooperative Systems Alan Dix

397

30

Viewpoints, Actionpoints and Spatial Frames for Collaborative User Interfaces Steve Benford & Lennart E Fahlen

409

Author Index

425

Keyword Index

427

Preface: HCF94 — You Probably Haven't Seen It All Before This volume contains the full papers and invited papers from the 1994 conference of the British HCI Group (a specialist group of the British Computer Society). It is a partial record of a more extensive conference that includes tutorials, panels, short papers, posters and demonstrations. Human-Computer Interaction has been consolidating over the last few years. As a result, many recent conferences have tended to be bland, without really high points or really low points. The invited papers are forward looking and avoid blandness in that way. There are, however, many forward looking full papers. Moreover, the 1994 conference will be remembered as one that moved HCI forward in some areas, rather than just documented its current state within the established areas.

Proceedings Structure The proceedings begin with the invited papers. This apart, the usual compromises of programme structure have not been carried forward into these proceedings. The full papers in this volume have been reorganised into broad topics. As ever in HCI, these topics are not disjoint and some papers could have easily have gone in one section as another. Readers interested in specific topics are thus advised to read through the full contents to avoid missing papers that may be of interest to them. The second group of papers covers all stages of interactive systems development (properly iterated of course) from requirements capture and specification, through to evaluation. There are also papers on specific design issues, including application areas such as computer-assisted learning. The third group of papers is the largest, and covers design knowledge at all levels of abstraction in interactive systems, from low level presentation to high level metaphors. Novel modalities such as eye-control and sign language are covered, as are 'intelligent' features such as adaptation and agents. The fourth group of papers focuses on modelling the elements of HCI: humans, computers and their interaction. The hope is that such models will be of direct relevance to design. The papers in this section present work at various stages of such relevance. The fifth set of papers addresses the problem of capturing designs and related information. The growing scope of this aspect of HCI is reflected in wide range of topics covered in the five papers.

x

Preface

The last set of papers address one of the more demanding areas of HCI work. ComputerSupported Co-operative Work (CSCW) has been divided from birth between technological utopianists and workplace democrats. The papers in this section largely avoid these entrenched positions, reporting either small focussed pieces of practical work in CSCW or attempts to gain a better understanding of two important issues in CSCW.

Paper Selection As with some past BCS HCI conferences, referees from previous years' conferences were replenished with an open invitation to referee papers. Such an approach balances the usual aristocracy with consituencies who are generally denied influence, even though they constitute the bulk of the audience for the proceedings, as well as the bulk of the conference delegates. Industrial HCI practitioners, academics who are fairly new to HCI, and graduate students all have perspectives that programme chairs need to know about. This is not to deny the real expertise of our well established referees (several of whom responded to the open invitation), and their judgements were given proper weight during paper selection. However, each paper was refereed by at least one new referee, and their reactions were often very useful. The selection process was completed in six weeks. The final selections were made by the editors of these proceedings. We would like to thank referees for their prompt and informative reports. We would also like to thank Russel Winder for his continued highly professional work on the production of these proceedings, and his measured forebearance with those of us who let him down with deadlines.

In Summary Old HCI hands will know there can't be a summary. HCI is too diverse. Readers new to HCI should at least skim all the papers (and the keyword index) to get a feel for current topics and approaches. Old hands should do the same, because what was once up in the air may now be more down on the ground, and what is now up in the air will be a refreshing surprise. We enjoyed seeing these proceedings through from the initial submission of 73 papers. We hope that you will enjoy reading the invited and accepted papers. Lastly, we hope that those of you at the conference will have a profitable and enjoyable week in Glasgow. We want HCI'94 to be a memorable conference. Gilbert Cockton Stephen Draper George Weir Glasgow, August 1994.

Preface

xi

The International Programme Committee Klaus H Ahlers, ECRC, Munchen, Germany. James L Alty, LUTCHI Research Centre, Department of Computer Studies, Loughborough University of Technology, Loughborough, England. Farath Arshad, Information Modelling Programme, University of Leeds, Leeds, England. Michel Beaudouin-Lafon, LRI, Universitd de Paris-Sud, Paris, France. David Benyon, Open University, Milton Keynes, England. Thomas Berlage, GMD, Germany. Paul Booth, University ofSalford, Salford, England. Simon Buckingham Shum, Department ofPsychology, University of York, York, England. Berardina De Carolis, Universitd di Ban, Italy. John Cato, Software Design & Build Ltd, Bristol, England. Alison Cawsey, GIST, University of Glasgow, Glasgow, Scotland. Stephen Clarke, GIST, University of Glasgow, Glasgow, Scotland. Lynne Colgan, Software Design & Build Ltd, Bristol, England. Jonathan Crellin, University ofWolverhampton, Wolverhampton, England. G P Crum, Information Systems Group, Staffordshire University, Stafford, England. Donald L Day, Syracuse University, USA. Keith Dickerson, RACE Industrial Consortium, Brussels, Belgium. Alan Dix, The University of Huddersfield, Huddersfield, England. Mark D Dunlop, University of Paisley, Paisley, Scotland. Jonathan V Earthy, Lloyd's Register of Shipping, London, England. Stefania Errore, Universitd di Bari, Italy. David England, GIST, University of Glasgow, Glasgow, Scotland. Janet Finlay, University of Huddersfield, Huddersfield, England. Stephen Gale, AIT Ltd, Henley-on-Thames, England. Bill Gaver, Royal College of Art, London, England. Simon Grant, City University, London, England. Phil Gray, GIST, University of Glasgow, Glasgow, Scotland. T R G Green, MRC Applied Psychology Unit, Cambridge, England. Mark van Harmelen, University of Manchester, Manchester England. Jonathan Headland, BTLabs Human Factors Unit, Ipswich, England. Bill Hefley, Carnegie Mellon University, Pittsburgh, USA. Hans-Jiirgen Hoffmann, Department of Computer Science, University of Darmstadt, Darmstadt, Germany. Martin Hofmann, SAP AG, Germany. Andrew Howes, MRC Applied Psychology Unit, Cambridge, England. John A Hughes, Department of Sociology, Lancaster University, Lancaster, England. Kai Jakobs, Technical University of Aachen, Aachen, Germany. Chris Johnson, HCI Group, Department of Computer Science, University of York, York, England. Peter E Jones, The University of Western Australia, Perth, Australia. Sara Jones, University of Hertfordshire, Hatfield, England. Solange Karsenty, DEC Paris Research Laboratory, Paris, France. Alistair C Kilgour, Department of Computing and Electrical Engineering, Heriot-Watt University, Edinburgh, Scotland. Mark Kirby, University of Huddersfield, Huddersfield, England. Darren Van Laar, Department of Psychology, University of Portsmouth, Portsmouth, England. Darryn Lavery, GIST, University of Glasgow, Glasgow, Scotland. Paul Luff, Department of Sociology, University of Surrey & Rank Xerox Cambridge EuroPARC, England. Miles Macleod, NPL DITC, Teddington, England.

xii

Preface Ray McAleese, ICBL, Heriot-Watt University, Edinburgh, Scotland. Charles van der Mast, Delft University of Technology, The Netherlands. Philippe A Palanque, LIS Universite Toulouse I, Toulouse, France. Caroline Parker, HUSAT Research Institute, Loughborough University of Technology, Loughborough, England. Fabio Paternd, CNUCE — CNR, Pisa, Italy. John Patterson, GIST, University of Glasgow, Glasgow, Scotland. Roberto Polillo, Etnoteam & Universitd di Milano, Milan, Italy. Martin Prime, Rutherford Appleton Laboratory, Didcot, England. J van Putten, Cap Volmac, Huis ter Heide, The Netherlands. Judith Ramsay, Cognitive Science, Roskilde University, Denmark. D S Riches, BNR Europe Ltd, Harlow, England. Ian Ritchie, Heriot-Watt University, Edinburgh, Scotland. Chris Roast, Computing Research Centre, Sheffield Hallam University, Sheffield, England. Tom Rodden, Computing Department, Lancaster University, Lancaster, England. Zsofia Ruttkay, Faculty of Maths and Computer Science, Vrije Universiteit, Amsterdam, The Netherlands. Martina Angela Sasse, University College London, London, England. Dominique L Scapin, INRIA, Paris, France. Matthias Schneider-Hufschmidt, Siemens AG, Miinchen, Germany. Brian Shackel, HUSAT Research Institute, Loughborough University of Technology, Loughborough, England. Tara Smith, Nickelby Ltd, Glasgow, Scotland. Bob Steele, School of Computing and Management Sciences, Sheffield Hallam University, Sheffield, England. Alistair Sutcliffe, City University, London, England. Peter J Thomas, University of the West of England, Bristol, England. Manfred Tscheligi, University of Vienna, Vienna, Austria. Lorna Uden, School of Computing, Staffordshire University, Stafford, England. Dave Usher, InterAction Consultancy, Bath, England. Jean Vanderdonckt, Institut d'Informatique, Facultes Universitaires Notre-Dame de la Paix, Namur, Belgium. Peter Ward, Information Modelling Programme, University of Leeds, Leeds, England. Clive P Warren, British Aerospace pic, Sowerby Research Centre, Bristol, England. John Waterworth, Institute of Systems Science, Singapore. Andy Whitefield, University College London, London, England. Frank Wilson, Interaction Design Ltd, Welwyn Garden City, England.

Invited Papers

The Future of Graphic User Interfaces: Personal Role Managers Ben Shneidermant & Catherine Plaisant* ^Department of Computer Science and Institute for Systems Research * Human-Computer Interaction Laboratory at the Center for Automation Research University of Maryland, College Park, MD 20742, USA. Tel: +1 301 405 2680 Fax: +1 301 405 6707 EMail: {ben, plaisant}@cs.umd.edu Personal computer users typically manage hundreds of directories and thousands of files with hierarchically structured file managers, plus archaic cluttered-desktop window managers, and iconic representations of applications. These users must deal with the annoying overhead of window housekeeping and the greater burden of mapping their organizational roles onto the unnecessarily rigid hierarchy. An alternate approach is presented, Personal Role Manager (PRM), to structure the screen layout and the interface tools to better match the multiple roles that individuals have in an organization. Each role has a vision statement, schedule, hierarchy of tasks, set of people, and collection of documents. Keywords: personal role manager, desktop metaphor, graphic user interface, coordination, computer-supported cooperative work (CSCW).

1. Introduction The transition from the first generation command line interfaces (such as DOS 3 or UNIX) to second generation point-and-click graphical user interfaces (GUIs) was accompanied by an important metaphorical shift. The older systems required users to understand computerdomain concepts such as executable binary software (the .EXE or .COM files), file naming rules, and hierarchical directories. The designers of second generation GUIs presented users with more meaningful metaphors and supported direct manipulation interactions (Shneiderman, 1982). The graphical user interface offered a desktop with applications represented as icons, documents organized into folders, and even a trashcan as an affordance

4

Ben Shneiderman & Catherine Plaisant

for the delete action. This visual representation of the world of action made objects and actions visible, permitted rapid, incremental and reversible actions, and emphasized pointing and clicking instead of keyboarding. The current third generation approach emphasizes a "docu-centric" design (Microsoft's Object Linking and Embedding or Apple's OpenDoc Architecture), unified suites of software, and "information at your fingertips" through hypertext linking. Documents become more important and applications fade into the background. The enriched documents contain multiple object types such as text, drawings, photos, spreadsheets, sound, animation, and even video with links across documents to share common objects. Actions that earlier had required opening an application, such as spell checking, thesaurus reference, or faxing a document, are now integrated into the unified docu-centric interface. While these are useful steps away from the underlying technology and more in harmony with the users's perceptions of their work, larger steps are needed to reach the fourth generation in the evolution of user interfaces. We believe that the natural progression is towards a "role-centered" design which emphasizes the users tasks rather than the documents. This is in harmony with the current movement toward computer-supported cooperative work and groupware. These tools are aimed at coordination of several people performing a common task with a common schedule. Our goal is to substantially improve support for individuals in managing their multiple roles in an organization. Each role brings them in contact with different sets of people for separate hierarchies of tasks following independent schedules. Our goal is to improve performance and reduce distraction while working in a role, and facilitate shifting of attention from one role to another. This proposal for a Personal Role Manager is at an early stage of development. It emerged from a research project with the World Bank to explore future desktop environments. Our screen mockups and MacroMind director scenarios are merely the first step in a long path towards commercial software development. We hope that by describing this work at an early stage we can elicit feedback from colleagues and encourage others to refine, expand, and apply our concept.

2. Previous Efforts Earlier efforts on roles have come more from the perspective of the organization (Biddle & Thomas, 1979; Roos & Starke, 1981; Sarbin & Allen, 1968). Singh & Rein (1992) write that: "Briefly stated, role theory views individuals as occupying positions in organizations ... Roles are the building blocks of organizational structures and are descriptive of such organizational phenomena as division of labor and specialization." As a result the focus is on coordination among roles within the organization. Although there is some recognition that an individual might have multiple roles, the emphasis is on the assignment of individuals to a single organizational role. The ensuing coordination theory and technology applies Petri net methods and other formal specifications to develop useful methods for human-human coordination (Singh, 1992; Singh & Rein, 1992). This managerial emphasis places greater concern on ensuring that organization goals are met, rather than facilitating the individuals' tasks of managing their time, organizing their information, and communicating with peers or subordinates. The user interface designs

The Future of Graphic User Interfaces: Personal Role Managers

5

that are suggested by this established research direction serve the needs of managers more than subordinates. Recent attempts to help users structure their work within multiple roles focus on personal schedulers to manage time and on tools to manage documents. Most users will group related documents in directories or folders, and can create screen displays with multiple related windows. The 'rooms' notion (Henderson & Card, 1986) does support the storage and retrieval of groups of windows — for example, Rooms for Windows (Xerox), Dashboard (Hewlett-Packard), and the Unix-based HP-VUE (Hewlett-Packard). The rooms notions could be expanded to support coordination across windows such as triggered deletions, synchronized scrolling, hierarchical browsing, and hypertext linking (Shneiderman, 1992). Other useful concepts have emerged from studies of how knowledge workers organize and use information (Kidd, 1994;Malone, 1983).

3. Personal Role Manager: An Overview In our proposed Personal Role Manager, each role has a vision statement (a document that describes responsibilities, quotas, goals, etc.) that is established by the user or manager. The explicitness of the vision can simplify the training and integration of new personnel into the organization and also facilitate the temporary covering of responsibilities among employees (for vacations or parental leave). For example, a professor may have roles such as a teacher of courses, advisor to graduate students, member of the recruiting committee, principal investigator of grants, author of technical reports, and liaison to industry. In the teacher role the professor's vision statement might include the intention to apply electronic mail to facilitate a large undergraduate course. Files might include homework assignments, bibliography, course outline, etc. The task hierarchy might begin with tasks such as choosing a text book and end with administering the final exam. The subtasks for administering the final exam might include preparing the exam, copying the exam, reserving a room, proctoring the exam and grading it. The set of people include the students, teaching assistants, bookstore manager, registrar, and colleagues teaching other sections of the course. The schedule would begin with deadlines for submitting the book order to the bookstore manager and end with turning in the final grades to the registrar. Similarly, a World Bank employee may have the role of a task manager (e.g. handling two projects: drinking water in Mali and a dam in Kenya), a role as a domain expert on steel construction, a member of the task force on information management, be the Lotus 1-2-3 peer training leader, and also organize the holiday party. Our approach was stimulated by our experiences in managing complex projects with many participants. We also observed and interviewed experienced users to understand what their needs are and how current systems fail to support their tasks. While there are various scheduling, time management, address book, document management packages available, the coordination of these efforts is often under-emphasized. The Personal Role Manager (PRM) that we propose would simplify and speed the performance of common coordination tasks, in the same way that GUI interfaces simplify and speed file management tasks. We believe that the PRM provides a novel foundation and will generate refinements and extensions. The key to PRM is organizing information according to the roles that an individual has in an organization. In our mockup, when users are working in a role, they have most

6


relevant information visually available. These visual cues remind them of their goals, related individuals, required tasks, and scheduled events. The initial layout of roles may be established by a manager for a new employee, but then the employee can adjust, combine, or split roles as the demands change. Screen management is one of the key functions of the PRM. All roles should always be visible but the current focus of attention could occupy most of the screen. As the user shifts attention to a second role, the current one would shrink and the second one would grow to fill the screen. Users could simultaneously enlarge two roles if there were interactions between them. 3.1. Vision

Statement

Each role has a vision statement that reminds the users of their goals. As a professor, my teaching role might have a vision statement about my desire to "increase class participation by collaborative methods, improve teamwork on term projects by requiring regular management meetings, prepare careful notes to facilitate future teaching of the same course, and coordinate with my teaching assistants by weekly meetings and email discussions." Such vision statements would likely be personal, but they could become a useful basis for discussions with peers or superiors.

3.2. Set of People When acting in a given role users interact with a set of people that is a subset of the large number of people who might be in a personal or organizational phone book. Making the role-relevant group of people continuously visible (for example with names or small photos on the border of the large screen) has at least two benefits. First, the images will act as cues to remind the user of the need to inform, request or communicate with that individual (similar to seeing someone in the hallway which triggers some communication to coordinate work). Second the images act as active menus to initiate phone, fax, or email communication. For example, a document can be dragged and dropped onto an image triggering email plus a log of the action. Providing direct access to those people without the need of a directory search speeds performance and reduces cognitive load. 3.3. Task Hierarchy Tasks are hierarchically organized into subtasks using an outlining tool, or other display of tree structured information. The professor role may have a task for each of several courses or the principal investigator role may have tasks for multiple grants. Each course has multiple subtasks such as writing the syllabus, ordering textbooks, giving exams, and preparing final grades. A World Bank employee can have two or three projects to manage, each with multiple subtasks. The task hierarchy acts as a to-do list, and is linked to the schedule calendar to remind the user of upcoming deadlines. 3.4. Schedule Each role has an associated schedule that is a component of a user's master schedule. When viewing a role, the user initially sees only the role-related schedule. For example, when in the professor role, the semester schedule is visible and when in the principal investigator role, the 2-year grant schedule is visible. Schedules can be combined to reveal a master schedule to allocate time and ensure that travel, vacations, and required meetings are blocked off on every schedule.

The Future of Graphic User Interfaces: Personal Role Managers

7

4. Related Developments The Personal Role Manager is potentially an important development, although it must be refined and its viability tested. However, there are a number of other important developments that will influence the PRM and other initiatives. The central development appears to be the increased emphasis on visual information seeking (Ahlberg & Shneiderman, 1994) and the dynamic queries approach (Shneiderman, 1994). These both are extensions of the direct manipulation principles of visual display of the world of action, rapid, incremental and reversible actions, and immediate (within 100 ms) and continuous display of the results of actions. These approaches start with a complete overview of a database, followed by zooming and filtering to narrow attention, and then pointing to get details-on-demand. By effective organization of visual displays, large amounts of information (3000 or more objects on the screen at once) can be absorbed, patterns recognized, and anomalies spotted. The remarkable human capacity for visual perception is underutilized in most contemporary applications. To gain the benefits of visual information seeking larger displays will be necessary for many applications. The current 640x480 pixel standard display is inadequate for many tasks and larger display such as 1280 x 1024 pixels will be helpful. Of course high performance computation to support animation, smooth zooming, and rapid panning will also be necessary. Even with larger display spaces, careful screen management could dramatically improve performance. Dialog boxes should appear close to but not on top of related information. Dialog boxes should automatically disappear as tasks are completed. Improved image browsers (Plaisant, Carr & Shneiderman, 1994), hierarchical browsers, and other coordinated displays are possible (Shneiderman, 1992). Improved input devices, gestural methods, and two-handed input (Bier et al., 1994) will also speed performance on the larger and denser displays. The proportion of computer input by keyboard will decline as pointing techniques mature, as designers create improved interfaces, and as increased fractions of relevant information are available online. Highspeed network connections will continue to spread more rapidly than we can anticipate and participation will increase. Video email, video-conferencing, video-information services, and video- entertainment will spread as the network connections increase, hardware/software improves, and prices decline. 5. Vision Statement, Again Computer scientists have an appreciation for recursion and therefore it seems fitting to consider a vision statement for the role of futurist. We in the computer sciences and related disciplines will earn greater respect if we increase our attention to serving human needs, as opposed to focusing on technology. Future developments could be more regularly guided by our fundamental values and societal goals. Allowing individuals to function more effectively inside larger organizations seems vital to improving health care, education, social services, etc. User interface developments are more likely to contribute to shaping a better world if there is an open discussion of goals, participatory design involving multiple viewpoints, and thoughtful consideration of the social impact (Shneiderman, 1990). We cannot guarantee a better world

8


through advances in science or new technologies, but if we explicitly attend to the social impact in our work we can more often bend the technology to serve genuine human needs.

Acknowledgements We gratefully appreciate the support of the World Bank for this opportunity to brainstorm about the future. We recognize the important assistance provided during the interviews with 18 World Bank personnel and the encouragement of Michael Sharpston.

References Ahlberg, C & Shneiderman, B (1994), "Visual Information Seeking: Tight Coupling of Dynamic Query Filters with Starfield Displays", in Proceedings of CHI'94: Human Factors in Computing Systems, B Adelson, 5 Dumais & J Olson [eds.], ACM Press, pp.313-317. Biddle, B J & Thomas, E J (1979), Role Theory: Concepts and Research, Krieger Publishing. Bier, E A, Stone, M C, Fishkin, K, Buxton, W & Baudel, T (1994), "A Taxonomy of See-through Tools", in Proceedings ofCHI'94: Human Factors in Computing Systems, B Adelson, S Dumais & J Olson [eds.], ACM Press, pp.358-364. Henderson, A & Card, S K (1986), "Rooms: The Use of Multiple Virtual Workspaces to Reduce Space Contention in a Window-based Graphical User Interface", ACM Transactions on Graphics 5 (3), pp.211-243. Kidd, A (1994), "The Marks are on the Knowledge Worker", in Proceedings of CHI'94: Human Factors in Computing Systems, B Adelson, S Dumais & J Olson [eds.], ACM Press, pp.186-191. Malone, T (1983), "How do People Organize their Desks? Implications for the Design of Office Information Systems", ACM Transactions on Office Information Systems 1(1), pp.99-112.

Plaisant, C, Carr, D & Shneiderman, B (1994), "Image Browsers: Taxonomy, Guidelines and Informal Specifications", Department of Computer Science, University of Maryland, Technical Report. Roos, L L & Starke, FA (1981), "Organizational Roles", in Handbook of Organizational Design Vol I: Adapting Organizations to Their Environments, P C Nystrom & W H Starbuck [eds.], Oxford University Press. Sarbin, T R & Allen, V L (1968), "Role Theory", in Handbook of Social Psychology (2nd Edition), G Lindzey 6 E Aronson [eds.], Addison Wesley. Shneiderman, B (1982), "Multi-party Grammars and Related Features for Designing Interactive Systems", IEEE Transactions in Systems, Man and Cybernetics 12 (2), pp.148-154. Shneiderman, B (1990), "Human Values and the Future of Technology: A Declaration of Empowerment", SIGCAS Computers & Society 20(3), pp.1-6, Keynote address, ACM SIGCAS Conference on Computers and the Quality of Life CQL'90, Reprinted in ACM SIGCHI Bulletin (January 1991). Shneiderman, B (1992), Designing the User Interface: Strategies for Effective Human-Computer Interaction (2nd Edition), Addison Wesley. Shneiderman, B (1994), "Dynamic Queries for Visual Information Seeking", IEEE Software, (in press). Singh, B (1992), Interconnected Roles (IR): A Coordination Model, MCC, Austin, TX, USA. Singh, B & Rein, G (1992), "Role Interaction Nets (RINS): A Process Description Formalism", MCC, Austin, TX, USA, Technical Report CT-083-92.

Beyond the Workstation: Mediaspaces and Augmented Reality Michel Beaudouin-Lafon Laboratoire de Recherche en Informatique — CNRS URA 410, Batiment490—Universite de Paris-Sud, 91 405 ORSAY Cedex, France. Tel: +33(1)69 416910 Fax: +33(1)69 4165 86 EMail: [email protected]

The embodiment of computers in desktop workstations has had a tremendous impact on the field of HCI. Now that mice and graphics displays are everywhere, the workstation defines the frontier between the computer world and the real world. We spend a lot of time and energy transferring information between those two worlds. This could be reduced by better integrating the real world with the computer world. This article describes two approaches to this integration: Mediaspaces, which allow people to communicate through an audio, video and computer environment, and Augmented Reality, which adds computational power to real world objects. The argument is made that the success of these approaches lies in their ability to build on fundamental human skills, namely the ability to communicate with other people and the ability to interact with objects in the real world. Keywords: mediaspace, augmented reality, multimedia, video, virtual reality, paper interface, gesture input, metaphor, paradigm.

1. Introduction Over the last decade, computers have evolved from mainframes to networks of personal computers and workstations. The range of users and uses of computers has expanded dramatically. Today, a computer is perceived more as an appliance than as a 'machine'. A key aspect of this evolution has been, and still is, the development of the field of HumanComputer Interaction. HCI has complemented, and sometimes driven, the evolution of the technology to make computer systems easier to use by a wider variety of users in a larger number of contexts.

10

Michel Beaudouin-Lafon

As most researchers in HCI know, this picture is a bit idyllic and much still needs to be done to improve the scope and usability of computers. At the same time, new buzzwords keep cropping up as the solutions to all the problems: multimedia and virtual reality are currently popular. In a world where GUIs have become passe, multimedia is already here and virtual reality is supposed to touch everyone tomorrow morning, people seem to have forgotten what these technologies are for. Expectations seem driven by the hype and the technology is treated as inevitable and fated to happen. Meanwhile, users continue to fight with the real world, with real computers and real human beings, and experience one of the two extremes of computing in the nineties: being a guru or being a wimp. It seems that, once again, the user has been forgotten in the process. What will these new technologies bring to the average user? Why should they change to these new technologies when it has been so hard to manage the current ones? All too often, using a new technology requires new skills. New skills are hard to acquire and in general they are acquired at the expense of other skills. Many people who use computers a lot, including me, now have a hard time writing a letter by hand. My typewriting skills have been learnt at the expense of my handwriting ones. So there is a real question of which skills are required to operate a computer, and which skills are going to disappear. This question is complex because it is hard to determine which skills are more important than others. Should we stop learning handwriting in school and switch to touch-typing? The latter position seems extreme, though, and has consequences far beyond the scope of HCI. A less controversial approach would let us continue to use our 'standard' skills to interact with computers. Indeed, many researchers have been working on these issues, e.g. speech and handwriting recognition. But we still face a sort of human barrier. Despite the number of speech and handwriting recognition systems, it is still impossible to use them to interact with a computer in a 'natural' way, i.e. as easily as with a human being. So we are faced with a problem. On the one hand, it is expensive to learn new skills. On the other hand, we cannot use all of our existing skills to interact efficiently with the computer. But this is not as hopeless as it might seem: a third approach lies between these two extremes. This article describes how we can build upon our existing skills by adapting them to new situations and developing them in new directions, rather than just replacing them or trying to use them as is. I will apply this approach to two sets of skills: our ability to communicate with other humans and our ability to interact with objects in the real world. These sets of skills respectively underlie the notion of mediaspace and the concept of augmented reality.

2. Mediaspaces The term mediaspace was coined in 1985 when a fixed audio-video link was set up by Xerox researchers between two sites, one in Palo Alto, California and the other in Portland, Oregon (Olson & Bly, 1991). Video links had been used previously for videoconferencing. However they had a very different feel to them: they linked dedicated rooms, required booking in advance, and were used primarily for important meetings. In contrast, Xerox's mediaspace was set up as a permanent link between the commons areas of two labs. The goal was to support both formal and informal communication between the lab members located at each site and to preserve the sense of community they would have had if they had worked in the same location.

Beyond the Workstation: Mediaspaces and Augmented Reality

11

Unlike a typical videoconferencing set up, the mediaspace supported a variety of uses. People had both formal and informal meetings and met each other accidentally, much as they would in the hallways of a shared building. The participants found that the mediaspace let them work together in a much more effective way than with conventional videoconferencing facilities. Before Xerox's mediaspace, an installation called the Hole-in-Space (Galloway & Rabinowitz, 1980), demonstrated the propensity for people to spontaneously communicate over an audiovideo link. The Hole-in-Space was an audio-video link relayed by a satellite between a street in New York City and a street in Los Angeles. Despite the lack of advertising or explanation as to what it was, people spontaneously used it to communicate. Initial uses were triggered by curiosity and led to exchanges between people who did not know each other. After a few days, some people even used it to meet with friends and relatives. At the CHI'94 conference in Boston, Garry Beirne and I installed a similar system, called Eye-on-CHI, that linked two remote sites of the conference premises. We observed people interacting spontaneously with people they did not know; student volunteers used it to handle their jobs more efficiently; and friends and colleagues met and chatted as if they were in the same hallway. These experiments show the tremendous power of setting up long-term audio-video links. They allow for almost the full bandwidth of human communication, which no other medium can provide. Such links are obvious to use, since there is no special user interface and no technology to operate. More recent mediaspaces have introduced computers into the system, for three different purposes. First, some users want access to a variety of connections instead of a single fixed link. So, most media spaces now have computer-controlled switches with cameras and monitors in each user's office (Bellcore, 1993; Buxton & Moran, 1990; Mantei et al., 1991). Various schemes have been tried to control such systems, with particular emphasis on user-controlled access rules to protect users' privacy (Gaver et al., 1992). The user interface for providing access control has proven to be difficult to design. A variety of metaphors have been tried, including telephones, doors, rooms, and hallways. Unfortunately, they don't work well in practice. The real world doesn't require such an explicit interface: the negotiation to begin communicating is part of the communication itself, rather than a separate protocol. The various services that media spaces provide, such as 'glance', 'video-phone', and 'cruise', fail to build upon our natural skills for engaging in a conversation. For example, when establishing a video connection, the other person's image suddenly appears on the screen as if she were popping up a meter away from the user. In the real world, there is an approach phase, which serves to gradually establish communication, according to a set of well-understood social conventions. Failure to reproduce these transitions in a mediaspace can result in rude invasions of each other's personal spaces, which in turn reduces the use of the system. The second use of computers consists in replacing the analog audio-video networks with digital ones. Some systems have a mixed network, where local connections are analog and long-distance connections use a video codec. Current digital video technology introduces degradations (lower frame rate, perceivable delay, lossy compression) that are far more disruptive for users than those introduced by analog video (color smearing, image going black and white, etc.). When the frame-rate drops and the transmission delay increases, many clues about human communication are lost or don't work, making the users feel uncomfortable. (The same phenomenon occurs with long-distance calls that are relayed by satellite, in which the speakers must adjust to a half-second delay). This indicates that the less disruptive

12


degradation of a signal depends not only on the contents of the signal (e.g. audio vs. video) but also on the use. When people are talking to each other, it's critical to minimize delays. On the other hand, delays have little importance if people are simply watching a pre-recorded video over the network. In my opinion, real-time digital video cannot yet provide the quality required to support good human communication over an audio-video link: we need faster machines and better networks. The third, and most important use of computers is to complement audio-video connections. Having just an audio-video link is insufficient for most tasks that involve some form of collaboration, since the users need to share artifacts in addition to seeing and talking to each other. A number of groupware systems have been developed for that purpose, most notably shared editors (shared sketching surfaces, shared text editors, shared drawing tools, etc.). When used without an audio or video link, these tools are almost unusable or meaningless. When used in conjunction with a link, they reveal the full power of the concept of mediaspace. This is best exemplified with the ClearBoard system (Ishii & Kobayashi, 1992) where the shared surface and the video images are cleverly overlaid to produce the effect of two people drawing on two sides of the same transparent glass. This system provides gaze awareness (i.e. I can tell what you are looking at), a skill we use constantly in face-to-face communication with someone else about a particular artifact. When a system succeeds in building upon our existing communication skills, it becomes transparent, i.e. its complexity becomes invisible or non-important to the users. This is, in my opinion, the true goal of a mediaspace: to be invisible. A mediaspace can be defined as the conjunction of a system that supports human communication (e.g. an audio-video network) and a system that supports sharing of artifacts (e.g. shared drawing tools). Together, they provide the ability to create shared contexts that reproduce real-life situations in which one can talk to somebody and use a blackboard or a piece of paper to explain a point (Bly, Harrison & Irwin, 1993). This requires careful integration of the two systems, a task that very few mediaspaces have undertaken. Montage (Tang & Rua, 1994) shows a simple example of such an integration: while they are connected, users can start a shared drawing tool. If one person is absent, an electronic note can be left on that person's screen. The success of this integration will be one of the keys to the success of mediaspaces, and is a bigger challenge than it seems. Another key will be the integration of synchronous and asynchronous communication. Virtually all group tasks combine situations where several users work jointly and situations where they work individually and communicate asynchronously either explicitly through email or implicitly by recording changes to a shared document. If mediaspaces are to support group work, they have to support such patterns of use. Within the shared context that a mediaspace provides, many things may happen. How does a user keeps track of them? In the real world, the answer is through awareness and notification. Awareness allows us to passively maintain a mental representation of a phenomenon (e.g. who is in the lab) which we then use to make decisions. Notifications remind or inform us that something has happened or is going to happen (e.g. a deadline is coming), and trigger further action on our part. Awareness can be provided by a mediaspace by collecting information and making it available to the users. For example, the Portholes system (Dourish & Bly, 1992) grabs images from each camera at regular intervals and displays them to the end users who can be globally aware of who is there. (Issues of privacy are addressed with the general


13

access control mechanism used in the mediaspace). In our mediaspace at LRI, we make these images available as a Mosaic document on the World-Wide Web, so that colleagues in different organizations can tell whether we are present and available. In addition, we use these images to directly access the mediaspace. The most natural way to enter somebody's office through the mediaspace, once an image of that office is on the screen, is to click on it. This demonstrates that awareness tools can be not only a useful part of a mediaspace but also a means of making the system more transparent by providing direct access to other functions of the mediaspace. Notifications are harder to provide because they may require an interpretation of the data transported over the mediaspace. It may be difficult to determine whether or not the condition triggering the notification has been met. For example, in order to be notified of the presence or absence of a person in her office, the system needs to interpret the contents of the image of that office. An additional problem with notifications is that they are not passive: users need to specify the notifications they want and how they want to handle them, which can rapidly become cumbersome. In the real world, we rely on a limited set of notifications (mostly regular reminders, deadlines and alarms) and we have little control over their generation and methods of handling them. In our experience, it is better to keep mediaspace notifications to a minimum and to minimize their programmability, for the sake of simplicity. When designed well, a system draws its power from simplicity rather than complexity. Complexity generally reveals a poor design, one that has not taken into account the characteristics and needs of its intended users. Note that, apart from some notifications, data exchanged over a mediaspace is not processed or interpreted, but is merely transported from one point to another in the network. This is an important characteristic of a media: to provide a means for transporting and distributing information without being concerned about its contents. It is important however to recognize that the medium does affect the contents. For example, Gaver (1992) examines the affordances of video in the context of mediaspaces, and concludes that it is impossible for them to be totally transparent. The range of information that is conveyed over an audio-video connection is so wide that extracting it automatically by computer seems hopeless. However, it is possible to extract a set of indicators from this data that can be used by the system in various ways. For example, relatively simple image processing techniques can be used to determine the number of people in a scene (Ueda et al., 1993). Similarly, it is possible to segment an audio track into utterances (Hindus & Schmandt, 1992) which can be used to automatically index the conversation for later access. Such techniques can be used to create minutes of a meeting or help analyze video data (Mackay & Davenport, 1989) and will undoubtably be integrated to future mediaspaces. The approach taken by mediaspaces differ from that of multimedia in that it focuses on the end rather than the means. Many multimedia systems focus on the means (mix video and audio clips, texts and images) without a proper definition of the end. A mediaspace uses audio, video, text, graphics, and images and yet it is more than a multimedia system: it is an integrated environment based on human communication that supports a wide range of uses based on our everyday skills. We can think of it as a new medium, rather than a collection of existing media.

14


In conclusion, mediaspaces show that there is a great potential in exploring means for creating new media with computers. They also break the barrier of the conventional workstation by adding an eye and an ear, and by creating a new role for it: the support of human-human communication through human-computer interaction.

3. Augmented Reality Like mediaspaces, Augmented Reality creates a new role for the computer, which can be described as supporting our interaction with real-world objects. Nowadays, the workstation marks the frontier between the computer world and the real world. Means for exchanging information between those two worlds are very limited when compared with the means to circulate information within each of these worlds. The complementarity of the computer and the real world is quite striking. For example, a newspaper can be carried around, folded, annotated and easily browsed, while its electronic counterpart is easy to index, search, duplicate, cut and copy. It is certainly not surprising that computers have been developed to support tasks that were difficult to carry out in the real world. But it is the case that in the process, the computer world has been treated more as a replacement of the real world than its complement. We can see this with the mythical paperless office: computers have generated more paper rather than getting rid of it. We also see this with the current Virtual Reality fad. Although the definition of virtual reality differs according to different authors, it is generally accepted that a VR system features some sort of immersion: the user's actions are all interpreted by the computer and the user's perceptions are all under computer control. Although this approach certainly has valuable applications in areas such as telepresence, simulation and some forms of data visualization, it is not applicable to interactive systems at large: one can hardly imagine a secretary wearing gloves and goggles to type a memo or read e-mail or a layman entering the bank's cyberspace to withdraw money from an ATM. Even if these technologies become much smaller and easier to use, the concept of immersing oneself in the computer's world ignores the on-going problem of interacting with the real world. Although the research community admits these limitations, the media are still hyping VR to fuel the public's fantasy. (Look at the recent Aerosmith 'Misery' video for an ironic look at the latest in VR fantasy.) This is not surprising. Any technology whose goal is to get rid of the real world is bound to grab the attention of the wider public. After all, ridding ourselves of the constraints of the real world is understandably intriguing. Artificial Intelligence was similarly overhyped some years ago. Advertised as the technology that would make machines as intelligent as people, it promised a way of ridding ourselves of the constraints that other people impose upon us. Augmented Reality explores a much more pragmatic route, although it is not any simpler technologically. AR recognizes that people are used to the real world and that the real world cannot be reproduced accurately enough on a computer. So AR builds on the real world by augmenting it with computational capabilities. Instead of sucking the user inside the computer world, as in VR, Augmented Reality overlays the computer world onto the real world or embeds it within the real world. This general paradigm is best understood through examples. The Digital Desk (Wellner, 1993) is a canonical example of an augmented reality system. The Digital Desk is a conventional desk with a camera and a video projection system mounted overhead. The camera is used to track objects and the user's gestures on the desk while the


15

projector is used to display virtual objects (windows) on the desk. A microphone is installed in the desk to detect finger taps from the user, which are used to deliver commands to the computer. The Digital Desk allows for a variety of interaction schemes that mix real objects (mostly paper) and computer objects. For example, a user might be reading a paper memo with a list of numbers on it and decide to make some calculations. The Digital Desk can display a spreadsheet window onto the desktop (with the projector), let the user select a list of numbers (with the camera and microphone) and drag them to the projected worksheet (with the camera). An optical character recognition system interprets the numbers and enters them into the spreadsheet, which the user can continue working with, as projected onto the desktop. The Digital Desk can also be used as a drawing tool. The user sketches a drawing on paper, which is then digitized by the camera and can be duplicated, scaled and edited by direct manipulation on the desktop. By linking two Digital Desks, users can cooperate on design tasks using the desk as the shared surface rather than a computer screen. The Digital Desk can also be used for specific tasks such as complex graphics design and video editing (Mackay et al., 1993). It is easy to imagine other uses of the Digital Desk, which demonstrates the effectiveness of the approach. By marrying our everyday skills for interacting with real-world objects with the ability of the computer to do certain tasks, the resulting system is more than the sum of its parts. As with mediaspaces, the key is integration that leads to transparency. Users don't think of the Digital Desk as a desk-wide computer screen combined with a deskwide scanner, an OCR system and a hand-controlled mouse, they think of it as an enhanced desk, a tool that lets them keep their existing work practices while simplifying their tasks. Another example of an Augmented Reality system is the Charade system that we developed at LRI (Baudel & Beaudouin-Lafon, 1993). Charade allows a user to give a presentation by controlling it with hand gestures. Actions such as going to the next slide or to any particular slide, highlighting a part on a slide, and triggering animations are all specified by hand gestures addressed to the screen. The user can define the gestures that Charade recognizes. Unlike many other systems, Charade recognizes real gestures, not just positions of the hand and arm. The range of recognizable gestures is therefore quite large and users can select the gestures that feel most natural to them. Typically, the gesture vocabulary exhibits a mix of iconic and non-iconic gestures. Non-iconic gestures are those that could be used to give a gestural order to an assistant turning the slides. For example, opening the right hand, palm to the right, and sweeping it to the right is clearly understood as going to the next slide. Similarly, pointing an index towards the screen means highlighting a part of the slide. Iconic gestures specify a command through a particular code. For example, we could decide that closing the fist means closing the presentation. Obviously, non-iconic gestures are preferable to iconic ones but cannot cover, in most cases, the vocabulary that is necessary to operate the system. Our experience indicates that the best gestures involve tensing the hand. This allows Charade to detect the user's intention when issuing a command, and helps the system discriminate between gestures used to support the presentation and gestures used to command the system. Yet this discrimination is achieved without any explicit marking, other than the gesture itself. This is the most important aspect of the system: experiments in which users gave real presentations with Charade showed that they issued commands very naturally and that the recognition and discrimination rate of the system was better in a real setting than in a raw test. Charade can be applied in other environments as a component of an augmented reality system. In addition to controlling a computer display, Charade can control real objects. For example,

16


at home, appliances such as VCRs, TV sets, answering machines, microwave ovens, lights, require either direct operation or the use of a remote control. Charade would allow users to remotely operate each appliance through hand gestures. The large control and display panels in control rooms could be replaced (or complemented) by hand gestures. In summary, what Charade does is to enhance real-world objects with the ability to respond to hand gestures. Real-world objects can also be used to enhance the computer, such as the props used by Hinckley et al. (1994) to orient 3-D objects and define cutting planes for the analysis of medical data. Or the computer can be used as a portable detector to access information embodied in familiar objects such as books and maps (Fitzmaurice, 1993). In other words, the general approach is to think of objects that we use in our everyday life and to imagine ways to enhance them with computational power. Implementing augmented reality systems is not without difficulties. Computers are not designed to keep track of the real world. One major difficulty is that of registration: objects in the real world must be registered with the system, associated with their electronic counterpart, tracked as they move and registered again when they reappear at a later time. For example, the Digital Desk needs to register the paper documents present on the desk, and Charade needs to register the so-called active zone where the object in question lies. This may require complex image recognition techniques or clever use of sensors. Since the goal is to enhance conventional objects, solutions to these problems should not require a complete re-engineering of those objects. Therefore passive marks on the objects are preferred to active devices, even though the latter are easier to use. Another technical difficulty is that of capturing input. In the Digital Desk, the camera cannot OCR text in normal-size fonts because it lacks definition. One solution is to have two cameras, one of them zoomed in on a part of the desk. This however is not satisfying since it affects the transparency of the system. In Charade, the user's gestures are captured by a VPL Dataglove, a clumsy and encumbering device. Gestures could be captured by one or several cameras, maybe with the help of a passive glove, but this has not been tried yet. Such difficulties can and will be overcome if we increase our research efforts. Although these systems are early prototypes, they exhibit reasonable performance. At the same time, their problems are far less than those faced by virtual reality systems: complementing the real world is much easier than creating an exact duplicate of it. In addition to being extremely difficult, copying the real world turns out very limiting. After all, should we really be trying to build airplanes that flap their wings in order to fly? An advantage of Augmented Reality is that its design space is much larger and has greater scope for creativity. In addition, the integration of mediaspaces with augmented reality, which starts being investigated, promises new and exciting perspectives.

4. Conclusion The embodiment of computers in desktop workstations has had a tremendous impact on the field of HCI. Graphics displays and mice have provided such a rich environment for the development of user interface metaphors that the users and the designers are in awe of the magic box. Like the heroes in the movie "Until the end of the world" (Wenders, 1991), we have been captivated by the images we have produced and have forgotten the real world.


17

It is time to get out of the box. There are other people in the real world and we constantly communicate with them. They rely on us and we rely on them. So let the computer help us communicate more effectively, with our full range of communication skills. There are objects in the real world and we depend on them for virtually all our activities. If the computer is not good enough to replace paper, let the computer enhance paper. Since we rely on so many appliances to live our everyday life, let the computer help any one of them to be smarter. Mediaspaces and Augmented Reality are an exciting challenge for HCI. They push the limits of the technology we have, they require an even larger multi-disciplinary approach, they question our beliefs of what a computer (and computation) is, and they open a wide new design space for enhancing people with technology.

Acknowledgements The ideas presented in this article have been influenced by many people, among whom I would particularly like to acknowledge Bill Buxton, Bill Gaver, and Wendy Mackay. I want to add many additional thanks to Wendy Mackay for helping with the wording and the articulation of ideas in the paper.

References Baudel, T & Beaudouin-Lafon, M (1993), "Charade: Remote Control of Objets Using Free-Hand Gestures", Communications of the ACM 36 (7), pp.28-35. Bellcore (1993), "The Touring Machine System", Communications of the ACM 36(1), pp.68-77, Bellcore Information Networking Research Laboratory. Bly, S A, Harrison, S R & Irwin, S (1993), "MediaSpaces: Bringing People Together in a Video, Audio and Computing Environment", Communications of the ACM 36(1), pp.28-47. Buxton, W & Moran, T(1990), "EuroPARC's Integrated Interactive Intermedia Facility (IIIF): Early Experiences", in Multi-User Interfaces and Applications, S Gibbs & A A Verrijn-Stuart [eds.], NorthHolland, pp.11-34, Proceedings of IFIP WG8.4 Conference, Heraklion, Greece, September 1990. Dourish, P & Bly, S (1992), "Portholes: Supporting Awareness in a Distributed Work Group", in Proceedings ofCHI'92: Human Factors in Computing Systems, P Bauersfeld, J Bennett & G Lynch [eds.], ACM Press, pp.541-547. Fitzmaurice, G W (1993), "Situated Information Spaces and Spatially Aware Palmtop Computers", Communications of the ACM 36 (7), pp.39-49. Galloway, K & Rabinowitz, S (1980), "Hole-In-Space", Mobile image videotape. Gaver, W(1992), "The Affordances of Media Spaces for Collaboration", in Proceedings of CSCW92: Conference on Computer Supported Cooperative Work, J Turner & R Kraut [eds.], ACM Press, pp. 1724. Gaver, W, Moran, T, MacLean, A, LOvstrand, L, Dourish, P, Carter, K & Buxton, W (1992), "Realising a Video Environment: EuroPARC's RAVE System", in Proceedings ofCHI'92: Human Factors in Computing Systems, P Bauersfeld, J Bennett & G Lynch [eds.], ACM Press, pp.27-35.

18


Hinckley, K, Pausch, R, Goble, J C & Kassell, N F (1994), "Passive Real-World Interface Props for Neurosurgical Visualization", in Proceedings of CHI'94: Human Factors in Computing Systems, B Adelson, S Dumais & J Olson [eds.], ACM Press, pp.452-458. Hindus, D & Schmandt, C (1992), "Ubiquitous Audio: Capturing Spontaneous Collaboration", in Proceedings ofCSCW'92: Conference on Computer Supported Cooperative Work, J T\irner & R Kraut [eds.], ACM Press, pp.210-217. Ishii, H & Kobayashi, M (1992), "ClearBoard: A Seamless Medium for Shared Drawing and Conversation with Eye Contact", in Proceedings ofCHI'92: Human Factors in Computing Systems, P Bauersfeld, J Bennett & G Lynch [eds.], ACM Press, pp.525-532. Mackay, W, Velay, G, Carter, K, Ma, C & Pagani, D (1993), "Augmenting Reality: Adding Computational Dimensions to Paper", Communications of the ACM 36 (7), pp.96-97. Mackay, W E & Davenport, G (1989), "Virtual Video Editing in Interactive Multimedia Applications", Communications of the ACM 32(7), pp.802-810. Mantei, M M, Baecker, R M, Sellen, A J, Buxton, W A S & Milligan, T (1991), "Experiences in the Use of a Media Space", in Proceedings of CHI'91: Human Factors in Computing Systems (Reaching through Technology), S P Robertson, G M Olson & J S Olson [eds.], ACM Press, pp.203-207. Olson, M H & Bly, S A (1991), "The Portland Experience: A Report on a Distributed Research Group", International Journal of Man-Machine Studies 34, pp.211-228. Tang, J C & Rua, M (1994), "Montage: Providing Teleproximity for Distributed Groups", in Proceedings of CHI'94: Human Factors in Computing Systems, B Adelson, S Dumais & J Olson [eds.], ACM Press, pp.37-43. Ueda, H, Takafuni, M, Sumino, S & Nagasaka, A (1993), "Automatic Structure Visualisation for Video Editing", in Proceedings ofINTERCHI'93, S Ashlund, K Mullet, A Henderson, E Hollnagel & T White [eds.], ACM Press, pp.137-142. Wellner, P (1993), "Interacting with Paper on the DigitalDesk", Communications of the ACM 36 (7), pp.87-96. Wenders, W (1991), Until the End of the World, Time Warner, Movie.

Methodology of Interactive Systems Development

Transferring HCI Modelling and Design Techniques to Practitioners: A Framework and Empirical Work Simon Buckingham Shum & Nick Hammond Human-Computer Interaction Group, Department of Psychology, University of York, Heslington, York YO1 5DD, UK. Tel: +44 (0)904 433165 Fax: +44(0)904 433181 EMail: {sjbsl,

nvhl}@unix.york.ac.uk

The human-computer interaction (HCI) community is generating a large number of analytic approaches such as models of user cognition and user-centred design representations. However, their successful uptake by practitioners depends on how easily they can be understood, and how usable and useful they are. We present a framework which identifies four different 'gulfs' between HCI modelling and design techniques and their intended users. These gulfs are potential opportunities to support designers if techniques can be encapsulated in appropriate forms. Use of the gulfs framework is illustrated in relation to three very different strands of work: i. representing HCI design spaces and design rationale; ii. modelling user cognition; and iii. modelling interactive system behaviour. We summarise what is currently known about these gulfs, report empirical investigations showing how these gulfs can be 'bridged', and describe plans for further investigations. We conclude that it is desirable for practitioners' requirements to shape analytic approaches much earlier in their development than has been the case to date. The work reported in this paper illustrates some of the techniques which can be recruited to this end. Keywords: theory-based design, evaluation, design rationale, cognitive modelling, formal specification.

22

Simon Buckingham Shum & Nick Hammond

1. Introduction The human-computer interaction (HCI) community is generating a large number of analytic, usability-oriented approaches such as cognitive modelling and user-centred design representations. Three critical factors which will determine whether any of these approaches makes any impact on design practice are their intelligibility to practitioners, and their utility and usability. We propose that although most HCI models still have a long way to go in their development before being sufficiently mature to be employed in everyday design activity, it is not too early for model developers to sensitise themselves to the needs and resources of their target design communities. The AMODEUS-2 project (AMODEUS-2,1992) addresses this issue. Our goals are to develop, relate and communicate to designers a range of theoretical modelling and analytic techniques for HCI design. The approaches with which we are working model user interface architectures (Coutaz, 1987), user cognition (Blandford & Young, 1993; May, Barnard & Blandford, 1993) and user-system interaction states (Barnard & Harrison, 1989), formally specify interactive system properties, (Abowd & Dix, 1992; Harrison & Thimbleby, 1990) and represent the structure of HCI design rationales and design spaces (Bernsen, 1993; MacLean et al., 1991; Timpkaetal., 1993). There are two main thrusts in AMODEUS-2. The first is development of the individual modelling and design approaches, such as extending the scope and complexity of issues an approach can address and reducing the cost in time and effort to perform modelling. The second thrust is that of transferring the modelling approaches to design practitioners and evaluating their intelligibility, utility and usability. Developers of abstract design representations, such as those listed above, should be thinking from the start about the eventual users of their techniques, just as the HCI community urges designers to 'involve the user' from initial conception of the software artifact (Gugerty, 1993; de Haan, van der Veer & van Vliet, 1991). The following questions summarise our concerns, motivated by those expressed by designers at a modeller-designer workshop (Shum & Hammond, 1993): • How can we communicate what different models do? • How do different models relate to each other? • How do different models fit into the development process? • What process is involved in applying a modelling/analytic approach? • What knowledge and expertise is needed to make effective use of a modelling approach? • How useful is the modelling to development teams? This paper describes how we are investigating these issues. In the first section we present a framework for identifying opportunities for communicating analytical HCI techniques to designers. This centres around the notion of gulfs between researchers and practitioners which can be bridged by appropriate encapsulations of the process or product of a particular research approach. The gulfs are then instantiated in relation to three very different strands of work: an approach to representing HCI design spaces and design rationale, and approaches to modelling user cognition and interactive systems. We describe what is currently known about these gulfs, empirical studies which have been conducted showing how these gulfs may be 'bridged', and outline further investigations which are underway.

Transferring HCI Modelling and Design Techniques to Practitioners

23

design practice encapsulated modelling process/product designer encapsulation gulf modeller .modelling ' process j rproductj

HCI modelling approach /design framework Figure 1: A good encapsulation embodies elements of modelling process or product in a form which can be comprehended and utilised by designers.

2. The 'Gulf Crisis': What do Modellers and Designers Have to Talk About? At present very few designers use theoretical HCI modelling techniques in practice (Bellotti, 1988). HCI practitioners and researchers tend to have different goals (design vs. science), methods (finding good-enough vs. optimal solutions) and languages (concrete orientation vs. abstract orientation, e.g. artifacts and scenarios vs. issues and tasks). The conceptual tools which HCI analysts bring to bear on a problem are thus often very different from those which typical user interface designers in the field utilise (Barnard, 1991; Long & Dowell, 1989). If we believe that theory still has something to offer HCI design practice, albeit not based on a simplistic model which ignores the differences between design and science, then we must explore what we shall refer to as encapsulations of theoretically-based approaches which can be used by design teams. It is this research-practice gulf which our current work aims to narrow, in the context of communicating and assessing the contribution of HCI modelling to HCI designers. We use the term encapsulation (as a process) to mean: Re-expression of the representations with which HCI modellers work, in order that members of the design community can comprehend and apply the results of modelling, or go on to use the approaches themselves. Thus, an encapsulation is the product of the encapsulation process, and will comprise some subset of the modelling process or product, highlighting particular features of interest to designers. An encapsulation is meant to be used by designers (Figure 1). The encapsulation intended for use by designers varies considerably depending on the modelling approach. We now develop this idea in order to articulate what an encapsulation might offer, and to whom.

24

Simon Buckingham Shum & Nick Hammond targetted user of modelling approach

existing design practice: informal/formal analysis process using existing tools/methods

_^~

usability analysis/ design decisions

encapsulation gulfs

modeller

model-based analysis/ design recommendations

informal/formal modelling process

Figure 2: The structure of the encapsulation gulfs framework, showing two design processes: that of existing design practice, and that of HCI modellers. The potential for communication between the two parties is conceived as a set of dividing gulfs which can be bridged by appropriate encapsulations.

targetted

user of modelling approach

encapsulation gulfs

existing design practice: informal/formal analysis process using existing tools/methods V

X payback gulf

- how easy is it to encode information in terms of the approach's formalisms?

gulf model/ framework used by expert


problem description

- is it useful to reason with some of these formalisms even without using the approach fully?

-informal/formal modelling process

consultancy gulf is it useful?


Figure 3: Three key gulfs between designers and modellers. To make elements of the analytic process available to designers, the cost gulf and payback gulf must be bridged. To communicate the product of a modelling analysis, the consultancy gulf must be bridged.

2.1. Identifying the Gulfs to be Bridged If one imagines a designer and a modeller working on the same HCI problem, their analysis processes can be represented as two activity streams (Figure 2), separated by an encapsulation gulf which represents the potential for communication. As will be illustrated, the targeted user of modelling approach will vary depending on the nature of the modelling. This template is elaborated in Figures 3 and 4. Firstly, Figure 3 shows three key gulfs between modeller and designer, which will be instantiated differently for each modelling approach. The most obvious gulf to bridge is the consultancy gulf, so called because the modeller works in the role of an expert 'consultant' who delivers a detailed modelling analysis of the problem. This gulf asks if the results of modelling are useful, that is, intelligible, relevant, and applicable. However, depending on the approach, designers may also be able to make use of the "informal/formal modelling process" as shown in Figure 3. The cost gulf asks about the demands placed on designers in using the model's formalisms, and the payback gulf

Transferring HCI Modelling and Design Techniques to Practitioners targetted user of modelling approach

existing design practice: informal/formal analysis process using existing tools/methods

25


-V

- how much of the modelling approach does target user need to know to understand tool's questions?

encapsulation how intelligible are the constructs used by gulfs the tool to represent the problem?

automated modelling tool (if any)

- does simply using the tool assist design reasoning (i.e. apart from what it might output at the end of the consultation)?

is it useful?

\ description of problem to modelling tool

— modelling process


Figure 4: Some modelling approaches have automated modelling tools, which introduce a new gulf to be bridged.

about the potential benefits for design reasoning which this might provide, independent of the utility of the consultancy modelling analysis which the approach could provide. This pair of 'down-up' arrows captures the two-way communication or 'dialectic' which a good design representation should engender; the effort of encoding ideas within a formalism should aid in the formulation of, and reflection on, those ideas. Finally, for some approaches, the main modelling-designer interface is meant to be a software tool — e.g. an expert system, or a Programmable User Model (Young, Green & Simon, 1989). An additional modelling stream appears for these approaches therefore, to represent the gulfs associated with a modelling tool intended for use by the designer, as shown in Figure 4. The 'reflective process' described earlier in relation to Figure 3 can be understood most obviously in relation to design representations which the designer directly uses. However, as Figure 4 emphasises, the process of using a modelling tool may also direct reasoning in helpful ways.

2.2. The 'Prerequisite Gulf: Establishing Common Ground A gulf which has remained implicit thus far, but which is prerequisite to the investigation and successful bridging of the other gulfs, is the ease of communicating to designers the purpose and (at a general level) the underlying principles of a given approach. In the same way that two people cannot effectively work without establishing a shared understanding of relevant concepts, and respect for each other's ability, successfully bridging this gulf establishes sufficient common ground and trust that the designer is both willing and able to try using the approach (thus engaging other gulfs). Designers are unlikely to trust any of the other forms in which they encounter an approach if sufficient basic information has not been communicated. Added to the top of each gulf analysis, therefore, is the 'prerequisite background gulf. We are currently evaluating one way of encapsulating this kind of 'prerequisite knowledge' through the use of executive summaries and short worked examples (Buckingham Shum et al., 1994a). These are intended as background for designers who are collaborating with AMODEUS so that they can better understand the modelling analyses they receive, but also as introductory material for other interested parties. As we investigate encapsulation requirements further, we will gain an understanding of how much designers need to know about an approach, and in what forms this knowledge seems to be best presented.

26

Simon Buckingham Shum & Nick Hammond

2.3. Organisational Factors So far, 'design' has been treated within the framework as though it did not exist within an organisational context. The gulfs analysis suggest that an encapsulation will be 'good' if it is representationally sufficient (expressing the modelling in appropriate scope and detail), and cognitively compatible (e.g. making important information salient). Critical as these requirements are, they are not sufficient. To understand more fully the context in which a model will be used, we need also to ask questions such as How compatible is a modelling encapsulation with current representations and practice? and What levels (individual! grouplproject!organisation) will the introduction of a particular encapsulation impact? Within our own work, we have found that rather than merely widening the gulfs, these factors may prevent them from ever being investigated, if modelling encapsulations are clearly incompatible with the design teams' existing practices, or introduce excessive overheads. Such considerations add weight to the importance of shaping modelling approaches — even as they are being developed — to the characteristics of the environments in which they will be used. 3. Using the Framework for Motivating Empirical Studies The gulfs framework motivates several different kinds of investigation, depending on what is being offered to designers by an approach. Two clear categories of gulf can be identified, depending on whether modelling process or product is being encapsulated. These gulfs are being investigated in different ways: • Process gulfs (cost/payback gulfs) relate to the question: are there any aspects of the modelling process which practitioners might find useful? Encapsulations which fall into this category are evaluated by detailed study of their usability and utility. Examples of these which will be illustrated below are a design rationale representation scheme, and a cognitive modelling expert system. • Product gulfs (consultancy gulf) relate to the question: is the modelling analysis comprehensible and useful to designers? Encapsulations which fall into this category are being evaluated by documenting designers' feedback on modelling of HCI problems they are facing. Examples of these which will be illustrated below are cognitive modelling, and formal specification. For each approach, the associated gulf analysis will be presented. This overview will then be elaborated by examining each gulf in turn — what do we currently know about the obstacles to successful transfer and use which gulf X represents, and how can they be overcome? 4. Design Space Analysis and Design Rationale A Design Rationale (DR) is a representation of the reasoning behind design decisions. The challenge of finding ways to represent DR which are both usable and useful to its creators and other designers has created much interest recently within the HCI community — e.g. (Carroll & Moran, 1991; Herbsleb & Kuwana, 1993; Lee, 1990). The particular DR approach with which we are working is Design Space Analysis using the semiformal QOC (Questions/Options/Criteria) notation (MacLean et al., 1991). A Design Space Analysis (DSA) consists of a network of Questions identifying key design issues, Options providing possible answers to each Question, and Criteria for assessing and comparing the Options. Figure 5 shows the graphical, semiformal QOC notation used for structuring DSA.

Transferring HCI Modelling and Design Techniques to Practitioners

27

Argument Argument

Arqument

\ Option • ( b : = 2 ) .

73

Manuals as Structured Programs

[ a. Flowgraph drawn abstractly.

Stopped

j

b. Flowgraph drawn as part of the diagram of a tape recorder.

Figure 3: Two representation of a flowgraph.

The following Pascal, with numbered lines, can be represented as a flowgraph (see Figure 3): while n< 10 do n :=n + 1 ; write(n);

(1) (2) (3)

As suggested by Figure 3, the same flowgraph can also be used for the following simplified extract from a user manual, where we have used the word 'goto' to show how the user can move from paragraph to paragraph as the manual is read ('executed'). In many ways, this is merely a compiled while command, and is further support to the argument that users follow programs, very low level programs at that, which happen to be written as manuals. Play:

press Pause to pause the tape goto Pause

(1)

press Stop to stop playing the tape goto Stop Pause:

Stop:

press Pause to resume playing the tape goto Play

(2)

(3)

Any edge (u, v) in a flowgraph can be replaced by a flowgraph (G, u, v): this is a composition operation called nesting, and is how more complex programs are constructed, given the appropriate building blocks. Sequential composition (in Pascal, using a semicolon) is simply

74

Mark Addison & Harold Thimbleby

nesting on the flowgraph P 2 , which is •—>•—>•. For a given set of control structures, it is sufficient to consider nesting on vertices with one out edge (since the vertex uniquely specifies the edge to be replaced); however this entails treating as distinct each form of various program control structures (such as the case statement and guarded commands, which require vertices of arbitrary out-degree). Given a flowgraph, a problem called decomposition is to determine how it might have been constructed from smaller, nested sub-flowgraphs. It is easy to prove that the maximal decomposition is unique. To measure program complexity, a program is decomposed (using standard algorithms: see below) into the maximal set of flowgraphs. Various metrics are then possible: for example the smaller the components (or the greater the number of them for a given size of program), the more structured the program. The metrics can be normalised by the size of the program to obtain complexity densities. One can also immediately establish what control structures have been used, and easily indicate any that are non-standard.

5. An Experiment As an experiment, we took the finite state machine description of a submachine of a domestic video cassette recorder. This description can equally be viewed as a manual fully documenting the machine (Thimbleby, 1993). We took the off-with-tape-out state as start and end states (as described above) and hence obtained flowgraphs. We applied a flowgraph decomposition algorithm (Fuchs, 1993; Lengauer & Tarjan, 1979) to the graphs. The somewhat surprising result was that there was no proper decomposition. (A proper decomposition is a decomposition into at least two non-empty sub-flowgraphs.) Moreover, there was no proper decomposition no matter what state was taken as the start/stop state. A more impressive way of saying this is that one might have saved a human manual writer from the impossibly tedious — and in this case, fruitless — job of searching for a well structured manual. Not one of the billions of possible distinct manuals that might have to have been examined by hand (by manual methods?) can be considered structured. If we take manuals as isomorphic to directed spanning trees, the number above is a count of the distinct labelled directed spanning trees rooted at the start vertex of the relevant flowgraph. For our small example system the number is over 10 million for each root. Given that many everyday systems are far larger, the space of their manuals is astronomical, and automatic means of optimising manual design are necessary. Of any group of paragraphs in any manual, more than one must refer to paragraphs outside the group, or elsewhere other paragraphs must refer to more than one inside the group. These correspond to gotos out of the group or gotos into it, not counting the start and end of the group. If the manuals were to be made more verbose or more extensive by documenting states as sequences of paragraphs (instead of one paragraph per state, or one paragraph per state x button, as assumed above), then the flowgraph decomposition would at once decompose these as Pi subgraphs (i.e. strictly linear flowgraphs with no branches); the quotient graph would then not be further decomposable. That no such Pi were found in the original case


75

shows that, in user interface terms, there were no states in which one could only next do one thing and that one thing could be done no other way. What the results mean, especially give that one can modify the system to be documented in various ways (e.g. by not writing a manual for all of it, as we assumed here), will be considered more fully below.

6. Discussion If a large computer program was found to have no proper flowgraph decomposition, one would justifiably complain to the programmer that it was unstructured; it is likely that the programmer could not easily explain what the program did; it is likely that nobody else could understand the program at all easily; and it is likely that the computer anyway does something other than what the programmer intended. The implications of unstructured program writing are serious. If the manuals for the VCRs we studied were written by humans, and we view them as 'programs' for their users to achieve the various functions the VCRs implement, then there is a very strong possibility that those writers did not understand what they had done, or, rather, they would not know what a user would do with what they had done. A programmer may 'know' what he or she has programmed or intended to program, but they may be surprised at what the computer does: this is why debugging programs is so difficult. Likewise, users may not apply the manual instructions as intended (or any aspect of the design as intended): this is why user interfaces need to be 'debugged' in real tests with users. What they had done was likely to be incorrect in details, and, further, that they would be unable to explain what they had done. Even if by some chance the manual was correct, it is extremely unlikely that anyone else would be able to maintain it. All this is the programming experience. In user interface terms, additionally it means that there would be little reason to suppose a user could understand the manual either. (There is little debate about whether computers understand their programs!) The structured programming experience is that trained programmers have difficulty reading and comprehending badly structured programs: how much harder would users (untrained as programmers) find unstructured manuals? A crucial point is the following: If there is no proper flowgraph decomposition, then a reader of the manual cannot use divide and conquer to help understand the manual. No part of the manual can be considered in isolation (i.e. 'dividing' the manual results in components that are incorrect or incomplete). A solution to the correctness problem of manuals is to construct the manual automatically. This solution has been proposed and demonstrated by Thimbleby (1993), though this does not directly address the problem of its complexity. (Arguably it exacerbates it, since it guarantees arbitrarily complex manuals are correct.) Since the decomposition of a manual into flowgraphs is unique and independent of its syntactical form, its complexity (in this regard) can only be improved by changing the system. Thus one has to use a system development environment where such complexity measures can be established early enough in the design cycle so that they can impact the design itself; this was also proposed by Thimbleby (1993).

76


6.1. Limitations of Flowgraphs Flowgraphs are just one of many ways of approaching the manual/program analogy. They have limitations for both software engineering and manual analysis. An important feature, however, is that the limitations are very precise. Flowgraphs not only provide an approach for measuring complexity but also provide a demarcation for the factors being measured. Flowgraph based analysis will be particularly relevant when the user is following a completely documented procedure, as in error recovery, fault diagnosis or servicing applications. In applications where the manual is intended to lead to understanding of system principles, other approaches would seem more appropriate. Flowgraphs are not specially appropriate for declarative manuals (ones that declare properties or invariants of the user interface, as opposed to specific procedures). Although there is no flowgraph decomposition in the example studied, one might respond that if certain edges were deleted then a structured manual could be obtained. For example, the Operate button switches the VCR off [sic] in many (but not all) states. Deleting edges to off would reduce the number of gotos out of many states. In fact, one can search for the 'best' edges to delete; and the manual would have to discuss these separately, as well as any exceptions (e.g. Operate does something different when the VCR is off). What is most interesting, however, is that a purely automatic procedure (flowgraph decomposition) has identified task-related issues, in turn which would raise various design trade-offs. Orthogonal components are common in user interfaces (e.g. a TV sound on/off component is usually orthogonal to the TV channel control), but flowgraph decomposition only considers nesting, not Cartesian product. (One can consider flowgraph decomposition of orthogonal components: essentially applying flowgraph metrics to independent parts of the manual separately.) Undo causes similar problems; indeed undo is not handled very well by finite state machines — push down automata are better models. 6.2. Aren't All Programs Equivalent to Structured

Programs?

Bohm & Jacopini (1966) showed that any (conventional imperative) computer program can be converted to [what we might now call] a structured program using a limited number of simple control structures. In their original terms, they showed that however complex a flowchart or Turing Machine program, it can always be converted to a program using a few (specifically: 3 by their first result, or 2 by their second) simple control structures. Their result was an important one in convincing the programming community that there was no program that had to be written in an unstructured form. It is sometimes forgotten that their result showed more precisely that any program can be converted to a structured program perhaps only by way of introducing state variables. For example any while loop from which you want to do an exit in the middle can be converted to a well-structured while loop, with a body that is a simple conditional (see Figure 4), by introducing a state variable. In manual-writing terms, this means that any manual can be converted to a 'structured manual' but perhaps only at the expense of introducing one or more variables. One is thereby shifting the complexity of the manual 'sflow structure to the cognitive load of remembering the states

Manuals as Structured Programs while test do begin dol ; if stop then goto cheat; do2 end; cheat:

77 flag := true; while test and flag do begin dol ; if stop then flag :- false else do2 end

Figure 4: Using state variables to permit structured programming. The program fragments shown are equivalent (assuming flag is not used elsewhere!), but the one on the left is notflowgraphdecomposable.

of the respective variables. It may be that a few variables (7±2!) are an acceptable tradeoff, though it seems more sensible for the system to display the state of the variables at the appropriate places by using indicators — though this approach would exacerbate the standard problem of system/manual synchronisation. As Dijkstra (1968) noted for the Bohm & Jacopini result, a translation of a bad structure by their mechanical method to one that avoids gotos does not necessarily achieve a program that is any clearer. We are not suggesting our approach helps write or generate better manuals; it merely measures something that indicates whether a manual may be badly written. That programs can be well-written without gotos is now a matter of history, and, as with manuals, it is essentially a stylistic rather than a theoretical issue.

7. Conclusions We have suggested that the design of (and evaluation of) user manuals may be approached using methods derived from software engineering. This approach suggests various approaches and metrics, which can be taken directly from software engineering. Moreover, the arguments that justify programming metrics can be applied with equal force in the user manual case. Program complexity measures correspond (amongst other features) to the likelihood of correctness and intelligibility of manuals, both of which one would usually wish to optimise simultaneously! It follows that manuals may be improved by such methods; designers and manual writers can now easily question the trade-offs involved in unstructured features of a user interface. The appropriateness or otherwise of a particular metric will depend on the user's tasks and what should be optimised by the design. Specifically, this paper considered flowgraph decomposition, which is only one of many plausible complexity metrics. From software engineering, they are known to be a good measure of complexity from the writer's and maintainer's points of view. An important advantage of flowgraph-based complexity measures is that they can be automated. Just as some computer programs are best written in an unstructured way (for example, because efficiency is paramount, or because they are microprogrammed and use some unconventional ideas of control flow), it is certainly not the case that structured methods are a universal panacea. Some user interfaces will benefit from structured manuals, some won't. Games, teaching systems (CAL, CBT) and high security systems are common examples of systems where the user is supposed not to understand everything; on the other hand, safety-critical systems, office systems and many consumer products are generally supposed to be ones that

78


are easy and reliable to use under the sorts of assumptions that our approach makes. Users should therefore have structured manuals, well structured in the software engineering sense. In this paper we have provided some initial ideas on how that may be achieved.

Acknowledgements This research work has been conducted as part of an SERC-supported project "Systems, manuals, usability and graph theory" Grant No.GR/J43110.

Appendix The following Mathematica (Wolfram, 1991) definition is the submachine of the JVC HRD540EK VCR used as an example in the paper. Its interpretation is as follows: States are numbered from 1. In state s t a t e L a b e l s [ [s] ] , pressing button b u t t o n L a b e l s [ [b] ] changes state to t r a n s i t i o n s [ [ s , b ] ], or does nothing if this is zero. {buttonLabels -> ( "Play", "Operate", "Forward", "Rewind", "Pause", "Record", "Stop/Eject", "Tape In"), stateLabels -> { "Fast Forward", "Off Tape In", "Off Tape Out", "On Tape In" "On Tape Out", "Play Forward", "Play Pause", "Play Rewind", "Play Tape", "record 030", "record 060", "record 090", "record 120", "record 150", "record 180", "record 210", "record 240", "Record Pause", "Record Pause 030", "Record Pause 060", "Record Pause 090", "Record Pause 120", "Record Pause 150", "Record Pause 180", "Record Pause 210", "Record Pause 240", "Record Tape", "Rewind Tape"), transitions -> { (0, 2, 0, 0, 0, 0, 4, 0), (0, 4, 0, 0, 0, 0, 3, 0), {0, 5, 0, 0, 0, 0, 0, 4), (9, 2, 1, 28, 0, 27, 5, 0), (0, 3, 0, 0, 0, 0, 0, 4), (9, 2, 0, 0, 7, 0, 4, 0), (9, 2, 1, 28, 0, 0, 4, 0), {9, 2, 0, 0, 7, 0, 4, 0), {0, 2, 6, 8, 7, 0, 4, 0 ) , {0, 2, 0, 0, 19, 11, 4, 0), {0, 2, 0, 0, 20, 12, 4, 0), {0, 2, 0, 0, 21, 13, 4, 0), (0, 2, 0, 0, 22, 14, 4, 0), [0, 2, 0, 0, 23, 15, 4, 0), {0, 2, 0, 0, 24, 16, 4, 0), {0, 2, 0, 0, 25, 17, 4, 0), (0, 2, 0, 0, 26, 27, 4, 0), (27, 2, 0, 0, 0, 19, 4, 0), (10, 2, 0, 0, 0, 20, 4, 0), (11, 2, 0, 0, 0, 21, 4, 0), (12, 2, 0, 0, 0, 22, 4, 0), (13, 2, 0, 0, 0, 23, 4, 0), (14, 2, 0, 0, 0, 24, 4, 0), (15, 2, 0, 0, 0, 25, 4, 0), (16, 2, 0, 0, 0, 26, 4, 0), (17, 2, 0, 0, 0, 18, 4, 0), (0, 2, 0, 0, 18, 10, 4, 0), (0, 2, 0, 0, 0, 0, 4, 0))


79

Note. This transition table was determined by experiment from the device, and not derived from the manual. We have included it here for several reasons: it is actually very tedious to determine; it is good practice to include experimental data; we may have made mistakes. This last point is most interesting: of course, we don't think we've made a mistake — the specification behaves, so far as we can tell, in a 'reasonable' way. But it may be incorrect. This is a central problem of user interface design: we think we have a user interface design (here, a simulation of a VCR), we have tested it, yet we may still be wrong. This serious problem arises in our case because manufacturers do not specify VCRs formally.

References Bohm, C & Jacopini, G (1966), "Flow Diagrams, Turing Machines and Languages with Only Two Formation Rules", Communications of the ACM 9 (5), pp.366-371. Carroll, J M (1990), The Nurnberg Funnel: Designing Minimalist Instruction for Practical Computer Skill, MIT Press. Dijkstra, E W (1968), "Go To Statement Considered Harmful", Communications of the ACM 11 (3), pp.147148. Draper, S W & Oatley, K (1992), "Action Centred Manuals or Minimalist Instruction? Alternative Theories for Carroll's Minimal Manuals", in Computers and Writing, State of the Art, P O'B Holt & N Williams [eds.], Intellect Press, pp.222-243. Fenton, N E & Whitty, R W (1986), "Axiomatic approach to Software Metrication through Program Decomposition", The Computer Journal 29(4), pp.330-339. Fuchs, N (1993), "An Improved Algorithm for Deriving the Decomposition Tree of a Flowgraph", Structured Programming 14 (3), pp.93—101. Jackson, M A (1983), System Development, Prentice-Hall. Lengauer, T & Tarjan, R E (1979), "A Fast Algorithm Finding Dominators in a Flowgraph", ACM Transactions on Programming Languages and Systems 1 (1), pp.121—141. Macromedia (1992), Authorware Professional Tutorial, Macromedia Inc, California, USA. Payne, S J (1991), "Display-based action at the user interface", International Journal of Man-Machine Studies 35 (3), pp.275-289. Runciman, C & Hammond, N V (1986), "User Programs: A Way to Match Computer Systems and Human Cognition", in People and Computers: Designing for Usability, M D Harrison & A Monk [eds.], Cambridge University Press, pp.464-481, Proceedings of HCI'86. Thimbleby, H W (1990), User Interface Design, Addison Wesley. Thimbleby, H W (1993), "Combining Systems and Manuals", in People and Computers VIII, J L Alty, D Diaper & S Guest [eds.], Cambridge University Press, pp.479-488, Proceedings of HCI'93. Wolfram, S (1991), Mathematica (2nd edition), Addison Wesley.

Improving Education through Computer-Based Alternative Assessment Methods Jody Paul Department of Computer Science and Engineering, University of Colorado at Denver, Campus Box 109, Denver, Colorado 802173364, USA. Tel: +1 (303) 556 8425 Fax: +1 (303) 556 8369 EMail: [email protected] Assessment contributes to the educational process of students but only a small fraction of the full potential is typically realized. The primary impediment to realizing greater benefit is the infeasibility of implementing more effective alternatives in the resource-limited settings typical of modern educational environments. We are developing a system architecture that exploits hypermedia technology to overcome serious limitations of traditional assessment methods. The architecture addresses the design of cost-effective confidence-measuring and performance-testing assessment vehicles using hypermedia-based studentsystem interaction. In this paper we describe the conceptual foundation, its embodiment in prototypes, and preliminary results from classroom tests. Keywords: educational assessment, hypermedia, computer-assisted education.

1. Introduction The educational experience can be enhanced by using assessment methods as techniques for evaluation and as guides for instructors and administrators in curriculum design and teaching methods (Airasian, 1991). Unfortunately, standardized assessment methods do not discriminate between finer-grained states of knowledge nor do they adequately reflect the ability of students to apply what they've learned. In addition, since the assessment instrument significantly influences instruction, alternative assessment methods are needed to better address fundamental educational goals. Past attempts to address these problems and goals on a large scale using traditional technology have proven infeasible primarily due to the high costs of providing adequate, standardized materials and controlled, responsive

82

JodyPaul

environments. In this paper we present alternatives that exploit the characteristics of modern hypermedia-capable computer systems to achieve the desired goals in a cost-effective way. 2. Assessment Direct contributions of assessment include: establishing and revealing status ('knowing what you know'), diagnosis of weaknesses ('knowing what you don't know'), comparative assessment with larger populations (answering' where do I stand?'), assimilation of knowledge into internal cognitive frameworks ('pulling it all together'), and the exercise of higher-order cognitive abilities (such as application of knowledge in alternate contexts and synthesis of discrete concepts). Indirect contributions, typically viewed in the context of instructors and administrators, include establishing and revealing students' knowledge states for problem diagnosis (of individual students, instructors, institutions, curricula, or instructional methods), grading, certification, and value-added measurement (such as required for accreditation and funding). Standardization, necessary to meet these objectives, has traditionally been at odds with the goals of accurately revealing finer granularity of knowledge state. For example, multiplechoice tests, the most widely used method, "can mask a large variety of states of knowledge and can introduce guessing on the part of the student." (Sibley, 1974) Further, such tests are of limited value in determining students' higher-order cognitive skills, such as the ability to apply what they have learned in different problem-solving contexts (Priestley, 1982; Sweezey, 1981). Resnick & Resnick (1992) report that the nature of standardized tests now in use is 'fundamentally incompatible' with the goal of improving students' higher-order abilities and that alternative assessment methods are needed. Although traditional assessment methods fail to adequately achieve the desired objectives (Kellaghan, Madaus & Airasian, 1982), alternatives that use traditional assessment techniques have prohibitive resource requirements. For example, achieving timely, customized feedback requires an impractical student-teacher ratio. Likewise, providing all students with complete, controlled laboratory environments is often costly (e.g. expensive instruments or materials), sometimes dangerous (e.g. hazardous materials or dangerous procedures), or otherwise infeasible (e.g. hypothetical or extra-terrestrial environments). Another significant aspect of assessment is that it drives instruction ('teachers teach to the test'). This has a subtle but profound impact: assessment methods that do not involve higherorder cognitive abilities, such as problem solving, do not encourage teachers to emphasize those abilities. Evidence suggests that because standard assessment methods do not address these levels of cognition, such abilities are neglected in general (Resnick & Resnick, 1992; Shepard, 1992) and that there exists the need for a "reformulation of assessment to help not hinder the effort to teach thinking." (Shepard, 1992) 3. Hypermedia Technology and Assessment Alternatives The use of hypermedia-capable computer systems enables more effective ways to achieve the desired goals, even in resource-limited educational settings. A well-integrated hypermedia

Improving

Education

...

83

support-base provides interactive multimedia capability coupled with the ability to link and navigate through domain and pedagogical information. Appropriate use of multimedia extends the involvement of the student (bringing additional senses into play and employing both 'left-' and 'right-brain' faculties), facilitates confidence-measuring protocols, and provides practicable performance assessment (evaluating students' ability to solve problems that include the performance of particular tasks or procedures (Priestley, 1982)). These modes make it easier for students to use the system, enhance the presentation of problem context, and support the simulated environments used for performance assessment. Our investigation of the feasibility of developing such alternative assessment tools blends techniques from education, artificial intelligence and human-computer interaction (Paul, 1993; Paul, 1994). We combine essentials of assessment and performance testing (Airasian, 1991; Dworkin & Dworkin, 1988; Freedle, 1990; Gifford & O'Connor, 1992; Kellaghan, Madaus & Airasian, 1982; Priestley, 1982; Sternberg, 1992; Sweezey, 1981) with student modeling (Murray & Woolf, 1991; Nwana, 1990; Wood & Holt, 1990) and principles of interface design (Grosz, 1978; Laurel, 1990; McKeown, 1983; McKeown, Wish & Mathews, 1985; Perry & Voelcker, 1990; Tognazzini, 1991). We have also adopted the goal of developing an architecture that applies to a large variety of subject areas, thereby providing additional economic advantage since the same physical systems can be used for several application areas.

4. Confidence Measuring The goal of confidence measuring assessment is to more accurately measure students' true knowledge states. Typical multiple-choice examinations require students to respond with what amounts to a probability distribution restricted to 0 and 1 values such as the response shown in brackets in the following example: Which of the following should be performed after making a minor modification to a module of a working system? A. Top-down testing B. Regression testing C. Ad-hoc testing By restricting responses to 0 or 1, we lose the ability to discriminate between states of knowledge such as "I strongly believe B to be correct", "I believe C to be incorrect but can't distinguish between A and B" and "From what I know, each alternative seems equally likely to be correct". Any scheme allowing students a wider range of responses must also encourage students to report their true knowledge states. Admissible scoring systems (also known as reproducing or proper scoring systems) are those that encourage students to respond with exactly their perceptions of the probability distribution. (Brown, 1970; Brown & Shuford, 1973; Sibley, 1974) We have adopted the following 3-alternative logarithmic admissible scoring system that depends only on the probability Px ascribed to the correct answer X, where n and k are normalization and range constants: Score if A is correct = n + k log(3 PA) Score if B is correct = n + k log(3 Score if C is correct = n + k log(3

84

JodyPaul

Figure 1: Confidence-measuring response template.

Brown & Shuford (1973) demonstrated that people who know they are to be rewarded according to this type of scheme are encouraged report the probabilities they believe in 'rather than shading them one way or the other to exploit the scoring system.' In essence, any response which varies from a student's true belief requires placing bets that the student considers unrewarding or failing to place bets the student considers rewarding. That is, this scheme is admissible and is one of the 'scoring systems which encourage honesty.' (Brown & Shuford, 1973) The confidence-measuring prototypes address university-level software engineering education. We have constructed prototypes that use both textual and multimedia vehicles (including sound and action sequences) to present problem context and feedback. These were implemented on the Macintosh™ platform using HyperCard™ as the integration substrate and QuickTime™ for real-time audio-visual presentation. Our basic assessment model centers on the use of 3-alternative questions and the 16-region response template shown in Figure 1. Proximity to a vertex corresponds to the degree of belief that the answer indicated by that vertex's letter is correct. The regions are colored for easy identification and, as the student moves the pointer over different regions, graphical and textual feedback provides the interpretation of that region. The decision to use 16 regions is based on initial testing with alternatives ranging from 4 regions to screen pixel resolution (about 4500 regions). The 16-region template provides sufficient information for useful discrimination among students' knowledge states, avoids artifacts such as students' motor-skills in trying to fine-tune positioning of the pointer, and has intuitive correspondence between the visual regions and their interpretation. The decision to use 3-option multiple-choice items follows the evidence reported in educational and psychological literature indicating greater efficacy of using 3 options rather than a higher or lower number of alternatives (Costin, 1972; Lord, 1977; Owen & Froman, 1987; Trevisan, Sax & Michael, 1991). The interpretation of each selectable region is shown in Figure 2. The values shown indicate the strength of belief that the correct answer is 'A', i.e. PA. Values are analogous with respect

Improving Education ...

85

Figure 2: Confidence-measuring response template, showing strength of belief PA associated with each region.

Figure 3: Confidence-measuring response template, showing values awarded for each selectable region in the case where the single correct answer is 'A'.

to each vertex for answers of ' B ' or ' C . Distance from a vertex is directly proportional to the degree of belief that the answer corresponding to that vertex is incorrect. The values awarded for each selectable region in the case where the single correct answer is 'A' are shown in Figure 3. Scoring is analogous with respect to each vertex for correct answers of ' B ' or ' C . We reinforce the 'feel' for this scoring system by giving students dynamic feedback about their selections and scores. The information collected using this technique can be used in many ways. A simple interpretation of the averaged scores provides a rough indication of students knowledge state. A realism function (Brown & Shuford, 1973) or external validity graph (Sibley, 1974) can be used to detect how aware a student is of his/her own knowledge state. That is, we can determine

86

Jody Paul

if the student appears to overvalue or undervalue his/her information and disclose the loss in score attributable to this bias in assignment of probabilities (also called 'lack of realism' or 'labeling error') and the loss attributable to lack of information about the subject matter. A particular pattern of responses may also be detected, providing additional information about the knowledge state of the student. This collection of information can be used to generate appropriate feedback, including navigation through problems if standardization is not required. We are currently investigating the use of case-based reasoning to help diagnose students' difficulties and provide customized corrective and directive advice.

4.1. Preliminary Experiments We collected data during two sessions of student usage of one of our confidence measuring prototypes. The subjects for one session were six graduate students currently enrolled in a software engineering course, the subjects for the other were twelve undergraduate students who had taken a software engineering course the previous semester and were currently enrolled in a software project course. Students were asked to fill out a pre-experience questionnaire, then were given a five-minute introductory lecture concerning the basics of confidencemeasuring assessment and how to use the computerized system. They then practiced with a sample exam that consisted of six questions of general trivia knowledge. Once they had completed the sample exam, they took the actual exam which contained twenty-seven questions covering a wide range of software engineering topics. After answering the final question, subjects received feedback on their aggregate score, the score they would have received if the test had been graded by conventional multiple-choice criteria, and how often they apparently undervalued or overvalued their knowledge. Finally, subjects were asked to complete a post-experience questionnaire and were encouraged to discuss their experiences. Sixteen subjects (89%) preferred the interactive confidence measuring method to standard multiple-choice exams. The predominant reasons for their preferences were (1) the ability to demonstrate partial knowledge and get credit even when some uncertainty remains; and (2) the test was perceived as 'more accurate' in showing what the test-taker really knows. The same number of students found the system easy to understand and easy to use. Eleven subjects (61 %) felt that the test was more accurate than standard multiple-choice and six (33%) felt that it took less time. Fifteen (83%) were comfortable with both the number of regions (16) and the number of alternatives (3). Two students (11 %) indicated that they would prefer fewer regions, while one student (5.5%) indicated a preference for more regions. The prototype did not provide immediate feedback during the exam as to the correct answer nor did it allow students to go back to earlier questions. Several students commented on these limitations and suggested that one or the other of these two features should be included. A few students also mentioned that they would have liked to have had more time to practice with the system. Since this type of response and scoring system is not yet typically encountered by students, we must ensure that students develop appropriate intuitions regarding the characteristics of this type of assessment. That is, we want students to understand that 'gaming' the test corresponds to faithfully reporting their true beliefs. Our prototypes attempt to address this issue by including training modes and using continuous graphical and textual feedback. Based on the comments from our test groups, we are currently working on improving both the training aspect and interface feedback in an attempt to shorten the time required for students to 'come up to speed' in using the system.

87

Improving Education ... Computer-Based Performance Assessment in Chemistry, Qualitative Analysis

This is a solution containing one of the 3 ions Ag\ Pb2+, and Hg22+. Using the qualitative analysis scheme diagrammed at right, determine which one.

is :•£•:

You have 10 ml of solution to work with.

add reagent to solution heat solution separate solution wash precipitate test pH put test tubs in wash put test tube in rack retrieve test tube from rack Cancel R»viev Instructions

Rtvitw Chjrt

N»xt Operation

Figure 4: Sample task-performance environment — Chemistry (1).

At this time we are analyzing the collected data to identify trends or biases with respect to realism functions. We are also using the data to assist the design of a case-based reasoning component to help diagnose students' difficulties and provide customized corrective and directive advice.

5. Performance Testing An added dimension to automated assessment provided by the architecture concerns the ability to do performance assessment, in which the student is given a task to perform in a 'virtual world' provided by the system. For example, where physical observables must be simulated (e.g. an interview with an end-user or a chemical reaction) a video sequence may be displayed. The use of such simulation is indicated when actual performance tests are impractical due to cost, danger, the serious consequences of mistakes, or the impossibility of arranging actual performance situations. By simulating performance conditions the system controls most of the variables in the testing situation and we can standardize the assessment across students and administrations (Priestley, 1982). Our performance assessment prototypes address chemistry and software engineering domains. Figures 4 and 5 show snapshots of task-performance displays and interaction from the chemistry version. Figure 6 shows a snapshot from the software-engineering version. In the chemistry prototype, students carry out qualitative analysis to determine which ions are present in an unknown solution. This problem-solving task is suitable for high-school or first-year college chemistry laboratory assessment. In the software engineering prototype, the student observes a customer explaining his concerns, then develops a top-level data-flowdiagram (DFD) using a palette of available objects and diagramming tools.

Jody Paul Computer-Based Performance Assessment in Chemistry. Qualitative Analysis

U j Original j

[Heated |

|w/HCl |

OHctiue

ORctiue

® Hctiue

R»vi*v Instructions

Next Operation

Select "Next Operation" to continue Select "Done" to identify solution ( Done

Figure 5: Sample task-performance environment — Chemistry (2).

A key issue that arises when attempting to assess the task-performance of a student concerns what to measure. We could choose to look only at the result achieved, but that information may not be sufficient. For example, it may be possible to achieve the result by inappropriate means, such as using an insertion sort when a bubble sort is requested. Alternatively, we could look at the specific command sequence issued by the student. In this case, there may be an infinite variety of functionally equivalent sequences. Our approach, instead, is to look at intermediate-goal satisfaction, which presumes a set of necessary sub-goals that must be satisfied to achieve the major goal.

6. Observations We are developing an architecture for educational assessment that overcomes serious limitations of current assessment techniques. Blending hypermedia with techniques from education, artificial intelligence and human-computer interaction, the architecture will provide instructors with automated assessment tools that respond dynamically to students' actions, indicate students' confidence in their answers, discriminate among students' knowledge states, and provide the ability to assess problem-solving behavior in task-performance contexts. Although we have focused primarily on software engineering and chemistry education, we believe that the methods and techniques developed for and employed in the prototypes are not confined to these domains of application and we are investigating application to other curricula. Such flexibility allows the same machines to be used for several different educational areas at different times. Coupled with the simulated task-environment aspect of the architecture, this provides the potential for institutions to realize significant savings in an era where costeffectiveness has become increasingly important.

Improving Education . . .

89

Requirements Collection: Customer Interview & Context Diagram

user

"A c

Staff Rtport

Task Description

Figure 6: Sample task-performance environment — Software Engineering.

Acknowledgements Special thanks to Patricia Supon and Sue Lewis for their contributions to this effort. References Airasian, P W (1991), Classroom Assessment, McGraw Hill. Brown, T A (1970), "Probabilistic Forecasts and Reproducing Scoring Systems", RAND, RM-6299-ARPA. Brown, T A & Shuford, E H (1973), "Quantifying Uncertainty into Numerical Probabilities for the Reporting of Intelligence", RAND, R-l 185-ARPA. Costin, F (1972), "Three-Choice Versus Four-Choice Items: Implications for Reliability and Validity of Objective Achievement Tests", Educational and Psychological Measurement 32, pp.1035-1038. Dworkin, A & Dworkin, N (1988), Problem Solving Assessment, Academic Therapy Publications. Freedle, R [ed.] (1990), Artificial Intelligence and the Future of Testing, Lawrence Erlbaum Associates. Gifford, B R & O'Connor, M C [eds.] (1992), Changing Assessments: Alternative Views of Aptitude, Achievement and Instruction, Kluwer (Academic Press). Grosz, B J (1978), "Discourse Analysis", in Understanding Spoken Language, D E Walker [ed.], North-Holland, pp.235-268. Kellaghan, T, Madaus, G F & Airasian, P W (1982), The Effects ofStandardizedTesting. Evaluation in Education and Human Services, Kluwer (Academic Press). Laurel, B [ed.] (1990), The Art of Human-Computer Interface Design, Addison Wesley.

90

Jody Paul

Lord, F M (1977), "Optimal Number of Choices Per Item — A Comparison of Four Approaches", Journal of Educational Measurement 14(1), pp.33-38. McKeown, K (1983), "User-Oriented Explanation for Expert Systems", Department of Computer Science, Columbia University, CUCS-85-83. McKeown, K R, Wish, M & Mathews, K (1985), "Tailoring Explanations for the User", Department of Computer Science, Columbia University, COCS-172-85. Murray, T & Woolf, B P (1991), "A Knowledge Acquisition Tool for Intelligent Computer Tutors", ACM SIGART Bulletin 2 (2), pp.9-21. Nwana, H S (1990), "Intelligent Tutoring Systems: An Overview", Artificial Intelligence Review 4(4), pp.251— 277. Owen, S V & Froman, R D (1987), "What's Wrong with Three-Option Multiple Choice Items?", Educational and Psychological Measurement 47, pp.513-522. Paul, J (1993), "Hypermedia-based Interactive Student-Assessment System (HISAS): Concept and Architecture", in Educational Multimedia and Hypermedia Annual 1993, H Maurer [ed.], Association for the Advancement of Computing in Education, pp.415-421. Paul, J (1994), "Alternative Assessment for Software Engineering Education", in Software Engineering Education, J L Diaz-Herrera [ed.], Springer-Verlag, pp.463-472. Perry, T S & Voelcker, J (1990), "Of Mice and Menus: Designing the User-Friendly Interface", IEEE Spectrum 26(9),pp.46-51. Priestley, M (1982), Performance Assessment in Education and Training: Alternative Techniques, Educational Technology Publications. Resnick, L B & Resnick, D P (1992), "Assessing the Thinking Curriculum: New Tools for Educational Reform", in Changing Assessments: Alternative Views of Aptitude Achievement and Instruction, B R Gifford & M C O'Connor [eds.], Kluwer (Academic Press), pp.37-75. Shepard, L A (1992), "What Policy Makers Who Mandate Tests Should Know About the New Psychology of Intellectual Ability and Learning", in Changing Assessments: Alternative Views ofAptitude Achievement and Instruction, B R Gifford & M C O'Connor [eds.], Kluwer (Academic Press), pp.301-328. Sibley, W L (1974), "A Prototype Computer Program for Interactive Computer-Administered Admissible Probability Measurement", RAND, R-1258-ARPA. Sternberg, R J (1992), "CAT: A Program of Comprehensive Abilities Testing", in Changing Assessments: Alternative Views of Aptitude Achievement and Instruction, B R Gifford & M C O'Connor [eds.], Kluwer (Academic Press), pp.213-274. Sweezey, R W (1981), Individual Performance Assessment: An Approach to Criterion-Referenced Test Development, Reston Publishing Company Inc. Tognazzini, B (1991), Tog on Interface, Addison Wesley. Trevisan, M S, Sax, G & Michael, W B (1991), "The Effects of the Number of Options Per Item and Student Ability on Test Validity and Reliability", Educational and Psychological Measurement 51 (4), pp.829837. Wood, P H & Holt, P D (1990), "Intelligent Tutoring Systems: An Annotated Bibliography", ACM SIGART Bulletin 1 (1), pp.21-42.

Visual Programming in a Visual Domain: A Case Study of Cognitive Dimensions Francesmary Modugno*, T R G Green* & Brad A Myers* t Carnegie Mellon University, 5000 Forbes Ave, Pittsburgh, PA 15213, USA. EMail: {fmm, bam}@cs.cmu.edu ^MRC Applied Psychology Unit, 15 Chaucer Road, Cambridge CB2 2EF, UK. EMail: [email protected] We present a new visual programming language and environment that serves as a form of feedback and representation in a Programming by Demonstration system. The language differs from existing visual languages because it explicitly represents data objects and implicitly represents operations by changes in data objects. The system was designed to provide non-programmers with programming support for common, repetitive tasks and incorporates some principles of cognition to assist these users in learning to use it. With this in mind, we analyzed the language and its editor along cognitive dimensions. The assessment provided insight into both strengths and weaknesses of the system, prompting a number of design changes. This demonstrates how useful such an analysis can be. Keywords: cognitive dimensions, end-user programming, programming by demonstration, visual language, visual shell, Pursuit.

1. Introduction A visual shell (or desktop) is a direct manipulation interface to a file system. Examples include the Apple Macintosh desktop and the Xerox Star. Although such systems are easy to use, most do not support end-user programming. Pursuit is a visual shell aimed at providing programming capabilities in a way that is consistent with the direct manipulation paradigm. To enable users to construct programs, Pursuit contains a Programming by Demonstration (PBD) system (Cypher, 1993). In a PBD system, users execute actions on real data and

92

Francesmary Modugno, TRG Green & Brad A Myers

the underlying system attempts to construct a program (Myers, 1991). Such systems have limitations: feedback is often difficult to understand, disruptive or non-existent; and programs often have no representation for users to examine or edit. Pursuit addresses these problems by presenting the evolving program in a visual language while it is being constructed. Unlike other visual languages, which explicitly represent operations and leave users to imagine data in their heads, Pursuit's visual language explicitly represents data objects using icons and implicitly represents operations by the visible changes to data icons. There are three properties we believe will make it possible for novices and non-programmers to learn to construct and recognize Pursuit programs. First, programs are specified by executing real actions on real data. The Pursuit PBD system extends the direct manipulation paradigm to enable users to specify programs in much the same way that they invoke operations — through direct manipulation. Second, programs are represented in a visual language in which the data and operations of a program look very much like the actual objects and changes users see on the desktop when constructing the program. Hopefully, this will make Pursuit programs look more familiar than programs written in a textual language or in a visual language that does not reflect the interface. Finally, during a demonstration, the program appears incrementally as the user executes each operation. In this way, users learn interactively how data and operations (i.e. program syntax) are represented in Pursuit. This paper reports our experience using cognitive dimensions (Green, 1989; Green, 1991) to determine how well Pursuit meets its design criteria, and gives a designer's perspective on how useful such an analysis is. The assessment provided support for several design decisions and revealed potential weak areas of the system, prompting a number of system changes.

2. Related Work on Visual Shells There have been several approaches to adding end-user programming to visual shells. Some visual shells contain a macro recorder — e.g. SmallStar (Halbert, 1984); QuicKeys2, MacroMaker and Tempo II for the Macintosh; and HP NewWave — that makes a transcript of user actions that can be replayed later. Although effective in automating simple, repetitive tasks, macro recorders are limited because they record exactly what users do—only the object that is pointed to can be a parameter and the transcript consists of a straight-line sequence of commands. To generalize transcripts, some macro recorders produce a representation of the transcript in a textual programming language for users to edit. However, this requires users to understand a programming language that is significantly different from the desktop and does not take advantage of the visual aspects of the interface. Some visual shells have invented a special graphical programming language (Borg, 1990; Haeberli, 1988) to enable users to write programs. Most of these languages are based on the data flow model, in which icons represent utilities and lines connecting them represent data paths. Unfortunately, most contain no way to depict abstractions or control structures. The types of programs users can write are quite limited. In addition, these languages require users to learn a special programming language whose syntax differs significantly from what they see in the interface. Finally, constructing programs by wiring together objects is quite different from the way users ordinarily interact with the system.

Visual Programming in a Visual Domain

93

3. Motivation For Pursuit's Approach The goal of Pursuit is to add end-user programming into a visual shell in a consistent way. This section briefly reviews some motivation and history for our design decisions. 3.1. Extending Programming

to

Non-Programmers

We set out to create a visual shell that would enable users to access the underlying power of the computer to help with their tasks, without requiring that they learn complex programming skills, or many special 'little languages' or commands. We were particularly interested in providing this power to people who use computers frequently, but who might not want to learn how to program (we refer to these users as 'non-programmers'). We began by studying the login files of and shell scripts written by computer scientists at Carnegie Mellon and by informally interviewing some non-programmers, such as secretaries and administrators. Our goal was to determine the types of programs users wrote and the types of tasks nonprogrammers do that could be automated. Our studies showed that most shell scripts (including those written by programmers) were very simple, repetitively executing a few commands over a set of files. Often, the files were related in a simple way, such as all being . t e x files or all being edited on a certain day. The informal discussions revealed that many of the repetitive tasks that non-programmers do, such as backing up files, were similar in form to the shell scripts. 3.2. Incorporating

Principles of Cognition

Our goal became to design a system that would simplify this type of programming. Therefore, we created a language that emphasizes the manipulation of sets of objects related in some specific way and that minimizes the use of explicit control constructs such as loops and conditionals, since novice and non-programmers often have difficulty with them (Doane, Pellegrino & Klatzky, 1990). In addition, as we describe throughout the paper, the language and editor incorporate some of the same principles of cognition that have made spreadsheets successful (Lewis & Olson, 1987): familiar, concrete representations; immediate feedback; suppressing the inner world of variables and computation; and automatic consistency maintenance. We feel that this will help users learn to understand and use the language. 3.3. Providing Editable, Visual Feedback Finally, we designed the language to serve as the main form of feedback between the PBD system and the user. Pursuit contains an inference mechanism1 that can detect loops over sets of data, branches on exit code conditions, and common substring patterns (Modugno, in preparation). Since all inference mechanisms will sometimes be wrong, it is important that users know what the system has inferred. Having a good representation of the program during the demonstration gives users full knowledge of the system's inferences at all times in an unobtrusive way. Users can verify these inferences and help guide the PBD system to which features of the examples should be generalized. There are other forms of feedback we could have chosen: dialog boxes (Halbert, 1984); questions and answers (Maulsby & Witten, 1989; Myers, 1988); textual representation of the code (Lieberman, 1982) changing the appearance of actual interface objects (e.g. anticipation highlighting (Cypher, 1991)); animation (Finzer & Gould, 1984); and sound (Lieberman, 1993). However, our approach has several benefits 1

This research does not focus on improving inferencing. It focuses on other limitations of PBD systems: feedback, representation and editing. The techniques described here are independent of the inference mechanism.

94

Francesmary Modugno, T RG Green & Brad A Myers

file.tex

Figure 1: The main data types in Pursuit: A) a file; B) a set of files; C) a folder; D) a set of folders.

rename report

report

talk

Figure 2: The representation of the operation rename paper talk. The first panel shows the icon representing the file paper located in the report folder before it is renamed. The second panel shows the same file after the rename operation. Notice that the icon's name has changed. This change represents the rename operation.

over these other forms. Unlike dialog boxes and the question and answer style, it is not disruptive, since the user does not need to respond to it. Unlike programs represented in a textual language, it does not require the user to learn a language that is very different from the interface. Finally, unlike anticipation highlighting, animation and sound, there is a static representation for users to examine and edit.

4. Pursuit's Visual Representation Language Visual shells are easy to use because of the constantly visible, concrete, familiar representations of data objects and the illusion of concretely manipulating these objects. Unfortunately, this 'conceptual simplicity' is often lost when programming is introduced: users interact with the system visually, but usually program it off-line in a textual programming language. Users must develop two very different bodies of knowledge: one for interacting with the system and one for programming it. Pursuit attempts to bridge this gap. By allowing programs to be specified by demonstration and by representing programs in a visual programming language that reflects the desktop, users can apply knowledge of the interface and its objects to the visual language and its objects when constructing, viewing and editing a program.

95


papers

papers

papers

|

d a t e - TODAY|

A

B

Figure 3: A) A visual declaration binding the set to be all the . t e x files in the p a p e r s folder that were edited today. B) The copy operation.

The Pursuit visual language2 combines elements of the comic strip metaphor (Kurlander & Feiner, 1988) and the visual production system (Furnas, 1991). Familiar icons are used to represent data objects, such as files and folders. Sets are represented by overlaying two icons of the same type (see Figure 1). Two panels are used to represent an operation. The prologue shows the data objects before the operation and the epilogue shows the data objects after (see Figure 2). A program is a series of operation panels concatenated together, along with representations for loops, conditionals, variables and parameters. Because two panels per operation result in long, space inefficient programs, Pursuit contains space saving heuristics that combine knowledge of the domain with information about operations. These and other features of Pursuit are illustrated in the following examples.

4.1. Example 1 This example illustrates how to write a program to backup all the . t e x files in the p a p e r s folder that were edited today. To backup the files, the user copies them to the b a c k u p s folder and then compresses the copies. To create a program to automate this task, the user demonstrates the actions on a particular set of files. During the demonstration, the underlying PBD system constructs a program. Figures 3-5 show the developing program during the demonstration. Figure 3A is a visual declaration. It appears when the user executes the c o p y operation, and defines the scope of the set variable. The icon on the right represents the set of all . t e x files in the p a p e r s folder that were edited today. The icon on the left is the icon used in the program to represent this set. The string "date = TODAY" is an attribute. It constrains the set to those files edited today. Attributes allow for abstract sets of objects and indicate the PBD system's inferences. In addition, users can directly edit them to specify properties of data objects or to fix incorrect inferences. Attribute strings can be simple arithmetic expressions defining a single value or a range or values (e.g. "256 < size < 1024") and can contain variables, system constants such as "TODAY" or "USER", and references to attributes of other objects. Our current focus is not on designing the most visually appealing language. Instead, we are exploring the utility of this particular language paradigm for non-programmers. Hence, our visual representations appear primitive. However, we recognize that the visual presentation will necessarily influence the acceptance of our language, and thus deem it important to explore that avenue. Currently, however, we do not have the resources to invest in graphic design.

96

Francesmary Modugno, T RG Green & Brad A Myers copy papers

papers

move backups

Figure 4: After the user drags (moves) the copies to the backups folder, the third panel appears. Notice that in the program the set of copies icon has moved from the p a p e r s folder to the backups folder, reflecting the changes the user has seen in the actual interface when the real copies were moved.

copy

move;compress

Figure 5: The completed program. The compress operation is represented by the difference in the height and the name of the icon for the copies in the second and third panels. This difference is similar to the change in appearance of the icons for the real files that the user would see in the actual interface, where the compress operation replaces a file's icon with a shorter icon and appends a ".Z" to its name. The shadow beneath the third panel indicates that it represents multiple operations. Clicking on it reveals the individual operation panels.

Attributes and sets minimize the need for loops, conditionals and variables. For example, to define the above set in a traditional programming language, one would have to write code to loop through all the files in the p a p e r s folder and test to see which ones had names ending in . t e x and were modified today. Attributes and sets make this looping and testing implicit, thus hiding some inner computations from users. Figure 3B shows the first two panels as they appear after the user opens the p a p e r s folder, selects the files to be copied and copies them. After the user moves the copies to backup folder, the new panel in Figure 4 appears depicting the move. Only one panel is added because Pursuit notices that the epilogue of the copy contains the prologue of the move operation. Determining when to combine the prologue of an operation with the epilogue of the previous operation is an example of a Pursuit space saving heuristic.

97


n

jit>Ut>,U-« I

Figure 6: The two cases used to demonstrate a conditional program. The first two panels show a successful execution of the copy operation, demonstrated on the a b s t r . t e x file. The remainder of the program shows the unsuccessful execution, demonstrated on the b i b l i o . t e x file, and corrective actions the user wants the program to take in that case. Between the second and third panels is a gap, indicating that the user is manipulating a different data object. The black square on the prologue of the second copy operation (Panel 3) indicates that the operation failed. The predicate following explains why — the existence of a file with the name c o p y - o f - b i b l i o . t e x . The dialog box icon indicates that the operation popped up a dialog box to the user. Clicking on the icon, pops up the dialog box displayed when the operation failed. To correct the operation, the user deletes the error causing file (Panels 4 and 5) and re-executes the copy operation (Panels 6 and 7).

Finally, the user selects all the copies and compresses them. Figure 5 shows the completed program. Another heuristic determines when several operations can be represented in a single panel. The shadow beneath the third panel of Figure 5 indicates that it contains both the move and compress operations. Clicking on it reveals the individual panels for the two operations. Figure 5 also illustrates an advantage of having icons represent data: icons minimize the use of explicit variables, and remove a level of indirection that variables introduce. To identify an icon in a program, Pursuit assigns it a unique color. Although an icon's appearance may change throughout the program, its color remains the same. For example, in the second panel the icon representing the output of the copy operation has the name "copy-of . t e x " and is in the p a p e r s folder. In the third panel, the same set has the name "copy-of - . t e x . Z" and is in the backups folder. Users can tell that the two icons represent the same set because they have the same color. Color serves the same purpose as a variable name in textual programming languages. 4.2. Example 2 This example illustrates how Pursuit automatically creates conditionals and loops. When an operation fails, Pursuit cannot construct an epilogue panel. Instead, it creates a conditional marker (e.g. the black square on the right side of the third panel in Figure 6) and a branch connector with an annotation (or predicate) stating the condition for that branch. In this example, the copy operation failed because a file with the required output name already existed, so the annotation is 'exists' plus a named file icon. To build a conditional program, the user demonstrates the program's actions on two examples — one in which the operation succeeds, and one in which it fails. When the user begins to demonstrate the program on a third example, Pursuit recognizes a loop. It asks the user to verify the two loop iterations by highlighting the panels in Figure 6. It thenfinishesexecuting the loop and updates the program. The updated program (shown in Figure 7) is an example of an explicit loop containing an explicit conditional.

98


I .t«....li

I 'nfatimiuJLaitt

exists , copy-of tex

I

copy-of-I ,tex I

Figure 7: The Pursuit program that copies each * . t e x file in the p a p e r s folder. If the copy operation fails because of the existence of a file with the output file name, the program deletes that old output file, and re-executes the copy operation. Users can see the other possible outcomes of the copy operation by clicking on the conditional marker.

The panel below the visual declaration (FOREACH . . . IN . . . ) states that the loop executes over all the files in the declaration set. The operations in the body of the loop are surrounded by the large loop rectangle. The conditional marker on the right edge of the prologue of the c o p y operation indicates that the program branches at this point. The first branch (labeled "no errors") is taken when the c o p y operation executes successfully. The lower branch is taken when the c o p y operation fails because a file with the output file name ( c o p y - o f < n l > . t e x in this case) already exists.

5. The Pursuit Editor Pursuit contains a visual language editor so that users can fix incorrect inferences, add or delete operations, change attributes, add loops and conditionals, select and name parameters, etc. The editor is similar to a direct manipulation text editor. Data objects are selected by clicking on them, and operations are selected by clicking and dragging the mouse across their panels. Once an object is chosen, appropriate editing commands appear in the edit menu (see Figure 8). For example, operations can be cut or copied into the cut buffer and pasted into another section of the program, or they can be wrapped in a loop. File and folder objects can be edited to add, remove or change attributes, or to make them into parameters. To help maintain consistency, edits are immediately propagated throughout the program. For example, if the user changes the name of a file set, all instances of the set and any members

99

Visual Programming in a Visual Domain EDIT cut copy

copy: move.

paste

papers

|

papers

compress |

backups

backups

|_

backups

property sheet

1

add branch continue demonstration add stop point

I

paper.tex

insert into loop

copy-of-I

loaoer :ex Z 1

make explicit loop undo

Figure 8: The user has selected a sequence of panels (indicated here by the outer gray rectangle) to wrap in a loop.

backups

nm'in I

|

backups

|

l:::\:\, I

Figure 9: Adding a user defined branch. The upper branch shows the path to take when the file p a p e r , t e x in the backups folder is older than the file p a p e r , t e x in the p a p e r s folder. The user is adding another predicate to the branch construct by selecting from the menu of predicate templates. The menu choices provide templates for the user to further edit in order to construct a predicate.

of it are immediately updated. If an operation that produced an output file is deleted, then all subsequent operations that involve that output file are highlighted and the user is informed that deleting the operation can lead to an invalid program. Users can also select a point in the program and add operations either by copying them from another point in the program or by demonstrating them. Operations can not be drawn from scratch. User defined branches (similar to the Lisp cond construct) can also be added to the program by inserting a branch template and constructing the predicates via the predicate menus (Figure 9). The types of predicates users can construct represent some of the common predicates found in our informal study of shell scripts, as well as more general predicates we as designers and users felt were important to have in a language. After editing the program, it can be saved. Users indicate parameters by clicking on those objects in the program that represent the actual parameters. For example, clicking on the p a p e r s folder in Figure 7 indicates that the folder over which the program executes is a parameter to the program. Once saved, a program can be executed by indicating its arguments and selecting the program from the menu of user defined programs. Programs can also be edited and re-saved, or deleted.

100

Francesmary Modugno, T RG Green & Brad A Myers Dimension

Informal Definition

Viscosity

resistance to change

Hidden Dependencies

important links between entities are not visible

Visibility and Side-by-Side-ability

ability to view components easily

Diffuseness/Terseness

succinctness of language

Imposed Guess-Ahead

constraints on the order of doing things

Closeness of Mapping

closeness of representation to domain

Progressive Evaluation

effort required to meet a goal

Hard Mental Operations

operations that place a high demand on working memory

Secondary Notation

extra information in means other than program syntax

Abstraction Gradient

types and availability of abstraction mechanisms

Role-Expressiveness

the purpose of a program component is readily inferred

Consistency

similar semantics are expressed in similar syntactic forms

Figure 10: The 12 cognitive dimensions identified by Green & Petre (submitted for publication).

6. Cognitive Dimensions of Pursuit Cognitive dimensions (Green, 1989; Green, 1991) of an information artifact provide a framework for a broad-brush assessment of a system's form and structure. By analyzing a system along each dimension, the framework provides insight into the cognitively important aspects of the system's notation and interaction style, and could reveal overlooked usability issues. In understanding the cognitive dimension framework, it is important to understand that any programming system is composed of its notational structure and its support environment. By notation we mean the the symbols that the user sees and manipulates. By environment we mean the mechanism available to manipulate these symbols. Cognitive dimensions apply to the entire system because the way the user interacts with the system is determined by both the notation and environment for manipulating that notation. One aspect of Pursuit that makes it an interesting case study for cognitive dimensions is that unlike all previous systems studied in which programs are statically defined with a text or visual language editor, Pursuit's PBD system and incrementally evolving program provide a highly interactive and dynamic environment. The processes of programming, testing and debugging are intertwined. Green & Petre (submitted for publication) list 12 cognitive dimensions (see Figure 10) and apply them to a set of contrasting programming languages. We considered Pursuit in light of these dimensions to see how far it was from a region of the design space suitable for its intended purposes. In some cases it scored well; in other cases it scored badly, prompting changes to the system's design.


101

6.1. History of the Analysis Before presenting our analysis, it is beneficial to explain how the analysis was done. The first author (and designer/implementer of Pursuit) began by reading Green (1989), Green (1991) and an early version of Green & Petre (submitted for publication). She then spent a day thinking about how each dimension applied to some typical programs users might create with Pursuit. Finally, she spent the next few days writing an early version this paper, and sent it via electronic mail to the second author. The second author and his colleague David Hendry, both experts in cognitive dimension, perused paper descriptions of Pursuit and the Pursuit video (Modugno & Myers, 1994). After watching the video continuously through, they ran it again, stopping frequently to ask questions such as "How can the user change that bit of program?" and "How does the user know what cases will be needed to set up a demonstration?". They then went over the list of dimensions, asking for each one how it applied to the programming exercise they had watched. Because they were already experienced in this activity, they needed very little time over and above the time taken to read the papers and watch the video. The second author agreed with the analysis done by the first author, and added only the closeness of mapping analysis. Our experience shows that cognitive dimensions are a powerful, easy to use tool. The first author was able to do a detailed assessment in less than a day, without ever having used the technique before. Furthermore, the technique provides both experts (the second author) and novices (the first author) the ability to examine an artifact, and proceed quickly to a high level (e-mail!) discussion of it. We attribute this to the compact shared vocabulary of cognitive dimensions not found in other evaluation techniques. Finally, the technique provided several insights to the designer, even though she had been working with the system for over two years! 6.2. The Analysis To save space, we present only the highlights of our analysis, practically unchanged from the original version. 6.2.1. Viscosity The Pursuit visual editor makes it relatively simple to cut, copy and paste operations in the program; to add branch and conditional constructs; to add and delete parameters; and to modify the attributes of objects. However, because the main method of program specification is by demonstration, to add new operations to a program users are forced to place "stop points" and re-execute the program on another piece of data. This can place a heavy burden on users who have to place the stop points in the correct position and insure that the state of the desktop is such that the program will execute and follow the desired path. A way to avoid this problem would be to expand the visual editor to include a method to construct an operation's representation. For example, the editor could contain a menu of all system operations. The user could select a particular operation, have a template of its prologue and epilogue appear, and edit the template to contain the correct data objects. 6.2.2. Hidden Dependencies There are two dependencies in Pursuit: between data objects and between operations. Data object dependencies define a relationship between two objects based on some shared attribute, which is most often the objects' names. For example, in the program in Figure 7 the output file and the predicate file (i.e. the files named c o p y - o f - < n l > . t e x ) both are derived

102


from (depend on) the loop input file (named < n l > . t e x ) . Operation dependencies define a relationship between two operations based on some shared data object. For example, the move and c o m p r e s s operations in Figure 5 depend on the c o p y operation. Both these dependencies can lead to problems when editing programs, since editing the dependency causing objects or operations can affect the dependent objects or operations. For example, suppose the user changes the name attribute of the set of files in Figure 7 to be all . mss files. The loop, output and predicate files must also be updated. Similarly, if the user deletes the c o p y operation in Figure 5, then the move and c o m p r e s s operations would become invalid. To avoid dependency problems, Pursuit contains two features. The first is the automatic propagation of attribute edits. Whenever the user edits an object, all dependent objects are automatically updated. The second feature is automatic notification of invalid operations. Whenever the user deletes a dependency causing operation, Pursuit highlights the dependent operations in the program and asks if these operations should also be deleted so that users can see how deleting a single operation affects the entire program. While these features address the problems of editing dependency causing objects and operations, they do not make users aware of these dependencies until an editing action is taken. A mechanism that shows the dependencies (for example, by highlighting the dependent objects or operations) of an object or operation could help users see the possible effects of their actions before they do anything and could decrease mental load when programming. Consider, for example, the current Pursuit editor and the mental burden on the user in the following scenario. Imagine a fairly long program in which the output of a c o p y operation is not used until several operations later, and the distance between the two operations is such that both are not visible in the program window at the same time. In this case, the dependency between the operations is not visible at any one time. Users must rely on their memory to identify the dependency when scrolling through the program. Similarly, if the user changes the name of the output file so it no longer contains the copied file name string, then the dependency between the output and input files can only be discovered by tracing the output file icon backwards through (possibly several) operation panels. In both cases, dependency tracking mechanism would simplify the user's search and memory requirements.

6.2.3. Visibility, Side-by-Side-ability and Diffuseness/Terseness The entire Pursuit program can be viewed in a scrollable window, making all of the program readily accessible. However, the portion of a program that can be viewed at any one time is limited to the width of the program window. Space saving heuristics, such as combining several operations into one panel, increase the amount of the program that is visible at any one time, and the ability to reveal the individual operations of a composite panel insures that the entire program can be viewed at the level of granularity of individual operations. In addition, the ability to pop up a data object's property sheet by clicking on any of its icons in the program allows users to view readily information and attributes of the object. Finally, the use of color to uniquely identify an icon makes it easier to identify and locate data objects when scanning a program. There are many ways to improve visibility. One way is to allow multiple views of the program so that users can simultaneously view semantically related but distant parts. To simplify accessing information about a data object, it would be helpful to be able to display the object's declaration in a separate window. This could reveal dependencies, such as


103

subsets of file sets, that are not evident in a property sheet. Finally, allowing complete program structures, such as loops and paths of a branch, to be collapsed into a single icon as well as adding the ability of users to select groups of panels to be collapsed into icons would increase the portion of the program visible at any one time, and could provide a global overview of a program's structure.

6.2.4. Imposed Guess-ahead Pursuit imposes certain order constraints on the programmer. A component is a sequence of operations that may contain data dependencies. During a single demonstration, a component can only be developed top-down. This is by nature of the demonstration specification method. However, between components that contain no dependencies with each other, there are no constraints. To remove constraints between components that contain data dependencies, there are two requirements. First, the user must be able to arrange the state of the desktop so that each component can be successfully demonstrated. Second, the editor must be augmented to allow for two different data objects (i.e. two icons of different colors) to be made into the same data object. For example, the user can demonstrate moving and compressing a file as a single component. Then the user can demonstrate copying a file as another component. To make the two components into the single program of copying a file and moving and compressing the output, the user needs a way to indicate that the icon representing the input to the move operation is the same as the output icon for the c o p y operation. In this way, the user can demonstrate pieces of the program in any order and then paste them together into a correct program without having to be constrained by the ordering of operations. This would make programming in Pursuit more amenable to the "top-down with deviation" programming process exhibited in other end-user programming domains (Davies, 1991; Visser, 1990). It is interesting to note that the demonstrational specification technique can both decrease and increase the look-ahead necessary to write a program. Demonstrating a program on existing data objects, without considering all possible problems the program may encounter, decreases look-ahead. The user simply interacts with the system in the usual way. However, if the user desires to construct a program so that it "always" works, then, like all programmers, they must be able to consider all possible data conditions that the program may have to deal with. In the PBD model, the state of the system to be arranged so that the demonstration will encounter these situations. Such a look-ahead requirement is burdensome. To decrease this burden, Pursuit incorporates two features: exit branch exposition and incomplete path exposition. Exit branch exposition allows users to view all possible outcomes of a particular operation by clicking on a conditional marker. For example, clicking on the conditional marker in Figure 7 displays the predicates for the remaining exit branches of the operation. The user then demonstrates what the program should do in each case. Incomplete path exposition displays the predicate for a particular exit branch of an operation when that branch is encountered during execution. When an operation of an executing program has an outcome not included in the program, Pursuit displays the program, adds the undemonstrated branch, highlights it, and asks the user what to do: abort the program execution; abort the execution of this data object; or allow the program to be augmented by the user demonstrating the new path the program should take. Both exit branch exposition and incomplete path exposition decrease the look-ahead requirement imposed by the PBD specification technique because the user is no longer forced

104

Francesmary Modugno, T R G Green & Brad A Myers

to think of all possible paths the program can take. Instead, the user can demonstrate the program in the current state and then explore ways to augment the program by examining each operation's exit branches. Furthermore, the program need not be edited immediately. Instead, the user can do so any time the program is run and a forgotten path is encountered. In addition to the constraints imposed by the demonstrational specification technique, the Pursuit programming model of sets and set manipulation also imposes some constraints. Consider the example (Section 4.1) in which the user copies, moves and compresses all the . t e x files in a folder. Suppose that one of the files originally copied was a b s t r . t e x and that the b a c k u p s folder contains a file with the name c o p y - o f - a b s t r . t e x . Z.Then the c o m p r e s s operation will fail on that set member. Thus, the users 'plan' to manipulate the set of files in order to construct the program was incorrect. To successfully demonstrate this program, the user must examine the state of the system and notice the problem causing file. Then she must demonstrate the program with two examples — one in which the c o m p r e s s operation succeeded and one in which it failed — so that Pursuit could infer the explicit loop (similar to the example in Section 4.2). This places a large look-ahead constraint on the user, since for very long programs involving multiple operations in multiple folders, the user would have to carefully inspect the system's state, remembering various data states and operation outputs throughout. Such a search would most likely exceed working memory capacity. A similar problem arises when the user demonstrates a set of operations on a file set and afterwards realizes that the set selection criteria cannot be expressed via the set attributes, but must be explicitly tested for in a user-defined branch. Imagine the frustration as the user exclaims "Darn, I should have used only a single file!". That is, to have correctly demonstrated the program, the user would have had to demonstrate it on a single data object, then edit the program to add the branch and wrap it in a loop. This requires that the user completely understand beforehand how to express selection criteria. To address these two problems, Pursuit needs a mechanism to convert a sequence of set operations to an explicit loop containing the operations. The loop's iteration set would be the data set for the original operations. Such a mechanism should automatically infer an explicit loop whenever an operation applied to a set has different outcomes for different set members, and should be available for users to invoke whenever they need to make a sequence of set operations into an explicit loop. In this way, users are less constrained to examine the system state or to fully know how to express certain selection criteria before a demonstration.

6.2.5. Closeness of Mapping In most language designs a close fit is thought desirable, because novices are expected already to know the domain. We believe that in shell programming novices will not know the domain and that it would be better if Pursuit hid some of the more idiosyncratic features. Pursuit is necessarily driven by some domain requirements, such as the various possible outcomes of operations, but it is mildly abstraction-tolerant: in Figure 5 the third panel probably conforms closer to the user's conceptual structure than the multiple operations that it encloses. For example, the user probably believes that when compressing a file the output is still the same file, only its contents are compressed. In reality, compressing a file produces an entirely different file whose contents are the compressed form of the original file's contents, and removes the original file from the system.


105

6.2.6. Progressive Evaluation In Pursuit, the program is constructed while it is being executed, so that the user can immediately see its effects. Even when editing or revising a program, users add operations by demonstrating actions so that they can see the results of the program. Incomplete programs are easy to execute — indeed programs as they are being constructed are incomplete programs executing. Thus, the programming process provides (almost) immediate means of evaluating progress and seeing results quickly, making for high gratification.

7. Related Work on Evaluation Techniques There has been much work on evaluating designs for interfaces; Green & Petre (submitted for publication) compare cognitive dimensions (CDs) with several other techniques. The nearest in spirit to CDs appears to be Heuristic Evaluation (HE) (Nielsen & Molich, 1990), which, like CDs, attempts to substitute a very small number of general principles for the huge volumes of detailed guidelines in common use. The 9 heuristics of HE, such as "use simple and natural dialogue", "speak the user's language", "minimize user memory load", etc., are to be applied by a small team, carefully considering all aspects of an interface and searching for problems. While both HE and CDs use the same processes (sit and think) there are many notable differences: i. The focus of HE is on interacting with an application, whereas CDs focus on users building an information structure (e.g. a program, a musical score, or even a longdistance telephone number). ii. HE is best used by HCI specialists, but CDs are designed to make sense to ordinary "choosers and users". As noted above, in this instance CDs were usefully applied by the first author, a computer scientist with no specialist HCI background. iii. Like other evaluation techniques, HE focuses on surface details, thus restricting its applicability; In contrast, CDs can be used to assess the information structure of a design long before any coding takes place. iv. Heuristics like "be consistent" are likely to be useful only at a low level (e.g. screens A and B are different), whereas "look for imposed guess-ahead" deals with structural problems of a very different order. v. But the most telling difference is that the 9 heuristics of HE are simply based on experience, whereas the CDs are based on a model of cognitive processes in design; in consequence: a. the CDs address problems that are overlooked by the HE — e.g. the need to rebuild information structures is a part of the design process, now recognized to be at least partly opportunistic; b. the terms of the CDs can in principle be grounded in analyses of information structure or in psychological theories of, say, parsing; and c. the CDs address the problems of trade-offs between choices, such as what happens if viscosity is reduced by adding abstractions.

106

Francesmary Modugno, T R G Green & Brad A Myers

8. Status and Future Work A prototype of Pursuit has been implemented using Garnet (Myers et al., 1990). This prototype has been used to evaluate the Pursuit design along the cognitive dimensions. Doing so has already revealed several ways to improve the design. Using the prototype, we have also done some informal user studies, which have provided important feedback to improve the system. For example, Pursuit initially contained a heuristic that sometimes eliminated the prologue of the first operation of a program. Since several people had difficulty understanding program in which this heuristic was applied, we eliminated it. Further work is needed to refine the heuristics for generating attributes and operation panels and to provide ways for making programs more concise. In addition, user studies are planned to evaluate the visual language itself as well as the entire Pursuit system to determine how well it helps users automate tasks. In these studies, users will construct programs for some tasks using Pursuit. These results will be compared with users doing the same task in the Pursuit visual shell but whose program representation language is an "English-like" textual language similar to the one found in SmallStar. This will help us evaluate whether or not the visual representation really does help simplify the programming process. We also plan to compare the recognizability of programs in both the Pursuit visual language and the "English-like" textual language. This would enable us to evaluate how easy it will be to write a program by modifying an existing one, to identify a program long after it is written, or to share programs between users. Indeed, if a system like Pursuit becomes popular, one can imagine that few programs will be written from scratch, but will instead be copied and modified from existing programs. The ability to recognize and tailor programs would thus become very important.

9. Conclusion Pursuit is a visual shell designed to provide much of the common programming power currently missing in visual shells in a way that is consistent with the direct manipulation paradigm. By combining the techniques of Programming by Demonstration with an editable, visual representation of programs, users can create abstract programs containing variables, loops, and conditionals. The goal is to enable users to access the underlying power of the computer by interacting with it the way they normally do — by executing real actions on real data objects — thus reducing the 'programming' skills they need to learn. The aim of the cognitive dimensions framework is to provide a vocabulary for discussion to assist designers and evaluators. It readily revealed potential improvements to Pursuit, and so in the present case it certainly achieved its goal. In particular, it revealed unnoticed weaknesses in viscosity, hidden dependencies and (quite unexpectedly to the designer) imposed guess-ahead; a convincing demonstration of how useful the framework can be.

Acknowledgements The authors thank Mitchum D'Souza, Bill Hefley, David Hendry, David Kosbie, David Kurlander, James Landay and Marian Petre for enriching comments on this work. We also thank the reviewers for their insightful comments. This research is partially funded by NSF grant IRI-9020089 and by a grant from AAUW.


107

References Borg, K (1990), "IShell: A Visual UNIX Shell", in Proceedings of CHV90: Human Factors in Computing Systems, J C Chew & J Whiteside [eds.], ACM Press, pp.201-207. Cypher, A (1991), "EAGER: Programming Repetitive Tasks by Example", in Proceedings ofCHI'91: Human Factors in Computing Systems (Reaching through Technology), S P Robertson, G M Olson & J S Olson [eds.], ACM Press, pp.33-40. Cypher, A (1993), Watch What I Do: Programming by Demonstration, MIT Press. Davies, S P (1991), "Characterising the Program Design Activity: Neither Strictly Top-down nor Globally Opportunistic", Behaviour & Information Technology 10 (3), pp. 173-190. Doane, S M, Pellegrino, J W & Klatzky, R L(1990), "Expertise in a Computer Operation System: Conceptualization and Performance", Human-Computer Interaction 5, pp.267-304. Finzer, W & Gould, L (1984), "Programming by Rehearsal", Byte Magazine 9 (6), pp.187-210. Furnas, G W (1991), "New Graphical Reasoning Models for Understanding Graphical Interfaces", in Proceedings of CHI'91: Human Factors in Computing Systems (Reaching through Technology), S P Robertson, G M Olson & J S Olson [eds.], ACM Press, pp.71-78. Green, T R G (1989), "Cognitive Dimensions of Notations", in People and Computers V, A Sutcliffe & L Macaulay [eds.], Cambridge University Press, pp.443-460, Proceedings of HCI'89. Green, T R G (1991), "Describing Information Artifacts with Cognitive Dimensions and Structure Maps", in People and Computers VI: Usability Now!, D Diaper & N Hammond [eds.], Cambridge University Press, pp.297-316, Proceedings of HCI'91. Green, T R G & Petre, M, "Cognitive Dimensions as Discussion Tools for Programming Language Design", (submitted for publication). Haeberli, P E (1988), "ConMan: A Visual Programming Language for Interactive Graphics", ACM SIGGRAPH, pp.103-111. Halbert, D C (1984), "Programming by Example", Computer Science Division, University of California, Berkeley, CA, USA, PhD thesis. Kurlander, D & Feiner, S (1988), "Editable Graphical Histories", in Proceedings of 1988 IEEE Workshop on Visual Languages, pp.127-134. Lewis, C & Olson, G M (1987), "Can Principles of Cognition Lower the Barriers to Programming?", in Empirical Studies of Programmers: 2nd Workshop, G Olson, S Sheppard & E Soloway [eds.], Ablex, pp.248-263. Lieberman, H (1982), "Constructing Graphical User Interfaces By Example", in Proceedings of Graphics Interface '82, pp.295-302. Lieberman, H (1993), "Mondrian: A Teachable Graphical Editor", in Proceedings ofINTERCHI'93, S Ashlund, K Mullet, A Henderson, E Hollnagel & T White [eds.], ACM Press, p.l44. Maulsby, D L & Witten, IH (1989), "Inducing Programs in a Direct Manipulation Environment", in Proceedings of CHI'89: Human Factors in Computing Systems, K Bice & C H Lewis [eds.], ACM Press, pp.57-62. Modugno, F, "Pursuit: Programming in the User Interface", School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA, PhD Thesis, (in preparation).

108

Francesmary Modugno, T RG Green & Brad A Myers

Modugno, F & Myers, B A (1994), "Pursuit: Graphically Representing Programs in a Demonstrational Visual Shell", in Proceedings ofCHI'94 Video Program. Myers, B A (1988), Creating User Interfaces by Demonstration, Academic Press. Myers, B A (1991), "Demonstrational Interfaces: A Step Beyond Direct Manipulation", in People and Computers VI: Usability Now!, D Diaper & N Hammond [eds.], Cambridge University Press, pp. 11 -30, Proceedings of HCI'91. Myers, B A, Guise, D A, Dannenberg, R B, Vander Zanden, B, Kosbie, D S, Pervin, E, Mickish, A & Marchal, P (1990), "Comprehensive Support for Graphical Highly-Interactive User Interfaces: The Garnet User Interface Development Environment", IEEE Computer 23 (11), pp.71-85. Nielsen, J & Molich, R (1990), "Heuristic Evaluation of User Interfaces", in Proceedings ofCM'90: Human Factors in Computing Systems, J C Chew & J Whiteside [eds.], ACM Press, pp.249-256. Visser, W (1990), "More or Less Following a Plan During Design: Opportunistic Deviations in Specification", International Journal of Man-Machine Studies 33 (3), pp.247-278.

Evaluating Evaluation Methods A Dutt, H Johnson & P Johnson Department of Computer Science, Queen Mary and Westfield College, University of London, Mile End Road, London El 4NS, UK. Tel: +44 (0)71 975 5238/5224 Fax: +44 (0)81 980 6533 EMail: {hilaryj, pete}@dcs.qmw.ac.uk In HCI the aim of evaluation is to gather information about the usability or potential usability of a system. This paper is principally concerned with evaluating the effectiveness of two discount user inspection evaluation methods in identifying usability problems in a commercial recruitment database system with complex interface and system functionality. The two specific inspection methods investigated are heuristic evaluation and cognitive walkthrough. Several comparisons are made between the number, nature and severity of usability problems highlighted, the time needed to employ the methods and the ability to generate requirements for re-design. The results indicate that the methods are best considered as complementary and both should be employed in, but perhaps at different stages of, the design process. Keywords: evaluation, usability inspection methods.

1. Introduction The development of a successful interactive system depends on a formula of iterative design and early and continuous evaluation. However, industry's response to conducting evaluations has been patchy (Johnson & Johnson, 1989; Rosson, Maass & Kellogg, 1988). Many industrialists remark that the reasons for this are the cost of employing evaluation methods and the expertise necessary. Another reason is the cumbersome and complex nature of evaluation approaches, especially task analytic approaches such as TAG (Payne & Green, 1986), TAL (Reisner, 1981) and GOMS (Card, Moran & Newell, 1983). Additionally, evaluations are seen as providing information about what is unsatisfactory, but are less useful in generating information that can be used to facilitate more usable and fewer re- designs. Researchers therefore, must assess the effect of using current evaluation methods within the industrial development process, develop future methodologies and tools that require a limited training period and can be far more easily accommodated within the development process.

110

A Dutt, H Johnson & P Johnson

Furthermore, by developing evaluation methods which are more effective in stimulating redesigns of improved usability, the number of iterations required to make a system reach an acceptable level of usability and hence the cost of applying evaluation methods, may be reduced. It is the assessment of evaluation methods themselves, as opposed to evaluation of an interface with a prescribed method, which concerns us. Our research goal is therefore to evaluate evaluation methods. To address the problem of meta-evaluation it is necessary to devise a research programme which includes various stages. For instance, it is necessary to review different approaches to evaluation highlighting their uses, advantages and disadvantages. It is also necessary to employ these different methods to evaluate a range of interfaces, thereby assessing their overall effectiveness at drawing attention to usability problems. The next step is to develop criteria not only to assess evaluation methods but also for evaluation methods to satisfy. These criteria will take account of, for example, the ability to generate re-designs of greater usability and the context of use of the evaluation method. In addition, an integrating framework is needed which is able to indicate the relative and complementary uses of different methods. Finally, there is a need for on-line tools to aid evaluation to overcome the problems highlighted by designers. These tools could take the form of an expert, advisory or consultative system, with a further range of tools aiding in data collection and analysis, and re-design suggestions.The result would hopefully be a set of procedures which would allow evaluation to occur frequently within the design process. Thus far the research conducted within this programme has first, evaluated the use and effectiveness of two usability inspection methods in identifying usability problems. The study provided data directly addressing the problems of time and thus cost, and expertise necessary. Second, the contribution of the evaluation methods to providing information and requirements for re-design, has been assessed. This is one departure from current evaluation research. Finally, we have compared the effectiveness of employing evaluation methods for providing information to guide re-design with the contribution made by using task analysis to identify user requirements for the re-designed system. This last study addressed a fundamental problem with current evaluation methods, that is, evaluating without access to user goals and characteristics. The study successfully demonstrated the complementary benefits of using both task analysis and evaluation methods for re- designing systems. This study provides a more significant departure from current research in that it provides an assessment of the value of evaluation and iterative design against the use of a task analysis method, Task Knowledge Structures (TKS), (Johnson & Johnson, 1991; Johnson et al., 1988) in generating information and requirements for the re-designed system. For the sake of brevity, this paper is only able to outline initial research undertaken towards achieving early stages in this research programme, i.e. studies of the use of usability inspection methods in evaluating a complex and widely-used, commercial interface.

2. Recent Research in Evaluation Much recent research has been concerned with discount usability inspection methods. These methods include specialist reports, usability and cognitive walkthroughs and jogthroughs (Karat, Campbell & Fiegel, 1992; Lewis et al., 1990). Evaluators using these methods can assess design specifications, early mock-ups, prototypes or fully implemented systems. The

Evaluating Evaluation Methods

111

major characteristic of inspection methods is that they draw on expert knowledge to provide judgements about system usability. The walkthrough method outlined in Lewis et al. (1990) was developed from work by Poison & Lewis (1990). Wharton et al. (1992) state that cognitive walkthrough is inappropriate for testing high functionality applications due to the repetitive form-filling. This discourages evaluators from using the method and they recommend someone with an HCI or Cognitive Science background as a member of an evaluation team consisting of three or four people. Further limitations is that the method does not match well with current software development practice. The method does not generate solutions for the problems identified and it is limited by focusing on lower level interface issues. To solve the latter problem, the walkthrough could be combined with other evaluation methods to allow for higher level evaluation. Also, the selection of the tasks to be undertaken by the evaluators is critical to the success of the evaluation and the quality of the problems diagnosed. Guidelines and heuristic evaluation also belong to this expert analysis group. Guidelines are used in ad hoc evaluations, based on, for instance, Smith & Mosier (1986). Nielsen & Molich (1990) reduced these guidelines to a list of nine usability rules or principles assumed to contain most usability problems. They demonstrated in a series of experiments that heuristic evaluation delivers the best results when several people conduct the evaluation independently and then aggregate their findings. Usability specialists were found to be better than those without usability expertise at finding usability problems. These authors recommend using three to five people to evaluate an interface heuristically. Nielsen and Molich state the weakness of their approach is that they have only evaluated fairly small-scale interfaces in a completed design that just needed polishing up. The benefits of heuristic evaluation is that it is cheap, quick, intuitive and requires minimum learning time. Therefore, it can be easy to motivate people to use it. The method does not require too much planning unlike observational, experimental or survey methods of evaluation and it can be used early in the design process. The disadvantages are that problems are identified without providing any direct solutions. The method tends to be biased by the thinking of the evaluators and does not generate breakthroughs in the evaluated design. It may identify 'false' problems not related to usability thus leading the designer in the wrong direction. Some heuristics are difficult to interpret and the use of heuristic evaluation in the context of the tasks end-users perform, or will perform on a proposed system, are not considered. Recent research has also been concerned with comparing the effectiveness of various inspection methods, for instance, Nielsen & Phillips (1993) compare heuristic, formal and empirical methods in predicting user performance. On the other hand, Jeffries et al. (1991) compared the effectiveness of heuristic evaluation, usability testing, guidelines and cognitive walkthrough in identifying usability problems. Their results suggested heuristic evaluation performed by several usability specialists is cheaper and identifies more of the serious problems than any other evaluation method. Their results also demonstrated that heuristic evaluation identifies the greatest number of the least severe problems, many of which were problems of 'taste'. The success of the heuristic evaluation in this study is probably due to the fact that three usability experts were used to carry out the heuristic evaluation over a period of two weeks, while for each of the other methods, groups of novice evaluators and one or two experts were used for a shorter period of time. Despite this, Jeffries et al. still concluded that heuristic evaluation and usability testing have advantages over cognitive walkthrough. Irrespective of this criticism, the participants of the CHI92 workshop on usability inspection

772


methods agreed that heuristic evaluation was the more easily learnt and better inspection method, compared to the cognitive walkthrough. However, they knew little about whether usability inspection methods described problems customers would have in the workplace or were related in any way to ultimate customer satisfaction. None of the afore-mentioned research papers considered the effectiveness of the evaluation methods in facilitating better re-designs. Furthermore, even though recent research favours heuristic evaluation there is a shortage of literature which considers the use of heuristic evaluation on realistic, complex interfaces. Nor is there any evidence of its ability, nor of the cognitive walkthrough, to produce designs of improved usability within the software development process. It would therefore be interesting to see how effective data from both heuristic evaluation and cognitive walkthrough really is at finding both usability problems and design solutions for a range of both simple and highly complex interfaces. The following study is an initial attempt to answer the first of these questions.

3. Evaluating Evaluation Methods Study The aims of the present research were first, to select two out of a range of evaluation methods to compare, and secondly, apply them to an implemented system. For this activity to be an advance on the state of the art, the chosen implemented system must be widely used in industry, and therefore commercially available with a highly complex set of functions and correspondingly complex interface. The system to be evaluated was used by a recruitment agency Devenco (UK) Ltd to assist staff in providing high-calibre personnel for the construction industry. The system is a sophisticated PICK network package running on the PICK operating system. It holds information pertaining to the recruitment process, i.e. information on applicants who have sent in their CVs; client companies, who send in job vacancy details that they require Devenco to fill, and job vacancies. It allows a consultant to find all the applicants suitable for a particular job or find all the jobs suitable for a particular applicant. It supports a large range of general management features such as the production of management reports and standard letters. Access to the system is by password only. 3.1. Selecting Evaluation

Methods

There were considered to be too many evaluation methods to compare in a single study. Given the large number of methods available, it was necessary to establish selection criteria for choosing two methods that could provide a reasonable evaluation of the system. The factors considered were number of subjects required to perform the evaluation, usability expertise of the evaluators, the length of time needed to carry out the study and finally, the ability to satisfy the constraints of the environment in which the studies were to take place. Heuristic evaluation and cognitive walkthrough were the two methods which best met these criteria.

3.2. Study Details — Heuristic Evaluation The first part of the study involved evaluating the recruitment system using heuristic evaluation, modified by Nielsen & Molich (1990) for use by software developers during the software development process. Three evaluators were selected to perform the evaluation. Two evaluators were Advanced MSc students in HCI at QMW, the third evaluator was an AMSc student in Distributed Systems.


113

The two from the HCI stream had experience in evaluating the usability of system interfaces and were familiar with the heuristic procedure. The third evaluator had no experience of human factors issues and no knowledge of the heuristic evaluation method. Each evaluator had his/her own terminal from which he/she could perform the evaluation. The heuristic evaluation required the evaluators to investigate the part of the system which directly related to the recruitment process, and look for problems that they judged as violating any one of the nine usability principles from Nielsen & Molich (1990). The evaluators were provided with answer sheets to write down their findings. The sessions lasted between one and a half and two hours. After this, they participated in a group discussion about the evaluation that lasted for a further fifteen minutes. 3.3. Study Results — Heuristic

Evaluation

The results show that more of the discovered problems violated the 'Simple And Natural Dialogue' heuristic than any other heuristic. Second in this order was the 'consistency' heuristic. However, only two problems were uncovered which violated the 'Minimise Memory Load' heuristic and one which violated the 'Good Error Message' heuristic. In the case of the latter heuristic, the problem identified was listed as two problems, where one problem was an example of the other. This highlights the fact that evaluators had no way of knowing when one problem was a generalisation of another problem. As a result, sometimes the evaluators would list a number of problems that reflected a more general problem in the system. On such occasions, it was left to the skill of the person doing the analysis to replace these problems with their more general form. Some of the heuristics were difficult to understand and therefore could not be applied as successfully as others, for instance the 'consistency' heuristic caused problems for the evaluator without human factors experience. He also had difficulty understanding the difference between the 'Simple And Natural Dialogue' heuristic and the 'Speak The User's Language' heuristic. One evaluator from the HCI stream found difficulty in establishing how to systematically apply the 'Minimise The User Memory Load' heuristic. 3.4. Performance of the

Evaluators

Performance of the evaluators varied: the two evaluators from the HCI stream discovered more problems than their colleague from the DS stream. It was very difficult to decide which evaluator found the most severe problems. In order to obtain accurate information relating to the comparative severity of the problems uncovered, another study had to be conducted using independent judges with no experience of the system being evaluated. During the group discussion, evaluators described some of the problems they encountered in the system. Overall opinion was that the system was not user-oriented: there were no error messages, no indication of what to do when a particular screen appeared and no help to get out of any difficult situations. One of the evaluators from the HCI stream pointed out that because the system used jargon related to the recruitment business, this caused problems when conducting the evaluation. The evaluators agreed that they did not have sufficient knowledge to evaluate the system specifically in terms of the recruitment consultants' needs or requirements of his/her job. One criticism which could be levelled at this study is that the number of evaluators is insufficient. This might be the case if there was any foundation to the assumption that the

114


fewer the evaluators, the greater the variability. We have no data which could be used to test this hypothesis. However, there are several factors which contributed to our choice of three evaluators in each group. First, there were time constraints imposed by using a commercial interface. Secondly, the number of evaluators satisfies the bounds recommended by Nielsen & Molich (1990). Thirdly, the number of evaluators is in the range which is regularly used in the literature, see studies by Wharton et al. (1992), making our results directly comparable. Fourthly, Nielsen & Landauer (1993, p.212) argue that one would rarely evaluate a single user interface to the 'bitter end' without applying iterative design to fix the usability problems found by the first few evaluators. Since this is the first evaluation of this interface, a small number of evaluators follows this rubric. Finally, Nielsen & Landauer (1993) argue that the highest ratio of benefits to costs, for medium-large software projects, (the only figures supplied) is for 3.2 test users and 4.4 heuristic evaluators. Our choice of three evaluators per group is close to these numbers. Another potential problem relates to the experience of the evaluators. Although the evaluators with an HCI background were familiar with research and the guidelines related to heuristic evaluation, none of the evaluators had ever undertaken a heuristic evaluation before. There are two reasons why such evaluators were chosen. The underlying rationale for the first reason is provided by Nielsen (1992). Nielsen argues that one way of evaluators building up the usability expertise necessary for optimal performance as heuristic evaluators is observation of test users. Both HCI evaluators had considerable experience of empirical studies involving endusers evaluating designs and therefore satisfied one possible pre-requisite for being optimal heuristic evaluators. A further reason relates to the fact that we conducted the study to consider how well evaluation methods might be integrated into the software development process. In this context, it would be interesting to see how evaluators who might mirror evaluators in industrial teams, i.e. with some human factors experience but without experience of the specific evaluation methods, might fare. Additionally, there is no reason to assume that the potential improved performance resulting from the development of expertise of the heuristic method is any different to that in the cognitive walkthrough method.

3.5. Summary — Heuristic Evaluation A total of 37 usability problems, which covered all parts of the database system, were identified by the three evaluators in this study. The number of problems was taken only after the problems had been examined, duplicate problems had been removed and common problems generalised. The fact that the 'Good Error Messages' heuristic yielded the lowest number of violations does not mean that it was the hardest to apply: this can only be judged by considering the evaluators behaviour and performance while applying the heuristic. In the case of the 'Good Error Messages' heuristic, this was one of the easiest to apply and two of the three evaluators identified the main violation of the heuristic, i.e. there are no error messages. The heuristic evaluation tends to identify a number of problems that are of a common type, but it does not indicate which problems are common to other problems and the general problem that arises from them. For example, the evaluators found that there was no onscreen indication of how to create new files or of how to get to the search facility. These two problems are examples of a more general problem present in the system; that is, the system does not show all the commands available on a screen. The evaluators overlooked the fact that the two problems they had discovered were of a common type and there was nothing to indicate the link between the problems in the heuristic study. It was left to the skill of the


115

experimenter to interpret the results and identify the common problem. Heuristic evaluation does not allow evaluators to accurately test how well the system performs in relation to the tasks that a recruitment consultant has to perform. This could be due to the fact that under the present method, evaluators are required to explore the system without any structure. Furthermore, it does not provide any evidence to a system developer of which problems actually affect the performance of tasks undertaken by the end-users. 3.6. Improvements

to Heuristic

Evaluation

An improvement to this evaluation method would mean providing some structure to the evaluation, either by giving the evaluators more information about the end-users tasks, insisting that one evaluator be familiar with the application domain, or by performing the evaluation using a set of tasks that exemplify those performed by the recruitment staff. While performing each task, evaluators would assess the usability of the system according to a set of heuristics that take into account the ability of the system to match the users' goals. Another improvement concerns the training of evaluators on the heuristics, which for most subjects are difficult to apply. 3.7. Study Details — Cognitive Walkthrough

Method

The recruitment system was again evaluated by three people, this time using a modified version of the cognitive walkthrough method. In order to make the results directly comparable, the three evaluators were chosen so that the range of experience among them in usability issues was identical to the evaluators used in the heuristic study. One evaluator was taken from the DS stream at QMW. The second was from the HCI stream and was familiar with the terminology used in the cognitive walkthrough. The third evaluator had more knowledge of HCI issues than the other two, in particular those concerning usability measurement. He was also familiar with the cognitive walkthrough method and its terminology. None of the evaluators however, had ever used the cognitive walkthrough method to evaluate interfaces. The version of the walkthrough used is similar to the cognitive walkthrough presented in Lewis et al. (1990), but differs in the type of interface that it is applied to, and extensions that are incorporated into it. These extension originated from recommendations given in Wharton et al. (1992). The extensions are: i. The tasks that were given to the evaluator were listed in order of complexity, with the simplest first and the most complex last. ii. The tasks were chosen from results of the task analysis to reflect key system functionality. iii. Evaluators were told that there was no need to fill in a form for each keystroke action and an 'action' could be higher-level than a keystroke. They were also told that they did not have to fill in a form for an action in too much detail, provided the action was identical to another action. In such cases, they should focus on problems incurred while performing the action. iv. Two evaluators are familiar with the cognitive walkthrough and its terminology. v. The walkthrough form was altered and an additional section was provided for the evaluator to keep track of any problems (unrelated to the task being performed) noticed while carrying out a task.

116


The evaluators were given an example task to perform and for each action performed in undertaking the task, a walkthrough form was completed. The example task allowed the evaluators to become acquainted with the walkthrough method and receive answers to queries he or she may have about the method. Once the evaluators were confident enough to begin the evaluation, a set of six tasks were presented and for each action of the tasks performed, the evaluators had to complete a walkthrough form. 3.8. Results — Cognitive

Walkthrough

The cognitive walkthrough identified 32 usability problems. The count was taken after all duplicate problems had been removed and all problems of a common nature had been generalised. The cognitive walkthrough identified usability problems which reflect many task-related problems for users rather than low severity problems of 'taste' or of very minor significance. The results provided written evidence of the parts of the system that have poor usability. The walkthrough method has the additional advantage of focusing on tasks that are representative of the tasks that the recruitment staff have to perform and increases designers' awareness of users' tasks. However, analysis can take up to twice as long as analysis time for heuristic evaluation depending on how well evaluators have completed the walkthrough forms. The disadvantage of the walkthrough method, in common with the heuristic method, is that it sometimes identifies problems of a common type, but provides no indication of which problems are generalisations of other problems.

3.9. Performance of the Evaluators All three evaluators had difficulties with some of the questions which naturally affected their responses to the questions. One evaluator could not understand the question 'User can access needed information in system response?' This question had to be rewritten for him. Other problems occurred in distinguishing the difference between the following questions: 3. How will user access description of action? Problem accessing? Why/why not? and 4. How will user associate description with action? Problem associating? Why/Why not? All the evaluators found the task of completing their forms exceedingly tiresome resulting in lack of detail on the forms. Another potential reason for this lack of detail was that all of the evaluators found that by the time they had filled in a form for a particular action, they had lost track of the next actions they had to perform to complete a task.

3.10. Summary — Cognitive Walkthrough The cognitive walkthrough method can be a effective evaluation method as it identifies taskrelated problems rather than problems of 'taste'. This allows system designers to focus resources on problems that severely affect the user's ability to execute tasks rather than spending resources on less important problems of 'taste'. It has the additional advantage of increasing awareness among system designers of the importance of user issues, in particular, users' tasks. This in turn might result in the cognitive walkthrough being very useful in reducing the communication gap between usability experts and software designers.


117

Although the cognitive walkthrough yielded fewer usability problems than the heuristic evaluation, the difference in number was small (five). Furthermore, some of the problems discovered in the heuristic evaluation were listed twice or more if they violated two or more heuristics. From the data viewed thus far, cognitive walkthrough, provided a few improvements are made, appears to be capable of delivering data that can be used to develop more usable designs. Nevertheless, there are still problems with the method. First of all, the walkthrough form for a single action is not itself 'user-friendly'. It does not take into account the tasks the evaluator has to perform during an evaluation and adds heavily to the evaluator's cognitive load. The walkthrough form used in the study consisted of two pages and contained questions that were difficult to interpret. The difficulty of interpretation can be attributed to the unnatural phrasing of the questions. Often, the time required to complete one of these forms resulted in the evaluator forgetting the next action he/she had to do to complete a task and the form tended to focus on lower level interface issues. It was also noticed that the quality of the results for the cognitive walkthrough depends on the person doing the analysis and their expertise at recognising and interpreting problems from the data in the forms. The results lead us to suggest a number of improvements to the cognitive walkthrough method such as: i. Rephrase questions so that they are comparatively simpler to understand and less ambiguous. ii. Put all questions on one sheet of paper even if it is larger than a sheet of A4. iii. Allow evaluators to write down their goal structure and the list of actions necessary to achieve each subgoal, either on the walkthrough form or on some review card for easy referral. iv. A more futuristic improvement would be to develop a computer-supported walkthrough tool that recorded the actions made by an evaluator in interacting with the system being evaluated, recorded all screen changes, assisted in completing the form and supported analysis of the data collected, perhaps in the form of a database containing all data collected and a database manager facility.

4. Severity of Usability Problems Identified In addition to time, number of problems identified, it is also important to consider the capability of the two evaluation methods to highlight major usability problems, i.e. those which will crucially affect usability. It was considered that an unbiased assessment of the severity of the problems identified was necessary. Consequently, a further study was carried out whereby nine independent judges ranked the usability problems in terms of their severity. The usability problems were randomly allocated identification numbers and each problem was written on a separate card; the cards were to be sorted with the most severe at the top and least severe at the bottom. Frequency tables were used to show if there was consistency across judges. The results indicated that there was consistency in the ranking of the most severe problems, but not in terms of the least severe, or 'taste' problems. Three problems were almost universally considered to be severe, and were not listed in the ten least severe problems; these were:

118

A Dutt, H Johnson & P Johnson i. 'system sometimes logs the user out if he/she types in an invalid command'; ii. 'there are no confirmation messages'; iii. 'the system has no error messages'.

All three of these most severe problems were identified by the heuristic evaluation, the cognitive walkthrough identified one of these problems, i.e. the last one. It is surprising that the cognitive walkthrough failed to identify the second problem. However, the identification of the first problem of the system logging the user out, may well be a random fault that did not come up in the evaluation. Identification of this usability problem is reliant on the evaluator typing in an invalid command. 5. Requirements for Re-design One aspect of the research we have conducted needs further analysis, that is the ability of the two evaluation methods to highlight problems which lead to requirements for re-design. There were 18 user requirements for re-design that were a result of conducting the heuristic evaluation, and 24 the cognitive walkthrough. On first sight the cognitive walkthrough appears to be better able to provide a basis for re-design. However, the results are not conclusive in that there is no straight path from evaluation, problem identification, requirement to solve the problem, and problem solution. 6. Conclusion We have already seen that it is difficult to compare on numeric grounds alone, the performance of the two evaluation methods. In terms of number of usability problems, more were found in the heuristic evaluation than the cognitive walkthrough. However, we noted that there were usability problems in the heuristic method which were counted twice since they violated more than one heuristic. The number of problems found is therefore not in isolation a reliable metric for evaluating evaluation methods. The number of problems found is relatively low given the quality of the interface. This could well be due to the way in which the problems were counted, i.e. after removal of duplicate problems and generalisation of common problems. Generally speaking, there is a lack of information in the literature as to what constitutes a problem which would provide guidance in the counting phase. In fact counting usability problems, which might be taken as one way of assessing alternative interfaces, is very much down to the expertise and consistent judgement of the person who analyses the evaluators' findings. A further potential measure of the effectiveness of the evaluation methods is the time to use the method in order to identify problems which exist. On time to use the methods, the difference was less than an hour, the heuristic evaluations took one and a half to two hours, plus time for a group discussion, and the cognitive walkthrough between two and a half and three hours. Again the difference is relatively small, not enough for a design team to choose between the two methods since they tend to identify complementary rather than overlapping usability problems. The time only becomes a problem when many evaluations of different interfaces are required and here the small difference might result in a choice other than the cognitive walkthrough. One question which concerns us relates to whether or not the evaluation sessions were comparable and therefore our results are valid. This question arises because the evaluators


119

in the different evaluation conditions spent a different amount of time on the interface. The heuristic session was shorter than the cognitive walkthrough session. Additionally there was no guarantee that the evaluators in the same or across groups spent the same amount of time on interface states or even followed a similar path through the different interface functions. It was never our intention to constrain the evaluators in this way. If so then the study would have been designed entirely differently in order to reflect the different constraints we wished the evaluation methods to be tested under. The intention was rather to see how long it took the evaluators using the different methods to be satisfied that they had identified all the usability problems. Time rather than a variable to be controlled was one of the factors we wished to investigate. The evaluators were free to stop or continue the evaluation session as they wished and under these conditions we consider the sessions to be comparable given the context for conducting the study. Both methods appear to be equally difficult to apply. It would be interesting to see if they are any easier to apply than task analytic evaluation approaches. The heuristic evaluation method contains heuristics which are hard to distinguish one from another, and which in some circumstances appear to be contradictory. On the other hand, the cognitive walkthrough contains tiresome form-filling activities and increases evaluators' cognitive load. The improvements we suggested to both methods would be beneficial to evaluators and subsequently to the designers of the evaluated systems. The nature of the usability problems highlighted was also not a good indicator of which is the better method. The heuristic method, as intended, identified general problems and lower level instances of those problems. The cognitive walkthrough successfully identified usability problems faced by users performing the specific tasks the system supported. However, it is not known whether the design optimally supported those tasks. The heuristic evaluation was better at identifying the most severe high-level usability problems with the cognitive walkthrough identifying lower-level issues. Yet, the cognitive walkthrough appeared to be better at providing a basis for re-design. Further investigation of the specific requirements from the evaluation studies is in hand, along with an investigation of the requirements identified by the task analysis. The results reported in this paper lead us to conclude that rather than the evaluation methods being considered as alternatives, that they be treated as complementary. It is important to take into account the context of use of an evaluation method, i.e. the purpose of the evaluation and at which stage of the design process it is to be used. Our results suggest, providing a task analysis has been conducted to guide system design, that early in the design process an evaluation method which focuses on high level design principles is needed. Alternatively, second or third re-designs need to focus more on tasks the user needs support in performing, and also on lower level design issues. In terms of criteria, more than one set of criteria appears necessary, governed by the role played by the particular evaluation method and at what stage in the design process. The study reported here is the first of a number of studies which comprise the research programme outlined in the introduction. It demonstrates that usability inspection methods do have a part to play in evaluating complex interfaces and that they might succeed in playing a role in the software development cycle. However, modifications need to be made to both methods in order that they are easier to use and provide a more solid foundation for re-design.

720


Acknowledgements Thanks are due to the six evaluators who took part in this study, and to Devenco (UK) Ltd. We also thank the Cognitive Science Tri-Council for funding the AMSc in HCI studentship awarded to Anita Dutt. Full details of usability problems and data are freely available from the second author.

References Card, S K, Moran, T P & Newell, A (1983), The Psychology of Human-Computer Interaction, Lawrence Erlbaum Associates. Jeffries, R, Miller, J R, Wharton, C & Uyeda, K M (1991), "User Interface Evaluation in the Real World: A Comparison of Four Techniques", in Proceedings ofCHI'9I: Human Factors in Computing Systems (Reaching through Technology), S P Robertson, G M Olson & J S Olson [eds.], ACM Press, pp.119-124. Johnson, H & Johnson, P (1989), "Integrating Task Analysis into System Design: Surveying Designers' Needs", Ergonomics 32(11), pp.1451-1467. Johnson, H & Johnson, P (1991), "Task Knowledge Structures: Psychological Basis and Integration into System Design", Acta Psychologica 78, pp.3-26. Johnson, P, Johnson, H, Waddington, R & Shouls, A (1988), "Task-related Knowledge Structures: Analysis Modelling and Application", in People and Computers A'.DM Jones & R Winder [eds.], Cambridge University Press, Proceedings of HCI'88. Karat, C-M, Campbell, R L & Fiegel, T (1992), "Comparison of Empirical Testing and Walkthrough Methods in User Interface Evaluation", in Proceedings of CHV92: Human Factors in Computing Systems, P Bauersfeld, J Bennett & G Lynch [eds.], ACM Press, pp.397-404. Lewis, C, Poison, P, Wharton, C & Rieman, J (1990), "Testing a Walkthrough Methodology for Theory-based design of Walk-up-and-use Interfaces", in Proceedings of CHI'90: Human Factors in Computing Systems, J C Chew & J Whiteside [eds.], ACM Press, pp.235-241. Nielsen, J (1992), "Finding Usability Problems through Heuristic Evaluation", in Proceedings of CHI'92: Human Factors in Computing Systems, P Bauersfeld, J Bennett & G Lynch [eds.], ACM Press, pp.373-380. Nielsen, J & Landauer, T (1993), "A Mathematical Model of the Finding of Usability Problems", in Proceedings ofINTERCHI'93, S Ashlund, K Mullet, A Henderson, E Hollnagel & T White [eds.], ACM Press, pp.206-213. Nielsen, J & Molich, R (1990), "Heuristic Evaluation of User Interfaces", in Proceedings of CHI'90: Human Factors in Computing Systems, J C Chew & J Whiteside [eds.], ACM Press, pp.249-256. Nielsen, J & Phillips, V L (1993), "Estimating the Relative Usability of Two Interfaces: Heuristic, Formal and Empirical Methods Compared", in Proceedings ofINTERCHI'93, S Ashlund, K Mullet, A Henderson, E Hollnagel & T White [eds.], ACM Press, pp.214-221. Payne, S J & Green, T R G (1986), "Task-Action Grammars: A Model of the Mental Representation of Task Languages", Human-Computer Interaction 2 (2), pp.93-133. Poison, P & Lewis, C (1990), "Theory-based Design for Easily Learned Interfaces", Human-Computer Interaction 5, pp.191-220. Reisner, P (1981), "Formal Grammars and Human Factors Design of an Interactive Graphics System", IEEE Transactions on Software Engineering 5 (2), pp.229-240.


121

Rosson, M B, Maass, S & Kellogg, W (1988), "The Designer as User: Building Requirements for Design Tools from Design Practice", Communications of the ACM 31, pp. 1288-1297. Smith, S L & Mosier, J N (1986), "Guidelines for Designing User Interface Software", The MITRE-Corporation, Software Report MTR-10090 (aka ESD-TR-86-2768). Wharton, C, Bradford, J, Jeffries, R & Franzke, M (1992), "Applying Cognitive Walkthroughs to More Complex User Interfaces: Experiences, Issues and Recommendations", in Proceedings of CHI' 92: Human Factors in Computing Systems, P Bauersfeld, J Bennett & G Lynch [eds.], ACM Press, pp.235-241.

Crafting Interaction: Styles, Metaphors, Modalities and Agents

A Comparison of Placement Strategies for Effective Visual Design Jean Vanderdonckt*, Missiri Ouedraogo* & Banta Ygueitengar1 * Institut d'Informatique, Facultes Universitaires Notre-Dame de la Paix, Rue Grandgagnage 21, B-5000 Namur, Belgium. Tel: +32 (0)81 72.49.75 Fax: +32 (0)81 72.49.67 EMail: [email protected] * Ministere de la Fonction Publique et de la Modernisation del'Administration, 03 BP 7006, Ouagadougou 03, Burkina Faso. Tel: +22630.22.86 1

do D. Ngarboldoum, Meridian Biad, B.P.87 N'djamena, Republic of Tchad. Tel: +235 51.43.14 Fax: +235 51.23.45 The development of graphical user interfaces for interactive applications is subject to a series of well-known problems which could be relevant of the domain of visual design. This typically includes the problem of placing aesthetically interaction objects (IO) according to principles applied in placement strategies. This paper first reviews the problem of IO placement and shows the rationale for the most significant placement strategies found today. It then tries to compare six such strategies along several dimensions and mathematical relationships with respect to three points of view: the designer's point of view, the human factors expert's point of view, and the user's point of view. Keywords: arrangement, dimensioning, interaction object, localization, interaction object placement, placement strategy, visual design.

126

Jean Vanderdonckt, Missiri Ouedraogo & Banta Ygueitengar

1. Introduction The problem of placement concerns the spatial position of interaction objects (10) such as edit box, radio button, list box, . . . with respect to visual elements in a screen layout. Effective placement can be achieved through a particular placement strategy involving a certain amount of visual techniques such as proximity, alignment, separate reference, centering, and conformity. For instance, Galitz suggests a placement strategy where IOs should be placed according to their nature: "All elements on a screen should be located in a unique and consistent position. These elements are: title, screen identifier, screen body (including caption, data, section headings, completion aids, prompting), status or instructional messages, error messages, command field or area." (Galitz, 1992) In this paper, the placement is defined as the description of a composite 10 (e.g. a form container widget) for displaying IOs to be placed relatively to one another. Composite IOs generally allow IOs to resize themselves as the composite resizes. IOs can be placed in relative positions either by percentage or by absolute distances. Every 10 is univocally determined on its top, bottom, left and right sides. 1.1. How to Define a Placement

Strategy?

On one side, there is a need for knowing how to present data, graphics, images in general (e.g. on paper). This knowledge is generally expressed in terms of geographical, physical or spatial principles such as adjacency (Galitz, 1992), proximity (Galitz, 1992; Kim & Foley, 1990), similarity (Kim & Foley, 1990), proportion (Kim & Foley, 1990; Marcus, 1992), format (Galitz, 1992; Lauer, 1990; Marcus, 1992), page layout grid (Kim & Foley, 1993; Marcus, 1992; Tarlin, 1990; Tufte, 1983), symmetry (Galitz, 1992; Lauer, 1990), ordering (Lauer, 1990), . . . Basics design principles are found extensively in (Galitz, 1992; Lauer, 1990; Marcus, 1992; Tufte, 1983). A summary of visual techniques that could be applied is reported in (Vanderdonckt & Gillo, 1994). On the other side, it is necessary to interpret these principles (which are independent of the domain of user interfaces) into rules or guidelines to be followed. It is not so easy to determine a realistic set of rules from these principles in order to work with properly (Kim & Foley, 1990). However, it could be done by translating some of these principles into a comprehensive placement strategy. In order to characterize this translation, quantifiable dimensions and relationships are needed. 1.2. Dimensions

and Relationships

in Placement

Strategies

The placement is decomposed into three parts: 1. The localization is interested by logically positioning IOs in the container. It covers position, alignment, justification. Localization could be mainly achieved through: — consistency, e.g. the position of 10 should be compatible with the users' conventions, should be consistent in format; — sequentiality, e.g. most frequently used IOs should be located first; — screen image, e.g. IOs should be equally located in all four quadrants of the container.

A Comparison of Placement Strategies for Effective Visual Design

127

2. The dimensioning goes in for the uniformization and standardization of 10 dimensions. It deals for instance with the length of abbreviations, the maximum number of characters per label, the length of an edit box, the item number in a list box, the harmonization between length and height for a dialog box. 3. The arrangement takes into account the 10 orientation and constraints related to the logical ordering of IOs. Arrangement should be logical as much as possible (e.g. by preference, by consensus, by physical property, by data flow), should emphasize visual cues, should care about aesthetics and should reduce ocular movements and screen density About 300 guidelines related to these dimensions have been gathered in a corpus ergonomicus (Bodart & Vanderdonckt, 1994). It is also sound to define mathematical relationships to characterize the practicability, the workability and the applicability of visual principles into the three above dimensions. These relationships will be helpful to compare different placement strategies. In the following list of relationships, the abbreviation 'iff' denotes 'if and only if: • Left justification: two IOs are left justified iff their left abscissa are identical. • Right justification: two IOs are right justified iff their right abscissa are identical. • Upper justification: two IOs are upper justified iff their left ordinates are identical. • Bottom justification: two IOs are bottom justified iff their right ordinates are identical. • Horizontal centering: two IOs are horizontally centered iff the ordinates of their centres are identical. • Vertical centering: two IOs are vertically centered iff the abscissa of their centres are identical. • Horizontal uniformity: two IOs are horizontally uniformized iff their lengths are identical. • Vertical uniformity: two IOs are vertically uniformized iff their heights are identical. • Horizontal equilibrium: three IOs are horizontally equilibred iff the horizontal interobject distances are identical. • Vertical equilibrium: three IOs are vertically equilibred iff the vertical inter-object distances are identical. • Diagonal equilibrium: three IOs are diagonally equilibred iff their centres are distributed on a same line. • Proportional equilibrium: three IOs are proportionally equilibred iff they are uniform and equilibred either vertically or horizontally. • Total equilibrium: three IOs are totally equilibred iff they are equilibred either vertically or horizontally, and uniform vertically and horizontally. For example, the dialog box reproduced in Figure 1 shows four rows of IOs that are equilibred. Labels are left justified, vertically uniform and vertically equilibred, therefore proportionally equilibred. Drop-down list boxes are right and left justified, vertically and horizontally uniform, vertically equilibred, therefore totally equilibred. Pushbuttons (Ok), (Cancel), and (Help) are totally equilibred in order to avoid different probabilities of selection which namely depends on the pushbutton dimensions. However, these pushbuttons are not vertically centered.

128

Jean Vanderdonckt, Missiri Ouedraogo & Banta Ygueitengar Change System Settings DJiplay:

| Super VGA (800x600.16 color.)

Keyboard:

|Enhanced 101 oi 102 key US and Non US keyboard.

Mpute:

|Micm.ofl. oi IBM PS/2

Network:

| Novell NetWare ( v < 0 )

111 lil III IPl

IZMZJ Iffr** 1 Figure 1: A dialog box with proportionally equilibred labels and total equilibred drop-down list boxes.

By demonstration Label: -Group boK-

T«xt Zone Texte

Figure 2: Placement by demonstration.

The characterization of these relationships is not really new since the eleven first relationships are provided as spatial manual arrangement options by most structured graphics editors and graphical user interface builders. But what is more interesting is to try to incorporate these relationships in placement strategies as a way of quantifiable measurement.

2. Manual Placement Strategies Placement strategies that should be manually applied are now considered in order to highlight the differences existing between such manual strategies and systematic strategies that could be included in computer-aided or automated tools for placing IOs. Manual placement strategies are performed when a designer is free to place IOs in the composite IO. This situation arises with interface builders (e.g. interface toolkit editors) where IOs could be created, placed and modified according to the designer's point of view. Such strategies also include placement by demonstration: Peridot (Myers, 1988), attempts to establish left, right, upper or bottom justification of IOs by using global rules defined by demonstration. Druid (Singh, Kok & Ngan, 1990) and Excel's Dialog Editor (Microsoft, 1992, p.263) both place IOs according to the implied current design situation. If the designer creates a first label, it is placed automatically with some margins. If an edit box should be added, it will be automatically placed on the right of the label with some extra space, assuming that the previously entered label identifies the edit box. When the designer creates another label after the previous one, the tool guesses that the new label and the previous one should be left justified. The problem with this technique is that, if the designer fails to follow a predefined traditional sequence, a bad IO placement may result.


129

For instance, if the designer creates a group box, an edit box and a label, the last label will be placed beneath the edit box (Figure 2). Moreover, these tools may require a lot of physical manipulations rather than logical ones. But, once these manipulations have been recorded, they can be replayed as many times as necessary. Gilt (Hashimoto & Myers, 1992) solves the manipulation problem by introducing a graphical tab, which is an absolute placement position, and a graphical style incorporating property and placement attributes. Redefining an existing graphical style modifies all IOs placed with that style.

3. Computer-Aided and Automated Placement Strategies As we can see, demonstrational strategies really become operational once all necessary visual techniques have been exemplified, thus requiring significant manipulation at the beginning. It stands to reason that the price of genericity may be considered here as the cost of this demonstration task. Thus, placement strategies providing a higher level of assistance are to be investigated. Model-based tools, like HUMANOID (Szekely, Luo & Neches, 1993), are design tools where user interfaces are described by giving a model of their presentation (including the placement of IOs) and their behaviour. HUMANOID provides a modelling language allowing designers to express how IOs should be manipulated. In particular, the 'Presentation' slot of the templates defines how IOs have to be laid out according to their nature. The three dimensions and all the mathematical relationships can therefore be theoretically preserved if they are included in the model. The biggest advantage of this approach is that the placement strategy is completely described and formalized in the model. Modifying the placement strategy in the model automatically regenerates the new placement of IOs, allowing the designer to work at a design level rather than at a physical level. This is why HUMANOID could be considered as a computer-aided placement strategy. If such tools say with which mean the placement of IOs can be formalized, specified, they do not necessarily say which mathematical relationships and which graphical principles should be so formalized. Of course, not all problems can be equally settled at the same time. Tarlin (1990) recalls us that modern word processors support user-defined documents through templates called style sheets which is a variant of underlying layout grid for structured documents. These templates specify various aspects of a document page such as its size, section header appearance, body text appearance, and the number of columns on the page. Why do we not reuse style sheets for placing IOs in dialog boxes? This is what Tarlin did when he completely visually redesigned 'Print' and 'Options' dialog boxes for X-Windows (Figure 3). Dotted lines represent the vertical and horizontal lines for defining a layout grid. The layout grid technique has been widely applied in computer-aided and automated placement strategies, but with different extents and interpretations. Such strategies are the onecolumn strategy applied in UIDE (de Baar, Foley & Mullet, 1992), the one or two-column strategy in GENIUS (Janssen, Weisbecker & Ziegler, 1993), the balanced two-column strategy in TRIDENT (Bodart et al., 1994), the shape strategy in DON (Kim & Foley, 1993), the Layout Appropriateness (Sears, 1993), and the Right/Bottom strategy in TRIDENT (Bodart et al., 1994).

3.1. Purposes and Limits of the Comparison Rather than analyzing all currently available tools for placing IOs, the comparison will focus on the six mentioned placement strategies according to three points of view and an example to see how they can be performed.

130

Jean Vanderdonckt, Missiri Ouedraogo & Banta Ygueitengar ;Number of Copies:

OK

;Page Range: From | Cancel

IPrint Format

;Printer Options..

PostSoript(R) Ip2 lpl Ip3 fbatch

OL

OL

10 D

;Onentaoon

©Default

Page Size] Default Side-s

Default

jNumber Up

Sheet C ount

I'iJASiWi-SllAafi

Cancel

jLayup Definition

I>M.GBX-Civil- status L-^RBX-Civil-status •EBX-Organization- code —•EBX-I dentific ation-numb er •EBX- Affiliation- typ e •EBX-M e de cine-man •SLB-Service •GBX-Room-tjrpe • • • •

'—•RBX-Ro om-typ e s SLB-Regimen MSG-Infotmation PBT-OK PBT-Cancel

Figure 4: Hierarchy of the Interaction objects to be placed in the example.

The designer's point of view includes the adequacy of the generated user interface with respect to placement dimensions and mathematical relationships identified in Section 1.2, the automation level (computer-aided or full automatic), the flexibility (capability to adapt the placement on the fly), the tailorability (ability to tailor a given placement according to a personal conventional scheme), and the level of assistance and control provided by the strategy and/or the tool implementing the strategy. The human factor expert's point of view includes the adequacy of the generated user interface with respect to visual techniques. These visual techniques namely encompass: • screen density: how unused blank spaces are optimized; • fragmentation: how IOs are imbricated, overlapping; • regularity: how regular is the layout grid; • consistency: how consistent are the placed 10 across multiples cases. The user's point of view is characterized by a subjective satisfaction, the visual appealing and the compatibility with the user's level of experience.

3.1.2. Example of a Placement Problem Each strategy will be applied on a small example taken in the field of a hospital admission application. The hierarchy of predefined IOs is depicted in Figure 4 with the appropriate information ordering to be respected. DBX denotes a dialog box, GBX denotes a group box, EBX denotes an edit box, RBX denotes a radio box, SLB denotes a scrollable list box, MSG denotes a message area and PBT denotes a pushbutton.

132

Jean Vanderdonckt, Missiri Ouedraogo & Banta Ygueitengar Date of the day Name Firstname Birth date Complete address Phone number Sex Civil status Organization Code Identification number: Affiliation type: Medicine man Service Room type: [Single room |Two beds room | Four beds room| Regimen: g ] Message:

Figure 5: The example resulting from UIDE.

4. Placement in UIDE UIDE (de Baar, Foley & Mullet, 1992) includes rules for mapping data objects belonging to the application design to 10 belonging to the user interface design. Two main steps are emphasized: the selection of 10 and their layout. UIDE's rules automatically lay out selected IOs in a parent window in order to create consistent layout which are compliant with a set of guidelines. These embedded guidelines constitute the first incursion of visual principles for producing a layout. A virtual grid is also used to insure proper 10 justification, separation and adequate window dimensions. No particular assumption is made for applying this strategy. The placement strategy mainly rests on a visual design with six principles: 1. vertical placement of IOs one after another in a row; 2. right justification of labels; 3. left justification of IOs; 4. regular space between labels and prompts, between prompts and IOs; 5. centering of standard push buttons; 6. normalized margins for all the four edges. From the designer's point of view, since the placement strategy is straightforward, the generation of dialog boxes is very easy and rapid (Figure 5). From the human factors expert's point of view, the localization and the dimensioning of IOs in a dialog box are very simple and the arrangement is more or less non existent since all IOs are vertically placed on the same level, therefore avoiding any visual structure, grouping, and continuity. However, left and right justification, proportional equilibrium implies a very clean, easy to read and to follow layout which is intrinsically kept consistent. If the number of IOs within a dialog box increases, the sizes of the composite 10 may become huge or too large to be managed into one separate composite 10. A lot of screen space is left unused. From the user's point of view, the best and cleanest placement does not necessarily mean the easiest to use.

133

A Comparison of Placement Strategies for Effective Visual Design AcUvtuui

I

•.-•%•

•

;

mip |

JMK

Figure 6: The example resulting from GENIUS.

5. Placement in GENIUS GENIUS (Janssen, Weisbecker & Ziegler, 1993) works similarly to UIDE except that: 1. the placement is governed by values input in property sheets (e.g. the data type, range, and condition values); 2. the placement is determined first for each 10 (or 10 group like a radio button with four items) separately, and, second, group by group; 3. the 10 sequencing follows the ordering of attributes specified in entities and relationships. No particular assumption is made for applying this placement strategy. It supports five principles: 1. vertical placement of IOs into one or two columns; 2. left justification of labels; 3. left justification of IOs; 4. vertical uniformity and equilibrium between IOs and groups of IOs; 5. total equilibration of the pushbuttons. Once again, from the designer's point of view, the resulting placement is fairly straightforward since all the IOs flow into one or two vertical columns. The rationale behind is that reading and 10 scanning are faster if they are vertical rather than horizontal. The message is processed like every other 10, not like a message. Flexibility is not supported, but the placement is provided instantly (Figure 6).

134

Jean Vanderdonckt, Missiri Ouedraogo & Banta Ygueitengar Title

Interaction objects

Interaction objects

First Column

Second Column

Figure 7: The layout grid in the two-column strategy.

From the humans factors expert's point of view, the spaces between IOs in each column are constant. Therefore, the composite 10 is very vertical. Thus, a lot of blank space is used and visual groups (e.g. the Patient group) are not visually isolated. This system seems to work efficiently for generating dialog boxes with a moderate number of IOs (typically, not more than 20). What will happen if the number of IOs increases? First, this can potentially lead to a vertically-arranged dialog box where placement is purely sequential, that is no appropriate placement according to the user's task. Second, the dialog box could grow like an awful stack of IOs.

6. Placement with Two-Column Strategy The rationale behind the two-column strategy is to equally distribute IOs into two columns, but with different vertical equilibration (Bodart & Vanderdonckt, 1994). This strategy is therefore a bit more general than GENIUS'S strategy. In fact, proportional and total equilibrations are considered as the most complete and aesthetic spatial relationships. These relationships are also the most difficult to apply for the two columns since different types of IOs are involved. The summarized steps of the two-column strategy (Bodart & Vanderdonckt, 1994) are the following: 1. the characterization of the 10 sequence: the contents of all IOs are identified (identification and descriptive labels, prompt and field); 2. the fixation of visual parameters: recommended standard space between two lines or rows, between label and prompt, prompt and field, font height; 3. the computation of standard dimensions of the IOs: all IOs are first pushed in a vertical stack whose dimensions are calculated recursively as the sum of the heights of individual IOs; 4. the computation of the dimensions of the two columns: the above stack is bipartitioned by finding regular proportions between two balanced columns; IOs are distributed in each column with respective dimensions and vertical uniformity/equilibrium if possible; 5. the computation of the internal proportion: after adding title, message area and separators if necessary, the current proportion is computed; 6. the adding of standard and custom pushbuttons, drawn buttons, and icons (if any).


135

Admission Date of day: •Patient

Name: Firstname: Birthdate : Complete Address : Phone number : Male O Female i—Civil statue— • Unmarried OMarried O Windowed O Divorced

Organization code I I Identification numbed I Affiliation type I I Medecine man : I I

Service : I -Room type 9 Single room O Two beds room O Four beds room

I |T]

Regimen

Figure 8: The example resulting from the two-column strategy.

The goal is to choose whether these buttons will be arranged in column on the right or in a row at the bottom of the composite 10. These buttons are automatically dimensioned with total equilibrium; 7. the selection of appropriate margins: margins result from an optimization problem based on the internal proportion which is forced to converge to one of the proportion recommended by Marcus (1992). The layout grid to be reached by the two-column strategy looks like the one represented in Figure 7. This two-column based strategy produced the dialog box of Figure 8. This strategy is now analyzed: • flexibility: the strategy provides a tree with physically arranged IOs and supports the information ordering since the 10 ordering reflects the information ordering. This strategy is rigid since it proposes a single version of the future dialog box according to a constant layout; • button placement: the strategy always places buttons either horizontally at the bottom of the dialog box or vertically on the right of the dialog box. Though always consistent and visually isolated, these placements require a lot of space; • column placement: the strategy attempts to share out IOs equally in the two columns to preserve balance and repartition. Thus, if the number of IOs is high, a visually appealing dialog box might result; if the number of IOs is very low, a flat unappealing dialog box might be viewed to the point that horizontality becomes irrelevant; • proportion: the strategy uses a convergence to a recommended proportion. This could resize global dimensions if the current dialog box has bad dimensions, but this could potentially lead to an increase of the global dimensions. But the dimensions are large regardless the unused grayed regions showed in Figure 8. • justification: the strategy automatically establishes all possible justifications increasing the feeling of unity;

136

Jean Vanderdonckt, Missiri Ouedraogo & Banta Ygueitengar ADMISSION

MSG-lnformation ,. Patient .

(41)

Complete address:

Service:

(31)

Regimen:

(30) j

Firstname:

(30)

Organization code:

Birthdate:

(29)

Identification number :

Name:

(27)

Affiliation type:

Phone number:

(25)

Date of day:

(29) | (29)| (26) | (24) |

r- Civil Status -

| • unmarried

(16) |

|Q single room

(25)

| • divorced (12) |

| g two beds room

(211

| • married (10}

|Q four beds room

an

[•windowed^) Medecine man:

(17) |

rSex llmale (13) | D female (12"T| I

OK

|

|

Cancel

|

Figure 9: The example resulting from DON.

• grouping: the strategy lays out IOs in each group recursively, thus preserving the grouping principle and the initial 10 sequence. On the other hand, this two-column strategy does not necessarily generalize particularly well to either the one-column or the multi-column case (as in a newspaper). This approach produces consistent layout but does not consistently produce visually appealing layouts. The two-column strategy lacks both generality and reusability in other contexts. This strategy have been so defined to conform to a predefined grid so that changing the number of columns, as recommended by Tarlin (1990), modifying the title or pushbutton location is not supported. The last goal for this strategy was to build a narrow, specific tool for form-filling applications, and therefore more sophisticated than a large general tool where precision and consistency could be endangered.

6.1. Placement in DON DON is more devoted to the layout generation and evaluation (Kim & Foley, 1990; Kim & Foley, 1993). All selected IOs subject to a dialog box are collected in a pool. At each step, the dimensions of IOs are examined two by two. IOs are laid out according to the longer, smaller or similar dimensions leading to groups of progressively decreasing sizes. We now analyse both strategies on different point of views (Figure 9): • flexibility: the strategy provides a tree with physically arranged IOs, but could generate an inconsistent IO ordering since the information ordering and/or structure is not taken into account. DON is not necessarily rigid since it suggests numerous placement versions;


137

• forms and dimensions analysis: only DOM deals with 10 dimensions. This generates a more aesthetic interface by minimizing the surface of remainder areas, but logically related IOs can be separated if their dimensions are not similar; • overflow solution: in the two-column strategy, this problem is supposed to be solved in the step of 10 selection. In DOM however, overflow solutions are performed only if necessary during the placement. If 10 dimensions and margins cannot be reduced enough to accommodate the IOs, DOM is forced to go back to the previous step of selecting IOs; • button placement: DOM's strategy is the most parameterizable and flexible. Moreover, buttons are placed wherever remainder areas are left; • column placement: DOM generates a placement where IOs are sorted by decreasing dimensions. On one hand, the designer benefits from improved layouts with minimized blank spaces and multiple possible layouts to choose from; on the other hand, the designer looses the information structure since the generation is only shape-based. DOM takes the visual 10 organization much more into account: balance, symmetry, space usage and density, size limits and ratio are immediately supported and evaluated. DOM automatically computes some aesthetic measures. It gives the designer complete feedback for deciding if a possible layout receives good scores for these measures. Combining automatic generation and evaluation of layouts have been proved very useful for providing the designer relevant assistance (Kim & Foley, 1993). One of the great feature of DOM is that the designer can choose a particular placement at his/her convenience if necessary and act with the editor at his/her convenience. As we saw, DON's strategy is very 'visual design' oriented, since only the shapes and dimensions of the IOs are taken into account for the placement. The limits of such a visual strategy could be highlighted: one can wonder to what extent (i.e. beyond conventional formfilling interaction styles) it makes sense to optimise the placement of IOs without considering the conversation on the user interface. For example, the conversation states, coding techniques, activation and deactivation of 10 by the user or the interactive application, dynamic transitions between IOs could be considered. We will see in Section 3.6 how dynamic task aspects such as the transitions between IOs may influence the arrangement of IOs.

6.2. Placement with Layout Appropriateness Sears developed a Layout Appropriateness (LA) method that can take advantage of a simple task description rather than a complete one (Sears, 1993). This metric is able to evaluate a layout, given different parameters detailing how users perform their tasks in the layout. These parameters integrate: the set of IOs, the sequences of performed actions, and their frequence. The LA metric is able to serve as basic measure for computer-aided searching of a best layout, called LA-optimal. LA metric remains the only one in its category. We will see now how the sequences of performed actions and their frequence may have an effect upon the placement. The two most frequent sequence of actions have been represented on the result of the twocolumn placement strategy (Figure 10). The single solid line represents the most frequent sequence of actions among the IOs. The double solid line represents the second most frequent sequence of actions. These sequences are not optimal because the user first types the name and firstname of the patient, then the other informations. Then, the patient's id. is provided because it is an identifier, but the 10 named 'Identification Number' is not the first of the

138


Start

Figure 10: The two most frequent 10 path on the two-column placement.

second column, and so on. The user finishes the form by providing the date of day when the patient informations are no longer necessary. Then the (Ok) pushbutton is pressed. We therefore conclude that the 10 ordering in the columns does not match the most frequent sequence of actions since the user has to come back several times. The LA strategy tries to overcome this situation. Figure 11 shows a LA-optimal layout for our problem. Several modifications have been introduced: • the most frequent accessed IOs are placed first; • the radio boxes (i.e. Sex, Civil status, Room type) have been re-arranged horizontally to reduce distance between the items; • the 'Female' item is placed before the 'Male' item, assuming that 52% of the population consists of females and 48% of males; • two new semantic groups have been identified due to the sequence of actions (i.e. Organization and Medical care); • the 'Date Of Day' edit box and the message area are put at the end of the layout since they are the least frequent IOs. Applying the LA-algorithm automatically leads to a placement in which the 10 scanning is minimal. This is the strongest point of the strategy, but it precludes that the 10 ordering by frequence is significant from the user's point of view. If the semantic structure of the information is not equal to the most frequent sequences of actions performed on these informations, the user may get lost, especially if these sequences of actions change from one user population to another. Otherwise, the user often prefers compact layouts such as in Figure 11. LA is useful for other purposes: • LA allows any size of 10, even though this seems to rely on the LA-optimal searching technique;

A Comparison of Placement Strategies for Effective Visual Design rPatientName : f_ Firstname: L~ Birthdate: Complete Address: Phone number: S e x : # Female O Male Civil status : #Unmarried OMarried O Widowed O Divorced

139

— OrganizationOrganization code : I I Identification number: I I Affiliation type : • I—Medical careService : L.

IHJ

Medecine man : | | Room type: #SingleQ Two beds O Four beds Regimen I IffH Date of day:

Figure 11: The example resulting from the Layout Appropriateness.

• the evaluation of appropriate dimensions for a particular 10 (e.g. the visibility of a list box, the preference for a drop-down object): for instance, if the 10 is frequently used and has small dimensions, a larger target could be suggested. Similarly, an 10 that is rarely used could be made smaller to save screen space; • the evaluation of a same layout manipulated by different populations of users, and not just one which is the most frequent; • the evaluation of a paper layout without requiring much user resources, programming efforts, and costly task descriptions.

6.3. Placement with Right/Bottom Strategy If the semantic structure of the information is not equal to the most frequent sequences of actions performed on these informations (the assumption for LA strategy is no longer valid) and if the semantic structure consists of a large amount of informations, another strategy that could be applied is the Right/Bottom strategy (Bodart et al., 1994). The main hypothesis of this strategy is to provide an initial tree of IOs to be laid out. The IOs are then arranged by placing the next 10 at the right or at the bottom of the previous one. The algorithm is summarized as following: */ the total length does not exceed the limit then place Si+1 with horizontal sequencing three cases are to be considered 1. height (Si) = height (Si+1) apply proportional uniformization 2. height (Si) > height (Si+1) i/Si+1 = edit box then if Si = list box or edit box then apply bottom justification else apply upper justification 3. height (Si) < height (Si+1) if available space is sufficient then apply bottom justification else maximize upper justification else place Si+1 with vertical sequencing.


140

Patient Admission Dale of day

Organization code:

rPatienl Name :

Identification number: Affiliation type:

Fiittname :

Medicine man :

Biithdale :

Service:

Complete address :

r-Room typeyp Single O Two beds Q Four beds

Phone numbei:

rSex

Disorientation

The relationship between the problems of cognitive overhead and disorientation is strong and direct: it has been suggested that part of the disorientation faced by the user in current systems is caused by the cognitive overhead imposed on the navigation. This observation is denoted here by: Cognitive Overhead —> Disorientation. Charney (1987) for instance, explains that by giving the user control of the sequence to be read, hypertext burdens the reading process: users may become overwhelmed by the available choices and lose track of their position in the underlying network. Gygi (1990) has directly related the problems, leading to cognitive entropy, of managing the complexity faced by the users: as a result the user becomes disoriented. The PIO approach, described next, is based on the fact that a reverse effect of this relationship is expected to occur: by diminishing cognitive overhead conditions the disorientation problem is reduced.

3. The PIO Approach When interacting with a hyperdocument, the user often performs tasks in a cyclic mode, as illustrated in Figure 1. This user's cyclic set of tasks corresponds to the operations which lead to the occurrence of one transfer of contexts.

148

Maria da Graga Campos Pimentel

\ Context initial identification of browsing alternatives choice of one activation of the corresponding operation analisys of the results

Context resulting

{ Figure 1: Transfer of contexts in the user's cyclic set of tasks.

The main objective of the Previewing Information Operation (PIO) approach, described below, is to tackle some of the causes of cognitive overhead imposed on the user whilst interacting with a hypertext system. Some overheads related to the user-hypertext interaction can be summarized: • Transfer of contexts in hypertext. Interacting with the system facing context transitions is likely to be the main activity of a hypertext user. For example a user, when selecting a link, is transferred from the context containing the source of the link, to the context containing the destination of the link. The cognitive effort demanded for overcoming the associated transfer of contexts is repeated at each navigational operation. • Identification of the link type. Most systems have at least two distinct types of hypertext link. Therefore the user has, when identifying the link, to distinguish among the available types, to assess what the resulting information would be before deciding whether or not to select the link. • Identification of the source of the link. An overhead may be related, for example, with the processes of understanding the source and/or scope of a link. An example was given by Shneiderman et al., when illustrating the problems related to the attachment of links in graphical information: "in a map of the United States, does a particular icon refer only to New York City, or to the entire state of New York?" (Shneiderman etal., 1991, p . l l ) • User's commitment. It is a burden to the user to be conscious that, once the selection is performed, the resulting context will have to be analysed before deciding whether or not to continue through that branch of the navigation. • Undo procedure. Undoing of navigational operations in hypertext is a crucial point to permit the user to explore the hyperdocument. In the case of the user deciding not to continue through a branch in the navigation after selecting a link, the undo of this selection will have to be activated. This is another cognitive compromise in the navigation: in fact, if undo procedure is too demanding the user will be reluctant to explore. Ideally, the undo of each operation should be easily achieved so that, instead of being a problem to the user, it should represent a stimulus for the user to explore by using the operation.

Evaluation of Alternative Operations for Browsing Hypertext

149

• Temporary disorientation. When a transfer of contexts occurs, the user faces a situation of momentary disorientation before being able to access the contents of the destination information. This type of overhead has been named in this paper as temporary disorientation overhead. One of the first tasks for the user is to identify what the new information is, and what information is no longer available. In a frame based system like KMS the identification may represent a straightforward task: in most (navigational) operations the whole frame containing the activated link is replaced by the new frame. In a multi-window platform, as for instance NoteCards, the user has either to position the new window first (an additional overhead: to find a convenient position) or to recognize what the target window is. Situations where the new information is inserted in situ, as in Guide (Brown, 1989a), may also represent an overhead for the user when it is necessary to recognize what is or is not new after the insertion has been done. • Understanding the relationship established by the link. The user has to identify what relationship exists between the source and destination information presented, a relationship which may not have been made totally clear in the context containing the source of the link (Landow, 1991). Prior to that, it may be necessary for the user to understand what the destination information itself means.

3.1. Approach The goal of the PIO approach was to affect the chain Cognitive Overhead —> Disorientation by reducing some of the causes of the cognitive overheads discussed above. The approach adopted was to offer the user two alternative ways of activating browsing operations associated with context transitions. Figure 2 presents a scheme of the normal operation in Part (a), and the schemes of the two alternative operations proposed: PlO/Consult in Part (b) and PIO/NoGo in Part (c). Part (a) shows that, during the standard operations, the user is transferred from source to destination contexts at once. When the PIO operations are activated, as indicated in the middle block in Parts (b) and (c) of Figure 2, the user enters the previewing stage and gains access to the previewing information. At the previewing stage, the user has interchangeable access to both source- and destinationrelated information by flipping between them. The contents of the previewing information are intended to help, for instance, clarify the semantics of the link and/or to reduce temporary disorientation. The difference between the PlO/Consult and the PIO/NoGo is that, in the former, the user may leave the previewing stage by choosing either to complete or to abandon the selection. In the PIO/NoGo operation, however, the user is not allowed to complete the transition: the only possibility is to return to the original state. The purpose of the operation is to offer a safe option for the user, whereby only access to further information on the link is permitted. In the experiment described below it is interesting that, although the PIO operations made the interface more complex, this did not prevent their use. The PIO approach is discussed in detail in (Pimentel, 1994), along with the implementations described below.

150


normal

source

resulting

link

context

context

selection (a)

destinationrelated source PlO/Consult

i

context

complete

resulting context

(b)

abandon

h

sourcerelated information

previewing stage

destinationrelated information

PIO/NoGo

source

resulting

context

context \

(c)

return

sourcerelated information

Figure 2: The PIO operations.

3.2. Implementations Implementations of the PIO approach were carried out in both a card-based hypertext platform, HyperCard (Apple Computer Inc, 1987), and a scroll based platform, UNIX Guide (Brown, 1989a). The experiment described in this paper was done using a standard version of UNIX Guide and a research version, which implemented the PIO approach for a number of hypertext facilities. In the PIO version, each type of button in Guide — replace, glossary and action — was associated with previewing information. If the user pressed and held the mouse button (instead of simply clicking it) in order to select the link, the destination-related information would be presented. (Refer to the figures at the end of the paper.) The results of both normal activation (click on mouse button) and the destination-related information associated with each type of the hypertext button is summarized below:


151

• Replace button: normal selection causes the associated replacement text to be expanded in situ, and to be presented to the user as ordinary information. The destination-related information consists of the presentation of the replacement, highlighted, while the user kept the mouse held down. Refer to Figures 3 to 6 at the end of the paper. • Glossary button: normal activation causes the definition of the button to be presented on a separated view at the bottom of the current window. During the previewing stage, i.e. while the user holds the mouse button down, the user is presented with a pop-up window containing the associated definition (refer to Figures 7 and 8). • Action button: normal activation results in the execution of a previously programmed action. During the previewing stage, additional information on the action to be performed is presented in a pop-up window while the user holds the mouse button down. In all the alternatives above the user still had access to the source-related information, which consisted of the information as it was before the activation of the link, with the addition that the hypertext button was presented highlighted (as in Figure 5): this was achieved by moving the mouse-cursor to the left. The viewing of the source- and destination-related information could be repeated as many times as wished. In the case of the PlO/Consult, when a decision was taken about whether or not to select the link, the user would release the mouse button whilst the corresponding context was presented.

4. Experiment The approach adopted in the experiment was to: • observe if there were any differences between the time taken by the groups to perform a given task; • observe significant differences and similarities occurring between the two groups, with regard to the evaluation of the systems; • observe the usage of the PIO generally. This paper presents data concerning the time taken to perform the task, and the main results obtained with regard to comparisons between the two groups, as well as within the PIO group.

4.1. Method 4.1.1. Subjects The 30 subjects participating in the experiment were all volunteers. Most of them were beginning a Master of Science Computing conversion course at the Computing Laboratory at UKC. The aim of targeting this group is that they represent a population which does not have Computer Science as a main background but which, at the same time, is interested in applied aspects of computing to aid their careers. Moreover, a mixed experience with computer usage and user-computer interface styles is to be expected. The subjects participating in the experiment were divided into two groups: the first using the standard version of the UNIX Guide system, and the second group using the modified version of Guide. From now on the standard version shall be referred to as the Control version, and the modified version as the PIO version. The subjects were randomly allocated to the groups and were equally likely to have had the same general experience concerning computer usage.

152


4.1.2. An Overview of a Session Each experimental session was composed of the following phases: • Informal background consultation. Initially the subjects were asked questions about their previous contact with different user interface styles (e.g. Macintosh and Windows), as well as previous contact with a hypertext system or other relevant computer software. • Introduction to the hypertext concept and introduction to usage of the system. These two phases were integrated: the subjects were introduced to the hypertext concept by using Guide and, at the same time, were told about hypermedia and other hypertext interface platforms, particularly multi-window and frame-based systems. The form of the presentation was standardized as much as possible, in an effort to keep bias to a minimum. • Free browsing. The user was presented with a version of the hyperdocument Xtutor, a tutorial on the X Window/twm interface for novices. This tutorial, a hierarchically structured document, was chosen for several reasons. First, it was a ready and available real word hyperdocument (Strang & Tardivel, 1992) — the version used had an introductory session on how to use the hypertext facilities removed. Secondly, it had already been used in the context of an experiment (Langford, 1994); as a consequence it was known that users should be able to go through part of the document without any problems, even in the short period of time available. Thirdly, its contents presented useful information for the majority of the subjects who volunteered for the task: later they would have to use the X Window environment at UKC. • Task directed browsing. In this phase the task/questionnaire set was handed to the subject, and explained. Some subjects might possibly have already used the Xtutor hyperdocument before. In order to limit the help that this may have given to these subjects, it was decided that the task should involve the search for information within respect to the structure of the document. The subjects were asked to try to find the answers to the questions presented in the task sheet by navigating in the hyperdocument. This would require them to navigate within several parts of the document, using the three main types of hypertext link available in Guide. They were then asked to fill in the questionnaire. The time to complete the task was recorded and the results are discussed later in Section 4.3. The first two volunteers were considered as pilot subjects for the experiment: one used the Control version and the other used the PIO version. Observation of their difficulties was very useful in adjusting the planned times of each phase, as well as identifying problems with the task and questionnaire. Both task and questionnaire were revised in the light of the pilot subjects' performance. • Questionnaire. The questionnaire was divided into four parts: 1. contents and concepts; 2. computing expertise, age and English language domain; 3. evaluation of system used, answers within a 1-7 scale; and 4. evaluation of system used, open questions.


153

Parts 1 and 2 were aimed at identifying the background experience of the users. Part 3 of the questionnaire requested the users to state their evaluation of the hypertext structures used (for example, if the different buttons types were easy to use and to understand), as well as the whole system (e.g. whether the system was easy to use or whether it was difficult to remember what the last steps were). In Part 4 the users were asked to produce general comments on the system (for example, what the most difficult or helpful aspects of the system were). 4.2. Results Concerning Sample and

Population

Data from the questionnaire show that both groups had roughly the same previous contact with the contents of the tutorial and with the hypertext concept; about the same age (mean for total group was nearly 28) and classified themselves as having about the same level of computer expertise (the MSc students were finishing the first month of their conversion course). Also of interest was information concerning the subjects' previous experiences with various user-computer interfaces. Data obtained in an informal interview carried out at the beginning of each session confirmed that there was no significant difference between the two groups. The overall data supports two important points in the analysis of the experiment. Firstly, the users in the two groups had approximately the same background as far as the user interfaces considered were concerned. Secondly, the whole group, containing novices and experienced users, as well as several levels in between, is representative of a diverse population. 4.3. Results Concerning Task Time Between

Groups

The time required to perform the task was recorded manually. The mean time for performing the task for the PIO group was about 27% less than for the Control group (means 7.25 and 9.93 minutes, respectively). An analysis of the variance, done by means of a t-test, gave as results: t = 1.83, df = 24, p = 0.08 < 0.1 (2-tailed). Although these results are not considered to be statistically significant for most studies, the p = 0.08 value supports a trend towards an important difference. The fact that the PIO group encountered a more complex interface has to be considered. Although the aim of the PIO operations is to help the user, the first contact with the interface is quite complex: the user has to learn what all the possible options are and how to use them. This point has been, in fact, mentioned in the users' comments. It is clear that an interface where the selection of a link is done by just a click demands less learning effort on part of the user than the several options offered by the PIO operations. This effort is expected to diminish as the user gains experience with the interface. Furthermore, the PIO group has also dealt with the Smooth scrolling operation. The research version had implemented, besides the PIO operations, a mechanism to control the scrolling of the text by pointing on a text area and pressing a mouse button. This would activate a continuous scrolling of the text which could be controlled by moving the cursor upwards and downwards. 4.4. Results From a Study Between Control and PIO

Groups

Results have shown that there were no significant differences between the two groups with regard to the measurements obtained by the users' evaluation of the system. Nevertheless, the indication of a trend by the analysis of the time taken to perform the task supported the

154


A

The system was easy to use

B C D

The system was easy to use It was easy to undo a replace button You 'knew where you were'

E

You 'knew where you were'

The replace button facility was easy to understand It was easy to undo a replace button It was easy to undo a glossary button It was difficult to decide whether or not to select a button It was easy to recognize what the new information was when selecting a button

Control

PIO

.1266

.5590

t

.3662 .1806

.7139 .6846

X

-.0734

-.6162

.4886

.6389

t t t

tsign < 0.05, \ sign < 0.01 (2-tailed) Table 1: Correlations.

undertaking of additional comparisons between the two groups. A number of correlation analyses were conducted; the results, presented in Table 1, are discussed in the following sections. 4.4.1. Correlation (A) The presentation of the previewing information may have influenced the users in their evaluation of how easy it was to use the system. The ease of understanding the link relationship was one of the main goals of the design of the PIO operations. The significant correlation, found in the PIO group and absent in the Control group, provides evidence for the suggestion that the goal has been achieved.

4.4.2. Correlation (B) Among the requirements for the design of the PIO was that the user should be offered an easy way to abandon the link selection. Cognitive overheads would be reduced if the operation could make it easier to undo a selection and to encourage the user to explore the available links. Therefore another of the main goals of the design of the PIO was that the user should have link selection operations that were easy to undo. The fact that correlation (B) is significant to the PIO group only suggests that the availability of the PIO operations concerning the undoing operations have influenced the user's rating of how easy it was to use the system.

4.4.3. Correlation (C) This is another relationship found significant within the PIO group only. It is interesting because, for the PIO group, the undoing of distinct types of buttons is achieved in the same way (when only the previewing perspective is considered). Hence, as the linear relationship is absent for the Control group, the subjects in this group may have treated the selection of the two types of button as distinct operations. The advantage of the PIO approach is that, by being able to treat the interfaces similarly, the operations are seen by the users as consistent and, as a consequence, more natural to use.


A B C D E

Dependent

Independent

PIO: all PIO/NoGo: all PIO/NoGo: replace PIO/NoGo: glossary Undo: all

Click: all Undo: all Undo: finished Click: glossary Click: all

r

adj

F

.26 .25 .23 .26 .47

5.96 5.46 4.99 5.96 13.25

155 SignF

.0297 .0361 .0436 .0297 .0030

.05 .05 .05 .05 .005

Table 2: Regressions within the PIO Group. Notes: • PIO: all corresponds to all PIO operations — PIO/Go (holding time longer than .60s), PIO/Back, PIO/ConsultGo, PIO/ConsultBack, PIO/NoGo — considered all types of hypertext buttons; • PIOINoGo: all corresponds to the user of the PIO/NoGo upon all link types; • PIOINoGo: replace corresponds to the user of the PIO/NoGo upon replace button. Ditto for PIOINoGo: glossary; • Undo: all to the sum of the undo operations performed (finished or abandoned); • Undo: finished to the sum of thefinishedundo operations performed; • Click: all considers all standard click selections issued (holding time less than or equal to .60s).

4.4.4. Correlation (D) The main hypothesis of the PIO approach is that it would make it easier for the user to make link selections, and this would reduce the transfer of context and temporary disorientation cognitive overheads. This would reflect on the chain Cognitive Overhead —> Disorientation. The significant relationship (D) suggests that the PIO approach, by helping with the task of deciding whether or not to select a link, has also influenced the users' feeling of orientation during the navigation.

4.4.5. Correlation (E) By placing the previewing information under the user's control, the temporary disorientation overhead related to the selection of the link was expected to be reduced. The relationship (E) suggests that the overall user's orientation was also correspondingly affected by the rate of how easy it was to recognize the new information.

4.5. Results From Study Within PIO Group Background recording implemented in the version used by the PIO group made it possible to obtain a record of the selections performed (including details of the operation used and the type of button), as well as a record of the undo operations activated (both finished and abandoned). Regression analyses were conducted to find out whether standard navigation operations would be predictors for the alternative PIO operations. The results, summarized in Table 2, are presented next.

156


4.5.1. Regression (A) The PIO operations, considered individually, were not predicted by the standard operation of selection (click). Nevertheless, when all operations are considered, the results indicate that the number of click selections performed is a predictor for the number of PIO operations issued by the user. This is probably one of the most important results obtained by this experiment, since it states that the PIO operations, as a whole, will represent a significant part of the operations used for link selection.

4.5.2. Regression (B) The PIO/NoGo, all button types considered, is predicted by the number of Undo operations issued (finished or abandoned). As the PIO/NoGo presents the destination of the link and automatically performs the undo, they represent an exploratory selection by the user. The proposal of the undo operation, in general, not only has a corrective aim but also an exploratory one (Thimbleby, 1990). The result indicates that the PIO/NoGo will represent a significant part of the undo operations used in the navigation. This is further confirmed by the result presented next.

4.5.3. Regression (C) The number of PIO/NoGo issued upon replace buttons (the most commonly used type) was predicted by the number of effective (not abandoned) undo operations. The PIO/NoGo operation has an exploratory aim, and so has the effective undo operation. The result further reinforces that the user, when exploring, will use the PIO/NoGo as an alternative operation to undoing.

4.5.4. Regression (D) The most specific predictor found was related to the glossary button: the number of PIO/NoGo operations issued by the user is predicted by the number of click selections. This result indicates that, for the glossary button type, the PIO/NoGo would represent part of the effective operations of selection issued (in contrast with the exploratory selections indicated in the two previous cases). This result confirms the user's tendency of effectively using the PIO/NoGo operation when the destination information is going to be presented for a short period of time.

4.5.5. Regression (E) The result (E) states that the number of standard click selections was, as expected, a predictor of the number of (abandoned or completed) undo operations issued. Some related data available in the literature, concerning the KMS frame based system, states that: "Backtracking is a frequent activity — for every move forward there tends to be a compensating move back." (Akscyn, McCracken & Yoder, 1988, p.830)


157

The regression (E) suggests that the number of performed undo operations is about half the number of selections achieved by click only. There are differences relative to the interface platforms, and to the availability or not of the PIO operations; this in effect does not permit direct comparisons between these two results. It would be interesting to investigate results concerning the number of click selections and undo operations in a Control group. This would permit a comparison of the relationship between the number of undo operations and click operations depending on the availability of the PIO operations. Moreover, it would be interesting to have such investigations carried out in a frame-based interface. The work reported in (Pimentel, 1994), referring to the implementation of the PIO operations in the HyperCard system, makes it possible for future investigations into these factors.

5. Conclusion Work related to the previewing approach is the implementation of the description note in Hyperties — which offers the user an option to abandon the selection after seeing a resume of the destination (Shneiderman, Kreitzberg & Berk, 1991), and the fast undo in KMS — which offers fast response on selection and backtracking as alternative to previewing (Akscyn, McCracken & Yoder, 1988). Wright observed that changing screens to access information only a click away can impair the reader's memory processes, even if the related undo is achieved by just another click (Wright, 1991). The PIO approach extends the resume and undo models by: a. applying the PIO approach to any browsing operation leading to context transition; b. presenting both source- and destination-related information in the previewing stage; c. complementing the options available by defining the PIO/NoGo alternative. In the experiment described, users have not only used the alternative operations, but have also found their interaction facilitated by the presence of the previewing mechanisms. These results suggest that the purpose of the PIO approach has been achieved: focus on the user-hypertext interaction has diminished the overall navigation problems. Future research is planned towards evaluating the previewing approach to secondary navigation modes in hypertext, as well as other interactive systems in which the user is involved in tasks leading to transfers of contexts.

Acknowledgements This research was carried out whilst the author was a PhD student at the Computing Laboratory, University of Kent at Canterbury, UK, under the supervision of Prof Peter J Brown. Dr Mike Forrester, Dr Duncan Langford and Ana Torres helped on the several phases involving the experiment described. Mike also made major contributions to the first version of this paper. This work has been supported by CAPES-Brazil under grant 813/90-8. (From August 1994, the author can be contacted at the Brazilian address.)

158


References Akscyn, R M, McCracken, D L & Yoder, E A (1988), "KMS: A Distributed Hypermedia System for Managing Knowledge in Organizations", Communications of the ACM 31 (7), pp.820-835. Apple Computer Inc (1987), HyperCard User's Guide. Baird, P & Percival, M (1989), "Glasgow Online: Database Development Using Apple's HyperCard", in Hypertext: Theory into Practice, R McAleese [ed.], Intellect, pp.75-92. Beeman, W O, Anderson, K T, Bader, G, Larkin, J, McClard, A P, McQuillan, P J & Shields, M (1987), "Hypertext and Pluralism: From Lineal to Non-lineal Thinking", in Proceedings of Hypertext'87, ACM Press, pp.67-88. Bernstein, M & Joyce, M (1992), "Contours of Constructive Hypertexts", in Proceedings ofECHT92, ACM Press, pp. 161-170. Brown, P J (1989a), "A Hypertext System for UNIX", Computing Systems 2(1), pp.37-53. Brown, P J (1989b), "Do We Need Maps to Navigate Round Hypertext Documents?", Electronic Publishing: Origination, Dissemination and Design 2 (2), pp.91—100. Charney, D (1987), "Comprehending Non-linear Text: The Role of Discourse Cues and Reading Strategies", in Proceedings of Hypertext'87, ACM Press, pp.109-120. Conklin, J (1987), "A Survey of Hypertext", Microelectronics and Computer Technology Corporation, Austin, Texas, USA, Tech Rep STP-356-86, revision 2, this is an expanded version of IEEE Computer 20(9), pp.17-41. Dillon, A, McKnight, C & Richardson, J (1993), "Space — The Final Chapter or Why Physical Representations are not Semantic Intentions", in Hypertext: A Psychological Perspective, C McKnight, A Dillon & J Richardson [eds.], Ellis Horwood, pp.169-191. Frisse, M, Cousins, S B & Hassan, S (1991), "WALT: A Research Environment for Medical Hypertext", in Proceedings of Hypertext'9], ACM Press, pp.389-394. Gygi, K (1990), "Recognizing the Symptoms of Hypertext and What to do About It", in The Art of HumanComputer InterfaceDesign,B Laurel [ed.], Addison Wesley, pp.279-288. Halasz, F G (1988), "Reflections on NoteCards: Seven Issues for the Next Generation of Hypertext Systems", Communications of the ACM 31 (7), pp.836-852. Jones, R A & Spiro, R (1992), "Imagined Conversations: The Relevance of Hypertext, Pragmatism, and Cognitive Flexibility Theory to the Interpretation of "Classic Texts" in Intellectual History", in Proceedings of ECHT92, ACM Press, pp.141-148. Landow, G P (1991), "The Rhetoric of Hypermedia: Some Rules for Authors", in Hypermedia and Literary Studies, P Delany & G P Landow [eds.], MIT Press, pp.81-103. Langford, D (1994), "Evaluating a Hypertext Tutorial: An Examination of the Application Xtutor", CVCP/USDU, ITTI Project, UK, Tech Rep ISBN 1 85889 007 1, EMail order: [email protected]. McKnight, C, Dillon, A & Richardson, J (1989), "Problems in Hyperland? A Human Factors Perspective", Hypermedia 1 (2),pp.l67-178. Nielsen, J (1990), Hypertext and Hypermedia, Academic Press.


159

Pimentel, M G C (1994), "A Framework for User-Hypertext Interaction and Alternative Operations for Browsing", University of Kent at Canterbury, UK, PhD thesis, PostScript version to be available upon request. Remde, J R, Gomes, L M & Landauer, T K (1987), "SuperBook: An Automatic Tool for Information Exploration — Hypertext?", in Proceedings of Hypertext'87, ACM Press, pp.175-188. Shneiderman, B, Kreitzberg, C & Berk, E (1991), "Editing to Structure a Reader's Experience", in Hypertext/Hypermedia Handbook, E Berk & J Devlin [eds.], Intertext (McGraw Hill), pp.143-164. Shneiderman, B, Plaisant, C, Botafogo, R, Hopkins, D & Weiland, W (1991), "Design to facilitate browsing: a look at the Hyperties workstation browser", Hypermedia 3 (2), pp.101-117. Strang, W & Tardivel, G (1992), "Xtutor Hyperdocument", public computer archive available via anonymous ftp from unix.hensa.ac.uk:misc/unix/unix-guide/guidejdocs. Thimbleby, H W (1990), User Interface Design, Addison Wesley. Wright, P (1991), "Cognitive Overheads and Prostheses: Some Issues in Evaluating Hypertexts", in Proceedings of Hypertext'91, ACM Press, pp.1-12.

160

Maria da Graga Campos Pimentel El xtutor AN INTRODUCTION TO THE X WINDOW SYSTEM LEAVE this Tutorial

FORWARD /HACK

UneUP/l.lneDOWN

Return to START

Welcome to this short Introduction to the X Window System. You may select any one of the following options by moving the mouse cursor on to the option of your choice and operate the mouse according to the Instructions received. 1. What Is tho X Window System? 2. Demonstration of the X Window System 3. BASIC WINDOW OPERATIONS

3.1 WHERE ARE THE WINDOWS? 32 USING THE CURSOR 3.3 A GRACEFUL EXIT 4. The Mouse Menus 5. Further Window Operations

about this document

Figure 3: UNIX Guide showing Xtutor. Replace buttons are shown in bold in main area.

AN INTRODUCTION TO THE X WINDOW SYSTEM LEAVE this Tutorial

FORWARD/HACK

UneUP/LlncBOWM

Return to START

Welcome to this short Introduction to the X Window System. You may select any one of the following options by moving the mouse cursor on to the option of your choice and operate the mouse according to the Instructions received. 1. What Is the X Window System? 2. Demonstration ol the X Window System 3. BASIC WINDOW OPERATIONS [3.1 WHERE ARE THE WINDOWS On the screen there are several rectangles, each of which is a'window*. Their names and oosltlons will depend on the default settlnq of your host computer. Most provide three windows which display a shell prompt such as '%', Each of these windows is a terminal emulator, and has the name of the window in the top left corner. This text is displayed In a window called 'xtutor1. At the top right is another type of window, called the Icon window, which consists of horizontal bars each containing the name of a window. Some systems also create an "xaliaa window', a small rectangle, with the name 'xallas' across the top. The uses of these two windows will be described later] 32 USING THE CURSOR

Figure 4: Destination-related information of WHERE ARE THE WINDOWS?

Evaluation of Alternative Operations for Browsing Hypertext AN INTRODUCTION TO THE X WINDOW SYSTEM LEAVE this Tutorial

FORWARD /BACK

UneUP/LineDOWN

Return to START

Welcome to this short Introduction to the X Window System. You may select any one of the following options by moving the mouse cursor on to the option of your choice and operate the mouse according to the instructions received. 1. What is the X Window System? 2. Demonstration ol ths X Window System 3. BASIC WINDOW OPERATIONS

|3,1 WHERE ARE THE WIND0WS7I 3.2 USING THE CURSOR 3.3 A GRACEFUL EXIT 4. The Mouse Menus 5. Further Window Operations

about this document

Figure 5: Source-related information of WHERE . . . WINDOWS?

6

numeric

continuous

[1,60]

Introduction

Interaction object

read

edit box

write

label

simple

list box with simple selection

multiple

list box with multiple selection

simple

list box

multiple

list box

simple

scale

>60

read

edit box

write

label

Table 1: Decision table of selection rules in GENIUS.

UIDE (de Baar, Foley & Mullet, 1992) goes a step further by expressing selection by utilizing the production rules contained in decision tables. These tables are stored in a file which is editable by the designer. Selection rules therefore become modifiable. TRIDENT (Bodart et al., 1993; Vanderdonckt & Bodart, 1993) graphically represent selection rules in a selection tree. With visible selection rules, designers are allowed to see graphically how the rules are processed; to see why a particular interaction object has been chosen; and to act if necessary. Selection rules can then be categorized along four dimensions: semantic dependence, explicitness, modifiability, and visibility. DON, UIDE, GENIUS and TRIDENT illustrate that the current trend in automated user interface generation is oriented more to a knowledge-based approach to selection rules. In this approach, specific models are created of: the application domain, and sometimes the application itself; the interaction objects belonging to the user interface components; and the user model. For our purposes, two models will be introduced: a data model and an abstract interaction object model. In this approach, selecting interaction objects becomes a matter of writing mapping rules between application components and interaction objects components (Gray et al., 1992). Having provided inspiration, these references will now serve in defining widely applicable selection rules. Another potentially helpful source for identifying selection rules attributes is the information about selection guidelines found in many standards and styleguides. For instance, Section 9.1 of Bellcore's styleguide (Root & White, 1993), entitled 'Matching Controls to user Tasks', deals with controls for commands and operations, for 'n of Many' choices, and for '1 of Many' choices. Mayhew (1993) also recommends the simple selection guidelines shown in Table 2.

166

Frangois Bodart & Jean Vanderdonckt Interaction object

Circumstances

option buttons

in dialog boxes, < 4 choices, choose one, attributes

check boxes

in dialog boxes, < 4 choices, choose many, attributes

list boxes

in dialog boxes, > 4 choices or dynamically changing choices attributes/objects

drop down list boxes

in dialog boxes, in forms, when field has a reasonable number offixedchoices, attributes

command buttons

in dialog boxes, dialog control actions

toolboxes

in application windows, data attributes

Table 2: Selection guidelines from Mayhew (1993).

3. A Generalized Definition of Selection Rules 3.1. A Data Model The data model consists of an object-oriented model (Figure 1) where application data can be divided into seven data types (i.e. hour, calendar, boolean, graphical, integer, real, and alphanumeric). These data types are considered to be the most frequent in businessoriented applications (Bodart et al., 1993) (e.g. office automation, administration, database applications). However, they are also considered to be incomplete when working with other dedicated applications (e.g. graphical editors, engineering laboratories). Each piece of general data is composed of several attributes: • a data name: this is an identifier (e.g. PersonSize); • the data length: the number of characters, if relevant (e.g. 2); • an identification label: this unique label identify the data to input/output and its nature (e.g. Size of the person:); • a description label: this label specifies some constraints that must be verified by the data in order to be valid, such as a unit measure, a valid domain or interval; • the type of domain (as the limits in UIDE): whether the domain of the data is unknown, known, or mixed (that is a known domain with some unknown values the user can supply); • the mandatory status: whether the data is required or not; • the interaction (as introduction in GENIUS and content in UIDE): whether data is to be used as input, as display or used in both ways; • the number of possible values (Npo for short) (as the range of values in GENIUS): this number can be 1, 2 , . . . or N if infinite (Vanderdonckt & Bodart, 1993); • the number of principal values (Npv for short): the number of values which are most frequently used (Vanderdonckt & Bodart, 1993); • the number of secondary values (Nsv for short): the number of values which are less frequently used such that the sum of principal values and secondary values is equal to the number of possible values (Npo = Npv + Nsv);

167

On the Problem of Selecting Interaction Objects

General Data DataLength DataName DescriptionLabel Domain IdentificationLabel Interaction Mandatory NumberPossibleValues NumberPrincipalValues NumberSecondaryValues NumberValuesChoose Selected AIO DisplaySelected_AIO

AlphanumericData

GraphicalData

ExpandableDomain

BooleanData

CalendarData

Real Data

Orientation

Preference ScreenDensity UserLevel

Precision U . . l_ .

Figure 1: The object-oriented data model.

HourData

IntegerData

Preference ScreenDensity UserLevel

ContinuousDomain Orientation

168

Frangois Bodart & Jean Vanderdonckt • the number of values to be chosen — as in (Mayhew, 1992), GENIUS and UIDE: this can be 1,2, . . . or N if infinite. If this number is equal to 1, the data follows a simple course: after having chosen a piece of data, the user can only select one value at a time to manipulate that data. If the number is greater than 1, the data follows a multiple path course: the user may wish to select two or more two values simultaneously; • the selected interaction object: interaction model (see below).

a pointer to a suggested entry in the abstract

Different data types inherit the above mentioned attributes either by specialization (for example, the AlphanumericData object add the ExpandableDomain slot) or redefinition (for example, the domain is known for the GraphicalData objects, Npo = Npv = 2, Nsv = 0, Nvc = 1 for the BooleanData object). Specialization attributes include: • domain expandability (Exp for short): whether the user can add own values; • the precision (as in UIDE): whether the number of digits, or decimals is important; • the continuity of the domain (Cont for short): whether all values are spread in a continuous range of values (e.g. V x > 7, [1 . . . 10]); • the data orientation (as in DON): whether the data is more conveniently represented horizontally, vertically, circularly or in an undefined way; • the screen density: whether the amount of data measured by presentation unit is high, low; • the user level: whether the user experience level is beginner, novice, intermediate, expert or master; • the selection preference: whether the user has a preference (or physical skill) for typing in data rather than selecting it. The last three attributes normally do not belong to the specification of application data, but they are provided apart. They are still inserted in the case of hour and calendar data types because the semantics of these cases are better known than the others. Some attributes (e.g. the identification label, the mandatory characteristic) will not serve for selection. They are introduced here for specifying the data model and for final generation of user interface.

3.2. An Abstract Interaction Object Model An Abstract Interaction Object (AIO) Model (Vanderdonckt & Bodart, 1993) rather than a Concrete Interaction (CIO) Model is introduced to address three problems: 1. different AIO can be used for input/display a same class of application data; 2. several presentations (or CIO) can be found for one particular AIO: for instance, a same check box can have a lot of different presentations (Figure 2); 3. the AIO classification should be free of presentation considerations: behavioural considerations should be abstract across different platforms (Johnson, 1992). Defining an AIO involves abstracting the behaviours of CIO in order to be independent of a particular physical toolkit or library of interaction objects. Six sets of AIO have been identified: action objects, scrolling objects, static objects, control objects, dialog objects and feedback objects (Vanderdonckt & Bodart, 1993). These sets have been arranged in a hierarchical object-oriented model where each AIO is identified by a name, by general and


169

Figure 2: Different CIO for one AIO.

specific attributes (e.g. AT_AIOJLength), by abstract events and by primitives. This definition allows us to select interaction objects across a wide variety of computing platforms because the behaviour is considered in the abstract, rather than in terms of the different presentations.

3.3. A Generalized Definition of a Selection Rule The ordering of attributes in each selection rule has importance when writing decision tables. Attributes are sorted out from the most general to the more particular in order to minimize lines and columns in decision tables and to allow an iterative refinement of the ergonomic quality of AIO. The further right the attribute is placed, the more specific it is. A selection then has the general following form: IF Attribute^Valuei AND Attribute2=Value2 AND . . . AND Attribute n =Value n THEN AIO is [ AIO_name] At = I ... n Attributei=Value< => AIO=[ AIO_name] where Attribute^ is any attribute of the data model, AIO_name is the the designation for any AIO in the AIO model.

4. A Corpus of Decision Tables A corpus of decision tables has been developed according to a hierarchy of types of selection rules (Figure 3). Selection rules fall into two categories: rules for input data and rules for displaying data. Similar types of selection rules are found in both categories: rules for input/displaying elementary data (7 supported data types), rules for taking into account the physical environment, rules for input/displaying specific kinds of data, group of data, and list of data (elementary and composite). Selection rules for window and dialog box are included in the 'display' division. Discussing every decision table that might be applied would be impossible. Therefore we refer to the technical report where all tables are explained (Vanderdonckt, 1993). We would rather focus on some decision tables and examples in order to illustrate the main ideas which prevail in the creation of the tables. These ideas are expressed in the form of underlying principles for selection rules. In order to introduce, to define and to illustrate these principles, we propose to show decision tables for input of integer data (elementary).

170

Frangois Bodart & Jean Vanderdonckt input

• elementary data • environment consideration • specific data • group of data - elementary data in a list - list of data

•hour — calendar — boolean — graphical — integer — real — alphanumeric

- display

• elementary data - specific data • group of data - list of data . environment consideration . composite data

hour — calendar — boolean — graphical — integer — real — alphanumeric

— selection of window I— selection of dialog box Figure 3: Hierarchy of selection rules.

4.1. Example of Decision Tables The label of each column from Table 3 represents an attribute taken away from the data model except for the last one which represents the selected AIO which taken from the AIO model. Table 3 depicts selection rules for choosing AIO where the domain is known and the choice is simple (i.e. Nvc = 1). The first row can be read as: if the number of secondary values is strictly positive (Nsv > 0), then the selected AIO is a list box (Figure 4a). The second row can be read as: if there are no secondary values (Nsv = 0) and if the domain is expandable (Exp = yes), then the selected AIO is a combination box (Figure 4b,c). The third row means: if there are no secondary values (Nsv = 0) and if the domain is unexpandable (Exp = no) and if the domain is not continuous (Cont = no) and if the number of possible values is between 2 and 3, then the selected AIO is a radio button with 2 or 3 items (Figure 4d). These rules can be stated roughly in the same way, in this way respecting the generalized definition mentioned above. Table 4 exhibits selection rules for the input of an integer where the domain is known and the choice is multiple (i.e. the user can input several values: Nvc > 1). Tables 5 and 6 show selection rules where the domain of the data is unknown and mixed, respectively. 4.2. Underlying

Principles

4.2.1. Principle of Appropriate Item Division This principle aims at reducing information division by applying the appropriate AIO. Particular AIO are more appropriate for small, middle or large pools of items. For example, a radio button is more suited to a small number of mutually exclusive items; a simple list box is more convenient for an intermediate number of items while a scrolling list box (i.e. a list box with fast scrolling performed by double arrow buttons) is more appropriate for a large number of items. The reason a radio button is not suited to this situation is that it leads to a large interaction object. To solve this problem, DON suggests a list box if the number of items is greater than a cutoff-constant = 8 (Kim & Foley, 1993). GENIUS (Weisbecker, 1993) and Mayhew (1993) prefer to limit it to '> 6' (Table 1) and to '> 4' (Table 2), respectively. We

On the Problem of Selecting Interaction Objects Nsv

Exp

Cont

Npo

Precision

Orientation

>0 =0

171 AIO list box combination box

yes no

no

yes

[2,3]

radio-button with Npo items

[4,7]

radio-button with Npo items + group box

[8, Tm]

list box

[Tm+l,2Tm]

scrolling list box

>2Tm

scrolling drop-down list box

[1,10]

low

high

[11, Tm]

>Tm

vertical

scroll bar

horizontal

scale

circular

pie diagram

undefined

scale

vertical

vertical thermometer

horizontal

horizontal thermometer

circular

dial

undefined

horizontal thermometer

high

spin button

low

scale

high

spin button

low

vertical

scroll bar

horizontal

scale

circular

dial

undefined

scale

Table 3: Selection rules for the input of an integer where the domain is known and the choice is simple.

propose to select a radio button if the number of possible values (Npo) is bounded by 2 and 3 (Table 3, Row 3). A group box is then added if Npo is between 4 and 7 in order to clearly build a visual group (Table 3, Row 4). If the domain is expandable, an edit box is placed beneath the known values (Figure 5) (Table 6, Row 2). Beyond the limit of 8 items, a list box

172

Frangois Bodart & Jean Vanderdonckt Nsv

Exp

Npo

AIO

-0

no

[2,3]

check boxes

[4,7]

check boxes+group box

[8, Tm]

list box

[Tm+l,2Tm]

Scrolling list box

>2Tm

Scrolling drop-down list box

yes

Combination box

>0

List box

Table 4: Selection rules for the input of an integer where domain is known and the choice is multiple.

Domain

AIO

unknown

Profiled edit box

Table 5: Selection rule for the input of an integer where domain is unknown.

Npo

AIO

[2,3]

radio button with Npo items + edit box

[4,7]

radio button with Npo items + edit box + group box

[8, Tm]

drop-down list box

[Tm+l,2Tm]

scrolling list box

>2Tm

scrolling drop-down list box

Table 6: Selection rule for input an integer if domain is mixed.

MIDI Chanel:

Bits :

MIDI chanel:

Bits : O 4

5 6

Figure 4: Some examples of selection rules applications.

is recommended. The use of list box is also limited by Tullis' constant Tm = 50 so that if the Npo exceeds this constant, a scrolling list will be preferred instead.

173

On the Problem of Selecting Interaction Objects Receive Frequency O 29

MHz ( 10 n )

<S> 145 MHz ( 2 n ) O i»35 MHz ( 70 cm )

Figure 5: A group box surrounding a radio button and an edit box.

LlllllL Figure 6: (a) pie diagram; (b) horizontal thermometer; (c) vertical thermometer; (d) scale.

Date' 06. 09. 1993 Figure 7: (a) A dial; (b) A spin button.

4.2.2. Principle of Adapted Precision and Orientation The goal of this principle is to adapt AIO to the relative precision or orientation the user wants in inputing or displaying data. For example, Row 10 in Table 3 states: if there are no secondary values (Nvs = 0); if the domain is not expandable (Exp = no); if the domain is continuous (Cont = yes); if the number of possible values is between 1 and 10 (Npo G [1,10]); if the precision is low (Precision = low); if the orientation is circular (Orientation = circular), then the selected AIO is a pie diagram (Figure 6a). If the orientation is left undefined by the designer, a scale would be selected (Figure 6d) (Table 3, Row 11). If the semantics of the data specify high precision and a familiar horizontal representation (e.g. a duration, a period of time), a horizontal thermometer would be selected (Figure 6b) (Table 3, Row 12). If the familiar representation is vertical (e.g. a temperature, a potentiometer), a vertical thermometer would be selected (Figure 6c) (Table 3, Row 13).

4.2.3. Principle of Appropriate Interaction Style Here the goal is to promote compatibility with respect to research that has considered how to choose the appropriate interaction style for certain kinds of data. For example, experimental

174

Frangois Bodart & Jean Vanderdonckt ALT+F10 ± ALT+F9 LTD a in ALT+F11

Figure 8: A list box replaced with a drop-down list box.

studies have supported the hypothesis that when task involves an elementary calendar input, fill-in field techniques are faster than direct manipulation, even in the case of inexperienced users (Mayhew, 1993). When the same task involves the same type of data, but with difficult values, direct manipulation entry was indicated to be faster. In the same way, a dial seems to be of more use for entering an integer if orientation is circular (Figure 7a) (Table 3, Row 14) rather than a rotator.

4.2.4. Principle of User Adaptability As the name indicates, the purpose of user adaptability is to select AIO which offer user friendly performance with improved user guidance and feedback if the user level of experience is low (e.g. a spin button rather than an edit box: Figure 7b). As user proficiency improves (i.e. as the user level increases), AIO that operate more quickly and demand more agile manipulations are adopted (e.g. a profiled edit box). Taking the case of a user who prefers to click up an item from a list rather than typing it in a field, AIO that allowed direct manipulation would be favoured.

4.2.5. Principle of Screen Density Reduction The goal of this principle is to reduce the screen density as much as possible. The screen density might be evaluated by calculating whether the number of visual groups exceeds the user level's tolerancy limit. Should this limit be exceeded, the surface area of the groups might be reduced, thus satisfying screen constraints as well as avoiding human perception saturation. Some particular AIO can thus be down-graded.

4.2.6. Principle of Environment Consideration The environment consideration principles defines selection rules for considering the physical environment as an extension of the two previous principles. Thus, if considerations about physical presentation make it desirable, replacement ad reduction rules permit the substitution of an already selected AIO for another. These rules can be applied everywhere. For example, every list or combination box, scrolling or not, graphical or not, can be replaced with an equivalent drop-down list if the screen density is high or if the user prefers the selection method or if the user is inexperienced (Figure 8).

4.2.7. Principle of Specific Data Consideration The goal of this principle is to define selection rules for considering the specific data in peculiar or extraordinary situations. Eighteen similar rules have been gathered. However, they have not been incorporated into the described data model since they would introduce the specificity of many different contexts which are difficult to express in a model. For example, a scrolling cursor (Figure 9) might be suggested in a case in which an integer is to be entered into a continuous domain when the data is intended to represent the settings of an apparatus.

175

On the Problem of Selecting Interaction Objects Reglage

Figure 9: Some scrolling cursors.

Inspect/chanae records Social security number

12

First name Last nane

KARL MILLfiRD

Job title Salary Charge rate/hour

Title Street & City State & Zip

11 Konya Driue Pittsburgh PA 98723

Home phone

(455) 367 4561

Find

Department Business phone Extension

Next

Preuious

Change

Production 135 Food Analyst 35000 35.00

-SkillsNOrientation

B

Sales Financial

Marketing Technical

Finish

Figure 10: Data grouping.

4.2.8. Principle of Data Grouping The goal of data grouping is to define selection rules when considering the input/display of data groups. This is often the case where semantically related data have to be grouped in an appropriate AIO whose definition support aggregation of data. For example, multiple group boxes can be selected for surrounding all AIO selected for elementary data. Figure 10 depicts a user interface where group boxes group data about the person, about his/her job, about his/her skills/orientation.

4.2.9. Principle of Data Listing The goal of this principle is to define selection rules used in the consideration of data list input/display. A common applicable case arises when the user provides multiple instances of a same aggregation of data. For example, a repetitive dialog box can be selected to enter a data-series whose type and definition are the same (Figure 11). A normal table can be selected to input/display aggregates of data arranged in rows and columns on the condition that all data are elementary and can be expressed in simple edit boxes (Figure 12).

5. Conclusion Despite extensive research activity and experience in the selection of AIO, this work still suffers from several intrinsic drawbacks:

176

Frangois Bodart & Jean Vanderdonckt Input of multiple trips Trip Number: - Trip type

O ride O drive flight Distance

Figure 11: A repetitive dialog box for a list of data.

Newdata

D

1 .id • Id

" '

wh.npuf

* purprio*

- howhMr

-

- iobfui -

««pur

1-

l

4

2

2

2

1

7

140

5

a

3

2

2

2

140

5

•4

4

3

3

5

130

4

S

3

4

5

130

1

2

4

S

.*

Figure 12: A normal table for a list of data.

• a great number of selection rules (257 for input and 63 for display) have been defined and detailed to increase the precision and the appropriateness of selection rules. However, this number becomes so high that the range of values become cumbersome to manipulate and master; • the ergonomic richness of the selected AIO only becomes to be realized in cases where data are sufficiently specific. Given precisely limited circumstances, the most appropriate AIO can in such cases be deduced. Thus, the power of semantic formalization of the data model stops where selection rules for specific data begin. Therefore, the significance of these rules is not maximized. We believe that human perception of real world and these limits constrain the efficiency of the selection rules. On the other hand, a human actor who is an expert in a given field and thus is conversant with the fields' difficult-to-formulate tacit knowledge, this expert might be able to recommend the AIO that is most appropriate to specific circumstance. In other words, the time and effort to formalize this knowledge into selection rules would outweigh the benefits; • our corpus of selection rules was mainly based on results, conventions and assumptions gathered from the experimental study mentioned in our different references (Mayhew, 1992). The consistency between these selection rules is not always clear. A trade-off has been chosen when necessary; • the selection rules have been expressed according to a strict production scheme: applying the set of selection rules therefore leads to one particular AIO. Sometimes


177

alternative AIO should be considered equally. Fuzzy thinking should be added to express the likelihood of selecting a particular AIO rather than another one under particular circumstances; • selection rules will always remain incomplete since they are based on existing AIO. If new AIO are born, new selection rules have to be defined; • selection rules heavily depends on user habits and preferences: selecting the most appropriate AIO according to experimental criteria sometimes does not lead to a rational result. Experimental criteria are formal, but psychological preferences are not. It is not always explicit about the criteria used to judge how well the selection rules meet users' needs at different points in the interaction dialog; • selection rules really need a final user validation even though they are basically founded on proven theoretical and experimental results; • selection rules strictly for business-oriented application have been studied. If they are to be extended to other fields, recent research in the visualization of information according to the complex information types and structures should be considered. For our limited purposes, we can say that those selection rules have in most cases been proved sufficient. We do not get the impression that new rules would substantially improve the usability of a selected AIO, except under particular circumstances (e.g. for multimedia data). In the future, maybe another formalization will become more appropriate. But this conclusion largely coincides with a view held by many psychologists: good design and good selection of AIO in particular are difficult to formalise, and require an understanding of human needs, capabilities and expectations which cannot easily be expressed in logical selection rules.

Acknowledgements The authors would like to thank the anonymous HCI'94 reviewers for their very helpful comments and other members of the TRIDENT project: A-M Hennebert, J-M Leheureux and I Provot for their collaboration. This work was partially supported by the FIRST research program, Ref.RASE/SCHL319/Conv. 1487 and by the 'Informatique Du Futur' project of SPPS under contract N°IT/IF/1. Any opinions, findings, conclusions or recommendations expressed in this paper are those of the authors, and do not necessarily reflect the view of the Belgian Government.

References de Baar, D J M J, Foley, J D & Mullet, K E (1992), "Coupling Application Design and User Interface Design", in Proceedings ofCHI'92: Human Factors in Computing Systems, P Bauersfeld, J Bennett & G Lynch [eds.], ACM Press, pp.259-266. Bodart, F, Hennebert, A-M, Leheureux, J-M, Provot, I, Sacre\ B & Vanderdonckt, J (1993), "Architecture Elements for Highly-interactive Business-oriented Applications", in Proceedings of East-West International Conference on Human-Computer Interaction, ICSTI, pp.151-173. Gray, M H, de Baar, D, Foley, J D & Mullet, K (1992), "Coupling Application Design and User Interface Design", in Proceedings ofCHI'92: Human Factors in Computing Systems, P Bauersfeld, J Bennett & G Lynch [eds.], ACM Press, pp.657-658.

178

Frangois Bodart & Jean Vanderdonckt

Janssen, C, Weisbecker, A & Ziegler, J (1993), "Generating User Interfaces from Data Models and Dialogue Net Specifications", in Proceedings ofINTERCHI'93, S Ashlund, K Mullet, A Henderson, E Hollnagel & T White [eds.], ACM Press, pp.418-423. Johnson, J (1992), "Selectors: Going Beyond User Interface Widgets", in Proceedings of CHI'92: Human Factors in Computing Systems, P Bauersfeld, J Bennett & G Lynch [eds.], ACM Press, pp.273-279. Kim, W C & Foley, J D (1990), "DON: User Interface Presentation Design Assistant", in Proceedings of User Interface Software Technology'90 (U1ST90), ACM Press, pp.10-20. Kim, W C & Foley, J D (1993), "Providing High-level Control and Expert Assistance in the User Interface Presentation Design", in Proceedings ofINTERCHI'93, S Ashlund, K Mullet, A Henderson, E Hollnagel & T White [eds.], ACM Press, pp.430-437. Mayhew, D J (1992), Principles and Guidelines in Software User Interface Design, Prentice-Hall. Mayhew, D J (1993), "Designing with Graphical User Interface Standards", in INTERCHI'93 Tutorial No.29, p.68. Petoud, I & Pigneur, Y (1990), "An Automatic and Visual Approach for User Interface Design", in Engineering for Human-Computer Interaction, North-Holland, pp.403-420. Root, R W & White, E (1993), "Graphical User Interface Design Guidelines for Bellcore Software Products", Bellcore, Document SR-STS-002614. Seligmann, D D & Feiner, S (1991), "Automated Generation of Intent-based 3-D Illustrations", in Computer Graphics, pp.123-132, Proceedings of ACM SIGGRAPH'91. Vanderdonckt, J (1993), "A Corpus on Selection Rules for Choosing Interaction Objects", FUNDP Namur Institute of Computer Science, Technical Report 93/3, Electronically available via anonymous ftp from arzach.fundp.ac.be [138.48.4.5] in/pub/papers/jvd/Selection.ps.Z. Vanderdonckt, J & Bodart, F (1993), "Encapsulating Knowledge for Intelligent Automatic Interaction Object Selection", in Proceedings of INTERCHI'93, S Ashlund, K Mullet, A Henderson, E Hollnagel & T White [eds.], ACM Press, pp.424-429. Weisbecker, A (1993), "Integration von Software-ergonomischem Wissen in die Systementwicklung", in Proceedings of Software Ergonomie'93, pp.299-310.

Minimising Conceptual Baggage: Making Choices about Metaphor Ben Anderson, Michael Smyth, Roger P Knott, Marius Bergan, Julie Bergan & James L Alty LUTCHI Research Centre, Department of Computer Studies, Loughborough University of Technology, Loughborough, Leicestershire LEI 1 3TU, UK. Tel: +44 (0)509 222789 Fax: +44(0)509 211586 EMail: [email protected] This paper introduces a pragmatic model of the use of metaphor in humancentred system design. It identifies a number of issues implied by the model and describes an experimental investigation of a subset of these issues. Three interfaces were developed in order to compare the effect on perceived and actual user understanding of different metaphors for a common underlying functionality. The investigation demonstrates that it is not only important to ensure a close coupling between metaphor and system, but that it is equally important to consider the likelihood that users will make incorrect inferences from the metaphor. The paper concludes that the pragmatic model provides a technique that can be of use to interface designers in addressing this problem. Keywords: metaphor, telecommunications, multimedia.

1. Introduction One of the most important issues to be considered in system design is how best to convey the functional attributes and action-oriented possibilities of the system to the user via the human-computer interface. Current thinking suggests that a particularly powerful technique is the use of metaphors based on real world, and therefore familiar, objects and activities to represent system properties (Carroll, Mack & Kellogg, 1988; Douglas & Moran, 1983; Erickson, 1990; Hammond & Allinson, 1987; Hutchins, 1989; Kay, 1990; Nelson, 1990; Rogers, Leiser & Carr, 1988; Smyth & Knott, 1993; Wozny, 1989). These papers tend to be either reasoned discourses on the advantages and disadvantages of metaphor (Carroll, Mack & Kellogg, 1988; Erickson, 1990; Hutchins, 1989; Kay, 1990; Nelson, 1990; Wozny, 1989),

180

B Anderson, M Smyth, R P Knott, M Bergan, J Bergan &JL Alty Topic

Computer system

Vehicle

Real world

Figure 1: The psycholinguistic conception of metaphor as applied to Human-Computer Interaction.

or are attempts to support psychological theories of metaphor (Hammond & Allinson, 1987; Rogers, Leiser & Carr, 1988) using experiments carried out under the aegis of human-centred system design. This paper introduces a pragmatic model of the use of metaphor in human-centred system design developed as part of a larger investigation, funded by the RACE initiative (MITS: Project No. R2094), of the role of metaphor in multimedia telecommunications (Alty, 1993). This model is outlined using set theoretic concepts and illustrative examples from current human computer interfaces. The paper discusses a number of issues raised by this model and reports a subsequent experimental investigation of a subset of these issues, in the context of the design of a demonstration telecommunications system. The paper then provides an analysis of the utility of the model in characterising the relationships and interactions between system functionality and metaphor. It concludes with some suggested guidelines and techniques for the use of metaphor as a strategy for interface design.

2. A Note on Terminology Terminology from the psycholinguistic literature (Dent-Read, Klein & Eggleston, submitted for publication Lakoff & Johnson, 1980; Richards, 1936; Verbrugge & McCarrell, 1977) is used in order to describe the entities that are taken to interact in the phenomenon of metaphor. This terminology defines the object to which the metaphor refers as the 'topic', whilst the object that is invoked figuratively is known as the 'vehicle', and the relationship between the two is known as the 'ground'. This relationship is illustrated in Figure 1. Thus in the simple example of the now familiar 'desktop metaphor' — the operating system is the topic whilst the real world desktop is the vehicle. The work reported in this paper will focus on the relationship between features of the vehicle and the underlying system functionality. Consequently, the topic of the metaphor will always refer to the system functionality and will be represented in terms of the system. The choice of representation is intended to emphasise the relationship between topic and system in the context of interface design. In addition to this, a policy will be adopted of using italicised type for words that refer to vehicles that are used as metaphors in computer interfaces. Thus, the paper will refer to the Macintosh desk-top, complete with it's folders, files and wastebasket.

Minimising Conceptual Baggage: Making Choices about Metaphor

181

Functionality supported by the vehicle (V+)

Functionality not supported by the vehicle (V-)

Functionality supported by the System (S+)

Those features provided by the system and supported by the vehicle (S+V+)

Those features provided by the system and not supported by the vehicle (S+V-)

Functionality not supported by the System (S-)

Features implied by the vehicle but not supported by the system (S-V+)

Features not implied by the vehicle and not supported by the system (S-V-)

Table 1: Breakdown of the interactions between the set of vehicle features and the set of system functionality.

Figure 2: The interaction of vehicle and system as applied to Human-Computer Interaction.

3. A Pragmatic Model of Metaphor at the Human Computer Interface This paper introduces a pragmatic model of metaphor that draws on the concepts of system and user models (Fischer, 1991; Norman, 1986) and the terminology for the interaction of system and vehicle suggested by (Hammond & Allinson, 1987) and later utilised by (Rogers, Leiser&Carr, 1988) It is suggested that the activity of metaphor at the human computer interface can be described in terms of the intersection of two sets. The first set represents the features of the vehicle (V+), in this case some real world entity, whilst the second represents features of the topic, in this case the system functionality, (S+) (see Table 1 and Figure 2). This conceptualisation provides a number of distinctions which are illustrated using the example of the Macintosh wastebasket: S+V+ features Those features that lie in the intersection between the two sets. Thus S+V+ features are those features of the system that map completely on to features of the vehicle. In the wastebasket example these include the mapping of deletion to throwing in the bin and the ability to retrieve items that have been thrown away by taking them out of the bin if the bin has not been emptied.

182

B Anderson, M Smyth, R P Knott, M Bergan, J Bergan &JL Alty

S+V- features Those features that lie within the set of system functionality but do not intersect with the set of vehicle features. In the case of the wastebasket such features include the ability to eject disks by dropping them into the wastebasket (which would mean disk deletion if this mapped onto the vehicle features) and the appearance of the wastebasket on top of the desk rather than underneath it. S-V+ features Those features that lie within the set of vehicle features but which do not intersect with the set of system features. In other words these are features of the real world entity that do not apply in the context of the particular system under consideration. These features are characterised as the 'conceptual baggage' that the vehicle brings with it to the interface. Examples of such features include the fact that real world wastebaskets have a finite volume and so will fill up if they are not emptied (although this is trivially true of the Macintosh wastebasket), they tend to be found underneath desks rather than on them and documents currently in use can be thrown away. S-V- features Those features that are neither part of the system functionality nor the vehicle. Since this set is infinitely large it may seem pointless to define it. However, consideration of features that could be thought of as appropriate to the system in use (that is those features that may lie just outside the set of vehicle features or the set of system functionality) will be important when making choices about which real world entity to use as a vehicle. The intersection between the vehicle and system sets that provides these distinctions is illustrated in Figure 2. Based on these definitions it is suggested that there are a number of factors that influence the effectiveness of particular metaphors used at the human computer interface. These factors can be thought of as depending on the degree of overlap between the two sets of features. They include the extent to which the vehicle causes a user to make incorrect inferences about the system functionality; the degree to which the system's core functionality maps onto the core features of the vehicle and the extent to which the system functionality fails to map onto the vehicle features at all. The model implies that these factors are intimately interrelated. In Figure 2 for example, an increase in system features that can be characterised as S+V+ will cause a corresponding decrease in features categorised as S+V-. One of the most important of these factors, and the one on which this paper concentrates, is the likelihood that users will make inappropriate inferences about the functionality of the system from their understanding of the vehicle (Carroll, Mack & Kellogg, 1988; Douglas & Moran, 1983). This paper proposes that this problem occurs whenever there is a high proportion of S-V+ features compared to S+V+ features. In other words, the majority of the features of the vehicle are inappropriate to the context of system usage. Consequently, the users will have inappropriate expectations of the system behaviour. This problem is caused by what could be termed the conceptual baggage brought to the interface by the vehicle. Conceptual baggage can be thought of as features of a vehicle that are not utilised in a particular vehicle-system pairing. This idea is analogous to Laurel's argument that a representation of intelligent software agents should lead the user to appropriate expectations about the complexity of potential agent behaviour (Laurel, 1990). Similarly, a vehicle used in any human computer interface


183

should lead to appropriate expectations about the system by a user (Smyth & Knott, 1993). It is vitally important therefore that vehicles selected for use in interface design are those which can be thought of as minimising the S-V+ features. From Figure 2 it should be apparent that there are two strategies that could be used to effect this minimisation. The first is to select a vehicle with a restricted scope, the second is to expand the scope of the system so that it more faithfully maps onto the features of the vehicle. Both strategies will cause a reduction in conceptual baggage. In the latter strategy this would appear to encourage the creation of software that more closely follows the features of some real world vehicle. As many authors have noted, such an approach must be viewed with caution (Hammond & Allinson, 1987; Kay, 1990; Nelson, 1990) because blindly 'following the metaphor' can lead to restrictive systems in which the ability to do anything truly new is lost. A series of experimental investigations were conducted in order to examine the viability of this pragmatic model. More specifically, three different interfaces (i.e. vehicle-system pairings) were devised for use in a prototype telecommunications system in order to investigate the relative importance of the conceptual baggage of each vehicle in this context. The underlying system functionality was common to all three interfaces so that any differences in user behaviour and performance could be attributed solely to the vehicle-system pairings. Section 4 of this paper describes the underlying system functionality that forms the basis for the three system vehicle pairings. These pairings are described by Section 5, whilst the remainder of the paper reports the experimental investigations of these pairings, and the general implications for utilising metaphor as a strategy for interface design.

4. Description of System Functionality The prototype system used in these investigations was designed to act as an interface to an office based integrated digital broadband telecommunications infrastructure. More specifically, the system was designed to broadcast the availability state of all users of the system at any given point in time, and to enable users to make point to point audio-visual connections. Each user of the system was represented as a graphical icon which was available to all other users of the system via a directory mechanism. Communication between users of the system was initiated via these icons which were also used to display the availability state of the particular user. In order to provide an adequate simulation of such technology, the system, known as DOORS (MITS, 1994), was developed to utilise the audio-visual infrastructure and controlling software (Gaver et al., 1992) available at Rank Xerox EuroPARC. Preliminary analysis of office-based communications during early design suggested that a person's availability can generally be allocated to one of three states: 1. Available for communication. 2. Busy but interruptable. 3. Not available for communication (Anderson, 1994). As was noted above, the main focus of the DOORS system was on the analysis of three different vehicles. These vehicles formed part of a coherent family of metaphors (Alty & Knott, 1994) which could be extended to cover relevant aspects of office-based communication. The variable member of these families was a vehicle used to represent the user's availability state. The remaining members of these families were consistent over all the three interfaces.

184


Figure 3: Screen shot of one of the interface metaphors (Office Doors).

5. Description of Vehicle-System Pairings In order to describe the relationships between system and vehicle for each of the three pairings, it was necessary to explore the features of each of the vehicles with respect to the proposed system functionality. Techniques suggested by Carroll, Mack & Kellogg (1988) were used to consider the mappings between vehicle and system at the levels of 'tasks', 'methods' and 'appearances' in a representative set of scenarios. The results of this analysis were set in the context of the pragmatic model so that it was possible to allocate attributes of the vehiclesystem pairing to one of the four categories (see Section 3). The ease and immediacy of the allocation process formed the basis of the characterisation of each vehicle-system pairing. For example, office doors immediately provided a wide range of possible attributes pertinent to the initiation of point to point audio-visual connections, compared to the attributes associated with dogs. While it was acknowledged that such a process was highly subjective and would be unlikely to produce an exhaustive set of attributes associated with a vehicle-system pairing, it is however contended that this activity provided a valuable mechanism for exploring the potential of candidate vehicles and formed the basis for the characterisations discussed below.

5.1. Office Doors The first vehicle-system pairing adopted the office door as a vehicle for representing the availability of a user. Specifically, an open door corresponded to 'available for communication', & partially open door to 'busy but interruptable' and finally a closed door to 'not available for communication' (Figure 3). The characterisation of the relationship between this vehicle and the system is shown in Figure 4. In order that equivalent vehicle-system pairings could be constructed, the functionality underlying the interface was kept relatively simple. As a result of this strategy, and the fact


I

185

Features of vehicle

Features of system

Figure 4: Characterisation of Office doors/system pairing.

Features of system

n

Figure 5: Characterisation of Dogs/system pairing.

that the office door is a very rich vehicle in this particular context, there were a great number of features of the vehicle that were not supported by the system. The system functionality for example, does not allow doors to be locked. Thus it can be seen in Figure 4 that the proportion of S-V+ features compared to S+V+ features was relatively high. In addition, most of the system functionality was accounted for by features of the vehicle. From this characterisation, certain predictions about the patterns of subject performance of this system could be expected. Firstly it could be expected that subjects would find the system easy to use even if they had not encountered it before, not only because the metaphor seems contextually relevant, but also because the ratio of S+V- features to S+V+ features is quite low. For the same reason it could be expected that subjects would quickly explore the system and successfully utilise the underlying functionality. However, it would be predicted that over time subjects would become frustrated that features they might expect to be present in the context of this system were not in fact supported as the conceptual baggage of this particular vehicle-system pairing is quite high. Office doors was therefore considered a rich and appropriate vehicle in the context of this pairing.

5.2. Dogs The second vehicle-system pairing adopted the dog as a vehicle for representing the availability of a user. Specifically, an attentive dog corresponded to 'available for communication', a digging dog to 'busy but interruptable' and finally a sleeping dog to

186

B Anderson, M Smyth, R P Knott, M Bergan, J Bergan

ScJLAlty

Features of vehicle /

/

Features of system

V

J

Figure 6: Characterisation of Traffic-lights/system pairing.

'not available for communication'. The characterisation of the relationship between this vehicle and the system is shown in Figure 5. In this pairing, as in the previous case, there were also a great number of potentially relevant features of the vehicle that were not supported by the system. For example, Dogs could not be trained to allow communications from specified people. Thus it can be seen that the proportion of S-V+ features compared to S+V+ features was relatively high. Again, there was considerable conceptual baggage. However, it can be seen that very little of the system functionality was accounted for by features of this vehicle. Such a characterisation would lead to different predictions about the patterns of user performance. Firstly it would be expected that initially subjects would not find the system intuitive, not only because the metaphor seems less contextually relevant, but also because the ratio of S+V- features to S+V+ features was comparatively high. For the same reason it would be expected that even if subjects do explore the system and become familiar with the functionality, the boundary between S+V- and S+V+ features will be apparent. Dogs was therefore considered to be a rich but inappropriate vehicle in the context of this pairing.

5.3. Traffic Lights The third vehicle-system pairing adopted the traffic light as a vehicle for representing the availability of a user. Specifically, a green light corresponded to 'available for communication', an amber light to 'busy but interruptable' and finally a red light to 'not available for communication'. The characterisation of the relationship between this vehicle and the system is shown in Figure 6. In this pairing it can be seen that there were few potentially relevant features of the vehicle that were not supported by the system. Thus the proportion of S-V+ features compared to S+V+ features was relatively low. In this instance, there was considerably less conceptual baggage than in the previous two situations. As was the case with the dog, it can be seen that very little of the system functionality was accounted for by features of the vehicle. This characterisation would lead to further predictions about the patterns of subject performance. Firstly it would be expected that subjects would not initially find the system intuitive, not only because the metaphor seems less contextually relevant, but also because the ratio of S+V- features to S+V+ features would be quite high. For the same reason it would be expected that even if subjects do explore the system and become familiar with the functionality, the boundary between


187

S+V- and S+V+ features will be apparent. Finally, owing to the predicted lack of conceptual baggage it would be expected that the subjects would be better able to distinguish between S-V+ features and S+V+ features associated with this vehicle-system pairing. Traffic Lights was therefore considered to be a sparse vehicle with limited appropriateness in the context of this pairing.

6. Experimental Methods As stated in Section 3 of this paper, the objective of the investigation was to examine the viability of the pragmatic model by utilising the interface metaphors Office doors, Dogs and Traffic Lights. In order to compare and contrast the effects of each of the vehicle-system pairings, three independent groups of subjects undertook the same task that required usage of identical underlying telecommunications services. Experimental data was collected using a combination of verbal protocol, activity capture using video and questionnaire techniques. This paper will focus on the data generated by the questionnaire and will outline some preliminary findings. 6.1. Subjects All subjects were selected from the student population currently undertaking post-graduate conversion courses in the Department of Computer Studies at Loughborough University. Eighteen subjects were used in total, producing a subject pool familiar with the basic operational characteristics of graphical user interfaces. Each subject was randomly assigned to one of the three interface metaphor conditions. 6.2. Task In order to expose subjects to the underlying system functionality, it was necessary to ensure that each subject both initiated and received a number of different kinds of audio-visual connections. Each subject was presented with a task scenario which required them to act as a member of a small software development team together with a press release for an imaginary piece of software developed within the group. The press release contained a general description of the software product, together with a projected release date and a technical specification. The task required that the subject sought confirmation of information contained within the press release by contacting the appropriate team members from a directory provided. The roles of other members of the software team were played by experimental stooges who were, unknown to the subject, also in continuous voice contact with the experimenter who could externally monitor the progress of the subject. 6.3. Experimental

Set-up

The system that was provided for subjects consisted of a Sun SPARCStation 2 and a separate colour monitor for the audio-visual connections (see Figure 7). To enable subjects to make communication connections the system was connected to two similar remote nodes located within the building. Experimental data was gathered by means of two video cameras and a microphone. The video and audio signals were mixed and used by the experimenter to monitor the subject's performance from a different location. The experimenter could inform both the experimental stooges of the subject's progress throughout each trial using a 3-way conference telephone line.

188


Figure 7: Experimental set-up in use.

6.4. Data Capture Techniques As a means of ascertaining the nature of each of the vehicle's relationship with the underlying functionality, a questionnaire was designed for each interface metaphor. The questionnaire contained three sections. The first and third sections required subjects to answer short openended questions concerning the interface metaphor. For example, 'at any stage during the task would you have liked to refer to a manual, if so at which point?'. The second section consisted of 12 statements attributing features to the system. These statements were split into four conditions, each one corresponding to one of the categories defined in Table 1 of Section 3. In this section subjects were instructed to indicate their agreement or disagreement with the statement, together with a rating of their confidence in their answer (0-100%). Examples of the statements associated with each vehicle-system pairing in the four conditions are provided in Table 2. In the case of the S-V- condition the statements presented to subjects were consistent across all of the vehicle-system pairings. The inclusion of a confidence rating associated with each statement enabled a distinction to be made between the subject's perceived understanding and their actual understanding of the system functionality. Actual understanding was measured in terms of the number of correct answers to the questionnaire statements. Supportive evidence for the understanding of the metaphor could be provided by the written answers provided in Sections A and C of the questionnaire.

7. Results The data generated by the experiments fell into three categories. Section B of the questionnaire provided both the number of statements that subjects got correct, and the level of confidence associated with each of the answers. Sections A and C of the questionnaire provided subjective responses. Table 3 presents the first two categories of data for each condition and each vehicle.

Minimising Conceptual Baggage: Making Choices about Metaphor Condition

S+V+

S+V-

S-V+

s-v-

189

Statement

Vehicie Doors

It is possible to knock on a door to attract the owner's attention.

Dogs

It is possible to whistle at a dog to attract the owner's attention.

Traffic Lights

Green indicates that you can proceed to make a connection.

Doors

You can tell who it is if someone knocks on a closed door.

Dogs

Your dog can tell you who has tried to contact you.

Traffic Lights

You can tell who is trying to contact you if your colour is red.

Doors

Related doors can be grouped into corridors.

Dogs

Related dogs can be grouped into a pack.

Traffic Lights

Related colours can be grouped on a palette.

All

To make a connection you need to know the person's phone number.

Table 2: A sample of statements taken from the questionnaire for each vehicle in the four conditions.

Vehicle 1 Doors

Vehicle 2 Dogs

Vehicle 3 Lights

N

M

N

M

N

M

S+V+

12

72.23

10

66.77

16

70.28

S+V-

16

59.72

14

80.56

13

64.45

S-V+

3

59.72

5

66.67

10

56.56

S-V-

12

58.33

9

74.28

12

65.28

Table 3: Summary of results generated by Section B of questionnaire Where N - total number correct (Max. 18), M - mean confidence level (Max. 100).

The total number of correct answers for each condition and vehicle are presented in graphical form in Figure 8. A 1-way ANOVA for repeated measures was carried out on all of the confidence level data. This showed that there was no significant effect (F(3, 51) = 1.635, ns) across the four conditions in terms of the confidence level. If there was any inherent difficulty associated with the statements for any of the conditions, it would be expected that subjects would be

790

B Anderson, M Smyth, R P Knott, M Bergan, J Bergan &JL Alty 20

8

Doors 10

Dogs

3

Traffic Lights

1 S+V+

S+V-

S-V+ Condition

S-V-

Figure 8: Bar chart showing the number of correct answers in each condition for the three interface metaphors.

significantly less confident about their answers to statements in that condition. As can be seen from the data, no such significance existed. Therefore it was concluded that any significant differences in terms of the number of correct answers could be attributed directly to the subject's understanding of the interactions between system and vehicle features. A similar statistic was calculated for the number of correct answers across all of the data. This showed that there was a significant effect (F(3, 51) = 11.37, p = 0.0001) across the four conditions in terms of the number of correct responses, as is apparent from Figure 8.

7.1. Infra-Metaphor Comparisons For each metaphor two 1 -way ANOVA's for repeated measures were calculated across the four conditions on both the number of correct answers and the mean confidence levels. Further, a series of t-Tests were performed on each of the condition pairings. 7.1.1. Doors An analysis of variance of the number of correct answers for the four conditions showed a significant effect for the Doors metaphor, (F(3, 15) = 28.81, p = 0.001). Further, t-Tests for repeated measures revealed that there were significantly fewer correct answers for the S-V+ condition than the S+V+ condition (t(5) = 6.708, p < 0.0011); the S+V- condition (t(5) = 7.05, p < 0.0009); and the S-V- condition (t(5) = 6.708, p < 0.0011). An analysis of variance of the mean confidence level for each condition using the Doors metaphor, irrespective of whether the answer was correct, produced a non-significant result for the Doors metaphor, (F(3,15) = 1.693, ns). Individual t-Tests showed the only significant difference, in terms of mean confidence, to be between the S+V+ and S+V- conditions, where subject's mean level of confidence was significantly higher in the S+V+ condition, (t(5) = 3.504, p < 0.0172).

7.1.2. Dogs An analysis of variance of the number of correct answers for the four conditions showed a significant effect for the Dogs metaphor, (F(3, 15) = 3.534, p = 0.041). Further, individual


191

t-Tests revealed that there were significantly more correct answers in the S+V- condition than in the S-V+ condition (t(5) = 3.503, p < 0.017). Analysis of variance of the mean confidence level for subject's responses for all conditions showed no significant effect for the Dogs metaphor, (F(3,15) = 1.801, ns). However individual t-Tests revealed that a significantly higher mean confidence level was produced in the S+Vcondition than in both the S+V+ condition, (t(5) = 2.715, p < 0.042) and the S-V+ condition (t(5) = 2.712, p < 0.0422).

7.1.3. Traffic Lights An analysis of variance of the number of correct answers for the four conditions showed no significant effect for the Traffic Light metaphor, (F(3, 15) = 1.404, ns). Individual t-Tests revealed that there were significantly more correct answers in the S+V+ condition than in the S-V+ condition (t(5) = 2.739, p < 0.04). Similar to both the Doors and Dogs metaphors, an analysis of variance on the mean confidence levels in all conditions showed no significant effect, (F(3, 15) = 1.174, ns). Again, t-Tests revealed significantly higher mean confidence levels associated with the S+V- condition compared to the S-V+ condition, (t(5) = 2.63, p < 0.0465).

7.2. Inter-Metaphor Comparisons Central to the design of this experiment was the ability to compare between equivalent interface metaphors. In order to achieve such a comparison the underlying system functionality remained stable and the number of correct responses and confidence levels acted as dependent variables. The metaphors were compared across each of the four experimental conditions. A 1-way completely randomised analysis of variance of the number of correct responses within each of the 4 conditions showed a significant effect in the S-V+ condition for the three metaphors, (F(2,15) = 4.239, p = 0.0347). Further t-Tests within the S-V+ condition revealed that the number of correct responses produced by subjects using the Traffic Lights metaphor was significantly higher than those using the Doors metaphor, (t(10) = 2.907, p < 0.0157). An analysis of variance was also performed on the mean confidence levels for each of the 4 conditions. No significant effect was observed within any of the conditions across the three metaphors. However, individual t-Tests within the S+V- condition showed mean confidence levels to be significantly higher for the Dogs metaphor compared to the Traffic Lights metaphor, (t(10) = 2.244, p < 0.0486).

8. Discussion It is clear from the results that the intuitive nature of the Office door interface metaphor caused the subjects to make incorrect assumptions concerning the nature of the underlying system functionality. This was supported firstly by the fact that the number of correct answers in the S-V+ condition was significantly less than in all of the other conditions; secondly by the fact that the subjects were not significantly less confident about their answers to statements from this category; and finally by the fact that subjects generally reported no need for a manual in order to begin the task. This would imply that subjects were confident that they were able to distinguish functionality that was in the system but not covered by the vehicle, from functionality that was covered by the vehicle, when in fact this was found not to be

792


the case. The subjects exhibited a misplaced sense of confidence about their answers due to the richness and contextual relevance of this vehicle, which had the effect of masking the boundary of the mapping between vehicle and system. It would seem therefore that the Office doors vehicle, while providing a contextually rich set of resources, brought a considerable amount of conceptual baggage to this particular vehicle-system pairing. The effect of this baggage was exacerbated by the relative simplicity of the underlying system functionality. In the case of Dogs, subjects were better able to identify system functionality that was not supported by the vehicle, than functionality that was suggested by the vehicle but was not present in the system. This was supported by the fact that the number of correct answers in the S-V+ condition was significantly less than in the S+V- condition; and that subjects were significantly more confident about their answers to statements in the S+V- condition than both the S+V+ and the S-V+ condition. In contrast to the Office doors vehicle, it would seem that Dogs provided a rich set of resources that were largely inappropriate in the context of this particular vehicle-system pairing. This is indicated by the fact that subjects reported a need for a manual explaining the representations of system state at the start of the task. Thus, whilst a degree of conceptual baggage could be expected, the lack of contextual relevance caused the effect to be reduced. Finally in the case of Traffic Lights, subjects were better able to identify system functionality that was supported by the vehicle than functionality that was suggested by the vehicle but was not present in the system. This was supported by the fact that the number of correct answers in the S-V+ condition was significantly less than in the S+V+ condition; and that subjects were significantly more confident about their answers to statements in the S+V- condition than the S-V+ condition. In addition this last result indicates that the vehicle maps only to a small part of the system functionality causing subjects to be aware of the boundary between the two. Subjects did not find this vehicle at all intuitive as is indicated by the fact that the majority of them expressed a need for a manual to explain the representations of system state. Once the subjects became aware of the mapping between vehicle and system, actual understanding of the interactions was superior to that in either of the other two vehicle-system pairings. The Traffic Lights vehicle then, did not provide a rich set of resources. However the resources it did provide mapped tightly to a small subset of the system functionality. Consequently the effect of this vehicle's inherent conceptual baggage was not as marked as in either of the other vehicle-system pairings. The inter-metaphor comparisons provide supportive evidence for these conclusions by demonstrating that the Doors vehicle brought with it more conceptual baggage than either of the other two vehicles. This is shown by the pattern of data for the S-V+ condition where both the Doors and Dogs vehicles produced fewer correct answers than Traffic Lights, significantly so in the case of Doors. In addition the evidence supports the suggestion that Dogs was a rich but largely inappropriate vehicle as compared to Traffic Lights. This was shown in particular by the fact that subjects were significantly more confident of their answers in the S+V- condition, when using the Dogs vehicle-system pairing, than when using the Traffic Lights. This can be attributed to the clarification of the boundary between features of the system that were not supported by the vehicle, and those that were supported. The characterisation of metaphor presented in this paper is intended to inform the designer's choice of vehicle in the context of the system under development. In the system described, Traffic Lights would seem the best suited to represent a person's availability for communication


193

due to the high degree of contextual relevance, coupled with low conceptual baggage. Whereas, if the system was intended to provide a more comprehensive set of functions, for example access to remote databases, common file stores and meeting support tools, then Doors would be the more appropriate choice as this vehicle has the potential for extension in order to support additional functionality.

9. Conclusions These findings suggest that there is empirical evidence for the contention that, as was predicted by the pragmatic model, conceptual baggage plays an important role in the overall effectiveness of particular metaphors at the human-computer interface. Further, this study reaffirms the principle that the utility of potential vehicles must always be considered in the light of the system functionality that they are intended to support. This paper demonstrates that exploring potential vehicles in terms of the four conditions defined in Section 3 can provide a useful and repeatable technique to achieve this aim. The degree of conceptual baggage brought to the interface by a particular vehicle-system pairing is primarily determined by the nature of the system and the richness of the vehicle. Conceptual baggage can only be defined with respect to a particular vehicle-system pairing because when paired with different systems, the same vehicle will exhibit different degrees of conceptual baggage and contextual relevance. A rich vehicle, such as Dogs, might have been expected to bring with it a high degree of conceptual baggage. The impact of this baggage was in fact reduced because the vehicle was not contextually relevant. This paper concludes that it is not only important to ensure a close coupling between a potential vehicle and intended system functionality, but it is equally important to reduce the likelihood that users will make incorrect inferences about system behaviour based on their understanding of the vehicle. Thus it is imperative to keep conceptual baggage to a minimum. The pragmatic model introduced in Section 3 provides interface designers with a technique for assessing the amount of conceptual baggage in any vehicle-system pairing.

Acknowledgements The research reported in this paper was supported by RACE Project MITS (Project No.R2094). We thank our partners, in particular The University Surrey, for their help in clarifying some of the ideas presented in this paper. We gratefully acknowledge Paul Dourish, Mike Molloy and Alex Zbyslaw of Rank Xerox EuroPARC for their help with the implementation and evaluation of the DOORS system.

References Alty, J L (1993), "Cooperative Working and Multimedia Telecommunications: The Importance of Metaphor", in Proceedings of RACE IS+N Conference, RACE Directorate. Alty, J L & Knott, R P (1994), "A Formal Notation for Metaphor Description", LUTCHI Internal Report No. 94/M/LUTCHI/0163. Anderson, B (1994), "Cognitive Anthropology and User-centred System Design I: How to Enter an Office", LUTCHI Internal Report No. 94/M/LUTCHI/0162.

194


Carroll, J M, Mack, R L & Kellogg, W A (1988), "Interface Metaphors and User Interface Design", in Handbook of Human-Computer Interaction, M Helander [ed.], Elsevier Science (North-Holland). Dent-Read, C H, Klein, G & Eggleston, R, "Metaphor in Visual Displays Designed to Guide Action", (submitted to Journal of Metaphor and Symbolic Activity for publication). Douglas, S A & Moran, T P (1983), "Learning Text Editor Semantics by Analogy", in Proceedings of CHI'83: Human Factors in Computing Systems, A Janda [ed.], ACM Press. Erickson, T D (1990), "Working with Interface Metaphors", in The Art of Human-Computer Interface Design, B Laurel [ed.], Addison Wesley. Fischer, G (1991), "The Importance of Models in Making Complex Systems Comprehensible", in Mental Models and Human Computer Interaction 2, M J Tauber & D Ackermann [eds.], Elsevier Science (North-Holland). Gaver, W, Moran, T, MacLean, A, Lovstrand, L, Dourish, P, Carter, K & Buxton, W (1992), "Realising a Video Environment: EuroPARC's RAVE System", in Proceedings ofCHI'92: Human Factors in Computing Systems, P Bauersfeld, J Bennett & G Lynch [eds.], ACM Press, pp.27-35. Hammond, N & Allinson, L (1987), "The Travel Metaphor as Design Principle and Training Aid for Navigating around Complex Systems", in People and Computers III, D Diaper & R Winder [eds.], Cambridge University Press, pp.75-90, Proceedings of HCI'87. Hutchins, E (1989), "Metaphors for Interface Design", in The Structure of Multimodal Dialogue, M Taylor, F Neel & D Bouwhuis [eds.], Elsevier Science (North-Holland). Kay, A (1990), "User Interface: A Personal View", in The Art of Human-Computer Interface Design, B Laurel [ed.], Addison Wesley, pp.191-207. Lakoff, G & Johnson, M (1980), Metaphors We Live By, University of Chicago Press. Laurel, B (1990), "Interface Agents: Metaphors with Character", in The Art of Human-Computer Interface Design, B Laurel [ed.], Addison Wesley. MITS (1994), "Deliverable D6, Review of Demonstrators", Internal RACE deliverable, No R2094/NOKIA/D6/DS//L/006/bl. Nelson, T H (1990), "The Right Way to Think About Software Design", in The Art of Human-Computer Interface Design, B Laurel [ed.], Addison Wesley. Norman, D A (1986), "Cognitive Engineering", in User Centered Systems Design: New Perspectives on HumanComputer Interaction, D A Norman & S W Draper [eds.], Lawrence Erlbaum Associates. Richards, I A (1936), The Philosophy of Rhetoric, Oxford University Press. Rogers, Y, Leiser, R & Carr, D (1988), "Evaluating Metaphors for Use at the User-System Interface", in Proceedings of the First European Conference on Information Technology For Organisational Systems, Eurinfo'88, H-J Bullinger, E N Protonotarios, D Bouwhuis & F Reim [eds.], North-Holland. Smyth, M & Knott, R (1993), "The Role of Metaphors at the Human Computer Interface", LUTCHI Internal Report No. 94/M/LUTCHI/0165. Verbrugge, R R & McCarrell, N S (1977), "Metaphoric Comprehension: Studies in Reminding and Resembling", Cognitive Psychology 9, pp.494-533. Wozny, L A (1989), "The Application of Metaphor, Analogy and Conceptual Models in Computer Systems", Interacting with Computers 1 (3), pp.273-283.

Keeping an Eye on your Interface: The Potential for Eye-Based Control of Graphical User Interfaces (GUI's) Howell O Istancet & Peter A Howarth* t Imaging and Displays Research Group, Department of Computing Science, De Montfort University, Leicester, UK. EMail: [email protected] * Vision and Lighting Research Group, Department of Human Sciences, Loughborough University of Technology, Loughborough, Leicestershire LEI 1 3TU, UK. EMail: [email protected] This paper examines the issues surrounding the use of an eyetracker, providing eye-movement data, as a general purpose input device for graphical user interfaces. Interacting with computers via eye-movements is not in itself new, however previous work in the area has been directed towards interaction with purpose-built software which can take into account device limitations such as accuracy. This work investigates how one can interact with unmodified graphical interface software which normally requires mouse and/or keyboard input. The results of three experiments are discussed which have compared performance between the eyetracker and the mouse, and between different ways of emulating mouse button presses using the eyetracker data. The experiments as a whole consider a range of tasks from simple button presses to the more complex and demanding operations of selecting text, and they indicate the feasibility of using the eyes to control computers. Keywords: physically-challenged, eye-control, input device.

1. Benefits of Controlling Graphical User Interfaces by Eye 1.1. Overview The use of the eyes as a primary means of controlling input is appealing for a number of reasons.

796

Howell OI stance & Peter A Howarth

First, it can be considered as a 'natural' mode of input and by-passes the need for learned hand-eye co-ordination to effect operations such as object selection. The user simply looks at a screen object they wish to select rather than using a hand-held pointing device, such as a mouse, to position a screen cursor over the object. Second, one can expect performance benefits. If a user need only look at an object to acquire it, rather than having additionally to control and position a cursor by hand, speed of selection will be increased. Eye-based cursor control, in combination with other input devices for effecting a selection, holds the promise of increasing the narrow communication bandwidth from user to machine. Third, it allows for interaction in situations where the user is unable to use their hands. This restriction may be a result of the task itself, where both hands are used for other purposes, or may be due to physical disability on the part of the user. The most commonly available input devices make no provision for users with severe disabilities, many of whom may have little or no control over their limbs and may only be able to make controlled movements with their eyes. A low-cost eyetracker, using eye position as a means of issuing commands via the user interface, offers a means of enfranchising this group with respect to access to modern graphical interface software.

1.2. Previous Research in Relation to the Anticipated Benefits of Eyetracking 1.2.1. Naturalness of Eye Movement The advantage of 'naturalness' in the use of eye-gaze data is that some insight is gained into the user's current intention during interaction with a computer, because before carrying out any action, you first look at what you are interested in. However, without an independent selection protocol, the difficulty lies in whether to make the inference that a user also wishes to select the item that he or she is currently looking at, or whether to assume that they are simply browsing. In his work on naval command and control systems, Jacob (1993) investigated the advantages of natural interaction, via eye movement tracking, in the provision of 'non-command' interfaces. Non-command techniques attempt to facilitate interaction with the system without the need for the user to give it specific commands. This requires that the system makes some kind of inference about what the user wants to do by monitoring some data associated with user behaviour, such as eye movement, and responding accordingly. Consequently, Jacob rejected all eye-based selection mechanisms other than dwell-time (an extended fixation on a target) as he considered interactions that required external training, such as blinking, to be 'unnatural'. He studied the use of eye positioning of the cursor, and used both dwell time and an external hardware button as selection mechanisms. He presented and evaluated interaction techniques for object selection, menu item selection, control of scrolling text and switching the active window. The techniques described were based on a combination of eyetracker and mouse-button commands, and are not transferable for a general purpose input device for a 'command driven' GUI. This is because the target software was purpose-built to accommodate the different command protocols used. Eye-based control has been used in another example of non-command interaction using eye position. Starker & Bolt (1990) provided viewers of a picture with information about particular features by assuming that the parts of the image of interest to a particular user would be revealed by whereabouts on the picture they were looking. The features of interest were

Keeping an Eye on your Interface

197

determined by an index of 'focus of attention' which was calculated from matching eye-gaze position to features in the underlying image. Again, the problem here lies in producing decision rules about whether a person wants information about that part of the picture, or whether they are simply looking at it. 1.2.2. Anticipated Performance

Benefits

Ware & Mikaelian (1987) examined performance differences in terms of speed of response and error rates for simple target acquisition tasks. The screen cursor was moved under eyecontrol and different ways of issuing selection commands (screen button, hardware button and dwell-time) were compared for several different target sizes. Selection speed increased, and error rates decreased dramatically, when target size increased from 0.45 to 1.5 degrees of visual angle. Selection times of less than 1 second were recorded, and increasing the target size above 1.5 degrees produced no further improvement in performance. Ware and Mikaelian found that the dwell time and the hardware button selection protocol were equally fast and both were superior to the screen button condition in terms of mean error rates (22% in the screen button condition, 8.5% in the hardware button condition and 12% in the dwell time condition). They also observed that the eyetracker could be a faster selection device than the mouse for targets above 1.5 degrees. This conclusion was arrived at, however, by comparison with data reported by Card, English & Burr (1978) for mouse performance and not with data they had collected themselves for the same tasks. Hence this comparison is unlikely to provide a reliable indicator of the relative merits of the devices. One of the issues addressed in the research described in this paper is a direct comparison between mouse and eyetracker. 1.2.3. Special-needs

User Devices

The potential of the use of eye tracking for specialist input devices for physically-challenged users has already been recognised (Downing, 1985; Hutchinson et al., 1989). These systems do not necessarily use eye-position data to drive a screen-based pointing device, but may give feedback in different ways, such as highlighting the object being looked at on the screen. Downing developed an eye-gaze communicator and control system which also used dwelltime as a selection mechanism. He estimated that selection speeds with the device would enable a word generation rate of between 10 and 20 times that achievable with other devices used by a physically-challenged user group. Ten Kate et al. (1980) studied different methods of providing communication aids controlled by eye. One used an eye switch for the selection of letters whilst the other used eye position recording and fixations on screen-targets corresponding to letters. Shaw et al. (1990) presented a low-cost control device, based on the sensing of eye-winks, for controlling a wheel chair. Different combinations of lid positions for each eye were used, and each mapped onto specific command actions. Rosen & Durfee (1978) used encoding of sequences of changes in eye-gaze directions and mapped these onto a vocabulary of communication items. A significant feature in each of these systems was that the application or device being controlled by eye position was specially constructed for the purpose of improving communication or mobility, and consequently any limitations in the performance of the eyetracking device could be taken account of in the design. The challenge addressed in this research is enabling interaction with any software application running on a particular windows platform (in this case, MS-Windows™ ).

198

Howell O Istance & Peter A Howarth

1.2.4. Challenges of using Eye Control for GUI's There are many challenges which must be met if we are to produce a usable general-purpose device emulator for mouse and keyboard which allows interaction with a wide range of available, unmodified software. To do this, the following must be provided: • A sufficient degree of pointing accuracy with the eye-based device to permit a realistic level of interaction with system and application objects. • A means of signalling commands equivalent to key or button presses which is comfortable and will not interfere with device pointing accuracy. • A means of disabling control of the cursor (equivalent to taking one's hand off a mouse). • A degree of comfort which facilitates use of the input device for extended periods of time. • A device which is of low cost and as portable as a PC. In meeting these challenges, two points need particular consideration.

1.3. Implications of General Purpose Input Device Emulation The requirement to enable interaction with unmodified software has two consequences. The first of these is that the angular size of the targets on the screen will have been determined independently of the capabilities of the eyetracker. These are determined by the size and resolution of the monitor used, the viewing distance from the subject to the screen, the actual graphical user interface used, and the size and type of window interaction objects it contains (such as buttons, menus bars, scroll bars). The second consequence is that the eyetracker device should permit the emulation of all of the commands and operations used to interact with the software. Simply emulating single mouse button clicks is not sufficient. It should also be possible to emulate click-and-drag and double-click actions with different mouse buttons, and to emulate keyboard operations to facilitate text input and key-based command input. Several screen-based keyboard products already exist, and in these products buttons representing keys appear on screen. The buttons are activated by a simple click operation with a mouse or other pointing device and cause a keystroke event to be sent to the underlying window system. With suitable calibration, it would be possible to look at a keyboard template, located off-screen, either above or below the display monitor and map the recorded eye position to the corresponding key. The advantage of an on-screen keyboard is that the cursor can give feedback as to where the system thinks the present point of gaze is whilst this information would not be available in the off-screen version. The disadvantage of an on-screen keyboard is that part of the screen must be given over to the keyboard application, unless a two-screen solution is considered. 1.4. Signalling the Equivalent of Mouse Button

Presses

At present, there are two options for selecting the object at which the cursor points. One is to use an external device, such as a blow tube or a muscle-activated switch. The alternative is to use an eye-based protocol, where an 'unnatural' event is used to signal selection. Options available here are to stare or 'dwell' on a item for an abnormally long time, to close both eyes for a period longer than a natural blink, or else to use a wink. The challenge is to find a usable selection protocol which is relatively rapid, has a high hit-rate and a low false-positive rate.


199

Jacob (1993; Jacob et al., 1993) described the 'Midas Touch' problem associated with the use of eye-movement data in the context of 'non-command' interfaces, where dwell time is used to signal target selection. Essentially, the user can not look at anything for too long without also selecting it, as it is not possible to distinguish between the intention to select an object or merely to browse. This problem disappears when the user is required to make some explicit action in order to select an object. The approach adopted in this research is that in the absence of information about which faculties any individual possesses, it should be possible to achieve all interaction via the eyes alone. However, it should also be possible, if desired or necessary, to add external devices to the system for signalling commands so that the eyetracker is used for cursor positioning only.

2. Hardware Requirements and Usability Issues In this section, usability problems will be discussed which have been identified during the research carried out to date. In summary, the main usability issues for the eyetracker input device concern: • Ease and speed of initial device calibration. • Ease of setting up the cameras to obtain stable, well-focused images of individuals' irises and pupils. • The need for subsequent adjustment of initial calibration values and whether this can be done during use of the device by the user. • Postural constraints, tolerance to head movement and consequent discomfort and fatigue. • The level of training necessary to use the device effectively. • Individual characteristics of the user, such as droopy eyelids or long eyelashes.

2.1. Eyetracker Systems and Problems with Eyetracking In most activities, people are normally able to, and do, move their head whilst continuing to look at objects in their field of view. The consequence for a system attempting to record where the eye is looking is the disambiguation of eye movements needed to keep the object of regard the same when the head moves, and eye movements resulting from changing the object of regard. Eyetracker systems use a variety of techniques including corneal reflection and processing video images of the pupil to measure eye position with respect to a set of optical components. Cameras and other optical devices can be head-mounted and eye position measured with respect to the head. Head-mounted devices do not constrain head position, but are obtrusive in that the subject is required to wear a helmet or headband. If the line of gaze from the eye to some external reference point, such as a display screen, is needed, then the orientation and position of the head has to be measured as well as the position of the eye, and the two sets of data aggregated. If, on the other hand, the cameras and optical components are fixed in relation to the room then eye position can be measured directly with respect to the room and objects within it, such as display screens. However, room-mounted systems require the head to be kept within the field of view of the camera or optical measuring device. Normally the head would need to be constrained to enable statically mounted cameras to maintain a reasonable image of the pupil, and this is usually effected by means of a head-rest. Whilst acceptable in a laboratory,

200


this degree of constraint is not going to be generally acceptable in an input device, and will inevitably lead to postural fatigue after periods of extended use. Room-mounted systems do exist which allow a certain degree of head movement. The more expensive of these use servo-controlled mirrors to track the eye and maintain a stable camera image, although this is usually accompanied by some loss of measuring accuracy. The range of movement which can be tolerated is typically in the region of ±15 cm in the horizontal and vertical planes and even in the fore and aft plane (ASL Inc, 1992). Generally, the greater the degree of constraint on head position, the greater the possibility for more accurate measurement. Room-mounted systems will also have vertical and horizontal limits to the angular range over which the eye can be tracked. In some cases, the vertical range of measurement can be prohibitively small (a 15-degree range at a viewing distance of 500 mm translates into a vertical distance of 134 mm, smaller than the height of most display screens). 2.2. Device Accuracy and

Precision

Device accuracy can be considered as the distance between the computed line of gaze and the true line of gaze and is expressed in terms of visual angle. Device accuracy of 1 °, for example, at a viewing distance of 500 mm to a display screen would translate into approximately 9 mm on screen. Device precision, on the other hand, is related to the distribution of measurement points about a mean whilst the subject is fixating on an object, the mean of the distribution corresponding to the accuracy. This variation around the mean comes about because, in addition to any measurement device imprecision, the eye will naturally fixate on a number of different points around a object of regard between saccades. In trying to drive readily-available software the resolution of the output from the eyetracker to a device driver generally needs to be comparable with that of a mouse, although the actual pointing resolution required will depend on the underlying application. For example, pointing to the start of a word within a piece of normal-sized text is clearly more demanding than selecting a command button in a dialog box. In systems which use a processed video image of the pupil, a relatively large image is necessary to obtain the maximum resolution from the eyetracker as the range of horizontal and vertical eye movement corresponding to the bounds of the screen has to occupy as much of the range of the measuring instrument as possible. The consequence of this is that small head movements could cause the pupil image to go outside the field of view of camera, and data (and control) would be temporarily lost. All of the above factors will influence the stability of the cursor under eye control with respect to where the subject is looking. Two problems are manifest which make accurate control of the cursor difficult, and these correspond to the descriptions of device accuracy and precision. The first is an offset between the cursor and the point of regard, and the second is jitter in the cursor position. Clearly, the smaller the target to be selected, the greater the effect these problems will have. The extent to which these affect task performance will be discussed in Section 4. 2.3. The System Used in the Current

Research

The eyetracker system (Micromeasurements System 7000 pupillometer) is a binocular, infrared video-based machine, which measures eye position 60 times per second. The video image of each pupil is processed to enable the horizontal and vertical co-ordinates of the pupil centre


201

to be determined. In addition the pupil area is computed. To compensate for head movements a compensation algorithm examines the position of an image of a small infra-red light source reflected in the subject's cornea. The position of this image is then compared with the position of the pupil centre. Similar movements of the two occur during translational movements of the head, whilst differential movements of the two images occur during eye rotation. 2.3.1. Device Emulation Under

MS-Windows

The signal from the eyetracker is received on the serial port of the PC running the target application software. The incoming data is filtered and scanned before being processed by the device emulator software. The filtering is achieved by means of a simple buffering system, which can be adjusted to damp the jitter in the observed cursor position. The extent to which filtering techniques can be applied are constrained by the need for real-time control of the screen cursor. If the user perceives a lag in the cursor position then he or she is likely to look back at the cursor, causing the cursor to be 'chased' around the screen. The filtered data is mapped onto a screen co-ordinate, clipped to the boundary of the screen and used to update the cursor position. The stream of incoming position and pupil size data is scanned for temporal changes used to characterise a dwell or a wink. If such an event is detected then an appropriate window system message is generated corresponding to, for example, a mouse button press.

2.3.2. Binocular Versus Monocular Systems The fact that the eyetracker system is binocular is significant in that data from both eyes can be used firstly, to increase bandwidth by using data from each eye for different purposes and, secondly, to improve accuracy by having two estimates of eye position. The former advantage is utilised in facilitating click-and-drag actions (see Section 4.2). With respect to the latter point, a measure of convergence can be obtained which offers the prospect of introducing corrections for fore and aft movement of the head. This can be done through knowledge of the positions of both eyes when fixating on the same point, at a known viewing distance, when the device is calibrated. Fore and aft movements of the head would give rise to a discrepancy between the two eyes in the respective calculated eye positions, which will be a function of distance from the screen to the eyes. The advantage of a monocular system is that it is likely to be less expensive, as it would only need the one camera. Also, it is likely to be simpler to set-up and calibrate initially. The advantage of the binocular system is that it can provide additional information about the eyes, which can be used for increasing system accuracy and for facilitating selection. For example, with a monocular system one needs additional feedback to the user about 'button-up' or 'button-down' status, (see Section 4.3) during a click-and-drag operation, whereas this is not needed with a binocular system.

2.3.3. System Calibration Calibration involves establishing a zero point for the eyetracker data with reference to the PC screen, and a scaling factor to map from eyetracker co-ordinates to screen co-ordinates. The usability issues in initial calibration partly relate to the time needed to establish a stable image for both eyes and the time taken to execute the calibration procedure. The latter is usually very short (40-60 seconds) in comparison with the former (4-5 minutes for a new user). A number of individual features will influence the ease with which satisfactory images can be produced. These include eye colour, the normal position of the upper lid relative to the iris, and in some cases the use of eye make-up.

202

Howell 0 Istance & Peter A Howarth

If the user is to initialise the use of the device without help from another person then they will need to be given feedback about the quality of the pupil images available to the eyetracker in relation to the relative positions of the head and camera. The difficulty in subsequent correction by the user of drift from the initial calibration values is partly dependent on whether a badly located screen cursor under eye-control will be used for changing these values.

2.3.4. Head Movement, Head Position and Postural Fatigue At present, the system used for the research described here is run with the two cameras, mounted on tripods, each at an angle of about 25 degrees to either side of the head. This allows the subject an uninterrupted view of the host computer screen. The subject sits with their chin placed on a chin-rest, and their brow against a curved, padded, brow-bar. This arrangement allows the subject to make small lateral head movements (which, as explained above, do not affect cursor position) but does not allow excessive fore and aft head movement. Trials carried out with certain categories of disabled users have shown that the constrained posture necessary to maintain a stable pupil image is currently a major usability problem. This problem will be alleviated by using a redesigned head-rest. 2.4. Implications for a Low Cost Device In specifying a low-cost device, two solutions suggest themselves. The main problem to contend with in any system is that of head movement. A light-weight head-mounted binocular camera system with separate head-tracking to enable a world gaze position to be computed could be used. The technology to produce these relatively inexpensively has recently been developed for VR applications. Alternatively, one or two small video cameras may be mounted on a frame that fits onto the monitor of the machine to be controlled. This mount could also contain devices such as LEDs which could be used to give continual feedback to the user about the position of their head with respect to the cameras. 2.5. Usability Issues in Comparison

with Other Eye-Based

Systems

The problems encountered during the use of the present system have clear analogues with those reported elsewhere (Jacob, 1993). Shaw et al. (1990) reported that their eye-wink control device took 30 minutes to set up, which included establishing the measurement thresholds between the open and closed states of the eye. Jacob et al. (1993) made particular reference to the stability and repeatability of eye position measurements and concluded that the currently available commercial eyetrackers were still not adequate for use in practical applications with disabled users.

3. Summary of User Performance Investigations with the Eyetracker The concerns for the research programme to date have been as follows: i. The feasibility of using the eyetracker as a pointing device. This investigation examined performance when using the cursor under eyetracker control to point at simple targets on screen, and enabled a comparison (in terms of speed and accuracy) to be made with a mouse. ii. The different means of emulating mouse button actions using the eyetracker. iii. The use of the eyetracker for click-and-drag operations in the context of text selection, compared with a mouse.

203


2.3x0.8

3.2x1.8 4.2x2.7 Target Size (degrees of visual angle)

5.2x3.5

Figure 1: Selection times for different target sizes.

3.1. Comparison Between Eyetracker and Mouse for Pointing at Targets An experiment was conducted to compare selection speed when a cursor was positioned on a rectangular on-screen button using a mouse and the eyetracker respectively. For both means of cursor positioning, selection of the target was achieved by pressing a mouse button. Six subjects took part in the experiment, all of whom were novice mouse users. These subjects were used so that unskilled eyetracker users were compared with unskilled mouse users. Each subject completed 5 trials with a mouse and 5 trials with the eyetracker as the means of positioning the cursor. A trial consisted of 12 runs, 3 for each of the 4 target sizes. Each run was initiated by the subject selecting a 'start' button on the screen. The target then appeared at a random position and distance from the 'start' button, and the time taken to select the target was recorded. Mean target selection times are shown in Figure 1 averaged across all subjects and all selection distances. Eye control of the cursor leads to faster acquisition times where accuracy is less important (i.e. larger target sizes) but performance declines in comparison with the mouse as demands on accuracy increase and target sizes become smaller. The selection times in the eyetracker condition are significantly faster for the two larger sizes (p < 0.001 and p = 0.002 respectively) but not for the two smaller sizes. Eyetracker selection times in this experiment are longer than those reported by Ware & Mikaelian (see Section 1.2.2). Two factors can account for this difference. First, the distance to target appears to be much shorter in Ware & Mikaelian's experiment. In their procedure, all targets were permanently visible on-screen and were adjacent to each other in a column, with the start button situated in the centre of the column. Using this configuration one would expect shorter selection times. Second, the level of previous experience amongst subjects may have been different. One of their subjects was one of the authors and presumably was not a novice eye-tracker user. Again, one would expect better performance because of this. In conclusion, this first experiment has demonstrated that the eyetracker-based device can be used to point at simple screen targets, and that performance is broadly similar to that of a mouse for these targets. The eyetracker does then hold the promise of enabling faster acquisition than a mouse, providing that the targets are sufficiently large.

204

Howell O Istance & Peter A Howarth 3.01

•a c o o

A O

•

H/W Dwell Wink

1.5-

Target Size (degrees of visual angle) 1.0 0.9 x 0.9

1.5 x 1.5

2.1 x2.1

H 2.7 x 2.7

Figure 2: Selection times for different selection protocols.

3.2. Investigations of Protocols for Emulating Mouse Commands The second set of experiments investigated performance differences between alternative ways of emulating mouse commands. The target selection task was similar to the experiment described above. The following protocols were investigated: i. Wink: one eye was used for cursor positioning whilst the other eye was used to emulate mouse button states. Subjects could choose which eye they wanted to use for cursor control and which for button control. The eye open corresponded to button up and the eye closed corresponded to button down. Using the binocular system, this protocol offers the possibility of emulating click-and-drag operations as the cursor can be moved with the other eye whilst the 'button eye' is either open or closed. ii. Dwell: an extended fixation at the same place on the screen for a specified dwell time was treated as the equivalent of a mouse click. During the experiment three different dwell conditions were investigated, representing different combinations of dwell duration (from 100 msec to 200 msec) and sizes of area within which the cursor was considered to be stationary. In comparing different protocols we have only shown the results when the smallest area and the longest time (200 msec) were used, as the numbers of errors in this condition was significantly lower than the other two dwell conditions. It will be seen later that accuracy is a critical feature of user performance with the eyetracker. iii. Hardware Button (H/W): selection in this case was made using the mouse button. This condition was included to enable comparison with the results obtained in the first experiment and to enable the relative efficiencies of the eye-only protocols to be assessed. Four subjects took part in experiment 2. Each subject completed 5 blocks of 10 trials. A trial consisted of 3 runs of each of the 4 different sized target buttons, as in experiment 1. Within a trial, the order of target size presentation was randomised over the 12 runs. No initial training was provided, as part of the purpose of the experiment was to investigate performance changes over an extended number of trials. The data in Figure 2 shows selection time for each target size averaged over error-free trials and all subjects. Data from the first 20 trials has been excluded as performance improved


205

A O

70 n

•

H/W Dwell Wink

Trial Number 1-10

11-20

21-30

31 -40

41-50

Figure 3: Error rates for selecting smallest target (size 0.9 degrees) over repeated trials.

over these trials. It can be seen in this figure that selection times in the Hardware button condition were significantly faster than both of the eye-based conditions, for all target sizes. The selection time in the Wink condition was affected most by target size. Turning now to selection errors, it was found that there was no significant difference in error rates between the three protocols for the two larger targets, but performance did decrease for the smallest target. Figure 3 shows the error rates for the smallest target (0.9 x 0.9 degrees) used in the experiment for the three protocols across a total of 50 trials. Error rates are defined here as the percentage of trials where the first selection was not on target (i.e. the target button was missed). The data is averaged for blocks of 10 trials across subjects. There is a clear improvement in performance across all three protocols during the first 20 trials, after which there is no apparent further improvement. There is a fixed ordering of error rates between protocols, and averaging over the last 30 trials (after performance had stabilised) the error rates were Hardware Button 5.5%, Dwell 21% and Wink 40%. Although these error rates would seem high, we must view the performance in the light of the fact that this is the worst performance encountered, and that the error rate for larger targets was low. The higher error rate in the Wink condition and the increase in selection time for this condition is possibly due to movement induced in the 'cursor controlling' eye by closing the 'button' eye, which was observed to cause the cursor to be moved off target before the selection event was detected. The closure of one eye usually results in the lowering of the upper lid of the other eye, and if this partially obscured the pupil then the cursor on the screen would also be lowered — thereby causing an increase in errors. This problem can be overcome easily by simply taking the cursor position prior to the wink onset as being the screen point to be selected. Concerning the dwell protocol, there is trade-off between the size of area on screen within which the cursor must remain for it to be considered as 'stationary' and the risk of missing the target if it is small in comparison with the 'stationary' area. Selection times for the other dwell conditions investigated which used larger acceptance areas were faster than those shown

206


I

A

Real Time Wink

A

Moded Wink

H c o

t> Target Distance

T3 4.8mm

29.2mm

68.2mm

Figure 4: Selection times for different dragging distances.

in Figure 2, but the number of error-free trials was much lower and error rates consequently much higher. In conclusion, the Hardware button shows the best overall performance. The Wink protocol offers the prospect of a usable protocol with large target sizes and where the cost of a miss is small. It is relatively easy to play (and complete) the Windows game of Solitaire, which requires clicking and dragging piles of cards, using the Wink protocol. However, both of the eye-based protocols are associated with higher error rates for smaller targets, which presently limits the usability of the eyetracker. On the other hand, the relatively low error rate with the Hardware Button is encouraging for the use of the eyetracker for cursor positioning together with an external selection device, such as a blow tube, for target selection.

3.3. Investigation of Text Selection Under Eye-Control This experiment compared performance between the eye-tracker and the mouse for a more complicated task involving click-and-drag to select a piece of text. Using these two input devices, the effect of selection protocol ('real-time' wink and moded wink) and target size (font size) were studied. The 'real-time' wink protocol was the same as the Wink protocol studied in the second experiment. The moded wink is a monocular variation of this in which a first wink sends a button-down event and a second wink sends a button-up event. Moving the eye between winks with both eyes open is equivalent to moving the mouse with the button down. This means that only one eye need be tracked, and hence a monocular tracking system is sufficient, although as pointed out earlier additional feedback is necessary to indicate to the user whether the device is currently in 'button-up' or 'button-down' mode. Nine subjects completed 3 trials of 63 runs each, using both eyetracker and the mouse. Each trial was partitioned into two sets of runs containing 27 and 36 runs respectively. The first set contained 3 levels of distance to the start of the target text from the 'start' button and three levels of target text size. The second set contained two levels of selection protocol, three levels of distance of target text to be dragged over and three levels of target text size. Each subject was given a brief practice session of 20 minutes with each device. The subject was required to select a section of highlighted text from within a body of text, each trial being started with an on-screen button outside the text body. Selection required the cursor to be moved to the start of the target text, followed by a click action, followed by dragging the cursor across the text with the 'button' down and finally releasing the 'button' at

207

Keeping an Eye on your Interface 501

A O

• 0.37

0.46

0-62

Real Wink ModedWink Mouse

Target Size (degrees)

Figure 5: Error rates during dragging for different font sizes.

the end of the piece of text. Figure 4 shows the effect of the length of the text to be selected on the selection times for the input devices and protocols. The times are averaged over the three different font sizes used in the experiment. Considering speed of dragging (selection of the far end of the target text) the mouse was significantly faster than the eyetracker for all dragging distances. The moded wink condition for the eyetracker produced slightly, but significantly, shorter selection times than the real-time wink condition. It was noted during the trials that the closed state of the 'button' eye during dragging caused the fixation point of the cursor eye to be lowered slightly. The subject had to compensate for this which made the dragging procedure more difficult and time consuming. This is a similar artefact to that observed previously during selection of targets with the Wink protocol, in experiment 2. The 'target width' was the width of the area within which the first or the second 'click' action could be made. This area increased with font size. Although not shown in the figure, it was also found that selection times for the eyetracker conditions were reduced by increasing the target width whereas these times for the mouse were unaffected. In this experiment, as before, error rates are defined as the percentage of trials where selection attempt was outside the target area. Error rates for both devices and protocols are shown in Figure 5 for the different font sizes used and include errors made both in selecting the beginning of the text section and its end. The trials were conducted at a viewing distance of 600 mm so that the target sizes in Figure 5 correspond to target widths of 3.9 mm, 4.8 mm and 6.5 mm. The cost of a miss is, relatively speaking, very high in the context of text selection. If the error is made when anchoring the end of the selected text then the whole operation, including anchoring the start of the selected text, has to be repeated. For applications such as this one, the error rates with the eye-based protocols and the eyetracker are, at present, very high — even with large font sizes. This constitutes the main obstacle to the use of a mouse-emulator for text selection. However, areas for system modification have been identified which will result, it is anticipated, in a reduction of these error rates.

208


4. Conclusions A major influence on the error rates recorded for the eyetracker is the deviation between the point of gaze and the position of the cursor on the screen. This can be attributed to a combination of factors including noise in the measurement system, errors and drift in the initial calibration as well as jitter in the eye movements themselves. In the first two experiments, subjects were able to adopt strategies to compensate for small disparities, such as fixating to the side of the target and observing in peripheral vision whether the cursor was bounded by the target button. Use of a similar strategy was much harder in the case of text selection as it was necessary to ascertain which two letters bounded the cursor, which was far more difficult using peripheral vision. The accuracy and stability of the screen cursor with respect to the point of gaze will limit the extent to which one can achieve successful interaction with small application, and window system, objects. Work is at present underway to examine the use of adaptive filtering techniques to improve stability. An alternative route is to accept the inherent device inaccuracies and concentrate on virtual keyboard emulation as a means of interacting with GUI's. In general, authors of GUI software provide key-based alternatives to the mouse for interacting with control objects. Typically, within a dialog box, it is possible to use the tab key to move between control elements and use the cursor control keys for operations such as changing list selections. Menu items can be selected via modified character identifier (accelerator) keys (e.g.falt] + [c]). Small, customised, on-screen keyboards which enable direct access to these commands offer an alternative to the primary normal means of interacting with GUIs, namely via the mouse. Using this approach may be far more fruitful than attempting to develop a more accurate and precise system which would enable direct emulation of mouse input. Here, one is able to accept the lower pointing accuracy and precision of a low-cost eye-tracker together with a greater degree of tolerance to head movement. Using this approach, however, one has to accept the restriction that most interaction is accomplished by screen-based keys which map directly onto equivalent keyboard commands, rather than having direct control over the graphical interface. The ergonomic problems associated with the relatively high levels of visual and postural fatigue currently associated with extended periods of device use need careful attention. This presents a greater challenge if one is to satisfy the goal of a low cost system, whilst ensuring a sufficient degree of device accuracy and allowing the user greater freedom of head movement to alleviate postural and fatigue problems. This requirement makes the use of a head-mounted device attractive in spite of the additional obtrusiveness imposed by the device itself. Finally, the issue of initial calibration and correction of calibration values by the user needs to be addressed if a person is to be able to use the input device largely unaided. Although problems remain to be solved, encouraging progress has been made with respect to each of these challenges and the authors believe that eye-gaze interaction at low cost with GUI's is feasible. It now remains to determine the optimum approach to realise this.

Acknowledgements Several research students have made valuable contributions to the work described in this paper, in terms of ideas as well as implementation and running experiments. The contributions of Simon Paterson and Chris Schrimpl (Department of Computing Science, De Montfort


209

University) and Darren O'Connor and Simon Layton (Department of Human Sciences, Loughborough University) is gratefully acknowledged.

References ASL Inc (1992), Eyetracker Instruction Handbook, 3rd Edition, Applied Science Group Ltd. Card, S, English, W K & Burr, B (1978), "Evaluation of Mouse, Rate-Controlled Isometric Joystick, Step Keys and Text Keys for Text Selection on a CRT", Ergonomics 21, pp.601-613. Downing, A R (1985), "Eye-Controlled and Other Fast Communicators for Speech Impaired and Other Physically Handicapped Persons", Australian Physical and Engineering Sciences in Medicine 8 (1). Hutchinson, T E, White, P, Martin, W N, Reichert, K C & Frey, L A (1989), "Human-Computer Interaction Using Eye-Gaze Input", IEEE Transactions in Systems, Man and Cybernetics 19 (6). Jacob, R (1993), "Eye Movement-based Human-Computer Interaction Techniques: Towards Non-Command Based Interfaces", in Advances in Human-Computer Interaction, R Hartson & D Hix [eds.] #4, Ablex. Jacob, R, Leggett, J, Myers, B & Pausch, R (1993), "Interaction Styles and Input/Output Devices", Behaviour & Information Technology 12 (2), pp.69-79. Rosen, M J & Durfee, W K (1978), "Preliminary Report on EyeCom: An Eye Movement Detection and Decoding System for Non-Verbal Communication", in Proceedings of the Annual Conference on Systems and Devices for the Disabled, pp. 167-171. Shaw, R, Crisman, E, Loomis, A & Laszewski, Z (1990), "The Eye-Wink Control Interface: Using the Computer to Provide the Severely Disabled with Increased Flexibility and Comfort", in Proceedings of the 3rd Annual IEEE Symposium on Computer-Based Medical Systems, IEEE Publications, pp. 105-111. Starker, I & Bolt, R A (1990), "A Gaze-responsive Self-disclosing Display", in Proceedings ofCHI'90: Human Factors in Computing Systems, J C Chew & J Whiteside [eds.], ACM Press, pp.3-9. Ten Kate, J, Frietman, E, Stoel, F & Willems, W (1980), "Eye Controlled Communication Aids", in Medical Progress Through Technology #8, Springer-Verlag, pp.1-21. Ware, C & Mikaelian, H (1987), "An Evaluation of an Eyetracker as a Device for Computer Input", in Proceedings ofCHI+GI'87: Human Factors in Computing Systems and Graphics Interface, J M Carroll & P P Tanner [eds.], ACM Press, pp.183-188.

A Linguistic Approach to Sign Language Synthesis Alan Conway & Tony Veale Hitachi Dublin Laboratory, O'Reilly Institute, Trinity College, Dublin 2, Ireland. Tel: +353 1 6798911 Fax: +353 1 6798926 EMail: [email protected] This paper describes a linguistically motivated approach to synthesising animated sign language. Our approach emphasises the importance of the internal, phonological structure of signs. Representing this level of structure results in greatly reduced lexicon size and more realistic signed output, a claim which is justified by reference to sign linguistics and by examples of sign language structure. We outline a representation scheme for phonological structure and a synthesis system which uses it to address these concerns. Keywords: deaf sign language, phonological structure, human animation.

1. Introduction The sign languages used by the deaf are a striking example of the diversity of human communication. On the surface, visual-gestural languages appear entirely dissimilar to verbal languages. It is a common misconception that signs are a form of pantomime and that they cannot convey the same range of abstract meanings as words. However, research has shown that this is entirely untrue (Klima & Bellugi, 1979). Sign languages are languages in the full sense of the word with all the expressive power of verbal languages. In this paper we present an approach to the synthesis of animated sign language which focuses on the internal structure of signs. Several authors have discussed the translation of verbal language into sign language and the visual presentation of sign language via 3D graphics (Holden & Roy, 1992; Lee & Kunii, 1992; Patten & Hartigan, 1993). However, these authors seem to regard the sign as a unit which requires no further analysis. Sign linguists tell us that signs have internal structure and are built from more fundamental units. We argue that representing this level of structure in a synthesis system is essential for the synthesis of native sign languages.

272

Alan Conway & Tony Veale

We distinguish two types of signing, borrowed sign systems and native sign languages. Borrowed systems like ISE (Irish Signed English) are sign for word transcriptions of verbal language into the manual mode. Native sign languages, such as ISL (Irish Sign Language), BSL (British) and ASL (American), have evolved within deaf communities for day to day communication. These languages have developed grammatical structures, distinct from spoken language grammar, which are better adapted to manual communication (Kyle & Woll, 1985; Liddell, 1980). The sign-as-unit approach is adequate for the animation of borrowed sign systems, but synthesis of native sign languages requires consideration of the internal structure of signs. There are two factors which motivate this contention; lexicon size and fluidity of articulation. Regarding lexicon size; native sign languages are heavily inflected, and to avoid including every inflected form in the lexicon the system must understand sign structure and inflection rules. Regarding fluid articulation; observation of human signers shows that signs vary considerably in form during signed conversation. To mimic such variations the system must understand which elements of sign structure may vary, and which must be preserved.

7.2. Why Synthesise Sign? Before going on to explain our approach, we will briefly address the question: why synthesise sign? After all, the standard keyboard, mouse and text/graphics screen is a manual-visual interface which requires no powers of hearing. While this is true, and standard computer interfaces are useful tools for the deaf, the goal of human-computer interaction research is always to make machines more accessible to human users. Speech synthesis and natural language processing research have pursued this goal with regard to verbal languages, there is no reason why the native languages of the deaf should be excluded: for many of the deaf in English speaking countries English is a second language. At present there are no writing systems for ISL, BSL or ASL in common use, though several notations have been developed (Prillwitz et al., 1989; Stokoe, Casterline & Croneberg, 1965; Sutton, 1990). The only generally accessible way to display sign language information is in the form of moving visual images. Video clips only allow a limited number of pre-recorded messages to be displayed, and require a fluent human signer to be available to record them. Flexible display of large amounts of data requires sign synthesis. There are several practical applications for sign synthesis, the most obvious being in the field of education. Educational packages for pre-literate children using speech and graphics are common, similar packages for deaf children need sign display capabilities. Hearing people learning sign language would also benefit from such technology. Sign dictionaries, which currently rely on static figures or unfamiliar notations, can be replaced by interactive animated dictionaries. In the future, machine translation will allow verbal language text to be automatically translated into animated sign, providing valuable news and information services for the deaf when no human translator is available. Further ahead, two way speech to sign translation can provide confidential interpretation where a human interpreter is an intrusion, for example between a doctor and patient.

1.2. Overview of Paper In Section 2 we will outline the architecture of a sign synthesis system based on the principle of phonological structure representation. In Section 3 we will outline the phonological description language (PDL) which we have developed to represent signs in the system, and

A Linguistic Approach to Sign Language Synthesis

213

Lexical sign sentence

Animated Visual Display

Figure 1: Overview of the sign synthesis system.

discuss the organisation of the lexicon. Section 4 focuses on the issues of fluid articulation from PDL structures. Finally Section 5 concludes by summarising the current state of our research and indicating our plans for future work.

2. The Sign Synthesis System The objective of the sign synthesis system is to produce a concrete visual animation sequence from an abstract symbolic description of a sign language sentence. This is accomplished in three stages: lexical, phonological and phonetic. The input to the system is a lexical description of the sentence, specifying the order of signs in the sentence and the grammatical modifications to apply. The phonological level is an intermediate representation of sign structure, in terms of the primitive elements of the sign language. We have developed a phonological description language called PDL to represent this structure. The phonetic output of the synthesis system is a detailed script of the movements required to articulate the sentence, this is used to drive the animated model. The system structure is shown in Figure 1.

2.1. Lexical Input The input to the system is a lexical representation of a sign phrase. This identifies the signs which make up the phrase, and the order in which they are signed. For a spoken language like English a list of word tokens is a sufficient lexical description but sign languages require a more complex representation. As well as linear sequence, sign sentences may have parallel elements. Non-manual markers, such as facial expression and body movements, play an important role in syntax (Liddell, 1980), and occur concurrently with manual signing. In some circumstances, the hands may be used independently to simultaneously articulate two signs. We will use a notation borrowed from Liddell. Signs are denoted by their glosses, i.e. their approximate English translation, written in upper case. Modifications to signs are indicated by raised square brackets. Inflections are denoted by T , for example the phrase 'very sick' is translated in ASL by the sign SICK with intensive inflection, a tense initial hold and rapid movement, (Klima & Bellugi, 1979) This is written siCK [I:intensive] . Non manual markers

214


are written on a bar over the signs they co-occur with. The question 'did the dog chase the cat' is translated in ASL as DOG CHASE CAT q i.e. the sequence of signs DOG CHASE CAT with the special ASL 'question' expression. Holds and simultaneous signing are represented by horizontal bars, for example the BSL translation for 'small boy', which is made by signing BOY with one hand and holding while signing SMALL with the other is written: BOY gMj\LL • In Japanese sign language, the phrase 'eating while watching TV can be translated as WATCHING - TV» u s m § b ° m hands to sign simultaneously. 2.2. Phonological

Structure

The intermediate phonological level represents the internal structure of signs in terms of phonemic elements. The terms 'phonological' and 'phonetic' are used by analogy to their meaning in spoken language linguistics. Phonology deals with the primitive distinguishable units which a language uses to construct lexical items. In the case of spoken languages, these units are phonemes — the basic set of sounds which are regarded as distinct and recognisable elements of words. The phonemes of sign languages are body locations, hand configurations and movements. As with spoken languages, each sign language has its own set of distinct recognisable units in each of these categories, for example (Stokoe, Casterline & Croneberg, 1965) identifies 55 such units for ASL. It is important to distinguish the phonological structure of a sign from a phonetic description of a performance of that sign. Phonology specifies the primitive elements of a sign's structure which are necessary to distinguish it from other signs in the lexicon. A phonetic description specifies the details of a particular performance of a sign. Individual performances of a sign may vary in the phonetic details of execution while still being recognisable as the same sign, i.e. having the same phonological structure. The lexicon specifies the phonological structure of each sign using the PDL language outlined in Section 3. There are two key benefits to using a phonological representation, the first being lexicon size. As mentioned before, native sign languages are heavily inflected, so each sign will have many different forms, each expressing different meanings. The lexicon contains only the citation forms of signs, i.e. their neutral or uninflected form. A set of inflection rules is used to generate inflected forms from citation forms. The separation of the lexicon into citation forms and inflection rules is much more compact than storing every inflected form explicitly in the lexicon. This will be discussed further in Section 3. The second benefit is in the fluid articulation of signs. As we have pointed out, there is a difference between phonetic and phonological descriptions. A sign in the midst of a conversation may be phonetically very different from the same sign produced in isolation. Signers adjust the articulation of signs to fit into the flow of a sentence and make it more fluid, but these adjustments are not arbitrary. A sign may not be adjusted in ways which alter its phonological structure, for then it becomes a different sign. Knowledge of phonological structure allows the system to adjust its output for fluid articulation without transgressing the limits of sign flexibility. This will be discussed further in Section 4. 2.3. Phonetic Description

and the Output

Model

The output of the system is a 3D animated graphical sequence. The sequence is generated by a 3D model, or doll, controlled by a set of parameters which define its pose. The phonetic


215

description of sign articulation consists of a parallel stream of animation events which are fed to the animation engine controlling the 3D model. Each event specifies a change in one of the parameters. Our current model has the following parameters: facial expression (chosen from a fixed set), head orientation, upper body orientation, shrug angle, hand position, palm orientation, elbow raise and hand shape. Hand shape is specified as a set of 3 joint angles for each finger: bend and spread at the base knuckle and curve at the middle knuckle. The top knuckle angle calculated as 2/3 the curve, following (Rijpkema & Girard, 1991) The thumb is specified by spread, twist (rotation towards palm) and curve. Hand position is specified by a pair of locations: the target in signing space and the focus on the hand. Both these locations are specified by 3D vectors relative to a body segment. The arm position required to place the focus at the target is calculated by reverse kinematics, with the elbow raise parameter selecting from the circle of possible elbow positions. Thus the hand location 'index finger touching tip of nose' can be translated into 'target = tip of nose, focus = tip of index finger'. Orientations are stored using quaternion representation, which has implementation advantages for animation over more traditional matrix or Euler angle representations (Shoemake, 1985). The performance of a sign is specified as a sequence of events. Each event is a period of time over which a doll parameter is changing. It is defined by the time interval, the initial and final values of the parameter and interpolation information which indicates how the parameter changes. The interpolation data includes a velocity profile which indicates the rate of acceleration and deceleration of parameter change over the interpolation period. This determines whether motion appears smooth, tense or jerky. For location interpolation (i.e. hand movement) the shape of the interpolation path in space is also specified. Events arrive at the animation engine in parallel streams, one for each doll parameter. The animation engine maintains an event queue for each parameter and a queue of pending frames. Events arrive in order of their start time and are placed on the queues until there is at least one pending event on each queue. Then events which start at the current time step are processed, with the initial parameter values being stored in the current frame and future interpolated values stored in the frame queue. Any parameters left unset in the current frame are copied from their previous value. Finally the current frame is displayed and removed from the head of the queue, and the process is repeated.

3. Phonological Structure and the Lexicon The first reason for introducing a phonological level of representation to our sign synthesis system is to allow efficient organisation of the sign lexicon. Sign languages are heavily inflected but, unlike spoken languages, the inflections are not sequential suffixes or affixes but parallel modifications to the form of the sign. Inflections cannot be represented as separate articulations which are concatenated with signs. They can only be represented if we consider signs, not as indivisible units, but as structures made up of smaller elements. Inflection rules are operations on these structures which add, delete or replace elements. For example, in ASL the phrase 'sickly' or 'sickness prone' can be translated by the sign SICKt I: P redis P° sitional l The predispositional inflection can be applied to many other signs, e.g. SILLY, WRONG, DIRTY, QUIET and so on (Klima & Bellugi, 1979). This is a regular inflection which operates by adding circular movement.

216


To illustrate the importance of capturing these regularities, consider that Klima and Bellugi list 8 different aspectual inflections of ASL verbs and 7 inflections for number and distribution. Deictic inflection of directional verbs adds another inflection for every possible pair of subject and object. Thus the ASL verb LOOK-AT can be modified to mean 'watch regularly' (aspectual), 'look at each one' (number and distribution) or 'I look at you' (deictic). Further these inflections can be compounded, so the single sign LOOK-AT can be inflected to mean 'he stared at each of them in turn'. Considering all inflections and compounds, the sign LOOK-AT has literally hundreds of possible forms. It is not only wasteful to record all of these separately in the lexicon, it is clearly impractical. The problem of lexicon size can be tackled by including only uninfiected citation forms in the lexicon and storing a separate database of inflection rules. Inflected forms are generated by applying these rules to the citation signs. This allows the system to use a lexicon which is drastically smaller than one which explicitly stores all inflected forms. We will now outline the representation language for phonological structure which is used by our system and then explain how inflections can be captured by appropriate lexicon organisation.

3.7. The PDL Model of Sign Structure In order to represent phonological structures, we have developed a description language called PDL. The PDL model is not intended to be a detailed phonological model for a particular language, but a framework for representing the phonology of any sign language. We have therefore made the model as general as possible, and avoided language specific constraints. The model views signs as sequences of states and transitions, similar to the Location and Move segments in Sandler's hand-tier model of sign phonology (Sandier, 1989). Each state and transition specifies concurrent values for sign parameters so both simultaneous and sequential aspects can be expressed. A state represents a 'snapshot' at an instant in time, while a transition represents activity over a period of time. There are five independent parameters in the PDL model: hand shape (HS), palm orientation (ORI), hand location (LOC), elbow position (ELB) and non-manual markers (NMM). Each state or transition may specify values for any or all of these parameters. In terms of the classical analysis of signs, HS and ORI states together define hand configuration, LOC states define locations, and transitions define movements. Path movement is defined by LOC transitions, while internal movement is given by HS and ORI transitions. Elbow location is not mentioned in the ASL or BSL literature, but some ISL signs require it — for example the ISL sign KEEP is made by holding the fist at shoulder height and bringing the elbow in to the side of the body. The NMM parameter designates linguistically significant non-manual activity. PDL descriptions will be indicated graphically by putting states in rectangular boxes with concurrent parameters specified vertically. Transitions are marked on arrows between states. Blank states indicate no change from the previous value, unmarked transitions are defined by the states on either side, simply taking the shortest path between the two. Marked transitions (in particular LOC transitions) indicate a special manner of moving from one state value to the next. An example will clarify these points. The PDL representations of the ASL signs SICK and SICK [I: P redis P° sitional] are shown in Figure 3 (the signs are illustrated in Figure 2). The notation 'body-location « hand-location' indicates contact between the indicated location on the hand and that on the body, in this case the middle fingers touching the forehead and

217


SICK [I: predispositional]

SICK

Figure 2: The uninflected sign SICK and the inflected version siCK [l: P redis P osilional] , which can be translated as 'sickly' or 'sickness prone'.

NMM HS LOC ORI Left HS LOC ORI

Riqht

Pained-fane Bfint-mifldlfi Chest« middle

-Tap-^

Bent-middle Forehead« middle - T a p - ^ SICK

Painari-face Bent-middle Chest« middle

-circle-^

Bent-middle Forehead« middle - c i r c l e - ^ SICK P : predispositional]

Figure 3: PDL descriptions of the ASL signs SICK and siCK[I:PredisPositiona11, meaning 'sickly' or 'sickness prone'.

chest. Body locations without explicit contact indicate proximity of the hand to that location. One hand may also be located by contact or proximity in relation to the other. Note that the final states of both signs are empty, because they are the same as the initial states. The tap transition indicates that the initial contact is broken by a slight movement away from the body and then re-established, the circle transition indicates a circle which returns to its starting point. Also note that no ORI is specified, it is implied by the HS and the LOC values which specify contact between the middle finger and chest.

3.2. Lexicon Organisation Using PDL Signs are stored in the lexicon in their citation form as PDL sequences. Sign modification rules are implemented as transformations or editing rules which take a citation form PDL sequence and return the PDL sequence for the inflected form. However, there is an additional complicating factor which must be taken into account. According to Klima & Bellugi (1979) inflections operate on the 'underlying form' of a sign, which may have features not apparent in the surface form. The result of this is that the same inflection may have different effects on different signs. PDL descriptions represent surface form only, to deal with underlying form we take an object oriented approach to the sign lexicon. The lexicon is organised as a multiple inheritance

218

Alan Conway & Tony Veale T

Figure 4: Predisposilional inflection rule.

Right LOC I ~ X ~ I — ? — •

v > r

Right LOC Left LOC I Neutral

Figure 5: Predispositional inflection for underlying-two-handed signs in ASL.

class hierarchy, and modification rules are methods on this hierarchy. Each modification rule is defined over a class of signs within the hierarchy, override methods may be defined for subclasses with different behaviour. To illustrate this point, consider the predispositional inflection of SICK illustrated in Figure 2. As we can see from the PDL forms of SICK and siCK [I: P redis P ositional] (Figure 3), the inflection rule is as shown in Figure 4. The same inflection rule applies to the signs ROUGH and DIRTY, which also inflect for predisposition by changing their transitions to circle transitions. However, signs like WRONG and SILLY, which are one handed in citation form, react differently to this inflection. In the predispositional aspect these signs become two handed, as well as having a circular motion added. This difference in behaviour cannot be anticipated from the surface form of the sign; ROUGH and DIRTY are one handed like WRONG and SILLY, but they do not become two handed under inflection, they simply change their LOC transitions like SICK. The explanation is that signs like WRONG are underlying two handed signs, even though the citation form is one handed. Under inflection the underlying two handedness emerges. In the sign lexicon, such signs are made members of the class underlying-two-handed. Instead of the default predispositional method, underlying-two-handed signs are inflected using the overriding method shown in Figure 5. There are a number of other types of sign modification, but they can all be accommodated within the framework of underlying classifications and modification methods. For example directional verbs are modified for subject and object by replacing the initial and final LOC parameters of their PDL descriptions. Reversing verbs, an underlying subclass of directional verbs, also require reversal of their ORI parameters (Kyle & Woll, 1985) Some two handed ISL verbs undergo contextual modification. These verbs belong to the underlying class weaktwo-handed, and when sandwiched between two one handed signs they become one handed (for example the sign WANT in YOU WANT DRINK q , see Figures 6 and 7). To summarise the process of sign look-up: the system begins with a lexical sign sentence, containing sign tokens and modifiers. The citation form of each sign is looked up in the lexicon, giving an initial PDL structure. Each of the modifiers attached to the sign is applied using the methods appropriate for its classification in the underlying class hierarchy. Any necessary contextual modifications are similarly applied. Finally the signs are concatenated

219


YOU

WANT

DRINK

Figure 6: The citation forms of ISL signs YOU, WANT and DRINK.

Figure 7: The ISL sentence YOU WANT DRINK 'Do you want a drink?'.

into a PDL stream, adding any NMMs which are marked on the lexical input. The result of this process is a PDL sequence representing the entire sign sentence.

4. Fluid Articulation of Signs The second reason for introducing phonological representations into the sign system is to allow the production of more fluid and realistic sign output. If signs are generated in isolation the output has an artificial 'robotic' quality, even if the motion between signs is smooth and natural. This is because signs in conversation are rarely made in the same form as when they are produced in isolation. The performance of each sign is adjusted in the context of surrounding signs to reduce unnecessary movement. To illustrate this point, consider the ISL translation of 'do you want a drink?' In citation form the individual signs YOU, WANT and DRINK are as shown in Figure 6. However in the question YOU WANT DRINK q both the signs WANT and DRINK move upward considerably from their citation locations, as shown in Figure 7. WANT also becomes onehanded — this is a contextual phonological change as explained in the last section. However, apart from this hand deletion, the signs have not changed phonologically from their citation

220

Alan Conway & Tony Veale neutral D-hand fingers-out

YOU

-out-^-j

- down - •

WANT

§H

mouth C-hand tingers-out

tingers-up

DRINK

Figure 8: The phonological structure of the ISL sentence YOU WANT DRINK q .

form. YOU and WANT are phonologically specified as neutral space signs; they are made in the neutral region in front of the body. In YOU WANT DRINK they are still in neutral space, but they have moved up from the centre of the neutral space zone. Note that the sign DRINK does not move appreciably from its citation form, because this would alter its phonological structure. DRINK is phonologically specified as being made on the mouth, it must be signed at the mouth to be correct. In this sentence DRINK has a dominating influence over the preceding signs. YOU and WANT have flexible locations, so they are adjusted to accommodate the more tightly located DRINK. This example demonstrates the importance of phonological structure in generating correct, fluid output. It is crucial that the system understand what is a permissible variation in the performance of a sign; i.e. what variations may be made without changing the phonological structure. We will now show how PDL descriptions can be flexibly mapped to phonetic descriptions, taking account of contextual effects, to produce realistic output.

4.1. Mapping PDL to Animation Events To create animated sign output, PDL state-transition-state triples must be mapped onto animation events. However, this mapping is flexible; PDL parameters do not always map onto identical output values. PDL provides an outline description of a sign, defining the essential elements and the range of possible variation. In mapping PDL states and transitions onto animation events the system takes account of the adjacent states and transitions, and anticipates the influence of dominating signs later in the phrase. This is particularly important in mapping LOC values onto model co-ordinates. Each LOC state is associated with a zone in signing space, with a central co-ordinate and a radius of elasticity. The radius indicates the limits within which the hand must lie to be inside the zone. Zones like 'neutral', 'left-neutral', 'chest', 'face' have large radii compared to zones like 'mouth', 'nose' etc. The default mapping for a zone in the absence of contextual information is the central point. In the context of a PDL sequence the mapping is determined by the default mapping, the current doll position, the current state-transition-state triple and any dominating locations in the remainder of the sign phrase. A dominating location is one with a smaller radius of elasticity than the current location and a distant centre. Consider the example illustrated in Figures 6 and 7. After the application of the hand dropping rule to WANT, the PDL description for this phrase is as shown in Figure 8. The first statetransition-state triple is from the initial relaxed state via a default in-between transition into the first state of YOU. The default mapping for the neutral LOC of this sign is the centre of the neutral zone. However, scanning ahead we find a dominating mouth LOC at the start of the sign DRINK, so the default position is adjusted upward within the neutral zone toward the centre of the mouth zone.


221

Next the out transition of YOU leads to an unmarked state, indicating that the sign is still in the neutral zone. This time the determining factors are the current position (high neutral) and the movement implied by the transition. Motion specified in adjacent transitions overrides anticipation of dominating influences. However, the final position after this transition will still be high, since the starting point for the out movement was adjusted upward. Now we move to the first state of the sign WANT, via another default transition. WANT is another neutral zone sign, and the hand is already in the neutral zone, so no motion is necessary except to change the hand shape. The 'down' transition forces a lowering of the hand, and again overrides the dominating mouth state. Now we move to the final sign, which has a small, inelastic LOC — the mouth. There are no further dominating states to consider, so the initial state of DRINK gets it's default mapping. Finally the last transition — a wrist rotation to change orientation — is mapped, and the sequence is complete. This example illustrates how PDL descriptions can be flexibly mapped onto animation events to produce fluid, natural looking signs. Detailed rules of influence are the subject of ongoing research, but the approach outlined here provides a framework for a system which can mimic the environmentally conditioned variations of sign performance.

5. Conclusion In this paper we have stressed the importance of a phonological level of representation in a sign synthesis system. There are two reasons for this. First, it allows regular grammatical features of native sign languages, such as directional verbs and aspectual modifications, to be expressed as transformation rules in the system. As a result, the size of the sign lexicon is drastically reduced. Second, generation of fluid sign output requires that the system adjust the form of signs according to context to reduce the amount of unnecessary motion. This can only be done if the system knows which aspects of signs may be altered without creating errors, and which are essential to recognition of the sign. The distinguishing features of a sign are precisely those characterised by a phonological description. We have outlined a framework for the synthesis of animated sign language which is divided into three representation levels: lexical, phonological and phonetic. This breakdown is motivated both by linguistic parallels and implementation concerns. We have outlined the structure of a phonological description language (PDL), and shown how it can be used to solve the problems of lexicon size and fluid articulation. However, this is still only a framework. Implementation to date has focused on the phonetic level — building the output model and animation engine. The next step is to analyse the phonological elements and inflection rules required to represent a specific sign language, and implement them in the PDL formalism. Most of the examples quoted in this paper are from ASL and BSL, as these languages are well documented in the literature. In the future however, we intend to focus on ISL, as that is the native sign language of Ireland where this research is based. Co-operation with native signers will be crucial to the success of this endeavour, for ultimately they are the arbiters of what is acceptable sign and what is not. In developing a computational phonology for ISL, we will no doubt encounter unanticipated sign phenomena. However, we are confident that our framework is general enough to absorb new features while maintaining it's central philosophy. The structured use of space and motion

222


to encode grammatical information seems to be a universal feature of native sign languages, and one which challenges sign synthesis systems. We believe that this framework provides a sound basis for tackling that challenge.

References Holden, E J & Roy, G G (1992), "The Graphical Translation of English Text into Signed English in the Hand Sign Translator System", Computer Graphics Forum 11 (3), pp.357-366. Klima, E & Bellugi, U (1979), The Signs of Language, Harvard University Press. Kyle, J G & Woll, B (1985), Sign Language, The Study ofDeafPeople and their Language, Cambridge University Press. Lee, J & Kunii, T L (1992), "Visual Translation: From Native Language to Sign Language", in Proceedings of the IEEE workshop on Visual Languages, pp.103-109. Liddell, S K (1980), American Sign Language Syntax, Mouton. Patten, T & Hartigan, J (1993), "Automatic Translation of English to American Sign Language", paper presented at the 1993 National Conference on Deafness. Prillwitz, S, Leven, R, Sienert, H, Hanke, T, Henning, J & et al. (1989), "HamNoSys version 2.0 — Hamburg Notation System for Sign Language: An Introduction", in International Studies on Sign Language and the Communication of the Deaf #5, Signum Verlag. Rijpkema, H & Girard, M (1991), "Computer Animation of Knowledge-based Human Grasping", Computer Graphics 25 (4), pp.339-348. Sandier, W (1989), Phonological Representation of the Sign: Linearity and Non-linearity in American Sign Language, Foris Publications. Shoemake, K (1985), "Animating Rotation with Quaternion Curves", Computer Graphics 19 (3), Proceedings ofACMSIGGRAPH'85. Stokoe, W C, Casterline, D C & Croneberg, C G (1965), A Dictionary ofAmerican Sign Language on Linguistic Principles, Linstok Press. Sutton, V (1990), Sutton'sASLIEnglish, English!ASL Dictionary, Deaf Action Committee for Sign Writing.

Generalisation and the Adaptive Interface Jonathan Hassell & Michael Harrison Human-Computer Interaction Group, Department of Computer Science, University of York, Heslington, York YO1 5DD, UK. Tel: +44 (0)904 432765 Fax: +44 (0)904 432767 EMail: {jph, mdh}@minster.york.ac.uk Automated macro systems which apply re-use to a user's input are a possible solution to the problems of customising an interactive system to the needs of the user. More useful than simple re-use would be a system that makes use of general patterns in users' behaviour and encapsulates this knowledge for application in similar, yet unfamiliar, circumstances. This process we term generalisation. This paper outlines some issues involved in controlling generalisation and the presentation and interaction with these macros, and specifies applicable heuristics. Finally the architecture for building an adaptive agent to perform the whole process is presented, with an example prototype operating on UNIX command-line interaction. Keywords: adaptive interfaces, interface customisation, interface transformation, command reuse, agent modelling, initiative.

1. Introduction One example of demonstrational interfaces (Myers, 1991) — automated macro creation — has been shown to be a promising area for adaptive system research previously by Greenberg (1990) and Crow & Smith (1992). Crow & Smith have extended the simple re-use of previous command entries (the history/tool-based systems of Greenberg) from a single line to an inferred macro. Macros are a concept that users are already familiar with for automation. Re-use, however, is limited to situations corresponding exactly to those which have occurred before. Whilst it has been shown that these situations happen reasonably frequently (Greenberg & Witten, 1993a; Greenberg & Witten, 1993b) for single line re-use (a result which has not been investigated for multi-line re-use), both single-line and multi-line macro re-use break down in situations which differ slightly from the original. In both cases the

224

Jonathan Hassell & Michael Harrison

re-use system is of no help. This is not so important when the unit of re-use is small, as the overhead of manually entering the new command is correspondingly small. However, in the case of a larger unit of re-use (a twenty line macro differing in only one line, for instance) this re-entry overhead is considerable. It would seem that, if a more general version of the macro could be found which could cope with similar but different cases, this would be of greater value than the original macro. The process of attempting to find this more general form of any given macro can be considered as abstracting away the variable parts of the original macro leaving variable place-holders into which new values can be placed — a process we call generalisation. A second question that this paper addresses is that of how the system should adapt when an adaptation is deemed to be worthwhile. The success or failure of any adapted system hinges on finding acceptable forms for this adaptation to take which do not introduce unfortunate side-effects such as a lack of consistency and predictability in the interface. It is because of failures in this area that adaptive systems have derived much of their criticism. In this paper we will describe a general philosophy that should lie behind any adaptive process (that adaptivity should be embodied in an agent) and specify some requirements for presenting the buttons that are attached to the macros. This paper is split into three sections. Firstly it details the concepts used and some heuristics suggested for the process of generalisation. Secondly the paper suggests a framework and some requirements for the presentation of the macros created by the generalisation process and for the user's interaction with them. Thirdly, the techniques presented, whilst relevant to creating and presenting macros in any given macro language and presentation style, are shown in use in a prototype system which provides an additional automatic button-based macro interface to a UNIX command line. Finally some conclusions and suggestions for further work are given.

2. Generalisation Concepts 2.1. Generalisation — An Introduction The aim of this paper is to specify some rules for the process of providing the user with more general versions of their habitual tasks (which can be seen as a model of the user's work habits). Following Foley, Kim & Gibbs (1987) and Cypher (1991) we refer to this abstraction process as generalisation. In this case we will define generalisation as an abstraction away from specific instances of an expression to give their more general form and their variable parts 1 . How then is the process to differentiate between what parts of the macro are variable and what parts fixed? If two macros are taken which are seen as possibly two instances of the same generalised macro, the places in which the instances differ can be seen as the variable parts of a more general macro, and similarly the places where the two are identical are the fixed body of the new macro. To give a concrete example: if you have two functions sort ^string.array and sort-integerjarray, a more general form of the two might be sort.any.array with strings and integers as instances (variable parts); similarly, set .triangle .colour and set square.colour might be generalised to set shape .colour with triangle and square as instances. This more general macro can provide additional functionality by using inheritance. To take the shapes example: given the colour This can be seen in software engineering terms as the reverse process to inheritance.

Generalisation and the Adaptive Interface

225

operation for a square and a triangle and the generalised version (set shape .colour) of the two, if a new shape is introduced, say a circle, a set .circle-colour operation would automatically be generated by inheritance (by using circle as a new instance of setshape.colour). In practice, there should be a limit to the number of variables in the macro — if two macros are completely dissimilar the process would produce a generalised macro with no fixed section and many variable parts, which would be of no use. The setting of this variable limit must take into account the fact that the user will have to specify the concrete values of any variables extracted, before being able to execute the macro. For the macro's usage to be efficient this variable instantiation time should be minimised.

2.2. Generalisation Pivots In previous work the process of generalisation has been assumed to occur around the operator — in fact little attention has been given to any other possibilities for generalisation. There is, however, another choice — generalising around one of the operands. The choice of whether to take the operator or one of the operands as a constant we term selecting a generalisation base or pivot. For instance, in the colouring example we have generalised using the operation set shape.colour as a pivot — the fixed section of the resulting generalised macro. There is, however, another alternative: generalising around the object (the operand). For instance, we could have generalised using the object (e.g. the square) as a pivot and found all the operations that had been applied to that object. Of course, this technique is more difficult to adopt should there be more than one operand. The choice of which operand to make the pivot becomes an issue. Why might this distinction be useful and what questions does it imply? To tackle these questions we will give an example from the prototype that has been developed to study these issues. In this case the operators are the UNIX commands submitted to the command line and the operands are the data-files that they operate on. The benefits of generalising around one or other of these objects are detailed below. 2.2.1. Generalising around the Operator So what would happen if the operator was chosen to be the generalisation pivot? To give an example from the prototype: are generalised to

cc x where x has instances f i l e l . c and file2.c

Essentially this provides a history of the use of the command in question. The prototype interface uses this technique to give a history mechanism oriented around commands, giving examples of their previous usage and a mechanism for applying the commands to the file(s) of the user's choice by specifying new operands.

2.2.2. Generalising Around the Operand What sort of advantages would be achieved if the operand was chosen to be the generalisation pivot? This method can bring many benefits for modelling systems that are object-oriented or data-oriented. In these systems the data file (or object) is at the centre of the system and the various commands (or operators) are applied to it. Here then is an environment in which the operand is of greater importance and thus generalising around it may produce benefits.

226


To give an example: emacs filel.c

\

& cc filel.c

> are generalised to J

x f i l e l . c where x has instances emacs and cc

Clearly in this case the generalised form gives a history of the commands applied to the file in question and can be re-used to perform the same actions again. To extend the concept, if a new file is created it can inherit the existing generalised operations to allow the application of previous commands to the new file as a new instance. Of course, there must be a concept of file type here — defined as the data-type of the file with the commands that it is meaningful to apply to it — to guide this inheritance process.

2.3. Context The generalisation process requires an understanding of context. In fact the dependence on context can be detected by recognising algebraic and compositional properties of commands. Similar suggestions have been made by Thimbleby (1990, Chapter 4). There are a number of mathematical constructs that allow the exploration of the relationship of each command to those closely surrounding it. These may hint at restrictions on the extraction of macros from a log and the generalisations that can be meaningfully applied to those macros, by detailing the inter-relationships between command lines. These constructs can equally be applied to the relationship between macros, if the properties for a generated macro can be inferred from the properties of the commands it comprises of. These properties can also help to indicate where two macros that are syntactically different have identical semantics. This can help ensure that the macros created are efficient and that potential redundancy is removed. For example, due to the distributivity of l p over the end of line character: lp

filel

lp file2

is semantically identical to

lp f i l e l

file2

The use of pre- and post-conditions (suggested, by Foley, Kim & Gibbs (1987)) can also be of help in considering the demands of context because they provide a definition of the effect of a command. Here we present two views of context: general context and the special case of defaults and global variables.

2.3.1. General Context How does context affect the process of extracting and generalising well-formed macros? Context in this case can be seen as introducing implicit variables in the execution of a macro which do not need to be explicitly specified by the user. In the interface specification example, implicit variables were introduced into interaction by using one of two competing metaphors: "select the object(s) then select the operation on them" or "select the operation then select the object". Both these styles of interaction instantiate an implicit variable: in the first it is the selected object in question, as all subsequent operations are performed on it; and in the second it is the command, which is applied to all subsequent objects. In our example of logging the use of a UNIX command line, the context is hidden. This style of interaction precludes modes because one command line requires both operator and operands to be specified. However each command line is executed with the system in a certain state; a state


227

defined by the settings of the system (such as the path variable, the current directory and others — what are called environment variables in UNIX). The existence of various files in the current directory also changes the state — if a file has been created, the system is in a possible Jo -edit state. Do these states constitute a context in which the commands are interpreted? The current directory acts as a context for interaction. It is selected and modified in the same way as the object selection in the example above and provides the context in which any operators are executed. Foley, Kim & Gibbs (1987) use of pre-conditions for encapsulating this context information is of use here. The classification of commands into those that change the context and those that execute within it (that have it set as a pre-condition) is useful for making explicit whether the command is dependent on the context in which it is executed. This pre-condition, along with the commands to set and reset the context, allows the moded command to be executed as a unit regardless of the context preceding its execution. It would then be of use in the pre-selection of macros to check if they are not sensitive to their context and, if not, to package them with a pre-condition and context setting and resetting commands.

2.3.2. Defaults and Global Variables Secondly let us consider the more specific concepts of defaults and global variables. The use of defaults is common; they capture any information that is provided as an initial option for the user to accept or reject. It can be argued that the set of defaults for all the parts of the interface and the underlying application can be thought of as a global mode in which all of the interaction resides. This context is inescapable, but it may be altered by the user by 'changing the defaults'. Global variables can be seen to have an identical role. All interaction with a system is affected in some way by this set of defaults and global variables: the system environment. A generalisation algorithm must take into consideration the environment in which the macros were extracted. If a macro is captured in environment a, does it make sense to execute the macro in a new environment b without re-establishing environment a first? Clearly there will be a set of macros that do not rely on their environment at all and thus can be stored on their own without any concept of environment; for example, simple macros such as c d / . There are difficulties even in this example: if there is more than one root directory on the computer in question (more than one conceptual or physical disk drive or more than one file-server) then it is not possible to construct this as a macro without ambiguity. One possible solution is to store the set of defaults and global variables as part of the macro (the UNIX s e t would be useful, in the example case, to make these settings explicit and available for storage).

2.4. Definition of a button At this point we shall define the concept of a button — a term that will be used throughout the rest of this paper to specify a macro (a program in a given macro language) enriched with the pre- and post-conditions outlined above and presentation information required for the purpose of interaction. This can be seen as an extension of the concept of a button in the Xerox buttons interface (MacLean et al., 1990).

3. Generalisation Transformation Rules The concepts detailed above have suggested various heuristics that should be used in the macro extraction and generalisation processes of any system that attempts to automatically extract macros from a user log. These will be detailed in this section.

228


3.1. Context Requirements Buttons should be stored together with their pre- and post-conditions (calculated from the pre- and post-conditions of their contents) including: • Whether they change the context of the interaction by introducing implicit variables (for example: select line mode, select object, c d n e w _ d i r e c t o r y ) . • Whether they are themselves sensitive to the context of the environment in which they are to be executed or whether their function is absolute (for example: I s is context sensitive, whereas c d / and new document are context insensitive) For example: (NB. x(R)y is "any term in x not established in y"): Macro C = cl & c2 where cl - move object to x, y and c2 = rotate 90° pre(cl) = object selected and post(cl) = object now at x, y A still selected pre(c2) = object selected and post(c2) = object now rotated 90° A still selected pre(C) - object selected A (object selected ® object selected) = object selected post(C) = object now at 90° and still selected A (object now at 90° A still selected (g) object now at x, y A still selected) = object now at 90° A still selected A object now at x, y

The specification of what information should be stored in the pre- and post-conditions of a command is still a matter of ongoing research. The aim is to require the minimum amount of information needed to cope with dealing with the problems of context. 3.2. Generalisation

Heuristics

The algebraic notions mentioned above have suggested various heuristics which should be used to ensure that the macros created by the extraction and generalisation processes are efficient and not redundant. These heuristics are applied mostly to the macro instances that have been extracted from a user's log. Further research may prove that they may also be useful when applied to generalised macros — the abstractions rather than the instances. 3.2.1. Equalisation

Issues

Functional and Perceivable Equivalence We shall consider two notions of equivalence here: functional equivalence and userperceivable equivalence. Two commands or command sequences are defined to be functionally equivalent if their pre- and post-conditions are identical. Two commands or command sequences are defined to be perceivably equivalent if they are functionally equivalent and appear to have identical function to the user. For instance: I s Is is functionally equivalent to Is but not perceivably equivalent as the first macro displays two listings of the directory and the second displays only one. Generalisation Heuristic 1: If two buttons are functionally equivalent but are perceivable by the user as being different, they should both be retained. Generalisation Heuristic 2: Buttons that are simply repetitions of other buttons should be removed and a repeat n times option added to the original button.


229

Efficiency of a Button A button is defined to be more efficient than another button if it uses fewer commands than that button. 3.2.2. Algebraic properties All commands and buttons should have a set of properties for equalisation purposes: distribution, identities, inverse, idempotence, commutativity, associativity. These properties will be used at various stages in the extraction and generalisation process to remove redundancy in the macros being created. The set of properties should be specified initially for each of the commands in the macro language by the programmer, and then automatically inferred for any new buttons that are created. Distributivity The distributivity property of an operation allows the user two different ways of issuing commands: a simple step-by-step method for learners; and a more powerful method for experts. For example, if the user wishes to print two files (filel and file2) she could issue the command sequence: lp f i l e l lp

file2

or she could use the more powerful (and efficient) method: l p f i l e l distributes over the end of line character.

file2aslp

Generalisation Heuristic 3: If any two syntactically different buttons are functionally equivalent as a result of distributivity the least efficient button should be erased. Identities A command is an functional identity if it can be combined with any other chosen command and the effect is functionally equivalent to executing the command on its own. A command is a perceivable identity if it is a functional identity and appears as an identity to the user. Generalisation Heuristic 4: Commands that are perceivable identities should be removed from any macro (this will not affect its function). Inverses The inverse of any command can be combined with the command to produce an effect that is functionally equivalent to the identity. For example: cd one directory forward and cd . . backup filel and restore filel. Generalisation Heuristic 5: In any button, the redundant sequence of a command immediately followed by its inverse should be removed. Idempotence A command is idempotent if executing it more than once is functionally and perceivably equivalent to executing it once. This property allows the shortening of contiguous sequences of an idempotent command to just one instance of the command. This property, when applied to a log before macro extraction, ensures that any command sequences that are similar in everything but a repeated idempotent command are found to be equivalent and extracted. The

230


property also removes redundancy from any macros that have been created. Examples of idempotent commands are: c d / and new document. Generalisation Heuristic 6: Any sequences of an idempotent command in the log should be reduced to one occurrence of the command. Commutativity Commutativity is another useful property that can be used to match two buttons that are functionally and perceivably equivalent, yet syntactically different. Any command sequences that commute can be executed in any order to achieve the same result. To give an example: rm f i l e l rm file2

is equivalent to

rm file2 rm f i l e l

Of course the two sequences must not be separated by any other non-commutable command sequence. In practice there are constraints attached to the notion of commutativity — some commands are commutative in general but there may be some commands that are commutative in some conditions, for instance when working on distinct files. Operations that require some interaction with the user should not be termed as commutable (such as m o r e and e m a c s for example). Generalisation Heuristic 7: If two syntactically different buttons can be matched by using commutativity then one should be erased. Associativity Associativity states that bracketing (or grouping) is irrelevant in an expression. For example: cd /

/Cd /

more f i l e l p filel

\

$HOME/

is equivalent to

\

I more f i l e l J cp f i l e l $HOME

cd / is equivalent to

more f i l e l C

P

f i l e l

$ H0ME

(NB. brackets indicate commands executed by a button) Generalisation Requirement: The implicit associative grouping inherent in executing a button, rather than by entering the button's macro line-by-line using the original interface, should have no effect. Substitutivity Substitutivity requires that you can substitute equivalent values (or expressions) for each other. It is essential that macro names should be able to be substituted for their contents (and vice versa) as this is the process by which they perform their automation. This achieves referential transparency in the macro interface. Generalisation Heuristic 8: If two buttons can be substituted for each other the least efficient button should be removed.

3.2.3. Composition of algebraic properties In the same way that the categorisation of the various types of command relationships that exist in the macro language of the system being modelled is useful, the composition of the categorisations is also important. Are macros that contain two idempotent statements in sequence idempotent themselves? There are many other similar questions that need careful and rigorous consideration formally and theoretically to flesh out more guidelines for generalisation.


231

4. Presentation and Interaction with 'buttons' 4.1. AMO — A Framework for Agent Interaction In a previous paper (Dix, Finlay & Hassell, 1992), the AMO framework for considering adaptive interface design has been proposed. It states that one problem of current approaches is that they adapt the medium of interaction (the interface between user and computer) whereas in normal life the agents in the interaction (the participants) are the ones doing the adaptation. To give an example: when two people (the agents) are communicating down a telephone connection (the medium), if any adaptation is needed (because the communication between them is not successful or inefficient) it is the people who adapt, not the telephone line. Any adaptation of the medium in this case is annoying to the people and should be handled with great care (for instance, if the delay between one person speaking and the other hearing changes — for example, if a terrestrial link is suddenly no longer available so a satellite link must be used — there will be a period of apprehension whilst the people adjust to the new time lag). This is deemed to be the cause of much of the insecurity of adaptive interfaces, as it is this type of adaptation of the medium that has previously been attempted, depriving the user of a deterministic ground — a consistent working environment that is predictable and stable — in which to work. It is therefore proposed that any adaptation in a system should be embodied adaptation — the adaptation should be performed by a visible agent in the interface with which the user can communicate. This agent's operation can then be customised itself by the user, in particular in terms of modifying the results of the adaptation process and in the presentation of the results of the process. This provides a practical interface to which of Edmonds's (1987) categories of control to use: adapt automatically; prompt the user when an adaptation is possible; or adapt only on the user's prompting — cf. Eager (Cypher, 1991) and Flexcel (Thomas, 1993).

4.2. Equal Opportunity In this case the adaptation performed by the adaptive agent is to build a model of the user's interaction in terms of the buttons extracted from the user's interaction trace. The results of this adaptation are presented for user interaction, creating a problem of initiative; which party — the user or the agent — should hold the initiative in controlling this presentation? Clearly the user needs to be able to explore and execute this button-model, but it is also clear that the agent must be able to create and modify this model to keep it up to date. The problems introduced when both agents need to have control and initiative over the button workspace are not new or unique. Much HCI research has dealt with the concept of initiative. We have adopted the principle of equal opportunity for handling this issue (Thimbleby, 1990, Chapter 4) where both user and computer can perform the same actions in the workspace. Restrictions to this rule must apply, however, where it is inconsistent with AMO — for instance, only the user should be able to specify the size of the workspace. 4.3. Naming of Buttons and Hiding of Complexity The name used to reference automatically extracted macros is important. For macros in languages such as the UNIX CLI the contents of the macro will be meaningful as an initial default title for the macro, albeit a tedious one. However, in the case of extracting macros from a drawing package or any other direct manipulation interface, the user will be unfamiliar with the underlying macro language. Hence, the default naming of the automatically extracted

232


macro becomes more difficult. Certainly the interface should make it possible for the user to give the macro a meaningful name with which to reference it to hide the complexity of the macro. The same should happen with the naming of the parameter instances of a button. But in the case of originally presenting the macro when it has just been extracted there remains a problem. One possible solution is for the macro to be demonstrated (as in demonstrational interfaces such as Metamouse (Maulsby, Witten & Kittlitz, 1989)) or simply to give the macro a dummy name until it is assigned a meaningful one by the user. Another is to embed knowledge within the system to give a 'first guess' name to any given macro. This is still an area of ongoing research.

4.4. Requirements of the 'button' Interface The concepts of embodied adaptivity and equal opportunity detailed above have suggested some requirements on an interface to the buttons created by automatic extraction and generalisation2. Presentation Requirement 1 The agent which presents the buttons must be presented as a separate interface to the application being considered and should not change the original interface to the application in any way. This fulfils the requirement from AMO that the adaptivity in the interface should be embodied. Presentation Requirement 2 The user should be able to perform any of the functions that the extraction and generalisation system provides, thus supporting the equal opportunity concept above. Thus facilities should be provided for allowing the buttons in the system to persist (for example: options to load, save, create, list, edit, delete, rename, and single-step through the buttons should be provided by the presentation agent). Presentation Requirement 3 The user should be able to select which of the three possibilities for presenting new buttons (Edmonds' categories: automatic presentation, presentation on demand, and prompted presentation) should be used, at any time. This allows the user to specify how much control the agent has over the button workspace (customisation of the agent). Presentation Requirement 4 Help in the form of a user-specified explanation of a button's usage (with example instances) should be packaged with the button to document it's function so that it may be given to other users and used without the need to understand its contents (cf. Xerox buttons (MacLean et al., 1990), Microsoft Wizards (Microsoft, 1992, Chapter 12, p.395-397)) thus hiding the complexity of the button. The presentation of buttons which include variables should provide the facility to assign a meaningful user-definable name to a button's parameter instantiation(s). Presentation Requirement 5 The user should be able to group the representations of the buttons (e.g. the buttons) according to any user-criterion (a user project, for example). The agent should also be able to do this automatically according to some inferred functional grouping thus ensuring equal opportunity. This helps to reduce search time for the user. Buttons which have been produced using different 2

Not all of the requirements outlined in this section have been implemented in the MAGI UNIX prototype presented in Section 5.


233

generalisation pivots should be presented separately in such a way as users should clearly see which pivot has been used to avoid confusion. Presentation Requirement 6 The user only has control of the workspace (for example: screen-estate) for the presentation of the buttons. This can be thought of as a screen-management mode in which the requirements of the AMO philosophy override the equal opportunity concept. Presentation Requirement 7 There should be an automatic caching mechanism which is brought into use when the workspace specified by the user is not sufficient to present all the buttons. This button usage monitor should also attempt to recognise changes in task or project to bring appropriate buttons to the fore. Both the user and agent should have the ability to alter the arrangement of the buttons in the adaptive agent's presentation workspace, using the principle of equal opportunity over the workspace.

5. Examples of the Application of Generalisations 5./. An Example of Macro Extraction and Generalisation Figure 1 gives a schematic of the macro extraction and generalisation processes applied to an example user log (in this case, a portion of a trace of interaction created whilst developing the prototype).

5.2. An Architecture and Prototype Implementation We are building a prototype system, based around a general architecture for building an adaptive agent — the Macro Automation and Generalisation Interface (MAGI) — to test the concepts of generalisation and presentation outlined above.

5.2.1. The MAGI Architecture The system architecture has been kept as modular as possible to allow modules to be easily re-used. This allows alternative implementations for the separate modules to be tested interchangeably (for example: the prefilter could produce one-line or multi-line habitual actions; the button interface could be replaced by a menu or icon interface). The modules have also been designed to facilitate the changing of the process (and thus the macro language) being logged. Functions that are dependent on the specific domain language being used are separated from more generally applicable functions in each module. For example, the prototype could be modified to process word-processor macros or macros in any other language, simply by providing a different logging shell and altering the syntactic elements of the prefilter and generaliser and providing tables of algebraic properties for the new language. The architecture is composed of four communicating modules: the loggable shell; the prefilter; the generaliser; and the presentation agent. The Logging Shell The shell is a modified form of the existing interface of the application being logged (for example, the UNIX wash shell is used in the prototype). The shell logs any user input in a file — encoded in a given macro language — as well as executing the input as normal. This encoding process, whilst simple for systems such as a UNIX CLI, has many decisions to make in logging, for example, GUI interaction with a drawing package (for example the level of


234 User Log:

Extracted Macros:

cd button3/filter

Generalised Macros:

make editc make generalise_interactive make editc make generalise_interactive cp generaliseînteractive . ./working cd ../working run_new_system cd ../tmttons_with_sockets make editc make button_sock cp button_sock ../working

make editc

Copy to working directory button for generalise_ interactive

make button_sock

cp button_sock ../working cd ../filter cp generalise_interactive . ./working

Figure 1: Example of macro extraction and generalisation.

granularity of information). The problems of logging such interfaces have been investigated before by other authors, see for instance (Kishi, 1992), so this work only intends to set some requirements on the output of this encoding process. At present the exact form of these requirements has not been specified; it is an area for future work. The shell must also be able to accept input from the presentation agent as if it were entered by the user via the original interface. The Prefilter The prefilter takes the output from the shell — the logfile—and finds any habitual actions that it contains. These habitual actions are found using an algorithm specified by Crow & Smith (1992) that has been implemented and modified by the authors. All habitual actions (macros) are packaged as new buttons and sent to the generaliser for inclusion in the buttonfile. The Presentation Agent This module handles the display and interaction with the buttons in the buttonfile — it is the additional adaptive interface created specifically according to the user's perceived needs. It reads the buttons from the buttonfile and presents them to the user who may execute them or edit them (the edited buttons being sent to the generaliser to update the buttonfile). The

235


user input to origiial interface

logging shell

macro execution (keystrokes)

log of user kiput (logtile)

buttonfile (button repository)

newbuttons

buttons

buttons

I

I

presentation agent

adaptive interlace

edt/delete/add buttons

generaliser

Figure 2: MAGI adaptive system architecture.

presentation metaphor for the interaction is flexible — any metaphor that has a reasonable mapping to macro execution could be used (for example: menu-based; voice-interaction; 'drag and drop'; gesture; natural language; or the 'wizard' paradigm of Microsoft Excel (Microsoft, 1992, Chapter 12, p.395-397)) — the only requirement is that it has the expressiveness to articulate all of the actions that can be applied to a button (execution, argument selection, deletion, listing and editing etc.) with reasonable efficiency and clarity. The Generaliser The generaliser accepts its input from the prefilter and the presentation agent. It acts as an arbitrator for the two processes (prefilter and presentation agent) which are in contention to update the buttonfile. Any alterations — specified by the user in the presentation agent — to buttons already in the buttonfile are handled. Any new buttons extracted by the prefilter or created by the user in the presentation agent are added to the buttonfile and the generaliser attempts to generalise them with the buttons already in the file (applying the generalisation heuristics above). Any buttons that are redundant are also removed from the file.

5.2.2. The MAGI UNIX Prototype The MAGI UNIX prototype provides an additional adaptive interface that evolves automatically as users go about their daily routine. It provides a macro-based interface to the UNIX command line and uses a simple button metaphor as the agent's interaction style. The prototype extracts macros progressively from the log of the user's interaction and creates executable buttons in the agent window. Any buttons that the prototype creates are given

236

Jonathan Hassell & Michael Harrison MAGI Prototype - Adaptive Button Dock Load Buttons IS&ve Buttons [Options Set iQuit System Compile generalise.interactive

Figure 3: One compile cycle button.

MAGI Prototype - Adaptive Button Dock Load Buttons Compile generalise_int eractive

ISave Buttons

Compile

buttorusock

[Options Set

IQuit System

IHQSl^Q^H

HHHHI^Î

generalise, interactive button_sock User Defined

Figure 4: Two compile cycle buttons and their generalised version.

initial default names; thereafter the interface allows them to be renamed, deleted and their macros displayed on request. If a generalised button is appropriate it is created from the two original buttons, with the arguments of the two buttons as options and a further option of a user-defined argument presented as a drop down menu when the button is selected. The size of the window in which the buttons are displayed is under the exclusive control of the user; the system will never alter its size, altering the size of the buttons instead so that they can all fit in the window at any time. The buttons can then be used to accelerate input, or can be completely ignored (the system will never interrupt users by asking for information unless the user explicitly requests this by setting the prompted interaction style on). The only inconvenience to the user's normal work is the buttons' visual appearance on the screen, occupying screen-estate. Figures 3 and 4 give screen shots of the example in Figure 1 from the current implementation.

5.2.3. Conclusions The MAGI UNIX prototype is currently implemented on a Sun 3/50 running under the sunview windowing system, employing a York-based presentation tool—Presenter — for the presentation agent, and a modified version of the wash shell for user input. Although it has


237

not yet been subjected to a full rigorous test, we have used it as a replacement for the normal shell for a short period and have found it unobtrusive and useful as a way of speeding our interaction. As it is difficult to assess the prototype's merit over such short trials — much of the value of the interface only becomes apparent after a number of buttons have been created — a longer test is planned to run alongside our extensions and investigations of the prototype. Furthermore we are using the most up to date version of the prototype as a shell for its further development.

6. Conclusions 6.1. Summary of the Main Points This paper has proposed that adaptive interfaces produced by means of automatic macro production techniques can be usefully extended by performing a generalisation process on the macros to find ones whose function can be applied to similar, yet different, circumstances. It has outlined various formal techniques for looking at this generalisation process and suggested some heuristics for controlling the process. Secondly we have addressed the issue of how to present the results of the macro production process in such a way as to be non-destructive to the original interface allowing predictability of interaction with that interface. The presentation and interaction with any elements of an environment that are adaptive have been constrained to occur within the paradigm of an intelligent agent separate from the original interface (but communicating with it). A framework and some requirements for interaction with this agent have been outlined. Finally we have given an architecture for such an agent — MAGI — and presented a prototype system which uses automatic macro production to produce graphical buttons for UNIX command line interaction. 6.2. Extensions and Further Work We are currently working on extending the exploration of the heuristics by modelling a different system — a drawing package — to examine a language that has a more hierarchical interaction structure than UNIX CLI interaction. We also propose to examine the mapping of the generated macros to a different presentation paradigm — initially a simple menu paradigm. These studies should help to explore the generality of this approach to adaptive interface design when applied to differing macro languages and interaction paradigms. Further work suggested is to examine modelling interaction with more than one application (for example, using AppleScript™ as a macro language to model and automate interaction between applications). This approach to providing adaptive interfaces needs to prove itself in more rigorous testing but, should it achieve this, this should suggest that the techniques outlined in this paper have practical merit. As adaptive systems are becoming more widely accepted (see, for instance, the introduction of Microsoft's new Intellisense™ technology) a strong knowledge base for helping create adaptive interfaces is essential. The authors hope that the methodologies, frameworks, and heuristics presented in this paper can enrich this base.

Acknowledgement Jonathan Hassell is funded by a CASE studentship from SERC, supported by the Digital Equipment Corporation

238


References Crow, D & Smith, B (1992), "DB-Habits: Comparing Minimal Knowledge and Knowledge-based Approaches to Pattern Recognition in the Domain of User-Computer Interactions", in Neural Networks and Pattern Recognition in Human Computer Interaction, R Beale & J E Finlay [eds.], Addison Wesley, pp.39-63. Cypher, A (1991), "EAGER: Programming Repetitive Tasks by Example", in Proceedings ofCHI'9J: Human Factors in Computing Systems (Reaching through Technology), S P Robertson, G M Olson & J S Olson [eds.], ACM Press, pp.33-4O. Dix, A, Finlay, J E & Hassell, J P (1992), "Environments for Co-operating Agents: Designing the Interface as Medium", in Proceedings of DTI CSCW-SIG Workshop on AI, Autonomous User Agents and CSCW, Springer-Verlag. Edmonds, E A (1987), "Editorial: Adaptation, Response and Knowledge", Knowledge-based Systems 1 (1). Foley, J D, Kim, W C & Gibbs, C A (1987), "Algorithms to Transform the Formal Specification of a User-Computer Interface", in Proceedings of INTERACT 87 — Second IFIP Conference on HumanComputer Interaction, H-J Bullinger& B Shackel [eds.], Elsevier Science (North-Holland), pp. 10011006. Greenberg, S [ed.] (1990), "The Re-use of Software Components — A Tool-based Approach", University of Calgary, PhD Thesis. Greenberg, S & Witten, I H (1993a), "Supporting Command Re-use: Empirical Foundations and Principles", International Journal of Man-Machine Studies 39, pp.353-390. Greenberg, S & Witten, I H (1993b), "Supporting Command Re-use: Mechanisms for Re-use", International Journal of Man-Machine Studies 39, pp.391-426. Kishi, N (1992), "SimUI: Graphical User Interface Evaluation Using Playback", in Proceedings of the 16th Annual International Computer Software & Applications Conference, IEEE Computer Society Press, pp.121-127. MacLean, A, Carter, K, Lovstrand, L & Moran, T (1990), "User Tailorable Systems: Pressing the Issues with Buttons", in Proceedings ofCHI'90: Human Factors in Computing Systems, J C Chew & J Whiteside [eds.], ACM Press, pp.175-182. Maulsby, D L, Witten, IH & Kittlitz, K A (1989), "Metamouse: Specifying Graphical Procedures by Example", in Computer Graphics #23 #3, pp.127-136, Proceedings of ACM SIGGRAPH'89. Microsoft (1992), "Excel User's Guide 2: Worksheet Analysis, Exchanging Data, Customizing, Automating". Myers, B A (1991), "Demonstrational Interfaces: A Step Beyond Direct Manipulation", in People and Computers VI: Usability Now!, D Diaper & N Hammond [eds.], Cambridge University Press, pp. 11 -30, Proceedings of HCI'91. Thimbleby, H W (1990), User Interface Design, Addison Wesley. Thomas, C G (1993), "Design, Implementation and Evaluation of an Adaptive User Interface", Knowledge-based Systems 6 (4), pp.230-238.

Agent-Based Interaction Russell Beale & Andrew Wood School of Computer Science, University of Birmingham, Edgbaston, Birmingham B15 2TT, UK. Tel: +44 (0)21 414 3729 Fax: +44 (0)21 414 4281 EMail: {R.Beale, A.M.Wood}@cs.bham.ac.uk Agents are becoming widespread in a variety of computer systems and domains, but often appear to have little in common with each other. In this paper we look at different agent systems and identify what a generic agent should be composed of. We also identify the characteristics of a task that make it worthy of an agentbased approach. We then discuss the implications for the interaction of using agents, that is, the notion of a balanced interface, and briefly look at how an agent-based approach assists in two very different application domains. Keywords: agents, intelligent interfaces, groupwork, computer-supported cooperative work (CSCW).

1. Introduction The term agent has become increasingly widespread over the past few years. Unfortunately, it has no consistent definition and means many different things to different people. This paper considers the multifarious uses that these disparate agents are put to and tries to identify what, if anything, they have in common. With this commonality in mind, we identify the general properties of an agent and consider what makes a task 'agent worthy'. Agent-based interaction has consequences, leading to the notion of a balanced interface. This leads on to a description of the design of a generic agent and a look at a couple of applications that are benefiting from being treated in an agent-based manner. 2. Classifying Agents We can identify a number of categories that agents can be classified under, based on the functionality offered by the different types of agent. Note that this classification is not intended to be exhaustive by any means; what we desire is a way of structuring the terms so

240

Russell Beak & Andrew Wood

that they offer us a view of what it actually means to be an agent, and can identify the types of task that are suited to an agent approach. 2.1. User Agents Agents can be adaptive, learning, self-customising pieces of software. Examples include mail and news user agents; some of these 'intelligently' filter and file mail (Metral, 1993) whilst the Aide de Camp agent allows messages to trigger certain actions. It also defines agents that are not simply reactive but also proactive. User modelling approaches that monitor user behaviour and adapt accordingly can also be considered under this category. The MADMAN — Multi- Agent Diary MANager—(Eisenger & Elshiewy, 1992) is an example of a scheduling agent that allows automatic scheduling of appointments and resources. These agents, and the user ones, all involve tasks that the user could do for themselves but that are more conveniently done by the computer. Agents can be used to allow the customized presentation of information. These agents preprocess the data and display it in a way that can be unique for each individual user. Care has to be taken in CSCW that this doesn't lead to loss of shared context. 2.2. Agent

Guides

Agents that act as tutors or guides, supplementing user's knowledge with their own. The are able to provide the user with access to greater skills or expertise. Microsoft and WordPerfect have what they term Wizards and Coaches (WordPerfect Corporation, 1993) respectively that act as tutors when using the system. They are more advanced than a simple help system in that they take an active role in leading the user through the task, and are able to suggest better ways of doing certain tasks. Guides, from Apple Computers (Oren et al., 1990), lead the user around a hypermedia database, and have personas of their own; they are anthropomorphisms of the agent idea. 2.3. Autonomous

Agents

A number of agents exist that are able to work on behalf of their user without the need for any interaction or input from the user. They act without your presence, tirelessly performing tasks. Some of these autonomous agents scour the world's databases (Sculley, 1988, p.540-565), usually via gopher, gopher+, the World-Wide Web, or other generic access methods, returning interesting and relevant information. The long delays caused by slow networks, and the vast quantity of information to be searched make these tasks that could and should be done in the background. They are not ones that most users could really do for themselves without a lot of effort. This means that these sort of agents require skills and expertise as well as being able to navigate around the world autonomously. 2.4. Symbiotic and Cooperative

Agents

Agents that are 'in there with you' (Clarke & Smyth, 1993); these assist the current task by providing alternative views and additional relevant information. Examples include Eager (Cypher, 1991) which observes user interaction and extrapolates to provide useful defaults and macros, and the CAD Helper, described by Tognazzini (1992),

Agent-Based Interaction

241

which spots useful relationships in engineering drawings, such as start, midpoint, end and tangents to lines. Clarke & Smyth's (1993) room arrangement and Fischer et al.'s (1991) Critics utilise a large knowledge base of domain rules (room/kitchen layout, and design) to help users by criticising during the design process and suggesting alternative variations that they think are better and bear user consideration. These agents are similar to guides, but provide complementary information as well. 2.5. Anthropomorphic

Agents

These agents imitate humans; this grouping reflects a representation issue rather than one of capabilities. The Maas-Neotek Robots log into a multi-user game, and have the same capabilities as the human players (Foner, 1993). Their main role is to wander around the game, either partially directed by their user or totally randomly, mapping it out and therefore able to offer helpful advice at a later stage. Shneiderman is a vociferous opponent of anthropomorphic agents, and argues that creating more powerful direct manipulation tools is the way forward (Don, 1992). Alan Kay's view is that the next step from manipulation is management (Kay, 1990): however powerful a tool we have to do something, we would still rather someone else did it. Agents allow the user to assume the role of manager, partially specifying what has to be done and letting the agents get on with it. 2.6. Multi-agent

Systems

Agents are not necessarily just software; agent definitions very often include humans. This means that, for example, agent systems are of prime concern to CSCW, in which multiple human agents cooperate and communicate to solve mutual goals, and software that is developed to support this is often agent-like. Distributed artificial intelligence is also a major contributor to agent literature; how agents communicate, share information, and achieve intelligent behaviour is discussed. 2.7. Agent-Oriented

Programming

A programming approach which extends the notion of object-oriented programming to its (logical?) conclusion. Agents have mental states, beliefs (like object data), capabilities (like object methods), and commitments (which are new). A LISP-like language that supports AOP is available from Stanford.

3. Agents and the Agent-Based Approach Having collected together the tasks and domains in which we find agents, we can produce a list of attributes that are common to some or all of these task areas. Such tasks are suitable for an 'agent-based approach'. However, before we can do this we need to refine our understanding of what we mean by agent without concentrating on particular domain over another. The definition we will adopt is as follows: an agent can be defined as a relatively simple heterogeneous autonomous communicating software component; multiple agents are created which act together to undertake a task for the user. The advantages of using agents are many (Wood, 1993), but one of the strongest reasons for their use is that they collectively exhibit emergent behaviour, where the behaviour of the agent population as a whole is greater than the sum of its parts. Such emergent behaviour

242

Russell Beale & Andrew Wood

is common in the field of neural computation, in which back-propagation nets rely on the emergent properties of their connection weight matrices for their complexity and power. Simple processing units that do no more than communicate with neighbours, weight inputs, sum and threshold are able to achieve tasks as diverse as text-to-speech, low-level vision, character recognition and ECG analysis (Beale & Jackson, 1990). The attributes of an agent-worthy task can then be identified as: adapting and researching, demonstrating and guiding, and autonomy and asynchrony. Adapting — the task requires a degree of adaptability, so that the software has to refine its skills to learn new or better ways of accomplishing things. This includes learning to avoid failure and accommodating user preferences. Researching — the task is poorly defined, so the software has to explore a number of different options based on its current expertise. Demonstrating — the task involves passing on skills that the software uses. This encompasses the software teaching users how to do things, and also provides explanations of what the agent is up to. It also allows one agent to teach another. Guiding — the task requires some degree of co-operation between the agent and the user. The software could offer constructive criticism to the user as they work, or could assist them in working though a particular set of options. Autonomy — the task itself requires regular or constant attention, but not necessarily any user input, thus making delegation useful. An example of this is the monitoring of the system state with certain events triggering specific actions. Asynchrony — the task has a significant delay between its initiation and completion. These delays can be due to slow processing or to network delays, or to temporary unavailability of the required service. Not all these conditions are required for a task to benefit from an agent-based approach, though many of the domains identified contain most of them.

4. The Balanced Interface The notion of having these software agents that can be invoked to do the task for you is an appealing one, but one that has implications for the interaction. As far as any other software system is concerned, agents are treated just as users and have the same rights and permissions as their owners. Agents are able to go and do the task for you, and so in some respects are subservient. However, they are also able to guide you and work in close co-operation; in this mode they are more like equals. Since they access systems as you, they are your representatives and are viewed on the same footing as a human user would be. Agents can also demonstrate new things to their users, and so are teachers; now, they are superior. Agents and their users are more or less equivalent; sometimes one has the leading hand, sometimes the other. Agents view the world from our perspective, and this has to be reflected in the interface, where the agent has to be promoted to a level of equality with the user. This is an interesting move forwards for the desktop metaphor, as it suggests that willing agents, both human and software, can observe, criticise, praise, chide, and learn from each other; a symbiotic relationship between user and software.


243

5. Generic Agent Design The first thing that a competent software agent needs are the sensors to allow it to observe the environment in which it exists. This information can be identical to that which the user receives, but can be coded at different levels. To assist the agent to behave intelligently in interpreting the data, contextual information ought to be provided from whatever tool is currently being used. For example, providing data coded at a low machine level may be 'left button down at 224, 349' whereas adding context and coding a higher level rewrites this as 'pressed OK button'. Agents do not only exist, they interact with the environment. They therefore require effectors, which can range from low-level constructs similar to the keyboard and mouse that the user has, or can be more high level allowing it to specify 'quit application' rather than 'move to x, y, typectrl-xctrl-c'. Mindless interaction is not desired; we have to equip ideal software agents with a brain. This will take many forms, but is likely to allow adaptation, planning and memory facilities; for example, giving agents navigation and mapmaking facilities to allow them to move about the information space. In general, agents need to be able to form models of their environment. They also have to create models of other agents so that they can effectively interact with each other to exploit individual talents. Finally, agents need to know about communication, so that they can present information to their users in an intelligible and accessible manner, and exchange information with other agents in an efficient way. The interesting thing about such agents is that each component is implementable under current technology, so that fully-operational agent systems are not merely a desire but are realisable. Allowing the agents to adapt, refine and improve, automatically or under user control, we can create a holistic system in which the whole is significantly more than the sum of its parts. 6. Two Case Studies Using these ideas of agent-worthy tasks and agent interaction, we can apply them to different domains. Two examples are given below; data mining and the implementation of a desktop secretary.

6.1. Agent-based Data Mining Data mining, or knowledge discovery, is the term given to the exploration of databases in an attempt to extract relevant and interesting interrelationships that exist between variables or between causes and effects. There are a number of problems that make data mining a nontrivial task, however. Some database systems do not allow us easy access to the information they record, and have complex and clumsy interfaces that insist we produce unambiguous requests for information; yet it is often the case that we do not know what sort of request to make for we do not know what information we want to be looking at in the first place! The sheer quantity of information provides us with another problem; in a space with potentially thousands of dimensions, some of which are huge in extent, the question of where to start is as meaningless as where to go to next. Browsing in a database with many, many different independent variables recorded is in itself difficult, as we cannot directly visualise more than three dimensions at any one time. Storing this amount of data does not represent

244

Russell Beale & Andrew Wood

a technological problem, and indeed the more that is stored, the more likely it is that a relationship exists within the database. It is just difficult to work out where it may be found. Another problem is that the networkable capabilities of many current systems mean that the datasets to be examined may exist on many different machines, in a variety of forms. This means that a variety of access protocols have to be used in order to obtain the data in the first place, and then it has to be converted into a suitable format for integration with the other datasets currently under investigation. It is also the case that different data types offer different ways of looking at them. Different datasets do have characteristic visualisation techniques associated with them; sparse datasets use scatterplots whilst more complete ones can be rendered in 3-d with isosurfaces and shading (Beale & Pryke, 1994). This serves to show that even if we can access the subsets of data we want, browsing through it is not necessarily a simple matter. The usual approach to searching for interesting relationships in data sets is for the user to interactively browse through the system, choosing representations of the data that they think may show up interesting features, and observing the results on screen. The user manipulates the data, manoeuvring herself through the data space in search of particular regions that are interesting. This hyperspatial navigation is difficult, time-consuming, and often unprofitable. The problem of data mining is ideally suited to an agent-based approach as it has all the attributes of an agent-worthy task. Adapting to learn what is interesting and what is not is an integral part of an agent data miner, whilst discovering those successful techniques, and searching the data space are all part of the research. Providing the user with a view on the discovered data is part of demonstrating, whilst assisting in hypertextual navigation is guiding. Setting the agents off into the database, and allowing them to integrate datasets from different databases across the world require autonomy and asynchrony too. Current work is developing agent-based data mining systems based on these concepts.

6.2. Desktop Secretary Having a desktop agent that can manage your time, scheduling and rearranging meetings with multiple people is an example of a locally distributed application domain ideally suited to agent-based interaction. This task is inherently hard due to the difficulties of trying to satisfy multiple sets of time constraints that are only partially known, and by the asynchronous nature of the information transfer between geographically separated users. Each user has an agent that is able to communicate with other agents. Requests for meetings are passed from the user to their agents, who then autonomously communicate with each other to decide upon mutually convenient times. Agents have access to the user's calendar and to a database of preferences (for example, I don't like meetings before 11 am.) and can resolve conflicts to come up with suggested times. These are then confirmed by the users. Cancellations and reschedulings are dealt with by the agents in a similar manner, notifying other affected parties of changed arrangements. This has been found to work effectively over a local network (Rooney, 1994; Wood, 1993) and is currently undergoing extensive modifications to allow it to deal with more complex scheduling and conflict resolution. Referring to the taxonomy, adaptation occurs to accommodate user preferences, whilst the conditions that the task must satisfy are poorly defined, necessitating a search to find a suitable solution. Neither demonstrating or guiding are relevant, but autonomy and asynchrony are


245

central characteristics inherent in the interaction and are well served by the agent-based approach.

7. Conclusion Agents are a powerful metaphor in the search for more effective interfaces. They require a balanced environment in which to work, and whilst there are many different incarnations they share a common set of properties. Likewise, whilst they can be applied in many different domains, these all have particular characteristics that make them suitable for an agent-based approach.

References Beale, R & Jackson, T (1990), Neural Computing: An Introduction, Adam Hilger. Beale, R & Pryke, A (1994), "Visualisation for Data Mining", School of Computer Science, University of Birmingham, Internal Report. Clarice & Smyth (1993), "A Cooperative Computer Based on the Principles of Human Cooperation", International Journal of Man-Machine Studies 38, pp.3-22. Cypher, A (1991), "EAGER: Programming Repetitive Tasks by Example", in Proceedings ofCM'91: Human Factors in Computing Systems (Reaching through Technology), S P Robertson, G M Olson & J S Olson [eds.], ACM Press, pp.33-40. Don, A (1992), "Anthropomorphism: From ELIZA to Terminator 2", in Proceedings ofCHl'92: Human Factors in Computing Systems, P Bauersfeld, J Bennett & G Lynch [eds.], ACM Press. Eisenger, N & Elshiewy, N (1992), "MADMAN — Multi-Agent Diary Manager", ESRC-92-7i Internal report. Fischer, G, Lemke, A C, Mastaglio, T & Morch, A (1991), "The Role of Critiquing in Cooperative Problem Solving", ACM Transactions on Office Information Systems 9(3), pp. 123-151. Foner, L N (1993), What's an Agent, Anyway? —A Sociological Case Study, MIT Press. Kay, A (1990), "User Interface: A Personal View", in The Art of Human-Computer Interface Design, B Laurel [ed.], Addison Wesley, pp.191-207. Metral, M (1993), "Design of a Generic Learning Interface Agent", MIT, BSc Dissertation. Oren, T, Salomon, G, Kreitman, K & Don, A (1990), "Guides: Characterizing the Interface", in The Art of Human-Computer Interface Design, B Laurel [ed.], Addison Wesley, pp.367-381. Rooney, A (1994), "Diary Agents", School of Computer Science, University of Birmingham, BSc Dissertation. Sculley, J (1988), Odyssey: Pepsi to Apple, Fontana/Collins. Tognazzini, B (1992), TOG on Interface, Addison Wesley, Chapter 20. Wood, A (1993), "Desktop Agents", School of Computer Science, University of Birmingham, BSc Dissertation. WordPerfect Corporation (1993), WordPerfect.

Modelling Humans, Computers and their Interaction

Multi-Perspective Modelling of Interface Design Issues: Undo in a Collaborative Editor Richard M Young* & Gregory D Abowd* tMRC Applied Psychology Unit, 15 Chaucer Road, Cambridge CB2 2EF, UK. Tel: +44 (0)223 355294 EMail: [email protected] ^College of Computing, Georgia Institute of Technology, 801 Atlantic Drive, Atlanta, GA 30332-0280, USA. EMail: [email protected] Successful interface design respects constraints stemming from a number of diverse domains analysed by different disciplines. Modelling techniques exist within the individual disciplines, but there is a need for ways to weave together different techniques to provide an integrated analysis of interface design issues from multiple perspectives. We illustrate the relations and interplay between six different modelling techniques — two for system modelling, two for user modelling, one for interaction modelling, and one for design modelling — applied to a shared design scenario concerning the provision of an Undo facility for a collaborative editor. The resulting multi-perspective analysis provides a depth of understanding and a breadth of scope beyond what can be achieved by any one technique alone. Keywords: user modelling, system modelling, design rationale, interaction analysis, multi-disciplinary analysis, scenario analysis, undo, multi-user, editing.

1. Introduction Successful interface design requires the satisfaction of a diverse set of constraints stemming from different domains. One of the factors making interface design so challenging is the number and diversity of those domains, and the different disciplines that study each. Relevant domains include that of the computer, within which are the disciplines of computer science and software engineering; of the user, studied by disciplines such as psychology; of work, the topic of sociology and anthropology and other disciplines; and of design itself.

250

Richard M Young & Gregory D Abowd

Modelling techniques that can contribute to interface design exist in each of these domains. However, any one of these approaches tells only part of the story, and covers only some of the issues. There is a pressing need to combine modelling techniques derived from different disciplines and reflecting different perspectives in order to provide analyses with the scope and the depth adequate for guiding design. In this paper we present one route to such an integration, illustrated by the application of six different modelling techniques to a common design task.

/./. The Design Task A team of designers is developing a group editor intended for synchronous document editing. Early versions of the editor allow for only one insertion point into the shared document at a time, but future releases of the system will allow for true concurrent editing with multiple insertion points, one for each user. The designers feel it is important to provide some sort of Undo support facility so that users can easily recover from mistakes by reversing the effect of erroneous commands. They are unsure, however, how that Undo support facility should work. Some feel it should be global, defined relative to the document. In this case, one user would be able to undo the work of another user. Others feel that global Undo would be confusing; instead, a local strategy for Undo is suggested whereby users can reverse only their own commands. The designers can appeal only to their intuition in reasoning about Undo in such a new context, and in this case there is no clear consensus.

1.2. How To Help? What kind of analytic leverage can we bring to this debate? In examining the analyses of six widely different techniques, we are not looking to see which of them is best. Rather, we are concerned in this paper with the interplay between various techniques when applied to a common problem. We want to see both how different modelling perspectives can influence each other and also how collectively they inform design. We argue that the impact from a collection of techniques is greater than the sum of their individual contributions. So it is our purpose to report the results of an exercise in multi-perspective analysis for the problem of Undo in a collaborative editor. This question of local versus global Undo might on first encounter seem a rather specialised and narrow topic. However, Undo is special because it raises subtle issues of usability and has widespread implications for system implementation. It differs in this respect from other editing commands. To add an 'ordinary' command to an editor, such as a delete-word facility, raises no great conceptual or implementational concerns. But Undo is different. As we shall see, it is subject to strong constraints from both the system side and the user side, and from the interaction between them. In the next section, we briefly outline the modelling techniques which were applied to the Undo design scenario. The remainder of the paper will then discuss the multiple analyses of Undo. The original modelling activity was carried out in isolation by experienced practitioners of each technique. This was followed by a period of integration, during which various individual analyses were revised based on the results of the original modelling activity. Part of the integrated story has previously been presented in (Abowd & Dix, 1992), which concentrates on the interplay between just two of the techniques. This paper will provide an overview of all six modelling techniques and how they influence the overall design guidance for multi-user Undo.

Multi-Perspective Modelling of Interface Design Issues

251

2. Defining The Perspectives We first outline the various modelling approaches, providing a brief description and references to sources for further information. An obvious question to raise concerns the choice of techniques: why these, and why not others? These six approaches are discussed here because they were the ones represented within the original AMODEUS project. But that is hardly the end of the matter, since the project did not include a random set of approaches. In order to take part fruitfully in this kind of multi-disciplinary analysis, a modelling technique has to meet a number of requirements. It needs to be both sufficiently explicit and sufficiently compatible with the other approaches in its underlying commitments, that is, it can accept input from other approaches and produce output in a form usable by them. Not any old modelling technique will do. There is also a requirement that the set of approaches taken together provide some kind of coverage of the areas of HCI for which modelling techniques exist. We would argue that our six approaches do provide a good approximation to that coverage. The main omission we know of lies in the area of task analysis and work organisation, but there we are unaware of any modelling techniques that meet the above criteria. In summary, the set of modelling techniques for inclusion in this exercise was tightly constrained. 2.1. Formal System

Modelling

In the tradition of formal specification as a tool for software engineering, formal system modelling uses abstract mathematical descriptions of the system in order to expose properties that contribute to its perceived usability. The classic example of this technique is the algebraic black-box approach of the PEE model, introduced by Dix & Runciman (1985) and extensively documented by Dix (1991). The two key aspects of this technique are a relatively abstract mathematical description of the problem space followed by the formulation of precise principles of usability, such as predictability or robustness, by which one can discriminate candidates in that problem space. It is important to understand that the principles, while described as constraints on the system description, are actually motivated by how a user would perceive the system. 2.2. PAC Software

Architecture

PAC is a reference software architecture for structuring implementations of interactive systems based on a collection of hierarchically related agents. Coutaz (1987) first proposed the PAC model and Bass & Coutaz (1991) show how it can be used to develop high-level architectural descriptions of user interfaces. Though PAC also presents a system perspective, it is different from the formal modelling because it is more concerned with the developer of the interactive system and not so much with how the user perceives the system. The emphasis of a PAC analysis will be on how the system supports software engineering goals, such as modifiability and concurrency. Separation of presentation and application supports those goals, but PAC also maintains a controlled relationship between presentation and application in order to support the kind of rapid semantic feedback that is necessary in modern graphical user interfaces. 2.3. Programmable

User Model

In the Programmable User Model (PUM) technique, a designer is invited to program a simulation of a potential user performing a task with a proposed interface (Young, Green & Simon, 1989). This technique presents the user's perspective in terms of a programmable cognitive architecture embodying psychological constraints. In order to program the user model, the designer has to conduct an analysis of the knowledge the user needs in order to

252


perform the task. Difficulties with this knowledge analysis, or errors in the model's eventual behaviour, warn the designer of usability problems in the proposed design. 2.4. Interaction

Framework

Interacting Cognitive Subsystems (ICS) is a technique reflecting a resource-based view of the multiple subsystems involved in cognition (Barnard, 1987). ICS provides a model of perception, cognition and action, but unlike other cognitive architectures, it is not intended to produce a description of the user in terms of sequences of actions. Rather, ICS provides a more holistic view of the user as an information processing system. The emphasis is on determining how easy particular sequences of actions become as they are made more automatic and proceduralized within the user. Knowledge about how to apply the model and draw predictions from it is embodied in an expert system. To analyze a proposed interface, the designer consults the expert system, which asks questions about the user-relevant aspects of the interface and then makes predictions about user behaviour. 2.5. Design Space

Analysis

Design space analysis is a design rationale technique for structuring and recording the usability considerations for design alternatives. MacLean et al. (1991) define a notation, QOC, which represents local parts of a design space in terms of three kinds of components: the Questions being addressed, which serve to structure the design space; the different Options being considered, which represent the possible design choices; and the various Criteria against which the options are assessed. This form of design rationale provides a medium for communicating and explaining the design to others, such as later maintainers of a system or even users. 2.6. Interaction

Framework

So far we have described techniques which adopt exclusively a single perspective on HCI, whether of the system, of the user, or of design. The interaction framework (Barnard & Harrison, 1989; Blandford, Harrison & Barnard, 1993) represents a different perspective which focusses on the interaction itself. One common feature of other techniques is the use of state machine descriptions of either the user or the system. Transitions between states in these models represent significant changes in either the human or computer during the evolution of the interaction. The interaction framework suggests that we can model states of the interaction which are independent of human or computer detail and can be informed by the work domain of an interactive system. We can pinpoint an area of concern for any behavioural scenario by understanding the trace of interaction state transitions in that situation. The interaction space can then be mapped onto the user and system spaces to restrict their scope of analysis.

3. Multi-Perspective Analysis 3.1. Formal System Modelling A convenient place to start a multi-disciplinary analysis of our Undo design issue is with the formal system modelling, which provides a general, abstract characterisation of Undo together with a set of well-defined concepts and vocabulary for thinking and talking about its design. The formal modelling analysis begins with the summary by Thimbleby (1990, Chapter 12) of the work of earlier authors concerning single-user Undo. A critical insight in modelling Undo is to separate commands which provide the Undo facility from the other


253

commands in the system. The formal models distinguish between the history of all commands which the user has issued, including the Undo commands, and the effective script, which is the set of normal commands that have not been Undone and which therefore determine the current state of the document. There are various different proposals for how Undo might work. One of the schemes is known as truncate, in which the effect of Undo is simply to reverse the last action. A slightly more elaborate version is called truncate*, in which successive invocations of Undo gradually 'roll back' the sequence of user actions, by reversing progressively earlier actions. Another variant, familiar to users of the Apple Macintosh, is flip-truncate, in which successive Undos alternate the document between two states, with and without the most recent user action. There are also more elaborate schemes, such undo/skip/redo, which involve the selective undoing and redoing of individual user actions, and in their most general form are capable of an arbitrary editing of the effective script. In this paper we will stick with the various members of the truncate family, especially truncate*. The more complicated schemes seem not to raise any new issues for collaborative work, and there are anyway severe doubts about whether users can manage them effectively. At least in simple cases, the ideas developed for analysing single-user Undo can without difficulty be extended to multi-user Undo. They can be used, for example, to illustrate the difference between the local Undo and the global Undo defined in our scenario. Suppose that a user U has just inserted the word 'hello' and then inserted the word 'world' into a document, and that another user O has selected the 'hello' and deleted it, so that the document contains just 'world'. (Throughout this paper we will use the symbol 'U' to mean some user whose viewpoint we are adopting in describing a segment of behaviour — whose shoulder we are peering over, as it were — and ' O ' to refer to some other unspecified user or users.) Suppose that U now issues an Undo. In the case of local Undo, the effect will be to reverse U's own last action, which was to insert 'world', so the document is now blank. However, in the case of global Undo, the effect is to reverse the last action no matter who did it. The last action is in this case O's deletion of 'hello', so the Undo reverses that deletion and the state of the document is now 'hello world', with the first word selected. The system modellers adopted a concise algebraic notation to clearly define the Undo options and reason about them, some of which will be used later. This analysis has not yet provided any advice for selecting between local and global Undo. As argued in (Abowd & Dix, 1992), the only judgment that the formal model can make is that the global Undo option will always be possible, whereas the local Undo option can result in undefinable behaviour. Note also that the story just told assumes it is possible to arrange the actions carried out by different users in a definite order. That is also a point to which we shall return.

3.2. Programmable User Model The PUM user modelling approach, like the formal system modelling, begins with the singleuser case. This approach makes use of the concepts and terminology developed by the system modellers, but extends it by performing a knowledge analysis which shifts the focus to the users, and asks what they have to know in order to be able to employ the Undo facility correctly and with predictable results. The knowledge analysis reveals that users have to know four things about the Undo:


254 Single Insertion Point

Multiple Insertion Point

Local

Global

Local

Global

Kl

V

V

V

Problematic (4)

K2

V

Problematic (3)

V

Problematic (3)

K3

V

V

V

Disastrous (5)

K4

OK, provided t h a t . . . (2)

Problematic (1)

Disastrous (4)

Table 1: Summary of knowledge analysis.

Kl: What stream of activity is relevant. In the single-user case this is unproblematic, since it is the user's own stream of input. K2: How that stream is articulated into units. For example, in a text editor, does a unit consist of just a single word, or is it everything since the last mouse click? K3: Which unit is affected. For the truncate schemes, it is the user's last action. K4: The effect of Undo. For the truncate schemes, the affected action is reversed. The importance of knowing these things is easily illustrated. Suppose that the user is nearing the end of typing in a long paragraph, and notices a spelling error in the previous word. Does it make sense to use Undo as part of the process of correcting that word? The answer depends critically upon K2, upon knowing whether Undo will remove just that last word or else the whole paragraph. As with the system modelling, this knowledge analysis can be extended easily from the singleuser to the multi-user case. When we do that, we can ask, for each of the candidate designs described in our scenario, how well it lends itself to having the user know Kl through K4. The results are summarised in Table 1, in which a tick mark means that the knowledge requirement can be satisfied unproblematically for that particular design, while other entries range from a "Yes, b u t . . . " through Problematic to Disastrous. Consider for example what happens (1) in the case of a Multiple Insertion Point (MIP) design with local Undo. There is no difficulty with Kl to K3, but K4, predicting the effect of an Undo, can be problematic. Suppose that U has just typed in a new paragraph, when O notices that U has inadvertently omitted a word and helpfully adds it for her. U then changes her mind and invokes Undo to remove the whole paragraph. What is the result? Does O's single word remain, or does it vanish too? This difficulty can be avoided in the case (2) of a Single Insertion Point (SIP) design, provided that the scope of Undo is restricted to the current phase, i.e. the period during which the user has had control of the insertion point. Otherwise the same problems arise. A different problem exists (3) for global Undo. If U is to be able to Undo some of O's work with predictable results, then she must know how O's work is divided into Undoable


255

units. But U may not be able to tell that just by watching the results of O's work on her own (i.e. f/'s) screen. Suppose for example that units of activity are defined by successive mouse clicks. Then can U see the difference between: i. O making a mouse click and then typing five words; and ii. O five times making a mouse click and typing a single word? Probably not. The solution is presumably to enhance the display in some way so as to indicate the scope of the immediately Undoable unit. (Notice how a user-centered knowledge analysis has here uncovered a new issue for design.) In the case of MIP, global Undo has difficulty satisfying any of the four knowledge requirements. Suppose (4) that both U and O are simultaneously active and have just spent time typing in (separate) paragraphs. How are these to be regarded as forming a single stream of activity (Kl), and what would be the effect of an Undo (K4)? Question K3, knowing which unit is affected by an Undo, is afflicted with yet another hard problem (5) — race conditions. Suppose U decides to issue an Undo in order to reverse some action. In the interval between the time when she makes that decision and the moment when she actually presses the Undo key, O — working on a different part of the document and perhaps physically located many miles, or even thousands of miles, away — might issue some further action, which then under global Undo becomes the action Undone, instead of the one U intended. That would be puzzling and frustrating for both parties. 3.3. Interacting Cognitive Subsystems and the Interaction

Framework

ICS and the Interaction Framework work together in this instance to provide an informal but insightful analysis focussing on the circumstances under which and purposes for which users employ Undo. They propose two arguments. In a single-user application, the users become practised at applying Undo to reverse their own actions. Such a skill is automatised, so with a SIP design users are unlikely to begin a phase by invoking Undo. This argues for a local Undo, with its scope restricted to the current phase. Notice that this is the same conclusion reached by the knowledge analysis, although on different grounds. Second, with a SIP design, U's cognitive processing will be divided between following the changes that O is making to the document and thinking about what U herself plans to do. This will be frustrating for U, because she will have to keep several ideas in mind without immediately being able to do anything about them. Furthermore, if O has control of the insertion point, it will be hard for him to predict when he will get it back once he relinquishes it. He will therefore be reluctant to give it up until he has done everything he wants to, so he will probably stop and think before yielding control. This will make it even more frustrating for U, since she will see O having control over the insertion point but apparently not making much use of it. These two approaches suggest then — in line with some other analyses, but looked at from a different viewpoint — that with a SIP design, Undo should be local. This provides us more confidence in the assertion that local Undo is favored over global Undo. However, these two methods also provide us the input that any SIP design is likely to be frustrating for its users. 3.4. PAC Software

Architecture

We have just seen how a knowledge analysis derived from the PUM approach can reveal problems with a particular design and raise new design questions and suggestions. Several of

256


those issues are also addressed by the PAC software engineering approach. For a collaborative editor, the PAC model takes the form of a network of communicating Workstation agents, each of which has subagents dealing with the Document and the Participants. One of the dominant issues for PAC is a question of implementation (not interface design): whether the Workstations should access a single central representation of the document, or whether the document should have a physically distributed representation. However, several of these apparently Software Engineering issues turn out to affect the usability issues we have been examining. With MIP designs, as with distributed databases, one of the concerns is for the semantic consistency of the document. With a geographically dispersed network, transmission delays inevitably mean that U can try to take an editing action which is invalidated by something that O has done but which U has not seen. In other words, there can be race conditions in the network. From an implementation point of view the problem can be solved, perhaps by giving priority to actions received at a central representation on a first-come basis. But from a usability point of view, such a solution is arbitrary and would make the system's behaviour unpredictable by the user. The difficulty is analogous to the race conditions discussed in the PUM knowledge analysis. Originally that problem was described as arising from the delay between when U forms an intention to invoke Undo and when she presses the key. But the delay does not end there. Even after the key has been pressed, the signal takes measurable time to wend its way through the network, possibly across continents and oceans, to the location where the editing action takes effect. During that time, an action of O's can intervene and become the action reversed by f/'s Undo. This analysis of implementation considerations reinforces the conclusion derived from the knowledge analysis — that for MIP designs global Undo is highly problematic. Some of the major arguments in deciding between a central or a distributed representation for the document concern the system response times for providing feedback to the user. In a geographically dispersed network it is usually impossible to provide guarantees on the maximum transmission delay, so the need to provide immediate feedback for U's own actions argues strongly for keeping a representation of the document at each workstation. In fact it suggests an additional principle to further constrain the software design — the effect of an editing action should be locally computable. This question of feedback raises new design considerations. Consider what we will term the crossfeed, that is, the information given to U as a result of O's actions. Whereas the feedback given to U in response to her own actions needs to be done on a character-by-character basis, frequent updates to C/'s screen due to crossfeed from O will tend to make the display unstable and hard for U to use. One can imagine that less frequent updates, with a larger grain size, will be more acceptable. Hence a set of new design issues gets raised, concerning the desirable granularity of crossfeed, and its relation to the earlier issues about units of activity. Those questions are best pursued within the context of Design Space Analysis.

3.5. Design Space Analysis The Design Space Analysis offered by the design modellers provides a framework for keeping track of the argumentation made during the design process (and contributed to by the different modelling approaches). Much of the story told so far could have been cast in terms of such a design rationale. The original statement of the scenario, for example, describes an initial design space with two Questions, one being How many authors can input at once?, with the


257

Options one or many, the other being What should be the scope of Undo?, with Options local and global. Further design space analysis yields the observation that many of the options involve a trade-off between the two fundamental Criteria of, on the one hand, allowing each user an opportunity for independent work, and on the other, enforcing a shared focus on all users. The new design issues raised during the preceding discussion can be seen as extensions and refinements of the initial design space. For instance, item (3) of the knowledge analysis suggested that the display be enhanced so that U can see the scope of O's units of undoable activity. This would be encoded as a new design Question, What non-textual information is included in the display?, with the Options including Indicate units of activity, which satisfies the Criterion User knows how units are articulated. The discussion based on Software Engineering considerations introduced a new Question about the granularity of feedback and crossfeed with the novel possibility that it might be different for feedback and crossfeed instead of the default assumption that it is the same. Examination of the design space representation suggests the further possibility that perhaps the granularity of the crossfeed should be the same as the units of articulation, i.e. that [7's screen should be updated to reflect O's work by increments corresponding to the size of the Undoable units. Further analysis would seek to trace the consequences of taking that design choice, and to uncover relevant criteria to apply to it.

3.6. Formal System Modelling Again: Independence and Co-operation Problem (1) raised by the knowledge analysis concerned the issue of interference between the work of two users in the case of a MIP design with local Undo. Specifically, we asked what would happen if U tentatively types in some text and O inserts a missing word into that text before U decides not to keep the text and invokes an Undo to remove it. The formal system modelling approach can be applied — on a finer scale than was done above — to address issues of this kind. As we mentioned earlier, the only advice that formal system modelling analysis could provide on its own in this analysis was that on the basis of definability alone, the global Undo option is preferred. There has been overwhelming evidence from other modelling perspectives which demand the local option. If we are to provide true design advice, we must show how to resolve this apparent conflict between modelling perspectives. We begin by developing a simple algebraic definition of independence between two actions. Suppose that the set of user commands is represented by C, and the set of users by A. Any command presented to the system is then a labelled user command, denoted by the set C x A. The history is represented by the set of sequences of user-labelled commands, denoted by H — (C x A)*. If S represents the set of system states, we can define the state transition function as doit : (H x S) —> S. We can define independence of commands in these terms. For example, two commands a issued by U and b issued by O are independent if they satisfy the constraint:

doit(((a,U),(b,O)),s) = doit({(b,O),(a,U)),s). That is, the order of application of independent commands does not affect the resulting state. If two commands are independent in this way, then the kind of difficulty foreseen for MIP with local Undo does not arise. U can undo a as if b came before it in the history, whether or not that was actually the case. Exploiting such a property of independence was first discussed

258


by Ellis & Gibbs (1989) in their concurrency control algorithm for Grove. Prakash & Knister (1992) present a simple algorithm for performing local Undo when a kind of generalised independence condition of this kind is met. In such cases the relevant command can be moved to the end of the history by being transposed with later commands, and then subjected to a straightforward reversal. If a dependency is detected, the users are asked whether they want the interfering command Undone as well. However, this kind of strong independence is probably not what we want. The purpose of a collaborative editor is to support co-operation, not independence, and from the formal point of view co-operation is necessarily a form of interference! (Abowd & Dix, 1992). So the question becomes whether we can capture formally various notions of fine-grained inter-dependence. Suppose that U were working on the beginning of a paragraph and O were working on the end of the same paragraph. We would like the system to recognise this as an acceptable form of co-operation, and to allow U and O each to be able to Undo their own work. Such a context-sensitive analysis pushes the limits of what we currently know how to analyse formally. Much depends on having the right representations, both in the analysis and in the implementation. A sophisticated representation should be able to recognise inclusion relations between pieces of text, for example, which would allow the editor to deal correctly and automatically (that is, without having to ask) with the case described in problem (1) by removing O's inserted word along with U's tentative text.

4. Discussion and Conclusions Our aim in presenting the foregoing analyses has been to illustrate how the combined use of multiple modelling techniques, drawing on different disciplines and viewing an interface design problem from different perspectives, is better than the use of one technique alone. We argue against the idea that any one approach or technique is 'best'. Instead, we have shown how the interplay between different techniques brought to bear on a common design problem serves to clarify the relationships between the different techniques and improve the individual analyses. Concerning the design issue itself, the question of local versus global Undo in a collaborative editor, the collective analyses have yielded a fair degree of convergence on a preferred recommendation — namely, local Undo with multiple insertion points. However, to take this multi-disciplinary exercise as providing a single 'right' answer would be to seriously misunderstand the nature of the enterprise and underestimate its utility. The correct answer to this or any other design question depends intimately on the circumstances and details of the particular case, a point made many times by others and even for the particular case of Undo (Choudhary & Dewan, 1992). Rather than a single solution, the outcome of this analysis is a thorough understanding, from multiple perspectives, of the design options, of their consequences and trade-offs, of their impact on usability, and of other factors influencing the appropriate choices for interface design in particular cases. Using multiple approaches provides a depth of analysis and a breadth of scope beyond what could be achieved by any one technique alone. The analyses revealed a web of connections and interdependencies between the different approaches, and it is worth summarizing the main ones. As we told the story, formal system modelling provides the concepts and terminology drawn on by the PUM for its knowledge analysis. The knowledge analysis, together with the consensus of ICS and the Interaction


259

Framework in turn feeds back into the formal modelling for its fine-grain analysis of cooperation and independence, which then provides a tentative solution to one of the problems uncovered by the analysis. The PUM raises a new design issue, about providing the user with an indication of the undoable units. That issue and the user-centered criteria from the knowledge analysis feed into the design space analysis and help to structure the design space. The PAC software engineering analysis uncovers an issue which extends PUM's race condition problem into a general concern about synchronisation and consistency. PAC also raises a new question about the desirable granularity of crossfeed, from one user's work to another's display, and that question is also picked up by the design analysis and tentatively connected to the earlier question of indicating undoable units. Meanwhile, coming from a very different angle, the interaction framework and ICS user modelling techniques together provide arguments that support the conclusions of the other approaches but on independent grounds. Experience with multi-perspective modelling of Undo and other design tasks has demonstrated merits both for guiding design and also informing the modelling activity. We have demonstrated that a greater understanding of multi-perspective modelling can provide a design tool with analytic leverage. We have noticed, however, that the various techniques do not contribute equally to the overall analysis of different designs. In the Undo example of this paper, most of the mileage has come from the formal system modelling and PUM analyses, but that is not always so. The profile of contributions from the different approaches varies with different design tasks, so that in other cases those two approaches offer relatively less and other approaches more. Understanding the characteristics of design tasks and how they affect the profile of contributions is an open research question. We hope that the experience we have described with Undo demonstrates that this is also a very important research question.

Acknowledgements Much of the work reported here was carried out as part of the AMODEUS project, CEC ESPRIT Basic Research Action 3066, with the original single-perspective analyses being performed by members of the AMODEUS team. The extension to the system modelling in the light of the PUM analysis was done in collaboration with Alan Dix.

References Abowd, G D & Dix, A J (1992), "Giving Undo Attention", Interacting with Computers 4(3), pp.317-342. Barnard, P J (1987), "Cognitive Resources and the Learning of Human-Computer Interaction", in Interfacing Thought: Cognitive Aspects of Human-Computer Interaction, J M Carroll [ed.], MIT Press, pp.112158. Barnard, P J & Harrison, M D (1989), "Integrating Cognitive and System Models in Human-Computer Interaction", in People and Computers V, A Sutcliffe & L Macaulay [eds.], Cambridge University Press, pp.87-103, Proceedings of HCI'89. Bass, L & Coutaz, J (1991), Developing Software for the User Interface, Addison Wesley. Blandford, A E, Harrison, M D & Barnard, P J (1993), "Integrating User Requirements and System Specification", in Computers, Communication and Usability, P F Byerley, P J Barnard & J May [eds.], North-Holland, pp.165-196.

260


Choudhary, R & Dewan, P (1992), "Multi-user Undo/Redo", Department of Computer Science, Purdue University, Internal Report. Coutaz, J (1987), "PAC: An Object-Oriented Model for Dialog Design", in Proceedings of INTERACT 87 — Second IFIP Conference on Human-Computer Interaction, H-J Bullinger & B Shackel [eds.], Elsevier Science (North-Holland), pp.431-436. Dix, A J (1991), Formal Methods for Interactive Systems, Academic Press. Dix, A J & Runciman, C (1985), "Abstract Models of Interactive Systems", in People and Computers: Designing the Interface, P Johnson & S Cook [eds.], Cambridge University Press, pp.13-22, Proceedings of HCI'85, Norwich, September. Ellis, C & Gibbs, E (1989), "Concurrency Control in Group Systems", ACM SIGMOD Record 18 (2), pp.399407. MacLean, A, Young, R M, Bellotti, V M E & Moran, T (1991), "Questions, Options, and Criteria: Elements of Design Space Analysis", Human-Computer Interaction 6 (3 & 4), pp.201-250. Prakash, A & Knister, M J (1992), "Undoing Actions in Collaborative Work", in Proceedings ofCSCW'92: Conference on Computer Supported Cooperative Work, J Turner & R Kraut [eds.], ACM Press, pp.273280. Thimbleby, H W (1990), User Interface Design, Addison Wesley. Young, R M, Green, T R G & Simon, T (1989), "Programmable User Models for Predictive Evaluation of Interface Designs", in Proceedings of CHI'89: Human Factors in Computing Systems, K Bice & C H Lewis [eds.], ACM Press, pp.15-19.

Qualitative Models of User Interfaces Mark Treglown Institute of Educational Technology, The Open University, Walton Hall, Milton Keynes MK7 6AA, UK. EMail: [email protected] Analogy is an important factor in learning unfamiliar computer systems and problem solving when using those systems. Designers of computer systems can aid novice users by exploiting analogies and explicitly representing a model world with which the users are familiar as part of the user interface. Objects in the model world, and some operations that may be performed on them, are often analogous to those in the real world. We consider the qualitative reasoning approach to modelling people's knowledge of the real world and attempt to build qualitative models of objects and operations in the model world of a user interface. These models reveal features of existing systems that cannot be explained in terms of users' knowledge of the real world and suggest limits to direct engagement with on-screen objects. Keywords: analogy, qualitative reasoning, direct engagement.

1. Introduction Two principle paradigms have been employed in designing user interfaces to interactive computing systems, the ideas of the conversation metaphor, and the model world metaphor. In the conversation metaphor, users and systems engage in a dialogue, using languages of various complexities, about some unseen, but assumed, task domain. In the model world metaphor, the task domain is explicitly represented on-screen. Even with these direct manipulation interfaces, when users encounter them for the first time, as Carroll & Thomas (1982) suggest, by definition they do not have the knowledge required to successfully use the system. Instead, related knowledge is employed and is used as a metaphor for the material being acquired. While learning an unfamiliar system, users may devise analogies and employ them to describe the new system (Payne, 1991) but system designers may also deliberately make the metaphor explicit in the user interface; users may then carry knowledge of the familiar source domain used as the metaphor into the unfamiliar target of the model world represented. A number of user interfaces employ the real world as the metaphor for the task domain represented by the computer. In this paper, we attempt to build models of operations in these systems using a notation employed to describe people's qualitative models of the real

262

Mark Treglown

world, in an attempt to determine how successful the designers of these systems have been. We discover that some systems display behaviour which cannot be described according to everyday notions of causality, and questions are raised as to how users explain these systems' behaviours. Problems also arise from the need to implement metaphor-based systems on functioning hardware. A traditional view of system design, is to hide the underlying machine and to make the task the main consideration — which is strongly urged by Norman (1988) among others. Metaphors are intended to hide the machine, but as Smith (1987) found with the Alternate Reality Kit, the underlying computer system cannot be made invisible and often system-caused effects intrude and cause the metaphor to break down. We examine below how qualitative models of the user interface may describe the actual behaviour of the system by making reference to values that lie outside the metaphor and must be assumed by the user.

2. Artificial Intelligence Models of Computer Systems and Users Cognitive psychology has provide a number of models of users' knowledge of computer systems, the GOMS model (Card, Moran & Newell, 1983), being the most well known and exploited. This model, among other similar ones, is used to describe a user's task model (their 'how to do it' knowledge). Users are also said to form device, or functional, models ('how it works' knowledge) of systems. We are concerned here with representing knowledge about the device, in particular with the model world represented in the system's display as part of the system image. A number of attempts have been made to represent device models using knowledge representation techniques associated with artificial intelligence. Virvou (1992), for example, in her active help system, has modelled the semantics of UNIX commands using production rules similar to those of the problem solving system STRIPS. Similar models of system state changes, represented as the results of operators on a problem space, are employed in implementations of Programmable User Models — see for example (Blandford & Young, 1993) — and Cawsey (1992) employs simple causal rules to generate explanatory dialogues. Within these models, however, only the system state change is described, the fact that computer hardware is not infinitely fast (Dix, 1987) tends not to be noted, and that the system state may change over time is not represented. This consideration is also missing from attempts to use temporal logic to model and specify the behaviour of interactive systems. The need to be able to reason about time in human-computer interaction has been recognized, however, by Decortis et al. (1991), who note that artificial intelligence techniques provide useful descriptions and methods.

2.1. Qualitative Reasoning Qualitative reasoning is a branch of artificial intelligence. Its goal is to represent people's common sense knowledge about the physical world and physical systems. Qualitative models attempt to provide descriptions of physical systems and devices, and their behaviours, which do not rely on any external representation of the system or calculations. In attempting this, a number of issues relevant to human-computer interaction arise: • a qualitative model may allow system activity to be determined, that is, it is possible to deduce what is happening at a particular time; • a model may also be used for prediction, deciding what will happen in the future; and • a model may be used to deduce how a particular system state came about.

Qualitative Models of User Interfaces

263

process heat-flow Individuals: src an object, Has-Quantity(src, heat) dst an object, Has-Quantity(dst, heat) path a Heat-Path, Heat-Connection(path, src, dst) Preconditions: Heat-Aligned(path) QuantityConditions: A[temperature(src)] > A[temperature(dst)] Relations: Let flow-rate be a quantity A[flow-rate] > ZERO flow-rate OCQ + (temperature(src) - temperature(dst)) Influences: I-(heat(src), A[flow-rate]) I+(heat(dst),

A[flow-rate])

Figure 1: Process specification of heat flow, taken from Forbus (1984, p. 106).

2.1.1. Qualitative Process Theory The two approaches that have been adopted in developing qualitative models of physical systems which provide some means of describing a system's mechanism are the devicecentered and the process-centered approaches (Forbus & Gentner, 1986a). Despite being interested in modelling the computer system and its behaviour in qualitative terms we do not adopt a device-centred approach to modelling, see for example (de Kleer & Brown, 1984) Device-centred approaches to qualitative reasoning attempt to predict the behaviour of a system given qualitative and causal descriptions of device components and the structure of how the device's components are connected. The system's behaviour is described by the set of qualitative states each of the devices may be in at certain times given the possible behaviours of each of the devices and the structure of how the devices are inter-connected (the device topology). This device-centred approach to qualitative modelling has been applied to computer systems, but only in an attempt to analyse integrated circuits. Predicting a system's behaviour from the topology of the underlying, hidden, hardware examines systems at the wrong level of detail when considering an interactive user interface. In attempting to construct device-centred models of the objects that make up the model world represented in an interface, the amount of mechanism that must be inferred by users is considerable (Kuipers, 1984) and is a problem that requires, as yet not properly explored, investigation. Instead, as can be seen below, these on-screen objects can be described in a useful way if the process-centred approach to modelling is adopted. This approach reduces, but does not completely remove, the problem of system features having to be inferred by users. The qualitative reasoning notation used to describe the behaviour of an interactive user interface is Forbus' (1984) Qualitative Process Theory (QPT) notation. QPT models a physical system in terms of a number of objects which have properties or attributes, such as mass and volume, and in terms of processes which cause properties, or the state, of an object to change. Figure 1 shows an example process definition and is used to introduce the notation of QPT. This process definition describes the process of heat flowing from one object, the source, to another, the destination. A third individual is introduced within the definition which acts as a path over which heat may flow.

264

Mark Treglown

In many physical systems there may be a number of preconditions which must apply before a process can be said to be active, preconditions which lie outside the scope of QPT. Before water may flow from a tap for example, someone must turn the tap. Turning the tap cannot be predicted and is considered as an intervention from outside the system. In the case of the process shown in Figure 1, the heat path between the two objects must be aligned (arranged in such a way) so that heat may flow between them. An object in another building is unlikely to be noticeably heated by the heat source, for example. Quantity conditions define the threshold values that attributes must have before a process can become active. In the case of heat-flow, one body must be at a higher temperature than the other, and this fact is included in the definition shown in Figure 1.

3. Modelling Direct Manipulation Interfaces Olson (1992) states that users' mental models: " . . . are knowledge that the user has about how something works, its component parts, the processes, their inter-relations, and how one component influences another." The current view of mental models, as stated by many of the authors in the volume edited by Rogers, Rutherford & Bibby (1992), is that they are partial descriptions of systems. As was seen in the introduction to QPT, and the example process shown in Figure 1, the information that makes up the sort of knowledge described by Olson can be captured in QPT object and process definitions. Below, attempts to capture some of the operations familiar to users of interactive systems and some examples of real system behaviour are presented. These definitions, however, only consider a few aspects of a real system.

3.1. Objects in the Model World Qualitative Process Theory models view a system as a collection of objects with processes that act on them. This view is paralleled in object-based user interface designs (Foley & van Dam, 1982). The model world of a user interface displayed to the user in the form of the on- screen objects can also be modelled using QPT. On-screen objects have attributes which allow certain operations to be performed on them and which must be learned if the user is not to perform actions which give rise to semantic errors. A document may be printed, for example, as may a still graphics image (if there is a suitable printer attached to the system), but it does not make sense to attempt to print the data representing a video fragment. Whether an on-screen object may be directly printed is an attribute that not all objects will have. All objects will have a size, however, which determines whether or not they may be copied onto a storage volume which has a certain amount of remaining free space. An object's attributes may be learned by a number of means; users learn from manuals, from on-line help facilities, and from the object's name. Important means by which attributes are learned are from the design of the icon used to represent the object (Gittins, 1986), and through exploratory learning and trial and error experiences of using the system. Figure 2 gives some of the attributes associated with two types of object in a hypothetical multimedia system. If the attributes of objects that need to be learned by the user are collated, an object taxonomy similar to that described by Rosenberg & Moran (1984) is constructed. Such a taxonomy has been constructed for the system described by (Treglown & O'Shea, 1993), which is currently being programmed.


265

Quantity-Type(size) Quantity-Type(creator-application) Quantity-Type(size-if-run) Quantity-Type(printable-object) doc a document Has-Quantity(doc, size) Has-Quantity(doc, creator-application) Has-Quantity(doc, printable-object) video a video-fragment Has-Quantity(video, size) Has-Quantity(video, creator-application) Figure 2: Attributes of different types of objects in the model world of a user interface. Process delete-file Individuals: file an object, Has-Quantity(file, size) Preconditions: (T task-is-delete-file) QuantityConditions: A[size(file)] > ZERO Relations: Let delete-rate be a quantity A[delete-rate] > ZERO delete-rate OCQ- system-load delete-rate « Q + device-speed Influences: I- (size(file), A[delete-rate]) Figure 3: A process definition of deleting a single file from a storage device.

3.2. Modelling Change of Objects The basis of modelling physical systems within QPT is the notion of a process. Processes cause objects, or some attribute(s) of an object to change, and it is possible to describe a number of operations found in direct manipulation interfaces (as well as command language interfaces) as processes acting on objects. Figure 3 shows a process description for a single file being deleted from some storage device. Within this description, the amount of data that makes up a file is reduced at a rate proportional to the speed of the device on which the data is stored, (file deletion from hard disks is faster than from floppy disks). The rate at which the file is deleted also often depends on the current load to the underlying system. Determining how the system will behave when a threshold value for some attribute of an object is reached is achieved by limit analysis (Forbus, 1990). The outcome of the file deletion process, or the perceived outcome from, possibly mental, simulation, would be that the data would eventually disappear, and the object would cease to exist. The file deletion process becomes inactive at this point, and the influences placed on other object attributes or processes are removed. Another process that may be described in the QPT notation is the moving of a file from one location on a storage device to another, as is shown in Figure 4. The ability to capture a great deal of common sense knowledge about how files move, useful for both the system designer and the user, can be seen in this process. An example of this is the inclusion of the idea of a path over which the data that makes up a file travels. If the precondition that

266

Mark Treglown Process move-file Individuals: source-file an object, Has-Quantity(source-file, size) destination-file an object source-dir a directory destination-dir a directory, Has-Quantity(destination-dir, free-space) path a data-path, Connection(data-path, source-dir, destination-dir) Preconditions: (T task-is-move-file) Aligned(path) QuantityConditions: A[free-space(destination-folder)] > A[size(source-file)] A[size(source-file)] > ZERO Relations: Let move-rate be a quantity A[move-rate] > ZERO move-rate OCQ + device-speed(destination-folder) move-rate « Q - system-load Influences: I- (size(source-file), A[move-rate]) 1+ (size(destination-file), A[move-rate]) Figure 4: A process description of file movement.

the path between the source directory and the destination directory is aligned (connected in a suitable way) does not hold then it may be because the user has misspelt the name of either the source or destination, or because either the source or destination volume has not been attached to the system. An error message or a dialogue box requesting confirmation of the command may then be displayed. It has been suggested that one use of qualitative models of devices is to suggest ways in which a device may be used. This process definition explains why an often complex sequence of tasks must sometimes be performed by users to establish a data path between two storage volumes before the principle task of copying a file from one place to another may be performed. This suggests that such models may be of use in instructional material for novice users. It may also be noted that the requirement that the intended destination have enough space to contain the file being moved before the file movement process is able to become active is easily captured, and users may consider and perform sub-tasks which bring this condition about. In Figures 3 and 4, the process definitions of moving and deleting files were presented. While these definitions are influenced by the same quantities, there is no mechanism for the conditions that cause one process to become active to bring about the conditions that cause the other to become active. Usually a process becomes active when preconditions are satisfied which cannot be satisfied as a result of the effect of other active processes. In computer systems this may not be the case, however, as the aspect of the System 6 version of the Apple Macintosh Finder interface shown in the action-effect rule (Monk & Dix, 1987) in Table 1 demonstrates. When the user copies a file onto another storage volume, in this version of the user interface, the trashcan empties. Within the model world of the Desk Top, there is no visible, physical connection between storage volumes and the trashcan from which this behaviour may be predicted. Other examples of apparently non-causal system behaviour may be found, the phenomena of icons snapping to an invisible grid within a window when

Qualitative Models of User Interfaces Rl.

267

:: File icon appears in window associated with disk if window is open and file is copied across. The trashcan empties if 'full' and the trashcan icon shows 'empty'.

Table 1: An action-effect rule from the Apple Macintosh System 6 Finder interface. Process Motion(B, dir) Individuals: B an object, Mobile(B) dir a direction Preconditions: Free-Direction(B, dir) Direction-Of(dir, velocity(B)) QuantityConditions: A m [velocity(B)] > ZERO Relations: A[velocity(B)] OCQ- system-load Influences: I+(position(B), A[velocity(B)]) Figure 5: A simple process definition for describing motion in an ARK model world, after Forbus (1984, p. 133).

'dropped' onto the Desk Top, for example. By modelling systems using QPT it may be possible to identify such disparities between the system and the analogical source domain and correct them before a system is implemented. This suggests the need for empirical investigation of novices' success in learning systems which display behaviour which deviates from behaviour suggested by the analogy employed, and examination of the models (if any) that users construct to explain the system's true behaviour.

3.3. Modelling Real System Behaviour As can be seen in the process definitions above, QPT is able to capture how the state of objects change with time as a result of a number of operations available to users. QPT is also able to describe certain systems based on metaphors whose behaviour cannot be explained in terms of that metaphor. As was mentioned in the introduction, Smith (1987) suggested that all metaphor-based systems will display behaviour that cannot be explained in terms of the metaphor. The example he cites is the noticeable lag in updating a number of moving on-screen objects as the number of objects increases within an Alternate Reality Kit (ARK) simulation. The process describing a moving object, in a system such as ARK, can be seen in Figure 5. This simple definition describes a moving object uninfluenced by any external force. The object's speed depends, though, on the load on the underlying system, which is in turn is dependent on the number of moving objects present. An indication of the current load on the underlying system is absent from ARK and must be inferred by the user if they are to understand the behaviour of the system. This use of QPT demonstrates how a difference between the knowledge required to use the system and the knowledge structure suggested by the analogy exploited may be shown.

4. What Kind of Stuff are Programs Made of? This paper has explored the hypothesis of examining the model world of a direct manipulation user interface in terms used to describe the real world. Analogy plays a large component in

268

Mark Treglown

learning (Rumelhart & Norman, 1981) and analogies are explicitly used in the interfaces we are concerned with. Hence, we have considered the role of analogy in learning about physical domains (Forbus & Gentner, 1986b) in discussing some user interfaces. An important issue in the study of qualitative reasoning, and the related concern of naive physics, is the nature of the substances of which objects are made. It has been assumed in the process definitions shown that objects in the on-screen model world are made up of some stuff. It is an interesting question to ask what kind of stuff objects are made up of, and how users might understand on-screen objects. In modelling the processes that affect objects in an interactive user interface in ways analogous to physical systems, the nature of typical on-screen objects and how they behave are highlighted. On-screen objects typically behave as if solid, they may be picked up by the user by means of the on-screen pointer and moved. Within the process definition shown in Figure 4, when a file is moved from one storage volume to another, the data appears to flow, or be extruded from one location to another over time. If an object represents a file archive, then it may be split into a number of pieces, each represented by a separate object and rejoined. Examinations of the nature of substances in qualitative models (Collins & Gentner, 1990; Hayes, 1985) reveal that single views of substances are insufficient to explain the behaviour of substances when in particular states, or when subject to different processes. It can be asked if users apply different models of objects when performing different operations. One can ask questions about the type of stuff objects in the interfaces model world are made up of to account for the behaviour they demonstrate when subject to different processes such as copying and deletion. Might objects be subject to a phase transition from a solid state to a liquid when moved from one place to another?

4.1. Direct Engagement In discussing direct manipulation user interfaces, Hutchins, Hollan & Norman (1986) examine the notion of direct engagement where: "Actions apply to the objects, observations are made directly upon those objects, and the interface and the computer become invisible." (p.l 14) Hutchins et al. suggest that in order to produce a feeling of direct engagement a system needs to be responsive: "with no delays between execution and the results, except where those delays are appropriate for the knowledge domain itself." (p.l 15) Users can be seen to demonstrate a sense of direct engagement with a system. A striking example can be seen in videotaped studies of a multiple-user version of the ARK system conducted by the Open University and Rank Xerox EuroPARC in which two subjects shudder when their virtual 'hands' accidently touch in the computer's display. The discussion of QPT process models of operations found in interfaces which represent a model world suggest limits to the extent to which direct engagement may be discussed. The process definition of file movement given in Figure 4 assumes that this operation takes some time to accomplish and users form a model of the system in which the movement of data occurs over time. When considering Norman's (1984) discussion of stages of activity in computer use, there will often be a delay between the articulation and execution of a user's command and any results being visible. Evaluation of the results in comparison with the user's goals must be delayed until the results are available. In command language interfaces,

Qualitative Models of User Interfaces USER INTERFACE

—ti

269 move process

Imme diatei feedback, direct engagement

UNDERLYING SOFTWARE OBJECTS

1—K

move process As in Figure A

File moves over time Figure 6: Separation of user interface and underlying software.

and some systems which represent a model world, models of the way in which objects are altered by processes such as those shown in Figures 3 and 4 seem plausible in their inclusion of a temporal component to the way in which the object changes. The temporal component to these process definitions lessens the sense of direct engagement users have with the objects that the processes act upon. A sense of separation between the behaviour of on-screen objects and the behaviour of the software implementation of the objects can be aggravated by the form of the user commands employed to cause some operations to be performed. A user's intention to move a file from one place to another is often articulated by selecting it and dragging it to the desired location, an action which closely resembles the action that would be performed in the real world. Within the user interface, this move operation demands immediate display feedback as the object moves. This action, however, articulates a different move process, one modelled in Figure 4. This disparity is shown in Figure 6.

5. Conclusions Interactive user interfaces often represent a model world which is analogous to the real world. We have attempted to model some aspects of these interfaces using a knowledge representation notation used to model people's qualitative models of the real world. The difficulties in modelling some real system behaviour suggest that the analogies that were meant to be applied, cannot. Computer systems have a temporal behaviour, and time must form an important part of users' knowledge of the systems they use. We have presented models of system behaviour in which the way in which values of attributes of objects change over time is central. These models suggest that some ways of articulating commands, which are more consistent with the real world analogy aggravate the disparity between objects in the display and their underlying representation in software. The suggestion that the sense of direct engagement users perceive with the on-screen objects depends on immediate feedback between user input and results, limits the number of on-screen objects with which direct engagement may be felt. The QPT processes presented are hypothetical, but give realistic accounts of existing direct manipulation system behaviour. An obvious, and necessary, further step is to carry out empirical studies to examine if these models resemble those models (if

270

Mark Treglown

any) that users form of such systems, and to discover how users reconcile the behaviour of systems and the systems behaviour suggested by the analogies exploited.

Acknowledgements Macintosh and Finder are trademarks of Apple Computer Incorporated. I would like to thank Tim O'Shea, Debbie Stone and the anonymous reviewers for their helpful comments. Thank you once again to Kim Issroff for reading drafts. This work was supported by an Open University Studentship.

Appendix A. Use of QPT Notation Individuals

The objects involved in, and affected by a process.

Preconditions

Conditions which lie outside process definitions, usually suggesting some human intervention. Conditions of values of attributes that must apply before a process becomes active. Relations between values of attributes of object. Values are directly influenced by other values while a process is active. The amount (value) of some attribute of an object. The magnitude of a value (sign is ignored). Value is directly influenced by other values. The value increases while the process is active. Decreasing direct influence. The proposition, or condition, is TRUE.

QuantityConditions Relations Influences A[ ... Am [ . . . I+( . . . I -( ... (T . . .

] ] ) ) )

OCQ+

A value is qualitatively proportional to another. The value increases while the process is active, but the relationship is not as well defined as with a direct influence.

«o-

A value is inversely proportional to another.

References Blandford, A & Young, R M (1993), "Developing Runnable User Models: Separating the Problem Solving Techniques from the Domain Knowledge", in People and Computers VIII, J L Alty, D Diaper & S Guest [eds.], Cambridge University Press, Proceedings of HCI'93. Card, S K, Moran, T P & Newell, A (1983), The Psychology of Human-Computer Interaction, Lawrence Erlbaum Associates. Carroll, J M & Thomas, J C (1982), "Metaphor and the Cognitive Representation of Computing Systems", IEEE Transactions in Systems, Man and Cybernetics SMC-12 (2), pp.107-116. Cawsey, A (1992), Explanation and Interaction, MIT Press. Collins, A & Gentner, D (1990), "Multiple Models of Evaporation Processes", in Qualitative Reasoning about Physical Systems, D S Weld & J de Kleer [eds.], Morgan Kaufmann, originally appeared in Proceedings of the Fifth Meeting of the Cognitive Science Society, 1983.


271

Decortis, F, de Keyser, V, Cacciabue, P C & Volta, G (1991), "The Temporal Dimension of Man-Machine Interaction", in Human-Computer Interaction and Complex Systems, G R S Weir & J L Alty [eds.], Academic Press. Dix, A J (1987), "The Myth of the Infinitely Fast Machine", in People and Computers III, D Diaper & R Winder [eds.], Cambridge University Press, pp.215-228, also reprinted in Dix, A J (1991) Formal Methods for Interactive Systems, Academic Press. Foley, J D & van Dam, A (1982), Fundamentals of Interactive Computer Graphics, Addison Wesley. Forbus, K D (1984), "Qualitative Process Theory", Artificial Intelligence 24, pp.85-168. Forbus, K D (1990), "The Qualitative Process Engine", in Qualitative Reasoning about Physical Systems, D S Weld & J de Kleer [eds.], Morgan Kaufmann. Forbus, K D & Gentner, D (1986a), "Causal Reasoning about Quantities", in Proceedings of the 8th Annual Conference of the Cognitive Science Society, Lawrence Erlbaum Associates. Forbus, K D & Gentner, D (1986b), "Learning Physical Domains: Toward a Theoretical Framework", in Machine Learning: An Artificial Intelligence Approach Volume 2, R S Michalski, J G Carbonell & T M Mitchell [eds.], Morgan Kaufmann. Gittins, D (1986), "Icon-based Human-Computer Interaction", International Journal of Man-Machine Studies 24, pp.519-543. Hayes, P J (1985), "Naive Physics 1: Ontology for Liquids", in Theories of the Commonsense World, J Hobbs & B Moore [eds.], Ablex. Hutchins.EL, Hollan, JD & Norman, DA (1986), "Direct Manipulation Interfaces", in User Centered Systems Design: New Perspectives on Human-Computer Interaction, D A Norman & S W Draper [eds.], Lawrence Erlbaum Associates. de Kleer, J & Brown, J S (1984), "A Qualitative Physics Based on Confluences", Artificial Intelligence 24, pp.7-83. Kuipers, B (1984), "Commonsense Reasoning about Causality: Deriving Behavior from Structure", Artificial Intelligence 24, pp.169-203. Monk, A F & Dix, A J (1987), "Refining Early Design Decisions with a Black-Box Model", in People and Computers III, D Diaper & R Winder [eds.], Cambridge University Press, Proceedings of HCI'87. Norman, D A (1984), "Stages and Levels in Human-Machine Interaction", International Journal of ManMachine Studies 21, pp.365-375. Norman, D A (1988), The Psychology of Everyday Things, Basic Books. Olson, J S (1992), "The What and Why of Mental Models in Human-Computer Interaction", in Proceedings of Mental Models and Everyday Activity 2nd Interdisciplinary Workshop on Mental Models. Payne, S J (1991), "A Descriptive Study of Mental Models", Behaviour & Information Technology 10(1), pp.3-21. Rogers, Y, Rutherford, A & Bibby, P A [eds.] (1992), Models in the Mind: Theory, Perspective and Application, Academic Press. Rosenberg, J K & Moran, T P (1984), "Generic Commands", in Proceedings of INTERACT 84 — First IFIP Conference on Human-Computer Interaction, B Shackel [ed.], Elsevier Science (North-Holland).

272

Mark Treglown

Rumelhart, D E & Norman, D A (1981), "Analogical Processes in Learning", in Cognitive Skills and Their Acquisition, J R Anderson [ed.], Lawrence Erlbaum Associates. Smith, R B (1987), "Experiences with the Alternative Reality Kit: An Example of the Tension between Literalism and Magic", in Proceedings of CHI+GI'87: Human Factors in Computing Systems and Graphics Interface, J M Carroll & P P Tanner [eds.], ACM Press. Treglown, M & O'Shea, T(1993), "The Computer-Computer Metaphor for Multimedia Systems", in Proceedings of the 10th International Conference on Technology and Education. Virvou, M (1992), "A Human Plausible Reasoning Theory in the Context of an Active Help System for UNIX Users", School of Cognitive and Computing Sciences, The University of Sussex, Brighton, DPhil Thesis, published as Cognitive Science Research Paper No. 283.

Modelling Interaction using Template Abstractions Chris Roast Computing Research Centre, Sheffield Hallam University, 100 Napier Street, Sheffield SI 1 8HD, UK. Tel: +44 (0)742 533768 Fax: +44 (0)742 533161 EMail: [email protected] The use of formal system models in interface development is often criticised for failing to recognise the 'human' nature of interaction. This paper describes an abstract system model of interaction, termed the template model, which is designed to address this criticism (Roast, 1993). The template model relies upon the identification of template abstractions which are system components that have particular user significance. By employing template abstractions system properties can be linked closely to user requirements. Using this framework we describe two categories of system constraint that express general usability requirements output correctness and structural consistency. The appropriateness of the perspective offered by the template model and the requirements expressed using it are illustrated in the analysis of an electronic mail system. Keywords: formal methods, system modelling, interaction models, email.

1. Introduction To ensure that interactive systems are used effectively it is accepted that intended users should have a central role in system development. From the point of view of system specification and development, this concern is met by normally requiring that a system and its operating environment be modelled. In the case of HCI, users of a system are the principal environment with which the system interacts. Hence ideally developers should consider modelling users in conjunction with the system being designed — for example see (Barnard & Harrison, 1989; Barnard & Harrison, 1992). In practice efforts to model system users have not provided sufficient accuracy and generality to reliably support software development. Commonly user models are used solely as evaluation mechanisms, and are not in a form that would relate

274

Chris Roast

to a design representation. In contrast to this, formal system models offer direct design representations, and their the formal nature provides a degree of precision and accuracy that: 1. supports reliable development and refinement; and 2. emphasises constraints necessitated by the eventual artifact. This paper describes a system model of interaction (the template model) which introduces user oriented issues into interactive system development by including abstractions with a defined relevance and role for intended users. In this way we are able to consider 'the user' and express usability requirements without having to speculate about users' psychological or cognitive activities. In what follows, we introduce the template model of interaction and characterise two system properties that support ease of use: output correctness and structural consistency. A detailed description of the template model and properties expressed using it is given in (Roast, 1993). To illustrate the properties expressed using the template model we analyse the behaviour of one component of an electronic mail system.

1.1. System Modelling in HCI The modelling of interactive behaviour in HCI research has largely concerned itself with comprehending and explaining user behaviour. The aim of such an account of interaction is that an understanding of users and their use of interactive systems will inform the development process. Despite success at modelling the user, the models developed have not informed the development process. Models of users are employed as a means of evaluating design proposals (Barnard, 1985; Card, Moran & Newell, 1983; Kieras & Poison, 1985; Young & Whittington, 1990). This can be contrasted with the potential for models of interaction to directly structure and shape the design representation. We shall term this use of a model as a constructive use. The constructive application of user models of interaction in development is prevented by: 1. there being mismatches between the terms used by user models and those required for system development; and 2. user accounts of interaction often requiring a detailed description of the system being developed. The template model, described below, is designed to overcome these difficulties by requiring that the mismatch between user oriented terms and system components is explicitly addressed within development. The second of the above points is addressed by the ability to model systems in terms of abstractions that are uncommitted to details yet to be decided. System models enable specified requirements to be expressed in terms of abstractions that can be related directly to the system being developed while avoiding unnecessary detail. The constructive potential of system models means they can be introduced as a design representation early in the development process, and properties expressed using them can be asserted and verified throughout the process. The main problem for system models is that usability requirements are not easily specified using system concepts. Requirements such as ease of use and appropriateness for task are seen primarily as psychological properties and not as system properties. As a result, system models are normally seen as inadequate as a means of expressing usability requirements of systems.

Modelling Interaction using Template Abstractions

275

2. Abstract System Model The general requirements of interactive systems cannot easily be expressed as system properties, because of their psychological nature. Dix et al. have proposed that abstract models of interaction be used as a means of overcoming this problem, by providing a minimal formal framework with components germane to interaction (Dix, 1991; Dix et al., 1987; Harrison & Dix, 1990). The abstractions provided are designed to express properties of direct relevance to use, and thus avoid system details that may distract from a user centred perspective. In this paper we shall use a simple abstract system model similar to that of (Harrison & Dix, 1990; Roast, 1993)1. The model consists of: a set of system states S, one of which is identified as the initial state so £ S; a set of system outputs D; the output for any state is determined by a 'view' function v : S —> D. If the system is in a state s, then v(s) is used to denote that state's output. The state of the system is considered to change in response to inputs, there are two types of input that we wish to distinguish: 1. possible inputs generated by the user of the system K\ and 2. possible inputs not invoked by the user but by some external agent(s) E. Both user inputs and external inputs are modelled as functions on system states: i.e. for any input i e i ^ o r i g E w e have i : S —> S. If the system is in a state a and input i occurs the new state of the system is denoted by i(s). The overall abstract model of interaction is given

by the tuple:

(S,so,D,v,K,E).

The template model is based upon the use of template abstractions, these are system properties with a defined relevance to intended users. The use of template abstractions enables the expression of system properties that are realistic and representative of system use.

2.1. Template Abstractions To express system constraints which concern information of relevance to successful interaction we use abstractions of the state based model's components which are qualified by having relevance to the application domain and users' perceptions and tasks. The two types of template abstraction considered here are: results and display templates.

2.1.1. Results The principle focus of interactive activity is to manipulate and observe objects represented within the system. It is such objects which are the subject of users' individual tasks and sub-tasks. Based upon these objects, the users assess the successful completion of their tasks. For instance, while composing a letter in email, the task of ensuring a letter's content is correct may be determined by the characters and words forming the body of the message. Alternatively, the task of ensuring the letter is received correctly may depend upon details in the message's header. By contrast, more low level tasks such as correcting text will depend upon the current cursor location, and, say, the content of a cut and paste buffer. Within the template model those objects which are manipulated by the user and are to be represented by the system are referred to as results—after Dix (1991). As a system abstraction a result r is modelled as a function from states to a range of values (UT). 1

The distinguishing feature of the model presented in this paper is that it accommodates inputs that are not directly invoked by the user.

276

Chris Roast DISPLAYS

VALUES of dtat

David Lloyd Lunch? C Hall ABSENCE D Malton PC problems

"David L l o y d


"C H a l l

ABSENCE"

C Hall ABSENCE D Malton PC problems Admin PC problems

"C H a l l

ABSENCE"


undefined

Lunch?"

Figure 1: An illustration of a display template dtat applied to a number of displays. The display template dtat extracts a boxed item from within a list.

Definition 1: A result r is a function upon states which extracts information necessary for users to determine if a task (or sub-task) is complete.

(Given a state s, r[s) denotes the value of the result in that state.) The identification and specification of results can be seen as detailing the outcomes of task analysis, such as the Task Description Hierarchy of Diaper (1989).

2.1.2. Display Templates The notion of result offers a way of focusing attention upon system details relevant to successful use, display templates offer the similar facility when considering system output. Display templates are abstractions of system output that select display properties relevant to its perceived content. For example, the choice of colour scheme used by an interface is irrelevant to the display's content. Thus display templates would abstract over any particular colour scheme used. In general, we shall consider display templates as a means of referring to perceived display features such as menus, icons, cursors, selected items, etc. while 'hiding' details that are not significant to their recognition. For example, within a list of items the manner in which a selected item is depicted and its physical location on the display is immaterial to it being recognised as selected (see Figure 1). Thus, display templates extract output details recognised as potential sources of information (Harrison, Roast & Wright, 1989). In order to accommodate users' perception of system output, we shall identify a display template dt as a partial function from displays to a range of values


277

Definition 2: A display template dt is a partial function upon displays which extracts details that can be perceived by users as potential sources of information. dt:D-» Udt (Given a display d, if dt(d) is defined it denotes the value of the display template for that display.) Modelling display templates as partial functions accommodates the fact that whatever feature a display template extracts, it may not be a always present in the output. For example, a display template may extract a highlighted item within a list, and in the case when no item is shown highlighted (or the list is not shown) the display template is considered undefined. (See the example in Figure 1.) The particular details of any display template will depend upon an understanding of users' perceptions and expectations. The specification of individual display templates can draw upon existing research and empirical studies of users' perceptions and output techniques. 2.2. The Template Model The template model combines the basic state based model and the template abstractions described above. By employing template abstractions the model is implicitly dependent upon the tasks that a system is intended to support and users' perceptions. Definition 3: (S,so,D,v,K,E) templates Dt:

A template model consists of the state based model combined with a set of results R and a set of display {S,so,D,v,K,E,R,Dt)

Formulated in this way the template model provides a general framework for describing and reasoning about interaction. In contrast to other system models, the explicit incorporation of template abstractions focuses attention upon the assumptions and expectations regarding system use. One particular advantage of recognising template abstractions is that template model properties can be expressed which address a system's ease of use. Below we consider the expression of usability properties using the framework offered by the template model. 3. Template Model Properties 3.1. Supporting the User The first of the system properties for supporting ease of use is motivated from the fact that difficulties with interfaces are often attributable to inadequate information being output. To support the user's evaluation of system state it is necessary that particular information be communicated to the user. Within the template model, state information relevant to users is represented by the set of results (R) and reliable communication can be supported via display templates from (Dt). Thus one clear requirement is that results and display templates are coupled in some manner. In addition, for display template values to reliably indicate underlying result values, we require that result values can be determined from defined display template values.

278

Chris Roast Definition 4: The result r and the display template dt are related in an output correct manner (written OC(r, dt)) iff: /defined d(ai)) A \ V51 52 G 5 :

'

( defined

i

^

. . ,

) * ^

. . =

^

, , . .. *

^ ^

,, =

Thus, we obtain a one-to-one correspondence between display template values and result values whenever the display template is defined.

3.2. Structuring Interaction The second model property we shall introduce concerns a basic interactive requirement — interactive systems react to user inputs. In particular the way that systems react informs the user about events that have occurred. Successful interaction with most systems requires that users maintain some understanding of the system's state. Ideally, whenever the state changes, users will need to amend their understanding appropriately. Normally, the new state cannot be output in its entirety and what is output indicates the differences between the new and preceding state. We propose that changes to system results are of primary relevance. Thus, the difference between a new state and a preceding state which is communicated to a user should reflect changes in result values. We shall distinguish two forms of result changes: those that are invoked by user inputs and, those that are invoked by external inputs. In each case we describe how a display template can reliably indicate that a change has occurred. Definition 5: A result r and display template dt are structurally consistent

(written SC(r, dt)) iff:

Definition 6: A result r and display template dt are externally structurally consistent (written 5SC(r, dt)) iff:

Structural consistency requires that: If a user input alters the value of a result, then the display template associated with the result is altered also. The alteration to the display template is that its value changes and following the input it is defined. External structural consistency is a similar constraint, required to hold for external inputs. The template properties described have been expressed as general interface properties that support ease of use. The benefit of this generality is that it avoids unnecessary commitment to design details early within development. In order to illustrate these properties and their impact within a particular system, we present a template based analysis of an electronic mail system.

4. An Analysis of Email In the previous section, we claimed that the template properties of output correctness and structural consistency indicate general ways in which successful use can be supported.

Modelling Interaction using Template Abstractions File

1 1 1 1 1 1 1

Edit

-

Message

Folder

AndyBissett PETE COLLINGWOOD Chris Hall Reza Hajetni craig craig PALMENH Alain.Hui_Bon_Hoa Fabio Paterno SUSAN SMALL

Options Window

279 Help

Exam Board Some Dates STUDENT ABSENCE hird-Meeting Final Call for Papers - Jelia'9 Final Call for Papers - Europea RE: Primacy of Action availability of a paper eurographics workshop Unit leaders/exam boards

1/17/94 1/17/94 1/17/94 1/12/94 1/10/94 1/10/94 1/05/94 1/04/94 1/04/94 12/24/93

/CfiOAST Figure 2: An example DaVinci display.

This section will use these properties in the analysis of an existing commercial electronic mail system, D a V i n c i 2 . The analysis reported addresses the interface's support of the management of received messages, and illustrates instances of the template properties introduced. It is argued that discrepancies between the template properties and the interface behaviour identify particular interface problems which can be the subject of further analysis or design improvements. Employing the template model in an analytic manner requires that we identify system abstractions from the existing implementation which can serve as template abstractions. Since the system is 'given', template abstractions have not been developed from the analysis of intended users, their capabilities and context of use. As an alternative, we have identified and used system abstractions that D a V i n c i requires users to manipulate, recognise and evaluate, in order to complete particular tasks. In addition, the unavailability of a definitive specification of the system's behaviour means that our understanding of the general behaviour is drawn from experience with the system. If the system were the subject of a development project, the presence of a system specification would enable the verification of any assumptions about its general behaviour.

4.1. The Interface In general the D a V i n c i interface uses a number of windows, interaction is conducted by invoking, moving between and acting within windows. On occasions the system initiates dialogue windows, normally to confirm information or to prevent the violation of integrity constraints, such as sending a message without an address being specified. Our analysis is limited to one component of the interface, the management of messages received. This component allows users to view and manipulate a list of received message headers — each header shows the message's status, sender name, subject and date. The users Version 2.50a, produced by DaVinci Systems.

280

Chris Roast mail list: the list of received message headers will be represented by a result mail list.

•mail list : 5 —» item sequence current item: the message to which particular operations apply will be represented by a result

termed current: current : S -+ item To model the message list it is assumed the data type item represents message headers. Figure 3: Results evident from the message management interface of DaVinci. For each state of the system the results identify a list of items received and a current item respectively.

displayed list: the list of mail headers displayed in the window will be represented by the display template displayed list:

displayed list : D -+> item sequence highlight: the display template highlight is to extract an item which is highlighted:

highlight : D -+> item Figure 4: Display templates evident from the message management window.

are able to view, delete, reply to or forward a message by selecting the required header and invoking the appropriate command. When the number of items in the list cannot be shown all at once, the window upon the list becomes scrollable. Figure 2 shows a D a V i n c i display.

4.2. Template Abstractions We now consider what template abstractions are evident from the interface. For the interaction being examined, we identify two results: mail list and current, these are described in Figure 3. The two results extract state information upon which successful use of the system depends. The performance of various message management tasks will depend upon the user correctly evaluating and manipulating these abstractions. In the template model properties discussed the evaluation of results is supported by the use of display templates. For D a V i n c i we shall propose display features which serve as candidate display templates, these are described in Figure 4. If a system specification were available, then a formal definition of these template abstractions would be developed to enable the verification, or refutation, of template model properties. We now describe the behaviour of D a V i n c i illustrating cases where the template properties introduced earlier are adhered to, and cases where they are violated. It is proposed that cases where template properties are violated serve as a means of focusing more detailed design analysis. We first look at an instance of output correctness and then examples of structural consistency. 4.3. Output

Correctness

The notion of output correctness is a formulation of a familiar relationship between components of the system state and what is displayed. Informally it captures the idea that particular display properties can provide reliable unambiguous information about the state


281

Incoming messages: When the system receives a new incoming message, the message is added to the message list. On receiving a message the system prompts the user with a moded dialogue box containing the name of the sender. List navigation: The user can change which item is current using the arrow keys. The up (down) key makes the item above (below) the current item. If the current item is scrolled off the list window, the use of the up (down) key automatically relocates the window so the current is visible. Figure 5: Email behaviours illustrating structural consistency.

of a system. An apparent case of this within D a V i n c i is that the current item operated on by commands is the same as the highlighted item in the list. Formally, such a relation is represented by the proposal that for the display template highlight and the result current we have: OC(current, highlight): given two states with displays that have the same highlighted item, then the current item is the same in both states (and vice versa). Within the templates model's intended context of use, this proposed relation between current and highlight could be formally verified, or refuted, based upon a system specification. D a V i n c i ' s apparent adherence to this instance of output correctness is in fact contradicted by some of its behaviour. The displayed list truncates long sender names, thus if two email sender names only differ in their components that are truncated, then they appear the same in the displayed list. Therefore, a highlighted item does not necessarily determine a unique value of current, and OC(current, highlight) is contradicted. We claim that such contradictions may be a cause of usability problems, in this particular case users' actions may be focused upon the wrong item in the mail list. A violation of output correctness such as this can motivate detailed analysis, addressing the unspecified assumptions on which the design relies or re-considering system design decisions. Here we consider two possibilities: • The situations in which OC(current, highlight) is contradicted could be examined. If these contradictory situations were found to be improbable, or rare, to the extent that they would cause no significant usability problems, then our model of D a V i n c i could be refined to reflect this understanding. For instance, we may assume that of all the items received no two are identical even when their addresses are truncated. Such an assumption would not only reflect our understanding of probable usage, but it would also mean the situations that contradicted OC(current, highlight) would be considered unrealistic. • A second approach to the contradiction of OC(current, highlight) is to modify the system so that the contradictory cases cannot arise. For instance, the design decision to truncate the address within the display list can be questioned and other designs in which truncation is avoided may be considered.

4.4. Structural Consistency Structural consistency, in contrast to output correctness, concerns informing the user of changes within the system. To illustrate structural consistency we shall refer to three descriptions of system behaviour. The first two examples in Figure 5 give behaviours exemplary of structural consistency, whereas the last case in Figure 6 will be used to motivate more a detailed consideration of the system's design.

282

Chris Roast Item deletion: The user can delete the current item, as a consequence an adjacent item in the list becomes current. If the current item is scrolled off the list window, the delete operation has no visible effect. Figure 6: Email behaviour possibly violating structural consistency.

In the first example incoming messages (Figure 5), the arrival of a message alters the items in the message list, thus the value of the result mail list changes in response to an external input. The notion of external structural consistency proposes that this change is reflected in the display. The moded dialogue box indicates the arrival visually and can be interpreted as another display template (termed arrival box). Therefore this pattern of behaviour is consistent with a case of external structural consistency in which £SC(mail list, arrival box) holds. The example of list navigation in Figure 5 illustrates structural consistency with respect to user inputs. In this case the current list item is altered using the up (down) key, thus the result current changes value. The appearance or movement of the highlight gives a visual response to the operation. Thus the list navigation behaviour is consistent with SC(current, highlight). For this instance of structural consistency to hold it is necessary that any operation that alters current has the visual effect of altering highlight. The final example item deletion (Figure 6) illustrates system behaviour which contradicts this. When the current item is scrolled off the window and then deleted, both result current and mail list are changed yet: the display template displayed list remains unaltered, and the display template highlight remains undefined. The fact that list navigation is consistent with SC(current, highlight), while item deletion is not, suggests that deleting an item that is not in the view could be the source of problems for users. It is possible that users will be unaware of the effect of the delete operation. As with the violation of output correctness discussed above, the conflict between the cases of list navigation and item deletion can direct further analysis. We can address this conflict in two ways we can analyse the contradictory behaviour further, or we can consider modifications to the system behaviour that would avoid the conflict. Taking these in turn: # The situation described in item deletion which contradicts SC(current, highlight) occurs when the highlighted item is scrolled out of view before being deleted. If detailed analysis of this situation revealed that this behaviour does not arise in general use, then our model of DaVinci could be refined to reflect this feature of general use. For instance, we may explicitly assume that highlight is always defined prior to a delete operation, in which case SC(current, highlight) would hold. • The second approach to the conflict described is to modify the system so that the problematic behaviour described does not occur. In this case the deletion behaviour can be modified to be consistent with list navigation's behaviour, that is we can formally require that SC(current, highlight) holds. For example, we can propose that upon deleting an item, if necessary, the window automatically scrolls to the show the highlighted item. This section has given examples of the template properties of output correctness and structural consistency. We have shown how the behaviour of DaVinci is suggestive of particular instances of the template properties. However, the various interactive situations permitted in


283

D a V i n c i conflict with the properties. The two conflicts identified have motivated a more detailed consideration of: the assumptions made about normal/probable use; and, the design of D a V i n c i . Addressing these conflicts in this way illustrates the precision provided by the template model perspective and shows how it can impact specification and design. Thus, within conventional system development the template model can serve as means of focusing attention upon usage characteristics that may otherwise go unnoticed. 5. Constructive Development The previous section presented a template based analysis of an existing system. More generally, a potential benefit of the template model is that it can be used constructively in system development. Hence, its analytic use, although valuable, does not reflect its more general role in design. Being an abstract formal model of interaction, the template model can serve as a basis for expressing system requirements prior to the particulars of system appearance and behaviour being determined. As design details are made explicit during development, their impact upon the template perspective can be assessed without requiring a full implementation. In this respect, the value of the template model over other system models is its emphasis upon identifying template abstractions, and making them primary abstractions within a specification. In this way it is possible to avoid design decisions which may introduce conflicts such as those considered in Section 4. For example, we can speculate that in the development of an email interface, the template abstractions (such as those in Figures 3 and 4) would be formally defined and their intended user significance identified. The identification of template abstractions can be seen as a product of a form of task analysis. Based on these abstractions, requirements supporting ease of use can be expressed as template model properties, such as: OC(current, highlight),

SC(current, highlight) and £SC{mail list, arrival box). Following this, the suitability of particular design decisions can be formally assessed in terms of such assertions. As a result, the development can progress and avoid decisions about system behaviour and appearance which conflict with these requirements. In general the template model can be used to focus system development upon user oriented concerns and user oriented requirements early in development. In addition, its constructive use involves explicilty identifying some of the basic suppositions on which a system's successful use depends. 6. Conclusions In this paper we have described an abstract system model in which interface properties can be expressed independently of a fully fledged artifact. Conventional system models do not accommodate the user's perspective upon interaction, and as a result do not enable the expression of interface properties pertinent to usability. The template model developed overcomes this by introducing abstractions which are formally characterised and are required to correspond to psychologically relevant, or task relevant, information. This enables mathematical properties to be linked closely with user requirements, we have described two such properties output correctness and structural consistency. Output correctness ensures that when information relevant to task is output it is accurate and unambiguous. Structural consistency provides a means of relating observable system behaviour to changes in the state

284

Chris Roast

information relevant to the users' tasks. Taking an existing system and analysing it using concepts from the template model, we have illustrated the relevance of the template properties developed and the potential impact of adopting them in development. In general, we have shown the potential benefit of employing system models in design while accommodating a user perspective. The introduction of a user perspective into system models ensures the concepts on which successful interaction relies are given a precise description that respects the constraints of the artifact being developed.

References Barnard, P J (1985), "Interacting Cognitive Subsystems: A Psycholinguistic Approach to Short Term Memory", in Progress in the Psychology of Language, A Ellis [ed.] #2, Lawrence Erlbaum Associates, pp. 197258, Chapter 6. Barnard, P J & Harrison, M D (1989), "Integrating Cognitive and System Models in Human-Computer Interaction", in People and Computers V, A Sutcliffe & L Macaulay [eds.], Cambridge University Press, pp.87-103, Proceedings of HCI'89. Barnard, P J & Harrison, M D (1992), "Towards a Framework for Modelling Human-Computer Interactions", in Proceedings of East-West International Conference on Human-Computer Interaction, J Gornostaev [ed.],ICSTI,pp.l89-196. Card, S K, Moran, T P & Newell, A (1983), The Psychology of Human-Computer Interaction, Lawrence Erlbaum Associates. Diaper, D [ed.] (1989), Task Analysis for Human-Computer Interaction, Ellis Horwood. Dix, A J (1991), Formal Methods for Interactive Systems, Academic Press. Dix, A J, Harrison, M D, Runciman, C & Thimbleby, H W (1987), "Interaction Models and the Principled Design of Interactive Systems", in Proceedings of the European Software Engineering Conference 1987, H Nichols & D S Simpson [eds.], Springer-Verlag, pp.127-135. Harrison, M D & Dix, A J (1990), "A State Model of Direct Manipulation", in Formal Methods in HumanComputer Interaction, M D Harrison & H W Thimbleby [eds.], Cambridge University Press, pp. 129151. Harrison, M D, Roast, C & Wright, P C (1989), "Complementary Methods for the Iterative Design of Interactive Systems", in Designing and Using Human-Computer Interfaces and Knowledge-based Systems, G Salvendy & M Smith [eds.], Elsevier Science (North-Holland), pp.651-658. Kieras, D & Poison, P G (1985), "An Approach to the Formal Analysis of User Complexity", International Journal of Man-Machine Studies 22, pp.365-394. Roast, C (1993), "Executing Models in Human-Computer Interaction", Department of Computer Science, University of York, York, UK, PhD thesis. Young, R M & Whittington, J (1990), "Using a Knowledge Analysis to Predict Conceptual Errors in Text-editor Usage", in Proceedings of CHI'90: Human Factors in Computing Systems, J C Chew & J Whiteside [eds.], ACM Press, pp.91-97.

The Formal Analysis of Human-Computer Interaction During Accident Investigations Chris Johnson Department of Computer Science, University of York, Heslington, York YO1 5DD, UK. Tel: +44 (0)904 433376 Fax: +44 (0)904 432767 EMail: [email protected] Many safety-critical applications rely upon complex interaction between computer systems and their users. When accidents occur, regulatory bodies are called upon to investigate the causes of user 'error' and system 'failure'. Reports are drawn up so that the designers and operators of future systems will not repeat previous 'mistakes'. These documents present the work of specialists who are drawn from many different technical disciplines: human factors; forensic investigation; engineering reconstruction; computer simulation; etc. Thefindingsof these different experts are often separated into different sections. This creates a number of problems. Important evidence can be hidden within numerous appendices. The interaction between systems and users can be obscured by tortuous cross referencing schemes. There are occasional temporal ambiguities and inconsistencies between the different analyses. This paper presents ways in which formal methods can be exploited to address these problems. Mathematical notations provide means of representing and reasoning about the circumstances that lead to accidents in human machine systems. Executable logics can also be used to simulate event sequences. These simulations might be shown to other analysts. They can be used to encourage agreement on the course of events prior to more detailed investigations. Keywords: safety-critical systems, temporal logics, formal methods, simulation. 1. Introduction Accident reports are intended to ensure that the faults of previous systems are not propagated into future applications. For example, the Presidential investigation into the Three Mile Island

286

Chris Johnson

accident led the United States' Nuclear Regulatory Commission (NRC) to adopt a policy of minimal intervention (Pew, Miller & Feehrer, 1981). Whenever possible operators should not be required to intervene in order to preserve the safety of their system. It was stipulated that the high-head charging pumps, part of the emergency cooling equipment, must be run for at least twenty minutes after reactor scrams. In 1979, changes in the generation process employed by the North Anna reactor led to dangerous temperature profiles following a scram (Duncan, 1987). The operators were faced with a difficult choice. If they obeyed NRC regulations then the safety of the plant would be threatened. If they disobeyed the regulations then the plant could be saved but they would break the NRC conditions of operation. Plant management chose to disregard the regulations; a pump was taken off the coolant circuit and the emergency was resolved. This incident underlines the importance of accident reports within the design and management of safety-critical systems. Regulatory authorities use them to direct the operation of interactive systems. It also illustrates the need to consider both human and system requirements during accident analysis. An initial response to the operator 'error' at Three Mile Island led the NRC to rely upon systems engineering. Users were forced to intervene when systems engineering failed to guarantee the safety of the North Anna reactor. Accident reports present the findings of specialists who are drawn from many different technical disciplines: human factors; forensic investigation; engineering reconstruction; computer simulation; etc. These different strands of analysis are, typically, contained in different chapters. For instance, the UK Department of Transport's Air Accident Investigations Branch (AAIB) report into a near collision at Gatwick airport places meteorological and engineering details within the body of the document whilst the human factors analysis is placed in an appendix (AAIB, 1989). Reactor design and operator training are considered in different chapters of the Watt Committee's report into the Chernobyl accident (Worley & Lewins, 1988). This separation is useful in the sense that each section of a document reflects a particular perspective or focus. It also creates a number of problems. Many reports rely upon tortuous cross-referencing to draw together different strands of analysis. Others leave the reader wondering how the different findings of the experts can form a coherent picture of the events leading to an accident (Kletz, 1985). Further problems can be created by inconsistencies within the text of accident reports. The Kegworth investigation concluded that on-board systems failed to prevent pilots from shutting down a healthy engine (AAIB, 1990). In some places the engines are referred to by their numbers; 1 and 2. Elsewhere they are referred to as the left and right engines. Some of these inconsistencies arise from the verbatim reporting of witnesses when they describe the ways in which they interacted with their systems. Others have no such justification. This creates considerable confusion; the reader of the report is drawn into continual cross referencing to ensure that they understand the natural language description (Johnson, McCarthy & Wright, 1994). This paper argues that formal methods can be applied to address some of the limitations of conventional accident reports. Mathematical specifications are increasingly being recruited to support the engineering of complex applications. For example, both National Aeronautical and Space Administration (NASA, 1989) and the European Space Agency (Gamble, Taylor & Vazquez, 1991) are applying formal notations to represent and reason about on-board communications systems. Formal analysis is a necessary part of many design projects commissioned by the United Kingdom's Ministry of Defence (MOD, 1991). These techniques have also been applied to support interface development. Thimbleby has shown how finite

The Formal Analysis of Human-Computer Interaction ...

287

state machines might be used to analyse interactive dialogues with a range of commercial applications (Thimbleby, 1993). Wright, Fields & Harrison (1994) show how Hoare's Communicating Sequential Processes (CSP) notation can be used to describe interactive properties of safety-critical systems. Previous work has built upon this research by showing that executable logics can be used to specify and prototype human-computer interfaces (Johnson, 1993a; Johnson, 1993b; Johnson & Harrison, 1992). Formal notations provide a precise and concise means of representing traces of interaction. Executable subsets of mathematical specifications can then be used to derive partial implementations. These prototypes can be shown to system operators. They provide a medium of communication between the various members of concurrent design teams. Many of these benefits can also be extended to the analysis of accidents involving safety-critical systems. Logic can be used to represent the interaction that leads to a failure. These notations provide means of stripping out irrelevant detail which might otherwise obscure critical events during the course of an accident. Executable subsets of logic formalisms can also be used to derive simulations from abstract descriptions. This provides a means of ensuring that mathematical descriptions accurately capture the complex interactions that lead to accidents. Much of the work described in this paper is in its early stages of development. Many questions remain unanswered. We do not address ways in which formal notations might be used to develop cognitive models of system operators. This is a weakness in our work; expertise and experience play an important role in the response to system failures. Further research intends to address this limitation. Our intention here is to demonstrate that formal techniques can be pragmatically applied to real-world problems.

1.1. The Outline of this Paper Section 2 briefly describes the events leading to the Chernobyl accident. This is used as an example for the remainder of the paper. Section 3 goes on to show how logic can be used to represent some of the factors which contributed to the failure in this human-machine system. Section 4 builds upon this and uses the same notation to specify requirements that must be met if the accident is not to be repeated in similar systems. Section 5 exploits the logic formalism to represent and reason about techniques that might be used to satisfy these requirements. It is argued that the application of formal methods does not remove the need for skilled analysis. It is perfectly possible to specify techniques that would threaten the safety of an interactive system. Section 6 describes means of identifying these potential threats. Executable subsets of the logic notation can be used to directly derive simulations of the events leading to an accident. The Prelog prototyping tool has been developed so that prototype displays can also be used to determine whether changes to human-machine interfaces might reduce the likelihood of future accidents. Section 7 presents the conclusions that can be drawn from this research. Areas for further work are also discussed.

2. An Example At 01.23 hours on the 26th April 1986, Unit 4 of the Chernobyl Nuclear Power Station experienced a rapid, uncontrolled power release. This destroyed the reactor core and breached the containment buildings. Radioactive material was released into the environment for almost ten days. The events leading to this emission started with a test that was intended to determine whether the reactor could continue to supply power once it had begun to be run down. If successful, the pumps which supplied emergency coolant to the reactor could be powered

288

Chris Johnson

by that reactor until diesel generators were started. Similar tests had failed on two previous occasions because the power level had fallen too rapidly. Operator intervention provided the immediate causes of the accident: they disabled the plant protection systems and made several errors in the way that they conducted the tests. The effects of this intervention were exacerbated by design flaws in the construction of RBMK reactors. The events surrounding this accident are used to illustrate the remainder of this paper. This example is appropriate because it typifies many of the problems that frustrate investigations into human-machine failures. The causes were complex. They involved the interaction of operator 'error' and design problems. It has also been the subject of a number of international reports (USSR SCUAE, 1986; Worley & Lewins, 1988). These documents have been supplemented by analyses from a wide range of experts including medical authorities (Morrey et al., 1986), meteorological researchers (Persson, Rodhe & de Geer, 1986), human factors experts (Reason, 1990) and systems engineers (Potter, 1988). A further reason for analysing the causes of this accident is that the TACIS'92 (Technical Assistance To The Commonwealth of Independent States) project is continuing to bring the human-machine systems in the CIS nuclear industry up to western standards. One of the aims of this initiative is to install computer based management information systems in RBMK reactors, similar to that at Chernobyl (Dynan & Francis, 1993).

3. Informal Descriptions to Accident Analysis Logic provides a precise and concise means of representing the informal description of the previous section. One of the contributory factors to the Chernobyl accident was an operator input error (Reason, 1990). A 'hold power' order was not entered once the power reduction test had been initiated. This can be represented by the following clause. Intuitive readings for the logic will be given throughout the remainder of the paper: dangerous-power Jiold.omission : — operator -inputînitiate .power -reduction), -i operator -input(power -hold).

(1)

This states that a dangerous omission occurs if operators issue input to initiate a power reduction and they do not issue input to hold the power level. The logic notation can also be used to represent system behaviour. In our example, the input omission led to a programmed power reduction below the target level of 700-1000 Megawatts (MW). The power fell to 30MW. The RBMK reactors at Chernobyl were known to be unstable below 700MW: dangerous -power .reduction : — dangerous-power-hold.omission,

mega-watt .power Jevel(30).

(2)

This states that a dangerous power reduction occurs if the operators initiate the power reduction and omit the power hold command and the power level is 30MW. The previous clause illustrates an important limitation of first order logic as a means of representing interactive systems. There is no notion of sequence. A dangerous-power-reduction would be true if the power level reached 30MW and then the operators issued input to initiate the power reduction. This does not accurately describe the course of events leading to the


289

accident. The input was issued before the power fell below safe limits. Previous papers have described a number of different means of representing time within logic specifications of interactive systems (Johnson, 1991; Johnson & Harrison, 1992). For instance, it is known that the operators of the Chernobyl plant had stabilised the power at 200MW by 01.00 hours. This can be represented by the clause: response.dangerous.power.reduction(0100) : — operator ,input{initiate.power .reduction, 2310), mega.watt.power mega.watt.power

.level(30,2349), _level(200,0100).

(3)

This states that the operators respond to a dangerous power reduction if a command to initiate power reduction is issued at 23.10 hours and the power level is 30MW at 23.49 hours and by 01.00 hours the power is 200MW. Additional parameters might be introduced to represent the date, month and year on which these propositions were true. This approach addresses one of the major problems in existing accident reports. Temporal ambiguities often arise within the chapters produced by different teams of experts. For instance, the Watt report into the Chernobyl report contains the following lines: "The operator stabilised the power at 200MW (thermal) by manual control at 01.00 on 26th April... By 01.23 the operator had apparently stabilised the reactor system at least temporarily." (Worley & Lewins, 1988, p.20) The first sentence can be interpreted as meaning that the reactor stabilised at 01.00 hours. Doubt is created by the second sentence which appears several paragraphs later; manual intervention started at 01.00 hours and the reactor had not stabilised until 01.23 hours. This ambiguity illustrates the importance of identifying the actual sequence of events which leads towards an accident. The final recommendations of the Watt report embody the second interpretation; systems must present information about the state of the reactor during the delay between operator intervention and stabilisation. Other sections do not expand upon this point. It is not addressed in the chapter on reactor operation and training even though the delay would seem to be a significant stage during interaction with the system. The explicit parameterisation of clauses with time stamps, as in (3), can be used to avoid temporal ambiguity. These clauses provide a partial ordering or time-line for the events leading to an accident. For instance, the previous quotation might be represented as follows: manual.response.to

.power _reduction(0123) : —

operator Jnput(initiate.power .reduction ,2310), mega.watt.power .level(30,2349), operator Jnput{manual .control .initiate ,0100), mega.watt.power .level(200,0123).

(4)

This states that the operators respond to a dangerous power reduction if a command to initiate power reduction is issued at 2310 hours and the power level is 30MW at 23.49 hours and at 01.00 hours operators issue input to initiate manual control of the reactor and by 01.23 hours the power is 200MW. This clause makes explicit the observation that manual control started at 01.00 hours and that the power had reached 200MW by 01.23 hours. This avoids the ambiguity that was present in

290

Chris Johnson

the natural language description. Such clauses do not, however, provide a panacea for accident analysis. The Ukranian parliament has recently voted to cancel a 1991 resolution closing the remaining Chernobyl RBMK units. They will remain in operation until 2003 (Cruickshank, 1993). Safety requirements must be identified if previous faults are to be avoided in other systems.

4. Initial Analysis to Safety Requirements Logic can be used to represent requirements that might be imposed upon interactive systems as a result of accident analysis. For instance, from (4) we know that the power level reached 30MW at 23.49 hours. The danger might have been averted if the operator had issued a command to hold the level at a time when the power was above the critical limit: halt-power-reduction(2330) operator -input(initiate

: — -power .reduction, 2310),

operator Jnput(power -hold, 2330), mega-watt-power Jevel(700,2330).

(5)

This states that the operators respond to a dangerous power reduction if a command to initiate power reduction is issued at 23.10 hours and input is issued at 23.30 to hold the power level and the reactor was generating 700MW at 2330. Such clauses cannot easily be used to guide the development and operation of future systems. They refer to specific events at particular moments during an accident. Operators might have responded to the power reduction by issuing the power-hold command at 23.31 hours, 23.32, 23.33 etc. Analysts would have to repeat clauses such as (5) to describe the effect of operator intervention at each of these points in time. This problem can be avoided by the introduction of temporal logic operators. For example, the 0 (read as 'eventually') operator can be defined for a formula w as follows: Ow 3t > 0, w\t — true. In this definition w\t is intended to represent the evaluation of w at time t. Manna & Pnueli (1982) present a more complete introduction to the syntax and semantics of this logic. In contrast, the following clause shows how temporal logic operators can abstract away from the specific time points that were used to describe the accident scenario: eventual-halt Jn-power-reduction

: —

operator-input(initiate-power-reduction), ^{operator -input{power -hold), mega-watt-power Jevel(700)).

(6)

This states that the operators respond to a dangerous power reduction if a command to initiate power reduction is issued and eventually input is issued to hold the power level and the reactor is generating 700MW. The problem with this requirement is that the operator may not be sufficient to preserve the safety of the system. The requirement specifies that users must eventually stabilise the power at 700MW. It would be perfectly possible for the power level to dip below 700MW before the input is provided. Additional temporal operators can be exploited to avoid this limitation. Theli (read as 'until') operator can be defined for formulae, w\ and w2, as follows: w\Uw2 & V
The Formal Analysis of Human-Computer Interaction . . .

user.response.to-dangerous-power.reduction : — operator .input(initiate .power .reduction), (mega.watt.power.level(Level), Level > 700) U operator.input(power.hold).

297

(7)

This states that the operators respond to a dangerous power reduction if a command to initiate the reduction is issued and the power is greater than 700MW until a command is issued to hold the power level. Such requirements do not represent the final stage in accident analysis. These clauses must be used to guide the detailed design or re-design of similar applications if previous failures are not to be repeated. In other words, it must be possible to identify a range of techniques that might be employed to achieve safety requirements and break the chain of events leading towards failures.

5. Safety Requirements to Design Specifications Systems engineering might be recruited to achieve the requirements specified by clauses such as (7). In the aftermath of the accident "arrangements were made to avoid the operation of the reactor below quarter power except during start-up and shutdown" (Worley & Lewins, 1988). Automated systems can withdraw neutron absorbing control rods when reactor power falls below the 800MW threshold. This technique achieves the requirement of (7) by substituting system.input for operator .input: system.response .to .dangerous .power .reduction : — mega.watt.power.level(Level), Level < 800, sy stem.input^withdraw.control-rods).

(8)

This states that the system responds to a dangerous power level if the power is below 800MW and the system withdraws the control rods. It is important to emphasise that the use of formal methods does not remove the need for skilled analysis. The action of the automated system described in (8) would pose a significant threat to the safety of similar reactors. This point can be clarified by a description of the reactor physics which compounded the operator's input omission, described in (1). RBMK reactors have positive void coefficients. In other words, the water in the coolant system actually hinders the reaction. When the Chernobyl operators initiated the test they increased the flow of water to the coolant system. This was necessary in order to test the capacity of the pumping equipment, as described in Section 2. The additional water reduced the power of the reaction. Clause (4) described how the operator assumed manual control as the power level fell to 30MW. They attempted to stabilise the situation by withdrawing the neutron absorbing control rods, just as the system would have done in (8). This exacerbated the problem because once the control rods were re-introduced the coolant turned into steam. This created a high positive power coefficient as the dampening effect of the water was lost. In other words, the 'solution' described by (8) would have jeopardised the safety of the system. The use of the logic notation does not, therefore, guarantee the identification of safe and acceptable solutions. It does, however, provide a precise means of representing design alternatives. For

292

Chris Johnson

instance, the solution described in (8) might be rejected in favour of a protection system that prevented the control rods from being withdrawn. Systems engineers (Worley & Lewins, 1988) have suggested that the accident might have been avoided if operators had been faced with interlocks and additional protection systems: system .avoids _power .coefficients : — mega-watt-power Jevel(Level),

Level < 800,

(-i operator -input (withdraw .control.rods), -i system-input(withdraw-control-rods))U {mega.wattjpower Jevel(Level), Level > 1600).

(9)

This states that the system avoids dangerous power coefficients if the power is below 800MW and input to withdraw the control rods is blocked until the power is more than 1600MW. It is important to consider the impact which such systems engineering solutions might have upon the usability of a control system. The previous clause specifies that operators must not issue input to withdraw -control-rods until the power level rises. Such delays can lead to frustration and error (Kuhmann, 1989). Unpredictable behaviour is likely to occur if the system were to process a backlog of delayed input (Ellis & Gibbs, 1989). Delayed commands might take effect at inappropriate moments during interaction. The presentation of a large amount of contextual information is required before a user can resolve such instances of unpredictability. Logic can also be used to represent such presentation requirements. Warnings could be issued to inform the operator of the underlying behaviour of the system. A rod Jock Scon might warn users that the rods could not be withdrawn: system-avoids-and-presents-power-coefficients : — mega-watt-power Jevel(Level), Level < 800, (display(low -power .warning),part(low ôperator-input(withdraw.control.rods))

.power .warning, rod Jock -icon), U

(mega-watt-powerJevel(Level), Level > 1600, display (power part(power -level-resolved, rod-withdraw.initiated)).

-level-resolved), (10)

This states that the system avoids dangerous power coefficients if the power is below 800MW and input to withdraw the control rods is blocked and a rod lock warning is part of a low power display until the power is more than 1600MW and a rod withdrawal icon is part of a display indicating that power has stabilised. Previous sections have argued that a logic notation can be used to reason about the events leading to accidents in human-machine systems. This notation provides a means of representing design objectives for future systems. Logic can also be used to reason about alternative means of achieving these safety requirements. It is important to emphasise, however, that the formal representation of interactive techniques, such as that described by (10), will not guarantee the safety of an application. For instance, inattention and fatigue may prevent operators from observing error messages such as rod Jock.icon. Executable subsets of temporal logics provide a means of detecting these problems. Simulations and prototypes can be directly developed from clauses such as (10). They can be shown to the different parties involved in accident investigations. They can also be shown to the potential operators of safety-critical systems. They are amenable to experimental analysis. They can be used


293

to determine the cognitive, perceptual, physiological and sociological demands that a control system might place upon its users.

6. Simulation Design specifications, expressed in formal notations, provide the non-formalist with little idea of what it would be like to interact with a system. Prototypes provide a far better impression of the look and feel of a final implementation. Several research groups are investigating ways of deriving partial implementations from temporal logic specifications (Moszkowski, 1986). We have developed Prelog. This prototyping tool integrates the Tokio temporal logic metainterpreter (Fujita et al., 1986) and the Presenter screen management system (Took, 1991). This provides a means of directly introducing presentation details into clauses such as (10). For instance, the rod Jock .icon might be decomposed into a number of graphical primitives. The resulting logic specifications can be exploited to drive prototype implementations using Prelog: dimension(rod Jock Jcon, 0.02,0.02).

(11)

pattern(rod Jock Jcon, blank).

(12)

line(rod Jock Jcon, 0.2,0.3,0.4,0.33).

(13)

This states that the rod Jock Jcon is 0.02 by 0.02 units in dimension; these units are calculated relative to the size of the low .power .warning display. The background of the image is blank. Its image includes a line from (0.2,0.3) to (0.4, 0.33). Prelog is being developed in two directions. The first relies upon the use of probabilistic data to simulate the behaviour of stochastic applications. For instance, operator commands to start pumping will not always be successful if pumps occasionally fail. Our approach is appropriate because temporal frequencies lie at the heart of reliability analysis. Failure rates can be explicitly represented using extensions to existing temporal logics. Figures from Associated Octel predict 76 failures per operating year for compressors (Davies, 1985). Designers can specify that the probability of a particular compressor failing in the next second is 0.964 x 10~ 6 : failure.rate(compressor

.1,0.000000964)

(14)

The probability of compressor 1 failing in the next second is 0.964 x 10~6. Such failure rates have important implications for interface development. Wickens argues that users devise polling strategies to exploit the probabilistic behaviour of their system (Wickens, 1984). Less attention will be allocated to the observation of low probability errors, greater attention is allocated to the detection of high probability errors. Prelog has already been enhanced so that designers can use probabilistic clauses to simulate the mix of high and low probability failures which characterise accidents in complex systems (Johnson, 1993c). In our example, the probability of compressor .1 failing, 0.964 x 10~ 8 , is compared with a random number in the range [0.0, 1.0]. If the random number is less than or equal to the probability then the fact is assumed to be true; the pump fails. If the random number is greater than the probability then the fact is assumed to be false; the pump does not fail. Current work is exploring means of introducing utility measures into our system. Low probability, high cost failures, typically, have a greater impact upon safety than high probability, low cost errors.

294

Chris Johnson

The second area of research into the simulation of interactive, safety-critical systems focuses upon the introduction of environmental information. Accidents often occur because important information is hidden from the normal line of sight of system operators (Bignell & Fortune, 1991). The design of effective displays is of little value if users cannot see the devices which present them. Logic can be used to represent the position of a control panel within a working environment. Prelog is being extended so that three dimensional models of control rooms can be animated to show the layout of potential displays and input devices: location(feedwater .panel, 6.7,8.8). dimension(feedwater .panel, 1.5,0.5,1.5). present(low .power .warning, feedwater .panel)

(15) (16) (17)

The feedwater control panel is located at (6.7, 8.8) and has dimensions 1.5 by 0.5 by 1.5 meters and presents the low.power-warning display. This information can be used to reason about the allocation of data to presentation devices. High priority information and routinely monitored displays must be placed in locations that fall within the normal visual angle of system operators. Less important information may be placed on peripheral presentation devices. It is important to explicitly represent this allocation because these details are often neglected. For instance, the Electric Power Research Institute's review of nuclear power control rooms found meters that could not be read from normal working positions (Seminara, Gonzalez & Parsons, 1977). Critical displays were presented on the reverse of panels whose primary displays were devoted to non-critical information. Meters were obscured by the normal working positions of other operators. 7. Conclusion This paper has argued that formal notations provide a precise means of representing the events that lead to accidents involving interactive systems. Temporal logics can be used to avoid some of the ambiguity that arises in natural language descriptions of interaction. A further advantage is that the same notation provides a means of specifying design requirements which are intended to preserve the safety of similar systems. Finally, executable subsets of the logic notation can be used to derive partial implementations of potential interfaces. These simulations can be shown to other investigators. They can be used to encourage agreement on the course of events prior to more detailed investigation. They can also be used to assess the utility of potential changes to human-machine interfaces. Previous sections have not attempted to draw out general principles from the events leading to the Chernobyl accident. This is justified because accident reports are, typically, pitched at a relatively low level of detail. Other authors have used formal methods to describe generic techniques that can guide the development of many different interfaces (Dix, 1991; Thimbleby, 1990). Further work intends to apply this approach to a number of accident reports. For instance, substitution error was a contributory factor to both the Keg worth (AAIB, 1990) and Three Mile Island (Kirkpatrick & Mallory, 1981) disasters. Further work needs to provide methodological support for the notations and tools described in this paper. The translation of accident observations into logic clauses and the derivation of requirements relies heavily upon the skills of the analyst. As noted in Section 5, it is perfectly possible to produce specifications that threaten the safety of an application. It has


295

been argued that the development of simulations can help to identify such problems. One means of doing this might be to get analysts to "think aloud" as a Prelog prototype simulates the events leading to an accident. This would build upon the 'story boarding' techniques that are already widely used during accident investigations. Section 1 emphasised that our approach is in the early stages of development. It can be argued that the logic only provides a syntactic description. It does not capture the causes of particular traces of interaction. For instance, the formal notation has not been used to develop cognitive models of system operators. It would take a significant amount of work to resolve such limitations. For instance, Rouse reviews a number of techniques which have been used to model system operators (Rouse, 1981). Few, if any, of these can be applied to represent and reason about group cognition in multi-user applications. This is a research area in its own right. In anticipation of the products of such work, the scope of our approach has been more limited. The complexity of interactive, safety-critical systems has created an urgent requirement for pragmatic techniques that can represent the complex failures which lead to loss of life and environmental destruction.

Acknowledgements This work was built upon initial research conducted into Petri net representations of aircraft accidents with Dr John McCarthy, Department of Applied Psychology, University College Cork, Ireland and Dr Peter Wright, British Aerospace's Dependable Computing Systems Centre, University of York, York, United Kingdom. Thanks must also go to the other members of the Human Computer Interaction Group, University of York. This work is supported in part by Science and Engineering Research Council grant GR/J07686.

References AAIB (1989), "Report on the Incident Involving BAC 1-11 G-AYWB and Boeing 737 EI-BTZ on 12 April 1988 at Gatwick Airport", Her Majesty's Stationery Office, 2/89, Air Accidents Investigations Branch, Department of Transport. AAIB (1990), "Report on the Accident to Boeing 737-400 G-OBME Near Kegworth, Leicestershire on 8th January 1989", Her Majesty's Stationery Office, 4/90, Air Accidents Investigations Branch, Department of Transport. Bignell, V & Fortune, J (1991), Understanding System Failure, Manchester University Press. Cruickshank, A (1993), "Chernobyl Stays On-line", ATOM, p.2. Davies, K R (1985), The Assessment and Control of Major Hazards, Pergamon Press. Dix, A J (1991), Formal Methods for Interactive Systems, Academic Press. Duncan, K D (1987), New Technology and Human Error, John Wiley & Sons. Dynan, J & Francis, A (1993), "Smolensk Gets GOMIS", ATOM, pp.28-29. Ellis, C & Gibbs, E (1989), "Concurrency Control in Group Systems", ACM SIGMOD Record 18 (2), pp.399407.

296

Chris Johnson

Fujita, M, Kono, S, Tanaka, H & Moto-Aka, T(1986), "Tbkio: Logic Programming based on Temporal Logic and its Compilation to Prolog", in Proceedings of the Third International Conference on Logic Programming, E Shapiro [ed.], LNCS #225, Springer-Verlag, pp.695-7O8. Gamble, M, Taylor, C R (1991), "The CCSDS Protocol Validation Programme: Inter-Agency Testing using LOTOS", in Formal Description Techniques III, J Quemada & J Marias & Vazquez, E [eds.], NorthHolland, pp.319-326. Johnson, C W (1991), "Applying Temporal Logic to Support the Specification and Prototyping of Concurrent Multi-user Interfaces", in People and Computers VI: Usability Now!, D Diaper & N Hammond [eds.], Cambridge University Press, pp. 145-156, Proceedings ofHCI'91. Johnson, C W (1993a), "A Formal Approach to the Presentation of CSCW Systems", in People and Computers VIII, J L Alty, D Diaper & S Guest [eds.], Cambridge University Press, pp.335-352, Proceedings of HCI'93. Johnson, C W (1993b), "A Probabilistic Logic for the Development of Safety-Critical Interactive Systems", International Journal of Man-Machine Studies 39(2), pp.333—351. Johnson, C W (1993c), "Specifying And Prototyping Dynamic Human-Computer Interfaces For Stochastic Applications", in People and Computers VIII, J L Alty, D Diaper & S Guest [eds.], Cambridge University Press, pp.233-248, Proceedings of HCI'93. Johnson, C W & Harrison, M D (1992), "Using Temporal Logic to Support the Specification and Prototyping of Interactive Control Systems", International Journal of Man-Machine Studies 36, pp.357-385. Johnson, C W, McCarthy, J C & Wright, P C (1994), "Using Petri Nets to Support Natural Language in Accident Reports", Ergonomics, (to appear). Kirkpatrick, M & Mallory, K (1981), "Substitution Error Potential in Nuclear Power Plant Control Rooms", in Proceedings of the 25th Annual Meeting of the Human Factors Society, R C Sugarman [ed.], Human Factors Society, pp.163-167. Kletz, T A (1985), What Went Wrong? Case Histories of Process Plant Disasters, Gulf. Kuhmann, W (1989), "Stress Inducing Properties of System Response Times", Ergonomics 32 (3), pp.271-280. MOD (1991), "Requirements for the Procurement of Safety Critical Software", Ministry of Defence, MOD DEF-STAN 0055. Manna, Z & Pnueli, A (1982), "Verification Of Concurrent Programs: Temporal Proof Principles", in Proceedings of Logic of Programs 1981, D Kozen [ed.], LNCS 131, Springer-Verlag, pp.200-252. Morrey, M, Brown, J, Williams, J, Crick, M, Simmonds, J & Hill, M (1986),A Preliminary Assessment of the Radiological Impact of the Chernobyl Reactor Accident on the Population of the European Community, National Radiological Protection Board, CEC Contract 96-398. Moszkowski, B (1986), Executing Temporal Logic Programs, Cambridge University Press. NASA (1989), Advanced Orbiting Systems —Architectural Specification For The CCSDS Secretariat, National Aeronautic and Space Administration. Persson, C, Rodhe, H & de Geer, L (1986), "The Chernobyl Accident — A Meteorological Analysis of How Radionuclides Reached and Were Depositied In Sweden", Ambio 1 (16). Pew, R M, Miller, D C & Feehrer, C E (1981), "Evaluating Nuclear Control Room Improvements Through Analysis of Critical Operator Decisions", in Proceedings of the 25th Annual Meeting of the Human Factors Society, R C Sugarman [ed.], Human Factors Society, pp.100-104.


297

Potter, P (1988), "The Design of the Chernobyl Unit 4 Reactor", in The Chernobyl Accident and its Implications for the United Kingdom—Report Number 19 of the Watt Committee on Energy, N Worley & J Lewins [eds.], Elsevier Science (North-Holland), pp.9-18. Reason, J (1990), Human Error, Cambridge University Press. Rouse, W B (1981), "Human-Computer Interaction in the Control of Dynamic Systems", ACM Computing Surveys 13(1). Seminara, J L, Gonzalez, W R & Parsons, S O (1977), "Human Factors Review of Nuclear Power Plant Control Room Design", Electronic Power Research Institute and Lockheed Missiles And Space Company, RI-NP-1977-309. Thimbleby, H W (1990), User Interface Design, Addison Wesley. Thimbleby, H W (1993), "Combining Systems and Manuals", in People and Computers VIII, J L Alty, D Diaper & S Guest [eds.], Cambridge University Press, pp.479-488, Proceedings of HCI'93. Took, R (1991), "Integrating Inheritance and Composition in an Objective Presentation Model for Multiple Media", in Proceedings of Eurographics'91, F H Post & W Barth [eds.], Elsevier Science (NorthHolland), pp.291-303. USSR SCUAE (1986), The Accident at the Chernobyl Nuclear Power Plant and its Consequence, International Atomic Energy Authority, USSR State Committee on the Utilisation of Atomic Energy. Wickens, C D (1984), Engineering Psychology andHuman Performance, C E Merrill Publishing Company. Worley, N & Lewins, J [eds.] (1988), The Chernobyl Accident and its Implications For The United Kingdom — Report Number 19 of the Watt Committee on Energy, Elsevier Science (North-Holland). Wright, P, Fields, B & Harrison, M (1994), "Deriving Human-error Tolerance Requirements from Task Analysis", in Proceedings of the International Conference on Requirements Engineering, IEEE Publications.

Notations and Tools for Design

XUAN: Enhancing UAN to Capture Temporal Relationships among Actions Phil Gray, David England & Steve McGowan GIST (Glasgow Interactive Systems cenTre), Department of Computing Science, University of Glasgow, 17 Lilybank Gardens, Hillhead, Glasgow G12 8QQ, UK. Tel: +44(0)41 3304933 Fax: [email protected] Time is one of the most vital properties of an interface from a user's point of view, and the TAU project aims to explore how temporal properties of user interfaces affect their usability. This paper describes the XUAN notation of the specification of temporal behaviour. This notation also provides the basis for a software tool allowing not only specification but also rapid instantiation and modification of (small) user interfaces with defined temporal behaviour. This in turn will support rapid experimentation on users that varies temporal aspects on interfaces. In this paper we describe the features we have added to the UAN in creating XUAN in order to express temporal properties of tasks. Keywords: task description language, response time, specification.

1. Introduction Time is one of the most vital properties of an interface from a user's point of view but an aspect of interaction that is neglected by HCI theorists and practitioners. Work by Teal & Rudnicky (1992) has shown that users change their interaction strategies in response to varying response delays. This change in strategy is not accounted for in Norman's theory of action (Norman, 1986) or GOMS (Card, Moran & Newell, 1983). The use of multimedia systems and CSCW systems will mean that people will be faced increasingly with time-varying interactions. Our work in the TAU project provides an experimental basis for exploring issues of time in complex interactions. Informally we know that if mouse tracking is too slow, using the mouse becomes almost impossible. At a somewhat longer time scale, if a machine makes it quick to try things out, users will experiment rather than think carefully, and conversely slow responses mean that learning by experiment ceases to be practicable and the documentation must be much

302

Phil Gray, David England & Steve McGowan

better at allowing first time success. With the advent of faster, cheaper processors and of parallelism, ever faster responses can be achieved. When and in what way is this important to users? Previous research findings on the relationship between temporal aspects of interaction and usability still leave many design-significant questions unanswered. The corpus of empirical investigations in this area needs to be increased considerably, particularly with respect to the quantitative durations of system and user actions on which the above examples depend. The whole idea of interaction is in fact crucially dependent on time, since firstly it depends on sequence (alternating actions by two parties), and secondly it breaks down if the size of response times is too great. The key principle may be that people always try to schedule their own mental activity to keep themselves (their minds) occupied. If machine response times lengthen, people will cease to ask for optional information (cease to interact) and try to act without it, even though this is more mental work, and will be more error prone. For instance, they will type ahead if the system echo is delayed, even though sometimes a slip will mean long strings have to be cancelled. Next, they will do, or at least think about, other things while waiting; this means they are more likely to have forgotten crucial items (such as what they were doing, and what state the interaction is in) when the machine finally responds. However, some multi-tasking by people is generally forced on them externally, e.g. the phone rings. In summary then, response time should be kept below some threshold in absolute terms (i.e. so no subjective waiting time), but interleaving and suspension of human tasks must be expected and supported to some extent in any case. Early work by Miller (1968) raised some of the issues which are still crucial for interface designers. Published guidelines (Brown, 1988; Smith & Mosier, 1986) offer some help in considering general temporal issues such as response time. They offer no help in the detailed modelling of a particular application and it is often the particular in HCI that provides the key to usability. For example, both sets of guidelines mentioned reference MIL-STD-147C for the recommended response times to various user actions. However, as the standard makes clear, these times are highly application dependent. One of the few attempts to build software addressing the issue is the "cognitive co-processor" of Card, Robertson & Mackinlay (1991). What formal work there is has tended to concentrate on sequence, for example (Allen, 1984; Hartson & Gray, 1992), and not on the quantitative durations on which the above examples depend. Johnson (1991) shows how temporal logic is superior to first order logic in the specification and prototyping of concurrent interaction, though it presents a system-level view of interaction. UAN is quantitative, but addresses execution times for an expert user and a fixed strategy — not the response time of the machine, nor its effect on the user's cognitive strategy and decision processes, still less on the user's attitude and effective reaction. If the Teal model is correct — that users vary their strategies with the system's temporal behaviour — then there are implications for the precision of UAN. That is, predictions based on fixed interaction strategies will be inaccurate when users are forced to vary strategies. Thus a system which has a consistent logical behaviour but varying temporal behaviour will produce variations in user behaviour. We cannot, therefore, make claims about the usability of a system without examining its temporal behaviour. However, we are here not so much interested in predicting user times although Allen & Scerbo (1983) claim that in practice UAN only came within a factor of 2 as in measuring how they vary as a function of changes in machine response times.

XUAN: Enhancing UAN to Capture Temporal Relationships among Actions303 The Temporal Aspects of Usability (TAU) project aims to support investigation of temporal issues in interaction. There are two main phases to TAU: the development of a notation XUAN (extended User Action Notation) to express temporal properties of interaction, and secondly, the use of XUAN-specified tasks as a means of conducting small-scale experiments testing hypotheses about the relationship between time and interaction. This paper addresses the former phase and describes XUAN which is an extension of the User Action Notation (Hartson, Siochi & Hix, 1990). The extensions described below form the basis for a more precise, executable task description language and extend the range of actual interactive behaviours that can be described. The executable UAN (a subset of extended UAN) is the subject of a further report (McGowan, Gray & England, 1994). One of the main problems with the existing User Action Notation (hereafter referred to simply as UAN in italics) was that the temporal relationships within and across tasks were not stated explicitly, and were represented loosely by the ordering of tasks, feedback and system operations within the UAN's tabular task diagrams. Hence, temporal relationships were often represented only implicitly. Furthermore, temporal relationships within UAN concentrated primarily on user actions; thus neglecting the temporal aspects of task categories not considered user actions. Coutaz et al. (1993) provide further criticisms of UAN, particularly in the implicit expression of temporal relations between tasks and the lack of control structures in the notation. XUAN largely answers these criticisms. XUAN attempts to resolve the above limitations by partitioning a task description into separate sections that specify task details such as local and global variables, pre- and post-conditions, constituent sub-tasks and, most importantly perhaps, a component that describes the complete temporal relationships among the specified sub-tasks. It is in this explicit partitioning of all task aspects that XUAN improves upon UAN, particularly regarding temporal relationships both within and between tasks.

2. Tasks and Actions In the existing UAN, tasks consist of a set of temporally-related user actions. The temporal ordering among elements in the action set is specified in the UAN action language. For example, if Task T contains the action set { a l , a 2 } , the relationship of strict sequence would be expressed by: al,

a2 (usually separated by a newline in UAN)

Order independent execution of the set (i.e. all must be executed, but in any order) is shown with the operator '&': a l & a2 A full description of the action language can be found in (Hartson & Gray, 1992) but briefly the main temporal relations are: , & I| ->

Strict sequence Order independence Concurrent with Interruptible by Mutually interruptible Mutually Interleavable

(introduced in XUAN) (as indicated by direction of arrow)

304

Phil Gray, David England & Steve McGowan Task: click button user actions

feedback

move to button

cursor tracks

mouse bttn down

button highlited

mouse bttn up

button unhilited

dialogue state/ application action

execute bttn action

Figure 1: UAN task description.

User actions are either primitive actions, typically manipulations of physical input devices (pressing a key, moving a mouse), or tasks. Additionally, and optionally, a UAN action specification may be annotated with information about system feedback (viz., perceivable changes in system state), (non-perceivable) changes to user interface state and application-significant operations. In the following examples, a* means one or more repetitions of t a s k a. Syntactically, a UAN specification places its user actions in a vertically organised list, with annotations in columns to the right. Thus, consider a specification of clicking a typical screen button widget as shown in Figure 1. This approach, although possessing the virtues of simplicity and readability, is overly restrictive in a number of important ways. In particular, it makes it difficult to specify complex temporal relationships between user actions and system activity (the annotation device assumes that each feedback element (immediately) follows the action to which it is attached). Further examples of the difficulties forced by these restrictions are observed by (Coutaz et al., 1993). In XUAN, we have chosen to solve this problem by expanding the language, roughly similar in approach to Shneiderman's multi-party grammar (Shneiderman, 1982), so that system actions are fully-Hedged constituents of a task. That is, tasks are viewed as collaborative enterprises involving one or more people ('users') and one or more computer systems. Thus, a XUAN task is, like a UAN task, a set of the actions comprising the task, separated out into columns (or types) for some communicative purpose. The UAN tabular layout is used as one way of writing a XUAN description (see below for an alternative method). The left/right-top/down ordering of the table elements may be viewed as a partial representation of temporal ordering on actions and the columnar organisation as modelling the task agents. Figure 2 shows a tabular XUAN description of the same task as above. One consequence of treating user and system function similarly in the task description is that the somewhat ad hoc nature of system annotations of user actions is replaced by a symmetrical treatment of each task agent. Both perform visible actions and (hypothesised) invisible internal actions. These internal actions for the system have already been considered (dialogue state, application action); those for the user are similar (changes to STM, LTM, planning and other cognitive actions). From this stand-point XUAN resembles Memory-Cognition-Action tables (Sharratt, 1990). In fact, it has previously been suggested that the UAN could be so enhanced. While we believe that XUAN can in principle accommodate this development in a more

XUAN: Enhancing UAN to Capture Temporal Relationships among Actions305 User

Computer-based Agent

c

internal actions

Screen 5B-1 or Screen 4 B - -> Screen El-em3 -—> Screen 4 B - -> Screen 5B-1 if an input error has occurred, i.e. if the 'show user list' button was not selected (see Figure 9);

322

Kee Yong him & John Long

File

Edit

View

Special

Window atabase

Screen object Sec/Pd Folder etc.

Description

Design Attributes

Contains host and user security reports sent to the NMgr. Folder behaviour is a variant of Macintosh folders.

Detailed further by interface model descriptions (see Figure 11).

etc.

etc.

Figure 10a. Pictorial screen layout specification of Screen 3A

Macintosh User Interface Environment

Security Action Selection Menu OJ

Search Connections

Show Access Points

Screen Object

Description

(f

Select

J

Design Attributes

File (menu bar)

Offers 'Open' and 'Quit' menu items. 'Open' Behaviour as per standard allows the network manager to open host and Macintosh menu items. user reports. 'Quit' allows the manager to quit the security application.

Security Action Selection Menu

Allows the network manager to select an appropriate action, namely 'Search Connections' (for details of network connections), 'Show User etc.

Figure 10b. Pictorial screen layout specification of Screen 4B

Figure 10: Partial set of pictorial screen layout specifications for network security management management.

c. screen design and behaviour of screen objects. Figure 10 augments the specifications in Figures 8 and 9 with pictorial specifications of computer display screens, e.g. Screens 3A and 4B. The pictures may either be drawn on paper (to-scale or dimensioned), or prototyped using a computer-based tool. Additional information on the screen contents is provided in an accompanying table, and the behaviour of individual screen objects are described using structured diagrams (see Figure 11);

Structured Notations to Support Human Factors Specification . . .

323

Sec/Pd folder 1

Create folder

Delete folder

Folder life body Folder life-cycle

Fill folder action body

Activate alarm

Other folder action body

1 Receive file from Sec/IN folder

1 Other folder action cycle

Figure 11: Part of an interface model (Sec/Pd folder) for network security management.

Message number

Message

eml

Sorry, your log-on inputs are incorrect. Your session will be terminated.

em2

Please indicate a host and/or user report action by selecting either the 'Delete' or 'Pending' radio button.

em3

Please indicate the required security action by selecting a radio button from the 'Security Action Selection Menu'. Do this BEFORE clicking the 'Select' button.

em4

Please select a user name

etc.

Figure 12: Part of an error message index for network security management.

d. error, feedback and help messages. Figure 12 also augments the specifications in Figures 8 and 9 with a table of error messages. For instance, Screen El-em3 in the latter figures implies a display screen of pictorial format El (not shown) and an error message ID number 3. The content of the message may then be determined from Figure 12. Interactive task scenarios may thus be designed and documented appropriately using the above notations. An example for the case-study follows. Having been alerted to a security violation, the network manager is required to access the computer database to gather information on the network user involved. To this end, the manager double-clicks the security icon in Screen 3A to activate the network security application (see Figure 10a). The input 'consumes' Screen 3A and triggers Screen 4B (see Figure 8). Thus, a menu offering a selection of three actions is presented to the network manager, namely 'Search Connection', 'Show User List' and 'Show Access Points' (see Figure 10b). The manager then clicks one of the three radio buttons to indicate the desired selection and confirms the input by clicking the 'Select' button (see Figure 8). If a radio button was not selected prior to clicking the 'Select' button, an error message screen, namely Screen El-em3 is activated (see Figures 8, 9 and 12). However, if the inputs were made

324

Kee Yong Lim & John Long

correctly, Screen 4B would be 'consumed' and Screen 5B-1 presented (see Figures 8 and 9). Using the latter screen (not shown pictorially), the network manager may then specify what information on the network user should be extracted from the database and displayed. On the basis of the information, the manager would then decide whether the user should be contacted to establish possible causes of the security violation (see Figure 8). Appropriate responses to the violation are thus determined, e.g. if the event is due to a password mis-key, the manager may restore the user's account using other computer functions (not shown). This account completes the case-study illustration. Using these notations and documentation schemes, human factors design concerns may be specified comprehensively at various stages of system development. Consequently, a greater uptake of its contributions may be expected.

3. Conclusion The paper illustrates how structured notations can be used to support wider human factors involvement in system design. In particular, the notations may improve the effectiveness of human factors involvement in three ways, namely: a. by supporting more complete and detailed human factors specifications at various stages of system design. Since the specifications have been set explicitly within the system development context, human factors contributions may be incorporated more efficiently by software engineers; b. by supporting the elicitation of user feedback. In particular, the improvements in human factors specifications outlined in (a) above, would support more comprehensive user feedback throughout system design. Feedback elicitation is also facilitated by the graphical nature of structured notations since they are understood better by users. Thus, invalid design assumptions may be identified earlier and eliminated more effectively; c. by exploiting the benefits of a common notation. Thus, by recruiting structured notations from existing software engineering methods, human factors design specifications would be understood better by software engineers. For the above reasons, the uptake of human factors contributions would improve. Additional improvements would also accrue if an explicit human factors design process were established, e.g. via a structured human factors method (Lim & Long, 1992a; Lim & Long, 1992c; Lim, Long & Silcock, 1992). Such a method could also be integrated with similarly structured software engineering methods. Thus, the scope, format, granularity and timing of human factors contributions may be configured appropriately to support a more effective uptake.

Acknowledgements Part of the research was carried out for the Procurement Executive, Ministry of Defence (RARDE, UK). Views expressed in the paper are those of the authors and should not be attributed to the Ministry of Defence.

Structured Notations to Support Human Factors Specification ...

325

References Blyth, R C & Hakiel, S R(1989), "A User Interface Design Methodology and the Implication for Structured System Design Methods", in Proceedings of the IEE Conference on Command, Control, Communications and Management Information Systems. Brooks, R (1991), "Comparative Task Analysis: An Alternative Direction for HCI Science", in Designing Interaction: Psychology at the Human-Computer Interface, J M Carroll [ed.], Cambridge University Press. Carver, M K (1988), "Practical Experience of Specifying the Human-Computer Interface using JSD", in Proceedings of the Ergonomics Society 1988 Conference, pp. 177-182. Carver, M K & Cameron, J (1987), The JSD Method: A Framework for the Specification of the Human-Computer Interface, LBMS Ltd. Damodaran, L, Ip, K & Beck, M (1988), "Integrating Human Factors Principles into a Structured Design Methodology: A Case-study in the UK Civil Service", in Proceedings of the First European Conference on Information Technology For Organisational Systems, Eurinfo'88, H-J Bullinger, E N Protonotarios, D Bouwhuis & F Reim [eds.], North-Holland. Finkelstein, A & Potts, C (1985), "Evaluation of Existing Requirements Extraction Strategies", FOREST Project Report Rl. Fitter, M & Green, T R G (1979), "When Do Diagrams Make Good Computer Languages?", International Journal of Man-Machine Studies 11, pp.235-261. Hares, J (1987), "Methods for a Longer Life", Computer News/Databases, p. 18, August 6. Lim, K Y & Long, J B (1992a), "A Method for (Recruiting) Methods: Facilitating Human Factors Input to System Design", in Proceedings of CHI'92: Human Factors in Computing Systems, P Bauersfeld, J Bennett & G Lynch [eds.], ACM Press. Lim, K Y & Long, J B (1992b), "Computer-based Tools for a Structured Human Factors Method", in Proceedings of Conference on Computer-Aided Ergonomics and Safety. Lim, K Y & Long, J B (1992c), "Rapid Prototyping, Structured Methods and the Incorporation of Human Factors in System Design", in Proceedings ofEast-West International Conference on Human-Computer Interaction, J Gornostaev [ed.], ICSTI. Lim, K Y & Long, J B (1994), "The MUSE Method for Usability Engineering", (to be published by Cambridge University Press). Lim, K Y, Long, J B & Silcock, N (1990a), "Motivation, Research Management and a Conception for Structured Integration of Human Factors with System Development Methods: An Illustration Using the JSD Method", in Proceedings of the European Conference on Cognitive Ergonomics, pp.359-374. Lim, K Y, Long, J B & Silcock, N (1990b), "Requirements, Research and Strategy for Integrating Human Factors with Structured Analysis and Design Methods: The Case of the JSD method", in Proceedings of the Ergonomics Society 1990 Conference, pp.32-38. Lim, K Y, Long, J B & Silcock, N (1992), "Integrating Human Factors with the JSD Method: An Illustrated Overview", Ergonomics 35 (10), pp.1135-1161. Sutcliffe, A (1988), "Some Experiences in Integrating Specification of HCI within a Structured System Development Method", in People and Computers IV, D M Jones & R Winder [eds.], Cambridge University Press, Proceedings of HCI'88.

326

Kee Yong him & John Long

Sutcliffe, A (1989), "Task Analysis, Systems Analysis and Design: Symbiosis or Synthesis?", Interacting with Computers 1 (1), pp.6—12. Whiteside, J, Jones, S, Levy, P S & Wixon, D (1985), "User Performance with Command, Menu and Iconic Interfaces", in Human Factors of Computing Systems II, Borman & Curtis [eds.], Elsevier Science (North-Holland).

Discount Dialogue Modelling with Action Simulator Andrew F Monk* & Martin B Curry* * Department of Psychology, University of York, Heslington, York YO1 5DD, UK. EMail: [email protected] * Sowerby Research Centre, British Aerospace pic, FPC 267, Filton, Bristol BS12 7QW, UK. EMail: [email protected] A description of the high level structure of a user interface is an important part of any system specification. Currently the most common way of thinking about and recording this part of the design is through story boards and verbal descriptions, these may be imprecise and are difficult to evaluate. Action Simulator allows a designer to build simple models of the high level behaviour of the user interface. The models are easy to read and can be executed to give a dynamic view of the design. This makes it possible to 'run through' the actions needed to complete the users' work. A procedure for characterising the users' work that is suitable for this purpose is also sketched out in the paper. Action Simulator consists of an Excel spreadsheet and associated macros and is publicly available. Keywords: dialogue model, task model, work objective, decomposition, scenario, system behaviour, specification, spreadsheet.

1. The Need for Abstract Dialogue Models The design of software, like any other undertaking in engineering, involves the construction of a specification that includes models of various kinds. The reason engineers construct a blue print or specification before building the artefact itself is that the latter is difficult to change and so between gathering requirements and implementation a specification is built that is easy to change. Analysis and evaluation of the specification enables improvements to be made before implementation begins. Also like other engineering projects, software is extremely complex and so difficult to reason about. For this reason engineers build models that concentrate on some aspect of the design and abstract across others. For example, when thinking about the user interface designers may produce structured English descriptions of the top level

328

Andrew F Monk & Martin B Curry

dialogue or some kind of diagram indicating how different functions are accessed. These devices through which designers reason about human-computer dialogue can be thought of as abstract dialogue models. An abstract dialogue model is the part of a system specification that describes the high level behaviour of a user interface. We have chosen to concentrate on the higher level behaviour of the interface as this is a major difficulty for the designer using modern graphical user interfaces. To take an example, in the course of designing a Graphical User Interface (GUI) for an advanced photocopier one might think about the activity of specifying settings such as paper size, colour, toner density and the number of copies. A high level dialogue model is concerned with when the user can make changes to the different settings. It may be desirable to allow users to change the toner density and number of copies while the copying mechanism is running, but not colour or paper size. The abstract dialogue model largely determines the task fit of the design, that is, how well the system supports the users in doing their work. In contrast, a specification of the low level system behaviour would describe how the user achieves these settings e.g. what happens when the user clicks or types with the mouse cursor on a particular field. This level of design is equally important but has been made easier with the advent of style guides and their associated tool kits — e.g. (Apple Computer Inc, 1987; IBM, 1991). There has also been a lot of valuable research into how intelligent tools may further facilitate low level design — e.g. (Foley et al., 1991). The problem of designing the higher level structure of a user interface so that it fits the way users have to work has received much less attention. This paper describes a tool for 'discount' dialogue modelling. Too many of the techniques devised by academics for software engineers require such a large investment in training, and effort during use, that they unlikely to be justified in terms of the pay back they produce. There is a strong case to be made for discount procedures that have a low cost in training and use. When the cost is low the pay back is much more likely to justify it. This case has been made persuasively by Nielsen (1989) in the area of techniques for usability inspection but a similar case can be made for dialogue modelling. The next section describes some alternative notations for describing human computer dialogue. Section 3 describes Action Simulator, a publicly available spreadsheet with which a designer can specify and observe the behaviour of a dialogue model. Section 4 outlines how dialogue modelling might be integrated into the design process.

2. Alternative Notations for Specifying Human-Computer Dialogue At present a designer is most likely to record and reason about the dialogue model by drawing story boards or writing verbal descriptions. A story board sketches major features of screens and indicates how the screens change under different circumstances. This can be very effective but such informal descriptions have their problems. It is difficult for the designer to see the impact of a decision about one part of the design on the other parts. Also it is easy for a designer to miss important steps or get sucked into detailed low level design prematurely. Very informal notations of this kind make any kind of analysis difficult. In particular, it is difficult to check that the design is complete i.e. that with it a user can complete the work required. The commonest semi-formal notations used for thinking about human-computer dialogue are probably flow charts and state transition networks (STN). One will commonly see designers

Discount Dialogue Modelling with Action Simulator

//A3 or A4?

329

\\

Single or multiple copies?

Figure 1: STN for copier controls, sequential dialogue model, one choice at a time.

Figure 2: STN for copier controls, simultaneous access to multiple functions.

sketching an STN and some even get into the specification document. STNs have been used effectively to specify simple devices such as video recorders (Thimbleby, 1993) but they are not generally suitable for specifying modern graphical user interfaces. This is because they encourage the designer to use sequential dialogue structures and discourage multi-threaded and direct manipulation dialogue. An important characteristic of a GUI is that it provides the possibility of giving users accessibility to many functions simultaneously. A word processor is a good example of an application where users can choose between a large number of functions at all times. They can choose to change the font or style of a character, paragraph layout and so on. These functions are available throughout most of the dialogue the user has with the computer. This leads to what has been characterised as multi-threaded human-computer dialogue. Users may make some font changes some paragraph layout changes and then, on the basis of what they see, go back and do more font changes. Giving users access to multiple actions throughout the dialogue is generally considered to be a good thing as it allows flexibility in the way they carry out their work. Thus a good design

330


principle is to maximise it, except where there are strong external constraints on the order in which tasks are to be carried out. Describing this sort of human-computer dialogue with an STN is difficult. Consider again the problem of designing a user interface for an advanced photocopier. Let us say that the user can choose between two sizes of paper, two colours and between a single, or multiple, copies. If the user is forced to make these choices in a specific order then the dialogue can be easily represented as an STN (see Figure 1). The state of the system is 'Choice of tray' then 'Choice of colour' then 'Choice of copies'. In the kind of interface considered here these choices are normally simultaneously available and so it no longer makes sense to think about states corresponding to 'Choice of tray' etc. Here the actions we are concerned with are the actions that select a particular choice of paper tray or a particular choice of colour. The availability of these actions and the effect they have depends on a different kind of state, that is, the current state for a particular choice of options. One can only choose the A3 paper tray if the current choice is the A4 paper tray. This means that the STN must record all the eight possible states (A4, black, single copy; A3, black, single copy; and so on). There are a large number of possible transitions between these eight states and the STN soon gets out of hand. Figure 2 is just manageable but this example, with three binary choices has an unrealistically small state space. Normally there are many multiple, rather than binary, choices and the corresponding 'state explosion' is considerable. There are various ways of extending STNs to deal with this problem, one of the best known being State Charts (Harel, 1987). We have developed a textual notation using Prepositional Production Systems (Olsen, 1990) that we believe is equally easy to use — see (Curry & Monk, 1994a; Curry & Monk, 1994b). This is the basis of Action Simulator. STNs and PPSs have a sound mathematical basis and can be analysed formally — see (Olsen, Monk & Curry, 1994; Thimbleby, 1993). There have also been a number of attempts to apply formal notations from other areas of software design such as Z and CSP — see for example (Harrison & Thimbleby, 1990). Despite showing some promise, these formal notations and analysis techniques have yet to be encapsulated in such a way that they can be used easily by designers and so will not be considered further here. Finally, it will always be difficult to reason about a static description of dynamic behaviour. Of course a prototype allows the designer to observe the dynamic behaviour of the specification. However, most prototyping tools require that the low level design of screens is reasonably complete. As well as being impractical, adding the low level detail will often obscure the higher level behaviour. What is needed is a dynamic simulation of the design that does not require the designer to describe low level detail, this is what Action Simulator provides.

3. Action Simulator The approach is an extension of Monk's action-effect rules (Monk, 1990). A Prepositional Production System (PPS) dialogue model consists of a set of if-then rules where each rule specifies an action with pre- and post-conditions. A typical PPS rule is given below. Pre-condition: Ready(FALSE) Action and effect: GivelD - > {Display 'Ready'} Post-condition: Ready (TRUE)

GivelD is the action in this rule. This abstraction stands for the activities by which users identify themselves as valid users of the copier. At this stage in the process of design the precise way that the user should do this has yet to be specified, it might involve the insertion

331

Discount Dialogue Modelling with Action Simulator Conditions A4

Black

State

TRUE

TRUE

Request A3

TRUE

No. rules

6

No. conds

3

Single copy TRUE

FALSE Request A4

FALSE TRUE

Request Red

TRUE FALSE

Request Black

FALSE TRUE

Req. >1 copies

TRUE FALSE

Reset copies

FALSE TRUE

Figure 3: Action Simulator spreadsheet for the dialogue model expressed as an STN in Figure 2, simultaneous independent choices.

of some sort of electronic key or entering a password of some kind. The user action GivelD has a side effect that is to {Display 'Ready'}. Again precisely how the display will signal that the action has been successful has yet to be decided, the rule simply specifies that this should be signalled to the user. For this rule to be activated, and the action to have this side effect, the dialogue model has to be in a certain state, that is the condition ReadyO has to be FALSE. In addition to having the side effect of the action, activating the rule changes the state of the dialogue model. Changes are specified as post-conditions. Here the condition ReadyO is set to TRUE. Conditions appear as pre-conditions in some rules and post-conditions in others. Thus Ready(TRUE) will be a pre-condition in other rules that describe actions that can only be selected by a user when the dialogue model is in that state. Action Simulator re-casts the PPS notation in a tabular form. This makes it possible to implement the dialogue model as a spread sheet. In order to simplify the computation, so that it can be executed in a spreadsheet, the side effect is omitted and would be recorded elsewhere. It is then possible to express the rules as rows and columns, where the rows are actions and the columns conditions. More complex software tools can be envisaged, our objective here is to provide a simplified version of PPS to demonstrate our approach. Figure 3 shows how Action Simulator would express the dialogue model described using an STN in Figure 2. To create such a dialogue model the designer first lists all the actions

332


a user could take, here: 'request A3', 'request A4', 'request red', 'request black', 'request multiple copies', 'cancel multiple copies'. One then specifies the pre-conditions that govern when those actions are available, the first line in the PPS notation used above, and then the post-conditions, the final line in the PPS notation used above. In Figure 3 the top row lists the conditions (A4, Black and Single copy). Propositional Production Systems allow multiple value conditions but Action Simulator allows only binary conditions to simplify the computation required. Thus, the condition A4 is either true (A4 paper tray selected) or false (A3 paper tray selected). The current state of the dialogue model is indicated in the next row, that is the default starting state {true, true, true}. Each action has its pre-conditions specified in the row containing the action and its post-conditions specified in the row below that. Thus, Request A3 has the pre-condition A4-true and the post-condition A3-true. The other conditions are blank meaning that they can be true or false ('don't care'). This gives a PPS its representational power. In effect the pre-condition {true, blank , blank} specifies a region in the total state space consisting of four states. With more realistically sized examples with several multiple value conditions this is very efficient. The dialogue model specified in this way can be executed so that the designer can observe its behaviour. This ability to observe the dynamic behaviour of a static notation is very useful. Running the Process Rules macro builds a set of formulae to evaluate the pre-conditions of each rule against the current state of the dialogue model. The actions that are available, because their pre-conditions are satisfied, is indicated by a row of '*'s below the action. In Figure 3 they are: 'Request A3', 'Request Red', and 'Request > 1 copies'. The designer may then explore the dynamics of the model by selecting an 'available' action and running the macro 'Do Action'. This changes the current state according to the post-conditions for that action. For example, selecting Request A3 would change the condition A4 from true to false. The '*'s under 'Request A3' would then disappear as this action is no longer available. 'Request A4' is however available as its pre-conditions are now met and so it would have '*'s under it. Figure 3 is a very simple dialogue model where the functions, select paper tray, select colour and select number of copies are independently concurrent. Figure 4 presents a slightly more complex dialogue model for a copier. Readers might like to trace through the dialogue this affords. Notice, for example, that some actions can be performed when the copier is copying but others cannot. Notice also that some of the actions are available to the system/world rather than the user. '*Finished copying*' is an action coming from the copier mechanism signalling that the job is finished.

4. Positioning Abstract Dialogue Modelling in the Design Process As was explained in the introduction to this paper abstract dialogue models are intended to be used as a part of the specification for a new computer application. The specification is developed with and analysed against a requirements document. There are a number of methodologies for writing requirements. This section describes some ways this might be done in the context of a discount methodology. It takes features from existing methods and puts them together as a minimal configuration for the purpose of dialogue modelling. As such it should be easy to integrate with whatever techniques are currently used by an organisation at this stage of the design process.

333

Discount Dialogue Modelling with Action Simulator Conditions Ready State

FALSE

Give ID

FALSE

No. conds Copying FALSE

TRUE Cancel ID

4 One Copy

Normal Toner Density

TRUE

TRUE

TRUE

TRUE

TRUE FALSE

Req. >1 copies

TRUE

FALSE

TRUE FALSE

Reset Copies

TRUE

FALSE

FALSE TRUE

Darker

TRUE

TRUE FALSE

Cancel darker

TRUE

FALSE TRUE

Copy

TRUE

FALSE TRUE

'Finished copying'

TRUE

TRUE FALSE

Figure 4: Extended dialogue model for copier.

The level of abstraction of an abstract dialogue model is broadly compatible with a description of the user's job as might be found in the lower levels of a conventional hierarchical task analysis (HTA) (Annett et al., 1971). The problem is to devise a way of making such descriptions easy to generate and putting them into a form in which they can easily be used to guide the design of the abstract dialogue model. The procedure that we are exploring is outlined below — see also (Curry et al., 1993). Depicting the work context — The first view of user requirements should be a top level description of the work context. This serves to demonstrate that everyone who might be effected by the impact of the new system has been thought of. It should be written around a rich picture (Checkland, 1981). Some instructions for generating such work context are given in Table 1. We have found that this can be useful both as an aid to thought and a good way of introducing the design problem to the complete design team and other interested parties. The main output from this exercise is an understanding of who is going to interact with the system and the work the system is to support. In addition, the major responsibilities and concerns of these users, and others, is recorded. Having identified the potential users of the system one

334

Andrew F Monk & Martin B Curry A.

B.

C.

D.

E.

In the middle of a large sheet of paper, draw a figure of some kind to represent each of the kinds of people who will actually interact with the system. These are the user roles. There will probably be more than one, for example, there may be computer operators who enter data and managers who use summary displays. Add further figures for people who might be affected by the introduction of the system even though they don't have to operate the system themselves. These non-user roles may include, amongst others: supervisors, supervisees and peers of the user roles; also customers, management and the IT department who supply the relevant hardware and software services. Indicate the flow of work on the diagram with labelled arrows. For example a customer may make an enquiry which is entered into a database by an enquiry clerk and which then results in some action by a repairman. A supervisor makes summary reports using the same database and gives them in printed form to a manager who uses them in his reports to the board of directors. Indicate the major concerns of all the roles represented by writing a short phrase next to each e.g. management are generally concerned with cutting costs, operators with losing their jobs or being 'de-skilled'. On a separate sheet list each of the roles defined as drawings on the diagram and write for each a concise definition (not more than two or three sentences) of: i. Their responsibilities (people are responsible to someone for something). ii. Their work (this is simply an amplification of the work flow arrows, step C, for that role). iii. Their concerns (this is similarly an opportunity to explain and add to the concerns indicated in step D).

Each of the steps A to E will suggest changes and additions you want to make to what you recorded in the earlier steps, this is a good thing to do. You will almost certainly have to re-do the whole thing to make it presentable anyway. Table 1: Steps to be taken in recording the work context for a new application.

may go on to record their characteristics such as keyboard and mousing skills. Any HCI text book will provide a discussion of what is required — e.g.(Sutcliffe, 1988). The next step is to describe the tasks to be supported. The reason for generating these user-centred requirements is to obtain a description of the tasks to be supported that can be used to generate and evaluate an abstract dialogue model. The description of the work developed above is not suitable for this purpose because it is at a very high level of abstraction. For the rich picture this is appropriate because one is aiming for very broad scope. For the purpose of building an abstract dialogue model one needs to specify the work of the target user population in more detail but still to hold short of providing so much detail that one anticipates the design of the system. We suggest this is done in three parts: a work objectives decomposition (WOD), an exceptions list and illustrative scenarios. Work Objective Decomposition — Table 2 presents some instructions for generating a WOD. This is not unlike a conventional hierarchical task analysis (HTA). Generating an exceptions list — The WOD represents an ideal world without interruptions or problems. By listing possible exceptions to this one can anticipate some of the pitfalls that may make a design unworkable. Table 3. presents some instructions for creating an exceptions list. Most important, for these user centred requirements, are the user exceptions due to mistakes and interruptions.


335

You will need to create a WOD for each user role. A work objective describes a state of the world the user would like to achieve. For example, a secretary might have three top level objectives: 'Phone enquiries dealt with', 'Notices typed and distributed', 'Reports typed and distributed'. These top level objectives come from the rich picture. Next decompose these objectives into sub-objectives. It is only necessary to do this where the objective is to be supported by the planned computer system. For example, let us say that we are designing a user interface for an advanced photocopier. We have identified that these secretaries have no need for copying to achieve the objective 'Phone enquiries dealt with' and so this work objective is not decomposed. The same process may be repeated once more to decompose sub-objectives into sub-sub-objectives, if necessary. WOD for Secretary as user of copier 1. Phone enquiries dealt with 2. Notice distributed (staff distribution list) 2.1. Have notice from boss 2.2. Notice typed and printed 2.3. Have copies of notice for distribution 2.3.1. At copier with notice 2.3.2. ID accepted by copier 2.3.3. No. copies etc. specified 2.3.4. Copying started 2.3.5. Copies retrieved 2.3.6. Notice retrieved 2.4. Copies of notice in mail 2.5. Notice filed 3. Report typed and distributed An example is given above, note that: i. While these objectives generally correspond to processes i.e. 'Have notice from boss' corresponds to 'Get notice from Boss' we have found it makes it easier to think clearly if one always expresses the objective in terms of the state to be achieved. ii. While the objectives and sub-objectives are numbered for reference this does not imply they have to be carried out in a specific order. You should not consider the order in which things are 'normally' done or logical constraints on the order objectives have to be achieved at this stage as it may impeded creative solutions at the design stage. iii. Avoid deep hierarchical structures. You will often only need to go to sub-objectives, it should not be necessary to go further than sub-sub-objectives. iv. Notice that the objectives are still fairly abstract, 'ID accepted by photocopier' commits one to having some kind of login procedure but stops short of specifying how this will work. This should only be considered at the design stage. Table 2: Instructions for creating a WOD.

Writing scenarios of typical work — A scenario is a fictional, but nonetheless typical, story describing a user's work. These stories flesh out a WOD and exception list by including sequences of actions. They contain examples of typical data associated with real use. For example, an abstract description of the users' behaviour from the WOD 'No. copies etc. specified' might be fleshed out to 'Jean requests 15 copies to be made on the darker setting, using red ink on A3 size paper'. The stories should highlight crucial sections of the users' tasks. In addition they convey information about the context in which the user's work is carried out. It is useful to include sample documents and other data with a scenario e.g. photocopies of delivery notes and invoices, printout from other systems and so on. Wherever

336

Andrew F Monk & Martin B Curry A.

List the system exceptions, these will include breakdowns and 'correct' behaviour (e.g. an unacceptable ID when the user is logging in to a system). Work out where these exceptions could occur i.e. which objectives might the user be working on when they occur. B. List user exceptions due to the users making mistakes or changing their minds. Take each sub-objectives to be computer supported in turn. Ask yourself whether the user having achieved that objective might want to 'undo' it at some later stage and if so where such a decision is likely to be taken, i.e. which objectives might the user be working on at that point. C. List user exceptions due to interruptions. Most people interleave several tasks in their daily work. Ask yourself when users could be interrupted and what implications this could have for design. The higher level work objectives and the rich picture will suggest what these interruptions are likely to be. Again work out where in the WOD these interruptions could occur. Some examples for the photocopier design are given below. System exception: [£/] Incorrect-id (at 2.3.2) [£2] Paper jam (at 2.3.4, 2.3.5) [E3] Out of toner (at 2.3.4, 2.3.5) [E4] Out of paper (at 2.3.4, 2.3.5) User exception: [E5] User decides number of copies inappropriate or copies too light (2.3.3, 2.3.4) [E6] User interrupted by phone (at any point) Table 3: Instructions for creating an exceptions list.

possible these should be real documents and data. If real data is not available one can make something up and show it to real users and ask how it could be made more realistic. Even if it is not possible to get real users just making something up will serve to clarify one's ideas. Scenarios are generated from the WOD and the exceptions list. A good procedure is to start by writing one or two best case scenarios using the WOD, one can then elaborate these into further scenarios by adding exceptions. If carefully written the scenarios can encapsulate most of the understanding of the user's work captured by this user centred requirements exercise. Generating the abstract dialogue model — we recommend that the abstract dialogue model be developed iteratively and incrementally starting with some central task. Some rules are written in Action Recorder to represent how the system will support this central task. These rules are then executed and the behaviour of the model evaluated against the relevant parts of some of the scenarios. This will highlight points where the model could be improved either because dialogue cannot support the work or does so in an awkward fashion. When these 'bugs' have been sorted out the dialogue model can be more exhaustively evaluated against the relevant parts of the WOD and the exceptions list, again this is done by executing the Action Recorder model and playing through the required user actions. Because the designer is working at a fairly abstract level the model is small and relatively easy to change. Indeed it is possible to throw away the whole dialogue model and start again. The WOD, exceptions list and scenarios can be though of as a design for the users' work. They will be based on what the users do now but will also contain some innovations requiring changes in current work practice. Evaluating the dialogue model against this representation of the users' work may suggest further innovations that can be recorded by changing the WOD and scenarios. It might also suggest new exceptions. It is much better to re-design someone's


337

job explicitly with careful consideration than implicitly by accident, as often happens when IT systems are introduced. This evaluation of the model against scenarios, WOD and exceptions is the iterative part of the process. The incremental part of the process is to start at some central task and work out. For example, in designing the copier interface one might judge that controlling the print settings was the most important task. Having developed an Action Simulator dialogue model to support this central task one then might go on to add rules describing the login procedure and so on. Each time some rules are added to the model Action Simulator is executed to evaluate it against the relevant parts of the scenarios, WOD and exceptions list as before. The final result is a complete high level model to the human-computer dialogue for the application that is communicated to the next stage of design. Because it can be executed it should be readily comprehensible to the designers responsible for fleshing out the design into a full specification.

5. Discussion The research leading to this method for discount dialogue modelling was done in the context of a collaborative project with two industrial partners. The PPS notation and the various devices for thinking about the user's job were all used in case studies in the course of this research. These included the design, for a client outside the project, of a system for logging the movements of food in and out of a cold store. We are convinced that the techniques described here will be workable in at least some real design environments. Our claim that the techniques are easy to learn and easy to apply has yet to be substantiated but has been born out by our own experience, the experience of our project partners and others. Tables 1 to 3 were extracted from a short brochure for use in teaching. The facts that it was possible to do describe the techniques in the space of a few pages suggests to us that they should be easy to teach and to use but we need to document more experience of their use by people who were not involved in their development. We also need to measure and document the benefits of using the approach. There are several areas where there is scope for further development of the techniques described here. Firstly, it should be possible to provide more in the way of tools for building the models. Olsen, Monk & Curry (1994) outline several automated analyses that can be carried out on PPS dialogue models for such properties as: task completeness, reversibility of effect, accessibility, connectedness and the avoidance of deadlock. An Action Simulator like tool could be built to include these analyses. Secondly, the process by which an abstract dialogue model is fleshed out into a full design has received little attention in this paper. What an Action Simulator dialogue model communicates to this stage of design can be thought of as a model of how the user needs to conceptualise the system. Thus the user needs to conceptualise each of the actions specified as unitary even though they may have several components. Also, in order to predict when an action is available, or what effect it will have, the user must be aware of the conditions the dialogue model can be in. So the conditions in an Action Simulator model communicate to the designer what changes in the state of system must be made salient to the user. Adding this detail may be a creative process that could not easily be automated, however, it may still be possible to provide software tools that help. This in turn depends on obtaining a better understanding of the process of design.

338


References Annett, J, Duncan, K D, Stammers, R B & Gray, M J (1971), "Task Analysis", HMSO, Training Information Paper No. 6. Apple Computer Inc [ed.] (1987), Human Interface Guidelines: the Apple Desktop Interface, Addison Wesley. Checkland, P B (1981), Systems Thinking, Systems Practice, John Wiley & Sons. Curry, M B & Monk, A F (1994a), "Dialogue Modelling of Graphical User Interfaces with a Production System", Behaviour & Information Technology, (in press). Curry, M B & Monk, A F (1994b), "The Dialogue Design Tool: Better Programming Through Design", in Cognitive Models and Intelligent Environments for Learning Programming, E Lemut & G Detorri [eds.], Springer-Verlag, (in press). Curry, M B, Monk, A F, Choudhury, K, Seaton, P & Stewart, T F M (1993), "Summarising Task Analysis for Task-based Design", in Proceedings ofINTERCHI'93, S Ashlund, K Mullet, A Henderson, E Hollnagel & T White [eds.], ACM Press, pp.45-46. Foley, J, Kim, W, Kovacevic, S & Murray, K (1991), "UIDE — An Intelligent User Design Environment", in Architectures for Intelligent User Interfaces: Elements and Prototypes, J Sullivan & S Tyler [eds.], Addison Wesley, pp.339-384. Harel, D (1987), "Statecharts: A Visual Formalism for Complex Systems", Science of Computer Programming 8, pp.231-274. Harrison, M D & Thimbleby, H W [eds.] (1990), Formal Methods in Human-Computer Interaction, Cambridge University Press. IBM [ed.] (1991), Common User Access (CUA). Systems Application Architecture, Basic and Advanced Interface Design Guides, IBM technical publications. Monk, A F (1990), "Action-Effect Rules: A Technique for Evaluating an Informal Specification Against Principles", Behaviour & Information Technology 9, pp.147-155. Nielsen, J (1989), "Usability Engineering at a Discount", in Designing and Using Human-Computer Interfaces and Knowledge-based Systems, G Salvendy & M J Smith [eds.], Elsevier Science (North-Holland), pp.394-401. Olsen, D R (1990), "Propositional Production Systems for Dialog Description", in Proceedings of CHI'90: Human Factors in Computing Systems, J C Chew & J Whiteside [eds.], ACM Press, pp.57-63. Olsen, D R, Monk, A F & Curry, M B (1994), "Algorithms and Procedures for Analysing Usability", HumanComputer Interaction, (in press). Sutcliffe, A (1988), Human-Computer Interface Design, Macmillan. Thimbleby, H W (1993), "Combining Systems and Manuals", in People andComputers VIII, J L Alty, D Diaper & S Guest [eds.], Cambridge University Press, pp.479-488, Proceedings of HCI'93.

Executable Task Analysis: Integration Issues Conn V Copas & Ernest A Edmonds LUTCHI Research Centre, Department of Computer Studies, Loughborough University of Technology, Loughborough, Leicestershire LEI 1 3TU, UK. Tel: +44 (0)509 222691 EMail: [email protected] Fax: +44 (0)509 610815 Executable task models are considered to hold promise for increasing the impact of task analysis upon software engineering. However, this approach generates a number of integration issues when user interface management systems (UIMS) are employed at run-time. A number of models of integration are proposed and critiqued, employing a task scenario which involves the customisation of a geographic information system. Some current UIMS are shown to militate against this integration on account of the dominance which is accorded to the application. Some current task representations are shown to pose integration problems on account of both their procedural nature, and neglect of the application. One organising theme of the paper is that of executable task models being considered as particular cases of knowledge-based systems. Keywords: task analysis, user interface management systems, knowledge-based systems, geographic information systems, rapid prototyping, knowledge representation, formal methods, user modelling.

1. Introduction A truism of software engineering for interactive systems is that development should ideally proceed from a requirements analysis, and should incorporate user requirements. Task analysis (TA) is frequently regarded as the most formal of available methods for addressing user requirements. However, it is also generally acknowledged that the lack of integration between TA and software engineering techniques has reduced the design impact of TA. Recently, one promising development is executable task models, which take some task specification and generate a series of interaction screens (de Haan & van der Veer, 1992;

340

Conn V Copas & Ernest A Edmonds

Wilson et al., 1993). This approach has the advantage of demonstrating user requirements to the software developer in a concrete and unambiguous fashion. It may be distinguished from conventional rapid prototyping by the fact that the interaction is generated in a more principled fashion. However, the approach inherits a common limitation of 'throw away' prototyping, namely, that in order to provide realism, it is necessary to simulate the responses of an application in some fashion. This approach may be necessary if one considers that the prototyping will be used to define the functionality of some innovative application, but is less acceptable where one wishes to re-use existing applications software. One obvious response to this problem is to attach application callbacks to the prototype, but this begins to duplicate work in user interface management systems (UIMS). A more satisfactory approach is to investigate greater integration between TA and UIMS, as has indeed been proposed by TA workers (Wilson et al., 1993). In a reciprocal fashion, one of the frontiers of UIMS research is considered to be greater integration with TA techniques (Green, 1987; Neches et al., 1993; Olsen, 1987). One aim of this paper is to articulate some of the technical challenges involved in any integration between executable TA and UIMS. The method employed is to propose and critique a number of hypothetical design alternatives in order to clarify requirements for integration. Thus, the contribution of the paper is partly conceptual, but also consists of an investigation of feasibility, supported by system examples. A distinguishing feature of this paper is that run-time architectures are addressed. That is, whilst the current emphasis on using TA models to stimulate or generate interface designs is laudable, there is also scope for consideration of the technically more ambitious case of using embedded TA models to exert direct control over the interaction between user and application. Secondly, this paper seeks to establish clearer links between the concepts of executable TA and knowledge-based systems. Many knowledge-based systems (particularly expert systems and intelligent tutoring systems) contain a model of the knowledge in some task domain. Further, the practice of employing a separate inference engine ensures that the knowledge base will be both executable and potentially interactive. Given the relative maturity of this technology, it is surprising that more cross-fertilisation with TA has not occurred, possibly because of artificial distinctions between 'knowledge acquisition' and 'task analysis', an issue which is re-considered later. This paper is structured as follows. First, the current position in UIMS with regard to the incorporation of user tasks is considered. Next, recent developments in TA are considered in more detail. These reviews are followed by the proposal and critique of some architectural models which might support the integration of TA and UIMS. The discussion employs a particular task scenario, that of using a geographic information system (GIS) to support regional planning, and introduces some issues of customisation. The paper attempts to extract some implications for both UIMS and TA practices for those situations where integration is intended.

2. User Interface Management Systems This discussion commences not with a UIMS as such, but with the pioneering command language grammar (Moran, 1981). This model proposed that interface design could be regarded as a top-down, four-stage process, involving the analysis of tasks, semantics, syntax and lexicon. The model has been criticised for its lack of specificity and has subsequently

Executable Task Analysis: Integration Issues

341

been extended (Elliot, 1990), but notions of a linguistic approach to the design of interaction proved to be influential on subsequent developments in UIMS. In contrast to Moran's grammar, the classical Seeheim UIMS model (Pfaff, 1985) partitions interaction into three rather than four components: a presentation layer, a dialogue controller and an application interface. Notably, the explicit modelling of user tasks is neglected. Typically, tasks have been considered to be implicit in the dialogue model (and possibly the presentation layer as well). That is, the dialogue model contains both the abstract specification of certain presentation objects which should be meaningful to the user, and also manages the available sequence of interface actions which are intended to enable users to achieve their task goals. Dialogue control is arguably the most crucial aspect of any UIMS, and has also proved to be the most problematic. Contemporary practice is to avoid the hard-coding of interaction sequences as much as possible in favour of more declarative methods of specification. In other words, it is considered preferable for much of the dialogue to be generated at run-time by using, for example, constraint satisfaction mechanisms (Hudson & King, 1986) or knowledgebased planning techniques (Senay, Sukaviriya & Moran, 1990). A central feature of these approaches is the employment of a sophisticated interface application model, as exemplified by CREASE (Hurley & Sibert, 1989) or UIDE (Sukaviriya, Foley & Griffith, 1993). Typically, the model employs an object/action representation, with actions causing changes in other objects' states. Actions are associated with both pre and post conditions which refer to object state rather than to other actions explicitly. Thus, the relationship between actions is specified implicitly rather than explicitly, and facilitates the run-time generation of or reasoning about the dynamics of the interaction. These practices have had the effect of reducing specification load, but not without cost. For example, it has been observed that object-oriented programming may foster the delusion that satisfactory dialogue will emerge without the need for any specification effort on the part of the developer (Cockton, 1990). There have been similar objections regarding the lack of explicit representations of control flow in some specification techniques (Edmonds & Hagiwara, 1990; Hartson, 1989). It has also been observed that a stilted form of interaction may sometimes arise when one relinquishes fine-grained control over the dialogue in favour of automatic generation (Myers, 1992). UIMS are conventionally regarded as being a means of supplying the user with greater interaction liberty (so-called external control), in contrast to systems based upon application frameworks (Myers, 1992). However, within UIMS, it is possible to distinguish variations in commitment to this concept of external control, with some approaches in fact being relatively application-dominated. For example, even sophisticated dialogue generators such as UIDE (Sukaviriya, Foley & Griffith, 1993) are based upon a logical or rational analysis of application semantics, with considerations of usage being only implicit. The limitations of this approach are most clearly revealed if one considers the building of alternative interfaces to data-base systems. Although the data-base may quite properly be considered to be an application, its general-purpose nature also endows it with some of the qualities of an application-generator. Consequently, the building of an interface may be regarded on one level as involving the simultaneous construction of an 'application' (King & Novak, 1993). In other words, the satisfaction of new task demands does not necessarily involve extensions to application functionality, but instead may involve customisation of the application. A UIMS which is

342


unnecessarily driven by the application model, or which models the application at too low a level, is restricted to generating the one general-purpose dialogue for all customisation possibilities. A number of UIMS do aim to inject more of a user-centred element. The developers of ITS (Wiecha et al., 1990) have gone so far as to suggest that dialogue design should be the responsibility of a task domain expert. FOCUS (Edmonds & McDaid, 1990; Edmonds et al., 1992) allows application actions to be aggregated in order to present a higher-level interface both to the user and to the dialogue controller. IMN (Ege & Stary, 1992) employs a dialogue which appears to be based upon constraint satisfaction mechanisms between application actions, interface actions and higher-level tasks. Bass, Kazman & Little (1992) present a conceptual UIMS architecture which is said to be inspired by the task modelling of Moran (1981), although the distinction between user tasks and applications actions is not clear. All of these approaches hold potential for the incorporation of explicit task models into UIMS.

3. Task Analysis TA has historically developed in isolation from systems analysis, leading to controversial claims that TA is either redundant or, less critically, that conventional analysis tools may readily be adapted to this end if the analyst shows such an inclination (Benyon, 1992). On the other hand, TA (particularly cognitive TA) continues to tantalise with the promise that it might deliver elusive insights, and thus has a unique contribution to make to software design. It is the case that, until recently, mainstream systems analysis tools have tended to neglect the support of conceptual stages of design, in particular, in modelling the user's universe of discourse. This situation is changing rapidly with the advent of object-oriented analysis (Coad & Yourdon, 1991) and the possible incorporation of techniques such as conceptual graphs into systems analysis (Loucopoulos & Champion, 1990). Simultaneously, TA has begun to develop some relatively sophisticated modelling ontologies (Braudes, 1990; Tauber, 1990; Walsh, 1989). It is an open question which of these techniques will prove most successful at this point. There are two other recent developments in TA which are encouraging from a software engineering perspective. The first is the provision of CASE-like environments for supporting the analysis process (Diaper & Addison, 1991; Wilson et al., 1993). These environments are intended to reduce the workload and to enhance the consistency of analysis. Some attempt to support early design evaluation by incorporating models of user characteristics and limitations. More significantly, there have also been advances in task representation methods, with the development of rigorous task specification techniques (Hartson & Gray, 1992; Wilson et al., 1992). These techniques underpin the development of executable task models, and are the main concern of this paper. As a minimum, TA involves a task decomposition such that a hierarchical tree of sub-tasks is constructed (neglecting at this point technical distinctions between goals, plans and actions, which are often treated inconsistently in the literature). Some of those sub-tasks refer to computer-supported activities, and thus the identification of sub-tasks may suggest high-level functions which should be offered by the user interface. One variation at this point concerns the issue of whether user-initiated inputs to the system may be associated with any node in the tree, or only with the leaves. In the latter case, the analyst is primarily concerned with


343

modelling abstract user tasks, and makes less initial commitment to the modelling of the interaction sub-system. A second dimension of variation concerns the issue of whether sub-task activities are parameterised, thus suggesting conceptual objects which should be visible in the interface. Some methodologies include an object taxonomy as well, and thus overlap with the conceptual design techniques referred to above. However, the greatest emphasis, and the greatest number of reported achievements to date, are based upon a process-centred description of user tasks. That is, TA frequently seeks to capture naturalistic user sub-task transitions in the expectation that these should be reflected in any computer-based support. A third dimension of variation concerns the manner in which relationships between sub-tasks are represented. In the minimal case, the sub-task tree defines an implicit precedence ordering and, not surprisingly, this type of modelling may be seen as a derivative of mainstream systems analysis (Sutcliffe, 1990). More advanced techniques employ a variety of dependency and temporal operators, see for example (Hartson & Gray, 1992; Wilson et al., 1992). Given TA's concern for the dynamics of user tasks, it is natural to look towards a mapping between user task models and interface dynamics, i.e. towards dialogue control models. TA has been employed, for example, in order to identify undesirable dialogue modes in existing systems (Borkoles & Johnson, 1992; Johnson et al., 1992). A small irony arises at this point, namely, that a hierarchical, process-centred model of user tasks, incorporating explicit precedence operators, is bound to suggest a highly moded dialogue itself. In other words, TA appears to be relying on the fact that its top-down approach will suggest the correct set of dialogue modes, in contrast to less user-centred approaches. Moded designs are generally not popular with interface researchers, although neither is modelessness necessarily beneficial for some classes of user (Thimbleby, 1990). This issue is reminiscent of UIMS controversies over the problems and merits of explicit representations of control flow. TA generally appears to subscribe not only to explicit representations (some of the advantages of which have been discussed in Section 2 of this paper) but also to procedural representations. One constant danger, however, is that of the rigidity which may arise from over-specification and hard-coded activity sequences (Took, 1990). Thus, a challenge exists for task representation methods: to find the right specification balance, as pursued at a later point in this paper.

4. A Task Scenario Before commencing a discussion of some hypothetical architectures in the next section, we wish to present an application scenario in order to clarify our intentions. The scenario is that of a regional planner using a GIS in order to support decisions about the selection of potential development regions within an English county, as we are currently involved in front-ending a GIS using the FOCUS UIMS. The planner is required to generate potential development corridors such that a variety of constraints pertaining to environmental and civic considerations are satisfied. A number of data layers are available, including roads, waterways, population, vegetation and elevation. These must typically be transformed in order to reflect the user's particular decision criteria (e.g. through reclassification or aggregation), and thence must be fed to some function which computes candidate solutions. The problem is sufficiently ill-structured that neither an optimal nor an algorithmic solution is possible, and conflicts between intermediate solutions must be resolved. The solution generation process is iterative.

344

Conn V Copas & Ernest A Edmonds Knowledge Based Front Ends for GIS > Mop Mana; IBIIIBIli i

Location: leics

Map Informatjon - Raster

.,

„

Ma 1 uls

'

. P l a y.

Map Transformation _ Raster

•

" '

Terrain Analysis

--,

Kelp 0 > j

Mapset: cocvc

,

„

J

Figure 1: A general purpose GIS main menu.

Standard analytical facilities include buffering, overlays, generation of descriptive statistics, and cost-surface modelling. Visualisation facilities include perspective views of 'drainage' across virtual cost surfaces. Readers are referred to Vidale et al. (1993) for more details of this type of task. GIS typically provide an extensive set of data manipulation commands which must often be amalgamated, sometimes in conjunction with external operating system commands, in order to achieve a given task. The one task goal may typically be satisfied by various sequences of application actions, each of differing efficiency (Burrough, 1992). Two distinct categories of support may be identified. The first relates to knowledge of the capabilities and limitations of the particular GIS being employed for the planning task. Conventionally, the application model of a UIMS is intended to maintain an awareness of the semantics of the particular application to which it is interfaced. Consequently, a knowledge-based UIMS such as one developed with the FOCUS architecture could be expected to be of value here, and this type of support has in fact been implemented. The main menu of the system is shown in Figure 1. Some of the functions of this embedded application knowledge are to prevent the user from performing semantically dubious operations, and to advise on sensible uses of the application. However, users must still perform a mapping from their regional planning goals to the highlevel functions offered by the interface. We anticipate an additional requirement for online support for the regional planning process itself. Put differently, the above system (in common with many database management systems) presents a general-purpose interface to the user. It would be desirable, however, if the interface reflected the particular task of regional planning. The design options presented in the next section are motivated by this view.

5. Some Design Alternatives 5.7. Task Models as Intelligent Consultants Following on from the previous discussion, one immediate possibility for providing specific regional planning support is to provide a module with encoded knowledge of planning processes. That module may include knowledge which is typically associated with task analysis, such as the relationship between various steps in the planning process. In addition, that module would be expected to be interactive. To the user, the module appears as an additional feature of the application, e.g. as an intelligent help or tutoring option (presuming that the module is intended to be consulted passively). Architecturally, however, the module constitutes a distinct application, as depicted in Figure 2.

345

Executable Task Analysis: Integration Issues Application User

DIALOGUE •CONTROL

APPLIC ATION INTERFACE

Task Knowledge Base

Figure 2: General purpose dialogue, with task models acting as an application supplement.

Within GIS, there are instances of this approach of supplementing the functionality of the application with intelligent modules (Elmes, 1992; Raper, Linsey & Connolly, 1990). One major design challenge is to increase the degree of integration between the support module and the rest of the application. Typically, the user escapes into a distinct mode in order to interact with the supporting system, then returns to normal use of the application. In the ideal case, it is preferable if some mapping exists between both the entities and the actions of the two systems so that the support module: a. is aware of the current context when responding to queries; and b. may call the application for purposes of demonstration. The application interface is the logical place at which this integration should occur, although this requires some revision of the conventional functions of the application interface. In particular, knowledge of the characteristics of multiple applications is required, as has been achieved in the case of the back-end manager within the FOCUS system (Edmonds & McDaid, 1990; Prat etal., 1990). Conventionally, user inputs have been regarded as something which alters application state, and which it is the responsibility of the application interface to manage. This issue becomes compounded if one considers that user inputs might invoke some intelligent module which then makes inputs to the main application. That is, on exit from the help/tutoring system, it is necessary to restore the original states of both the main application and the interaction sub-system, referring here to both data state and display (or view) state. Broadly, there are three approaches for achieving this goal, in ascending order of sophistication: 1. Store the state in a buffer, and restore it upon exit. This approach is moderately satisfactory for data, but is problematic for displays. Restoring of views and interaction state typically involves more than the retrieval of screen dumps. 2. Store input histories, with an associated undo for each input. This approach suffers from the problem that some inputs may not necessarily possess a simple undo, and that a convoluted, repetitious undo sequence could occur. 3. Most flexibly, employ planning techniques (Senay, Sukaviriya & Moran, 1990) in order that the system may reason about the most effective means of restoring state. This approach necessitates a state-based application model. In summary, this approach of employing a task model as an independent ancillary to the application appears feasible, given the current state of technology. One disadvantage, however, is that the dialogue remains general-purpose, whereas many task analysts require that their models will have a more active effect upon the dialogue, at least. This possibility is considered next.

346


5.2. Implicit Task Models The conventional strategy for injecting task-specific characteristics into GIS is to undertake a program of customisation (Bundock & Raper, 1991). The basic approach is to present a specialised view of the application by tailoring some graphical interaction objects, constructing special-purpose screen sequences, and supplying command macros. Generally, the customised interaction is more constrained than the original and, in many ways, involves the construction of the illusion that the application has been designed for just one purpose. This strategy implicitly subscribes to UIMS ideals of separability between interaction subsystem and application. UIMS technology has so far made little impact on the GIS field, and so customisation typically involves the use of either interface builders or raw window manager programming. Extrapolating, however, GIS customisation represents nothing more than the prevailing approach in UIMS design. That is, if a UIMS was to be employed, the largest aspect of customisation is dialogue re-design as new tasks emerge (assuming that the UIMS is not so application-centred that this process is infeasible). This implicit modelling of user tasks is unsatisfactory on grounds of maintainability. These issues of task-specific customisation have received little exposure in the UIMS literature, possibly because researchers have historically concentrated on front-ending single-purpose applications and have assumed that the satisfaction of new task demands invariably suggests extensions to the application itself. There has been speculation that an ideal UIMS design environment would automatically generate the user interface, with user task models feeding that process (Green, 1987). This approach requires that executable task models do not merely function as throw-away demonstrators, but have a software link with the rest of the UIMS. Some of the challenges involved in constructing that link are discussed in the next section. 5.3. Logical Separation of Task Models and UIMS The preceding discussion has suggested a requirement for task models to be explicitly separated from the UIMS. This raises the issue of whether logical or physical separation is being considered, with the former case being considered initially. The broad function of the task model has been proposed (Green, 1987) and, to a preliminary extent, realised (Wilson et al., 1993). First, a mapping is required between entities identified in the task domain and the device-independent abstract interaction objects (AIOs) defined in the dialogue manager, and realised in a device-dependent fashion by the presentation manager (Edmonds et al., 1992). To date, this has involved the pragmatic strategy of supplying the task module with a fixed set of AIOs, although it is obviously attractive to speculate that these might be constructed automatically on the basis of the task model (Wilson et al., 1993). Secondly, a mapping is required between activities identified in the task domain and activities supported in the interface by the dialogue manager. The extreme top-down approach considers that a process-centred task model in fact defines the dialogue abstractly, and so dialogue is effectively an instantiation of the task model. It is encouraging that a relatively large amount of task analysis research has been concerned with representing the mappings which are proposed to exist between users' conceptual entities/actions and those present in existing systems, e.g. (Payne, Squibb & Howes, 1990). These principles ought to be exploited during design, i.e. in the case in which one wishes to synthesise a device model from a user conceptual model. On the other hand, a number of deficiencies may be noted with current techniques, including a lack of expressiveness

347


User

DIALOGUE CONTROL

APPLIC ATION INTERFACE

t

pplication

Task Knowledge Base

Figure 3: Task-specific dialogue, with the mapping between the task knowledge base and the UIMS providing reconfigurability.

in representing user concepts, and a lack of semantic in preference to syntactical content (de Haan, van der Veer & van Vliet, 1991). This approach to dialogue definition is not without its problems. First, as indicated previously, task models tend to suggest a highly moded sequence of activities. It has been found necessary to relax these constraints in the actual dialogue by replacing precedence relations between sub-tasks with an interleaving operator (Wilson et al., 1993). This is a pragmatic but inelegant response, and, in effect, discards some of the information in the task analysis. Secondly, this type of approach neglects that dialogue controllers, when interfaced to an actual application, must do more than support user input, and must manage the responses of the application. For example, a common function is to provide context-sensitivity by enabling/disabling AIOs on the basis of state information received from the application. In other words, this top-down approach towards dialogue definition should be tempered with some bottom-up considerations.

5.4. Physical Separation of Task Models and UIMS Figure 3 provides a high-level depiction of an executable task model which has been interfaced to some UIMS. This paper is primarily concerned with the situation of frontending an application which possesses a relatively large functional core, as is the case with GIS customisation. A Seeheim-inspired UIMS is presented for simplicity, but contemporary extensions are assumed. Event-driven dialogue is envisaged, with rule-based control and/or dialogue control sub-systems providing concurrency. An object-oriented interaction system is not considered to impose undue limitations, but it is acknowledged that an object-oriented application could involve revisions to this approach (ten Hagen & Soede, 1991). The link between the task model and the UIMS may occur either at compile-time or at runtime, although the latter approach is preferable. First, for developers, it would be convenient if the task base was interpreted at run-time, so that edits could be made without quitting the system. Second, for end-users, a high-level facility for reconfiguring the task knowledge base should be anticipated. It should be noted that the application interface also constitutes a knowledge base, and conventionally has been used to present (indirectly) a designer's concept of application semantics to the user. In contrast, it is anticipated that users will possess individual and possibly evolving views about the semantics of their own tasks, and thus this knowledge base should be regarded as supplying default options rather than firm constraints. The dialogue controller also constitutes a knowledge base (at least in systems such as FOCUS) containing, for example, knowledge about the enabling and disabling of AIOs.

348

Conn V Copas & Ernest A Edmonds PLAN DEVELOPMENT REGIONS

DECIDE SELECTION CRITERIA

TRANSFORM DATA

DECIDE SOLUTION FUNCTION

GENERATE CANDIDATE SOLUTIONS

RESOLVE CONFLICTS

Figure 4: An illustrative regional planning task decomposition, represented in the form of a tree. Higher level activities near the root refer to abstract user tasks, whereas lower level activities at the leaves refer to interface actions.

On the grounds of concurrency, communication between the task model and the dialogue model occurs through messaging. This proposal of two communicating, possibly rule-based, systems may be seen as both an instantiation of and an extension of Cognitive Complexity Theory (Bovair, Kieras & Poison, 1990). Further, the content of that communication consists of indications of either the sender's state or some target state of the receiver, in preference to imperatives. It is anticipated that this approach facilitates the use of either planning or constraint satisfaction techniques to overcome some of the rigidities which derive from hardcoded action sequences within task models, as discussed previously in the context of UIMS. Such an approach has implications for task representation methods suggesting, in particular, that a state-based representation, employing an ontology of pre and post conditions upon actions, has advantages over process-centred representations. One example of this approach towards task analysis is provided by ETKS (Borkoles & Johnson, 1992). To illustrate these points more clearly, a brief analysis of a GIS-supported regional planning task is presented, inspired by descriptions such as may be found in (Tomlin, 1990). A task decomposition is presented in Figure 4. Figure 4 contains a task decomposition, but intentionally remains uncommitted to any specification of sub-task dependencies. However, the task scenario which was previously described certainly suggests that these dependencies exist. For example, it would be illogical to attempt to generate candidate solutions without having first established one's selection criteria. Process-centred task representations may seduce the analyst into specifying that these sub-tasks must occur in a consecutive fashion. On the other hand, to specify that these sub-tasks may occur in an interleaved fashion discards some of the semantics of the task domain. A state-based representation offers the option that one may specify that the existence of some data is a prerequisite for the solution generation step, without suggesting that one sub-task necessarily commences when the other ceases.


349

6. Integration Issues This paper concludes by flagging some unresolved integration issues which require further research. First, it is necessary to consider how application responses (filtered through the dialogue manager) impact the task executive. Most basically, these responses signal to the task executive that some dialogue act has been completed, thus providing the stimulus for the executive to transition into a new state. Previous workers (Moran, 1981) have, in effect, regarded the task model as a higher level dialogue model. In the physically separate case, however, it is necessary to specify more completely how control is achieved. In the worst case, it is necessary to anticipate top-down and bottom-up tensions occurring within the dialogue controller, as both the application and the task executive compete for control of AIO state. A related problem is that these proposals suggest that the task model exerts its influence on a general-purpose dialogue in a background fashion. Alternatively, the top-down philosophy would maintain that even highly abstract user sub-tasks, which are associated with no application callbacks, should be represented directly in the interface (referring to the second level of Figure 4 above). Returning to Figure 3, the discussion so far has been silent regarding links between the task model and the application interface. Conventionally, this interface has been regarded as containing the definitive model of the application, at least for the requirements of the front-end. In bottom-up design practices, the intention is that alternative dialogue managers would be interfaced to the application through the one module. However, design practices which incorporate more of a top-down element suggest the possibility that the application interface might be required to change in order to accommodate alternative front-ends. An immediate example of this requirement occurs if one considers that the task analysis may suggest the need for high-level functions which are not presently supported. There are no compelling reasons for believing that this link between the task and application models should be operative at run-time and, in fact, to do so would subvert the functions of the dialogue controller. The main design implication is the suggestion of moving away from the concept that the application interface provides 'the' view of the application, towards an approach in which this interface provides one of a number of possible views as circumstances dictate. One example of this philosophy (Olsen & Dance, 1988) allows users to construct and thence store macros in the application model.

7. Conclusions The integration of executable task models and UIMS has been demonstrated to be both desirable and feasible, at least in a minimal sense. A number of alternative models of integration have been proposed. We intend to gain implementation further experience using the FOCUS infrastructure and hence to address some of the outstanding issues discussed above.

Acknowledgements Conn V Copas was supported by the Defence Science & Technology Organisation, Australia, during the preparation of this paper.

350


References Bass, L, Kazman, R & Little, R (1992), "Toward a Software Engineering Model of Human-Computer Interaction", in Engineering for Human-Computer Interaction, J Larson & C Unger [eds.], Elsevier Science (North-Holland), pp.131 -154. Benyon, D (1992), "The Role of Task Analysis in Systems Design", Interacting with Computers 4(1), pp. 102123. Borkoles, J & Johnson, P (1992), "ETKS: Generative Task Modelling in User Interface Design", in Proceedings of the Hawaii International Conference on System Sciences, B D Shriver [ed.] #2, IEEE Computer Society Press, pp.699-708. Bovair, S, Kieras, D E & Poison, P G (1990), "The Acquisition and Performance of Text-editing Skill: A Cognitive Complexity Analysis", Human-Computer Interaction 5 (1), pp.1-48. Braudes, R E (1990), "Conceptual Modelling: A Look at System-level User Interface Issues", in Taking Software Design Seriously, J Karat [ed.], Academic Press. Bundock, M & Raper, J (1991), "Customisation: From Tools to Efficient Working Systems", in Proceedings of Mapping Awareness'91, pp.101-113. Burrough, P A (1992), "Development of Intelligent Geographical Information Systems", International Journal of Geographical Information Systems 6(1), pp.1-11. Coad, P & Yourdon, E (1991), Object-oriented Analysis (2nd edition), Yourdon Press (Prentice-Hall). Cockton, G (1990), "Designing Abstractions for Communications Control", in Formal Methods in HumanComputer Interaction, M D Harrison & H W Thimbleby [eds.], Cambridge University Press. Diaper, D & Addison, M (1991), "User Modelling: The Task Oriented Modelling (TOM) Approach to the Designer's Model", in People and Computers VI: Usability Now!, D Diaper & N Hammond [eds.], Cambridge University Press, pp.387-402, Proceedings of HCI'91. Edmonds, E & Hagiwara, N (1990), "An Experiment in Interactive Architectures", in Proceedings of INTERACT 90 — Third IFIP Conference on Human-Computer Interaction, D Diaper, D Gilmore, G Cockton & B Shackel [eds.], Elsevier Science (North-Holland), pp.601-606. Edmonds, E A & McDaid, E (1990), Knowledge-Based Systems 3 (4), pp.221-224. Edmonds, E A, Murray, B S, Ghazikhanian, J & Heggie, S P (1992), "The Re-use and Integration of Existing Software", in People and Computers VII, A Monk, D Diaper & M Harrison [eds.], Cambridge University Press, pp.415-427, Proceedings of HCI'92. Ege, R K & Stary, C (1992), "Designing Maintainable Reusable Interfaces", IEEE Software 9 (6), pp.24-32. Elliot, C(1990), "Linguistic Models in the Design of Cooperative Help Systems", in Proceedings of 1NTERACT90 — Third IFIP Conference on Human-Computer Interaction, D Diaper, D Gilmore, G Cockton & B Shackel [eds.], Elsevier Science (North-Holland). Elmes, G A (1992), "Data Quality Issues in User Interface Design for a Knowledge-based Decision Support System", in Proceedings of the 5th International Symposium on SpatialData Handling #1, International Geographic Union Commission on GIS. Green, M (1987), "Directions for User Interface Management Systems Research", Computer Graphics 21 (2), pp. 113-116.


351

de Haan, G & van der Veer, G C (1992), "ETAG as the Basis for Intelligent Help Systems", in Proceedings of the 6th European Conference on Cognitive Ergonomics, G C van der Veer, S Bagnara & G A M Kempen [eds.], European Association of Cognitive Ergonomics, pp.271-283. de Haan, G, van der Veer, G C & van Vliet, J C (1991), "Formal Modelling Techniques in Human-Computer Interaction", Ada Psychologica 78 (1-3), pp.27-67. ten Hagen, P J W & Soede, D (1991), "Assembling a User Interface Out of Communication Processes", in User Interface Management and Design, M R Gomes, D A Duce, F R A Hopgood & J R Lee [eds.], Springer-Verlag. Hartson, H R (1989), "User-interface Management Control and Communication", IEEE Software 6 (1), pp.6270. Hartson, H R & Gray, P D (1992), "Temporal Aspects of Tasks in the User Action Notation", Human-Computer Interaction 7 (1), pp.1-45. Hudson, S E & King, R A (1986), "A Generator of Direct Manipulation Office Systems", ACM Transactions on Office Information Systems 4 (2), pp.132-162. Hurley, W D & Sibert, J L (1989), "Modeling User Interface-Application Interactions", IEEE Software 6 (1), pp.71-77. Johnson, P, Markopoulos, P, Wilson, S & Pycock, J (1992), "Task Based Design: Mapping Between User Task Models and User Interface Designs", in Proceedings of the 2nd Interdisciplinary Workshop on Mental Models. King, R & Novak, M (1993), "Designing Database Interfaces with DBface", ACM Transactions on Office Information Systems 11 (2), pp. 105-132. Loucopoulos, P & Champion, R E M (1990), "Concept Acquisition and Analysis for Requirements Specification", Software Engineering Journal 5 (2), pp.116-124. Moran, T P (1981), "The Command Language Grammar: A Representation of Interactive Computer Systems", International Journal of Man-Machine Studies 15(1), pp.3-50. Myers, B A (1992), "State of the Art in User Interface Software Tools", in Advances in Human-Computer Interaction, H R Hartson & D Hix [eds.] #4, Ablex. Neches, R, Foley, J, Szekely, P, Sukaviriya, P, Luo, P, Kovacevic, S & Hudson, S (1993), "Knowledgeable Development Environments Using Shared Design Models", in Proceedings of the 1993 International Workshop on Intelligent User Interfaces, W D Gray, W Hefley & D Murray [eds.], ACM Press, pp.63-70. Olsen, DR (1987), "Larger Issues in User Interface Management", Computer Graphics 21 (2), pp.134-137. Olsen, D R & Dance, J R (1988), "Macros by Example in a Graphical UIMS", IEEE Computer Graphics and Applications, pp.68-78. Payne, S J, Squibb, H R & Howes, A (1990), "The Nature of Device Models: The Yoked State Space Hypothesis and Some Experiments with Text Editors", Human-Computer Interaction 5, pp.415-444. Pfaff, G [ed.] (1985), User Interface Management Systems, Springer-Verlag. Prat, A, Lores, J, Fletcher, P & Catot, J M (1990), "Back-end Manager: An Interface between a Knowledge-based Front End and its Application Subsystems", Knowledge-based Systems 3 (4), pp.225-229.

352


Raper, J F, Linsey, T K & Connolly, T (1990), "UGIX — A Spatial Language Interface for GIS: Concept and Reality", in Proceedings ofEGIS'90, J Harts, H F L Ottens & H J Scholten [eds.], EGIS Foundation. Senay, H, Sukaviriya, P & Moran, L (1990), "Planning for Automatic Help Generation", in Engineering for Human-Computer Interaction, G Cockton [ed.], Elsevier Science (North-Holland), pp.293-311. Sukaviriya, P N, Foley, J D & Griffith, T (1993), "A Second Generation User Interface Design Environment: The Model and the Runtime Architecture", in Proceedings of 1NTERCH1'93, S Ashlund, K Mullet, A Henderson, E Hollnagel & T White [eds.], ACM Press, pp.375-382. Sutcliffe, A G (1990), "Integrating Specification of Human-Computer Interface with Jackson System Development", Information and Software Technology 32(10). Tauber, M J (1990), "ETAG: Extended Task Analysis Grammar — A Language for Description of the User's Task Language", in Proceedings of INTERACT90 — Third IFIP Conference on Human-Computer Interaction, D Diaper, D Gilmore, G Cockton & B Shackel [eds.], Elsevier Science (North-Holland), pp.163-168. Thimbleby, H W (1990), User Interface Design, Addison Wesley. Tomlin, C D (1990), Geographic Information Systems and Cartographic Modelling, Prentice-Hall. Took, R (1990), "Putting Design into Practice: Formal Specification and the User Interface", in Formal Methods in Human-Computer Interaction, M D Harrison & H W Thimbleby [eds.], Cambridge University Press, pp.63-96. Vidale, L, deSede, M, Prelaz-Droux, R & Claramunt, C (1993), "GERMINAL Offers a New Approach for Swiss Environmental Management", GIS Europe 2(1), pp.58-60. Walsh, P A (1989), "ATOM: A Means for Integrating Human-Computer Interface Design with Jackson System Development", in Task Analysis for Human-Computer Interaction, D Diaper [ed.], Ellis Horwood. Wiecha, C, Bennett, W, Boies, S, Gould, J & Greene, S (1990), "ITS: A Tool for Rapidly Developing Interactive Applications", ACM Transactions on Office Information Systems 8 (3), pp.204-236. Wilson, S, Johnson, P, Kelly, C, Cunningham, J & Markopoulos, P (1993), in People and Computers VIII, J L Alty, D Diaper & S Guest [eds.], Cambridge University Press, pp.217-231, Proceedings of HCI'93. Wilson, S, Markopoulos, P, Pycock, J & Johnson, P (1992), "Modelling Perspectives in User Interface Design", in Proceedings of East-West International Conference on Human-Computer Interaction, J Gornostaev [ed.], ICSTI,pp.210-216.

Beyond Data Models for Automated User Interface Generation Angel R Puerta, Henrik Eriksson, John H Gennari & Mark A Musen Medical Computer Science Group, Knowledge Systems Laboratory, Departments of Medicine and Computer Science, Stanford University, Stanford, CA 94305-5479, USA. EMail: {puerta, eriksson, gennari, musen}@camis.stanford.edu Researchers in the area of automated design of user interfaces have shown that the layout of an interface can, in many cases, be generated from the application's data model using an intelligent program that applies design rules. The specification of interface behavior, however, has not been automated in the same manner, and is mostly a programmatic task. Mecano is a model-based user-interface development environment that extends the notion of automating interface design from data models. Mecano uses a domain model — a highlevel knowledge representation that augments significantly the expressiveness of a data model — to generate automatically both the static layout and the dynamic behavior of an interface. Mecano has been applied successfully to completely generate the layout and the dynamic behavior of relatively large and complex, domain-specific, form- and graph-based interfaces for medical applications and several other domains. Keywords: model-based interface development, automated interface design, interface models, domain models, data models.

1. Introduction One of the areas that is receiving increased interest by researchers is that of model-based user interface development. This emerging technology is centered around the premise that a declarative interface model can be used as a basis for building interface development environments. The model-based approach facilitates the automation of the design and implementation of user interfaces. In addition, researchers have shown that an application's data model can be used effectively to generate the static layout of an application's interface (de Baar, Foley & Mullet, 1992; Janssen, Weisbecker & Ziegler, 1993). However, data models have not been applied to the generation of interface behavior specifications.

354

Angel R Puerta, Henrik Eriksson, John H Gennari & Mark A Musen

In this paper, we present Mecano, a model-based interface development environment that extends the concept of generating interface specifications from data models. Mecano employs a domain model to generate not only the layout of an interface, but also its dynamic behavior. Domain models are high-level representations of the objects and relationships in a domain. Because they explicitly declare domain characteristics that are not normally part of data models, a domain model offers the possibility of automating a larger part of the interface design process than what is feasible with regular data models. The rest of the paper is organized as follows. We first detail the concept of model-based interface development and its relationship with automated user-interface design. Then, we define and exemplify domain models and contrast them to data models. Furthermore, we explore the architecture of Mecano and describe the interface-behavior generation process. Finally, we relate this work to other research efforts and present a number of conclusions.

2. Model-Based User-Interface Development The design and implementation of user interfaces is an iterative process that cycles from design to development until a satisfactory product is achieved. There are a number of tools that support the different phases of user interface construction. Current tools either focus on the design phase, or on the development phase, and have a number of shortcomings that make interfaces difficult to build. The principal areas where current user interface tools falter are: • Lack of integrated design and development support. Design tools such as HyperCard do not support development, whereas development tools such as many UIMSs do not support high-level design. • Lack of support for dynamic behavior specifications. Conventional programming languages are the only way to specify most of the dynamic aspects of a user interface. • Low level of automation. Interface components such as windows and menus must be designed and specified one by one. There is no support for making global changes that affect all components or groups of components. • Poor lifecycle support. Design changes are difficult to propagate, and maintenance is time-consuming because every change must be applied manually to each interface component. The shortcomings of available user-interface tools are due to a common cause: Developers work at a low level of abstraction — with items such as windows, widgets, and programming constructs — but have no access to explicit representations of the design knowledge needed to create an interface. A solution to such shortcomings is to provide developers with declarative interface models, that allow manipulation of all facets of interface design at a high level of abstraction. The key idea is to use an interface model as the central component of an integrated design and development environment that supports all phases of user interface construction. Figure 1 shows a basic generic architecture for model-based interface development systems. The key component is a declarative interface model that represents the various characteristics of a user interface design, from presentation and behavior to user- and working-environment preferences. Because the interface model covers all facets of interface design and development, it can be used as a central repository of knowledge that software tools can

Beyond Data Models for Automated User Interface Generation Interface Model Dialog

355

Design-Time Tools

Presentation Task User/Environment Knowledge Representation

Application-State I Monitor I

f User I Interfacel

Figure 1: The basic architecture of a model-based user interface development system. The interface model is a central repository of interface design knowledge. A set of design tools manipulate the model to achieve a particular design. The run-time system implements the design in the interface model as a running interface. The run-time tools support the use of the interface — with functions such as help — by monitoring the state of the interaction through the application-state monitor.

access to perform functions related to interface construction. Thus, the set of designtime tools manipulates the model directly to build an interface design. Examples of such tools are model editors, design critics, and design-alternative generators. Similarly, run-time tools use the model to add support to the human-computer interaction process. Examples of run-time tools are help generators and performance-monitoring tools. The run-time tools are intrinsically related to the application-state monitor (which in some systems may be part of the run-time system). This component keeps track of the current, previous, and possible future states of the interaction, and communicates with the run-time tools to relay needed state information to those tools. The run-time system — in some systems called the interface generator — accesses the model to implement the design embodied in a particular interface model instance as a running user interface. Model-based systems create an integrated interface-development environment where developers move from a generic description of interfaces (the interface model) to a specific description of a single interface (the interface-model instance) using design-time tools. The systems then take the model instances and implement them as interfaces, adding the interaction support of the run- time tools. By centralizing all the design knowledge, and by abstracting such knowledge at a high level, model-based systems offer the opportunity to streamline the iterative process of interface construction, to allow implementation of global definitions and design changes, to support specification of dynamic behavior without the need for conventional programming, and to automate major parts of the interface-development cycle.

356

Angel R Puerta, Henrik Eriksson, John H Gennari & Mark A Musen High-Level Dialog Specification Interface Model (Design)

Figure 2: Generic framework for automated interface-generation environments that employ data models. The interface design is produced by tools that examine a data model and a dialog specification. The design may be represented implicitly or explicitly (as an interface model). The run-time system implements the design.

The few model-based systems that have been developed tend to fall into one of two categories: 1. systems that assist in the design process; and 2. systems that automate the interface-design process. Systems in the first category normally include design-time tools with advanced modelvisualization and editing capabilities. The underlying philosophy is to facilitate the task of model manipulation. Systems in the second category contain complex tools that can instantiate large portions of the interface model for a given interface. Their corresponding philosophy is to minimize the amount of effort needed for model manipulation during design of an interface. In general, design-assistance systems offer maximum design flexibility but increase developer effort, whereas design-automation systems minimize developer effort but offer less design flexibility, thus requiring additional custom-tailoring to complete satisfactory interfaces. Mecano is a design-automation model-based development system with special facilities to assist developers in customizing generated interfaces. In the next section, we examine the approach to interface generation in Mecano.

3. Automatic User Interface Generation One important type of design-time tool in model-based interface-development systems is that of automatic interface generators. Such tools partially specify an interface design from a higher-level specification, such as a data model, or a dialog representation. Of special interest to the goals of Mecano as an environment that can integrate the development of applications and interfaces is the use of an application's data model to generate the static layout of an interface (de Baar, Foley & Mullet, 1992; Janssen, Weisbecker & Ziegler, 1993).

Beyond Data Models for Automated User Interface Generation

357

Figure 2 shows a generic framework for automated interface-generation environments that employ data models. An intelligent program examines the data model and applies a set of design rules to produce a static layout design for an interface. Because the data model is shared between the interface design and the target application design, both designs can be coupled, and changes to the application design can be propagated easily to the interface design. The dynamic behavior of the interface, however, must be specified separately. This process can take many forms, from using a graphical editor to construct dialog Petri nets (Janssen, Weisbecker & Ziegler, 1993), to assigning sets of pre- and postconditions to each interface object (Gieskens & Foley, 1992). Although working with high-level dialog specifications is helpful to interface developers, it does not automate the design of dynamic behavior. For large interfaces, editing the dialog specifications is still a time-consuming task involving the definition of hundreds of actions and conditions, some of which may conflict with each other. The main reason that current data-model approaches cannot automate the design of dynamic behavior is that data models themselves are very limited in what they express. They are applied only to serve as a vocabulary to access the data structures of the application. The intelligent design tools that examine the data model can only make design decisions based on the information in the data model. Most of the dynamic behavior of an interface is domain-specific, but data models are not used to capture effectively the characteristics of a given domain. In Mecano, we intend to use domain models instead of data models to generate interfaces. A domain model is a representation that captures all the definitions and relationships of a given application domain and that subsumes the data model for the application. By substituting the data model in Figure 2 with a domain model, Mecano does not require dialog-specification editing and is able to generate complete dynamic-behavior specifications even for large interfaces.

4. Domain Models A domain model is a representation of both the objects in a domain and their relationships. As such, a domain model may include a data model of the domain. In the same spirit that interface models provide developers with access to a higher-level representation of design knowledge, domain models also allow access to a level of representation higher than that of data models. Whereas data models establish a vocabulary to access the data structures of an application, domain models establish a vocabulary to access the objects in an application domain. As we will detail in this paper, domain relationships are a key to determining the dynamic behavior of user interfaces. With Mecano, we exploit the relationships defined in domain models to generate dialog and layout specifications for user interfaces. The result is a theory of how to map domain concepts to interface designs through a series of mappings connecting a domain object, a domain characteristic, or a domain relationship to an interface design element — such as a window navigation tree, an interaction style, or a dialog constraint. Figures 3 and 4 show partial views of a model for the medical domain of therapy planning according to standard treatment protocols. We have defined this model using a framebased representation language that defines class hierarchies (Gennari, 1993). This framerepresentation language is used to define not only domain models, but also the interface modes that Mecano employs. Keeping both types of models in the same language improves shareability of the models with other groups, and facilitates the generation of interface-model instances from domain models.

358

Angel R Puerta, Henrik Eriksson, John H Gennari & Mark A Musen I Chemotherapy I

[ Follow-Up j Figure 3: Partial view of a medical domain model for therapy (protocol) administration. The part-of hierarchy can be mapped to the window navigation tree of an interface — an example of the type of mappings domain-to-interface that are exploited by Mecano to generate interface designs.

Slots Name Chemotherapy l£

Facets (type :string)

Algorithm Drug_Parts

(allowed-classes .drug)

Figure 4: Partial view of the slots and facets (properties) for the chemotherapy class of Figure 3. The slot type can be mapped to an interaction style (e.g. type string to text-field object).

5. The Mecano Architecture Figure 5 shows the major components of the Mecano architecture. This architecture follows the basic model-based system architecture shown in Figure 1 with some minor variations. The design-time tools include a model editor (for both domain and interface models), an intelligent designer to generate interface model instances (i.e. interface designs) from domain models, and an interface builder to custom-tailor the designs produced by the intelligent designer tool. There is no application-state monitor since this component is intended to be subsumed by the run-time system. Mecano integrates design, development, and maintenance capabilities in a single environment. It manipulates sharable objects (domain and interface models) that can be used, in the spirit of ARPA's Knowledge Sharing Effort (Neches et al., 1991), by other groups to generate interfaces in their host environments. Mecano also provides a degree of platform independence by producing textual interface model instances (i.e. interface specifications) that are implemented by a run-time system. In this manner, textual specifications can be generated in Mecano, and such specifications can be implemented by an appropriate run-time system in a different platform. The process of generating automatically interfaces within the Mecano context is depicted in Figure 6. The central concept is that of interface model instantiation where the intelligentdesigner tool processes a given domain model and creates an application- and domain-specific instance of the generic interface model. The instance is created by applying a series of


Design Tools Intelligent Designer

359

Model Editor

Interface 1 Builder

Figure 5: The main components of the Mecano architecture. The system delivers a development environment where all phases of interface construction, from design to maintenance, are supported. The architecture follows the basic model-based system architecture shown in Figure 1.

Figure 6: Generating interfaces in Mecano. The intelligent-designer tool instantiates the interface model for a given application and domain. It processes the domain model and creates the desired instance through a mapping of domain to interface characteristics. The run-time system implements the interface model instance by accessing an interface-specification language version of the instance.

mappings between domain and interface characteristics. An interface model instance is a fully represented interface design that is application and domain specific. Once such an instance is available, it can be implemented as an interface by feeding it to a run-time system in the form of an interface specification written in a declarative language.

6. Coupling Interface and Application Design The goal of separating an application from its interface at the development level has been established as a sound software-engineering principle. The separation allows for a degree of encapsulation of the design of both elements and minimizes the impact of application software changes on interface software. The interface and its target application, however, still share

360


a common data — or knowledge — representation. The application operates on the data and the interface allows end users to visualize those data. As a consequence, any changes that an application designer may make to a data model necessitate corresponding changes to the interface design. To propagate such changes more effectively, it is essential that both application design and interface design be coupled. The application data model is the obvious first candidate for the basis for coupling both designs; that approach has been taken by some groups (de Baar, Foley & Mullet, 1992; Janssen, Weisbecker & Ziegler, 1993). Furthermore, researchers are now exploring domain models, rather than data models, as the foundation for domain-specific software applications (Puerta, Tu & Musen, 1993). These researchers, however, have concentrated on application design but not on user-interface design. The Mecano approach to user interface generation has the added advantage of providing a direct way to achieve two highly desirable goals: coupling application design with interface design and automating the generation of complete interface designs — including static layout and dynamic behavior. Our laboratory is developing an environment to build domainspecific software architectures, called PROTEGE-II (Puerta, Tu & Musen, 1993) that uses domain models as the basis for application development. Mecano and PROTEGE-II will constitute an environment in which the coupling of application design and interface design is realized.

7. Generating Interfaces with Mecano The interface development cycle of Mecano is shown in Figure 7. After a domain and task analysis, a domain model is defined with the model editor shown in Figure 8. It is not necessary to build domain models from scratch for every application. For example, a domain model for medical therapy planning can be reused, with minor variations, in other applications. This is a significant advantage of Mecano over systems that design from data models because data models are difficult to reuse across applications. Once edited, the domain model is used to generate dialog specifications. These specifications have two levels in Mecano (Puerta, 1993; Puerta et al., 1994): • High-level dialog defines all interface windows, assigns interface objects to windows, and specifies the navigation schema among windows in the interface. • Low-level dialog defines specific dialog elements (widgets) to each interface object created at the high level and specifies how the standard behavior of the dialog element is modified for the given domain.

7.1. High-Level Dialog Generation The elements of the high-level dialog specification are generated by examining the class hierarchy of the domain model (see Figure 3) and the slots of each class (see Figure 4). Figure 9 shows an interface generated from the partial domain model shown in Figures 3 and 4. The complete medical domain model for therapy administration generates an interface with over 60 windows and hundreds of widgets. Note that the dialog for window navigation is established during high-level dialog design but that it can be refined, or augmented, at low-level dialog design time. The procedure to generate a high-level dialog design is as follows:


361

Generate High-Level Dialog

Iterate

Custom-Tailor Design

Generate Low-Level Dialog

Developer and Participatory Design Actions Mecano Processes

Figure 7: Mecano defines an iterative interface development process with a high degree of automation.

• Each class in the hierarchy is assigned a window. • Window navigation is established by searching the class hierarchy for links indicated by the allowed-classes facet in the domain model. For example, the Drug window shown in Figure 9 is accessed from the Chemo window because the Drug class is an allowed class for the slot DrugJPart. • Each window is assigned one interface object per slot in the class. After generation, the developer has the option of customizing the interface by splitting windows multiple objects into two or more windows. Interface objects are assigned actual widgets during low-level dialog design.

7.2. Layout Generation Layout generation is similar to that performed by other systems such as UIDE (de Baar, Foley & Mullet, 1992; Gieskens & Foley, 1992) and GENIUS (Janssen, Weisbecker & Ziegler, 1993) that use data models to generate the layout: • Each interface object defined at high-level design time is assigned a dialog element (widget) by examining the facets of the corresponding slot in the domain model. For example an object of type string is assigned a text field, an object of type Boolean is assigned a check-box widget, and an object of type string and cardinality multiple (i.e. the object can be multiple-valued) is assigned a list browser. • Each dialog element is placed on its corresponding window by a layout algorithm that observes interface design guidelines.

362

Angel R Puerta, Henrik Eriksson, John H Gennari & Mark A Musen Ontology Browser Aid it-a imt

i Dom»lnn»ineV' (Protocol

Da'3H iva link j

Classes: domaln-entitv

Inspect fpnisnpiicr-tjifpl&ts jjICtlSW-dP

[leli3io.'i!/-cntar,cn Ipinioiv-ti

r

Delete

IS Class n< me

E

Qass Inspector

Is Slot n»m *•

| chemotherapy

|numr j

Parents):

r o'.or.c*

Slots: r.ini.

Mj ij~ r

ffi 1.11 • ri;c r"=B : M J rant

I

In'o-i'ea t oh T " it T "3--= »y

|

m

Facet Inspector

r

>•>: s -'v--i

-1

Fdctli-

Figure 8: Developers edit domain models with a browser tool that allows definition, review, and inspection of models.

7.3. Low-Level Dialog Generation Elements of the low-level dialog specification are generated by examining the facets (properties) defined for each slot in the domain model (see Figure 4). These facets include part-of relationships among classes. • Each dialog element may be assigned actions beyond the standard behavior of the dialog element by examining the facets of the corresponding slot in the domain model. Examples of dialog-element actions include disabling editing in other dialog elements, and updating values in other dialog elements after a user input action (see Figure 9). Note that the specification of dialog-element actions is one of the important operations that are not automated in systems that rely on data models for interface generation. 7.4. Layout and Design

Revision

After Mecano produces generates an executable design, the developer conducts participatory design sessions with end users to custom tailor the design and make appropriate changes. Note that the required changes may necessitate editing of the corresponding domain model, and consequently, regenerating a new interface design.

363

Beyond Data Models for Automated User Interface Generation Update After User Input •

—

Display Graphical Editor

^^^^^^—

Interface Objects from Protocol Class Slots

Window for Protocol Class

High-Level Dialog Design Low-Level Dialog Design

Figure 9: A form-based interface generated from the domain models partially shown in Figures 3 and 4. The interface generated from the full domain model for medical therapy consists of over 60 windows and hundreds of dialog elements.

Mecano provides facilities for reapplying any customization done to a design before it was regenerated (Eriksson, Puerta & Musen, 1994). Thus, Mecano allows developers to experiment with early prototypes and to quickly revise stable designs. Our experience in participatory layout revision with end-users is that working sessions, even for large interfaces, can be completed is a few hours at the most.

7.5. Domain-Specific Graphical Editors In addition to the form-based interfaces shown in Figure 9, Mecano can generate layout and behavior specifications for domain-specific graphical editors. For example, consider the following slot information for the class Protocol: (slot algorithm (type :procedure) (allowed-classes :xrt :chemotherapy :drug))

The intelligent designer tool in Mecano examines this slot and creates an abstract interface object for the slot, then due to the type procedure in the slot, it maps the interface object to a graphical editor as its dialog element. The tool also defines three graphical objects to be used during editing, one for x-ray therapies (xrt), one for chemotherapies (chemo), and one for drugs using information derived from the allowed-classes facet of the algorithm slot. Figure 10 shows a graphical editor generated from the above slot definition.

364


Figure 10: A graphical editor for the specification of medical procedures. The available drawing objects and the constraints on their interconnectivity at run-time are determined by Mecano during the dialog-generation phases.

8. Related Work There are a number of model-based development environments reported in literature that are closely related to the Mecano effort. The TRIDENT (Vanderdonckt & Bodart, 1993) system bases the interface generation process on a user-task analysis that yields a user-task model and an application model that drive the generation process. TRIDENT automatically designs high-level dialog and presentation specifications that follow declarative interface design guidelines stored as part of the system's knowledge base. TRIDENT is geared toward assisting developers, in contrast to the automation emphasis on Mecano. Thus, users of TRIDENT must use extensively the available model editing tools to define in greater detail the task and application characteristics required for dialog generation. TRIDENT does not generate low-level dialog specifications. Another development environment centered around a user-task model is ADEPT (Johnson, Wilson & Johnson, 1994). This system allows development of interfaces through an evolutionary process of editing a task model, generating a prototype, and refining the task model. As with TRIDENT, it requires extensive editing and the specification of interface actions directly through the task model editing tool — a step automated for the type of interfaces generated with Mecano. The GENIUS environment (Janssen, Weisbecker & Ziegler, 1993) uses an entity-relationship data model, along with a graphical editor for dialog specifications, to generate interfaces. The data model, which can be edited graphically, provides the basis for the definition of the interface components and their layout. The graphical editor allows the review of dialog nets, a variation of Petri nets, that define the actions of the interface objects and the conditions that preclude or follow those actions. GENIUS is not designed to generate behavior specifications from high-level models (e.g. a task model).


365

The UIDE environment includes a tool for static layout generation from an extended data model (de Baar, Foley & Mullet, 1992). The specification of dynamic behavior, however, must be achieved by defining sets of pre- and post-conditions (Gieskens & Foley, 1992) for each one of the interface objects. HUMANOID (Szekely, Luo & Neches, 1993) defines an elaborate interface model that includes components for the application, the presentation, and the dialog. Developers construct application models and HUMANOID picks among a number of templates of interfaces to display the interface. The developer can then refine the behavior of the interface by editing the dialog model. HUMANOID assists, but does not automate, the generation of dynamic behavior specifications, and requires considerable additional developer effort to generate interfaces that do not conform to its templates, as is the case with most complex interfaces.

9. Analysis and Conclusions We have presented Mecano, a model-based development environment that extends the notion of using data models to drive interface-specification generation. The main advantages and contributions of Mecano are: • Use of domain models to drive interface-specification generation. Domain models make explicit domain information and relationships that are not included in data models. Domain models are reusable across applications. • Generation of both the static layout and the dynamic behavior of domain-specific, form- and graph-based interfaces, including relative large and complex ones, for multiple domains (e.g. medical treatment, elevator configuration). • A highly automated design environment that supports the full development cycle of an interface while coupling interface and application design. • Textual interface-model instances that can be made executable by multiple run-time systems, thus providing a degree of portability to the generated interfaces. • A basic theory of how to amp domain characteristics to interface design specifications. The automatic nature of the Mecano development process constricts the design space of the generated interfaces. We have emphasized participatory design revisions as part of the Mecano environment to balance in part the lack of human input in the generation phase. In addition, the generation of interfaces from domain models is most effective for interfaces with relatively fixed dialog structures — as is the case with the form- and graph-based interfaces supported by Mecano. Interfaces with highly complex and flexible dialog structures do probably need the existence of a detailed task model — such as those in ADEPT (Johnson, Wilson & Johnson, 1994) — to drive the generation of the interface. We are researching ways to combine the task modeling capabilities available in Mecano through its generic interface model with the domain-model driven approach to user interface generation. Overall, Mecano provides a combination of comprehensive design and development support, level of automation, and portability of generated interfaces that should form the basis for continued research into complete and effective interface development environments.

366


Acknowledgements This work has been supported in part by grants LM05157 and LMO53O5 from the National Library of Medicine, and by gifts from Digital Equipment Corporation. Dr Musen is recipient of NSF Young Investigator Award IRI-9257578.

References de Baar, D J M J, Foley, J D & Mullet, K E (1992), "Coupling Application Design and User Interface Design", in Proceedings of CM'92: Human Factors in Computing Systems, P Bauersfeld, J Bennett & G Lynch [eds.], ACM Press, pp.259-266. Eriksson, H, Puerta, A R & Musen, M A (1994), "Generation of Knowledge Acquisition Tools from Domain Ontologies", in Proceedings of the Eighth Banff Knowledge Acquisition for Knowledge-Based Systems Workshop, pp.7.1-7.20. Gennari, J H (1993), "A Brief Guide to Mattre and MODEL: An Ontology Editor and a Frame-Based Knowledge Representation Language", Stanford University Knowledge Systems Laboratory, Report KSL-93-46. Gieskens, D F & Foley, J D (1992), "Controlling User Interface Objects through Pre- and Post-conditions", in Proceedings of CHI'92: Human Factors in Computing Systems, P Bauersfeld, J Bennett & G Lynch [eds.], ACM Press, pp.189-194. Janssen, C, Weisbecker, A & Ziegler, J (1993), "Generating User Interfaces from Data Models and Dialogue Net Specifications", in Proceedings oflNTERCHI'93, S Ashlund, K Mullet, A Henderson, E Hollnagel & T White [eds.], ACM Press, pp.418-423. Johnson, P, Wilson, W & Johnson, H (1994), "Scenarios, Task Analysis, and the ADEPT Design Environment", in Scenario-based Design, J Carroll [ed.], Addison Wesley, (in press). Neches, R, Fikes, R, Finin, T, Gruber, T, Patil, R, Senator, T & Swartout, W (1991), "Enabling Technology for Knowledge Sharing", AI Magazine 12 (3), pp.36-56. Puerta, A R (1993), "The Study of Models of Intelligent Interfaces", in Proceedings of the 1993 International Workshop on Intelligent User Interfaces, pp.71-80. Puerta, A R, Eriksson, H, Gennari, J H & Musen, M A (1994), "Model-Based Automated Generation of User Interfaces", in Proceedings of the Twelfth National Conference on Artificial Intelligence, AAAI94. Puerta, A R, Tu, S W & Musen, M A (1993), InternationalJournal of Intelligent Systems 8 (1), pp.129-152. Szekely, P, Luo, P & Neches, R (1993), "Beyond Interface Builders: Model-based Interface Tools", in Proceedings of INTERCHI'93, S Ashlund, K Mullet, A Henderson, E Hollnagel & T White [eds.], ACM Press, pp.383-390. Vanderdonckt, J & Bodart, F (1993), "Encapsulating Knowledge for Intelligent Automatic Interaction Object Selection", in Proceedings of INTERCHI'93, S Ashlund, K Mullet, A Henderson, E Hollnagel & T White [eds.], ACM Press, pp.424-429.

Computer-Supported Cooperative Work

Designing a User Interface for Folding Editors to Support Collaborative Work Richard J King* & Ying K Leung* * Computer Power Group, 493 St Kilda Road, Melbourne, Victoria 3004, Australia. EMail: [email protected] * Centre for Systems Methodologies, Swinburne University of Technology, PO Box 218, Hawthorn, Victoria 3122, Australia. EMail: [email protected] Software engineering requires the production of hierarchically organised text-based documents, such as specifications, source codes, and manuals. Folding editors are effective tools designed to support such activities involving documents of any complexity. In a collaborative work environment, especially in a large software development team, folding editors offer many advantages over the conventional editing system. Surprisingly, the use of folding editors has often been overlooked. This could partly be attributed to the generally inadequate design of their interface and partly to the lack of effective facilities available to support collaborative work. This paper identifies some of the typical difficulties associated with sharing files in a software engineering teamwork environment and discusses the advantages of folding editors to overcome these problems. It then describes the rationale of a design of a user interface for folding editors to support collaborative work. Keywords: folding editors, computer-supported cooperative work (CSCW), groupware, user interface design, software engineering, CASE tools, collaborative writing.

1. Introduction Software development is an area of human activity requiring that people work together to solve a problem. Typically, teams of people are involved in the specification, design, implementation and testing of the many different modules that comprise a system under construction. Even though most tasks are assigned to one individual at any one time, many documents and program modules are written and maintained by multiple authors over the lifetime of the

370

Richard J King & Ying K Leung

software project. Software engineering requires the production of hierarchically organised text-based documents, examples of which are: specifications, schedules, designs, programs, test plans, on-line help and manuals. All of these documents have two common attributes: first, they are structured, more specifically, they are hierarchical in structure; second, more than one person will in general author them. Beaudouin-Lafon (1990) stresses the need for specific tools to support the collaborative development of software with small teams (2 to 5 people). The process of editing software modules is the most common reason for the team members to need to access and modify each other's files. There are two underlying problems associated with such a work environment. The first one relates to the large information space which needs to be presented on a small display screen; the flat view of the source code is invariably too large to enable the viewer to maintain an adequate overall context on a typical display screen. While windowing systems provide a simple means to overcome this problem, the user is often required to switch attention from one window to another if more than one window is opened. Further, as the number of overlapping windows increases, screen clutter results; the mental load imposed on the viewer who has to keep track of these windows and, at the same time, to interpret their contents can become unacceptably demanding. The second problem concerns the interaction of team members with the text documents during their development. Traditionally modifications are made to software modules after they are 'checked out' from a repository using a version/revision control system (VCS or RCS). The process of checking out software produces a local copy for the person to edit and usually also flags that module as 'locked out', so that no-one else can check it out for modification. Other members of the team can, of course, still have access to the module for reading, however. Even with the sophistication of a version control system, this process leads to frustration when modules are locked for long periods of time. The process breaks down when the current owner of a module is asked to do the editing for another member of the team who is unable to obtain a file. The management of many individual activities can become an exercise in group co-ordination with a consequent decrease in productivity of the team. In an attempt to solve this problem, some version control systems allow software modules to be checked out by more than one member of the team simultaneously. This can lead to a proliferation of versions which have to be merged back together when all modifications are complete, but worse, the success of the merge operations cannot always be guaranteed. The fundamental difficulty in such a collaborative work environment is the granularity of the lock: to add, change or delete only a single character in a file, exclusive access from other programmers must be gained for the whole file. A tool that allows the shared editing of software modules will be very useful for nearly all editing tasks performed by software engineers in large development teams. The problem described above is not unique to the software engineering world. Most editing activities that are large or complex enough to require a team approach will need to solve the problem of how to co-ordinate the various individuals required to work concurrently on the same text file. Folding editors, also known as structure or outline editors, are designed to support source code development work for projects of any complexity. In a collaborative work environment, especially in a large software development team, they can offer many advantages. They inherently provide a powerful information hiding mechanism to enable members of a software development team to maintain an overview of the document/source code under consideration.

Designing a UI for Folding Editors to Support Collaborative Work

371

Very often other team members are only required to have an understanding of the overall structure of the system; they do not need to know the detailed source code of every module in the project. Surprisingly, the use of folding editors in software engineering has often been overlooked. This could partly be attributed to the generally poor design of their interface and the inadequate facilities provided by these editors to support collaborative work.

2. Folds and Folding Editors When we read or look at a piece of text we do not see a sequence of characters. We see sentences, paragraphs and chapters, if the text is a letter or other written document. Even if the text is in a language with which we are not familiar we will still recognise the sentence, paragraph and chapter structure. Likewise, a programmer views the source code of a program as an organised collection of classes, procedures and functions and within these procedures and functions he creates and maintains structured blocks of declarations and statements. One logically expects to find an 'else' block with an ' i f statement, for instance, or a block of variable declarations near the beginning of a program. From a software engineering point of view the structure of a program is the essence of what we describe of a program to somebody else. We might start from the top (with the big picture) and expand our descriptions to describe lower and finer levels of detail. If a program is an implementation of a design, then the structure captures the design. This rationale applies to most other tasks where we describe or explain a thought, idea, or design. Although the development of these ideas and designs is often opportunistic and chaotic (especially when many people are involved) the document that explains these ideas and designs must be structured to explain the concepts clearly. Outlining modes are available in most of the popular word processors on personal computer systems. These outline editors are purpose built for the display and editing of structures based on the heading levels of the document. The content of the headings can be folded or unfolded to enable the user to have control over the level of detail to be viewed. An arbitrary block of text cannot be folded; folds must be created through the use of sub-ordinate levels of headings. Because these editors are first and foremost word-processors using a proprietary format in which the document is stored, they are not suitable as general purpose program editors. Halewood & Woodward (1988) described a syntax-directed, structured Pascal program editor called NSEDIT, which is based on the display and manipulation of Nassi-Schneiderman (NS) charts. The original work of NSEDIT has since been developed into GRIPSE (Halewood & Woodward, 1993) which attempts to cope better with the demands of displaying NS diagrams on limited size screens using a technique that the authors call zooming. Prior to NSEDIT, systems such as Eliot (Jesshope, Crawley & Lovegrove, 1985), LINCKS (Padgham & Ronnquist, 1988), the Inmos' occam Folding Editor (part of the Transputer Development System), Origami and foldED were produced. These systems differ greatly in their user interfaces and in the functionality provided. A fundamental choice confronting the implementors of collaborative systems is deciding between having a central file server or having a distributed system where each user has his or her own copy of the data. The Group Outline Viewing Editor, GROVE (Ellis, Gibbs & Rein, 1991), chooses to replicate keystrokes immediately to all users in the group but, unfortunately, does not support structured information. IRIS (Borghoff & Teege, 1993) is a collaborative

372


editor designed to assist with and improve Software Engineering project management. It supports distributed editing with replicated data and dynamic voting for consistency control as well as mechanisms for notifying users of each other's actions. Individually many of the above systems describe solutions to one or more of the problems of collaborative editing but all fail to present a unifying concept that is useful in both single and multi-user environments. Solutions that do not benefit the occasional collaborative worker who works mostly in a single user environment will not be adopted. Folds provide a means for the user to selectively and gradually introduce the capability to work collaboratively into an editing environment, rather than forcing the user to choose between an editor that is capable of collaborative work and one that is not.

3. Advantages of Folds Whilst there is a substantial difference in the look of folded or partially folded text compared to flat (unfolded) text, the biggest advantages of folds are in the creation of structured text. The text can be created in a top-down manner, progressively adding further details through a process of stepwise refinement. When applied to the process of writing software, the text body for the major folds (most often the functions and procedures) is written initially in pseudo-code using just fold headlines (the single line of text associated with each fold in its closed state). A fold headline is the equivalent of a heading in a word processor. Word processors that support outlining base the document structure on the heading levels. It should be pointed out that the folding feature in Macintosh's Finder and Microsoft Excel is a powerful and effective mechanism for visualising information with a hierarchical structure and it also helps to alleviate the inherent constraints of small display screens. As is apparent to anyone who has used an outlining mode, the speed with which a particular line of text can be reached is quite astounding. The user has very little need for the scroll bar when the document is well structured. A major benefit is the ability to work at a chosen level of detail both at the time of initially writing the text and subsequently when editing the text such as when searching for a problem (bug) or maintaining the code. It provides a means of implementing at the program source code level many of the desirable properties of good software engineering: • Functionality can be encapsulated in a fold. • Detail can be revealed/hidden at will. • Structure can be created and highlighted. • Programs become well-commented through the use of fold headlines. • Block comments have well-defined scopes. • Program development is by top-down and step-wise refinement. • High level design in the form of pseudo code evolves to real code without being lost. If a programmer decides not to take advantage of any of these benefits, a folding editor can still be used as a traditional editor of a stream of characters. This backward compatibility with editors with which programmers are familiar allows their gradual introduction into existing projects. Programmers are not faced with the choice of whether to use a folding editor or not, but rather with whether to use folds or not.


373

program sort; ... what this module does ... function shellSort ... function inputData ... function outputData ... main program

Figure 1: A folded view of a structured program.

4. Requirements of Folding Editors Borghoff & Teege (1993) stress that a structure editor must, in addition to allowing Structure Editing, offer what they call Basic Editing. This is all the functionality of a single-user non-folding editor. In addition, a folding editor must allow the editing of the structure of a document in a manner analogous to that of the text of the document. The three fundamental operations that must be supported are: • Denning a new part. • Removing an existing part. • Moving a part to another position in the structure. The editing of the structure must be co-ordinated with the basic editing. Navigation must be at both the basic and structured levels. Furthermore, folds must persist from one session to another.

5. Structure Commands in menus, toolbar buttons and through the keyboard must be provided to: • Create a fold. • Delete a fold. • Move a fold. • Expand a fold to view fold body. • Collapse a fold to view only the fold headline. • Navigate within the document and to move to previous / next folds. Because of what folds represent in a programmer's mind, they should be able to be easily and directly manipulated.

5.1. Macro Level Structure Creating folds at the macro level provides the ability to view the gross structure of a program. This structure is akin to a function chart which describes the 'big picture' of the program or module. All detail below the highest levels is hidden from view as shown in Figure 1. The high level folds provide a means of quickly stepping to the part of the program where the code is to be examined in more detail by opening the fold of interest.

374

Richard J King & Ying K Leung { find T in table }

{ find T in table }

while hash[H] > T do ... try next address

while hash[H] > T do if H = 0 then ... wrap to highest address else . . . decrement address

Figure 2: A program control structure shown folded and unfolded.

5.2. Micro Level Structure The objective of providing folds at the micro level is to support the hiding of complex code. The programmer reading the program can decide to what level to open the folds to understand the program. Fine detail, perhaps in a programming language with which the reader is not familiar can be ignored — it may well be sufficient to read the fold headlines. Figure 2 shows an example of a micro level fold. Fold headlines are preceded by an ellipse to facilitate easy identification and distinction from the program code. Note that the expanded code contains two nested folds. These can each be opened individually. They may lead to yet further folded code. The reader is free to open the folds to the depth of understanding or complexity required. 6. Multi-user Requirements A system supporting multiple users must be responsive at two levels: each individual user's response time must be short and the time to notify every other user of one user's modifications must also be short. The most important goals for a multi-user folding editor, over and above those for a single-user folding editor, are: • To minimise the presentation and manipulation differences between the single-user and multi-user editors. • To provide the multi-user support as extensions of the single-user interface. • To make the one tool the preferred choice for all editing. • To implement the notification mechanisms, like the fold marks, as unobtrusively as possible. • To allow individuals to come and go at will, i.e. to join and leave an editing session. • To provide indicators for locally edited folds, remote activity, remotely edited folds (which are locked until saved). • To lock (and unlock) folds explicitly to prevent other users from modifying parts of the text while they are being locally modified. • To show the previous unedited version of a fold once it has been remotely edited. 7. GUI Design Issues

7.1. Fold Presentation Style DiSessa (1986) asserts that structure relies on presentation as the major channel of communication to the user. Structured programs are only easier to understand than non-


375

structured programs if the structure of the program can be well presented. We have assumed that the programmer will fold the program text in a logical manner that highlights the structure of the program to best effect. The problem then is one of how best to display the folded and unfolded text. The folds are opened in a manner analogous to a surgical operation where only the folds necessary to get to the point of interest have been opened. The fold structures should intrude as little as possible on the overall look of the program. A possible solution is to adopt a scheme where the program is displayed with all folds in a closed state and each fold tagged in some manner (so that keyboard entry can rapidly open a fold). A fold is opened by entering the fold identifier in a similar manner to many DOS based menu selection programs or by clicking on the fold identifier with the mouse. This is the scheme adopted by Stromfors (1986) and Monk, Walsh & Dix (1988). When a fold is selected the chosen fold is opened and completely replaces the text currently displayed. There is no issue with displaying open/closed fold states. Folds are only ever seen in their closed states with the exception of the fold that is currently displayed. In a collaborative work environment, many design questions arise as to the best way to indicate: • Local and remote changes. • The remote users' foci. • That a fold has been modified and changes have not been saved by a remote user. • That a fold is open but not modified by a remote user.

7.2. Open and Closed Folds The major disadvantage of the above folding scheme is the jump in context whenever a fold is opened. Context is lost both entering and leaving a fold because the currently displayed text is completely replaced. This is a very similar situation to that found in hypertext systems where the user is transported from one context to another with almost total loss of bearings at each jump. Many researchers have reported on the need for hypertext navigation maps because of this loss of context (Beard & Walker, 1990; Delisle & Schwartz, 1986; Halasz, Moran&Trigg, 1987). In order to maintain context, that is to reveal the contents of one structure within the context of an enclosing structure, it is necessary to have the notion of open and closed folds and to have both states visible on screen at the same time. Folds which are collapsed and show only their headline are also referred to as closed. Folds which have been expanded to reveal their contents are also termed open. The use of ellipses as a leader for closed fold headlines (as in the examples below) seems very natural. In common usage it implies that more follows or is to be inferred. Many books containing pseudo code use this style; often however, with no text after the ellipsis (the text being implied). The design of the presentation style for open folds is not so easy. It is highly desirable to be able to indicate the beginning and ending of the opened fold, especially when the open fold body is longer than one screen.

376


7.3. Fold Manipulation A folding editor provides great challenge for taking advantage of a graphical user interface and making use of direct manipulation. The resulting interface must be compatible with existing editors. Cursor keys for moving around text must work as expected but should be naturally extended so the user can navigate the structure and manipulate the folds. Short cuts for fold operations are necessary when users want to keep their hands on the keyboard. Providing hotspots at the fold creases that can be clicked on with a mouse provides an intuitive interface for opening and closing folds to facilitate browsing of the fold bodies. The keyboard can and will often be used alone but browsing text is simplified and sped up if hotspots are introduced into the presentation space. These hotspots must be unobtrusive and cause minimal distraction to the display of the structure and text. Figure 3 shows an example of a folded view and an unfolded view of a program using the interface. Issues remain as to the best way to distinguish the fold headlines from the lines of text comprising the fold body, especially when the fold is opened, and as to whether distinguishing between them is even desirable. Another issue is whether fold headlines are able to be modified at all times. There is no good reason not to allow the fold headlines to be able to be modified at all times. One of the goals of this folding editor was to make the fold headlines look and feel very much like all other text and only differentiate it subtly where and when necessary. Treating fold headlines equally with other lines of text also makes for a less modal editor. Lines of text cannot always be treated equally. When a user marks a block of lines it is only allowable to start and end on the same fold level. The marked lines can however span any number of folds and nested folds. No edit or fold creation operations make sense on text which spans differing levels of folds. Use of the prototype editor indicates that two methods of creating folds should be supported. The first allows existing text to be marked and folded into a single headline. The second allows a fold with no body to be created, knowing that it will later be refined to a further level of detail. The prototype restricts fold headlines to a single line. This is in keeping with the nature of folding: reducing a block of text to a summary line. Allowing only a single line for the fold headline means that pressing the enter key in the middle of the fold headline will not split the line into two lines. Pressing the enter key inserts a blank line as the first line of the fold body. A desirable feature of folds is that they have a default left margin that is not necessarily the same as that of their parent fold. This follows from most block structured languages, encouraging an indented style of program layout. Currently many non-folding editors support this by aligning the cursor on a new line with the first non-blank character of the preceding line. Folds can support an indented block structure more directly and positively. Copying and cutting a fold to the system clipboard raises a question as to how folds should be treated externally to a folding editor. Both folded and unfolded formats need to be copied to the clipboard as the folding editor cannot know whether the fold is going to be pasted back into itself or pasted into a non-folding editor. The same question arises if we choose to print

Designing a Ulfor Folding Editors to Support Collaborative Work WinFold - [c:\richard\winfold\iconinf2.pas] File Edit Search View Fold Window Help program Iconlnfo;

[11

si H

377

1

J

declarations etc

a TMyApplication n methods m TMyWindow nethods m ®

main program

|l«»3«rf2« ool3 a. A folded

;

-

*

- /

program.

WinFold - [c:\richard\winfold\iconinf2.pas] File Edit Search Vjew Fflld Window Help program Iconlnfo; [*] declarations etc H] TMyApplication ffl methods H TMyUindou |5| methods [1 constructor 51 destructor ffl GetUindouClass SetUpUindou procedure TMyUindou.SetUpllindow; begin 13 amend system menu S3 s t a r t o f f displaying free Resources; I I Get Module Handles ThModuleGDI := GetModuleHandle('GDI'); | hModuleUser := GetModuleHandle('USER'); 11 s t a r t timer end; [3 UMQueryOpen d UHDraglcon [+] UMSusCommand M WMTimer

H

Paint

ID main program

b. Opening the folds of the program to reveal more detail. Figure 3: An example of a folded and an unfolded view.

the text of a folding editor. Do we get the text as if all folds were opened or exactly what the user currently sees if he scrolls from top to bottom of the file?

378

Richard J King & Ying K Leung Colour

Indicates

grey (not coloured)

fold unchanged; no remote 'interest'

yellow

local changes made in this fold; additional local modifications okay remote user has focus in this fold, but has made no changes to the fold. Watch out! Remote user has made changes to this fold. Local modifications prohibited (cursor remains at pointer if this fold is opened)

orange red

Table 1: Colour coding scheme of the fold markers.

In a collaborative work environment, fold interaction raises issues of how to indicate: • that an item has just been deleted by another user (For example, if a remote user deletes a fold (open or closed) visible in your local window, then how is this deletion relayed to you.); • the addition of an item by a remote user; • that a remote user is changing an item and consequently it is no longer editable locally. As has been pointed out by Borghoff & Teege (1993) it can be very distracting to have every change made by every remote user affect the local user's screen. Folds do provide a measure of protection from much of the character by character updates that would otherwise be visible in a non-folding environment as the local user will not see any changes taking place in a fold that is closed.

8. Final Design It is only possible to illustrate some of the features of the resulting folding editor in this paper. The essential collaborative feature that resulted from the design work was an extension to the fold hotspots (markers) to convey the status of both local and remote changes and remote focus. Colour coding is an effective means of representation as it requires no additional screen space and can be readily changed to an alternative iconic format. The plus and minus fold markers are colour coded as defined in Table 1. In the case where one fold marker must indicate two statuses the colour nearest the bottom of the table (red) takes precedence. The collaborative features of the final design of the interface is best illustrated by an example captured from the prototype folding editor as shown in Figures 4 and 5. In the following description reference is made to both a local user and a remote user. The term remote user is used to describe another user who may be in the same room or possibly, with a suitable network connection, on the other side of the world. It is even possible that one user could edit the file in multiple windows on the same workstation and be both local and 'remote' (from the editor's point of view) at the same time. Another feature is that the coloured fold markers float up to the enclosing folds if the fold they mark is closed. Likewise, the markers move inside to the changed line(s) within a fold when the fold is opened.

Designing a Ul for Folding Editors to Support Collaborative Work IB WinFold - [c:\richard\winfold\iconinf2.pas] o| File Edit Search View Fold Window Help program Iconlnfo;

379

HUM

1

Hj declarations etc U THyApplication Hj methods remote user has edited this fold H TMyWindow • methods -"'"""^ local user has edited this fold

B main program

T

| line 11 of 30 cd1

\

Figure 4: Remote (red) and local (yellow) change markers.

WinFold - [c:\richard\winfold\iconinf2.pasl File

Edit

Search

View

Fold

Window

Help

program Iconlnfo; ] declarations etc ID TMyflpplication methods local user opens fold edited by remote user

|p methods constructor constructor TMyUindow.Init(AParent: PUindowsObject; begin • do default window initialisation B set window style end; change markers float inside Hi destructor (1 GetUindowClass (1 SetUpUindow S] UMQueryOpen IS UMDraglcon \M UMSysCommand S WMTimer SI Paint

ine1*of37 col6, , , . _ |

Figure 5: Examining a remote user's changes.

The fold markers are dynamic: if while the local user looks on, the remote user saves his changes then the remote markers will revert to normal (grey). The local user would then be free to change any fold the remote user had altered in addition to anything the local user has already changed, of course. Any other users on the system would not be allowed to change what the local user has changed until the local changes are saved. The brief description above has introduced many extensions to the hotspot buttons in the single user folding editor. The following lists provides a brief summary of these features:

380

Richard J King & Ying K Leung • functionality is provided without consuming any extra screen space; • colour coding the fold hotspots indicates local/remote activity; • hotspot revision marks 'float' with fold opens and closures.

9. Concluding Remarks The majority of software engineers who have worked with either a commercial folding editor or who have informally evaluated the prototypes of the folding editor described here, find the folding concept both desirable and appealing. Some may be simply attracted by the novelty of the folding concept, but most are enthusiastic about the benefits of folding in general and quickly understand the advantages that a folding editor can provide in a collaborative work environment. The current prototype has demonstrated that a multi-user editor is possible using folds. Folds provide an elegant solution to many of the issues when multiple users are editing the same file. This paper has also demonstrated that the feature of folding in an editor (rather than this particular folding editor) is beneficial and desirable in both single-user and multi-user environments. Whilst a detailed usability study of the current design is underway, the number of design alternatives for both the presentation and interactive manipulation of folds provides fertile ground for further investigation. A classification of folded text presentation techniques could lead to a more wide-spread adoption of folding (outlining) beyond the areas of software engineering and word processing. The concept of having varying degrees of open-ness for folds, similar to the fisheye view proposed by (Furnas, 1986) should be explored. Other facilities to support collaborative undo mechanisms and version control would be a useful enhancement of the current system.

References Beard, D V & Walker, J Q (1990), "Navigational Techniques to Improve the Display of Large Two-dimensional Spaces", Behaviour & Information Technology 9 (6), pp.451-466. Beaudouin-Lafon, M (1990), "Collaborative Development of Software", in Proceedings of the IFIP Conference WG 8.4 on Multiuser Interfaces and Applications, North-Holland. Borghoff, U & Teege, G (1993), "Application of Collaborative Editing to Software Engineering Projects", ACM SICSOFT Software Engineering Notes 18 (3), pp.56-64. Delisle, N & Schwartz, M (1986), "Netptune: A Hypertext System for CAD Applications", in Proceedings of ACM SIGMOD International Conference on Management of Data, pp. 132-143. DiSessa, A A (1986), "Notes on the Future of Programming: Breaking the Utility Barrier", in User Centered Systems Design: New Perspectives on Human-Computer Interaction, D A Norman & S W Draper [eds.], Lawrence Erlbaum Associates. Ellis, C, Gibbs, S & Rein, G (1991), "Groupware: Some Issues and Experiences", Communications of the ACM 34(l),pp.38-58. Furnas, G W (1986), "Generalized Fisheye Views", in Proceedings ofCHI'86: Human Factors in Computing Systems, M Mantei & P Orbeton [eds.], ACM Press, pp.16-23.

Designing a Ulfor Folding Editors to Support Collaborative Work

381

Halasz, F G, Moran, T P & Trigg, R H (1987), "NoteCards in a Nutshell", in Proceedings ofCHI+GI'87: Human Factors in Computing Systems and Graphics Interface, J M Carroll & P P Tanner [eds.], ACM Press, pp.45-52. Halewood, K & Woodward, M R (1988), "NSEDIT: A Syntax-directed Editor and Testing Tool Based on Nassi-Schneiderman Charts", Software: Practice and Experience 18, pp.987-998. Halewood, K & Woodward, M R (1993), "A Uniform Graphical View of the Program Construction Process: GRIPSE", International Journal of Man-Machine Studies 38, pp.8O5-837. Jesshope, C R, Crawley, M J & Lovegrove, G L (1985), "An Intelligent Pascal Editor for a Graphical Oriented Workstation", Software: Practice and Experience 15, pp.1103-1119. Monk, A F, Walsh, P & Dix, A J (1988), "A Comparison of Hypertext Scrolling and Folding as Mechanisms for Program Browsing", in People and Computers IV, D M Jones & R Winder [eds.], Cambridge University Press, pp.421-435, Proceedings of HCI'88. Padgham, L & Ronnquist, R (1988), From A Technical to a Humane Environment: A Software System Supporting Cooperative Work, North-Holland. Stromfors, 0 (1986), "Editing Large Program using a Structure-Oriented Text Editor", in Advanced Programming Environments—Proceedings of an International Workshop, Conradi, Didriksen & Wanvik [eds.], pp.39-46.

Collaborative Use ofX-Windows Applications in Observational Astronomy Darryn Lavery*, Alistair Kilgour* & Pete Sykes' t Department of Computing Science, University of Glasgow, HLilybank Gardens, Hillhead, Glasgow G12 8QQ, UK. EMail: [email protected] * Department of Computing and Electrical Engineering, Heriot-Watt University, Riccarton, Edinburgh EH14 4AS, UK. EMail: [email protected] 1

Axon Networks Inc., Scottish Software Partners Centre, South Queensferry, Edinburgh EH30 9TG, UK. EMail: [email protected] This paper describes a case study in the design and prototyping of a system to support shared use of application programs in an X Windows environment. The primary aim was to satisfy the requirements for remote observation at Royal Observatory Edinburgh. The starting point for the software development was an existing tool 'Shared-X', developed to support window-sharing in X Windows. The paper describes the analysis of requirements for safe and efficient shared control in the remote observing situation. Previous work in groupware and application sharing is reviewed, and the architecture for the target system is related to existing taxonomies. The modifications that were necessary to the Shared X tool are described, in particular an improved and extended mechanism for floor control, which was found to be an important factor in the acceptability and usability of the system in the target domain. However limitations in the underlying X Windows architecture and having no access to the shared X source code prevented full implementation of the specification for shared telepointers. In conclusion the work highlights the importance of key issues in collaborative system design, including the importance of flexible and transparent mechanisms for floor control, the effective representation of status and control information in the user interface, the need for appropriate support mechanisms in the underlying window system (e.g. for multiple telepointers), and the increased complexity of evaluation with collaborative as opposed to single-user systems.

384

Darryn Lavery, Alistair Kilgour & Pete Sykes Keywords: computer-supported cooperative work (CSCW), groupware, realtime conferencing, shared window servers, Shared X.

1. Shared Observation The requirements for this study arose in the application domain of observational astronomy. For good observing conditions the world's major astronomical telescopes are usually located at high altitude, often in remote areas, to minimise atmospheric effects such as humidity, and interference from artificial lighting. The newest and largest telescopes are in either Hawaii or Chile. Traditionally observation is carried out using the 'visiting observer' method, where the observer travels to the site to perform the observations. Time and money have to be spent in reaching the site, and the visiting observer may have to contend with jet lag and mountain sickness. In an attempt to overcome these problems, telescopes today are being designed and built to support remote observation. Support staff at the telescope site, consisting of a telescope operator and staff astronomer, will perform the scientific programme. One or more remote observers will passively observe the progress of the scientific programme, examining control screens or images produced by the telescope and its instruments. Manual control is not usually granted to a remote observer as it is considered dangerous; unscheduled movement of the telescope could cause injury to the local operator. In special circumstances control may however be ceded, e.g. to an engineer performing diagnostics. Although the remote observer(s) are not in manual control they may have verbal control, as they can alter the course of the experiment by instructing the support staff. This is the procedure currently carried out at Royal Observatory Edinburgh (ROE). Telescope observations are performed at the Joint Astronomy Centre in Hilo (JACH), in Hawaii, and the astronomer remotely monitors the observations from an observation room at ROE. In this environment there is a clear need for application sharing — the remote observer needs to see the output of the programs which the telescope operator is running. The remote observer is usually an occasional visitor to ROE, and has little time for training in the use of observation and conference software. Any system to support remote observing must therefore support close collaboration and be both easy to learn and easy to use. Currently all telescope and instrument software runs under VAX/VMS using ASCII and Tektronix graphics code for output, and a program called TT/Recon (Software Partners) is used to multiplex the output to remote sites. Recently the observatories have started a migration to the use of UNIX workstations with X Windows and Motif as the interface. Because of the number and range of existing application programs, for most of which no source code is available, there were strong reasons in this case to opt for an approach which required no change to these programs, rather than attempting to develop special multi-user applications. A primary software requirement was for a window sharer which allows unmodified single user applications to be shared. A good window sharer offers the possibility to improve remote observing, for example in the area of data reduction, which involves taking the data from an observation and using techniques such as image processing to remove certain effects. Currently this is done live in an observation programme, with the aim of gaining rapid feedback quickly enough to be able to influence the further course of the observation programme and

Collaborative Use of X-Windows Applications in Observational Astronomy 385 is usually done by the remote observer. The staff astronomer may need to look at the data, which means the data must be saved and transmitted to the staff astronomer. Each may use different data reduction techniques, so there is no guarantee they will have consistent views. This task could be improved through window sharing: the reduction software could be shared, so guaranteeing that both parties have the same view of the data, and have a common basis for decision. Software used in Hawaii is tightly integrated with the telescopes and sensors installed at the site and cannot be run elsewhere without these close connections. Hence a sharing architecture based on a central point running the software was required. Several window sharers were investigated but none of them met the needs of remote observing. The one which came closest was Shared X. Before describing this in detail, we set the context by reviewing and analysing current approaches and architectures for supporting shared access and collaborative use of applications software.

2. Groupware and Application Sharing Application programs intended for shared use among a group of users, or utility programs designed to support collaborative problem-solving among geographically separated users, belong to a class of software known as groupware. The term groupware was introduced in (Ellis, Gibbs & Rein, 1991) to describe the products of Computer Supported Co-operative Work (CSCW), an area devoted to studying groups of people and how computers can help them work together. One class of computer-supported collaborative environment is the realtime conference (Ellis, Gibbs & Rein, 1991; Sarin & Greif, 1985) in which all collaborators are active at the same time, although physically dispersed. Support for realtime conferencing usually involves allowing participants to have shared access to one or more application programs. A key question, as discussed in (Ellis, Gibbs & Rein, 1991; Lauwers et al., 1990; Lauwers & Lantz, 1990; Sarin & Greif, 1985), is whether these applications need to be specially written with sharing in mind, or whether shared access can effectively be provided to applications designed originally for single-user access. There follows an analysis of the respective advantages and disadvantages of the two approaches.

2.1. Collaboration Aware Applications In this type of system the application is specially written to support multiple users, see for example RTCAL (Sarin & Greif, 1985) and Colab (Stefik et al., 1987b). Such systems allow parallel activity by each participant, and support different roles with respect to the application by different users. A role carries with it both privileges and responsibilities, e.g. one role may allow the temporary locking of a file to prevent others writing to it. Collaboration-aware applications may also allow different participants to view shared data in different ways, e.g. as a bar-chart instead of a table. A major drawback of this approach is that only applications that have been written for sharing may be used in the collaborative environment. A further problem with such systems is the inherent difficulties they pose for the software developer. Implementing single-user applications is difficult enough, but multi-user systems introduce a whole range of new challenges. Apart from the much more demanding user interface requirements, issues such as communications, synchronisation and concurrency have to be tackled, although some help is available through toolkits, see for example (Knister & Prakash, 1990; Roseman & Greenburg, 1992).

386

Darryn Lavery, Alistair Kilgour & Pete Sykes

2.2. Collaboration Transparent Applications This approach makes it possible to utilise existing applications without modification, by placing a conference agent between the applications and the participants (Ahuja, Ensor & Horn, 1988; Altenhofen, 1993; Crowley et al., 1990; Lantz, 1986; Lauwers et al., 1990; Lauwers & Lantz, 1990). The conference agent is responsible for floor control i.e. managing the input to an application from one of several users and for supporting early leavers and late joiners. Tools in this category vary from shared screen systems such as Augment (Engelbart, 1982) and TT/Recon (Software Partners) to shared window systems such as Rapport (Ahuja, Ensor & Horn, 1988), Shared X (Altenhofen, 1993) and MMConf (Crowley et al., 1990). The principal advantage of this strategy is that the only investment required is in the development of the sharer; no modification of existing application software is required. Users benefit from continuing to use tools familiar from single-user experience, rather than having to learn new collaborative applications to perform the same functions. Also, the need for data translation between private and collaborative applications is avoided. Once a user has mastered the art of sharing, all applications are shareable, as the act of sharing happens outside the application domain and is therefore the same for all applications. On the negative side, only a coarser grain of control is possible with collaboration-transparent applications, as only one participant can interact with the application at any one time. No controls or protocols for shared use are built in to the application itself, which has particularly serious implications for safety critical work. (Some aspects of the shared observation domain fall into this category.) In contrast to collaboration aware applications, use of collaboration transparent applications means that all conferees must have the same view of the application state and any data it is manipulating. This situation is known as WYSIWIS ("What You See Is What I See"), and may be regarded as an advantage, since the shared images act as a common frame of reference for the group. Strict WYSIWIS means that all participants see the same windows in the same place, and see the cursor movements of all participants. However, experience on the Colab system (Stefik et al., 1987a) has shown that strict WYSIWIS is too restrictive, and that for example participants should be able to place windows where they want to.

2.3. Architectures for Sharing A further subdivision among collaboration transparent systems arises from alternative architectures for the control of sharing. The distinction between replicated architectures (where multiple copies of each shared application are allowed), and centralised architectures (where only a single copy exists of each application) is discussed in (Ahuja, Ensor & Horn, 1988; Crowley et al., 1990; Lantz, 1986; Lauwers et al., 1990; Lauwers & Lantz, 1990; Sarin &Greif, 1985). i. Replicated Architecture In a replicated architecture, e.g. MMConf (Crowley et al., 1990), a copy of each application to be shared is present on the workstations of all participants, and the conference agent multiplexes the input of the floor holder to all copies of the application. This means that the same resources (applications, memory, site licences, files etc.) need to be present on all workstations. Better response to the user is achieved as only the input is transmitted along the network; the output from each instance of the application goes only to the local

Collaborative Use of X-Windows Applications in Observational Astronomy 387 window server. This approach is also more versatile, in particular where different varieties of workstation are involved, as the local copy of the application on each workstation can take care of local differences, e.g. in screen resolution, colour, etc. However there is difficulty with this approach in synchronising events across all workstations, and input consistency needs to be maintained, as experience with Rapport (Ahuja, Ensor & Horn, 1988) has shown. Integration of latecomers is also difficult with a replicated architecture, as they need to be given a copy of the current state of the application, which may mean recording all input events since the start of the application. Because of this, MMconf (Crowley et al., 1990) for example does not support latecomers. ii. Centralised Architecture In a centralised architecture, e.g. the later version of Rapport (Ahuja, Ensor & Horn, 1988), the output is multiplexed to all the participants. Software and space requirements are minimised, as files or applications associated with the owner of an application do not have to be present on all workstations. Some synchronisation problems remain, but these are not as severe as with replicated windows. Participants running different hardware and software may still be able to participate. As the application is 'owned' by the user who started it and others are merely supplying an input stream, there may be some unpleasant side effects concerning access privileges and availability of local resources. Further, response is heavily dependent on the throughput of the network as the output frequently generates a heavier load than the input to an application. Use of controls with rapid feedback, e.g. scroll bars, may be particularly onerous. Problems can occur in a heterogeneous environment with differences in screen size and colour maps requiring careful management in the shared environment.

3. The Shared X Window Sharer Shared X 2.1 (Altenhofen, 1993) is the commercial version of Shared X, a window sharer from Digital's assets library available for various UNIX workstations running X. Typically a user may have a window like Figure 1. Shared X provides facilities for starting shared applications, adding or removing users from a shared application, requesting and passing control of an application. Actions on applications are performed by selecting a window of the application with the mouse, and pressing the necessary button e.g. Pass Control. To get a copy of an application a user must ask permission from anyone who is viewing the application. Shared X is only required if the user wishes to start a shared application, and is not required to view another person's shared applications. From general observation, consultation with experts in remote observing, and a usability evaluation of Shared X, the following advantages and problems were discovered. Advantages: i. With the system currently in use at the Royal Observatory, the remote observer chooses which of the screens (telescope status, telescope control, instrument status etc.) to view. The telescope operator need perform no extra work in supporting a remote observer. With Shared X, in contrast, a potential conferee has to ask to be

388

Darryn Lavery, Alistair Kilgour & Pete Sykes -I

JjMotif Chalkbox on Calculator!!! Working in polite mode

nn

alistair

pete

darryn |

You have the chalk add user | delete user| new app

choose appj

quit

1

Figure 1: Floor control in Shared X.

given a copy of the windows of the application, by use of a communication channel. This presents more work to the telescope/instrument operator. Situations such as this, where one user is required to do extra work, but a different user gets the benefit, have been blamed for many groupware failures (Grudin, 1990). However this is not expected to be a problem as it is perceived that the telescope operator would not object to the redistribution of workload or to the monitoring which is a side effect of using a collaborative system. Remote observers will also benefit as they will have less work to do in starting up a remote observing session. ii. The internal architecture of Shared X makes it possible at any stage to share an application which was originally set up as private. This is a very useful feature, as it is not always possible to predetermine when sharing will be required. The lack of such a capability has been identified as a problem with other shared window systems (Lauwers & Lantz, 1990). Disadvantages: i. Telepointing, the ability to replicate the movement of the current user's pointer onto the screens of every observer, is available with Shared X. However as X Windows supports only a single pointer per user, if telepointing is in operation then other users will find their pointers are under remote control, effectively preventing them from interacting with any other application on their screen. ii. All users must have their copy of a shared window in the same place on the screen. iii. Shared X provides a 'Chalkbox' to implement floor control and user management. This Chalkbox appears to function well in a scenario where a number of users are tightly focussed on one task but is not suitable for less intensive intermittent sharing of applications. In particular, it does not support more than one application at a time and it does not inform the user by providing notification about events on any other shared application until that application is selected. iv. Shared X provides two floor control policies, but each has problems for remote observing:

Collaborative Use of X-Windows Applications in Observational Astronomy 389 f Application] Workstation and Chalkbox I Application] • Workstation and Chalkbox

I Application] — * •

/

Shared X bridge

' •*

all events

'

Workstation and Chalkbox

•*

control events

Figure 2: Architecture of Shared X.

• 'polite' With the polite mode the user makes a request for control and the current controller determines whether and when to hand over control. If the telescope operator gave control of the telescope away to a remote observer or engineer, they would have to rely on the other user voluntarily returning control subsequently. • 'anarchy' The anarchy mode allows any user to grab control of an application without asking. It is not clear that social constraints would be strong enough to guarantee that a remote observer would never unilaterally take control of the telescope, with possibly serious consequences for the safety of the local operator. Understandably, the amount of control granted to a remote observer is a sensitive issue to the telescope operator. Shared X is implemented as a client server model. The server (shared X bridge) multiplexes the output of one or more applications to multiple workstations, but only accepts input from one of them. (In practice all workstations send input, e.g. to request or pass control to another workstation.) In X Windows terms, the Shared X server looks like a standard X server, and shared applications can be started by a shell command with the display flag set appropriately. Adding a user to a shared application is done by specifying a workstation display name, although aliases can be used. The Shared X server displays a copy of the specified window on the originating workstation through its local X server. Hence the requesting workstation needs no special software. (Floor control is not enforced at this level.) One of the clients is the chalkbox. This provides the user interface to the Shared X server, through which users gain access to the Shared X functionality, such as starting a shared application, adding or removing users, and high level features such as floor control. Communication between chalkboxes and the Shared X server is implemented via the ICCCM (Interclient Communications Convention) technique (Nye, 1992, Chapter 12). Messages are written as special properties into one of the windows of the application whose sharing state is to be modified. The other windows of that application will also be modified automatically. The Shared X server monitors the window properties and takes the appropriate action. There

390


is no direct communication between the chalkbox and the Shared X server, so that a chalkbox may communicate to a number of Shared X servers simultaneously in one message, and without knowing their location. The Shared X architecture is illustrated in Figure 2. Chalkboxes can communicate with each other by monitoring the status of a shared window. However no locking facilities are provided, so race conditions can occur if two users try to alter the same property for the same application. This method of communication shapes the access control i.e. how a user gets a copy of the windows of an application. For example, a user has no handle on a remote shared application, so cannot alter the properties of its windows. No central system is provided for registering interest in an application. Several advantages accrue from Shared X's adoption of the client/server model. Since the Shared X server looks like a standard X server, it can be built into the window system, so any application can be shared. All existing applications become potentially shareable; it is the user's choice whether sharing of any particular application should take place. Further, since the chalkboxes are separate client processes, any of them can be modified to provide a user interface more appropriate to a particular domain, without affecting the rest of the system. 4. Usability and Security Requirements The remote observer is not allowed control of the telescope. However an engineer may be granted supervised control for diagnostic purposes. A structured control mechanism is needed for safety purposes. The telescope operator must be able to relinquish control, and for safety reasons they should be able to regain control without the engineer giving it up. This can be extended to allow for lists of trusted users. For example the collaborating astronomer could be allowed to get control, when the telescope operator has gone for refreshments. Astronomers visiting Royal Observatory Edinburgh usually have a couple of hours to learn to use the system. Therefore it should be easy for the novice to use but will not compromise the ease of use for an expert. Although informal training or assistance is given through the first few hours of the observation from a member of staff a goal would be to reduce the amount of training needed. 5. A Prototype Support System for Remote Observing Analysis of user requirements, and of the properties of Shared X as described above, indicated that the main deficiency of Shared X in the target application domain lay in its floor control mechanisms. In order to approach more closely the needs of the shared observing domain, an extended and enhanced floor control mechanism was designed, and embodied in a new chalkbox, with the following properties: • A combination of 'polite' and 'anarchy' floor control strategies is supported, in which only the owner of an application may take control of an application without the current floor holder voluntarily giving it up. The owner can also assign privileged status to any other user, which gives the same capabilities as the owner to that user. The owner may also remove a user from the privileged group at any time. • Anyone can add a user to the group sharing an application, but only the owner or a privileged user may remove a user.

Collaborative Use ofX-Windows Applications in Observational Astronomy 391

Help

Options You

XDesigner

alistair

DDD

ODD

bitmap

xterm

pete

ODD

X X

Remove User Give Control to Add Privilege

Figure 3: User interface of new prototype Chalkbox.

• When a user cannot be added to a shared application because they are not logged in or have not set permissions to accept a client from a particular workstation, they are placed on a 'pending' list, and the chalkbox keeps retrying to add them automatically, until either the request is successful, or is cancelled by the originator. • If the Shared X server is used as the default X server, then a private application may be converted into a shared application at any time. Desirable features which were not implemented in the prototype included: • Full telepointing facilities. • Facilities to reconnect when a user is disconnected accidentally, for example in the event of a network failure. As indicated in the discussion of Shared X above, full support for telepointing is not possible with the current release of X Windows, which supports only a single cursor per user. Figure 3 shows the user interface of the prototype chalkbox. A table is displayed which allows the user to see what is being shared, and who the sharers are. Along the top row are the users, and down the left are the applications being shared. So from the table it can be seen that this user is sharing three applications XDesigner, bitmap and xterm with Pete and Alistair. The intersections between the users and the applications show how the user is interacting with the application. For example, Alistair is controlling bitmap, as signified by the mouse. In particular: • The user owns (signified by the border) and is controlling (signified by the mouse) XDesigner, is viewing (signified by the monitor) bitmap, and is controlling and owns xterm.

392

Darryn Lavery, Alistair

Kilgour

& Pete

Sykes

• Alistair is viewing (as signified by the monitor) and is a privileged user of (as signified by the 'P') XDesigner, is controlling (signified by the mouse) and is the owner of (signified by the border) bitmap, and is viewing xterm. • Pete is a pending user (signified by the switched off monitor) of XDesigner (i.e. the system is trying to give a copy to him, but so far has been unable to do so), and is not viewing (as signified by the cross) bitmap and xterm. The table is dynamic, so the states of users and applications will change over time. For example, if Pete is added to the group sharing XDesigner, the icon will change from a cross to a monitor. Any copies of shared applications given to the user will automatically appear in the table, as new application rows. If a user is requesting control of an application, then the icon on the intersection of the user and the application will oscillate between a mouse and a monitor. At each position in the table there is a menu which can be revealed by clicking there. For example, to give control of the xterm application to Alistair, the user would click on the icon and select the Give Control to option. Similarly if no one (except the owner) is viewing the XDesigner application, the user can click on the XDesigner text to reveal a menu, and select the Make Application Private option. The application will then disappear from the table. When there are many users or applications, particularly if the user is sharing with disjoint user groups, the space occupied by the table may become too large. When this occurs scroll bars will be used. However requests for control may be missed if the intersection between the user and application is not in the view determined by the scroll bars. The blank space in the top left hand corner of the table will flash to indicate a request is being made.

6. Implementation Problems The prototype was developed using Motif on a SUN SparcStation. The interface was constructed using an interface builder called X-Designer™ (Imperial Software Technology 1992) and standard C. The functionality for sharing windows was provided by the Shared X server (shXbridge), part of Digital's Shared X tool. As indicated above, the additional functionality required to support the needs of the target domain was added by building a new chalkbox client. Two major problems were encountered during implementation.

6.1. Finding Shared Applications The applications a user is sharing must be found and entered into the table. This is a two part problem: the applications displayed on that user's display must be found, and then it is necessary to determine which of these are being shared. The main window of each of application needs to be identified, so that it can be used by the ICCCM mechanism to pass messages regarding application sharing. (It would also be convenient if this window were guaranteed to exist for the lifetime of the application.) However, there is no automatic way to determine which clients are using a display, as X Windows provides no function call to list all the applications. Further, when starting a new shared application there is no way to determine any of the windows that the new application has started, as there is no necessary connection between a window and the process that created it. Such a facility is needed because when an application is started, a property of the window, used by chalkboxes to communicate with each other, must be initialised. In contrast, Digital's

Collaborative Use of X-Windows Applications in Observational Astronomy 393 chalkbox forces the user to choose the application once it has started; initialisation of the window property takes place then. Further problems include the following: • There is no simple way of determining which windows belong to the same client. However, Shared X can be used to identify which windows belong together by writing into the status property of one window, then looking for windows with that marker, since the property is automatically propagated to all associated windows. (This feature proved useful later in dealing with telepointers.) • An application may have multiple windows, some being transient (dialogues). Even those that are not transient may be only temporary; there is no way to determine if a window is permanent or not. The method used was found by dissecting an XI1R5 client called xlsclients, which lists all of the clients connected to a server. It looks at all the top level windows, and if any of them has a WM.COMMAND property, then this is a main window of an application, and the content of the property is the name and arguments of the application. This was the method followed. However it depends on two assumptions: • That the window identified is the permanent window. • That all X applications set the WM.COMMAND property. According to (Nye, 1992, p.419) the property is used by session managers to indicate how that application can be started and set to its current state. Although there are not many session managers, this convention must be obeyed by all of them for the mechanism to work. However it has been found that some applications do not set WM.COMMAND, and so they are not compatible with the chalkbox.

6.2. Telepointers There should be one telepointer for each application, which should be controlled by the floor holder. It should appear to the other viewers when the floor holder moves the cursor over any exposed region of any of the windows of the shared application, but should not appear at that point for a particular viewer if their copy of the shared window is covered by another window. Problems encountered included: • As before, it was necessary to identify which windows belonged to a particular shared application. This can be via the Shared X server by the mechanism described above. • A shared window will have a particular window identifier, but the corresponding window is not guaranteed to have the same identifier on a different workstation. The window identifier is needed in order to work out if the window is covered by another window when moving the telepointer. Currently the Shared X server forces all participants to have shared windows in the same place. If this ever changed it would be necessary for the Shared X server to keep track of the different positions of the shared windows on the different workstations in order to work out the correct cursor position. This is not a trivial problem, but did not have to be addressed in the current exercise. • A partial implementation was written as an external program to the shared X server. Using the shape extension to X which allows non rectangular windows to be displayed, a window the shape of a pointer was moved around in response to mouse events from the floor holder of an application.

394


To summarise, trying to write a telepointer externally to the Shared X server is difficult: the only reliable strategy would be for the telepointer code to be part of the Shared X server, with support from the standard X server. In the exercise described here, we did not have access to the source code for the Shared X server, so were unable to explore this challenge further.

7. Current Status and Conclusions The current implementation is incomplete. A few minor bugs need to be fixed and the specification fully implemented. The performance of the current implementation could be improved with more development. A preliminary usability evaluation of the prototype system has been carried out, details of which will be reported separately. This confirmed that the system has the potential to meet the needs of remote observing. The prototype was favourably received by a sample of potential users, but first time users had problems due to the lack of affordance of the table elements in suggesting their pop-up menu capability. An improvement here might be to highlight the icon when the mouse pointer moves over it. Further research is also needed on effective ways of providing feedback to a participant from events generated by other users. The next stage would be to develop a second prototype and test it in a fully-fledged remote observing session. However, access to the Shared X code would be essential if the full benefits from the current testing were to be realised. Although focussed on the needs of a particular application domain, this study has highlighted general issues concerning both the interface design and the underlying architectural features necessary for effective exploitation of cooperative computing in a distributed windowing environment. These include better support from the window system in identifying windows associated with each application, multiple pointers for each user, and virtual mapping of windows onto screen positions (so that copies of shared windows need not be in the same physical location on each screen). Some of these may be addressed in future release of X Windows; we have not investigated how well they are addressed in other window systems. As with all interactive applications, performance is an issue. It is of particular importance in collaborative environments, where rapid feedback is essential to assure each user that their actions have been seen and accepted by all collaborating sites. Although improved network and workstation performance may in time solve most current speed problems, it may also be worthwhile to give attention to finding the bottlenecks and optimising the associated parts of the window and networks systems. The study also showed what we believe to be a general requirement for flexible yet secure floor control mechanisms, with an intuitive user interface giving a clear indication of which applications are shared, and how the sharing status may be changed. The client/server architecture of Shared X separates the floor control from the Shared X server, and hence allows a new floor control policy to be implemented independently of the server, without compromising the user's ability to share any other applications outwith the test domain. This feature is a major strength of the Shared X architecture. Finally, our experience reinforces the view that user interface design and user-centred evaluation are even more important in collaborative environments than in single user environments, and correspondingly more challenging.

Collaborative Use of X-Windows Applications in Observational Astronomy 395 Acknowledgements The work is derived from the first author's MSc dissertation (Lavery, 1993) and was funded by the SERC. The authors would like to thank Malcolm Stewart and Peredur Williams at the Royal Observatory Edinburgh for their active participation during this project. Joachim Schaper of Digital kindly lent the Royal Observatory a copy of Shared X, which made this project possible. We would like to thank the subjects who participated in the usability evaluations and interviews. We would also like to thank the anonymous reviewers for their helpful comments.

References Ahuja, S R, Ensor, J R & Horn, D N (Mar 23-25 1988), "The Rapport Multimedia Conferencing System", in Proceedings of the Conference of Office Information Systems, ACM Press, pp.1-8. Altenhofen, M (1993), SharedX User Manual, Digital. Crowley, T, Milazzo, P, Baker, E, Forsdick, H & Tomlinson, R (1990), "MMConf: An Infrastructure for Building Shared Multimedia Applications", in Proceedings ofCSCW'90: Conference on Computer Supported Cooperative Work, ACM Press, pp.329-342. Ellis, C, Gibbs, S & Rein, G (1991), "Groupware: Some Issues and Experiences", Communications of the ACM 34(l),pp.38-58. Engelbart, D (1982), "Towards High-Performance Knowledge Workers", in Proceedings ofOAC'82. Grudin, J (1990), "Groupware and Cooperative Work: Problems and Prospects", in The Art of Human-Computer Interface Design, B Laurel [ed.], Addison Wesley, pp.171—186. Knister, M J & Prakash, A (1990), "DistEdit: A Distributed Toolkit for Supporting Multiple Group Editors", in Proceedings of CSCW'90: Conference on Computer Supported Cooperative Work, ACM Press, pp.343-355. Lantz, K (1986), "An Experiment in Integrated Multimedia Conferencing", in Proceedings of CSCW'86: Conference on Computer Supported Cooperative Work, D Peterson [ed.], ACM Press, pp.267-275. Lauwers, J C, Joseph, T A, Lantz, K A & Romanow, A L (1990), "Replicated Architectures for Shared Window Systems: A Critique", in Proceedings of the Conference of Office Information Systems, ACM Press, pp.249-260. Lauwers, J C & Lantz, K A (1990), "Collaboration Awareness in Support of Collaboration Transparency: Requirements for the Next Generation of Shared Window Systems", in Proceedings ofCHI'90: Human Factors in Computing Systems, J C Chew & J Whiteside [eds.], ACM Press, pp.303-311. Lavery, D (1993), "Window Sharing Requirements for Remote Observing", Heriot-Watt University, Edinburgh, Scotland, MSc Dissertation. Nye, A (\992), Xlib Programming Manual, O'Reilly & Associates, Inc. Roseman, M & Greenburg, S (1992), "GROUPKIT: A Groupware Toolkit For Building Real-time Conferencing Applications", in Proceedings ofCSCW'92: Conference on Computer Supported Cooperative Work, J Turner & R Kraut [eds.], ACM Press, pp.43-50. Sarin, S & Greif, I (1985), "Computer-based Real-time Conferencing Systems", IEEE Computer 18(10), pp.3348.

396


Software Partners, TTIRecon: The VAX/VMS Terminal Reconnaissance Tool, Software Partners Inc. Stefik, M, Bobrow, D, Foster, D G, Lanning, S & Tatar, D (1987a), "WYSIWIS Revised: Early Experiences with Multiuser Interfaces", ACM Transactions on Office Information Systems 5 (2), pp.147-167. Stefik, M, Foster, G, Bobrow, D G, Kahn, K, Lanning, S & Suchman, L (1987b), "Beyond the Chalkboard: Computer Support for Collaboration and Problem Solving in Meetings", Communications of the ACM 30(1), pp.32-47.

Que Sera Sera — The Problem of the Future Perfect in Open and Cooperative Systems Alan Dix School of Computing and Mathematics, The University of Huddersfield, Queensgate, Huddersfield HD1 3DH, UK. EMail: [email protected] When the pace of interaction with a task is too slow, the user's execution/evaluation loop is broken. Feedback normally says what has happened. However, when the task is slow, nothing has happened yet — the interest shifts to what will have happened. This poses two problems for the user. Firstly, recalling the appropriate context when a response eventually arrives. Secondly, maintaining the expectation that the response will come and when, so that appropriate action can be taken if it fails to materialise. The design question is how to support these activities, of which the latter has received little emphasis to date. Keywords: cooperative work, computer-supported cooperative work (CSCW), delayed feedback, status/event, response time.

1. Raison d'etre A kitchen, somewhere in Cumbria, 7th November 1992, 8:37 am Alan puts bread under the grill and waits for it to brown. Email message received May 1993 From: forsyth Subject: To: alan yes

15:35:13

Wed 5 May 1993

2. Background A previous paper (Dix, 1992) discusses the problem of pace in cooperative systems. That paper concerned the relationship between the pace of communication channels and the pace of the tasks performed cooperatively using those channels. The pace at which communications

398

Alan Dix

occur through a channel is the pace of interaction and is determined in part by the properties and intrinsic pace of the channel and in part by the pace of the task. Mismatches of pace lead to problems, both when the pace of interaction is too slow for the task, and when it is too fast. However, people are skilled in developing coping strategies which alleviate these problems. This paper is related to the above work in that it concerns issues of time and pace of interaction. However, the focus here will be on the relationship between the pace of interaction (whether determined by communication channels or the task itself) and the pace of the users themselves. There are certain timescales which have qualitative differences for users. Two timescales will be important: • Short term memory — From a cognitive viewpoint, we have a timescale determined by short term memory (Dix et al., 1993). Because of interference, this timescale is itself influenced by other tasks. • Task switching — Either by choice, or because of interruptions, users may switch between tasks. This puts an upper time limit on the consecutive time spent on any particular task. There is a large body of research on the effects of response time on user performance and behaviour — e.g. (Dix, 1987; Shneiderman, 1984; Teal & Rudnicky, 1992). In most studies, the user is involved in a single task and is continually waiting for the system response. However, this paper will be primarily concerned with very long responses (from minutes to days) and the user will be expected to engage in other tasks between initiating an action and receiving a response.

3. The Broken Loop Norman's execution/evaluation cycle has been an influential model of the way humans interact with their environment (Norman, 1988). Norman considers seven stages of interaction, but for the purposes of this paper, a simplified version, suffices (Figure 1): • Execution — Having established goals, the user decides what to do and does it. In addition, the user will have some expectation as to the effect these actions will have on the system. • Evaluation — The user interprets the system's feedback in the light of the expectations generated as part of the execution stage. As a result of this goals may be modified and a new execution/evaluation cycle may begin. Notice that in this description, the user is very proactive acting with intention to bring about some desired state of the system. When the Norman model is applied to computer systems it is normally assumed that having completed the execution side the user then waits for the system feedback and then completes the evaluation stages. In such contexts, the system is assumed to be closed — the system is controlled by the user alone, with no interaction with other users or the external environment. Where the response time is suitably fast, the whole cycle can take place within the user's short term memory span. With cooperative or open1 systems this simple application fails. Consider the sending and receiving of an email message. This may be one 'cycle' within a task yet may take hours, 1

That is systems which interact with the external environment.

Que Sera Sera

399

goals

/

expectations

\ evaluation

execution

system

Figure 1: Norman execution/evaluation loop.

days or weeks to complete. Similarly, consider a factory. The pressure within a vessel is too low, so the operator turns on a pump, which will slowly increase the pressure. However, it may take minutes or hours for the vessel come up to the required pressure. In these examples, the cycle is no longer completed within short term memory span — the loop is broken. In a closed system feedback tells the user about what has happened. In an open system, the emphasis changes to what will have happened: that a reply will eventually have been received, that the vessel will have reached its required pressure. The user knows now what to expect and yet may have to wait a substantial time before the desired result is obtained. Arguably, models arising from industrial ergonomics are more appropriate here. These treat the user more as responding to stimuli from the system/environment: e.g. when the alarm rings, the operator turns off the pump. In a sense, this sees the loop as broken at the top (the user end) rather than at the bottom (the system). However, it radically changes the view of the user. The user is now reactive rather than proactive. This may often be the case in a large and complex system: the operator may not have a model of the system or any particular goals, but may be regarded as a cog within the whole. Perhaps a more acceptable way of looking at this would be within the remit of distributed cognition — the intentionality sits within the whole human/machine system rather than in the user per se. That is, one might argue that even when the user is acting purely reactively, one can see plans and intentions being acted out by the system as a whole. Whether or not there is a larger view of intentionality, many actions are part of larger planned sequences. So, we still need to retain the essentials of the Norman loop, that action is purposeful, designed by the user to carry forward some aim or goal. In this case, the user faces two problems: • Restoring context — By the time a response comes, the user may have forgotten what it was about. For example, consider the email message which began this paper: "yes" — yes what?

400

Alan Dix • Maintaining expectations — As well as remembering what should happen, the user must remember that it should happen at all. The user can no longer simply wait for feedback, but must remember that some response is expected in case a failure occurs and the response never comes.

Of these two issues, the first is reasonably well studied. In the groupware literature, we have examples such as Object Lens (Malone et al., 1987) where structured messages contain some indication of their source. At the level of social protocols, email users often include a copy of the senders message with their reply. In an industrial setting the fact that dials and alarms are associated with particular items of physical equipment acts as an element of context in its own right. However, the setting does not tell one why the event has taken place, the location may be a useful reminder, but the user still has to recall the rationale. Furthermore, in complex control rooms, where the alarms are no longer physically associated with devices, even establishing the locus of an event becomes a non-trivial task. In contrast, the overhead of maintaining the expectation of a response is not widely recognised, perhaps because it is not explicit in the normal feedback loop. The expected feedback can be of two kinds: event or status — see also (Dix et al., 1993, Section 9.4). Events, such as alarms or the receipt of an email message, can be assumed to be noticed when they arrive, but the user must take action if no event occurs within an acceptable time. Status feedback, such as a temperature dial, is more complicated. The required feedback is that the status eventually attains some desired value. The achievement of this value is an event in itself, a status-change event, but detecting that this event has occurred requires some positive action by the user. For both kinds of feedback an additional burden is placed on the user, but harder than simply restoring the context. In this case, the user has to recall unprompted that certain responses are expected to occur. Human memory is very efficient at recall from stimuli, but less so at this form of spontaneous recall. Note that both the Norman loop and the reactive model of the user are silent on the second of these two problems. The explicit evaluation in the Norman model reminds one that context is required, but there is no similar placeholder for maintaining expectations — it is implicit in the closed loop. Similarly, the reactive view emphasises the user's formulation of a response to a stimulus and thus suggest that some context may be required to frame that response. However, a purely reactive model does not suggest actions when no stimulus occurs. Note that in both open and cooperative systems, the user's own system is acting in a mediating role 2 . Rather than working on the system, the user is working through the system to the world beyond. The feedback which we have been discussing above has been that of the 'world' (other users and physical processes), but one should not forget the immediate feedback from the local system. The user needs confirmation of an action at two levels (Figure 2): • That the action has been received — from the system — fast. • That the action has had the desired effect — from the world — slow. In closed systems, the distinction between the two is often elided, but even there may be important (Dix et al., 1993). Indeed, the whole direct manipulation style is built upon the hiding of the distinction. In contrast, the designer of an open system will need to offer feedback at both levels and will need to distinguish the two. 2

This is similar to Hollnagel's (1993) 'linked HCI'.

401

Que Sera Sera application feedback ^

r—^ (

User

V

immediate feedback

^—-^C— user actions

Figure 2: Mediated interaction.

The rest of this paper will concentrate on the problem of maintaining expectations of responses as the problems of immediate feedback and restoration of context are relatively well understood.

4. Solutions The problem of delayed response is common in everyday life. We would therefore expect to find coping strategies adopted in various circumstances. This can lead to effective design guidelines as the system can offer support for the user's own, often effective, coping strategies. In addition, there are additional solutions which are made possible by the presence of a support system. The fact that interaction is mediated by the user's local system means that it can record the user's actions on the world and process the responses. This has obvious implications for restoring context for the users, but can also help in the maintenance of expectation. We can classify potential solutions by who or what takes responsibility for ensuring that the user's actions have their desired response: the user, the world (including other users) or the local system. We start with the responsibility firmly with the user — the aide-memoire.

Aide-memoire From the knotted handkerchief to the Post-It note and 'to do' lists, people are well aware of their fallibility and put things in their environment to remind them. However, looking more closely, one finds that these methods are principally used for reminders that something has to be done. For recording expectations about others, people are far more sloppy. Office procedures are a classic example, being typically event driven. Your solicitor sends a letter on your behalf, but does nothing until either a reply arrives, or you ring up to pester. The better professionals will put some note in their diary of when the response is expected. Even where a diary or calendar is used, they themselves require 'monitoring' and significant events can be missed (Payne, 1993). One can augment these normal forms of aide-memoire with electronic equivalents. For example, the "Cc:" facility on an email system can be used to automatically post outward going mail into ones own mailbox. It can then be left there until a satisfactory reply arrives. The two problems with such techniques is the cost of putting in the information and the noise from having too many reminders around.

402

Alan Dix

As a design concept, the aide-memoire can be applied in other contexts. For example, consider again the example of the operator wishing to increase the pressure in a vessel. Imagine adding a small red needle to the pressure dial, which can be set by the operator. When the operator opens the valve the dial is also set to the desired pressure. Then, when the operator next scans the control panel, any dial where the needle value and the actual value differ demand attention. Of course, the difference may be acceptable as sufficient time has not elapsed in which case the operator can simply ignore the discrepancy until next time the dials are inspected. Note how the use of an aide-memoire shifts the burden from maintaining expectation to restoring context. The fact that the aide-memoire is there reminds one that something should happen (or be done), which then means that one has to remember what it is. In the case of the needle on the dial, the aide-memoire is part of the items to which it refers, the only remaining job for the operator is to recall whether the target pressure should have been reached yet. There are problems with aides-memoire, consider an extreme case. One year you forget your sister's birthday, so you tie a knot in your handkerchief so that you don't forget next year also. The disadvantages are obvious — a (very dirty) handkerchief with 73 knots and you can't remember what any of them were for. When considering system support, the characteristic feature of an aide-memoire is that the information it captures is uninterpreted it is the user who assigns meaning to it. However, the electronic domain can add value, for example, the user may be also able to link a reminder to the screens/applications required for interpreting it.

Trust and Predictability The reason that many office procedures are event driven is that this deliberately removes the need to maintain expectations. The individual can perform a task, send results or requests to someone else, and then reach closure, not considering the task again until some further stimulus (usually piece of paper!) arrives. Such systems are built upon trust: having asked someone to do something, one trusts that they will perform the action, having requested information, one trusts that the reply will be received. Note that this effectively shifts the burden of responsibility from the sender to the recipient. Predictability of physical and electronic systems has a similar effect. If one knows that increasing the inflow to a vessel by 10% always increases the pressure by 5% within 20 minutes, there is no need to monitor the process. In some sense trust is a particular form of predictability. However, shifting responsibility to another individual has different implications to shifting it to a machine or process. In the former case, one has someone else to blame when things go wrong! Of course, things do go wrong, especially in an office situation where the chains may become quite long, often crossing corporate boundaries. In control engineering terms, using predictability allows one to swap closed loop control for open loop control. The problem of delayed feedback ceases to be a problem because feedback is no longer required — when something is asked to be done, one assumes it will be done. However, open loop control is inherently more fragile and demands total knowledge and trust. So, because things don't always happen as expected, even where open loop control is used, it will often be enclosed within some wider procedure where feedback is monitored.

Que Sera Sera

403

System support here would be ways of annotating requests to make the shift of responsibility more obvious. This is particularly a problem with email communications where different corporate (and individual) cultures put a different emphasis on the 'strength' of messages. In some email cultures, ignoring email is allowed in a way that ignoring a paper communication would not be. A system like Coordinator (Winograd & Flores, 1986) at least made these mutual expectations clear.

Tell the System Of course the system can do more to help if it knows what the user's expectations are. As the system mediates the feedback from the world, the user could explicitly tell it when certain responses are expected. For example, when sending an email message, one could have a 'reply by' field, which need not be sent to the recipient (but may be). This system could then alert the user if no reply is received by the specified time. The same technique can be applied to industrial situations, for example, telling the system that vessel A should be at an operating pressure of 4 bar within 20 minutes. The advantage is that the responsibility has now moved entirely to the system. If the reply is not received or the vessel does not come up to pressure, the system alerts the user. Once the user has told the system nothing needs to be done until some response or alert is received — a form of non-military 'fire and forget'. The problem with such explicit passing of responsibility is the effort it places on the users, both in terms of the time taken to enter the data and the complexity of externalising their expectations. For large scale tasks this may not be a significant barrier, especially if the applications are closely linked to familiar tools such as diaries or PERT charts where the significant times can be entered. However, as the pace of activity increases, the load becomes unacceptable. Even for email communication, with rates commonly between 20 and 100 messages a day, explicitly setting 'reply by' dates is likely to be taxing. An alternative is to make this passing of responsibility implicit. This can be achieved by system prediction followed by user confirmation. Various office tools already use predictive features as task accelerators, for example, the latest versions of Microsoft Excel have intelligent sum operators which guess what you want to sum. In a similar way, an email system could guess the expected reply date based on the normal rate of interchange with the recipient. This could then be used as a default for the 'reply by' field. Alternatively, user defined rules could be used in a structured message system to determine the reply time as a function of the recipient, message type etc. The important thing is that such techniques accelerate the process and require only confirmation (which may even be silence). In an industrial setting predictive interfaces are already used, but with an emphasis on planning. The operator enters planned actions and the system shows the projected changes based on some model of the physical process. Given the system has supplied predictions it is not unreasonable to expect it check those predictions and inform the user when the system falls outside those parameters. It is interesting to note that if predictive systems took on the responsibility to alert users, they would become more honest. Imagine a colleague gives you some information upon which you act. Later your colleague becomes aware that the information is wrong. You would be rightly aggrieved if your colleague did not inform you. This is precisely the situation with most predictive interfaces!

Alan Dix

404 BRITANNIA

180

300

CREWE 9 . 25

BAL770 5423 M/B737/C

T420

EGGW UA2 UB3 UB4 EGAA

Figure 3: Air traffic control flight strip.

5. Two Case Studies We have now seen several ways of dealing with the problem of delayed feedback. We will now look at what these tell us about two specific scenarios.

Air Traffic Control Air traffic control is a complex activity. It involves coordination between the controllers within the ATC centre and between the controller and the aircrew. Furthermore, it involves the physical movement of aircraft. That is it is open both by virtue of cooperative activity and physical processes. One critical part of the ATC process is the use of flight strips, small strips of coloured paper referring to individual planes (Figure 3). On this is printed and written various information, some constant (usually) such as the destination, and some variable such as the flight level (height). When the controller asks the aircrew to take the aircraft to a new flight level there is a delay before that height is reached, due to the communication and to the physical behaviour of the aircraft. Furthermore, the controller's attention cannot remain with that aircraft, but must shift to the many other tasks within the air sector. One would therefore expect to find some form of aide-memoire or other means to cope with the delayed feedback, and the obvious locus for such information is the flight strip. As predicted the flight strip is used in precisely this way: "When a controller gives an instruction to a pilot, for example to ascend to flight level 220, s/he marks this on the strip: in this case, with an upwards arrow and the number 220 (Figure 4.i). When the pilot acknowledges the instruction, the controller crosses through the old flight level on the strip {Figure 4.ii). When the new height is attained, the controller marks a check beside it (Figure 4.iii)." From (Hughes, Randell & Shapiro, 1992), figure references added. Note that the flight strip acts as an aide-memoire first for the completion of the communication task and then for the completion of the whole task. A quick scan of the flight strips (probably not even conscious) will reveal all those with two flight levels (indicating that confirmation is still awaited) or those with unchecked flight levels (confirmed, but not attained). Clearly any system to automate flight strips must either replicate this facility or produce some equivalent one. For example, if flight strips were on-line then the system could automatically check when aircraft did not reach their desired height. The choice between such alternatives is not simple as a single mechanism may fulfill several roles, but the analysis both points one towards the existing mechanisms and gives one a repertoire of alternatives.

405

Que Sera Sera i. Controller gives instruction to a pilot "ascend to flight level 220" 9 .37

BRITANNIA

BTN

180 220

^ ^

300

CREWE

9.25

CREWE

9.25

CREWE

9.25

BAL770 5423 M/B737/C

T420


ii. Pilot acknowledges the instruction 9.37

BRITANNIA

BTN

220 f

300

BAL770 5423 M/B737/C T420


Hi. New height is attained 9.37

BRITANNIA

BTN

/

/220 [

300

BAL770 5423 M/B737/C

T420


Figure 4: Flight strip is annotated to record expectations.

Car radio tuning We now look at a problem which arose during the design of a new car radio. The interface to the radio is located at the front of the car and has the familiar tuning buttons and frequency display. A user interface micro-processor handles this interface. However, the actual radio tuner is controlled by a separate micro-processor in a different part of the car. The two microprocessors communicate via a (not too fast) data bus. The data bus introduces delays and thus we are considering a mediated interaction as described earlier. The timescales involved are actually quite short, possibly a second or two at very worst, probably only a fraction of a second. However, the driver is (one hopes) involved in a variety of other, more demanding tasks, such as keeping the car on the road. Thus even these comparatively short timescales lead to a broken execution/evaluation loop. The specific design problem faced was the behaviour when the user pressed the tuning buttons. The interface controller would send a message to the tuner asking it to retune. Two options for the feedback were initially considered: i. Display the new frequency immediately the user pressed the button, ii. Wait until the tuner responds that it has retuned. The rationale for the former is that the user ought to gain immediate feedback, the justification for the latter is that the display ought always to reflect the actual state of the system. In fact, this is a highly complex situation. There are two feedback paths to the user: the frequency display and the sound of the radio itself. These two are usually in agreement, and the driver will be unaware of the distributed nature of the cars computing architecture, so when discrepancies arise there is liable to be some confusion. In addition, the visual feedback has two functions. On the one hand it gives feedback on events: "your button press has been noted". On the other hand it gives a continuous status display: "the channel you are listening to is 92.5MHz".

406

Alan Dix

From our previous analysis of mediated interaction we see that feedback is required at two levels: • Interface — that the user's button press has been registered • Application — that the channel has actually changed Option (ii) gives no immediate interface feedback. If the network lags are short enough this may not matter, but anything more than a few hundred milliseconds is unacceptable for button press feedback. The result is likely to be that the user will repeat the button press, leading to an overshoot of the desired station and 'hunting' behaviour. In fact, this scenario is more likely to affect the passenger as the driver may operate the radio by touch whilst keeping attention on the road. Tactile feedback from the buttons is thus likely to dominate for the driver. Application feedback comes through the sound of the radio in both cases. The justification for option (ii) says that there should be a status-status mapping between the state of the radio and the frequency displayed. However, where a status-status mapping is mediated by messages there is inevitably some discrepancy. In option (i) the frequency display would be ahead of the actual tuned frequency, but option (ii) suffers as the display would lag behind. Thus there will always be some period of discrepancy and if this is not to cause confusion some means of flagging it should be supplied. We now have two requirements: to show immediate feedback and to indicate discrepancy in the displayed frequency. We can now consider a third option: iii. Some instant display change to an 'undetermined' state — that is some display which indicates that change is in progress. There are various ways of showing this: a. Spinning dials — The digits of the display quickly alter as if they were searching for the new frequency. This would have the advantage of giving an immediate sense of change, but would mean that there would be no feedback of where that change is going. Thus if the user hit the wrong key (tune down rather than up), there would be no feedback until the system had stabilised. b. Flash the number—either the new number (answering the criticism of (iiia), or alternate between old and new. Possibly this would be too distracting. c. Busy indicator — Have an additional busy indicator so that the user knows that the action is not complete. Like the ATC case study, the final choice of solutions depends on a variety of factors (e.g. must use standard radio display screen), the essential feature is that the analysis highlights the central elements of the design.

6. Summary We have seen how in open and cooperative systems there is often a lag between a user's action and the associated feedback. This breaking of the traditional execution/evaluation loop puts extra burdens on the user: to recall the context when a response arrives, to maintain the expectation that an response should arrive. We looked at four potential solutions to the latter problem:

Que Sera Sera

407

• Aide-memoire. • Trust and predictability. • Explicitly telling the system — passing on responsibility. • Implicitly telling the system by confirmation of system predictions. We can analyse existing situations to uncover uses of these solutions and also use them to generate design alternatives. The former is important as otherwise existing coping strategies may be spoilt by the introduction of new support systems. Two case studies have revealed how these problems occur in practice and demonstrate some of the resulting design space. In both cases, the analysis suggests various alternatives, but the choice depends on other factors. The importance of the analysis is that from seeing that a phenomena or problem exists, we can go to its underlying cause and thus open up the possibilities for solution.

Acknowledgements This work was funded by SERC Advanced Fellowship B/89/ITA/220 and SERC grant GR/F/01895. Thanks to the anonymous referees who gave several pointers to related work and to those who have listened and commented on earlier verbal presentations of this material.

Raison d'etre (continued) A kitchen, somewhere in Cumbria, 7th November 1992, 8:39 am Alan awakened from reverie by the smell of burning toast.

References Dix, A (1992), "Pace and Interaction", in People and Computers VII, A Monk, D Diaper & M Harrison [eds.], Cambridge University Press, pp.193-207, Proceedings of HCI'92. Dix, A, Finlay, J, Abowd, G & Beale, R (1993), Human-Computer Interaction, Prentice-Hall. Dix, A J (1987), "The Myth of the Infinitely Fast Machine", in People and Computers III, D Diaper & R Winder [eds.], Cambridge University Press, pp.215-228, also reprinted in Dix, A J (1991) Formal Methods for Interactive Systems, Academic Press. Hollnagel, E (1993), "The Design of Reliable HCI: The Hunt for Hidden Assumptions", in People and Computers VIII, J L Alty, D Diaper & S Guest [eds.], Cambridge University Press, pp.3-15, Proceedings of HCI'93. Hughes, J A, Randell, D & Shapiro, D (1992), "Faltering from Ethnography to Design", in Proceedings of CSCW'92: Conference on Computer Supported Cooperative Work, J Turner & R Kraut [eds.], ACM Press, pp.115-122. Malone, T W, Grant, K R, Lai, K & Rosenblitt, R Rao and D (1987), "Semistructured Messages are Surprisingly Useful for Computer Supported Coordination", ACM Transactions on Programming Languages and Systems 5(2), pp.115-131. Norman, D A (1988), The Psychology of Everyday Things, Basic Books. Payne, S J (1993), "Understanding Calendar Use", Human-Computer Interaction 8(2), pp.83-100.

408

Alan Dix

Shneiderman, B (1984), "Response Time and Display Rate in Human Performance with Computers", ACM Computing Surveys 16 (3), pp.265-286. Teal, S L & Rudnicky, A I (1992), "A Performance Model of System Delay and User Strategy Selection", in Proceedings of CHI'92: Human Factors in Computing Systems, P Bauersfeld, J Bennett & G Lynch [eds.], ACM Press, pp.295-3O5. Winograd, T & Flores, F (1986), Understanding Computers and Cognition: A New Foundation for Design, Addison Wesley.

Viewpoints, Actionpoints and Spatial Frames for Collaborative User Interfaces Steve Benford* & Lennart E Fahlen* * Department of Computer Science, The University of Nottingham, Nottingham NG7 2RD, UK. Tel: +44 (0)602 514203 Fax: +44 (0)602 515254 EMail: [email protected] * Swedish Institute of Computer Science (SICS), Box 1263, S-16428, Kista, Stockholm, Sweden. Tel: +46 8 7521539 EMail: [email protected] Synchronisation is a key issue for collaborative user interfaces. An examination of current approaches, in particular the concept WYSIVVIS and the use of Video as a communication medium, highlights a number of issues in this area including lack of a common spatial frame of reference, lack of appropriate embodiment of users and inflexible and rigid communication channels between users. The paper then proposes a new framework for designing collaborative user interfaces which addresses these issues. This framework is based on the notion of a common spatial frame within which embodied users are free to move autonomously, being casually aware of each other's activities. Embodiment is considered in terms of both individual viewpoints and actionpoints (e.g. telepointers) within the display space. We propose that, in many cases, synchronisation of the spatial frame is necessary but synchronisation of viewpoints and actionpoints may actually inhibit collaboration. We finish by describing some prototype systems which provide one (of possibly many) examples of how our framework might be employed; in this case to create shared cooperative virtual environments. Keywords: synchronisation, video, WYSIWIS, virtual reality, multi-user interface, computer-supported cooperative work (CSCW).

410

Steve Benford & Lennart E Fahlen

1. Introduction Collaborative user interfaces, particularly shared workspaces, have been the focus of considerable research effort in recent years. Resulting systems include multi-user editors and drawing tools (Ellis, Gibbs & Rein, 1991; Foster & Stefik, 1986; Greenberg & Bohnet, 1991), shared screen systems and more specialised design surfaces (Ishii & Kobayashi, 1992). There has also been a growth in the use of multi-media technology to support communication and awareness between the users of such systems including conferencing systems (Sarin & Greif, 1985) and media-spaces (Gaver et al., 1992; Root, 1988). Determining an appropriate degree of synchronisation across multiple user interfaces is a key issue for such systems. At one extreme no synchronisation, where users may hold entirely unrelated views of the universe, may make collaboration virtually impossible. Perhaps more surprisingly, the other extreme, that of complete synchronisation, may also inhibit collaboration due to loss of individual autonomy and the overhead required to maintain common views. The nub of the synchronisation issue rests on the tension between group and individual needs, an issue across the whole of CSCW. This paper proposes a theoretical design framework for collaborative user interfaces that aims to achieve an appropriate degree of synchronisation. To peer ahead for a moment, this framework will stress the importance of providing a common spatial frame of reference for multiple users and the separation of this frame from the selection and presentation of displayed objects and the individual perspectives of the users. Our use of 'space' in this context is in a very general sense, i.e. an arbitrary mapping of any kind of dimensions of information into a landscape where some useful sense of location, direction and movement can be established (Benedikt, 1991). The framework will also focus on the issue of appropriate embodiment of users, including the separation of their viewpoints from their actionpoints. This will be combined with promotion of awareness of the presence, identity and activity of others in a common space, encouraging the replacement of communication rules with 'softy enforced' social conventions. Put in other words, restrictions and rules are imposed only if really necessary and then in a way that is as flexible and under user control as is possible. Finally, we will conclude our paper by describing some prototype multi-user virtual reality systems which demonstrate one example (and probably the most obvious one at that) of the proposed framework in operation. However, we first begin with the motivation for our work, provided by a examination of two current approaches to collaborative user interfaces, the WYSIWIS paradigm and the use of video as a communication medium. 2. Synchronisation Through WYSIWIS WYSIWIS ('What You See Is What I See') prescribes a tightly coupled way of synchronising multi-user interfaces and has been proposed as a foundational concept for shared systems (Stefik et al., 1987b). Under WYSIWIS, users see the same objects, in the same ways and in the same places. WYSIWIS is also often enhanced by the addition of individual telepointers. Several researchers have already expressed reservations over strict WYSIWIS. In particular, Stefik et al. (1987a) propose a variant called 'Relaxed WYSIWIS' which allows synchronisation constraints to be relaxed along four key dimensions. These are space, where WYSIWIS is only applied to only a subset of visible objects; time, where delays in updating views are allowed; population, where sharing may be limited to subgroups of the user population and congruence, where alternative views (e.g. visual variations) of objects may be possible.

Viewpoints, Actionpoints and Spatial Frames

411

We argue that WYSIWIS may in fact inhibit collaboration by binding people too closely together. In effect, it places too many constraints on people's freedom to move and act as autonomous individuals. Furthermore, maintaining synchronisation during concurrent multiparty activity requires considerable effort. This effort is either passed onto users or onto the underlying system, the latter in the form of complex locking mechanisms and heavyweight and costly networking protocols (which may in turn inhibit the establishment of shared workspaces over wide area networks). For example, turn-taking mechanisms control synchronisation in systems where there is one 'writer' and 'many' readers such as (Greenberg & Bohnet, 1991) and more collaboration aware systems often introduce explicit mechanisms for managing the sharing of displays such as (Rein & Ellis, 1991). Although relaxed WYSIWIS may go some way towards providing users with a degree of autonomy while maintaining a degree of synchronisation, we propose that it still imposes a fundamental and unnecessary constraint on users in that it synchronises their viewpoints within the display space. Put another way, it effectively glues their heads together. This falls into the trap of assuming that looking at a common object implies looking at it from the same perspective. This approach can be contrasted with that of Shu & Flowers (1992) who, in their work on collaborative three dimensional design, argue that users should have autonomous but mutually visible viewpoints within a common space. A viewpoint represents a users position within the display space and also their orientation and hence their focus of attention or gaze direction. Thus, viewpoints combine both the position that the user is looking at and the position that they are looking from within the display space. A further key aspect of viewpoints is that users are aware of each others viewpoints in a shared space and thus have some notion of where other people are working. Put more formally, viewpoints have the effect of directly embodying users within the display space (i.e. giving them a visible presence). We can contrast the notion of viewpoint with the use of telepointers in shared workspaces. Many WYSIWIS systems support the notion of autonomous telepointers which allow different users to indicate places of interest within the common display space. Like Shu and Flowers' viewpoints, telepointers aim to embody users within the space, to show where they are working and what they are currently doing. The difference between viewpoints and telepointers is that viewpoints indicate where users are looking whereas telepointers show where they are manipulating. Drawing on our experience of interaction in the real world, viewpoints correspond to the position of our head and eyes where as telepointers correspond to our hands. In fact, telepointers are just one example of what we chose to call actionpoints (see below). Furthermore, we note that viewpoints and actionpoints are not necessarily coupled together in a single individual and that they are rarely synchronised between different individuals. However, other people are generally aware of where we are looking and manipulating. In other words, real world interaction supports both autonomous viewpoints and actionpoints coupled with awareness of the viewpoints and actionpoints of others.

3. Synchronisation and Video Now we turn our attention to the use of video in collaborative systems, particularly its use within teleconferencing and media-spaces. Although the use of video may seem to be a separate issue from that of WYSIWIS, we argue that the synchronisation of user displays in a multi-party video interaction is also influenced by the above issues. The introduction of video between remote participants may be intended to meet several goals. First, video may enhance

412


audio in teleconferencing by allowing users to see each other, to make eye contact and to more effectively manage conversation (e.g. effecting turn taking and repairs) (Acker & Levitt, 1987). Second, video may be intended to enhance general awareness through the introduction of facilities such as glancing and office-sharing in media-spaces (Gaver et al., 1992; Root, 1988). Gaver (1992) has already discussed the affordances and some of the current limitations of video in some depth. We briefly highlight the following issues as motivating our work: 1. Lack of a common spatial frame — although video may link together separate physical spaces, it fails to provide a consistent common spatial frame between them. In other words, there is no common navigable space established in a video connection. Instead, video provides a non-traversable window (and usually a small one at that) between separate spaces. Participants using video therefore cannot easily establish common spatial terms of reference. As a more concrete example, the notion of making eye-contact in a video conference of more than two people is plainly nonsensical. Glancing at a camera will result in the glance being transmitted to many people — just who is being glanced at? Contrast this with the ability to tell who is attending to who around a real conference table by observing gaze direction. Interestingly enough, this problem does not become apparent until we consider groups of more than two people. 2. Lack of embodiment — people using video act as external observers peering into a space through a window. They are not actually embodied within the remote space. This may result in several problems. First, they are unable to easily move about in the remote space, experiencing it from all angles and perspectives. Second, lack of embodiment may cause social problems in the use of video technology (e.g. people feel that the are being watched and are unaware of the observer's presence or identity). Perhaps the general case is not so much one of sharing viewpoints, but is more how you perceive and relate to other collaborators. It seems that it is important to have a casual and non-threatening trustful awareness of others. The video tunnel's 'en face' view is not sufficient and might even be considered harmful (Rodden, Sommerville &Twidale, 1993).

4. A Framework for Collaborative User Interfaces This analysis of synchronisation issues has motivated us to propose a new framework for synchronising multiple user displays in CSCW systems. At present we offer this framework as a working hypothesis, backed up with some demonstration systems. Longer term evaluation is clearly required to test this hypothesis. The aim of our framework is to provide a clear separation between key design issues and to show how they relate to each other, thereby allowing designers to determine appropriate synchronisation policies. To summarise, previous sections have identified seven key design issues for collaborative interfaces: 1. Both too little and too much synchronisation of users displays may inhibit the ability to co-operate. In particular, WYSIWIS may involve too much synchronisation. 2. People might possess independent viewpoints and actionpoints within a common display space. 3. Viewpoints and actionpoints need not be synchronised between individuals. However, people should be aware of other peoples viewpoints and actionpoints.


413

4. People should be aware of each others presence, identity and activity (the latter supported via viewpoints and actionpoints). 5. A common spatial frame should be provided. Existing multi-user interfaces typically lack a common spatial frame. Instead, they provide synchronisation between sets of disjoint spaces. 6. Users should have an embodiment. Many existing interfaces fail to adequately embody users within the common display space. 7. Firm and fast mechanisms (or rules) for management and control of shared workspaces should be replaced by social conventions that are not enforced by the environment itself. We propose a new design framework for collaborative interfaces which addresses these issues. The framework consists of the following components.

4.1. Components of the Framework Selection A collaborative system may contain many objects, of which only a subset may be present in a users interface at a given time. Selection describes the process of choosing which objects are represented from the set of possible objects. A requirement on selection is that it should be lightweight and non-obtrusive.

Presentation Once selected, it is necessary to choose the mode of presentation of an object. In particular, each visible object may have many possible representations (e.g. a clock representation might be digital or analogue or a representation of temperature might be either colour or length/height). It should also be possible to agree on common representations. An architecture for managing multiple presentations of objects has already been proposed in (Bentley et al., 1992).

Spatial Frame The spatial frame refers to the spatial frame of reference used in each person's display mechanism. The spatial frame locates all objects within the display space by defining a co-ordinate system which allows the measurement of both position and direction. The spatial frame may be defined to have any number of dimensions. Thus, a spatial frame may represent a 2-D windowed interface, a 3-D virtual environment or indeed a more general N-dimensional data space (where dimensions are constructed from combinations of the attributes belonging to some set of objects). The notion of a common spatial frame means that users inhabit mutually consistent co-ordinate systems. Consequently, they can assign consistent relative positions to each other and to all other observed objects. In turn, this supports the use of consistent spatial language and conventions (e.g. if I observe that person A is looking at B, I can be sure that this is also true for you, for A and for B). It is possible that different users' spatial frames may only partially overlap. For example, two users may inhabit different multi-dimensional data spaces which overlap across a specific subset of their dimensions. In this case we can relate the notion of their common spatial frame to the set of dimensions which they share.

414


Embodiment Embodiment describes the way in which users are themselves directly represented within the display space. Notice that we consider users as existing within the space, not as observers looking onto it. Note that this does not necessarily imply the use of immersive virtual reality technologies, only that the user has some representation within the space. Embodiment conveys awareness of presence, identity and even activity to other people and is therefore fundamental to managing cooperative work. As a result, embodiment must satisfy a number of possibly conflicting goals including personalisation (i.e. easily conveying identity and allowing people to tailor their own images), the identification of position, direction, activity and also what can be called truthfulness (it may be confusing and harmful for objects to appear other than they are — e.g. lying about identity or suggesting capabilities not actually possessed, for instance showing ears when an object cannot hear).

Viewpoints A viewpoint represents where a user is attending within the display space (i.e. an individual's perspective). Thus, a viewpoint is analogous to the position and direction of a person's gaze within the real world.

Actionpoints An actionpoint represents a point within the display space where a user is interacting with some object. Examples of actionpoints are telepointers on a shared drawing surface, cursors on a shared editing surface and 'hands' within a virtual reality system. Thus, we separate the point of attention (viewpoint) from the point of manipulation (actionpoint). This is analogous to the separation between our eyes and hands within the real-world. Following on from this analogy, viewpoints and actionpoints may be independent of each other, although they may often be used in a loosely coupled way.

Awareness and Communication Awareness of the presence, identity and activity of others in a shared space is critical for establishing and subsequently managing cooperative work. Awareness may be peripheral or focused. It may also be casual or intentional. At the extreme, awareness leads to direct communication via some appropriate medium. Awareness relies on embodiment and includes awareness of both the viewpoints and actionpoints of others.

4.2. Synchronisation of Components Having identified key components of collaborative interfaces, we now return to the issue of synchronisation. More specifically, we consider the question "which of these components should be synchronised in order to best support cooperative work?" Although we argue that there is no hard and fast answer to this question and that the degree of synchronisation will depend upon the preferences of the people involved and the nature of the application, we propose a core solution which we believe will act as a guideline or starting point for most cases. As stated above, we believe that some synchronisation is needed, but that even relaxed WYSIWIS is too extreme. We are now in a position to phrase this argument more formally: The key problem with WYSIWIS is that it does not allow autonomy of viewpoint between different users within the common spatial frame (although it does usually allow autonomy of actionpoints).


415

As for video, we can state the main problems as follows. The key problem with video is that it provides no common spatial frame of reference within which users can be embodied and actionpoints and viewpoints can be provided As a result we propose the following general synchronisation policy: 1. Spatial frames should be relatively highly synchronised. This means that multipleusers should experience some common spatial frame of reference, allowing them to consistently place each other within the shared space and also to use spatial terms during interaction. It is important for cooperative work to establish at least one common spatial frame between participants. The degree of synchronisation might vary according to the number of dimensions involved. For example, it is possible that two users may share only a subset of available dimensions and, of these, only a further subset may be synchronised. Thus, we can conceive of different degrees of synchronisation of users' common spatial frame. 2. Individual viewpoints and actionpoints should not generally be synchronised between different people. In other words, each person should be free to control their own view points and actionpoints. There may be some cases where synchronisation would be useful (e.g. riding piggy-back and following someone else's view of a complex task). 3. Selection and presentation can be synchronised to varying degrees depending on the application. Thus, the extent to which users see the same objects presented in the same ways may vary. In particular, the framework supports the idea of multiple presentations of shared information as discussed in (Bentley et al., 1992). In general, the less synchronisation of selection and the less synchronisation of presentation between users, the harder it will be to collaborate. It is also important to consider how synchronisation is applied. Instead of enforcing synchronisation through hardwired mechanisms, support for awareness and communication allows participants to negotiate synchronisation with each other. Such negotiation may either be direct or may be indirect, the latter through a process of observing the position of others in the common spatial frame. Thus, firm rules and heavyweight mechanisms are replaced by 'softly enforced' social conventions which are 'applied' by users not by the computer. This equates much better to the way in which groups manage access to resources in the real world. For example, a whiteboard has no internal mechanism to limit or control multiple access and turn-taking. Instead, people subtly negotiate access based on an awareness of who is using and wants to use the whiteboard backed up with social convention (e.g. it is generally rude to interrupt people). In turn, this mechanism is supported by continual awareness of who is attending to and manipulating the board within a common spatial frame. The social conventions can be broken, although at some social risk. We will return to this example with our prototype systems below.

4.3. Sub-spaces and Unfolding Many user interfaces support the notion of subspaces (e.g. nested windows). We can relate this to our model through the concept of 'unfolding'. In his work on virtual reality, Michael Benedikt introduces the term unfolding to refer to the creation of sub-spaces within a Cyberspace, where each subspace is defined by its own set of dimensions (Benedikt, 1991). Our framework should also allow the creation of sub-spaces through a process of unfolding, where each subspace defines its own common spatial frame and may provide its

416


own synchronisation policy. The idea of having different synchronisation policies in different (sub) spatial frames provides our framework with further flexibility.

4.4. Examples of Use Before describing specific prototyping activities, we first briefly reflect our framework back onto a variety of different applications. First, we consider a virtual desktop application where the display constitutes a moving window passing over a large 2-D surface containing windows, icons and other objects. The virtual desktop itself represents the common spatial frame and should thus be synchronised across multiple users. However, the position of a given display and pointer/cursor should be autonomously managed. Furthermore, the positions of other peoples viewpoints and actionpoints should be made visible in some form so as to provide some sense of embodiment. A major difficulty here might be the limited view afforded by the display which fails to provide peripheral awareness of the presence and location of others. This might be solved by the provision of a separate map facility which affords an instant overview of the entire virtual desktop surface. A second application area might be that of video-conferencing or media spaces. Here the major problem to be addressed is the lack of a common spatial frame. Some mechanism is required for situating a set of video images within a common space and for navigating this space. One possible approach is suggested by Gaver (1992), e.g. the use of Mobile Cameras. Another might be to situate the two dimensional video images within some kind of 3-D display space. A third application area for our framework might be that of multi-user virtual reality systems. Indeed, in many ways this is the most immediately obvious target area and is the domain that has been chosen for our early prototyping work. The following section therefore describes several early collaborative virtual environments that realise the ideas described above.

5. Some Prototype Applications Several researchers have recently begun to experiment with multi-user virtual reality systems including Codella et al. (1992) with Rubber-rocks, the work of the ATR Lab (Ohya et al., 1993; Takemura & Kishino, 1992), the Habitat environment (Morningstar & Farmer, 1991), and E's (1992) Networked Virtual Art Museum. In addition, current commercial VR products offer some limited multi-user capability including dVS from DIVISION (Grimsdale, 1991), Superscape from Dimension and the World Toolkit from Sense 8. At Nottingham and SICS we have also begun experimenting with multi-user VR as a means of supporting co-operative work and as a way of realising the concepts proposed above. In this section we will introduce two current demonstrators which between them show how our framework can inform the design of CSCW systems. Of course, we need to stress once more that, although an obvious realisation of our framework, multi-user VR is not the only target domain for this work.

5.1. DIVE The first and most extensive demonstrator is the DIVE system developed at SICS (Fahlen et al., 1993). DIVE (Distributed Interactive Virtual Environment) is a distributed multi-user virtual reality system, suited for building and running applications in a high performance computer network setting1. DIVE has been used to implement a variety of applications Implementations of DIVE exist for several types of UNIX platform. For information about the availability of

DIVE please send EMAIL to [email protected].


417

High Capacity Backbone

Figure 1: DIVE architecture. In the figure there are 3 coexisting DIVE world contexts, made up out of (at least) 7 processes running on 3 different computers.

including teleconferencing, information visualisation and control of industrial robots. The system consists of a set of communicating UNIX processes, running on nodes (computers) distributed within a local or wide-area network, see Figure 1. The processes have access to a number of databases, each one describing a virtual context (or world). For each world, there exists an associated process group, consisting of all the processes that are currently members of that world. When a process joins a specific process group, it receives a complete copy of the actual world data. The different copies of the world database are then kept consistent by the use of reliable multi-cast network protocols so that the databases can be updated concurrently. A process can only be a member of one world process group at a given time but it is easy and quick to go between different worlds. Current DIVE applications realise the components of our proposed interface framework in the following way: DIVE provides a common spatial frame in the form of a three dimensional space which is shared by its users. At present, selection is synchronised so that all users see the same set of objects. However, their presentation may vary. In the simplest case, the appearance of an object depends on the perspective from which it is viewed as one would expect in a normal 3-D space. DIVE users are directly embodied through a virtual body which they move at will with a full six degrees of freedom (3 translational and 3 rotational). Each user may have a unique and arbitrarily complex embodiment as specified in their own configuration file. In addition to a graphical representation of the user, this embodiment also provides a set of 'strings', attached to external peripherals, which are used to freely and independently manipulate the users viewpoints and actionpoints. Possible peripherals include head-mounted displays, wands, joysticks, screens and mice. Two typical configurations might be an immersive interface where a single viewpoint is controlled by a 3-D tracker attached to

418


Figure 2: Teleconferencing in DIVE.

the headmount and an actionpoint by a 3-D tracker attached to a wand; or a non-immersive interface where a mouse is used to control both viewpoint (via a special graphical 'vehicle') and actionpoint. A non-immersive user may inhabit as many instances of their body as they wish, each having a separate display window. The screen-shot in Figure 2 is taken from a typical DIVE tele-conferencing environment and includes a range of different embodiments. There are five users in this scenario (the image is the current view of one of them) and between them they inhabit three forms of body. The first is vaguely humanoid in shape, even having a photograph texture mapped onto the 'face' to support identification. The overall body position as well as the separate head and one arm are all tracked and move to show the positions of the user's viewpoint and actionpoint. The second (in the middle of the shot) is a simpler 'blockie' also with a moving head to show viewpoint. Actionpoint is conveyed by a separate pointer which acts as a limb (not currently active in the picture). The third is a flat 2-D image of an external user which has been captured from a video camera. Figure 2 also shows the use of communication tools in DIVE, in particular, the inclusion of a fully-functional whiteboard. Thus, this DIVE scenario demonstrates a shared drawing surface, situated within a common spatial frame within which users are directly embodied and have unsynchronised, but mutually visible, viewpoints and actionpoints. It is interesting to compare this with 2-D shared drawing tools which use WYSIWIS. Awareness of other users is clearly provided by their embodiments. In addition, DIVE also supports a more complex awareness model where users negotiate mutual, non-symmetrical, levels of awareness across a range of media using two concepts called focus (a spatial field to represent attention) and nimbus (a field to represent the projection of activity). A detailed description of this awareness mechanism is beyond the scope of this paper and the interested reader is referred to (Benford et al., 1993; Benford & Fahlen, 1993). Finally, the concept of unfolding is supported in so far as users can move through portals from one DIVE universe into another at will. Universes can be connected into any arbitrary structure.


419

It should also be stressed that a demonstration or video sequence is required to really appreciate the awareness of other people's presence and actions that are provided in DIVE applications such as this.

5.2. MASSIVE Our second demonstrator, MASSIVE, represents an attempt to produce a collaborative virtual environment capable of scaling to larger numbers of inhabitants than many existing systems (Greenhalgh, 1994). Part of MASSIVE's approach to scale involves simultaneously supporting radically different styles of user interface. In order to achieve this, MASSIVE further relaxes synchronisation across some of the components of our proposed framework. MASSIVE allows users with different styles of user-interface to interact within a common spatial frame. In particular, the system supports both a 3-D graphics interface (using the DIVE visualiser code) and also a 2-D text interface. The 2-D interface takes the form of a MUD (Multi-User Dungeon) like representation of the world based on a 2-D scrolling window which allows its user to move across an infinite 2-D plane. Other users and objects are embodied by combinations of simple ASCII characters. For example, a user is embodied by the first letter of their name attached to a short line which points in the direction they are currently facing. Direct communication between users is also supported through a mixture of audio and text communication media. Our motivation for providing such a mixture of interfaces has been to show how users with access to quite different levels of technology can still interact within a common virtual world. Providing a 2-D interface also allows us to heavily populate worlds at minimal expense. We recently managed a session of six simultaneous users; although perhaps not yet a 'heavy' population, this is large scale by current VR standards. The architectural innovation that makes such heterogeneity possible is the manner in which the system dynamically brokers and connects 'media' between different users. Each user is represented by an object which exports a number of media which may be attached to other objects in order to interact (examples of media are graphics, text and audio). When objects get sufficiently proximate to one another the system compares their media and, if they possess any in common, establishes a connection between them. Thus, a text-interface user may also export a graphics embodiment which can then be seen by graphics interface users and vice-versa (an object can export an image to another object even if it is unable to display the medium itself). If two objects export no common media then they will not be able to interact and will be mutually invisible. Considering our proposed user interface framework, we can see that in order to achieve this heterogeneity, MASSIVE has relaxed synchronisation across both the spatial frame and selection components: • In the case of interaction between 2-D and 3-D users their spatial frames only partially overlap (across the 2-D plane). Thus, 2-D text users are effectively 'flatlanders' who inhabit just a limited slice of the 3-D users environment. Thus, synchronisation only occurs across a subset of all possible spatial dimensions. • The fact that objects might be invisible to one another (depending on whether they 'speak' any media in common) means that synchronisation of selection is effectively relaxed — not all users see the same objects.

420


In summary, MASSIVE synchronises spatial frames where they overlap, does not strictly synchronise selection or presentation and does not synchronise viewpoints or actionpoints at all. In addition, as with DIVE, it provides direct embodiment (in a variety of media) and awareness of others within the spatial frame.

5.3. Initial Observations We have presented two demonstration systems which implement our proposed framework. Our aim in doing this has been both to illustrate some possible applications of our ideas and also to indicate their technical feasibility. The status of these demonstrators is that DIVE is generally available as a VR platform for the research community and that MASSIVE exists as a laboratory prototype. Although demonstrated to be technically feasible, our proposed framework still remains little more than a hypothesis which has yet to be experimentally tested (e.g. by comparison with WYSIWIS). We believe that considerable technical problems have to be surmounted before this becomes a realistic possibility. In our opinion, the current demonstrators are still too crude to test anything other than people's reaction to the base technologies involved (indeed, this is true of nearly all VR systems). Problems with immersive technologies (lack of comfort, poor resolution and field of view); the crude appearances of some of our environments (lack of complexity and textures); as well as currently limited scaleability all have to be addressed before a useful evaluation of our framework can be made. Given these caveats, we are however prepared to offer a few subjective initial observations. First, direct embodiment of users appears to enhance awareness of others in a powerful way. Given an appropriate channel (i.e. audio) we have found it quite easy to associate simple 3-D graphical embodiments with other users (2-D may be another issue - see below). This leads to a second point — audio really does seem to be a critical requirement. Although the text channel allows communication, it is really too limited for helping synchronise realtime activity. Thirdly, those embodiments which show viewpoint and actionpoint by moving 'head' and 'limbs' driven by position trackers easily convey a sense of human presence and activity. We suspect that showing such dynamic movement may turn out to be one of the most significant factors in achieving this. Thirdly, interaction between users with radically different interfaces and the resulting relaxation in synchronisation of the spatial frame and selection have already led to some interesting side-effects. For example: • The presence of invisible users who may be engaged in hidden interactions with other visible users. • The ability for 3-D users to appear to teleport relative to 2-D users by moving off the 2-D plane and then entering it at some other location. • Differences in the treatment of personal space between 2-D and 3-D users. We have observed that, whereas 3-D users maintain a distance between themselves and others, 2-D users appear to have little sense of personal space. Indeed, they frequently stand 'in other peoples faces' and even occasionally walk through their bodies! We suspect that two factors may contribute to this effect. First, graphics users have a limited field of vision and so have to stand back to obtain a reasonable view whereas 2-D text users have a fully circular field of view. Second, the representation of the world seen by 2-D users may be too limited for them to identify the characters on the display with other people.


421

Whether these observations constitute serious long term problems remains to be seen. At present, we merely note them as interesting consequences of relaxing synchronisation between key components of our framework. In particular, they raise issues for the maintenance of synchronisation between radically different styles and capabilities of interface.

6. Summary Support for cooperative work requires an appropriate degree of synchronisation of user interfaces. Too little synchronisation and users have no common ground; too much and co-ordination becomes problematic. An analysis of two key CSCW technologies, WYSIWIS interfaces and video, suggested a number of issues to be addressed in this area. More specifically, we have proposed that existing approaches may fail to distinguish between individual viewpoints, actionpoints and having a common spatial frame of reference and may also fail to adequately embody users within the common display space. The paper then proposed a framework for the design of collaborative user interfaces which aims to overcome these problems. This framework identifies the following key components: selection, presentation, spatial frame, embodiment, viewpoints and actionpoints. Considering synchronisation, the paper suggested that the spatial frame requires a high degree of synchronisation but that viewpoints and actionpoints do not. Thus, users can be free to move autonomously within a common space. They should also be aware of the presence and movement of others. It also proposed that viewpoints and actionpoints need not be strictly coupled together for an individual. Furthermore, different synchronisation policies could be applied to separate sub-spaces through a process of unfolding. We then described two prototype implementations which show how the framework might be applied to the development of collaborative virtual environments and presented some initial observations from use in the laboratory.

Acknowledgements Very many thanks to Christer Carlsson and Olov Stahl of SICS for their work on the DIVE system and to Chris Greenhalgh of the University of Nottingham for designing and implementing the MASSIVE system. This work has been supported by the Commission of the European Communities via the COMIC project (ESPRIT Basic Research Action 6225) and also via the UK's Science and Engineering Research Council.

References Acker, S & Levitt, S (1987), "Designing Videoconference Facilities for Improved Eye Contact", Journal of Broadcasting and Electronic Media 31 (2), pp.181—191. Benedikt, M (1991), "Cyberspace: Some Proposals", in Cyberspace: First Steps, M Benedikt [ed.], MIT Press, pp.273-3O2. Benford, S, Bullock, A, Cook, C, Harvey, P, Ingram, R & Lee, O (1993), "From Rooms to Cyberspace: Models of Interaction in Large Virtual Computer Spaces", Interacting with Computers 5 (2), pp.217-237.

422


Benford, S & Fahl6n, L (1993), "A Spatial Model of Interaction in Large Virtual Environments", in Proceedings of ECSCW93, the 3rd European Conference on Computer-Supported Cooperative Work, Kluwer (Academic Press), pp.109-124. Bentley, R, Rodden, T, Sawyer, P & Sommerville, I (1992), "An Architecture for Tailoring Cooperative MultiUser Displays", in Proceedings ofCSCW'92: Conference on Computer Supported Cooperative Work, J Turner & R Kraut [eds.], ACM Press. Codella, C, Jalili, R, Koved, L, Lewis, J B, Ling, D T, Lipscomb, J S, Rabenhorst, D A, Wang, C P, Norton, A, Sweeney, P & Turk, G (1992), "Interactive Simulation in a Multi-Person Virtual World", in Proceedings ofCHI'92: Human Factors in Computing Systems, P Bauersfeld, J Bennett & G Lynch [eds.], ACM Press. E, C Loeffler (1992), "Networked Virtual Reality", in Proceedings of the 5th Multi-G Workshop, Kista, Stockholm, Swedish Institute of Computer Science (SICS), ISBN 91-7170-115-X. Ellis, C, Gibbs, S & Rein, G (1991), "Groupware: Some Issues and Experiences", Communications of the ACM 34(1), pp.38-58. Fahten, L E, Stihl, O, Brown, C G & Carlsson, C (1993), "A Space Based Model for User Interaction in Shared Synthetic Environments", in Proceedings of INTERCHI'93, S Ashlund, K Mullet, A Henderson, E Hollnagel & T White [eds.], ACM Press. Foster, G & Stefik, M (1986), "Cognoter, Theory and Practice of a Colaborative Tool", in Proceedings of CSCW'86: Conference on Computer Supported Cooperative Work, D Peterson [ed.], ACM Press, pp.7-15. Gaver, W (1992), "The Affordances of Media Spaces for Collaboration", in Proceedings of CSCW'92: Conference on Computer Supported Cooperative Work, J Turner & R Kraut [eds.], ACM Press, pp.17— 24. Gaver, W, Moran, T, MacLean, A, Lovstrand, L, Dourish, P, Carter, K & Buxton, W (1992), "Realising a Video Environment: EuroPARC's RAVE System", in Proceedings of CHI'92: Human Factors in Computing Systems, P Bauersfeld, J Bennett & G Lynch [eds.], ACM Press, pp.27-35. Greenberg, S & Bohnet, R (1991), "GroupSketch: A Multi-user Sketchpad for Geographically-distributed Small Groups", in Proceedings of Graphics Interface '91. Greenhalgh, C (1994), "An Experimental Implementation of the Spatial Model", The Department of Computer Science, The University of Nottingham, UK, COMIC Working Paper COMIC-NOTT-4-15. Grimsdale, C (1991), "dVS — Distributed Virtual Environment System", in Proceedings of Computer Graphics '9/, ISBN 0 86353 282 9. Ishii, H & Kobayashi, M (1992), "ClearBoard: A Seamless Medium for Shared Drawing and Conversation with Eye Contact", in Proceedings ofCHI'92: Human Factors in Computing Systems, P Bauersfeld, J Bennett & G Lynch [eds.], ACM Press, pp.525-532. Morningstar, C & Farmer, F R (1991), "The Lessons of Lucasfilm's Habitat", in Cyberspace: First Steps, M Benedikt [ed.], MIT Press, pp.273-302. Ohya, J, Kitamura, Y, Takemura, H, Kishino, F & Terashima, N (1993), "Real-time Reproduction of 3D Human Images in Virtual Space Teleconferencing", in Proceedings of VRAIS'93, IEEE, Seattle, Washington, pp.408-414. Rein, G L & Ellis, C A (1991), "rIBIS: A Real-time Group Hypertext System", International]ournal of Man-Machine Studies 34 (3), pp.349-368.


423

Rodden, T, Sommerville, I & Twidale, M (1993), "The DNP: Understanding and Supporting the Work of Designers'", in Proceedings of ECSCW'93, the 3rd European Conference on Computer-Supported Cooperative Work, Kluwer (Academic Press). Root, R W (1988), "Design of a Multi-Media Vehicle for Social Browsing", in Proceedings ofCSCW88: Conference on Computer Supported Cooperative Work, D Tartar [ed.], ACM Press, pp.25-38. Sarin, S & Greif, I (1985), "Computer-based Real-time Conferencing Systems", IEEE Computer 18 (10), pp.3348. Shu, L & Flowers, W (1992), "Groupware Experiences in Three-Dimensional Computer-Aided Design", in Proceedings ofCSCW'92: Conference on Computer Supported Cooperative Work, J Turner & R Kraut [eds.], ACM Press. Stefik, M, Bobrow, D, Foster, D G, Lanning, S & Tatar, D (1987a), "WYSIWIS Revised: Early Experiences with Multiuser Interfaces", ACM Transactions on Office Information Systems 5 (2), pp. 147-167. Stefik, M, Foster, G, Bobrow, D G, Kahn, K, Lanning, S & Suchman, L (1987b), "Beyond the Chalkboard: Computer Support for Collaboration and Problem Solving in Meetings", Communications of the ACM 30(l),pp.32-47. Takemura, H & Kishino, F (1992), "Cooperative Work Environment Using Virtual Workspace", in Proceedings ofCSCW'92: Conference on Computer Supported Cooperative Work, J Turner & R Kraut [eds.], ACM Press.

Author Index Abowd, Gregory D, 249 Addison, Mark, 67 Alty, James L, 179 Anderson, Ben, 179 Beale, Russell, 239 Beaudouin-Lafon, Michel, 9 Benford, Steve, 409 Bergan, Julie, 179 Bergan, Marius, 179 Bodart, Francois, 162 Buckingham Shum, Simon, 21 Conway, Alan, 211 Copas, Conn V, 339 Curry, Martin B, 327 Dix, Alan, 397 Dowell, John, 53 Dutt, A, 109 Edmonds, Ernest A, 339 England, David, 301 Eriksson, Henrik, 353 Fahlen, Lennart E, 409 Fulton, David, 37 Gennari, John H, 353 Gray, Phil, 301 Green, T R G, 91 Hammond, Nick, 21 Harrison, Michael, 223 Hassell, Jonathan, 223 Howarth, Peter A, 195 Istance, Howell O, 195 Johnson, Chris, 285 Johnson,H, 109

Johnson, P, 109 Kilgour, Alistair, 383 King, Richard J, 369 Knott, Roger P, 179 Lavery, Darryn, 383 Leung, Ying K, 369 Lim, Kee Yong, 313 Long, John, 313 McGowan, Steve, 301 Modugno, Francesmary, 91 Monk, Andrew F, 327 Musen, Mark A, 353 Myers, Brad A, 91 Ouedraogo, Missiri, 125 Paul, Jody, 81 Pimentel, Maria da Graca Campos, 145 Plaisant, Catherine, 3 Puerta, Angel R, 353 Roast, Chris, 273 Salter, Ian, 53 Shneiderman, Ben, 3 Smyth, Michael, 179 Sykes, Pete, 383 Thimbleby, Harold, 67 Treglown, Mark, 261 Vanderdonckt, Jean, 125, 162 Veale, Tony, 211 Wood, Andrew, 239 Ygueitengar, Banta, 125 Young, Richard M, 249 Zekrullahi, Solaleh, 53

Keywordlndex adaptive interfaces, 223 agent modelling, 223 agents, 239 air traffic management, 53 analogy, 261 arrangement, 125 augmented reality, 9 automated interface design, 353

CASE tools, 369 cognitive dimensions, 91 cognitive modelling, 21 collaborative writing, 369 command reuse, 223 computer-assisted education, 81 computer-supported cooperative work (CSCW), 239, 369, 384, 397, 409 cooperative work, 397 coordination, 3

data models, 353 deaf sign language, 211 decomposition, 327 delayed feedback, 397 design process, 163 design rationale, 21,249 desktop metaphor, 3 dialogue model, 327 dimensioning, 125 direct engagement, 261 domain analysis, 53 domain models, 353

editing, 249 educational assessment, 81 email, 273 end-user programming, 91 evaluation, 21, 109, 145 eye-control, 195

finite state machines, 67

flowgraphs, 67 folding editors, 369 formal methods, 273, 285, 339 formal specification, 21

geographic information systems, 339 gesture input, 9 graphic user interface, 3 graphical structured notations, 314 groupware, 369, 384 groupwork, 239

human animation, 211 human factors specifications, 314 hypermedia, 81 hypertext, 67, 145

initiative, 223 input device, 195 intelligent interfaces, 239 interaction analysis, 249 interaction models, 273 interaction object, 125,163 interaction object placement, 125 interaction tools and techniques, 163 interface customisation, 223 interface design, 53 interface models, 353 interface transformation, 223 interviews, 37

knowledge representation, 339 knowledge-based systems, 339

link selection, 145 localization, 125

manuals, 67 mediaspace, 9

428 metaphor, 9, 179 model-based interface development, 353 model-based interface tools, 163 multi-disciplinary analysis, 249 multimedia, 9, 67, 179 multi-user, 249 multi-user interface, 409

Keyword Index specification, 301, 327 spreadsheet, 327 status/event, 397 structured human factors method, 314 synchronisation, 409 system behaviour, 327 system design, 37 system modelling, 249, 273

object-oriented programming, 163

paper interface, 9 paradigm, 9 personal role manager, 3 phonological structure, 211 physically-challenged, 195 placement strategy, 125 previewing information, 145 problems in communication, 37 programming by demonstration, 91 Pursuit, 91

qualitative reasoning, 261

rapid prototyping, 339 realtime conferencing, 384 response time, 301, 397

safety-critical systems, 285 scenario, 327 scenario analysis, 249 selection, 163 shared window servers, 384 Shared X, 384 simulation, 285 software engineering, 369

task analysis, 339 task description language, 301 task domain, 53 task model, 327 task quality, 53 telecommunications, 179 temporal logics, 285 theory-based design, 21

undo, 249 usability inspection methods, 109 user interface design, 369 user interface management systems, 163, 339 user modelling, 249, 339

video, 9, 409 virtual reality, 9, 409 visual description, 37 visual design, 125 visual language, 91 visual shell, 91 visual thinking, 37

work objective, 327 WYSIWIS, 409

People and Computers

Computers For Smart People

Computers Helping People with Special Needs