The Six Disciplines of Breakthrough Learning: Second Edition; How to Turn Training and Development into Business Results (Pfeiffer Essential Resources for Training and HR Professionals)

Essential resources for training and HR professionals Essential resources for training and HR professionals Praise for...

Author: Calhoun W. Wick | Roy V. H. Pollock | Andy Jefferson

693 downloads 2732 Views 12MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Essential resources for training and HR professionals


Praise for the Second Edition of

The Six Disciplines of Breakthrough Learning “No other book in the last decade has been more important for the workplace learning field. The second edition is even better, incorporating new wisdom learned in the crucible of real-world practice.” —WILL THALHEIMER, PH.D., president, Work-Learning Research, Inc.

From left to right: Roy Pollock, Calhoun Wick & Andrew Jefferson Photo by Terence Roberts

THE AUTHORS Calhoun Wick is the founder and chairman of the Fort Hill Company. He is recognized nationally as a consultant, educator, and researcher on improving the performance of managers and organizations. Roy Pollock serves as chief learning officer for Fort Hill. Andrew Jefferson is president and CEO of Fort Hill. They authored the award-winning first edition of The Six Disciplines of Breakthrough Learning as well as Getting Your Money’s Worth from Training and Development. The Fort Hill Company specializes in helping companies earn a better return on their investments in training and development. Over the past ten years, Fort Hill’s 6Ds methodology and learning transfer management systems have helped many of the world’s leading companies ensure that learning gets transferred and results improve. The company is based in Wilmington, Delaware.

“The Six Disciplines brings together many of the most important principles of corporate training in an easy-to-understand, highly visual format. Readers will find this book filled with examples, models, and practical tools you can use to create high-impact learning solutions in your own organization.” —JOSH BERSIN, president and chief executive officer, Bersin & Associates “Wick, Pollock, and Jefferson have a well-proven formula for moving learning from an academic exercise to business results. Their 6Ds offer all leaders specific and concrete things that they can do to turn learning into results.” —DAVE ULRICH, professor, Ross School of Business, University of Michigan, and partner, The RBL Group “I introduced The Six Disciplines to GE’s Global Learning Council and I am very glad that I did. The Six Disciplines gives us a common language and a common process that work very well in GE.” —JAYNE JOHNSON, director, Global Leadership Development, General Electric “The 6Ds model provided a powerful framework for designing high-impact learning programs that helped to shape our approach to learning and to align our learning leaders across the organization. The second edition adds insights and tools that make this edition even more relevant, practical, and valuable.” —ROBERT SACHS, Ph.D., vice president, Learning and Development, Kaiser-Permanente “Embracing the Six Disciplines has allowed us to develop truly innovative high-impact leadership programs that have changed the trajectory of our firm.” —CAROL BONETT, vice president, leadership development officer, Morgan Stanley Smith Barney LLC

ORGANIZATIONAL DEVELOPMENT

The SIX DISCIPLINES of Breakthrough Learning

“Simply put, this book is a critical read for anyone who wants to ensure that the time and money spent in development produces results.” — TERESA ROCHE, vice president and chief learning officer, Agilent Technologies

SIX DISCIPLINES The

Wick Pollock Jefferson

2nd

Edition

of Breakthrough Learning

SIX DISCIPLINES The


When the first edition of The Six Disciplines of Breakthrough Learning was published, it quickly became the new standard for companies who were serious about increasing the return on their investment in learning and development. This thoroughly revised, updated, and expanded edition of The Six Disciplines of Breakthrough Learning reveals how training, OD, and HR professionals can enhance their contributions to the success of their organizations. The book is filled with practical, proven-to-work guidelines and examples that show clearly how to design, deliver, and document comprehensive learning experiences that improve leadership and management, sales, customer service, marketing, and other business functions. The authors—experts in the field of learning and development as well as business— explain the theories and principles that undergird the 6Ds approach and provide practical methods, tools, roadmaps, and checklists for bridging the “learning-doing” gap. The book’s guidelines are illustrated with dozens of case studies from successful companies on the cutting edge of results-driven educational performance.

2nd

Edition

How to Turn Training and Development into Business Results

Discover more at www.pfeiffer.com

Calhoun Wick, Roy Pollock, and Andrew Jefferson

The 6Ds encompass the entire process of converting learning into improved performance, from pre-course communications to the impact of the learning transfer climate. Designed to be used by a broad spectrum of professionals, the book’s knowledge can be applied to virtually any learning and development program. Most important, The Six Disciplines of Breakthrough Learning focuses on business and performance outcomes—the new “finish line” for learning and development.

fm.indd viii

6/17/10 3:53:24 PM

More Praise for The Six Disciplines of Breakthrough Learning “Learning professionals who apply The Six Disciplines in their day-to-day work will deliver undeniable value to any organization.” —NICOLE ROY-TOBIN, CHIEF LEARNING OFFICER, FINANCIAL ADVISORY SERVICES, DELOITTE SERVICES, LLP “Applying The Six Disciplines catapulted us from a respected training and development function to true business partners. In this second edition, these trusted and credible authors bring even greater clarity and context to the 6Ds. I highly recommend this book.” —DIANE HINTON, PLASTIPAK ACADEMY GOVERNING BOARD LEADER, 2010 CORPU EXEMPLARY PRACTICE AWARD FOR BUSINESS ALIGNMENT, PLASTIPAK PACKAGING, INC. “The Six Disciplines is the definitive ‘how-to’ guide that provides practical, useful howto templates and lots of examples that can meet any type of training measurement requirement. Now there is no excuse not to get started.” —J. SCOTT BOSTON, SENIOR VICE PRESIDENT TALENT MANAGEMENT & MCKESSON CENTER FOR LEARNING, MCKESSON CORPORATION “The Six Disciplines is truly a masterpiece of useful advice and guidance for leveraging results from training and development. It is by far the best book ever written on the topic and will no doubt exert an enormous positive influence.” —GEOFF RIP, STRATEGY DIRECTOR, INSTITUTE FOR LEARNING PRACTITIONERS (AUSTRALIA), AND PRESIDENT, CHANGELEVER INTERNATIONAL “In today’s world, great learning and development professionals can be defined in three words . . . effective business consultants. If you want to be viewed as a strategic partner whose learning and development solutions align to and impact business results, this is a must-read.” —GARY WALJASPER, HEAD OF ORGANIZATIONAL DEVELOPMENT, THE PRINCIPAL FINANCIAL GROUP

fm.indd i

6/17/10 3:53:21 PM

“I wondered why companies using The Six Disciplines framework stand out as top performers. Now I know. The second edition is filled with even more good examples, design concepts, and useful tools.” —SUE TODD, PRESIDENT AND CEO OF CORPORATE UNIVERSITY XCHANGE “I recommend this book to any trainer, be it a novice or an expert, as it forces them to look at the training delivery more holistically as an end-to-end process much beyond the classroom.” —DORAIRAJ SELVAN, PH.D., SENIOR VICE PRESIDENT, TALENT TRANSFORMATION, WIPRO TECHNOLOGIES, BANGALORE, INDIA “The second edition of The Six Disciplines contains a treasure trove of powerful ideas for facilitating the transfer and application of new learning into the workplace. Few have done more than Cal Wick and his colleagues to bridge the gap between the growing body of academic understanding of the ’transfer problem’ and still-reluctant learning and development practitioners.” —ROBERT TERRY, PH.D., MANAGING DIRECTOR, ASK EUROPE “This is a must-have for the shelf of all professionals who are charged with bringing change to their organization and actually seeing a return on investment!” —BEVERLY KAYE, FOUNDER & CEO, CAREER SYSTEMS INTERNATIONAL; COAUTHOR, LOVE ‘EM OR LOSE ‘EM; AUTHOR, UP IS NOT THE ONLY WAY “As businesses evolve and adapt in response to greater global complexities, those organizations that focus on the transfer and application of learnings and insights into business results will stand apart. The Six Disciplines offers a mindset and approach to the next iteration of learning as a competitive advantage.” —ANDREW WOJECKI, PH.D. “The Six Disciplines describes and illustrates six principles practiced by companies that earn the highest returns by efficiently converting learning into business results. A truly valuable book!” —KEN BLANCHARD, COAUTHOR, THE ONE MINUTE MANAGER AND THE SECRET The Six Disciplines transcends generations, learning styles, culture, and technology as an enduring and practical way to assure that learning matters to the business. —MARYANN BILLINGTON SENIOR PARTNER, KORN/FERRY LEADERSHIP AND TALENT CONSULTING

fm.indd ii

6/17/10 3:53:22 PM

ABOUT THIS BOOK

This book is for everyone who is a provider, sponsor, purchaser, or consumer of corporate training and development. It describes a proven set of processes and tools that achieve a breakthrough in corporate education and that significantly improve the return on the investment that companies make in learning and development.

Why Is This Topic Important? Staying competitive in today’s business climate depends on know-how, human capital, and the ability to learn quickly at both the individual and organizational levels. Companies invest heavily in training and development—more than $50 billion annually in the United States alone—in an effort to improve their leadership, efficiency, quality, customer service, and so forth. There is compelling evidence that these investments can and do pay dividends. There is equally compelling evidence, however, that the business benefit of learning and development can be much greater than most organizations currently realize. Substantial value is being left on the table in the form of “learning scrap”—training and development that is never transferred to the work of the organization in a way that improves performance.

fm.indd iii

6/17/10 3:53:22 PM

What Can You Achieve with This Book? This book describes and illustrates six disciplines that together represent a breakthrough in corporate training and development: (1) define business outcomes, (2) design the complete experience, (3) deliver for application, (4) drive learning transfer, (5) deploy performance support, and (6) document the results. By practicing these six disciplines, you will be able to design, deliver, and document learning and development programs that produce greater business impact and higher returns on investment.

How Is the Book Organized? In the Introduction we provide a brief overview of the six disciplines of breakthrough learning. We then dedicate a full chapter to each discipline, exploring it in-depth and providing recommendations and tools to maximize its contribution. Insights from exceptional business and learning leaders, as well as case studies, are used to illustrate key concepts. At the end of each chapter, we provide checklists for implementation and action items for both general managers and learning leaders, because maximizing the return on investment from learning and development requires a true partnership between line management and learning professionals.

The Instructor’s Manual for the second edition of The Six Disciplines of Breakthrough Learning is available free online to qualified college and university instructors. If you would like to download and print out a copy of the guide, please visit: www.wiley.com/college/wick.

fm.indd iv

6/17/10 3:53:23 PM

About Pfeiffer Pfeiffer serves the professional development and hands-on resource needs of training and human resource practitioners and gives them products to do their jobs better. We deliver proven ideas and solutions from experts in HR development and HR management, and we offer effective and customizable tools to improve workplace performance. From novice to seasoned professional, Pfeiffer is the source you can trust to make yourself and your organization more successful.

Essential Knowledge Pfeiffer produces insightful, practical, and comprehensive materials on topics that matter the most to training and HR professionals. Our Essential Knowledge resources translate the expertise of seasoned professionals into practical, how-to guidance on critical workplace issues and problems. These resources are supported by case studies, worksheets, and job aids and are frequently supplemented with CD-ROMs, websites, and other means of making the content easier to read, understand, and use.

Essential Tools Pfeiffer’s Essential Tools resources save time and expense by offering proven, ready-to-use materials—including exercises, activities, games, instruments, and assessments—for use during a training or team-learning event. These resources are frequently offered in looseleaf or CD-ROM format to facilitate copying and customization of the material. Pfeiffer also recognizes the remarkable power of new technologies in expanding the reach and effectiveness of training. While e-hype has often created whizbang solutions in search of a problem, we are dedicated to bringing convenience and enhancements to proven training solutions. All our e-tools comply with rigorous functionality standards. The most appropriate technology wrapped around essential content yields the perfect solution for today’s on-the-go trainers and human resource professionals.

w w w. p f e i f f e r. c o m

fm.indd v


6/17/10 3:53:23 PM

fm.indd vi

6/17/10 3:53:24 PM

THE SIX DISCIPLINES OF BREAKTHROUGH LEARNING

fm.indd vii

6/17/10 3:53:24 PM

fm.indd viii

6/17/10 3:53:24 PM

Y THE SIX DISCIPLINES OF BREAKTHROUGH LEARNING How to Turn Training and Development into Business Results SECOND EDITION

Calhoun Wick Roy Pollock Andrew Jefferson Foreword by Al Switzler Afterword by Jim Kouzes and Barry Posner

fm.indd ix

6/17/10 3:53:24 PM

Copyright © 2010 by John Wiley & Sons, Inc. All Rights Reserved. Published by Pfeiffer An Imprint of Wiley 989 Market Street, San Francisco, CA 94103-1741

www.pfeiffer.com

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400, fax 978-646-8600, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, 201-748-6011, fax 201-748-6008, or online at http://www.wiley.com/go/permissions. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. Readers should be aware that Internet websites offered as citations and/or sources for further information may have changed or disappeared between the time this was written and when it is read. For additional copies/bulk purchases of this book in the U.S. please contact 800-274-4434. Pfeiffer books and products are available through most bookstores. To contact Pfeiffer directly call our Customer Care Department within the U.S. at 800-274-4434, outside the U.S. at 317-572-3985, fax 317-572-4002, or visit www.pfeiffer.com. Pfeiffer also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Page 361 constitutes a continuation of the Copyright page. Library of Congress Cataloging-in-Publication Data Wick, Calhoun W. The six disciplines of breakthrough learning: how to turn training and development into business results / Calhoun Wick, Roy Pollock, Andrew Jefferson; foreword by Al Switzler.—2nd ed. p. cm. Prev. ed. entered under title. Includes bibliographical references and index. ISBN 978-0-470-52652-1 (hardback) 1. Organizational learning. 2. Employees — Training of. I. Pollock, Roy V. H. II. Jefferson, Andrew McK. III. Title. HD58.82.S59 2010 658.3'124 — dc22 2010014180 Acquiring Editor: Matthew Davis Production Editor: Michael Kay/Susan Geraghty Editor: Rebecca Taff Editorial Assistant: Lindsay Morton

Manufacturing Supervisor: Becky Morgan Graphic Design: Michael Aspengren Cover Design: Michael Aspengren Editorial Assistance: Kathleen Myers

Printed in the United States of America HB Printing 10 9 8 7 6 5 4 3 2 1

fm.indd x

6/17/10 3:53:25 PM

CONTENTS

Foreword

xiii

Introduction: The Six Ds

1

D1 Define Business Outcomes

19

D2 Design the Complete Experience D3 Deliver for Application D4 Drive Learning Transfer

65

109 161

D5 Deploy Performance Support

213

D6 Document Results 255 Coda

319

Closing Thoughts: Learning Is the Master Skill 329 Jim Kouzes and Barry Posner References Index

333

345

About the Authors

359 xi

TOC.indd xi

6/17/10 3:44:47 PM

To our clients, for allowing us to work with them and learn from them in their pursuit of excellence; to our employees, for making it happen; and to our families, for encouraging us to pursue our dreams.

TOC.indd xii

6/17/10 3:44:48 PM

FOREWORD

B

ecause you’re holding this book, I know that you’ve had experience with training and learning. You’ve no doubt experienced both the positives and negatives. On the positive side, you’ve been in courses where you learned insights and skills that, when you applied them to real issues, helped you achieve important results. If you’re like me, you cherish those experiences. And, if you’re like most people, you’ve also seen the negatives—training experiences where powerful learning occurred, but then . . . nothing: no application, no follow through on commitments, and, not surprisingly, no results. This negative aspect of the learning and training industry is widespread, deep, and broad. And these problems are not just known by executive management, HR management, and trainers; they are seen and felt by millions who sit through training classes but don’t see the value. Action plans and good intentions fade like mirages; no one seems to care. And after years of such experiences, they conclude that too much of training is a waste of time and money. It is something you endure so some box somewhere can be checked. But there is hope. And you hold in your hand an essential key to the solution. The Six Disciplines of Breakthrough Learning by Cal Wick, Roy Pollock, and Andy Jefferson is a readable, memorable, and practical approach to maximizing the positives and overcoming the negatives in learning and development. These authors, who are themselves experienced trainers and managers, truly understand that corporate learning is about changing people’s behavior to produce desired xiii

FOREWORD.indd xiii

6/17/10 3:40:12 PM

xiv

Foreword

outcomes for the organization. This book contains a wealth of information, ideas, examples, and steps to help you achieve that goal. This new second edition includes dozens of new examples, citations, and research studies that make it even more valuable. There was a time, not so long ago, when learning and organization development professionals dismissed the idea of linking interventions to corporate results. We felt our job was to develop people because it was the right thing to do. Period. But we were wrong. Helping people develop and grow is still the right thing to do; but it must also deliver results. As the authors explain, business leaders are charged with investing corporate resources in the way that produces the best possible outcomes for the organization’s key stakeholders—customers, employees, and shareholders. In today’s intensely competitive business climate, no organization can squander time or money on activities that fail to produce value. That means that, as learning professionals, we have to set our sights on a “new finish line.” We need to continue to train people well. But in addition, we need to be sure that newly trained people actually behave differently on the job and that their new behaviors lead to demonstrable improvements in the results our companies are trying to achieve. Anything short of that is ultimately entertainment and therefore expendable. As I read this new edition, I was impressed with the many tools, steps, and insights that are included. Here are a few points I’d like to highlight: • The six disciplines make sense; they are theoretically sound. Cal, Roy, and Andy are mature researchers, have spent years in the trenches, have identified the obstacles, have shared very useful real-world cases, and have shared specific strategies. I could clearly see the evidence that they have studied and how they have helped hundreds of companies achieve breakthrough learning. Seldom is a book so well conceived and so complete. • This book is a large and powerful tool box. These authors are eminently pragmatic. So beyond the sound theory and the sage perspective, they share and explain checklists, scorecards, assessments, and action points for learning leaders and for line leaders. There are diagnostic tools and advice about traps to avoid. • And there are wonderful ah-ha’s and chuckles. Of course, what is an epiphany and what is humorous will vary from one individual to another. But I had my share of both, and I know you will also. The second edition of The Six Disciplines of Breakthrough Learning is a remarkable book. I am certain it will continue to make a powerful contribution to the entire training industry, as did the first edition.

FOREWORD.indd xiv

6/17/10 3:40:12 PM

Foreword

xv

I am also convinced that the investment you and your team make in reading, discussing, and applying the concepts in this book will make a positive and measurable difference in connecting training and learning to the business results that matter. The Six Disciplines of Breakthrough Learning will allow you to make a muchneeded contribution more often, on a grander scale than ever before; it will help you be one of those influencers who are changing the world for the better. June 2010

FOREWORD.indd xv

AL SWITZLER, Best-selling author of Crucial Conversations Provo, Utah

6/17/10 3:40:13 PM

FOREWORD.indd xvi

6/17/10 3:40:13 PM

THE SIX DISCIPLINES OF BREAKTHROUGH LEARNING

FOREWORD.indd xvii

6/17/10 3:40:13 PM

FOREWORD.indd xviii

6/17/10 3:40:13 PM

INTRODUCTION: THE SIX Ds The training you provide must contribute—visibly and substantially—to fulfillment of customers’ business strategies. —VAN ADELSBERG AND TROLLEY

T

hroughout our careers, we have been convinced of the strategic importance of learning and the contribution that corporate training and development programs can and should make. But we have also been deeply concerned that their actual impact is frequently far less than their true potential. Ten years ago, we started on a mission to understand—and to find ways to remove—the impediments to learning realizing its full potential and strategic contribution. We have worked with hundreds of organizations, large and small, and thousands of different programs. We have been privileged to be part of breakthrough learning programs—initiatives that helped propel their companies to a higher level of performance and that delivered results of undeniable value. But we have also observed programs that produced minimal or even negative impact, most often for the lack of learning transfer. New skills and knowledge were taught, but never applied to the actual work of the organization. When we compared the differences between these two extremes, we found that breakthrough learning is the result of a disciplined and systematic approach, executed with passion, excellence, and a commitment to continuous improvement. There is no “magic bullet”—no one, simple fix that transforms corporate learning from the periphery to central strategic importance. Seven years ago, we distilled the critical practices that characterize high impact initiatives into the 6Ds (Figure I.1), which we described in The Six Disciplines of Breakthrough Learning (2006). 1

intro.indd 1

6/17/10 3:43:02 PM

2

The Six Disciplines of Breakthrough Learning

FIGURE I.1.

THE 6Ds THAT TURN LEARNING INTO BUSINESS RESULTS

Case in Point I.1 The 6Ds at GE Jayne Johnson, director, Leadership Education, for GE Global Learning, Crotonville, introduced the Six Disciplines to GE. “Part of my role in leading the Global Learning Council is to share best practices among the group. I read the Six Disciplines book and fell in love with the methodology. It just makes so much sense to start with the outcome in mind and work backwards. The concepts really resonated with me, so I introduced the Six Disciplines to the Global Learning Council by having one of the authors come out to Crotonville and conduct a session for us. Looking back now, I’m very glad that I did, because the 6Ds lives on across GE Global Learning. “GE is very much a metrics-driven organization, very results-oriented. The 6Ds gave us a common language to reinforce all our efforts and ensure that we are making an impact with the courses we run. Before we knew about the Six Disciplines, whenever somebody went to one of our classes, we had them put together an action plan based on all the things they had learned and wanted to implement upon their return home. “Once they left our grounds here in Crotonville, we really had no idea what they did with it. Occasionally, we would check in, but there was no consistency. So, what the Six Disciplines did was give us a more consistent and thoughtful approach to ensuring that participants continue to think about what they committed to do in the class and as a result, we have a higher percentage of people following-through on them. The Six Disciplines gave us a common language and a common process that makes so much sense. It works very well in GE.”

Since the first edition of this book, learning organizations in many leading companies have adopted the 6Ds as the organizing principles for their training and development efforts. They have found them to be a powerful mnemonic and a common language to leverage best practices across their learning organizations (see Case in Point I.1).

intro.indd 2

6/17/10 3:43:03 PM

Introduction

3

This new edition of The Six Disciplines of Breakthrough Learning has been extensively revised. We have incorporated new research and examples of best practices from innovative companies on the leading edge of getting better results from learning. We have also incorporated many of the new tools and guides we developed in conjunction with the 6Ds Workshops. What follows is a brief introduction to each of the Six Disciplines. Each begins with a “D” to make them easier to remember and apply. While implementing the principles of any one of the 6Ds individually will enhance results, the greatest improvement is achieved when all six are used together; there is synergy among them. In the remainder of the book, we dedicate a full chapter to each discipline, exploring it in depth and providing examples and tools to maximize its contribution.

Define Business Outcomes A fundamental premise of this book is that human capital is the single most important source of competitive advantage in today’s increasingly knowledgebased economy. Competitive pressure requires organizations to continuously improve the quality of their products and services and the efficiency with which they are delivered. Maintaining competitive advantage through human capital requires ongoing investment in employees’ development, both so that they stay current in a rapidly changing world and so that they stay with the company. According to James K. Harter, Ph.D., Gallup’s chief scientist for workplace management, one of the best predictors of retention is whether an employee feels he or she has opportunities at work to learn and grow (Robison, 2008). Corporate-sponsored learning represents an investment that companies make to enhance their human capital and so ensure their future. Companies expect that investment to pay dividends in terms of greater effectiveness, improved productivity, enhanced customer satisfaction, better commitment, higher retention, and so forth. That means that all company-funded learning opportunities—whether classroom-based training, e-learning, informal learning, executive coaches, tuition reimbursement, and the like—ultimately serve a business purpose. We should say at this point that, throughout the book, we use “business” and “corporate” to refer to the larger organization that sponsors learning and development initiatives. These are meant to also include government agencies and not-for-profit enterprises. Although they are not “businesses” in the usual sense, continuous learning is nevertheless essential for these organizations to fulfill their missions, and it must be managed in a business-like manner to be effective. Whether or not the organization is expected to produce a profit, the fundamental

intro.indd 3

6/17/10 3:43:04 PM

4


rationale—and management’s expectation—is that performance will in some way improve following a training and development program or other learning opportunity (see Figure I.2). In other words, as workplace learning professionals, “We are not in the business of providing classes, learning tools, or even learning itself. We are in the business of facilitating improved business results” (Harburg, 2004, p. 21). The extent to which organizations are willing to fund learning, and the regard in which it is held, depend on the extent to which learning initiatives deliver on management’s expectations for improved performance. Therefore, the First Discipline—and perhaps the most critical—is to clearly define the business outcomes expected from each and every learning initiative. We do not mean learning outcomes or learning objectives. Most programs already have well-defined learning objectives that articulate what participants will learn or what they will be able to do by the end of the program. Those are still necessary for designing the instruction, but they do not answer the fundamental questions that business leaders are interested in: • How will this initiative benefit the business? • How will we know? FIGURE I.2.

MANAGEMENT EXPECTS TRAINING TO IMPROVE PERFORMANCE

el op me nt

Effectiveness

New (and Higher) Level of Performance

Current Level of Performance

Before

intro.indd 4

g Trainin

v De d an

After

6/17/10 3:43:07 PM

Introduction

5

Whereas learning objectives explain what participants will know or be able to do at the end of the program, business outcomes specify what they will do on the job, and the benefits to the business. Clearly defining the expected business outcomes has many advantages: • It makes learning a more strategic function, since the relationship to the mission of the organization is clear. • It increases the motivation to learn in adults by answering the “what’s in it for me?” question. • It increases the likelihood of investment by making the business value explicit. • It calls attention to the shared responsibility of learning and line managers; onthe-job results can only be obtained with their support and reinforcement. Enterprises that have implemented D1 across their learning organizations have discovered that they achieve much greater buy-in, not only from management, but also from program participants themselves. Getting clear about D1— the desired business outcomes—makes designing a more effective intervention easier. It is also prerequisite to effectively documenting the results (D6). Finally, having clearly defined outcomes allows learning organizations to win: They can unambiguously demonstrate their value because they know what success means to the business. In the chapter on D1 we underscore the importance of making sure that there is open, transparent, and readily apparent alignment between needs of the business and the goals of learning initiatives. We provide guidelines for distinguishing between learning and business outcomes and for distinguishing between performance problems that can, and cannot, be improved by training. We include tools and guidelines for having the dialogue with business leaders needed to ensure linkage. We underscore the benefits of understanding the value chain of learning, of mapping the intended impact, picking the right problems, and managing expectations.

Design the Complete Experience A second theme running through this book is that converting learning into business results is a process, not a one-off event. Learning organizations need to be much more explicit and deliberate about the process by which learning is transformed into results than has been the norm in the past (see Figure I.3). Process improvement requires considering all of the factors that affect the outcome and singling out those that have the most profound influence for special

intro.indd 5

6/17/10 3:43:08 PM

6


FIGURE I.3.

TRAINING AND DEVELOPMENT NEEDS TO DESIGN THE COMPLETE EXPERIENCE

© Sidney Harris/Condé Nast Publications/www.cartoonbank.com.

attention. The Second Discipline of breakthrough learning, therefore, is to design the complete experience. The emphasis here is on complete, which means including what happens before and after the formal periods of instruction as part of the design. Historically, instructional design systems and corporate learning organizations focused primarily on “the course”—the period and method of instruction—with relatively little attention paid to what happened before and, especially, after the instruction. The research results are clear, however. The program’s “surround”— what happens before and after training—is as important as the instruction itself in determining the outcome. The “transfer climate” in the participant’s workplace has a particularly profound impact; indeed, it can make or break the value of any learning program.

intro.indd 6

6/17/10 3:43:10 PM

Introduction

7

The Second Discipline—designing the complete experience—recognizes that, from the participants’ points of view, the learning experience begins long before the formal course. It ought also to continue long afterward, until they have improved their performance and produced results (see Figure I.4). The Second Discipline demands a new paradigm: The learning organization is responsible for optimizing the learner’s total experience—not just what happens in the classroom (or its virtual, electronic, or informal equivalent). Maximally effective programs approach learning holistically and systemically; they pay special attention to the impact of the participant’s manager and the work environment on learning transfer and results. Designing learning initiatives that comprise all the factors that influence the results—including those outside the traditional scope of training and development—is more important now than ever; learning organizations are increasingly being funded on the basis of the business value they generate (or fail to generate). In the chapter on D2, we examine what makes up the “ complete experi- The real finish line for learning ence” and which elements most impact is the delivery of business learning transfer and the creation of busiresults. ness value. We suggest methods and tools to optimize outcomes, some of which challenge conventional thinking. We argue that learning organizations need to redefine what it means to “finish a course.” The participants’ work is not done when they reach the end of an online module or the last day of class; the real finish line for learning is the documented delivery of business results. We show that improving the transfer climate and providing post-instructional performance support are particularly rich opportunities for a breakthrough. FIGURE I.4. THE COMPLETE LEARNING EXPERIENCE ENCOMPASSES MUCH MORE THAN THE PERIOD OF INSTRUCTION

The complete learning experience hear about program

invitation or sign up

preparation

instruction: “the course”

learning transfer and practice with support

achieve results/ mastery

Historical focus of training and development

intro.indd 7

6/17/10 3:43:10 PM

8


Designing for the complete experience allows learning organizations to realize their full promise of delivering value to the organization. Since training and development programs take time and cost money, everyone benefits when they are planned and managed in a way that maximizes the likelihood of success.

Deliver for Application The Third Discipline that characterizes high-impact learning programs is that they are delivered to facilitate application. That is, their instructional designers begin with the end in mind—what participants are supposed to do differently and better—and then consciously select learning strategies that help participants bridge the learning-doing gap (see Figure I.5). Delivering for application means selecting what to teach and how to deliver it most effectively based on the desired business outcomes and the behaviors needed to achieve them. It means using instructional approaches, technologies, and supporting strategies that accelerate learning transfer and application on the job. At the core of the Third Discipline is the principle that learning creates value only when it is applied; therefore, the way in which learning is delivered should reflect and facilitate the way in which it will be used. In Chapter D3, we look at innovative ways that progressive learning organizations are bridging the learning-doing gap by making the relevance of the FIGURE I.5. THERE IS ALWAYS A GAP BETWEEN LEARNING AND DOING; THE GOAL OF D3 IS TO BRIDGE THAT GAP

© 2008 Fort Hill Company. All rights reserved.

intro.indd 8

6/17/10 3:43:11 PM

Introduction

9

material clear; showing how each element is connected to real business issues; motivating application by answering the “what’s in it for me?” question; and helping participants prepare and plan for on-the-job application. We review insights into what makes learning memorable and provide tools to map the chain of value and monitor the perception of the program’s utility.

Drive Learning Transfer Because companies invest in learning to address business needs and opportunities, learning objectives are, in effect, business objectives. They should be treated as such. In any well-managed company, systems are in place to set, measure, monitor, and reward achievement of business objectives. Historically, however, Learning objectives are, in there have been no such mechanisms for effect, business objectives. learning-transfer objectives (see Figure I.6). Program participants were allowed to set them and forget them. Small wonder relatively few were achieved. “Talk to any group of laymen or professionals about what’s broken in the current learning and development process, and most will tell you it’s the lack of serious posttraining follow-through” (Zenger, Folkman, & Sherwin, 2005, p. 30). Learning transfer is the process of putting learning to work in a way that improves performance. The Fourth Discipline that characterizes high-impact learning organizations is that they drive the transfer of learning back to the work of the enterprise. They do not leave it to chance or individual initiative. Instead, they put in place systems and processes to actively encourage and manage the FIGURE I.6.

LEARNING TRANSFER IS THE WEAKEST LINK IN MOST TRAINING AND DEVELOPMENT PROGRAMS

Copyright © Grantland Enterprises; www.grantland.net. Used with permission.

intro.indd 9

6/17/10 3:43:13 PM

10


transfer process. The Fourth Discipline includes making sure that participants set (or are assigned) the right objectives, that they transfer their learning to the work they do, and that their managers hold them accountable for doing so. In the chapter on D4, we introduce the concept of learning scrap and the high cost of doing nothing to ensure learning transfer. We explain the elements that define the transfer climate and determine the results that training ultimately delivers. We review what it takes to improve performance and discuss breakthroughs in the management of the learning transfer process made possible by technology. Finally, we provide case examples of how supporting learning transfer enhances the value of already effective programs.

Deploy Performance Support Companies that are serious about getting a return on their investment in learning and development understand that the job isn’t finished until the new knowledge and skills are successfully applied in a way that improves performance. They recognize that getting better at anything requires practice and that when trying anything new, people need support and coaching (Figure I.7). FIGURE I.7. TRAINING AND DEVELOPMENT PROGRAMS CREATE GREATER VALUE WHEN THEY INCLUDE BOTH ACCOUNTABILITY AND SUPPORT

Business Value Created

Accountability Plus Support

intro.indd 10

Accountability Only Support Only

6/17/10 3:43:16 PM

Introduction

11

To maximize the probability of delivering meaningful business outcomes, highly effective learning organizations practice the Fifth Discipline: They deploy various forms of ongoing performance support after the instruction. They work with senior leaders to develop a culture in which managers understand that they have a responsibility to support learning. They “put their money where their mouths are” by reallocating some of their resources from pure instruction to instruction plus performance support for transfer and application. In the chapter on D5, we make the analogy between product support and performance support for learning transfer. We explore the three sources of support—materials, systems, and people—and provide examples. We discuss the need to balance accountability and support and the new demands this places on the learning organization and line management. We focus special attention on the participant’s manager, because he or she has a profound impact on whether learning is applied or scrapped. We confront head-on the reasons that managers don’t coach more to support the use of training and we provide specific steps to make the transfer climate favorable for results.

Document Results The bottom-line questions that must be answered about any learning and development initiative are these: Did it make a positive difference? Did it achieve the results for which it was designed? Was it worth it? The Sixth Discipline is to document results in a relevant, credible, and compelling way that justifies further investment and supports continuous improvement. Evidence of results is needed to justify continued investment in any business iniLearning organizations should tiative. Learning is not exempt from this be models of continuous requirement, especially in a time of finanimprovement. cial exigencies. In today’s business climate, companies must continuously improve the effectiveness and efficiency of all their business processes to remain competitive. Learning organizations should be models of continuous improvement. But continuous improvement is possible, and continued investment is warranted, only when there is unequivocal evidence that the initiative generated value. The requisite data are those that document outcomes of importance to the business—not just activity (people trained, courses taught), learner satisfaction (reaction), or even the amount learned (see Figure I.8). The Sixth Discipline—documenting results—is essential to support a cycle of continuous learning, innovation, adaptation, and improvement. The results of

intro.indd 11

6/17/10 3:43:17 PM

12


FIGURE I.8. THE OUTCOMES OF INTEREST TO THE BUSINESS ARE ON-THE-JOB BEHAVIORS AND RESULTS

Timeline from Learning to Results Training or other structured learning

Acquisition of new skills and knowledge

Transfer learning to work

Most common measures of training

Improved performance / business outcomes

Measures that really matter to the business

one program are the raw material for the next cycle of defining outcomes, designing experiences, delivering, driving, deploying, and documenting. A never-ending cycle of reinvention and renewal ensures that corporate education keeps pace with the changing competitive environment, workforce, and business needs. In the chapter on D6, we discuss why learning organizations must document results. We differentiate between metrics needed to manage the learning organization and actual results that matter to the business. We provide guiding principles for program evaluation and advice on what to measure, how to collect and analyze the information, and, especially important, how to market the results.

Summary Learning programs are investments that a company makes to enhance the value and effectiveness of its human capital. Management has a fiduciary and ethical responsibility to ensure that those investments produce a return: results that improve performance and competitiveness. We have identified Six Disciplines of Breakthrough Learning—the 6Ds— that characterize high-value, breakthrough learning and development initiatives (see Figure I.9). Learning organizations that have adopted the 6Ds as their operating principles and that have practiced them diligently have increased the contribution that learning makes to their company ’s success (see Case in Point I.2). As a result, 6Ds learning organizations have enjoyed a corresponding increase in the recognition and support they receive.

intro.indd 12

6/17/10 3:43:18 PM

Introduction

FIGURE I.9.

13

THE SIX BREAKTHROUGH LEARNING DISCIPLINES

• Link program objectives to business needs • Describe what participants will do differently • Agree on definition of success

• Recognize transfer as a critical part of the process • Treat learning objectives as business objectives • Put in place systems and processes to support transfer

• Include all four phases of learning

• Make relevance and utility of content clear

• Plan and manage learning transfer process

• Provide time for practice with feedback

• Redefine finish line as on-the-job results

• Engage participants’ managers • Provide job aids and performance support systems • Ensure availability of feedback and coaching

• Use methods that make learning memorable

• Measure what matters to sponsors (D1 outcomes) • Use insights to drive continuous improvement • Market the results to key stakeholders

In the following chapters, we explore each of the Six Disciplines in depth and provide practical tools and advice for their implementation. Each discipline is illustrated with case examples and insights from corporate leaders. Our experience in helping companies implement the 6Ds has renewed our optimism about people, learning, and organizations. We have been privileged to work with talented and creative learning and line professionals and to see breakthrough results; we believe that we are at the beginning of a true renaissance in corporate education. We are confident that you will extend the principles articulated in this book to achieve even greater success. We look forward to hearing your stories.

intro.indd 13

6/17/10 3:43:18 PM

14


Case in Point I.2 The 6Ds at Kaiser Permanente Founded in 1945, Kaiser Permanente is the nation’s largest not-for-profit health plan, serving 8.6 million members. In the rapidly changing field of health care, training and development are especially important to help Kaiser Permanente achieve its mission of providing quality care for its members and their families and contributing to the well-being of its communities. Robert Sachs, Ph.D., is vice president of learning and development. He introduced the Six Disciplines at Kaiser Permanente because of “the focus on outcomes, the whole idea of creating impact, and the concept of designing the complete experience. It isn’t just what happens in the instructional walls—whether it was a classroom or e-learning or whatever—that matters, but how you get people ready and how you help them translate what they learn into results. “Historically, we’re a pretty decentralized organization, so we probably went about doing our learning in just about every imaginable way. The 6Ds provided a framework that was easy to understand and comprehend and teach so that people could apply it. Both the vision and strategy were that we all would use this common framework to think about our work. “I’ve got a group that does a fair amount of instructional design for our core learning programs, and so they began to modify their instructional design framework and methodology to incorporate the Six Disciplines. It’s hard to make the impact totally tangible from the dollar and cents standpoint because a lot of what we have applied this to so far has been general leadership training. But we are clearly getting people who are actively following through with tangible effort to translate things that they take away from a program into real action. “We’ve done some work on ROI, and there is still opportunity to refine our measures of business impact, but we do see people being more effective at managing their teams, cutting down the amount of time that it takes to work on issues of employee relations, and making meeting times more efficient and effective as a result of applying some of the skills they’ve gotten in their programs. We also see people taking on more strategic issues and broadening their strategic relationships. Those have been some of the tangible results and outcomes.” Alan Jang, senior manager of the learning solutions team, said, “In order for learning solutions in Kaiser-Permanente to be successful, meaning achieving measurable results that impact the business, we had to create a new finish line for training. Training doesn’t end at the end of the course; it should also include follow-through and support. The [Six Disciplines] give us the framework to do that” (quoted in Chai, 2009).

intro.indd 14

6/17/10 3:43:19 PM

Introduction

15

Action Points For Learning Leaders • Evaluate the extent to which you are practicing the Six Disciplines. • Select a program for which you are responsible that has strategic importance and great potential contribution. • Score the selected program using the 6Ds scorecard (Exhibit I.1). • Decide which discipline, if strengthened, will produce the greatest improvement the quickest (“low-hanging fruit”). • Use the relevant chapters of this book and your organizational knowledge to develop a plan. • Present your findings, the target, your plan, and rationale to the relevant management team. • Ask for the resources and cooperation needed to implement your recommendations. • Evaluate the results and repeat the process to drive continuous improvement. For Line Leaders • Think about critical business needs that can be addressed (at least in part) by a learning and development program; then complete the worksheet in Table I.1. • In the first column, labeled “Critical Business Needs,” write the most pressing business opportunities or challenges that training can help address. • In the second column, labeled “Conditions of Satisfaction,” describe the results you require to consider the program a success—what people will do better and differently and the business impact this will have. • In the third column, “Acceptable Evidence,” describe the information you will need to be sure the program is working. What kinds of data will you consider credible and relevant? • Show the completed worksheet to your head of learning and development. Ask whether it is possible to deliver the results desired. Then work together to make it happen. • If you are already investing in learning and development (through an internal unit or external vendors) use the 6Ds Scorecard (Exhibit I.1) to identify the most promising areas for improvement. Ask your learning leader to do the same and compare your results. • Use the relevant chapter(s) of this book to jointly develop a plan for improvement.

intro.indd 15

6/17/10 3:43:19 PM

16


Exhibit I.1 6Ds Learning and Application Scorecard Use this tool to evaluate the readiness of a program to deliver results. For each of the items below, check the box that best describes the program using the following key:

The program design covers the entire process from invitation to on-the-job application and measurement of results.

1 2 3 4 5

3.

The relevance of each section to the business is explicitly stated; application is stressed throughout, as is the expectation for action.

1 2 3 4 5

4.

A robust process and sufficient time are provided for participants to set strong goals, plan for action, and prepare communications to others.

1 2 3 4 5

5.

After the program, participants are reminded periodically of their objectives and their opportunity to apply what they learned.

1 2 3 4 5

6.

Participants’ managers are actively engaged during the post-program period. They review and agree on objectives and expect and monitor progress.

1 2 3 4 5

7.

Participants continue to learn from each other after the program. Materials and advice are available to help them accomplish their goals.

1 2 3 4 5

8.

Participants can easily engage coaches for feedback, advice, and support. The group is tracked and supported during the application (post-program) phase.

1 2 3 4 5

9.

On-the-job results are measured based on desired outcomes identified prior to the program.

1 2 3 4 5

An active continuous-improvement process is used to strengthen the preparation, program, and learning transfer.

1 2 3 4 5

10.

Deliver

2.

Drive

1 2 3 4 5

Deploy

The business needs are well understood. Anticipated on-the-job results of the training are clearly defined and measurable.

Document

1.

Design Define

1 = Not at all 2 = To a small extent 3 = Somewhat 4 = To a large extent 5 = To a very great extent

Total Score:

Scoring Instructions:

Add the numeric value (1 to 5) of all boxes selected. The maximum possible score is 50. Use the table below to assess the readiness of the program to deliver valuable follow-through results.

Points > 45 33–44 < 32

Assessment Excellent likelihood of measurable results and return on investment from program. Action: Continue disciplined approach to learning; strengthen lowest-scoring items. Moderate likelihood for positive results, but return may be less than optimal. Action: Strengthen weakest items to raise score to > 44. Valuable results and adequate return on investment is less likely. Action: Revise program in a systemic way.

© 2008, Fort Hill Company. Used with permission.

intro.indd 16

6/17/10 3:43:20 PM

Introduction

TABLE I.1. Critical Business Needs (That Training Can Help Address)

intro.indd 17

17

THE LEARNING FOR RESULTS WORKSHEET Conditions of Satisfaction (What Will Be Different if the Program Is a Success?)

Acceptable Evidence (What Data Are Needed to Show the Program Is Working?)

6/17/10 3:43:20 PM

intro.indd 18

6/17/10 3:43:20 PM

DEFINE BUSINESS OUTCOMES Management must always, in every decision and action, put economic performance first. It can only justify its existence and its authority by the economic results it produces. There may be great non-economic results: the happiness of the members of the enterprise, the contribution to the welfare or culture of the community, etc. Yet management has failed if it fails to produce economic results. . . . It has failed if it does not improve, or at least maintain, the wealth-producing capacity of the economic resources entrusted to it. —PETER DRUCKER

I

n the final analysis, organizations invest in training and development and other corporate-enabled learning initiatives for one and only one reason: to improve the performance of individuals and, thereby, the organization as a whole. Effective learning contributes to the success of the enterprise in many ways: directly— through enhanced job performance—and indirectly—through better retention, recruitment, employee commitment, and so forth. Organizations seek to facilitate learning as a purposeful business activity to ensure their own success and the success of their people. Learning agility and speed are essential to remaining competitive, perhaps even viable, in an increasingly knowledge-driven global economy. According to Warren Bennis, Distinguished Professor of Business Administration at USC’s Leadership Institute: “It isn’t about ‘treating people nice;’ it’s about helping develop people to be at their very best, because that’s the only way that companies are going to succeed. It’s human capital that is the basis for competitive advantage” (Bennis, n.d.). As one of the most famous CEOs at Proctor & Gamble, Richard “Red” Deupree, put it: “If you leave us our buildings and our brands but take away our people, the company will fail” (quoted in Dyer, Dalzell, & Olegario, 2004, p. 159). 19

CH001.indd 19

6/17/10 3:22:11 PM

20


Thus, the true measure of success for corporate training and development initiatives is the extent to which they enhance the value of the organization’s human capital and help it achieve its mission. That is a new and more demanding standard. In the past, “delivering a great training program was sufficient. But not anymore. Global competition, market pressures for performance, and the recent economic downturn have moved the finish line. There is a new criterion for training success: improved performance on the job” (Wick, Pollock, & Jefferson, 2009). Delivering great learning experiences is still important, but only as part of a process that delivers great outcomes for the business. And the only way to know whether the learning organization has achieved its goal is to agree in advance on the outcomes that matter most and the definition of success. Thus, the First Discipline (D1) of breakthrough learning is to clearly define, in partnership with line leaders, the desired business outcomes. In this chapter we underscore the importance of shifting the focus of training and development from learning outcomes to business outcomes and we provide tools and guidance for making the transition. Topics include: • • • • • • • •

Begin with the end in mind Express outcomes in business terms Pick the right problems Translate needs into deliverables Avoid training traps Manage expectations A checklist for D1 Action points for learning and line leaders

Begin with the End in Mind One of the habits of highly effective people is that they begin with the end in mind (Covey, 2004, p. 96). The same principle applies to highly effective learning organizations. Corporate training and development initiatives should always be a means to an end, not ends unto themselves. The specific “end” to be achieved depends on the nature and drivers of the business and its opportunities, challenges, and environment. Regardless of the specific objective to be achieved, however, all corporate learning is aimed at improving the performance of the business in some way (see Case in Point D1.1 and Figure D1.1). Thus, helping employees learn to deliver more effective leadership, reduce accidents, provide superior customer service, accelerate product development, enhance teamwork, give more effective

CH001.indd 20

6/17/10 3:22:13 PM

Define Business Outcomes

21

Case in Point D1.1 Learning as a Business Strategy One company that really understands the strategic nature of learning is Ingersoll Rand. As Rita Smith, vice president of enterprise learning, puts it: “We’re here for only one reason: to help drive business outcomes. We need to understand the business strategy, key strategic drivers, external threats, and financial metrics. We literally need to be bilingual, speaking the languages of both learning and business” (Smith, 2008). Ingersoll-Rand’s CEO, Herb Henkel, sees learning as a key strategic lever, so much so that he has made it an integral part of the strategic planning process: “When we go through the strategic planning process, we come up with ideas, strategies, and visions of where we’re going to be. Then we decide what to invest in to get the things we want. So I look at how many dollars we spend on bricks and mortar; how many on developing new products; and how much training we need to be able to meet our goals. Built into the planning process is the assumption that there will have to be some kind of training. So we consider it no different than we would anything else in terms of investment decisions” (quoted in Bingham & Galagan, 2008).

presentations, and so forth, are all undertaken with the ultimate goal of improving the company’s financial health and performance. The effectiveness of a learning intervention, then, is the extent to which it helps the organization achieve the “end in mind.” It follows that effectiveness can only be assessed—and success declared—if the goals of the intervention are clearly defined at the outset and those goals are aligned with and contribute to organizational effectiveness. Goals Are Prerequisite for Success In the absence of clearly defined, business-relevant objectives, learning organizations can never “win” in their quest for recognition and resources. Sporting events are fun to watch because everyone knows the rules and what “winning” means. Races have clearly marked finish lines. Other competitions have established goals and scoring. No one would pay to watch a football game if the rules were decided after the game or if winning were based on which team enjoyed themselves the most. No one would invest in stocks in the absence of accounting principles and bottom-line measures of success. Similarly, an effective commission plan drives sales behavior because it spells out how salespeople can increase their financial rewards. Contracts include specified performance levels to avoid future disagreement about deliverables. The

CH001.indd 21

6/17/10 3:22:15 PM

22


FIGURE D1.1. EFFECTIVE TRAINING PROGRAMS ALWAYS KEEP THE BUSINESS OBJECTIVE IN MIND

evidence required to support a specific therapeutic claim is agreed upon with the FDA in advance of clinical studies. Management by objectives works when individuals meet with their managers to define goals that are specific, measurable, achievable, relevant, and time-bound. The objectives for a learning and development initiative should be no different. Management and the learning organization need to work together to define the “Conditions of Satisfaction” early in the development process (see Case in Point D1.2). The promised results and the way in which they will be measured

CH001.indd 22

6/17/10 3:22:15 PM


23

Case in Point D1.2 Conditions of Satisfaction When we asked Richard Leider, award-winning coauthor of Whistle While You Work and Claiming Your Place at the Fire, about the importance of defining objectives, he told us that it was vital and went on to explain: “We teach leaders how to create what we call COS—conditions of satisfaction. What are your conditions of satisfaction? What is it that we are supposed to do differently after this and by when? What is it that you want delivered by when? Or created by when? “You could call it accountability, but when leaders lead, they are customers. For leadership development, the line leader is a customer. She makes a request; she puts out certain conditions of satisfaction. And so this whole notion of leader as customer translates into the training, and therefore the follow-up practice; leaders have certain conditions of satisfaction for training. “So often leaders are not clear about their conditions of satisfaction. There is a certain language and a certain rigor that leaders need to learn in order for meetings and training and transactions to be effective. It really clears up all that murkiness. You could say it is common sense; well, the fact is—look where the breakdowns are.”

should be agreed to as part of the design, not as an afterthought. The point is that learning and development can never compete successfully or truly claim success if the goalposts are not set out, fixed in position, and agreed to in advance. Only then do those responsible for implementation have the opportunity to excel. Business results must be the touchstone for defining learning and developBusiness results must be ment objectives, the “ true north ” the touchstone for defining against which programs are designed, learning and development implemented, and measured. Any other objectives. organizing principle is likely to get the initiative off course, because as David Campbell put it: “If you don’t know where you are going, you will probably end up someplace else” (Campbell, 1974). That learning and development programs should be designed with the end in mind hardly seems like a revelation. For at least twenty years, virtually all instructional design models—ADDIE and its many derivatives—have stressed the importance of starting with analysis and defining learning objectives. Similarly, “focus on outcomes” is the first standard of the International Society for Performance Improvement’s Human Performance Technology (International Society for Performance Improvement, 2002, p. 1). Fifteen years ago, the Robinsons

CH001.indd 23

6/17/10 3:22:17 PM

24


published the first edition of Performance Consulting, which recognized that the real responsibility of professionals in human resources, learning, and organizational development roles was “enhancing people’s performance in support of business goals” (Robinson & Robinson, 1995). Yet our experience across dozens of companies and hundreds of programs suggests that, to this day, most learning initiatives do not go far enough in defining their ultimate purpose. Most are still content to define objectives “at the end of this program” when in fact, “It is not sufficient that people attend a training program and acquire skill; what ultimately matters is that these individuals apply the skills on the job so that their performance improves and the business benefits” (Robinson & Robinson, 2008, p. 2). In other words, the “finish line” for learning is not at the end of the program, but much later, on the job, after learning transfer. The Robinsons’ quote sums up the distinction between the practice of D1—defining business outcomes—and the more common practice of defining learning outcomes, which often fail to make the link to the ultimate purpose explicit. Effective programs define their objectives in terms of on-the-job behaviors and business results; less effective programs merely define what will be learned or covered. Susan Burnett, currently senior vice president for talent and organizational development at Yahoo!, explains: “One of our strategic conclusions at Deloitte was that we needed to design and deliver outcome-based learning (see Case in Point D2.3). What I tell my team is that you have to change the conversation you have with your business partners. If you talk about what kind of training they need, you’re going to get a laundry list. It’s got to be all about the business outcomes; that’s the power of the Six Disciplines—it starts there. “If I ever write a book, it’s going to be about why learning and development organizations get so disconnected from the business and how to fix it. I often feel that this is a profession without a professional process” (Burnett, personal communication, 2009). The Six Disciplines define a process to keep learning connected to and focused on the business—a process that starts by getting crystal clear about purpose and outcomes.

Express Outcomes in Business Terms Throughout this discussion, we will repeatedly emphasize that goals for learning initiatives need to be expressed as “business outcomes.” Drucker’s famous dictum with which we began this chapter is uncompromising: Management’s responsibility is to ensure that every investment a company makes generates economic value. Business is, in the end, about financial return.

CH001.indd 24

6/17/10 3:22:18 PM


25

While Drucker was writing about for-profit companies, leaders of non-profit and government organizations must also ensure that every expenditure they approve contributes to the goals of the organization, even though these do not include “profit” as such. Training in leadership, ethics, diversity, and so forth are laudable (have merit) in their own right, but they must also pay dividends (have worth) in terms of lower operating costs, greater productivity, enhanced employee commitment, improved consumer confidence, and so forth. Programs that help participants improve their personal performance also pay dividends in greater job satisfaction, motivation, discretionary effort, and retention. What we are saying is that every corporate-sponsored learning initiative must (ultimately) produce a positive financial return, directly or indirectly; it is the only acceptable justification for continued investment. That may strike some readers as crass and mercenary, especially those who believe that helping people develop and grow is the “right thing to do.” We agree that it is the right thing to do and a corporate obligation. At the same time, we are convinced that the ability to define and deliver business benefit is the only way for learning and development to prove its value. The best defense against having the training budget cut is to show convincingly that doing so ultimately hurts the bottom line and reduces the probability of organizational success. “Learning is essential only to the degree that it contributes more to performance than other allocations of scarce resources” (Danielson & Wiggenhorn, 2003, p. 20). Results Can Be Measured We should pause here to address a common misperception. A frequent excuse for failing to define the expected results of learning and development is that “they cannot be measured.” A comment supposedly made by Jack Welch to that effect is often cited as proof. What he actually said was “We will not measure our learning systems in the same way that we do other business initiatives. To do so would only trivialize them” (quoted in Baldwin & Danielson, 2000, emphasis added). We do not deny that it can be difficult to isolate and quantify the specific contribution of training. Many factors influence business results: overall economic climate, success of marketing, competitors’ actions, new market entrants, and, in many businesses, even the weather. But those same factors also confound the evaluation of sales, marketing, and other business activities. Those functions are nevertheless held accountable, and rewarded for, delivering on their promises. That something is difficult to measure with precision, or that it is influenced by numerous other factors, is no excuse for not defining and evaluating outcomes.

CH001.indd 25

6/17/10 3:22:18 PM

26


As we will discuss more fully in D6, the fundamental “claim” or promise of training is that “ if you give the learning organization money and your employees’ time, we will help improve their performance.” Such a claim requires evidence to back it up. By analogy, consider health claims. When a pharmaceutical company develops a new drug, it must specify the drug’s claims—what it supposedly treats or cures. The company must then provide the regulatory authorities in the countries in which it wants to market evidence that shows beyond a reasonable doubt that the drug does, in fact, do what it claims to do. Data must be obtained from real patients in real time. How well a drug works in real patients, however, is influenced by many other factors—lifestyle, concurrent diseases, stage of illness, and genetic background (Figure D1.2). Patients don’t necessarily follow the directions correctly and the outcomes may be quite hard to measure. No matter. If a company wants the rights to market a drug, it has to design and execute an evaluation that unequivocally supports its claims, these difficulties notwithstanding. Before the Food and Drug Administration was established, patent medicine purveyors made any claim they wanted. The result was the “snake oil salesman” who made extravagant claims in order to sell quack remedies to the gullible— remedies that would supposedly cure whatever ailed them. To avoid promulgating such unsupported remedies in education, a consortium of the American Educational Research Association, American Psychological Association, and the National Council on Measurement in Education prepared the “Standards for Measurement is an integral Educational and Psychological Testing.” part of human endeavor. The standards require that measures (and by inference, human resource programs based on those measures) be reliable and valid predictors of the value to individuals and organizations. The standards acknowledge the difficulty of reliably defining and measuring the outcome of educational programs, but they are clear that such difficulty does not provide an exemption. The same is true in business. “Experience with several CEOs and top teams, and regular discussion with their advisors, convince me that the rejection of measures in learning is a dead end. Measurement is an integral part of human endeavor, and its power can’t be cancelled out by high-sounding sentiment” (Bordonaro, 2005, p. 229). To claim that the effects of learning and development cannot be measured is wrong-headed and potentially dangerous; if the positive effects of learning and development cannot be measured, then presumably neither can the negative effects of reducing or eliminating it. Failure to routinely document the business value of training and development undoubtedly contributes to the practice of making training budgets among the first to be cut in periods

CH001.indd 26

6/17/10 3:22:19 PM


27

FIGURE D1.2. MANY FACTORS BESIDES TRAINING AFFECT BUSINESS RESULTS JUST AS MANY FACTORS BESIDES A DRUG AFFECT MEDICAL OUTCOMES Regulators adverse rulings

Suppliers

Competitors mergers / acquisitions

higher costs shortages

sales tactics

approval time consolidations

new products

negotiating power

new regulations

pricing

Business Results consolidations

advertising

training

targeting negotiating power

public relations

impact of economy

retention commitment

sales effectiveness

engagement

Buyers

Sales/Marketing

Human Capital

Drug

Physician

Patient

active ingredient

accuracy

dosage form

weight compliance

thoroughness

formulation side effects

genetics

expertise

age

communication

Medical Outcome other medications

lifestyle

stage of illness

complications

caretakers

Environment

CH001.indd 27

concurrent maladies

Illness

6/17/10 3:22:19 PM

28


TABLE D1.1.

COMPARISON OF LEARNING OBJECTIVES AND BUSINESS OUTCOMES

Traditional Learning Objectives

Business Outcomes Objectives

Timing

End of course

On-the-job

Focus

Abilities and capabilities

Behaviors and results

of belt tightening. The discipline of always starting with business outcomes is an important safeguard against this eventuality. Because it defines the conditions of success as outcomes that matter to management, they will be more reluctant to give them up. Business Outcomes Versus Learning Objectives Making the shift from defining learning objectives to defining business outcomes is, on the one hand, a subtle change for the learning organization, but it has profound implications. Defining business outcomes differs from the traditional practice of defining learning outcomes in two important dimensions: timing and focus (Table D1.1). The vast majority of course objectives that we see are learning objectives that define capabilities—what participants will know or be able to do at the end of the course of instruction. In contrast, the business-centric outcomes we advocate focus on performance—what participants will do on the job—their behaviors and the results they will generate (Table D1.2). That distinction is crucial, because learning only creates value when it is transferred and applied to the work of the individual and the organization. “High-quality learning and training do not necessarily translate into business results. Enterprise learning must be driven with the end in mind: the business results to be achieved” (Vanthournout & others, 2006, p. 17). Defining learning objectives is still an essential part of the instructional design process, but they are subordinate to the business objectives; they are a means to the end, not the ultimate objective. Being explicit about the business outcomes that a learning program is intended to achieve has many advantages: • It makes explicit the business value of the time and money that will be invested. • As such, it garners greater support from both managers and participants. • It helps management prioritize training in the same way it weighs other investments (see Case in Point D1.1).

CH001.indd 28

6/17/10 3:22:21 PM


29

• It answers the “what’s in it for me?” question for participants and therefore satisfies one of the principles of adult learning: the need to know Why? • It highlights the need for managerial involvement, since managers control the environment in which transfer and application must occur. • It provides alignment with the business and a clear criterion against which to evaluate potential approaches: Is this the best way to achieve the result? • The outcomes are stated in a way that makes how to measure them apparent. “The minute that the trainer or researcher takes on the performance perspective, the whole thought process becomes that of an organizational stakeholder or organizational systems expert. . . . The performance perspective creates rational performance goals and the matrix of connections of variables and levels within the organization needed to ensure the appropriate actions and performance” (Swanson, 2003, p. 122). It’s Not Just About Money Defining outcomes in business terms does not, however, mean that every outcome must be stated in financial terms or that ROI is the only measure of interest to business leaders. Such reductionism is not always necessary or desirable (as we TABLE D1.2. EXAMPLES OF BUSINESS OUTCOMES VERSUS LEARNING OBJECTIVES

CH001.indd 29

Learning Objective

Business Outcome

Be able to describe the leadership styles and development stages of the situational leadership model and match the appropriate style to a developmental stage

Increase the productivity of your team and your managerial effectiveness by correctly utilizing the situational leadership model

Be able to apply the Six Sigma DMADDI model to a business process

Reduce the number of errors in the order entry process by applying Six Sigma tools and processes

Be able to explain each of the 6Ds and give an example of their application in corporate learning

Reduce the amount of learning that goes unused by applying the 6Ds to learning program design and execution

Demonstrate how to give effective feedback to a direct report

Increase employee commitment by giving more frequent and more effective feedback

Show how to use reframing to overcome customers’ objections

Increase sales by discovering and addressing customers’ objections

6/17/10 3:22:22 PM

30


will discuss in greater detail in D6, documenting results). Suffice it to say here that “business outcomes” include a wide range of potential metrics in addition to financial measures, such as increased customer satisfaction, greater employee commitment, enhanced leadership effectiveness, more productive sales approaches, improved work quality, and so forth. While these are not direct financial measures, business leaders understand that they are key contributors to financial performance and are often leading indicators of future business success. Indeed, business leaders may not want, or give credence to, efforts to more directly monetize the outcomes of training (Redford, 2007). A study at Ashridge Business School, for example, found that line leaders were actually more flexible about what defined acceptable outcomes than were learning leaders (Charlton & Osterweil, 2005). The authors concluded: “Sponsors may not be as wedded to proof of financial ROI as many HR professionals assume” (p. 13). If a clear causal link has already been established between certain behaviors (for example, better coaching of subordinates) and financial returns (lower employment costs as a result of greater retention), then it may be less expensive, more useful, Sponsors may not be as and more credible to document increases wedded to proof of financial in the desired behaviors rather than to ROI as many HR professionals apply an abstract financial transmogrifiassume. cation. Charles Jennings, head of global learning at Reuters, agrees: “A customer services manager will not care about ROI, but they will want to understand how learning improves customer satisfaction” (quoted in Redford, 2007). By “business outcomes” we mean verifiable outcomes of relevance to the business of the organization. We mean defining a goal of the program as, for example, “participants’ direct reports will receive more frequent and more effective coaching” rather than “participants will be able to demonstrate the SBA approach to providing feedback.” The former is concrete, measurable, and linked to business needs. The latter could be tested and measured, but that misses the point. The issue is not whether the new approach was learned and even demonstrable, but whether it is used in the participant’s daily work in a way that benefits the organization. “New skills and knowledge alone do not add value; they must be applied, then nurtured until improved performance can be counted on consistently to produce an important job result” (Brinkerhoff & Apking, 2001, p. 4). A program that will increase customer satisfaction is worthy of investment; a program that delivers only attendee satisfaction is not. Learning and development’s task is not completed until learning has been transferred and applied in a way that produces results; the measure of its success is in business terms.

CH001.indd 30

6/17/10 3:22:22 PM


31

“Effective training occurs when it is fully transferred and when performance achieves or exceeds business goals” (Wall & White, 1997, p. 169).

Pick the Right Problems The essence of military success is to concentrate the right resources on the right battlefield at the right time. The essence of success for a learning organization is to concentrate the right resources on the right issues at the right time. The more limited its resources, the more important it is for a learning organization to focus on those issues and opportunities that have highest potential return. But how do you know which are the right issues? There are five keys: • • • • •

Understand the business Understand the current situation Do your market research Translate needs into desired outcomes Enforce priorities

Understand the Business There is no substitute for understanding the business. For training and development to add value, learning leaders need to understand the fundamental nature of the business in which their organization is engaged. That is not to say that they have to be experts in finance, or business strategy, or marketing, or manufacturing. They do not all need to rush out to earn MBAs. But they do need to have a clear sense of how their organization fulfills its mission, makes money, and differentiates itself in the marketplace. In their book, What CEOs Expect from Corporate Training, Rothwell, Lindholm, and Wallick (2003) list the seven competencies that CEOs expect from workplace learning professionals. “Business knowledge” heads the list. Business leaders want learning professionals who understand business in general. But what they really want is learning professionals who understand their specific business, who can clearly and succinctly explain the business model of their company or division and its most important business drivers and challenges. They want learning leaders who understand what keeps the business leaders up at Business leaders want learning night, the human capital limitations to professionals who understand growth, and therefore how learning and business. talent development can best help the company achieve its goals.

CH001.indd 31

6/17/10 3:22:23 PM

32


Plastipak Academy received Corporate University Xchange’s Exemplary Practice Award for Business Alignment in 2010. According to Academy Leader Diane Hinton, “Plastipak Academy’s success is built on our reputation as a business function that drives real business results through targeted learning solutions. We talk about business issues with leaders, not just in the context of learning and development. And, we emphasize collaboration throughout the learning development and implementation process, leveraging leader involvement and learning transfer to achieve business results.” Test yourself. Can you answer the questions in Exhibit D1.1? If so, congratulations, you are in touch with your business. If not, or if you struggled with some of the questions, then you have the opportunity to improve your performance by deepening your understanding of your organization’s business. How can you deepen your knowledge about the business and thereby enhance your value as a business partner? There are ample learning opportunities on the job and in the course of your work. The first and critical requirements are interest and curiosity. Once you become genuinely interested in how the business works and curious about the kinds of questions in Exhibit D1.1, you will find plenty of

Exhibit D1.1 Self-Test of Business Knowledge Answer the following questions about your business: 1. The most important source of our revenue is:

2. The most important driver of our growth is:

3. The core elements of our strategy are (list):

4. Our main competitor is:

5. The greatest threat we face is:

6. The greatest human capital challenge we face as a company is:

CH001.indd 32

6/17/10 3:22:23 PM


33

information and opportunities to deepen your understanding in business plans, reports, and conversations with business leaders. Boudreau (2010) makes the point that the more that HR leaders are able to explain what they do using the business models that line managers use intuitively, the better they will be understood. An example would be relating managing a talent pool to an inventory management model. Business leaders understand the kinds of tradeoffs that must be made in setting inventory levels for various products or raw materials; there are direct parallels to managing the talent inventory. Likewise, business managers understand process management and process improvement models in manufacturing; explaining the parallels to the learning process will help them appreciate why treating learning as an isolated event will always sub-optimize the results. An example of such a model is given in Figure D1.3. It illustrates the concept that there are only two fundamental paths to improving business results: (1) increase revenue (sales volume); and (2) increase productivity (reduce the cost of providing products and services). For example, sales training that generates more effective sales behaviors contributes to the sales volume path. Safety training that reduces time lost in accidents contributes to productivity through lower costs and greater output. It is important to note that it is not the training as such that contributes to the organization’s success, but the actions that flow from training. Indeed, training itself actually increases cost and lowers productivity unless it is transferred to work and translated into productive behavior. Improved actions and changed behaviors necessarily precede business impact (Figure D1.4). This sequence of events is relevant to both defining outcomes (D1) and documenting results (D6). A change in behavior will be the first evidence that the program is having the desired effect. It is therefore a leading indicator of business impact. FIGURE D1.3.

PATHWAYS TO IMPROVING BUSINESS RESULTS

Profits

Increase Sales Volume

Market Development

CH001.indd 33

Market Penetration

Increase Productivity

Decrease Input Costs

Increase Output

6/17/10 3:22:24 PM

34


On the other hand, a program that fails to change behavior is unlikely to deliver results and needs to be examined to understand the root cause(s) of the breakdown. The flow of causality is from left to right, from training to behaviors to results (Figure D1.4). Therefore, learning and development programs must be planned in the opposite direction, from desired results, to the behaviors required to produce them, to the kinds of learning experiences and exercises needed to create the capability to perform optimally (Figure D1.5). To design highly effective learning interventions, you need to know the behaviors—in particular, the high-leverage or vital behaviors—that have the greatest impact on the desired results (Patterson, Grenny, Maxfield, McMillan, & Switzler, 2008, p. 28). A key task of the analysis phase of the ADDIE, TDQAS, and other instructional design approaches is to identify the critical knowledge, skills, and abilities required to perform required tasks and then use these to inform the design process. Training, of course, is only part of the process of learning and adopting new and more effective behaviors. In D2, we will talk about the importance of designing the whole experience to optimize the behavioral change. In D3, we will discuss how delivery affects learning transfer, and in D4 and D5, how the transfer climate regulates the conversion rate of learning to performance. FIGURE D1.4. LEARNING AND DEVELOPMENT PRODUCES RESULTS THROUGH NEW AND MORE EFFECTIVE BEHAVIORS BY THE PARTICIPANTS

Successful training and development

More effective or efficient actions/behaviors

Improved business results

FIGURE D1.5. DESIGN LEARNING PROGRAMS STARTING WITH BUSINESS OUTCOMES AND THE BEHAVIORS REQUIRED TO ACHIEVE THEM

Experiences needed to produce them

CH001.indd 34

Required changes in actions/behaviors

Desired business outcomes

6/17/10 3:22:25 PM


35

The point here is that learning interventions ultimately contribute to the business by making it more efficient (increasing productivity, reducing waste) or by making it more effective (driving revenue growth) and that the intermediate step in either path requires a change in behavior to new and better ways of doing things. Which path and which behaviors are most critical depend on the organization’s business model, climate, and state of development. Therefore, it is essential to understand the current circumstances. Understand the Current Situation Once you understand the “big picture” business fundamentals of your enterprise, you are in a position to take the next step, which is to discover the highest value needs or opportunities that training and development could help fulfill. It is important to invest this effort upfront. Successful companies invest substantial time and effort to discover their customers’ most important unmet needs. In the Innovator’s Solution, Christensen and Raynor (2003) explain: “When customers become aware of a job that they need to get done in their lives, they look around for a product or service that they can ‘hire’ to get the job done.” Therefore, “companies that target their products at the circumstances in which customers find themselves, rather than the customers themselves, are those that can launch predictably successful products” (p. 75). In effect, line organizations “hire” learning and development to do a job they need done. The more fully the learning and development group understands the challenges that line managers face—the business ends they are trying to achieve—the better they will be able to deliver solutions that the organization values and rewards with continued investment. Indeed, Swanson (2003) argues that neglect of front-end analysis is one of the most common causes for training transfer failure. Which improvement targets offer the greatest potential for return is very much time- and situation-specific. They depend on a company’s history, environment, market, competitors, product life cycle, cost structure, and so forth. Circumstances change. The most pressing business needs or opportunities this year are likely to be different than last year’s. Therefore, the best investment of learning resources (the programs that will deliver the highest value) are also likely to have changed. Corporate learning organizations need to continually scan the environment and adjust their offerings if they are to remain relevant. In a well-run business, the largest shares of available advertising, sales, marketing, and other resources are committed to those product lines with the greatest potential for growth and profitability. Learning organizations should follow suit, deploying the greatest portion of their resources to opportunities and issues

CH001.indd 35

6/17/10 3:22:26 PM

36


that are “mission critical.” That may require taking resources away from—or even discontinuing—programs of lesser importance. Making such hard and vital resource allocation decisions is the essence of strategic management. But how do you decide which opportunities are most promising? Do Your Market Research Companies decide which customer needs offer the most attractive opportunity for their products and services through market research. Multiple sources of data are consulted. Studies, interviews, and focus groups are conducted to understand what customers want, how much they are willing to pay, how they will judge success, and how they should be segmented and targeted. Defining customer needs for learning and development initiatives should follow a similar approach. Since most key customers for learning and development are internal, the needs analysis should be faster and less expensive than for a consumer product, but for a major program it should be no less thorough. Two sources of information are available to identify the most valuable training interventions: “explicit knowledge” contained in business plans and reports, and the “tacit knowledge” of leaders, managers, and employees. Both will be needed to make an informed decision. You should begin by reviewing all of the “explicit” published information (strategic plans, business plans, task forces’ or consultants’ reports, and so forth) before taking busy managers’ time for interviews or focus groups. There are three reasons for this. First, it demonstrates your respect for business leaders’ time and the pressures they face. You should never waste a manager’s time asking him or her to go over things you could have easily read for yourself. Second, you will be afforded more respect—and probably more time—if it is clear that you have done your homework and have generated ideas of your own. Third, with the business plan as background, you will be able to ask more intelligent and penetrating questions and thus reach a deeper level of understanding than if you simply show up and ask to be enlightened. Get a copy of the business unit’s plan and read it carefully. Review any other relevant company documents, such as a strategic plan, consultant’s report, and competitive analysis. Read actively with an inquisitive mind. • • • • •

CH001.indd 36

What is the business unit trying to accomplish? What are the business leaders’ key objectives and initiatives? What do they list as the competitive threats or downside scenarios? What keeps them up at night worrying? What are the causes of suboptimal performance that education could help resolve?

6/17/10 3:22:26 PM


37

• Where are there inefficiencies or inconsistencies that waste significant amounts of time and money? • Where could an effective training or development program reduce the risk and improve the likelihood or magnitude of the success? The goal is to understand the business’s greatest unmet needs and opportunities and to formulate your own ideas about where learning could add value, rather than just waiting to be told. Understanding the business plan allows you to be proactive, to move from “order taker” to strategic business partner and performance consultant. Once you have reviewed the available documents and formulated some ideas about key needs, it is time to get out and speak with line leaders. Suggested interview guidelines are given in Exhibit D1.2. Avoid the temptation to interview only other learning leaders or the heads of human resources; the information you gather will be insufficient. However well-informed and well-intentioned they may be, they are one step removed from the action. Likewise, you cannot rely on a single manager’s point of view (see “Inadequate Input” below). You have to talk to the people who are in the arena, the people whose careers depend on delivering results. That’s crucial, because learning organizations that achieve real alignment and deliver against the needs of the business enjoy strong support, even in the face of financial exigencies (see Case in Point D1.3). Translate Needs into Desired Outcomes The goal of your discussions with business leaders is to define the desired outcomes in areas in which learning can help close a critical gap or seize a valuable business opportunity. Since the original publication of The Six Disciplines, thousands of learning professionals have used the Outcomes Planning Wheel (Figure D1.6) to help guide and structure discussions with line leaders about training needs. This simple, four-step approach works well both for prospective discussions with leaders about potential training solutions, as well as to respond to a request for training. We have modified it slightly from the original based on our experience and suggestions from readers and clients. The Wheel is simple to draw—on a flip chart, for example—to help facilitate a group discussion, but also works well one-on-one, even over the phone. Although the questions are numbered and initially approached in order, answers to one question often belong in another quadrant or lead to follow-up questions. Be flexible and modify the flow as necessary to get the information you need; do not feel slavishly bound to the form.

CH001.indd 37

6/17/10 3:22:27 PM

38


Exhibit D1.2 Interview Guidelines for Discovering Business Needs Preparation • Do your homework; read the relevant business plans, reports, and related materials. • Schedule your interview with the business leader in advance; state the objective and time required. • Know what you want to get out of the interview before you begin. The Interview • Follow the classic sales opening: meet and greet, state the value, propose an agenda, check for agreement. • Start on time and end on time. • Use open questions, check for understanding by restating; probe for the deeper issues. • Summarize what you understood from reading the plan and check your understanding: “From what I was able to read, it seems like the most important things you are trying to accomplish are X, Y, Z. Did I summarize that correctly? What have I missed?” • Identify the main gaps between current and desired performance. Are there areas in which people lack key skills or capabilities that will make it difficult to achieve the objectives? What one improvement in terms of people’s skills or behaviors would most help the leader achieve his or her objectives? • Are there new initiatives, opportunities, or strategies that will require new knowledge or skills to execute? • Use the Outcomes Planning Wheel (Figure D1.6) to help identify the links among overall business needs, desired behaviors, confirmatory evidence, and conditions of success. Follow-Up • Immediately following the interview, summarize the discussion in writing. • Include the answers to the four questions of the Outcomes Planning Wheel as an efficient way to set out your understanding of the linkages between the results desired and the capabilities needed to achieve them. • Send a thank-you note to the person you interviewed and include a copy of your summary (“Thank you very much for your time. I found our discussion very helpful and enlightening. I attach a brief summary of the key points. Please let me know if I missed or misunderstood anything.”). • The purpose is four-fold: • Summarizing your notes will encourage you to reflect on what you have learned and better cement it in your memory. • The document will be a useful reference as the planning process proceeds. • Your summary acknowledges that you valued the leader’s time and input. • Finally, sharing your summary with the person you interviewed provides him or her with the opportunity to correct any oversights or misunderstandings, which will help you create a superior solution.

CH001.indd 38

6/17/10 3:22:27 PM


39

Case in Point D1.3 Learning to Effect Change Health care is a rapidly changing industry. The pace of change places unprecedented demands on its leaders. Skills and approaches that were successful in the past are no longer adequate; new leadership competencies and insights are needed. Ray Vigil, the chief learning officer for Humana, Inc., believes strongly that the leadership development strategy must be an integral part of the overall business strategy; it must begin with the end in mind. Because these conditions have been met at Humana, the Leadership Institute is viewed by the top management as an important enterprise-wide tool with tremendous positive business impact. “The CEO wanted to take a very traditional company and move to a consumer-centric point of view, approaching problems and the industry very differently, so that we produced competitive advantage by a disruptive change strategy in the way we go to market. But historically, leaders in health care have not had the same kind of leadership development focus that other industries have had. So the big challenge we had initially was how to take a management team that had been successful with very traditional functional skills and get them to realize that they needed to embrace a very different vision; they had to develop the capability to work across the enterprise, not just in their function. “Our CEO had been working with the senior leadership team to develop an enlightened view and a strategy for becoming a consumer-centric company. While leaders understood the strategy intellectually, they were having difficulty understanding what they had to do differently. “Most people would say, ‘Gee, if you understand something, it ought to be obvious that you take action on it.’ But I think it is a very big challenge for people to execute a strategy that is very different from the mindset and skills that brought them success in the past. “From the beginning, when our CEO said, ‘I know I need a chief learning officer,’ I interpreted that to mean that he had a business strategy that was going to require a great deal of change leadership, and he needed a learning strategy to complement the business strategy. I was able to convince my team that we needed to think about what we were doing in a strategic sense. We needed to really drill down on where the CEO wanted to take the business and what it was going to take to get us there. What did the future state look like? How do you implement and execute that strategy with the human capital we have? “We devised something we called the touch-point program to identify the key intersections, the key leverage points with the organization that could make a difference. We identified the key stakeholders and made them part of the design team. They helped us design the whole process, including the evaluation and after-action review. When it came time to implement, everybody was shocked that there was such great acceptance, but that was because we had identified the key leverage points and through the touch-point campaign had incorporated their interests and thinking into the program. Because we started with an outcome of use and what the end state was supposed to look like, we were able to integrate all the pieces into a coherent strategy and an integrated experience for the participants.”

CH001.indd 39

6/17/10 3:22:28 PM

40


FIGURE D1.6. OUTCOMES PLANNING WHEEL

4.

1.

What are the specific criteria of success?

What business need(s) will be met?

3.

4 1 3 2

What or who could confirm these changes?

2. What will participants do differently and better?

© 2010 Fort Hill Company. Used with permission.

As you move from a reactive training role into one as a performance consultant, remember that many business leaders are used to simply “ordering” training and have not thought deeply about the questions at the heart of the Outcomes Planning Wheel. You will need to ask some probing and clarifying questions to achieve the mutual level of understanding necessary to develop a truly effective program. You may even encounter resistance from some managers who may feel challenged or threatened by your interest in trying to understand the real rationale behind “we need a program.” The best approach we have found is to assure the leader up-front that you have the company’s best interest in mind and that the reason for the interview is to maximize the probability of success by saying something like: “Because training

CH001.indd 40

6/17/10 3:22:28 PM


41

will consume participants’ time and company money, I want to be sure we deliver the best possible results for you. I’d like to ask you a series of questions to be sure I understand the business drivers behind the request and the outcomes you really want from the training. That will help us design the best and most cost-effective solution.” Throughout the conversation, practice active listening. Stop and check for understanding periodically: “Let me be sure that I have understood you correctly. [Restate the needs, behaviors, etc. as you understand them]. Is that correct?” Likewise, do not be afraid to admit you do not understand something: “Could you explain what you mean by . . . .” “Help me understand . . .” is a very useful phrase that can be used to genuinely seek understanding, but also as a graceful way to point out inconsistency or confusion, as in: “Help me understand how the training course you have requested is linked to your business needs.”

The Four Questions There are four key questions to the Outcomes Planning Wheel. Each is discussed in turn below. 1. What Business Need(s) Will Be Met? The first question is about the business needs that will be met (or opportunities realized) if the course is a success. The goal is to gain clarity about the real business issue or opportunity behind the request for training. This is critical not only to inform the design of the program, but to make sure that training is an appropriate part of the solution (see “Training Traps” below). Being able to tightly couple training to specific business needs increases the buy-in of the participants and their managers. Learning organizations that have adopted the 6Ds find that they enjoy greater support for their programs when they include the business purpose in the course description rather than when they simply list the topics covered or learning objectives. For example: “This course will help you increase the productivity of your team by . . .” or “This program will help you increase sales by . . . .” You will probably have to ask a series of clarifying questions to get to a definition of business need that is specific enough to be actionable. A manager’s first answer to this question often simply asserts the need for training, “We need a training program on the order entry system,” as opposed to stating the business need behind the proposed solution or what issue the training program is meant to resolve. The challenge in such cases is to help the client reframe the question: “to transition from a focus on the solution to a focus on business and performance results” (Robinson &

CH001.indd 41

6/17/10 3:22:29 PM

42


Robinson, 2008, p. 171). You need to probe for the ultimate, measurable, businessrelated outcome behind the proposed training: “Can you help me understand what is behind your request for training? If the training is successful, how will the business benefit? For example, will we experience fewer errors, greater speed, lower cost, or something like that?” The perceived need for training usually originates because management believes that something is not happening that should be (“the sales clerks are not asking customers about related supplies”) or that something is happening that should not be (“too many mistakes,” “inappropriate behavior,” and so forth). Your goal is to draw out the underlying perception and why it is a problem: the cost or lost opportunity for the organization. A second common situation is for the initial answer to define the desired result, but in terms too general to be useful, for example, “We need to increase sales.” In this case, you need to probe for details, for the intermediate steps or aspects of the process that require attention: “You have been very clear about the urgent need to increase sales. Can you say more about the specific aspects of the sales process that you feel we most need to address or improve?” 2. What Will Participants Do Differently and Better? This second question is designed to help identify the vital behaviors or actions needed to effect the desired results. The core concept is the quip often ascribed to Einstein: “One definition of insanity is to continue doing the same thing and expect a different result.” In other words, people have to do something in a different way to produce a different result: Just ‘knowing” or “understanding” is not enough. Participants have to put their new knowledge and skills to work in ways that are different, better, and more effective than what they were doing previously. The learning organization needs to know what the desired behaviors are in order to design effective instruction and help create a positive “transfer climate” (see D4) that supports those behaviors on the job. Ask the client to describe the desired changes in behavior: “If the training is a success, and we were to watch how people perform their jobs afterward, what would we see them doing that is different and better?” Another useful approach is to ask about the behaviors of top performers: “What do the top performers do that lesser performers don’t? If the training is successful, more people should do what the top performers do. What would that look like?” 3. What or Who Could Confirm These Changes? The goal of this third question is to start the dialogue about how to validate whether or not the training is producing the desired outcomes. The core concept is that the only way that learning organizations can succeed is if they know how

CH001.indd 42

6/17/10 3:22:30 PM


43

success is defined by the customer. The latter phrase is in italics because, as we will explain in greater detail in D6, it is the customer who ultimately decides whether the initiative was a success or failure. Engaging the customer (which, in business, usually means the department that is paying the bill) in a discussion of what could be measured facilitates the discussion of what should be measured and how success will be defined (Question 4, below). The right time to have those discussions is in the project definition phase since it affects everything else—from designing the complete experience (D2) to documenting the results (D6). The initial discussion of Question 3 should be a shared brainstorming session. “How can we be sure the training is working? That is, if we get the training and the transfer climate right, the participants’ performance should begin to improve. Who will notice these changes first? How can we find out whether people are really using what they learned? What else will change or be measurable?” You may need to “prime the pump” here by offering some suggestions to help your client think about potential outcomes and how they could be confirmed. Examples include: • If we were to actually observe participants doing their jobs, would we be able to see the change? What would we look for? • Who would notice a change as part of their normal interactions with the participants, for example, customers, managers, or direct reports? Who could we ask? • Would any of the business metrics that we track routinely (sales, quality, customer satisfaction, and so forth) change? Which? It turns out that while there are a very large number of potential outcomes, there are only a small number of types of outcomes and ways to assess them. Use Table D1.3 to help guide your brainstorming. In the early stages of the conversation, the goal is to explore a number of options for measuring the outcome. However, sometimes the changes that the customer wants to see (for example, change in turnover) will take so long to become evident that the impact of training will be obscured by other factors. In such cases, encourage the client to help you identify “leading indicators.” Leading indicators are the earliest evidence that things are going in the right direction. Your follow-up discussion might sound like this: “We agreed at the beginning of our discussion that improving retention is a long-term business need and goal. The problem with relying solely on that to measure the impact of training is that we won’t see a significant change in retention rates for months. By then it will be hard to figure out whether it was the training or something else. What could we measure sooner that would give us an indication that the training is having the desired effect? A survey? Employee commitment scores? 360-degree feedback?”

CH001.indd 43

6/17/10 3:22:30 PM

44


TABLE D1.3.

MAJOR CATEGORIES OF POST-LEARNING OUTCOMES AND WAYS TO DOCUMENT THEM

Type of Outcome

Potential Data Sources

Potential Data Collection

Change in behavior

Participants themselves Participants’ managers Customers Trained observers

Survey Interview Observation

Improved opinion by key stakeholders

Customers Direct reports Managers Others

Satisfaction survey Interviews Focus groups

Improved business metrics

Company IT system Independent tracking resource

Data extraction Data purchase

Improved work product

Samples of work

Expert review or observation

Identifying and assessing leading indicators is important to get an early read on whether the training is having the desired effect and to support continuous improvement of subsequent classes. 4. What Are the Specific Criteria of Success? Once you have listed all the possible outcomes that might be measured, it is time to agree on what will be measured and when, and how much of a change is required to consider the program a success. It is vital to get agreement on the “conditions of satisfaction” in advance since these are the deliverables section of the contract between the learning organization and line management. The discussion of Question 4 should be primarily a selection process, winnowing down the possible choices identified in Question 3 to the critical few. It should include a high-level discussion of how they will be measured; for example, will self reports suffice? Your goal is to come to agreement with the business leaders on the key outcomes and acceptable approaches to documenting results, not to develop a detailed evaluation plan—that is the work of D6. For example, suppose that the ultimate business need is to reduce employee turnover, and that you and the sponsor have agreed (as the research shows) that a leading indicator of intent to stay is a good working relationship between the employee and his or her supervisor. For a management training program, then, you might define the specific measure of success as: “On a survey three months

CH001.indd 44

6/17/10 3:22:31 PM


45

after the program, at least half the direct reports of attendees agree that their relationship with their manager has improved.” For other kinds of programs, the criterion for success might be a change in a tracked business metric (quality index, customer satisfaction, average sales), while for others, a manager’s evaluation of the participant’s performance might be an adequate measure of success. There is no one right answer. The goal of Question 4 is to come to agreement on what the program is supposed to achieve in terms that the customer considers relevant and credible and which the learning organization believes it can achieve given the resources available and the level of support for learning transfer. Defining the criteria of success should be a “give and take” discussion. The learning organization should not blindly accept impossible targets—since sooner or later it will be called to account. While committing to specific levels of achievement may feel uncomfortable and foreign to learning professionals, it is, as Basarab (in press) points out, the norm in business. Sales leaders have to commit to achieving specific levels of sales, marketers to specific market share, CEOs to profitA clear set of business ability and growth targets, heads of outcomes from training manufacturing to lower cost or higher enhances the credibility of the quality. A known target allows the learnlearning organization. ing organization to optimize the learning experience and to implement continuous improvement. And, perhaps most important, a clear set of business outcomes from training enhances the credibility of the learning organization, earning it a real seat at the table and its long-sought goal of being embraced as a true business partner. Create Co-Ownership Once the desired business outcomes have been clarified, there is one more critical question to discuss: “Besides training, what else needs to be in place to achieve these results?” This is a critical question to raise because training is never the whole answer. To get a lasting change in behavior requires coaching and encouragement from managers; support when people run into difficulties; and recognition and reward systems that are aligned with the desired actions. As Mosel pointed out more than fifty years ago: “It is top management, through the organizational climate or reward structure it creates, that is really doing the training, regardless of what the training staff does. The training administered by the training staff ‘sticks’ only if it coincides with what top management is teaching every day” (Mosel, 1957, emphasis in original). Use the final “what else” question to begin to educate the sponsor on the extent to which factors in addition to training (see D4: Drive Learning Transfer)

CH001.indd 45

6/17/10 3:22:31 PM

46


influence the success or failure of the effort. “We can certainly teach people how and why these behaviors are important and we will put in place systems to support and document them. But what else needs to happen to support and encourage these behaviors on the job? I’m thinking about everything from coaching to tools to reward systems.” Another useful way to help clients think about the post-training environment is to ask: “What currently gets in the way of people performing in the desired fashion? What barriers can we reduce or eliminate to increase the probability of success?” Your goal is to help the sponsor realize that the business is co-responsible for the success or failure of any training initiative. You need them to commit, as their part of the “contract,” to creating a supportive transfer climate. Training needs clearly defined business outcomes to succeed, but it also requires a supportive environment and the cooperation of management. Enforce Priorities Once you begin to think proactively about the areas in which learning could contribute to the success of the business, you will discover many potential programs that could make a meaningful difference. By no means should all of these be pursued. Learning organizations—like other business functions—need to pick their targets carefully. No company has sufficient resources to pursue every market opportunity; no learning and development group can take on all the potential places in which performance could be improved through training. The danger of trying to pursue too many opportunities simultaneously is that you under-resource all of them with the end result that none make a meaningful contribution. It is better to execute a small number of important initiatives well than to execute a large number badly. The general rule of thumb is that the more limited your resources, the more discriminating you must be, directing your efforts where they are likely to produce the greatest return. Targeting your resources on opportunities that have the greatest potential necessarily means saying “no” to some requests. Learning organizations cannot—and should not—attempt to satisfy every training request or pursue every opportunity, especially when saddled with fixed or shrinking resources. There needs to be a rigorous and defensible basis for moving ahead with some opportunities and denying or delaying others. In this regard, a learning advisory council or steering committee can be very helpful: Senior executives, from multiple business units and corporate functions, should regularly meet and provide input on what needs to be done (and win implicit buy-in in the process). They will help push programs through the approval and budgeting process and provide timely and accurate feedback

CH001.indd 46

6/17/10 3:22:31 PM


47

on program effectiveness. Educational programs will carry significant weight if they are supported by the advisory board and board support will increase the credibility of the entire executive education function. (Saslow, 2005, p. 45)

Three factors must be weighed in the prioritization process: • The upside potential. This is the positive business impact that will be realized if the program achieves the business outcomes for which it was created. • The downside costs of killing or delaying a program. These might include lost opportunity, adverse legal or regulatory exposure, inadequate leadership pipeline, and so forth. • The probability of success. A program might well have significant potential upside, but if line management is unwilling to invest the time and resources necessary to create a favorable transfer climate, then the probability of achieving the projected results is low and the expected payback should be significantly discounted. To help ensure that such factors are taken into consideration, some companies, such as Randstad, use a business unit sign-off form than prompts managers to think about the magnitude of the problem and the value of resolving it (Bersin, 2008b). One of our biopharmaceutical clients requires business units to force rank potential training opportunities in terms of their potential business value. Regardless of the specific mechanism used, the key is to have a process that involves business leaders in making sure that scarce training resources are focused on the highest-value opportunities.

Avoid Training Traps The human development field is rife with traps for the unwary. In our consulting practice, we encounter too many learning organizations that have succumbed to one or more of the many potential pitfalls (Case in Point D1.4). Some of the most common pitfalls are listed below, with suggestions on how to avoid them, because nothing is more tragic than to have designers, facilitators, and technicians pour their energy and creativity into a program that doesn’t lead to valued results. It’s what Peterson and Nielsen call “fake work” and it is tragic because it is just as hard as real work and because it consumes time, resources, and people’s careers, even though it doesn’t actually advance the organization’s mission (Peterson & Nielsen, 2009, p. 4).

CH001.indd 47

6/17/10 3:22:32 PM

48


Case in Point D1.4 Detailed Roadmap to Unclear Destination A major corporation consulted us about how they could increase the impact of a soon-tobe-launched management development program. We started through the Outcomes Planning Wheel. We asked: “What’s the driving business issue? How will the company benefit by having someone attend the program? What will participants do better and differently as a consequence of attending?” There was an awkward silence. The chief of learning turned to the program director. The program director deferred to the instructional designer. “That’s a good question,” they all said. But no one had the answer. They had a detailed plan—practically to the minute—of what was going to happen during the course. They knew which leadership models they were going to use. They had purchased a custom simulation and hired big-name speakers. They had a list of learning objectives—what knowledge participants would acquire—but no one could explain how these were linked to the business of the business. They were about to launch the program to hundreds of mid- and upper-level managers, but no one responsible for putting it together could explain the key business drivers or how attendance would increase productivity, generate new revenue, or otherwise create value for the company. This was a very smart, dedicated, hard-working, professional, and talented team, yet they had fallen into the trap of focusing so hard on the “how” that they had lost sight of the “why.” Their experience suggests just how easy this is to do and why learning and line leaders alike must keep coming back to the business rationale and define the intended outcomes in business terms.

How can such a thing happen? Why do even savvy learning organizations get off track? Our experience suggests that there are five key training traps that must be avoided: • • • • •

Training as cure-all Having a program to have a program Confusion between means and ends Laudable intent Inadequate input

Training as Cure-All Not every performance problem is amenable to training. One of the most common reasons that training fails to deliver satisfactory results is that the problem

CH001.indd 48

6/17/10 3:22:33 PM


49

being addressed was not one that could be solved by training in the first place (Phillips & Phillips, 2002). If the productivity of customer support personnel is low, for example, because the computer system they use is slow and unreliable, no amount of training will noticeably improve the situation; better results will be achieved by expending resources to upgrade the software, equipment, or both. If, on the other hand, productivity is low because support personnel do not know how to use the software efficiently or take advantage of shortcuts, then the right training with the right performance support will produce significant improvement. Unfortunately, many managers’ knee-jerk reaction to any performance issue is “We need a training program,” without really analyzing the root causes of the problem. Any number of deficiencies can result in suboptimal performance; training can only address performance issues related to a lack of skills or knowledge (Stolovitch & Keeps, 2004, p. 11). Mager and Pipe (1997) suggested that the acid test is: “Could they do it if their lives depended on it?” If the answer is yes, “then you can forget training as a potential solution. After all, ‘teaching’ people what they already know how to do isn’t going to affect the performance discrepancy” (p. 93). If people know how to do something correctly, but they aren’t, then their poor performance stems from a lack of motivation, unclear expectations, insufficient feedback, or other managerial or systems failures (Exhibit D1.3) and training is not the solution. Learning professionals, in their roles as performance consultants, need to be sure that they have investigated and eliminated other causes of poor performance first. “Training is expensive to design and deliver; it should be the last, not the first, intervention the HRD professional and the organization should consider in order to improve employee performance” (Broad & Newstrom, 1992, p. 5). Nothing is more demoralizing to the learning organization, more frustrating to participants, or more damaging to learning’s reputation than to take on a training assignment that is doomed to failure because the real problem is a breakdown in management or systems. We have become increasingly concerned about the misuse of training to try to solve all manner of unrelated performance problems; it is giving learning a black eye. In our workshops, we ask learning professionals what percent of their programs fail because training was not the right solution to the problem. Estimates are consistently 10 to 25 percent, and in one company, 40 percent. That suggests that somewhere between one-tenth and one-quarter of all training programs are doomed to failure because the real issues have not been identified or addressed. Training is being used as a sop by management to give the appearance that something is being done. No wonder there is often cynicism about the effectiveness of training. Workplace learning professionals must not let themselves be used as patsies.

CH001.indd 49

6/17/10 3:22:34 PM

50


Exhibit D1.3 Causes of Poor Performance That Can Be Solved Without Training UNCLEAR EXPECTATIONS ❑ If employees do not know exactly what is expected of them, make sure performance standards are made clear. Suboptimal performance is often due simply to inadequate understanding of what is wanted.

LACK OF FEEDBACK ❑ If employees have not received unambiguous feedback on their current performance, see that they get it. Training is sometimes used to avoid providing direct feedback on sub-par performance. Even if people understand what is expected, they need feedback on their performance in order to improve it.

LACK OF INCENTIVE ❑ If there is no positive recognition or reward for doing things right, see that there is. Training cannot solve a motivation problem. Being recognized for a job well done is a powerful motivator. Yet an astounding number of employees say they have not received any positive feedback for months on end.

LACK OF CONSEQUENCES ❑ If there aren’t currently any negative consequences for not performing, make sure there are. If there are no consequences for not performing in the desired fashion, most people will take the path of least resistance, and attempting to train them to do otherwise is a fool’s errand. One poor learning professional in one of our workshops had been asked to put together a half-day program on hardhat safety because workers were not wearing their hardhats on the job site. We asked what happened if they were on site without a hardhat. The answer was “nothing”—which was the problem—not a lack or knowledge or skill.

INADEQUATE TOOLS, NO ACCESS TO NEEDED INFORMATION, OVERLY COMPLICATED PROCESSES, AND SO FORTH ❑ If there are other obstacles to satisfactory performance—such as inadequate tools or insufficient access to information—these have to be addressed directly; they cannot be resolved by training. Use training only to address issues stemming from inadequate knowledge or skills that cannot be solved by simple job aids or other performance support.

CH001.indd 50

6/17/10 3:22:34 PM


51

So what do you do when you are asked to create a training program when you know that training is not the right solution to the problem? You follow Stolovitch’s advice and say immediately, “I can help you solve your problem” (Stolovitch & Keeps, 2004, p. 16). Because you can, even though it won’t be by providing training. Indeed, if you can show, through skillful questioning and analysis, that the problem is not a lack of knowledge or skills—and therefore one that can be Even when training clearly is solved more effectively and less expenpart of the solution, it is never sively with a non-training solution—you all of the solution. have helped your client, the company, and the learning organization a great deal. If, on the other hand, the first words out of your mouth are, “I am not sure that training is the right solution,” you may be right, but you may be seen as negative and uncooperative, and the manager may just go hire a training vendor to provide the wrong solution anyway. You will have missed an opportunity to educate your client and to help your organization and yourself. Remember that even when training clearly is part of the solution, it is never all of the solution. In the next chapter (D2), we will consider the criticality of designing the complete experience, which includes all four phases of learning, together with the performance support systems and environmental factors that determine whether or not a program delivers its objectives. Because learning and development is increasingly being held accountable for demonstrating meaningful business impact, learning organizations must become adept at avoiding assignments for which training is not the answer. We Need a Program When the main purpose of having a program is “to have a program,” the initiative is in trouble from the start. This problem typically begins when someone in upper management decrees that the company needs to have a program on something or other (see Figure D1.7). That isn’t itself a problem when the impetus is a real business need that can be addressed by training, such as insufficient skills in marketing, leadership, or sales. But programs also spring from causes that are less results-driven, such as the need to “have some training” as part of a meeting, a business book extolling the latest theory, a consultant, a conference, a competitor’s program (especially if it attracted media attention), or a pointed question from the board (“What are you doing to accelerate innovation?”). Whatever the inciting event, the assignment is passed down the line as, “The boss wants us to put together a program on X.” Well-meaning subordinates

CH001.indd 51

6/17/10 3:22:34 PM

52


FIGURE D1.7.

TRAINING TRAP 2: WE NEED A PROGRAM

© 2010 by Randy Glasbergen; www.glasbergen.com. Used with permission.

scramble to fill the order. Experts are consulted; vendors are vetted; models and theories are debated. After the expenditure of much time and energy, a “custom” curriculum is designed; logistics are planned; speakers are hired. The only detail that has been overlooked is the business driver: What results are expected beyond “having a program?” How will “having a program” benefit the business? What outcomes are we trying to achieve? It is very easy, especially in large organizations, for the program—which usually means “the event”—to become an end in itself. When this happens, success becomes defined as the number of attendees and the scores on endof-event evaluations. But neither of these drive the organization forward nor justify the cost. The higher in the organization that the idea for the program originated, the less likely that it will be challenged. “Having a program” will become an unquestioned organizational imperative. Jack Welch is reported to have said that one of the problems with being CEO is that “You ask for a cup of coffee and they go out and buy Colombia.” If the CEO wants a program, what human resources or organizational development staff member is going to have the temerity to ask: “Why?” or “What’s the payoff for the company?” Yet those are exactly the kinds of questions that must be asked if learning and development is going to succeed.

CH001.indd 52

6/17/10 3:22:35 PM


53

Susan Burnett, senior vice president for talent and organizational development at Yahoo!, told the story of being interviewed at The Gap: “When I was being interviewed, the CEO said, ‘I want Merchant University.’ And I said: ‘Why?’ If I had not had that conversation, I would not have known.” (See Case in Point D1.5.)

Case in Point D1.5 Good News, Bad News When Susan Burnett became senior vice president of talent and development for Gap, Inc., she got some good news. “I came in on the day they were finalizing the budget and sat down with my new boss. She showed me the budget and she said, ‘Hey, I have some good news for you. The learning team got $6 million in incremental funds to produce Merchant University, Design Academy, and a new on-boarding process.’ I said, ‘OK, cool, six million dollars, that is great. But why does the business need these programs? What business results are we supposed to produce for Gap, Inc?’ “And so I interviewed people, talked to managers. I am voraciously interested in how the business works and how we make money—always have been. It comes from years of line management. So here was the perfect opportunity. I was a new leader, in a new business, and I needed to really understand the drivers for success. “I learned about Gap’s business strategies for growth, operations, and people. As I talked with the executive team and their leadership teams, I learned that Gap’s transformation would come from building new leadership capabilities in our people and new organizational capabilities in product development, supply chain, and IT. I learned that the merchant job was central to our brand’s success, and that Gap had invented the role of the merchant in the late 1980s, evolving from buyer to merchant leader. I also saw that the current reengineering of the product pipeline would require the reinvention of the merchant role again. And I learned that turnover of our new employees and our key merchant talent was unacceptable. “I was excited that my learning organization would have the opportunity to make a big difference in achieving Gap’s business objectives. We could stem the new employee turnover tide with an awesome hiring-to-on-boarding process that improved productivity and performance in the first ninety days. We could accelerate understanding of the new merchant role, the new concept to customer pipeline, and build the new merchant general management capabilities desperately needed by our business.” But then she discovered the bad news. The learning team was not connected to the priorities of the business. They were working hard to produce legacy programs, but they were not working on the real business issues. They were not connected to the business strategies and core changes the new leadership was driving. As a result, “I got a proposal for Merchant U that wasn’t connected to the transformation the business needed for success. I also saw that my team didn’t have the critical business relationships and connections that would get them the information they needed. It was a recipe for failure.”

CH001.indd 53

6/17/10 3:22:36 PM

54


The stronger the command-and-control culture of a company, the more likely everyone will exert best efforts to fulfill a program request without ever stopping to ask, “Why?” The result will be a learning initiative that lacks grassroots support because there is no clear line of sight to the mission and goals of the business. Line managers will grumble among themselves about the cost or wasted time, but most will go along with it in order to save their powder for bigger battles. A conspiracy of silence develops; the discussions of the program will all take place in what Connolly and Rianoshek (2002) call “pretense” mode—everyone saying the right things but no one addressing the real concerns. And the poor learning organization—having done its best to fulfill what it thought it was asked to do— will not understand why its contribution is not valued as highly as the management rhetoric suggested. Programs created for the sole sake of “having a program” are doomed. The antidote is to establish D1 as a guiding principle across the organization: No program, regardless of who requests it, will be executed without a clear rationale based on expected business outcomes. That will help create a climate in which it is normal to question the business justification for every proposed program and in which only those initiatives for which there is a compelling business need are executed. Confusing Means and Ends Just as “having a program” is inadequate as a rationale, “having given a program” is inadequate as an outcome. Many corporate learning organizations still report the number of people taught, hours of instruction, the number of e-learning courses completed, and so forth as if they were results. Learning management systems have contributed to the problem by making it easier to collect, generate, slice, and dice such data and turn them into PowerPoint slides and graphs. But such statistics are measures of activity, not productivity. They are measures of inputs, not outputs. The well-known human resources consultant Dave Ulrich related a meeting he had with the chairman and top human resources leaders from a large bank. “The training person said that 80 percent of employees have done at least forty hours in classes. The chairman said, ‘Congratulations.’ I said, ‘You’re talking about the activities you’re doing. The question is: What are you delivering?’” (quoted in Hammonds, 2005). The true goal of learning and development—and the standard against which it will increasingly be measured—is the extent to which it contributes to a company’s prosperity and competitiveness. The activities involved—courses, hours, instructors, coaching, podcasts, blog entries, and all the rest—are the means to an end, not the end itself. They are important for the learning organization

CH001.indd 54

6/17/10 3:22:36 PM


55

to track for the internal management of the function, but they are not the business outcomes for which the unit was created and funded. Confusing means with ends is hardly peculiar to learning and development. In any initiative, it is easy to become so focused on activities that the original purpose is lost. That is, “having lost sight of our objectives, we redoubled our efforts.” Activities are seductive; they are observable, quantifiable, and easy to measure, graph, and report. But they do not equate to value added. Writing about corporate change initiatives, Schaffer and Thomson (1992) said: “At the heart of these programs, which we call ‘activity centered,’ is a fundamentally flawed logic that confuses ends with means, process with outcomes.” A focus on activity rather than results leads to a situation in which “The performance improvement efforts of many companies have as much impact on operational and financial results as a ceremonial rain dance has on the weather” (p. 2). The fault is management’s, or, as the comics character Pogo said: “We have met the enemy and he is us.” For years, learning and development organizations have been treated as pure cost centers rather than contributors to productivity. The result has been undue emphasis on efficiency (cost containment) rather than efficacy (cost benefit), on activity (programs or hours of training) as opposed to results (increased sales, greater efficiency, better leadership, more effective marketing, higher quality service, and so forth). Training departments, consultants, and vendors do not promise specific results because, historically, they haven’t been required to. As long as companies are willing to pay for activity, only a fool would offer more. This is in stark contrast to the standards for line departments. If a sales manager requests funds to increase the size of the sales force, she must be able to demonstrate how this will be translated into more sales, not merely more sales calls. A sales manager is measured and rewarded for achieving the promised increase in revenue, not merely for increasing the size of the sales force. Similarly, if the head of manufacturing requests funds to retool a plant, he had better be able to justify the investment in terms of increased productivity, lower costs, higher quality, and so forth; and he had better deliver on those promises. That is the way business works; activity alone is not rewarded. Teresa Roche, CLO for Agilent Technologies, understands this. Together with the prestige of being asked by the CEO to lead a high-profile, transformational leadership program came increased pressure and accountability for producing results. Roche said she welcomes the scrutiny. “If I’m going to be given something to work on and somebody is investing dollars and I’m being held to account to that, I wouldn’t want it any other way,” she said (quoted in Prokopeak, 2009). Practicing D1 and D6 religiously—requiring that business outcomes be specified for every program and then documenting them—will allow learning

CH001.indd 55

6/17/10 3:22:36 PM

56


organizations to avoid the means versus ends pitfall. Doing so ensures that learning organizations are subjected to the same rigorous scrutiny to which a well-run business subjects its line operations, and that they are recognized and rewarded for producing results. It is a manifest disservice to both learning professionals and shareholders to apply a lower standard to evaluate educational programs than is applied to other business processes. Laudable Intent A fourth pitfall for learning organizations is programs that are positioned on the moral high ground. Challenging the value of a diversity, safety training, or leadership development program is akin to questioning the value of motherhood, patriotism, or equality. Managers who value their careers are reluctant to challenge politically correct programs that have laudable intent. The result is that learning and development initiatives on such topics are rarely subjected to the same kind of rigorous review and debate as other expenditures of comparable magnitude. Marketing plans, for example, are closely scrutinized and frequently challenged. The debate about a marketing program is not about whether it is good or necessary to do marketing, but rather about whether the proposed plan is the best possible approach and whether it is likely to deliver the promised results. There should be equally candid discussions about learning programs in leadership, diversity, total quality, or other areas of laudable intent. Such debate should not be about the merit of such ideals, which is a given, but should rigorously explore how well the program serves or will serve the organization and whether the approach is likely to produce the desired outcomes. If a program is not producing the desired results—no matter how politically correct or laudable—it should be scrapped and replaced (see Case in Point D1.6). Learning organizations that practice D1—keeping the focus always on the desired business outcomes— encourage discussion about the organizational goals for every program and the best means of achieving them. Inadequate Input We continue to be surprised by the number of major learning programs that are implemented without adequate input in otherwise well-managed companies. Some are developed entirely within the human resource or training organization and are put into practice with little or no input from line leaders. While it is true that human resources, organizational development, and learning organizations have deep expertise in their respective disciplines, they are not the consumers of

CH001.indd 56

6/17/10 3:22:37 PM


57

Case in Point D1.6 It’s Your Ship When Captain Michael Abrashoff took command of the USS Benfold, one of his early actions was to cancel the diversity training program—not because he did not believe that diversity was critical to the success of the Navy, but because the current program was not achieving the outcomes for which it was intended: reducing racial tensions and sexual discrimination and improving teamwork. Because he was focused on the business outcomes, he wasn’t content just to have a program that everyone attended. When it was clear that the program wasn’t producing the required results, he eliminated it. “I could have been fired for that, but in my view it was common sense that a program that produced such awful results was clearly ineffective. I had no intention of allowing anything ineffective on Benfold” (Abrashoff, 2002, p. 170). Instead, he replaced it with something more effective. A program should not be exempt from having to deliver results just because it has high ideals and laudable intent; indeed, because their inherent merit is so great, such programs should probably receive increased scrutiny to be sure they are achieving their lofty goals.

the program, nor are they directly accountable for the bottom line. Their perspective on the business is different from that of line leaders; they have less hands-on experience managing hard business metrics. If they consult only among themselves, they may design a program with strong learning objectives but only weak links to the essential business outcomes. Equally dangerous is relying on only a single source of information about the performance gaps and their causes. The Outcomes Planning Wheel (Figure D1.6) we provided has proven useful in facilitating discussions with business managers about performance needs and the outcomes desired. We do not mean to imply, however, that you should rely on just one point of view. A manager’s perspective is different than that of the people who actually do the work. The Robinsons (2008, p. 80) recommend consulting a minimum of two different sources for information such as employees, managers, customers, business documents, or operational data. The success of training—or any other intervention, for that matter—depends on truly understanding the problem you are trying to solve. It is therefore vital to invest the time and effort to gather the data you need to gain a comprehensive view of the business outcomes sought and the skills and behaviors required to achieve them. The payoff is that programs that are clearly linked to business strategies will enjoy strong support from senior line leaders and earn the learning organization a true “seat at the table” (see Case in Point D1.7).

CH001.indd 57

6/17/10 3:22:37 PM

58


Case in Point D1.7 Linking Learning and Strategy Plastipak Packaging is a global, $2 billion manufacturer of rigid plastic containers. Driven by the goal of enabling company leaders to manage the challenges and opportunities of rapid growth, the company launched Plastipak Academy in 2006. Its mission: deliver targeted learning solutions aligned with current business needs, equipping leaders enterprise-wide to grow the business. To change the paradigm about the role of learning, Academy leader Diane Hinton and her team have worked hard to hold conversations with senior managers that differ from what they typically expect from a learning unit. The goal is to ensure that learning at Plastipak is driven by business needs and strategy. As she explained: “We ask about their business challenges—not their training needs—to identify strategic priorities. For example, to jumpstart our needs assessment process, Mary Singos interviewed our company’s top twenty-six leaders about their business challenges during one-hour, one-on-one, gut-level conversations. She crystallized each dialog into need statements that the leader checked for accuracy and rated on business impact and priority. “We then compiled and presented the analysis—or the ‘voice of our leaders’—as a total picture of organizational needs and threats to each member of the executive steering team, one-on-one. With the executives’ guidance on priorities, we developed learning proposals for the top four business challenges that emerged. We solicited sponsorship from the executive(s) who ‘owned’ them. And in one case, we were able to forge a collaborative partnership between two executives who shared a similar challenge, but who previously held very different views on the solution. “The result is that we have moved the learning dialog from ‘training needs’ to ‘business needs.’ That has enabled us to establish a governance structure with executives who now collaborate to identify, set, and fund priorities. “Top leaders increasingly approach us or refer others to us to help them meet their pressing business needs. Our reputation for proactively doing the right things and doing them well has earned us a true ‘seat at the table.’ “For example, William C. Young, our president and CEO, requested that I be present during the annual corporate strategy meeting because, as he said, ‘I want the Academy to play an integral role in communicating and driving these changes.’ Mr. Young openly expresses the opinion that Plastipak Academy is a strategic lever bringing competitive advantage.” That is the kind of recognition to which all learning organizations should strive, but one that can only be earned by being business outcomes driven.

CH001.indd 58

6/17/10 3:22:38 PM


59

The International Society for Performance Improvement (ISPI) has defined human performance technology (HPT) as a systematic approach to solve practical problems of performance by the individuals and groups in organizations. The Society’s governing principles reinforce the importance of getting sufficient input to clearly define desired outcomes and performance gaps before designing interventions. The principles include: • A focus on outcomes • A systematic approach to the assessment of the need or opportunity • A systematic analysis of the work and workplace to identify the cause of factors that limit performance The HPT process begins with a comparison of the present and the desired levels of individual and organizational performance to identify the performance gap. A cause analysis is then done to determine what impact the work environment (information, resources, and incentives) and the people are having on performance. Once the performance gap and the causes have been determined, the appropriate interventions are designed and developed. (International Society for Performance Improvement, n.d.)

One way to ensure adequate input is to create a steering committee that includes both business and learning leaders to provide oversight and insight. Bersin (2008a) identified “alignment with the business strategy” as one of the six practices of highly effective development. At BNY Mellon, the top ten executives meet quarterly to review how leadership development is supporting the company’s business initiatives (Bersin, 2008a). At Sony, senior business leaders serve on Sony’s Talent Management Council, which helps design and evaluate Sony’s Integrated Leadership Curriculum. An executive from the council also attends the virtual wrap-up session in which participants report the business impact of working on their leadership goals. Many companies use senior leaders as instructors in strategic programs. Involving senior leadership in forging the strategy and also in the rollout ensures that the program has legitimacy and remains focused on business outcomes.

Manage Expectations Expectations matter. When you buy a product, you do so expecting certain benefits. As the saying goes: “People do not buy quarter-inch drills; they buy the expectation of quarter-inch holes.” Whether or not buyers are satisfied with their

CH001.indd 59

6/17/10 3:22:38 PM

60


purchases depends on whether or not their expectations were met. The same applies to training and development. Management doesn’t (or shouldn’t) buy courses; they buy the expectation of improved performance. Therefore, learning organizations need to be mindful of, and manage, the expectations of their customers Management doesn’t to ensure that they are in line with what can be buy courses; they buy the realistically delivered. Indeed, the Kirkpatricks expectation of improved have proposed that “return on expectations” is performance. the ultimate measure of training effectiveness (Kirkpatrick & Kirkpatrick, 2009, p. 89). Those who decide how much to invest in learning and development do so based on their anticipation of the value it will deliver: The greater the investment of time and money required, the higher the expectation. The higher the expectation, the more compelling the evidence of value created will need to be for the purchasers to feel they have gotten their money’s worth. Think about your own behavior: Your expectations of a $49 entrée are different than they are for a $9 one. You would hold the former to a much higher standard than the latter, would you not? The challenge for learning and development—as for anyone selling a product or service—is to promise enough to get the order, but not so much that it cannot be delivered. And the only way to find the right balance is to agree in advance on the expected business outcomes and definition of success.

Better Than Nothing As odd as it sounds, training programs have to be better than nothing. That’s because companies can choose to do nothing—not to invest, either because the cost of resolving the problem exceeds the expected benefit or because they have lost confidence in training’s ability to deliver meaningful business results. Learning and development organizations must demonstrate that they deliver substantially more value than no training at all. If they don’t, then “no training at all” will continue to be the option of choice in times of fiscal restraint (see Case in Point D1.8). So the final reason to always define business outcomes is to make it clear to management that investing in training and development is better than doing nothing.

CH001.indd 60

6/17/10 3:22:39 PM


61

Case in Point D1.8 No Evidence, No Funding When Nigel Paine took over as the director of training for the BBC, he discovered that the company was investing £1.5 million a year in executive education of one sort or another (Paine, 2003). But when he searched all the available records, he could find no evidence of a return on the investment. People who attended such programs did not necessarily advance more quickly, stay with the firm longer, or generate more business value. There seemed to be no rationale for why some employees were sent to training programs and others were not, no explicit expectations of what would be gained, no follow-through. Therefore, he placed a moratorium on all spending for executive education. Needless to say, there was a great hue and cry among providers, both internal and external. But when challenged, no one could present credible evidence that their solution provided more value than the alternative—doing nothing. One and a half million pounds is real money, even for an organization the size of the BBC. An investment of that magnitude will and should be scrutinized. Management has a fiduciary responsibility to be sure that an investment in learning and development is the best possible use of the money and that the resources could not be more profitably deployed elsewhere. This is a key point with respect to the first discipline of defining outcomes: Learning and development programs compete for resources not only with alternative educational providers and formats, but also with other departments and other uses for the money. Canceling a £1.5 million expenditure, for example, drops £1.5 million straight to the bottom line. Learning and development leaders need to show, credibly and compellingly, why shareholders are better served by investing money in learning and development rather than reporting it as profit and paying it out as dividends. In short, they need to show why training (which is expensive) is better than doing nothing (which, at least from a cost point-of-view, is free).

Summary The first, crucial, and frequently overlooked step in successful learning and development is to define the desired outcomes in business terms. Successful learning organizations prioritize the business needs that training can address and concentrate their resources on those that will have the greatest impact and probability of success. In collaboration with the key stakeholders, they define the rationale for the program in terms of the business outcomes to be delivered, not just the learning objectives to be achieved. They agree with the sponsors on the measures

CH001.indd 61

6/17/10 3:22:39 PM

62


of success before they design the complete program. This first critical discipline— defining business outcomes—is the keystone for the success of the entire training initiative. If program designers fail to clearly articulate the desired business outcomes, no amount of effort in subsequent stages can fully compensate; the future of the program, and perhaps the company itself, is put at risk. Use the checklist for D1 to ensure that training is the best solution and that the objectives for the training and development are stated as business outcomes. CHECKLIST FOR D1 ✓ ❑

Criterion The Overall Program

❑

The proposed training and development address a performance issue related to lack of knowledge or skill.

❑

Non-training solutions have been explored or tried and rejected.

❑

Environmental factors that will affect successful implementation (such as accountability, consequences, coaching, etc.) have been identified and discussed.

Each Program Objective ❑

Is clearly linked to a high-priority, high-value business need or opportunity.

❑

States the actual performance that will be achieved (as opposed to knowledge, ability, or capability).

❑

Specifies to what extent this performance standard will be met and by when.

❑

Uses business terms, concepts, and language.

❑

Clearly indicates how success could be measured.

Action Points For Learning Leaders • Never offer a program simply because you were asked to offer a program. • Always ask, “Why? What is the expected benefit to the company?” • This is vital: If you do not clearly understand the business need and the linkage between the training and filling the need, you cannot design an effective intervention. • Stand your ground when you believe training is the wrong solution. • Be a performance consultant; use the power of your expertise to help managers see there are better alternatives for achieving the desired result. • Read and understand the business plan. Be proactive in identifying areas in which learning and development could contribute. • Test your ideas with discerning line leaders. Make them your allies.

CH001.indd 62

6/17/10 3:22:40 PM


63

• Once you have decided to address an appropriate need with training, use the Outcomes Planning Wheel to help negotiate a clear “contract” with management that specifies in advance the objectives, methods, and how success will be determined. • Create co-ownership of results by helping managers understand that training will fail unless accompanied by concomitant changes in systems, managerial engagement, incentives, and so forth. • Review all the programs for which you are responsible to be sure each has objectives that are credibly linked to business imperatives. • Be proactive in managing the training portfolio. • Replace marginal programs with more strategic ones. • Propose to management realignment or even reduction of resources if appropriate. For Line Leaders • Review the key learning and development initiatives in the business unit for which you are responsible. • Are they clearly aligned with the most pressing needs of the business? • Are there critical needs that are not being addressed? • Are resources being spent on low-value programs that could be profitably redirected to higher-value initiatives? • If you conclude that the current learning and development initiatives are not aligned with the most important needs of your business, you and the learning leader share the responsibility for ensuring that they are. • Write down the business need that you would really like to see learning and development address and the results you want. • Then schedule a discussion with the head of learning and development. • Ask whether it is possible to address your needs with training. (Keep in mind that many performance issues in business are the result of systems or process problems that cannot be solved by training or that can be solved more quickly and less expensively in other ways. If the head of your learning organization is any good, he or she will tell you whether it is realistic to try to achieve the results you want through training.) • Work through the Outcomes Planning Wheel (Figure D1.6) with your learning and development partner. • Agree on the behavioral changes that are needed to achieve your objective and how they can be confirmed. • Ask learning and development to propose a plan for achieving these results. • Review it critically, using the 6Ds Scorecard (Exhibit I.1). • If appropriate, rebalance your learning and development portfolio to redirect resources to the initiatives with the greatest potential payoff.

CH001.indd 63

6/17/10 3:22:40 PM

CH001.indd 64

6/17/10 3:22:41 PM

DESIGN THE COMPLETE EXPERIENCE If you can’t describe what you are doing as a process, you don’t know what you’re doing. —W. EDWARDS DEMING

I

n a rapidly changing and highly competitive world, learning speed and learning agility are important sources of competitive advantage. In a special report sponsored by the ASTD, eLearning Guild, and Chief Learning Officer magazine, Clark and Gottfredson (2008) concluded: “The ultimate source of adaptive capacity, competitiveness, and self-preservation . . . [is] the ongoing ability of an organization to learn and apply its knowledge.” While learning encompasses much more than training and development, the accelerated and focused learning that organizations achieve through formal learning programs is a vital component of staying competitive. Given the strategic importance of learning and the magnitude of the annual investment in training and development, it is vital to maximize the value it generates and, thus, the return on that investment. Maximizing the output of training and development requires managing learning as a process rather than as an event. That means designing the complete learning experience, not just the instruction. In this chapter we examine what it means to manage the entire learning process holistically and systematically — to actively plan and infl uence what happens before as well as after the traditional boundaries of corporate education—and the benefits that accrue by doing so.

65

CH002.indd 65

6/18/10 3:44:21 PM

66


Topics include: • • • • • •

Many factors influence outcomes A new paradigm The four phases of corporate learning Checking the process A checklist for D2 Action points for learning and line leaders

Many Factors Influence Outcomes Learning doesn’t happen in a vacuum. What each person takes away from a given learning experience is shaped by many things, including his or her expectations, attitude, prior knowledge and experiences, learning style, aptitude, and emotional state. Similarly, numerous factors influence the extent to which people subsequently transfer and apply their knowledge. These include opportunity, encouragement, reinforcement, and early successes or failures (Figure D2.1). Since the success of corporate learning and development programs (that is, the generation of meaningful business outcomes) requires both learning and learning transfer, the design of any such initiative must comprehend the entire process—not just what happens in the classroom, e-learning sessions, or simulation. Research has shown that what occurs before and after the formal period of instruction is as important, if not more important, than what happens during the course itself (Broad, 2005, pp. 82–93; Saks & Belcourt, 2006). FIGURE D2.1. MANY FACTORS INFLUENCE THE LEARNER’S EXPERIENCE, IMPACT LEARNING TRANSFER, AND AFFECT RESULTS Motivation Prior experience

Expected value

Learning Learning style

Personal relevance

Emotional state

CH002.indd 66

Support/ reinforcement Opportunity to use

Reminders

T R A N S F E R Early successes

RESULTS

Competing priorities

Recognition/ reward

6/18/10 3:44:23 PM

Design the Complete Experience

67

A New Paradigm We need, therefore, a new paradigm about the scope of the learning organization’s responsibility: one that goes beyond “delivery of programs” to “delivery of results,” beyond “instructional design” to “experiential design,” one that includes all of the factors that influence outcomes. The concept of paradigms (accepted “truths”) and their power to shape thinking was popularized by Thomas Kuhn in his classic The Structure of Scientific Revolutions (1962). While paradigms are essential for what Kuhn called “normal science” and efficient day-to-day problem solving, there comes a point at which prevailing paradigms no longer fit the facts, are counter-productive to progress, and need to be discarded. We believe we have reached that point in learning and development. The classroom and the computer screen are boxes, both literally and figuratively. As long as learning professionals and business managers confine their thinking to “inside the box,” they place artificial limits on both the potential and the ultimate return of learning programs. Writing in the Journal of Organizational Excellence, Teresa Roche, CLO of Agilent Technologies, explained: “At Agilent, every department is expected to innovate, learn continuously, and deliver bottom-line results. Global Learning and Leadership Development knew it could not fulfill these expectations simply by delivering traditional programs in traditional ways—no matter how high the end-of-course ratings. To reap the full benefits of corporate training investments, it needed to broaden its perspectives about when, where, and how learning occurs” (Roche, Wick, & Stewart, 2005, p. 46). Broad and Newstrom (1992) took an important first step in this direction when they introduced a transfer matrix that defined three periods—before, during, and after the training—and three learning partner roles—the manager, the trainer, and the trainee (Table D2.1). The matrix challenged learning organizations to think more broadly in terms of what influences learning transfer and when. It specifically called attention to the way in which learning transfer—and subsequently, results—are influenced by what happens before and after the program and by players other than workplace learning professionals. TABLE D2.1.

THE ROLE-TIME TRANSFER MATRIX Time

Role

Before

During

After

Manager Trainer Trainee

CH002.indd 67

6/18/10 3:44:24 PM

68


Further research has shown that additional roles—for example, the learner’s work group—also influence outcomes (see D4). The support or lack of support from managers, peers, direct reports, executives, and others influences participants’ attitudes toward learning and its subsequent use, as does the overall climate and culture of the organization (Broad, 2005, pp. 29–30; Holton, Bates, & Ruona, 2000). The key point here is that the participant’s learning experience encompasses much more than what is included in traditional instructional design. Those who ignore these realities of learning in a corporate environment do so at their own peril; they put the success of the program in jeopardy. The International Society for Performance Improvement (ISPI) has, since its inception, emphasized the need to think holistically and systemically about human resources interventions. Its principles for human performance technology state: “Taking a systems view is vital because organizations are very complex systems that affect the performance of the individuals that work within them. . . . A systems approach considers the larger environment that impacts processes and other work” (ISPI, 2002). Isolated initiatives rarely solve business issues because business issues are inherently systemic in nature (Senge, 1990, p. 7).

A Four-Phase Process The process of turning learning into business results has four phases (Table D2.2). Broad and Newstrom (1992) labeled the first three “before,” “during,” and “after” training in their transfer matrix (Table D2.1). The problem with using “before,” “during,” and “after” is that it reinforces the event paradigm; it implies that “real” learning only takes place during the formal instruction period (during “the course”) and that what happens before is just “warm-up” and afterward merely a footnote. In actuality, all four phases are necessary for a program to be truly successful. More programs fail as a result of inadequate learning transfer (the third phase) than inadequate knowledge acquisition as such (see D4). Indeed, the most challenging aspect of converting training into results is getting the learning out of the instructional setting and onto the job, or—as we like to say—“the real work begins when the course ends.” To underscore the criticality of all four phases, many organizations are moving away from the “before, during, and after” terminology. For example, Pfizer refers to the three phases of its Advanced Transition Program as “on-ramp,” “ residential, ” and “ on - the - job application ” (Blee, Bonito, & Tucker, 2005, pp. 261–264). Standard Chartered Bank’s Leadership Essentials and Leading for Performance and Sony’s Integrated Leadership Curriculum have begun using the terms Phase I, Phase II, and Phase III, as have other writers (e.g., Zenger,

CH002.indd 68

6/18/10 3:44:25 PM


TABLE D2.2.

69

THE FOUR PHASES OF LEARNING

Phase I—Prepare Phase II—Learn Phase III—Transfer Phase IV—Achieve Descriptor Preparatory Reading and Other Assignments

Structured Learning/ Instruction/ Course

Learning Transfer and Application on the Job

Old term

Before

During

After

What’s included:

• Needs analysis • Participant selection • Course design • Evaluation plan • Invitation • Marketing • Meeting with manager • Background reading • Online exercises • Assessments

• Classroom, virtual, or online instruction • Discussions • Simulations • Exercises • Role play • Action learning

• Goal setting • Action planning • Follow-through • Debrief with manager • Teaching others • Deliberate practice • Reporting progress • Reflection • Collaboration

Stop, Assess, and Recognize Achievement

• Self-evaluation • Third-party assessment (manager, peers, direct reports, etc.) • Change in performance metrics • Improved work product • Recognition

Folkman, & Sherwin, 2005) to emphasize that these are simply stages of a continuum—an ongoing, coordinated learning process. Likewise, many learning organizations specifically eschew the term “prework” for the learning that needs to take place in Phase I. They argue that the very name “pre-work” suggests that it is not “real work,” that it is optional and unimportant. On the contrary, in optimally designed programs, completing the Phase I learning exercises is essential to participate fully and to extract maximum value from the rest of the program ( Jefferson, Pollock, & Wick, 2009), hence the concern for finding a less pejorative term than “pre-work” to describe the first phase of the learning-to-results process. In the first edition of The Six Disciplines, we defined three phases of learning. Our experience over the past five years has persuaded us that there are really four phases of the complete experience, with the final phase being a point at which to pause, assess, and recognize achievement (Table D2.2).

CH002.indd 69

6/18/10 3:44:26 PM

70


There are two key concepts to Phase IV: (1) that it represents a defined end point for a given cycle and (2) that it assesses and celebrates achievement. While assessment is typically thought of as separate activity, apart and distinct from the learning process, anyone who has ever prepared for a performance-based competition or taken a final examination in school knows how powerful such assessments are as a motivation to learn and to practice. You also know from your personal experience how much your learning was shaped by how the competition would be judged or by what was going to be included on the final examination. Including an assessment of what has been achieved as Phase IV of the complete learning process increases the probability of success because it focuses attention and energy on the outcomes of interest. “First, the fact that outcomes will be ‘inspected’ (evaluated, measured, monitored, or assessed) heightens the attention that all stakeholders—performers, managers, performance consultants, and others— give to carrying out their roles in the performance improvement intervention” (Broad, 2005, p. 115). Defining a specific time and method at which achievement will be assessed is a departure from current practice in most corporate learning. In sharp contrast to advanced degree and professional certification programs, an assessment is rarely a feature of corporate education. Peter Gilson, former chairman of Swiss Army Brands, Inc., described typical practice: “As a young corporate executive, I attended dozens of development programs, but no one ever once followed up with me to see what I The most information ever had done with what I learned. The most informacollected was how I rated the tion ever collected was how I rated the instructor” instructor. (personal communication, 2006). In the absence of clear expectations and the knowledge that achievement will be recognized, it’s small wonder that a great deal of corporate training is never applied. Participants should never be left in doubt about how they and the program will be assessed. Ken Blanchard related the story of how, when he was a young assistant professor, he used to give the students a copy of the final examination on the first day of class. Other faculty objected, but Ken persisted, arguing that since the final examination represented what he felt was most important for students to take away from his course, then it only made sense to let them know that from the beginning and energize and focus their learning accordingly. The other reason to include this fourth phase as part of the learning process is that assessment itself is a learning exercise. Assessments reinforce learning by requiring participants to retrieve and process information some time after the original learning occurred. A well-designed assessment provides vital feedback to learners on what they have achieved as well as what they still need

CH002.indd 70

6/18/10 3:44:27 PM


71

to work on. Just an honest self-assessment of achievement can be a valuable learning exercise. After much debate, we decided to describe the fourth phase as the “endpoint” of a given cycle. We do not want to imply that learning should ever stop, or that the quest for performance improvement should ever end. But we were persuaded that the benefits of putting a “stake in the ground”—of declaring a specific goal line for each learning program that everyone can strive for— outweighed any downside. Everyone in business understands that annual goals do not mean the business closes down at the end of the fiscal year; the cycle starts again immediately, as it should with learning. But anyone who has been around toward the end of a fiscal cycle knows how having specific targets and a timeline motivates action. The drive to achieve a goal—even when there is no monetary reward involved (completing a marathon, for example)—is a powerful intrinsic motivator for most people (Pink, 2009). Failing to define an end point for a learning cycle denies people the satisfaction of reaching a milestone and the benefits of stopping to reflect on what they have achieved. For these reasons, we believe that defining an end point at which participants’ achievements will be assessed, recognized, and celebrated should be part of the design of the complete learning experience (see Figure D2.2). FIGURE D2.2. THE FOUR PHASES OF LEARNING NECESSARY TO IMPROVE PERFORMANCE

Effectiveness

New (and Higher) Level of Performance Achieve Transfer Learn Current Level of Performance Prepare

Before

CH002.indd 71

After

6/18/10 3:44:27 PM

72


In the subsequent discussion, we use the neutral terms Phase I, Phase II, Phase III, and Phase IV. Although these are not very catchy or memorable monikers, they have the advantage of being broadly applicable to a wide range of program types, delivery formats, and objectives. We will append descriptors like “prepare, learn, apply, and achieve” as necessary to provide clarity and links to earlier works. For each phase, we examine the evidence of its importance and provide suggestions for strengthening it as part of D2—designing the complete experience.

Phase I: Prepare What participants take away from a learning and development program, what they transfer to their work, and the results they are ultimately able to achieve are all strongly influenced by what happens during Phase I, long before formal instruction—the traditional bastion of training—begins. This is true even for programs that have no specific Phase I assignments. Participants’ prior education and experience form the foundation on which to build new capabilities (National Research Council, 2000); their expectations of the program—and their managers’ expectations of them—influence their receptivity to new ideas and approaches.

Expectations Influence Outcomes Participants do not arrive at corporate educational programs like blank sheets of paper. They come with opinions, biases, and expectations. They may have read about the program online or in a course catalog; they may have heard about it from colleagues. These early exposures begin to shape their opinions about the program’s probable value and therefore their willingness to devote time and energy to it. Their assumptions about its value are also strongly influenced by their managers’ level of interest or indifference and by their experience with training programs in the past. Such preconceptions affect the outcomes in surprisingly powerful ways. A classic experiment done at MIT illustrates just how powerful, and how easily swayed, this effect is (Kelley, 1950). Students were asked to rate a discussion led by a substitute instructor. Each student was given a brief biography of the instructor prior to class. Unbeknown to the students, there were two different versions of the sixty-three-word biography, identical except for two words: one described the instructor as “very warm,” the other as “rather cold.”

CH002.indd 72

6/18/10 3:44:28 PM


73

At the end of the class, the students were all given the same questionnaire and asked to evaluate the session. As Ori and Rom Brafman write in Sway (2008), “Upon seeing the results, you’d think the students were responding to two completely different instructors. Most students in the group that had received the bio describing the substitute as ‘warm’ loved him. . . . Although the second group sat in exactly the same class and participated in the exact same discussion, a majority of them didn’t really take to the instructor” (p. 73). If just two words can sway the experience of learning that much, it is clear that participants’ preconception of the expected value of a corporate learning program—prior to any actual experience of the course itself—will strongly influence their experience in the program and their post-program opinion of its value. Tharenou (2001) showed that expectations—especially regarding the practical utility of the program—strongly influence an employee’s decision to participate in voluntary training. It follows that participants’ going-in expectations also impact their willingness to fully engage in learning exercises like simulations, role plays, online learning, and discussions. You know this from your own experience. Your Expectations strongly willingness to go to a movie, try a new resinfluence an employee’s taurant, or attend a conference is strongly decision to participate. influenced by what you have read or heard about it from colleagues and friends. Taken together, the research suggests that learning organizations will benefit by devoting more effort and attention to creating positive “going-in” expectations among participants because, to a surprising degree, such opinions become selffulfilling prophesies. To a great extent, participants get out of a program pretty much what they presume they will. If they expect it to be a high-value, worthwhile learning experience, then that is usually what they experience. If they expect it to be a waste of time, it almost always is, at least for them. As the MIT study illustrates, two participants who participate in the same program but with different expectations will have different experiences and leave with different perceptions of the value they received (Figure D2.3). This phenomenon is so strong that learning and development planners ignore it at their peril. That is not to say that a great course cannot convert the skeptics, or that a truly awful program will not disillusion even “true believers.” But it is much easier to achieve the desired business outcomes when participants enter programs motivated to learn because they expect the content will help them with their jobs and because they know they will be held accountable for producing results. Thus, part of the Second Discipline involves managing the expectations created in Phase I—well ahead of what has traditionally been considered the start of a training program.

CH002.indd 73

6/18/10 3:44:29 PM

74


FIGURE D2.3. EXPECTATIONS INFLUENCE HOW DIFFERENT PARTICIPANTS BENEFIT FROM THE SAME PROGRAM

INPUT

OUTPUT

Learner 1:

Learner 1:

High expectations; strong manager support

“Great program, real impact. I use something I learned everyday.”

Learner 2: Low expectations; manager indifferent

SAME PROGRAM

Learner 2: “Waste of time, impractical, don’t expect to use.”

Phase I learning also includes background reading, online exercises, assessments, and other assignments that help accelerate and enrich the learning that takes place in Phase II, the formal or structured learning experience. Strengthening Phase I Because what happens in Phase I impacts the ultimate value created by learning and development, strengthening Phase I is an important aspect of an overall improvement effort. Three actions are key: raising expectations, engaging managers, and accelerating learning by getting together the right people with the right background knowledge at the right time in their careers.

CH002.indd 74

6/18/10 3:44:29 PM


75

Raise Expectations Whether participants perceive training as something essential for their success, as a reward, as a punishment, as a mini-vacation, or as a waste of time influences how seriously they approach it and therefore the likelihood of its success. Learning organizations can enhance expectations by applying marketing principles. Marketing departments spend a great deal of their time and money managing their product’s positioning and brand promise—what customers think of when they hear the product’s name and what they expect when they purchase it. Strong brands like Coca-Cola, Crest, Lexus, and iPhone have strong positions. Their key benefits come immediately to mind and shape perceptions of value, price, and alternatives. A brand’s reputation positively or negatively influences buying decisions and customer satisfaction. So does training’s brand. Effective marketing organizations know what their target customers value and use all the communication channels at their disposal to associate their brands with what customers desire most. The goal is to create a positive association between the brand and something that the customer desires, and then to embed that association deep in customers’ psyches. Developing a strong “brand” for training—one that is perceived as integral to the business and essential to one’s career—will enhance the ability of the learning organization to contribute to the business (see also D6: Sell the Sizzle). But what do employees desire from training? The principles of adult education, as well as studies of why employees attend training, indicate that adults are motivated to learn things that they believe will be of practical benefit in their lives and careers. Yet most course descriptions emphasize the features of the program—how long it is, what will be covered, the learning objectives, and instructional method—but rarely the benefits—how it will help the participant and the organization. Thus, defining business outcomes (D1) not only informs program design, but also helps communicate (raise expectations of) the value of attending. There is a big difference in perceived value between a program advertised as one in which “you will learn effective time management techniques” (a feature statement) and the same program billed as one that will “enable you to get more done in less time, with less stress, leaving more time for the things you enjoy” (a benefit statement). We once, only half in jest, suggested renaming a confined space safety course as “how to get home to your loved ones alive.” The point is that people buy expected benefits; they do not buy features. “Too many people state the features of their offering and expect the buyer to join People buy expected benefits; the dots and understand the value or benthey do not buy features. efit” (Dugdale & Lambert, 2007, p. 163).

CH002.indd 75

6/18/10 3:44:30 PM

76


To get buy-in and garner support for learning programs, you need to be explicit about the “why”—the benefits that will accrue to the participants. The power of making the business benefit explicit was demonstrated in a workshop we did in Asia for one of the Fortune 10 companies. Following the workshop, one of the participants re-wrote all of the course descriptions in her division to emphasize the business and personal benefits (outcomes) rather than just the features. As a result, interest in the programs from potential participants and support for attendance from their managers increased noticeably, even though the programs themselves had not changed at all—only the way in which they were positioned and described. Improve the Invitation Another good opportunity to positively influence the par-

ticipants’ expectations is through the invitation process. Many invitations to attend corporate training and development programs read more like a jury summons than an opportunity to participate in something of value. One client we worked with (who shall remain nameless) ran a top-quality, intensive development workshop for its highest-potential employees. Candidates for this highly selective program were invited only after a rigorous vetting process. Nevertheless, something went badly amiss in the invitation process. Not only did the participants have no idea what an honor it was to be invited, but some actually thought they were there for remediation! Review the way in which people are invited (or directed) to attend programs in your organization. Are the benefits to the participants clear? Does the invitation come from a person in authority they know and respect? Does it raise their expectations that they will gain something of value by attending and, at the same time, does it communicate the high expectations the company has of them? Or does it appear to be just another impersonal email from some faceless bureaucracy? Consider more enticing ways of communicating the invitation, especially for strategically vital programs. Some companies use a personal invitation from a senior leader. At McKesson, for example, the executive vice president of human resources personally calls each participant in the Leaders Teaching Leaders Program to confirm his or her commitment and participation (Boston, Allred, & Cappy, 2009). Others are even more creative: UBS Bank repurposed a marketing video, redubbing it to create a high-energy invitation to a key leadership program. The invitation to attend is one of your early opportunities to “sell” participants on the benefits; it is an important element of Phase I that helps set the tone for the learner’s complete experience. If you don’t want your program to be viewed as just another boring task, don’t promote it that way. Secure Management Support Visible support from management greatly

increases the probability of success. It’s especially important in Phase I because employees at all levels are pressed for time. Overwhelmed by competing priorities,

CH002.indd 76

6/18/10 3:44:31 PM


77

they look to their managers for indications of what is most important and then apportion their efforts accordingly. If they perceive that a program has strong managerial support, it will receive more effort and attention. If their managers appear indifferent or skeptical about the value of the training, then participants will give it the minimum possible amount of time and effort. Participants’ direct supervisors, in particular, have a profound effect on whether or not training produces results. That effect exerts itself before instruction ever begins (discussed in Exhibit D2.1 below). This should come as no real surprise, given the overall impact that a person’s direct supervisor has on his or her work priorities, experience, and job satisfaction. Kouzes and Posner (2007) said it well: If you’re a manager in an organization, to your direct reports you are the most important leader in your organization. You are more likely than any other leader to influence their desire to stay or leave, the trajectory of their careers, their ethical behavior, their ability to perform at their best, their drive to wow customers, their satisfaction with their jobs, and their motivation to share the company’s vision and values (p. 338, emphasis in the original).

Exhibit D2.1 Managers and Phase I of Learning When polled about which time-role combination had the greatest impact on learning transfer, trainers themselves agreed that the manager’s involvement in Phase I—prior to the formal coursework—was the single most powerful influence on whether the training ultimately would be transferred and applied (Broad & Newstrom, 1992, p. 54). This perception has been confirmed by many others. For example, Brinkerhoff and Montesino (1995) found that participants who had discussions with their managers before and after training reported significantly higher levels of skill application and felt more accountable for applying new learning. Feldstein and Boothman (1997) compared high- and low-performance learners and identified eight factors that characterized high-performance learners. Half of these factors were related to the influence of the manager. For example, 75 percent of the high-performance learners reported that their supervisors had expectations of improved performance following training, whereas only 25 percent of low performers did. In a follow-up study, a system was implemented to increase the pre- and post-course interaction with managers. The result was that both the learners and their managers reported much higher rates of transfer. At American Express, “high improvement” learners were four times more likely to have had a one-on-one conversation with their managers than the “no improvement” group (American Express, 2007). The bottom line is that managers directly impact the outcome of training—and that that influence begins in Phase I.

CH002.indd 77

6/18/10 3:44:31 PM

78


They did not say it, but they could have easily added, “and their expectations with respect to training and development.” If a person’s manager shows no interest in his or her development, or worse, clearly begrudges the time that he or she will spend in a program, that telegraphs a strong message that jades the participant’s opinion and reduces his or her willingness to participate before the program even begins. Therefore, ensuring that participants know, feel, and believe in management’s support starting in Phase I is vital to maximizing the return on educational investments. Credible support is easier to recruit and sustain if the First Discipline—defining the expected business outcomes—is followed. Line managers are more willing to have their people attend training programs, and are more supportive of learning transfer, when they perceive a strong alignment between the needs of the business and the training curriculum. Bersin (2008b) found that alignment and management support were the two key factors that determined organizational impact (p. 82). As such, he encouraged learning organizations to assess the degree of alignment of both individual programs and the learning function as a whole with the needs of the business. Louis Carter, founder and president of the Best Practice Institute, conducted a study of organizations that achieved sustainable results from leadership development. Support and participation of senior management was the highest-ranking success factor (Carter, Ulrich, & Goldsmith, 2005, p. 421). Critical senior leadership behaviors for success included: • • • •

Allocating funds for the initiative Modeling behavior consistent with strategy Integrating the initiative into the strategic plan Facilitating education or training (p. 444)

Support needs to go beyond mere words. If a learning and development program is to be effective in bringing about organizational change, then the most senior and most visible leadership must model its precepts. Senior managers need to complete the program themselves, embrace its principles, use the recommended approaches to management, and incorporate concepts from the program in their day-to-day activities. If they do not, they seriously undermine the program’s impact and credibility (see Case in Point D2.1). Participants are very adept at sensing whether or not programs have the wholehearted support from the leaders in the organization. The karmic seeds of success or failure are planted before the program is even launched by the extent to which it enjoys management support that goes beyond mere rhetoric; managers of attendees must be prepared to support the program by their actions. And one of the most effective actions they can take is to have a short, focused, structured

CH002.indd 78

6/18/10 3:44:32 PM


79

Case in Point D2.1 When the Video Doesn’t Match the Audio A biotechnology firm we worked with had sustained dramatic growth for a number of years. Managers had been rapidly promoted as the business grew, but with very little formal management training and, because of the pace of growth, limited on-the-job experience and mentoring. As a result, most middle managers led by the seat of their pants, mimicking the entrepreneurial style of the founder. Then the company encountered market turbulence and missed its forecast significantly. Its stock dropped precipitously and the senior leadership realized that the lack of professional management was a serious impediment to continued prosperity. So they had human resources design and implement a five-day program to help managers increase the effectiveness of teamwork, foster innovation, improve efficiency, and create ownership for results through delegation. The senior management team strongly endorsed the program and made stirring speeches about its importance for the future of the company. When it came time to attend, however, they were “too busy” to attend the full program. They asked organizational development to put together a special half-day “executive edition” that included neither preparatory work nor follow-through. The result was predictable. The senior managers never mastered the material. They therefore failed to incorporate the processes or terminology into their own leadership. So, for example, while middle managers were exhorted to expand creative thinking by conducting brainstorming sessions in a particular way, their own managers failed to do so. In fact, many actions by the company’s senior leaders were directly contrary to the program’s concepts and principles. Needless to say, the program failed to create the hoped-for change or generate a return on investment. Indeed, the “do as I say, not as I do” attitude of the senior leaders not only undermined the program’s effectiveness, but it generated cynicism among middle managers about both the senior leadership and the value of training and development.

discussion with their direct reports in Phase I, in advance of the formal instruction (Jefferson, Pollock, & Wick, 2009). If senior managers are unwilling to make the commitment to provide this level of support, the whole premise of the program needs to be reexamined. The learning leadership must question whether they have truly defined their company’s most pressing needs and developed a program that addresses them. Launching a program in the absence of strong management supLeave no doubt in port undermines its probability of success participants’ minds about and potentially damages the reputation of what is expected. the learning organization as a whole.

CH002.indd 79

6/18/10 3:44:32 PM

80


Redefine the Finish Line An important objective of Phase I should be to leave no doubt in participants’ minds about what is expected of them. They should come to the course with the clear understanding that the privilege of attending an educational program carries with it the responsibility to apply it to improve their performance (Figure D2.4). Follow-through on training and development should be part of what Bossidy and Charan (2002) call a “culture of execution,” because “Without execution, the breakthrough thinking breaks down, learning adds no value, people don’t meet their stretch goals, and the revolution stops dead in its tracks” (p. 19, emphasis added). A quantum jump in performance would result if everyone who attended a training program knew that he or she would be expected to execute against the learning transfer goals and achieve results, just as he or she would against other business objectives. When employees are afforded an opportunity to learn, expectations regarding their subsequent performance go up (Jefferson, Pollock, & Wick, 2009). If their performance remains unchanged while expectations rise, their performance rating relative to expectations will actually decline (Figure D2.5). And yet learning organizations continue to inadvertently reinforce the mistaken notion that attendance is sufficient and that when the instruction ends, the participant’s work is done. In fact, the real work of turning learning into results is just beginning.

FIGURE D2.4. THE OPPORTUNITY TO ATTEND TRAINING RAISES EXPECTATIONS FOR IMPROVED PERFORMANCE

© Charles Barsotti/Condé Nast Publications/www.cartoonbank.com. Used with permission.

CH002.indd 80

6/18/10 3:44:33 PM


81

FIGURE D2.5. EXPECTATIONS GO UP AFTER A TRAINING PROGRAM. IF PERFORMANCE REMAINS THE SAME, THEN PERFORMANCE RELATIVE TO EXPECTATIONS DECLINES

Performance Relative to Expectations

Negative variance

Before Training

Performance

Expectations

Performance

Expectations

Positive variance

After Training

After Jefferson, Pollock, and Wick, 2009. Used with permission.

The problem starts with the agenda, which, for the vast majority of learning programs—whether classroom, virtual, or online—includes only Phase II. A more accurate representation of the process would be a timeline that includes all four phases of the complete experience (Figure D2.6). Indeed, we have taken the position that: “We should never again speak of a three-day, one-week, or even three-week agenda. Every program should be at least three months long” because for most kinds of corporate training, it takes at least that long to establish new habits and produce results (Wick, Pollock, & Jefferson, 2008). The paradigm that “the course is the thing” is further reinforced by the last agenda item for most programs. It typically says something like “program ends,” giving the misleading impression that participants have completed their

CH002.indd 81

6/18/10 3:44:34 PM

82


FIGURE D2.6. A PROGRAM TIMELINE THAT RESETS EXPECTATIONS BY SHOWING THE COMPLETE LEARNING EXPERIENCE

Begin 1

2

3

Weeks 4

5

6

7

8

End 9

10

11

12

13

Meet with manager Complete preparatory work Participate in course Follow up with feedback providers, coach, manager

Transfer and apply learning to work Assessment of results Plan continued improvement

responsibilities simply by “showing up.” That notion is driven home by the habit of handing out certificates and mementoes and awarding credit at the end of the class or the e-learning program. Recognizing how counterproductive this practice is—since at the end of instruction the organization has sustained the cost but not received any benefit— more enlightened organizations have begun moving the finish line from the last day of class to the documentation of results (see Case in Point D2.2). The real work of turning learning into results begins when the formal instruction ends. In subsequent chapters we discuss specific ways to facilitate learning transfer and application (Phase III). These approaches are more effective when the expectation for putting learning to work has been clearly established from the onset, in Phase I. It is essential, therefore, to reset participants’ expectations so that “successful course completion” is understood to mean achieving sustained performance improvement, not just reaching the end of instruction. Get the Right People into the Room A key difference between teaching adults and teaching children is that adult learners bring much richer and more varied experiences to their education. These experiences are a powerful framework on which to build and a source of practical examples and real-world knowledge.

CH002.indd 82

6/18/10 3:44:34 PM


83

Case in Point D2.2 Resetting the Finish Line Home Depot conducted a series of large-scale learning events to help its store managers run more efficient and effective operations. At the end of the three-day forum, participants were presented with a very handsome crystal trophy. The president of Home Depot Canada, Annette Verschuren, realized that doing so sent entirely the wrong message. The award should not be for attendance, but for implementing ideas or practices that actually improved store operations. So, in her unit at least, store managers received their trophies only when they could document at least one action they had taken as a result of attending the Store Managers’ Forum that demonstrably improved their store’s performance—a nice example of redefining what it means to complete a course from the end of instruction to actual implementation and results. Another fine example comes from Bill Amaxopoulos, leadership program manager for the Chubb Group of Insurance Companies. Bill has moved to a phased approach in which attendees and their managers must attend pre-course webcasts together and commit to follow-through activities. Then, on a three-month post-course webcast, each participant summarizes his or her goals from the program and subsequent accomplishments. They receive credit for completing the course only after they participate in the follow-up assessment call.

Thus, helping participants build on their experiences and share “lessons learned” are important precepts of adult education. The very richness and variety of experience represented in any given class, however, presents a significant challenge for corporate learning professionals. In high school or even college, educators can assume a much more homogenous set of experiences and educational backgrounds. In corporate education, however, groups frequently include learners who differ widely in age, years’ experience, educational background, work histories, and, in an increasingly global marketplace, even mother tongue. Although this diversity offers fertile ground for discussion and insight, it also increases the difficulty of delivering an effective learning experience for each participant. Designing the complete learning experience, then, includes defining the background that participants need to have to benefit from the program, as well as the learning experiences they need in Phase I to establish a common foundation. In academic universities, upper-level courses have clearly established prerequisites for enrollment, such as, “Students must have successfully completed Calculus 101.” This is much less frequent in corporate universities and executive education programs—perhaps for fear of limiting enrollment. The downside of

CH002.indd 83

6/18/10 3:44:35 PM

84


having no prerequisites, however, is that the program must necessarily be taught at the “one-size-fits-all” level, with the result that it may fit no one perfectly—too elementary for some and too advanced for others. Recognizing this dilemma, a number of companies are moving toward courses offered in a coordinated sequence. This approach allows their learning organizations to offer introductory courses to those new to a role or function, as well as advanced programs that explore topics in greater depth. One company uses a pyramid icon in its course catalog to illustrate where each course fits in the sequence from basic to expert. Pfizer’s Learning Center structured its leadership development programs into series of courses called Leading Edge for first-time managers, followed by the Advanced Transition Program when they are promoted to a leader of leaders (Kontra, Trainor, & Wick, 2007). Unilever developed a five-part leadership series, and Sony’s four-level Integrated Leadership Curriculum was modeled after The Leadership Pipeline (Charan, Drotter, & Noel, 2001). Such approaches are consistent with Deloitte’s strategic conclusions about corporate learning (see Case in Point D2.3). Dividing learning and development into sequential programs with prerequisites allows corporate educators to better meet the needs of their learners and to take advanced programs to higher levels of excellence. Prerequisites to advanced programs might include not only other courses but also specific work experiences, such as having led a project team or managed those who manage others. The shift toward holding learning organizations accountable for outcomes, rather than just activity, should encourage us to define and set prerequisites more often, since allowing the wrong people to attend a program dilutes its effectiveness and reduces the value it generates. Get Everyone on the Same Page Learning is also strengthened when Phase

I includes specific assignments or experiences that help bring everyone to a common level of understanding and background. In our opinion, every program should have some required preparation, be it reading, e-learning, simulations, assessments, teleconferences, or other learning experiences. Chris Jenkins, learning and development manager for U.S. Bank Wealth Management Group, concurs. He feels so strongly about the importance of preparation and follow-through that he thinks programs that don’t include them shouldn’t be given at all. Preparation allows instructors to go further and faster. It makes discussions and group exercises richer and more meaningful. But the amount of preparatory work assigned must be weighed against the time available. If there are too many assignments to complete, or more reading than can reasonably be done by managers who already have more than full-time jobs, the value of the preparation will be diminished.

CH002.indd 84

6/18/10 3:44:35 PM


85

Case in Point D2.3 The Deloitte U.S. Firms’ Six Strategic Conclusions Over 40,000 professionals of Deloitte LLP and its subsidiaries (Deloitte U.S. Firms) provide audit, consulting, financial advisory, risk management, and tax services to selected clients. Needless to say, continuous learning and development are essential for the organization’s professionals to remain current and provide the high-quality service and advice for which Deloitte is famous. To assist the Deloitte U.S. Firms’ talent development organization in helping its people stay always one step ahead, the organization applied its rigorous strategic planning process to its own learning agenda. The process generated Six Strategic Conclusions that are being used to guide the Deloitte U.S. Firms’ learning and development efforts: 1. Outside-In (External focus) Learning and development are designed to anticipate and deliver the key capabilities valued now and in the future by external constituencies including clients, regulatory bodies, and the talent market. Business objectives inform talent development; business leaders are actively engaged in setting the agenda. 2. Customized Learning solutions are customized to individual learners’ needs and career aspirations. Programs are tailored for different levels of proficiency as well as industry, sector, professional level, and job role. 3. Leading-Edge Learning design and delivery incorporate the latest research, technology, and innovation. The focus is “action learning” rather than “passive listening.” Learning continues beyond the classroom and is sustained with performance support tools. 4. Integrated Learning programs provide connection points and networking opportunities. Assignments and talent deployment take into account individual development needs and career interests. Business leaders, managers, and counselors drive follow-through on learning programs. 5. Outcome-Driven Desired outcomes and measurement plans are documented before program design begins and include knowledge and skills acquisition, learning transfer and application, and business impact. Client satisfaction with capabilities and industry evaluations of effectiveness are the ultimate measures of learning’s success. 6. Synchronized All the “moving parts” support, rather than conflict with, one another. Communication and collaboration among businesses, service lines, industry groups, and talent development ensure consistency and avoid duplication of effort.

CH002.indd 85

6/18/10 3:44:36 PM

86


The time pressures that discourage participants from doing Phase I preparation must be counterbalanced by incentives for completing it and by persuasion regarding its value. For example, if participants are expected to complete reading or other assignments in Phase I, it is incumbent upon the instructors to make explicit use of them in Phase II—ideally in such a way that those who did not do the assignments wish they had. If Phase II is structured in such a way that completing the preparatory work makes the program much more valuable (or if the lack of preparation becomes embarrassingly obvious), the whole organization soon learns this. Subsequent groups will be better prepared, having heard the word to be sure to do the homework. This is not unlike the reputation that specific courses and professors acquire in college. (“She is a great professor, but you’ll be sorry if you go to class without doing the reading!”) Preparing learners to maximize course time in Phase II should not be confined simply to reading assignments. The range of learning opportunities is far richer. Think beyond traditional reading assignments to more experiential learning. For example, have participants interview customers and record their perceptions, spend time in manufacturing or in customer service, interview leaders of other divisions about successes or failures, visit a competitor’s store, or work with the competitor’s products. Take advantage of technology and have participants work through an online simulation, assessment, or other e-learning module. Look for experiences that stimulate their curiosity and imagination or shake them out of their current comfort zones. One organization we worked with challenged its marketing managers to spend two hours on the phone as a customer service representative trying to explain incentive programs. It was a real eye-opener: they came away with a whole new perspective and a commitment to design programs that were easier to convey. There are so many potentially interesting and innovative Phase I learning opportunities that the challenge is not to get carried away. A leadership transition program at one company had eight different Phase I assignments that included reading a whole book, completing several online assignments, and conducting a halfdozen interviews. Needless to say, when surveyed, very few participants indicated that they had done all of the assignments. Phase I learning plans need to respect the time pressures of today’s business environment; every minute spent needs to add value. Do not, for example, ask people to read an entire business book. Few have the time, and the perceived magnitude of the task alone will deter some from starting. Identify the key sections that will be discussed and assign these only. Phase I work, no matter how relevant and well-focused, produces no learning unless it is done. Therefore, designing the complete experience includes designing incentives to help ensure that preparation is completed and its importance is appreciated. Here again managerial support is vital. The managers of participants need to be aware of what preparation is expected and buy into its importance.

CH002.indd 86

6/18/10 3:44:36 PM


87

Ideally, they ought to meet with their direct reports before they attend to discuss the preparatory exercises, goals for attending the program, and expectations for outcomes. As a step in this direction, many companies now require the manager to approve attendance at educational programs. Some, like Ingersoll Rand, go further and encourage the participant and manager to sign a learning “covenant” in which both agree, in advance, on the key learning objectives and how progress will be assessed. This is analogous to performance management (management by objectives) for business goals. In the learning covenant, the manager also makes a commitment: to provide ongoing advice and support as needed to achieve the agreed-on goals. Where such contracts are taken seriously, the results of the program are enhanced. We know of at least one company that feels so strongly about the importance of a Phase I discussion between managers and participants that if participants arrive at a key program without a learning contract co-signed by their manager, they are put on a plane and sent home. Needless to say, they only had to do that once. Diane Hinton and Mary Singos of Plastipak Academy have customized a complete process and tools for supervisor involvement that begins in Phase I. Called “High-Impact Learning and You” it includes an overview, exercises, and job aids for both participants and supervisors to help them understand their roles and derive greater value from training investments. Supervisors are expected to take four key actions: 1. Create a line of sight on the impact map linking skills to business goals for their team and the specific associate who will attend the training. 2. Prepare the associate by hand-delivering the training packet and meeting to discuss the marked-up impact map and the learner-prepared “My Roadmap to Results” to agree on learning targets for driving key results. 3. Hold a fifteen-minute, post-training meeting to plan actions for moving from learning to results. 4. Support new skill application on the job through ongoing coaching and feedback. There is also a role for technology in Phase I. Many 360 - degree feedback and other assessment instruments are available online. Pfizer, for example, has participants in its Advanced Transition program complete a unique online leadership assessment survey to establish a baseline for transition performance (Blee, Bonito, & Tucker, 2005, p. 261). Standard Chartered Bank uses an online learning transfer management system (Friday5s) to help ensure completion of Phase I learning in its leadership programs. The system provides automated reminders about the preparatory assignments far enough in advance

CH002.indd 87

6/18/10 3:44:36 PM

88


to facilitate their completion. Reminders have proven especially useful for assignments that cannot be done at the last minute, such as having participants define learning objectives jointly with their managers, interview a senior leader, or listen to customer service calls. In programs without such reminders, some participants inevitably postpone their preparation until the last minute—about as productive as trying to read War and Peace the night before the final in a Russian literature class. In courses that use online Phase I support systems, participants are asked to record the insights from their preparation in a shared learning space. Instructors are able to access the database to review the results and select relevant examples for use in Phase II of the program. The shared visibility (the author of each entry is identified) provides clear accountability; everyone involved with the program can see who did, and who did not, complete the required preparation. The social networking aspect of this shared preparation helps build a sense of collegiality prior to the program. It also provides instructors with group-specific examples they can use to add relevance and credibility to the content.

Phase II: Learn We discuss Phase II (“the course” or structured learning period) in more detail in the next chapter (D3: Deliver for Application). In terms of planning the complete experience, the critical issues are to: • Ensure congruence between the learning experiences in Phase II and the ultimate business outcomes sought. • Build on and reinforce the preparatory work of Phase I. • Use instructional approaches that are appropriate to the required behaviors and skills. • Honor principles of adult learning. • Prepare managers and others essential to support learning transfer so that they fulfill their roles. • Ensure that participants are prepared and able to apply what they have learned. • Make the transition between Phase II and Phase III strong and seamless. When these conditions are met, it is possible to construct a value chain that links each learning topic and exercise through the intermediary steps to the ultimate business objective of the training (Table D2.3). Brinkerhoff and Gill (1994) originated the concept of developing such impact maps; they have proven to be an invaluable tool for delivering high impact learning (Brinkerhoff & Apking, 2001).

CH002.indd 88

6/18/10 3:44:37 PM


89

TABLE D2.3. EXAMPLE OF A VALUE CHAIN SHOWING THE LINKS BETWEEN A PHASE II EXERCISE AND THE DESIRED BUSINESS OUTCOMES Learning Experience • Multi-round, multi-player simulation of market dynamics in response to changes in pricing • Debrief of results and why they occurred • Repeat of simulation to see if lessons were learned

Ultimate Business Purpose

Capabilities Created or Enhanced

On-the-Job Behaviors

Leading Indicators

• Ability to project price-volume ratios and impact on profitability • Ability to anticipate competitors’ reactions and think through subsequent impact • Understanding of how pricing affects market value

• Fully analyze impact of price reductions on profitability in both short and long term • Anticipate competitors’ likely reactions • Think two or three moves ahead instead of one

• Improved • Improved profitability pricing decisions • More strategic use of pricing • Better balancing of price and volume • Higher gross margins

Strengthening Phase II The key opportunities to strengthen Phase II learning are to enhance the linkage between the instruction and the business goals and to ensure that the methods of instruction are congruent with the performance ultimately desired. One way to construct an explicit chain of inference is to ask a series of if–then statements beginning with the end in mind and working back to the kind of learning exercises needed. That is: • If this is the ultimate business outcome we are trying to achieve, then what do employees need to do to achieve it? • If they are supposed to do these things, then what knowledge, skills, and abilities must they have? • If they are to become proficient in a required skill, then what sort of learning experiences and practice do they need?

CH002.indd 89

6/18/10 3:44:37 PM

90


• Finally, if these are the kinds of learning experiences required, then which are best accomplished in Phase I, which in Phase II, and which in Phase III? This approach to selection of course content and instructional method has a number of advantages. It ensures that the ultimate business objective is the criterion for weighing what to include and what to leave out. It helps ensure alignment between exercises and the desired behaviors and ways of thinking. In other words, if the goal is to improve strategic planning and if strategic analysis is identified as a requisite skill, then merely listening passively to lectures on planning is incongruous with the objective. To become proficient at analysis, participants need to have the opportunity to practice their analytical skills with expert guidance and feedback. Adults are motivated to learn when the relevance is clear. Moreover, when learning and development programs are designed this way—with business outcomes always as the end in view—it is much easier for participants to see their relevance. The latter is important. As we shall expound on in the discussion of D3, adults are more motivated to learn and do so more effectively when the relevance of what they are learning is clear (Knowles, Holton, & Swanson, 2005). To improve training’s effectiveness, make the rationale for each learning exercise readily apparent. It should be possible to explain the decision for including each topic or exercise to learners, managers, sponsors, and other stakeholders in a short series of “so that” statements: “Your success and the company’s depend on making sound strategic pricing decisions. We included a business simulation so that you would have the opportunity to practice analyzing strategic alternatives, see the results of your decisions, and get expert feedback so that you will be better able to analyze strategic alternatives on-the-job.” Phase II is the best understood and most well-studied phase of corporate learning. There are numerous books and research studies on instructional design. Getting the instruction right is essential, but it is only one element of the complete learning experience. Its effectiveness is strongly modulated by what occurs beforehand and afterward and how well it is aligned with the needs of the business and learners.

Phase III: Transfer What happens (or fails to happen) in Phase III is the main arbiter of whether business results are achieved. Thus, enhancing the third phase of learning—learning transfer and application—has tremendous potential to enhance the overall

CH002.indd 90

6/18/10 3:44:38 PM


91

effectiveness of corporate education. No matter how great the learning experience was in Phase II, it generates no value unless it is used (Figure D2.7). Therefore, planning and facilitating Phase III learning transfer is in everyone’s best interest: the individual, the learning and development organization, the participant’s manager, and the company as a whole. It deserves much greater attention than it has received in the past. The Real Work Begins When the Course Ends It may seem odd to include “learning transfer” as a phase of the learning process. However, as we will show below, it is, in fact, the most important part of the learning process and one that makes a crucial difference to the success of any learning intervention. Geoff Rip, president of ChangeLever International, makes the point that so-called learning transfer is really a form of action learning in which people continue to advance their understanding, skill, and knowledge by applying the nascent learning from the instructional phase to real-life problems and objectives. According to Rip (personal communication, 2010), “This phase of learning is essential to the achievement of competence or proficiency since it takes significant practice and feedback to develop complex skills and for new behaviors to become habituated. We need to stop thinking of ‘learning transfer’ as just transferring what was learned, but as a critical step in creating learning and achieving learning objectives, which has profound implications from a design perspective.” Because transfer is a critical component of the overall learning pro-

FIGURE D2.7. PHASE III LEARNING TRANSFER IS THE WEAKEST LINK IN MOST LEARNING PROGRAMS

Copyright © Grantland Enterprises; www.grantland.net. Used by permission.

CH002.indd 91

6/18/10 3:44:39 PM

92


cess, we feel strongly that learning professionals need to take greater ownership of Phase III and exert more influence on the post-course environment. The literature on transfer of learning is clear: The post-course environment has a profound effect on whether learning gets translated into results. Newstrom (1986) was one of the first to study the barriers to effective learning transfer. Of the top nine he found, only three were related to the content and delivery of the course, while more than half were related to the post-course environment and support (Table D2.4). The single greatest barrier to learning transfer was lack of reinforcement on the job. Similarly, of the eleven reasons that training fails cited by Phillips and Phillips (2002), seven were related to the post-course environment: • • • • • • •

Training regarded as an isolated event Participants not held accountable for results Failure to prepare the job environment to support transfer Management reinforcement and support missing Failure to isolate the effects of training Lack of commitment and involvement of executives Failure to provide feedback and use information about results

More recently, studies by Cromwell and Kolb (2004) and at American Express (2007) and Pfizer (Kontra, Trainor, & Wick, 2007) have reaffirmed the impact of the post-course environment on the effectiveness of learning transfer and the ultimate success of the training and development effort.

TABLE D2.4.

IMPEDIMENTS TO TRAINING TRANSFER

Rank

Barrier

1

Lack of reinforcement on the job

2

Interference from immediate (work) environment

3

Nonsupportive organizational culture

4

Trainee’s perception of impractical training program

5

Trainee’s perception of irrelevant training content

6

Trainee’s discomfort with change and associated effort

7

Separation from inspiration or support of trainer

8

Trainee’s perception of poorly designed or delivered training

9

Pressure from peers to resist changes

Source: Republished with permission, Emerald Group Publishing Limited. www.emeraldinsight. com/jmd.htm

CH002.indd 92

6/18/10 3:44:40 PM


93

The importance of the post-course period should come as no surprise. The value of revisiting a topic at intervals (the spacing effect) “is one of the oldest and best documented phenomena in the history of learning and memory research” (Bahrick & Hall, 2005). Likewise, research on superior performance across a wide range of human endeavors has long identified the pivotal role of practice in achieving expertise (Ericsson, Charness, Feltovich, & Hoffman, 2006; Ericsson, Krampe, & Tesch-Romer, 1993; Ericsson, Prietula, & Cokely, 2007). These findings have been recently popularized by Colvin in Talent Is Overrated (2008), Gladwell in Outliers (2008), and Coyle in The Talent Code (2009), all of which demonstrate convincingly that deliberate practice contributes more to mastery than does “natural ability.” Given the well-known importance of the immediate post-training period to the overall success of learning and development, why haven’t corporate learning organizations done more to manage this critical phase? We believe there have been three main impediments: 1. The prevailing paradigms that treat training as an event and focus most of the instructional design on what amounts to event planning 2. The post-course period as “no-man’s land” between the learning organization and day-to-day management, with neither accepting clear ownership for learning transfer and results 3. The difficulty of (and historical lack of systems for) managing the process for large numbers of dispersed learners. These issues will be addressed in greater detail in the chapters on D4 (Drive Value is generated only Learning Transfer) and D5 (Deploy when the learning is applied. Performance Support). We touch upon them here as the rationale for why Phase III needs to be included in planning the complete experience and why learning organizations need to dedicate more resources to influencing this vital, but neglected, aspect of the process of turning learning into results. Strengthening Phase III Corporate education is not an event, like a play or a concert or a three-day cricket match. It is a business process that is expected to yield business results over time. Like every process, it has inputs and outputs. Significant inputs—investments of time and money—are consumed in the Phase I preparation and the Phase II

CH002.indd 93

6/18/10 3:44:40 PM

94


coursework. Value is generated only when the learning is applied in a way that produces improved performance. For a program to be considered a success, the output—in terms of value created by improved productivity—needs to be sufficiently greater than the status quo (that is, had no training occurred) in order to repay the investment in facilities, trainers, planning, travel, opportunity cost, and so forth (Figure D2.8). Treat Training as a Process, Not an Event Transfer and application (Phase III) should

not be thought of as something that happens after a program; it is part of the program, an integral and irreplaceable action-learning element of the process. Since it is the only part of the process that generates value (rather than consumes it), transfer is essential to repay the investment; it must not be left to chance or individual initiative. The entire learning process, including follow-through, must be managed from beginning to end like any other business process critical to an organization’s success. The program is not finished until the learning has been transformed into results.

FIGURE D2.8. THE COST OF CREATING AND DELIVERING A TRAINING AND DEVELOPMENT PROGRAM MUST BE OFFSET BY IMPROVED PRODUCTIVITY AND BUSINESS RESULTS

Output (returns)

Input (investments) Planning costs, materials, management time, salaries

Preparation

ROI

Facilities and travel, instructors’ salaries, participants’ salaries, opportunity costs

Coursework

Improved productivity

Application Learning transfer

CH002.indd 94

6/18/10 3:44:41 PM


95

Accept Responsibility for Phase III The second reason that Phase III, despite its importance, has not been part of program design is that no one “owns” it, in the sense of being held accountable for the outcome. Line managers, having themselves been acculturated to think of learning as an event, assume they have fulfilled their obligation by authorizing their direct reports to attend training. They mistakenly believe that it is the training department’s responsibility to be sure that learning generates results. Most managers greatly underestimate their impact on the outcome. Learning organizations, on the other hand, realize that they have no direct authority over participants once they return to their day-to-day jobs. They therefore have (erroneously) concluded that they are not responsible for what happens after Phase II either. The fact is, learning transfer is an essential phase of the complete learning experience, and the learning organization cannot claim that the training was a success until it has been successfully transferred to the work of the enterprise. Conversely, line managers cannot hold the training department accountable for results unless they do their part to support transfer, application, and assessment. The whole process succeeds or the whole process fails. In other words, managers and learning professionals co-own and are co-responsible for turning learning into business results, whether they realize it or not. While it is true that the learning organization has no direct authority over participants’ work activities or managers’ actions, that does not mean that they are not without influence. Bill Amaxopoulos, leadership program manager for the Chubb Group of Insurance Companies, put it well: “Maybe we can’t control the environment people return to, but we owe it to them to have a major influence over that environment.” And since learning organizations are increasingly being judged by business outcomes, it is in their own best interests to optimize Phase III. Companies that have embraced the Six Disciplines work actively to put systems, processes, and procedures in place so that line management and the learning organization share the responsibility for driving learning transfer. Learning organizations cannot “wash their hands” of responsibility for Phase III and still be successful. Adopt New Systems and Approaches The third reason that Phase III learning has been left largely unplanned and unmanaged is that until recently there were no practical methods to facilitate, monitor, and support the process. As we will discuss in chapters D4 and D5, new approaches and technologies, from teleconferencing to social networking to learning transfer management systems, have greatly reduced this impediment. The main barriers now to embracing Phase III as the greatest opportunity to enhance learning are outdated mindsets and misperceptions, rather than reality.

CH002.indd 95

6/18/10 3:44:41 PM

96


Provide Performance Support Consumer products companies recognize that customer support is an integral part of the “whole product” experience. Good customer support contributes to customer satisfaction and repeat purchases. So they invest in instruction manuals, troubleshooting guides, online help sites, and personnel to ensure that customers can use their products effectively and well. Learning organizations strengthen their contributions to their businesses when they provide analogous forms of information and support. Participants are very likely to have questions or encounter difficulties when they first attempt to apply what they have learned. If they can obtain answers or assistance quickly and easily, they will be more likely to continue to try to utilize their new knowledge and skills. If they run into a problem and can’t get help or have a question but can’t find an answer, they are likely to become frustrated, give up the effort, and revert to pre-training behaviors. Specific suggestions for implementing ongoing guidance and support are discussed in D5: Deploy Performance Support. The point to be made here is that support during the learning transfer period needs to be a conscious part of the overall design. It should be planned and executed with the same care as has been traditionally afforded to instruction.

Phase IV: Achieve Phase IV completes the learning cycle. Participants’ achievements in applying what they have learned are assessed and recognized; opportunities for continued growth are identified. Indeed, “to achieve” means “to bring to a successful conclusion, to accomplish.” It follows that achievement is only possible when there is a clear standard for performance and when that performance is assessed in some way. This is the true finish line for learning. Program participants should be informed at the outset how and when their achievements will be assessed. There are three solid reasons for including assessment as part of the overall learning design: 1. It makes expectations explicit. 2. It establishes a clear goal line for the learning experience. 3. Assessment is, itself, a learning experience. Assessment Makes Expectations Explicit Employees want to know what is expected of them. In a study we did with AstraZeneca, for example, there was a direct correlation between clarity of

CH002.indd 96

6/18/10 3:44:42 PM


97

expectations and job satisfaction (Figure D2.9). Lack of clear expectations is a frequent cause of poor performance (Mager & Pipe, 1997, p. 34) and a source of frustration and employee dissatisfaction. Yet relatively few learning programs are really clear about what is expected of parEmployees want to know ticipants in terms of job performance. In the what is expected of them. absence of clear expectations and assessment of whether those expectations have been met, it should be no surprise that much of training goes unused. By analogy, consider what would happen to a company that set no targets for its sales people and never bothered to keep track of who sold how much. The fact is that measures drive performance, or as the saying goes, “what you measure is what you get.” Knowing that there will be accountability for demonstrating achievement causes participants to take the learning more seriously and give greater effort to applying it to their work. To achieve the desired results, however, the assessment has to be relevant in method, content, and timing. Assessment Sets a Clear Goal Line The second reason that evaluating achievement should be part of the complete experience is that it demarcates a clear finish line for a training cycle and establishes explicit conditions of satisfaction. The importance of having a clear finish FIGURE D2.9. CHANGE IN JOB SATISFACTION WHEN EXPECTATIONS WERE MADE CLEAR

% of respondents

60% 50% 40% 30% 20% 10% 0% Less satisfied

CH002.indd 97

More satisfied

6/18/10 3:44:42 PM

98


line and accountability for learning is familiar to anyone who has ever audited a class. When you audit a class (take it without exams, assignments, or credit), you inevitably give it short shrift when the work piles up and deadlines are tight at the end of the semester. Since there is no assessment in a class that you audit, it is human nature to spend your time on the courses for which you will be held accountable. Corporate universities, by failing to assess achievement, are in effect allowing all their students to audit all their courses. In academic universities, however, you do not receive credit for just auditing a class. In most corporate universities, you do; all you have to do is show up. The problem is that employees are held accountable and are routinely assessed for other aspects of their work. Not surprisingly, those receive a much greater share of their effort, time, and attention, just as graded courses do compared to pass-fail or audited courses. Even setting aside the accountability aspect of assessment, having a challenge with a clear goal itself is motivating for most people (Pink, 2009). Think of the hours that people spend doing crossword puzzles or playing solitaire or video games in which the only objective is to “win” by reaching some arbitrary goal, which carries no reward other than the intrinsic satisfaction of having reached it. Why do we deny participants the satisfaction of knowing they have achieved the goals for which the program was created? Assessment as Learning Finally, the assessment process itself fosters learning in at least two ways: it requires retrieval, which improves recall through the spacing effect, and it provides feedback on performance, which is essential for improvement. Retrieval Practice An interesting and well-known aspect of human learning is

that the more often information is retrieved from memory, the easier it is to recall subsequently. Retrieval of information is a first and critical step in converting new knowledge and skills to action. Therefore, educational practices that enhance retrieval (remembering) facilitate learning transfer and performance improvement. Various kinds of assessments, and even just preparing for assessments, reinforce learning and improve recall by requiring learners to retrieve and process the information again after the original learning occurred. The effect is even more powerful when the retrievals are spaced out in time (see review by Thalheimer, 2006). Thus, assessment, especially when it takes place after some time has elapsed, itself helps reinforce and solidify learning. Moreover, most people, faced with an upcoming assessment, review the material or rehearse the skill in their minds, thus further reinforcing the learning.

CH002.indd 98

6/18/10 3:44:43 PM


99

Feedback Feedback is essential to improve performance. In the absence of feed-

back, it is impossible to know whether you are doing something right or wrong, whether you are improving, and what you need to do to get better. It is a vital Real experts seek out component of developing expertise: constructive, even painful “Real experts seek out constructive, even feedback. painful feedback” (Ericsson, Prietula, & Cokely, 2007). But not everyone is prepared to voluntarily seek such feedback. Here again, there is value in defining a specific time and process for assessing achievement; it provides participants with much-needed feedback on how well they have mastered the skills and material and what they need to continue to work on in the next cycle. Nature of the Assessment For assessment to achieve the goal of supporting learning in Phase IV, it must measure the behaviors, skills, and results that the program was intended to foster. Just as instructional methods must be selected based on desired outcomes, so must assessment methods. In the 1950s, Benjamin Bloom and colleagues developed a taxonomy for educational objectives (Table D2.5). The important point for this discussion is that assessment at any given level does not predict performance at higher levels. In other words, testing participants’ ability to recall specific facts, terms, concepts, or strategies tells you very little about their ability to actually apply those concepts. If the goal of a learning program is to have people apply their knowledge to deliver a more effective sales presentation, then the assessment must specifically test application in an appropriate scenario. Unfortunately, the vast majority of assessment questions—especially in e-learning programs—are at the lowest cognitive domain (recall) even when they are intended to assess application, analysis, or higher level skills: In general, the single most useful improvement you can make in writing test items is to write them above the memorization level . . . the vast majority of test items are written at the memorization level. In contrast, the vast majority of jobs require performance that is above the memorization level. This disconnect between testing practice and job performance is what often leads management to question the value of training and turns testing into a misleading indicator of performance. E.g., “How come you passed the course but can’t do the job?” is a common summary of the problem (Shrock & Coscarelli, 2007, p. 157, emphasis in the original).

For assessments to be valid and support the desired learning, they must evaluate the actual behaviors, skills, and results desired. In general, the principles of

CH002.indd 99

6/18/10 3:44:43 PM

100


TABLE D2.5.

TAXONOMY OF COGNITIVE DOMAINS

Level

Domain

Definition

Examples

6

Evaluation

Form judgments about the value or worth of information, products, ideas, or quality of work and be able to defend the basis of the opinion.

appraise, compare, contrast, criticize, defend, justify, support, validate, judge

5

Synthesis

Put information together in a different way or new pattern to produce something unique or original.

categorize, compile, compose, create, design, devise, formulate, predict, produce

4

Analysis

Break information into parts. Differentiate among facts, opinions, assumptions, and conclusions. Identify errors in logic.

break down, deduce, diagram, differentiate, distinguish, illustrate, infer, outline, point out, relate, subdivide

3

Application

Use previously acquired knowledge in a new way or to solve problems in a new situation

change, compute, demonstrate, develop, modify, operate, organize, prepare, relate, solve, transfer, use

2

Comprehension

Demonstrate a level of understanding by restating, organizing, comparing, translating, interpreting, or stating main ideas

convert, estimate, defend, distinguish, discriminate, explain, generalize, summarize, infer, paraphrase, predict

1

Knowledge

Ability to recall previously learned facts, rules, terms, strategies, etc.

define, describe, identify, label, list, match, name, select, state

After Bloom, Englehart, Furst, Hill, and Krathwohl, 1956.

an effective program evaluation apply to individual assessments; they must be relevant, credible, compelling, and efficient (see D6: Document Results). Assessments of individual achievement may be part of the overall program evaluation, but they need not always be. For example, a well-designed selfassessment—one that allows each participant to gauge his or her success in applying the course principles and that provides feedback to guide further improvement— would support Phase IV learning, whether or not the results were ever used as part of the program evaluation. So would a structured feedback session between manager and employee or a teleconference in which each participant recounts his or her achievements, stumbling blocks, and plans for further development. Phase IV means defining an end point at which to assess and recognize individual achievement, whether or not these are part of program evaluation (see below and D6).

CH002.indd 100

6/18/10 3:44:44 PM


101

The most important thing is that there be an assessment. A cycle of learning is truly complete only when it includes a point at which the participant’s achievements in “putting learning to work” are assessed and recognized. Assessment provides participants with a clear goal to strive for, definitive expectations for improvement, and feedback on their progress, all of which help increase the amount of learning that is transferred and all of which contribute to meaningful results.

Staple Yourself to the Learner Finally, you need to ensure that the learner’s complete experience will be “all of a piece” and constitute a coherent experience (see Case in Point D2.4). We have found the concept of “stapling yourself to the learner” to be a useful exercise in this regard.

Case in Point D2.4 Leadership Development with Impact One learning leader who truly understands the importance of designing the complete learning experience and delivering relevant results is Larry Mohl, CLO of Children’s Healthcare of Atlanta. Children’s Healthcare is one of the nation’s finest and fastest-growing providers of healthcare for children, treating more than half a million patients a year. Children’s rapid growth, combined with a focus on quality, in the face of the fi nancial pressures and complexity of today ’s health care environment, create a need for a steady supply of leaders with leadership and business skills in addition to their medical knowledge and expertise. To meet this challenge, Larry and his team created the Center for Leadership. At the onset, they defined clear and challenging measures of success for the Center that included: • Improved personal leadership • Competency improvement • Readiness for new or expanded role • Enhanced leadership pipeline • More internal promotions and “ready now” leaders • Operational impact • Efficiency, standardization, new or improved capabilities and services, etc. • Financial impact • Operational impacts on cost, revenue, and funding

CH002.indd 101

6/18/10 3:44:44 PM

102


Case in Point D2.4 Leadership Development with Impact (continued) Indeed, Larry accepted that continuing funding for the Center would be contingent on demonstrating progress against each of these four key conditions of success. To meet the need, the Center for Leadership designed four successive programs to address the new challenges leaders face as they advance through the leadership pipeline: supervisor certification, management certification, management acceleration, and executive experience. An example of a “complete learning experience” is the Executive Experience. This twelve- to eighteen-month-long program includes a holistic initial assessment; five sequential workshops with follow - through support: personal coaching, and action learning projects to continue learning during the intervals; and a rigorous final analysis using the success case methodology to document results and identify opportunities for continuous improvement. The Center’s holistic approach to development has been an outstanding success. Larry and his team have been able to document significant progress in all four key measures of success, from improved leadership ability, a stronger pipeline and more internal promotions on the one hand, to enhanced revenues, cost savings, and expanded services on the other. Larry shared the following insights: “To be successful, you have to build credibility with senior executives. That means you have to promise real results on the most pressing needs of the organization, measure what matters, and drive relevant impact. You have to be willing to show management both the good and the bad so that you can engage them in problem solving. “Most people have a difficult time connecting the dots between what they are learning, how it is being applied, and how it is impacting the operation. You’ll have to work to dig out the results and make them credible. Get a finance person on your core team and build both emotional and rational value. “Finally, don’t wait for the ‘big presentation’ to tell your story! Continually connect the dots between people, program, and impact and ask participants to do the same.”

The idea grew out of a landmark article by Shapiro, Rangan, and Sviokla (1992) in Harvard Business Review. They argued that the only way to really understand your customer’s experience (and how to improve it) was to figuratively “staple yourself to an order.” That is, physically follow an order through all the steps in your company to see how many times it was handled, how often it was set aside, how hard it was to find its status, where mistakes occurred, and so forth.

CH002.indd 102

6/18/10 3:44:45 PM


103

FIGURE D2.10. CHECK THE PROGRAM DESIGN BY IMAGINING YOURSELF STAPLED TO A LEARNER THROUGHOUT

The application of this idea to learning and development is to imagine yourself stapled to a learner as he or she goes through all four phases of learning (not just the detailed agenda of Phase II), from the invitation until assessment of progress weeks or months later (see Figure D2.10). At each stage, ask yourself: If I were the learner, • Would I understand what is expected of me? • Would I understand how this relates to other systems, slogans, and corporate initiatives? • Is it clear how the learning initiative relates to my work? Would I be able to use what is being taught? • Can I see the benefit for me personally? Why are they asking me to do this? • Could I connect each exercise to things I already knew or learned on the job or in other parts of the program? • What would influence my opinions and actions at this point in the process? • Where would I turn for help if I needed it? • What does my manager think? Does she support it? How would I know? • What is expected of me? How will I be evaluated? Will anyone know or care whether I use this stuff ? Whenever we have done this exercise with a client, they have discovered opportunities to make improvements that strengthened the overall experience and impact.

Summary The Second Discipline practiced by the most effective corporate learning organizations is that they include the complete learning experience in their plans—from

CH002.indd 103

6/18/10 3:44:45 PM

104


how the program is positioned in the invitation, to the preparation expected in Phase I, the instructional methods of Phase II, the accountability and support of Phase III, and how achievement will be assessed and recognized in Phase IV. Their program plans include how the contribution of all the key players—both inside and outside the learning organization—will be encouraged and coordinated. Designing the complete experience goes well beyond the traditional scope of corporate learning professionals. It will require learning new skills and abandoning long-standing paradigms. It is, in our experience, the only way to achieve a real breakthrough. Adopting this holistic approach to learning and learning transfer dramatically increases output and both the perceived and real value of educational efforts. Redefining the finish line as the delivery of results, rather than of the last day of class, is an invigorating challenge that offers substantial rewards. Use the checklist for D2 to ensure that the program design contemplates the “complete experience.”

The complete experience goes well beyond the traditional scope of corporate learning.

CHECKLIST FOR D2 PHASE I—Prepare

CH002.indd 104

✓ ❑

Element

Criterion

❑

Selection

The selection or enrollment process makes sure the “right people are on the bus”—meaning those with appropriate job experience and responsibilities to benefit from the program.

❑

Invitation

The invitation is clear and compelling. It explains the rationale for the program, summarizes its content, and sets expectations for its subsequent use. Ideally, the invitation comes from an influential and committed sponsor.

❑

Preparation (participants)

There is meaningful preparatory work—reading, exercises, simulations, performance feedback, etc.—that will help maximize the time spent in the learning program itself.

❑

Preparation (participants with managers)

A pre-program meeting with the participant’s manager is strongly encouraged (ideally, required). Guidelines and worksheets for that meeting are provided.

❑

Preparation (managers)

Managers are provided an overview of the program, its objectives, and the business needs being addressed, as well as step-by-step guidelines for fulfilling their role in maximizing results.

6/18/10 3:44:47 PM


105

PHASE II—Learn Element

Criterion

❑

Use of Phase I Learning

The preparatory work is utilized extensively in the program— so much so that those who did not complete it are at a disadvantage (or ideally, are not allowed to attend).

❑

Value Chain

There is a clear understanding among the design team and facilitators of how each component relates to the desired behaviors, capabilities, and expected business outcomes. These links are made explicit to the learners.

❑

Relevance

Relevant examples, stories, simulations, discussions, and so forth are included to help learners see how the material applies to their jobs. Current practitioners and/or prior graduates of the program are used to help underscore its utility.

❑

Practice

The agenda provides adequate time for learners to practice the desired skills or behaviors with supervision and feedback.

❑

Process Check

End-of-course evaluations include assessment of whether learners perceived the utility and relevance of the program and feel prepared to use it to advantage in their work.

PHASE III—Transfer ❑

Performance Support

Resources are committed to ensuring that learners can get help in applying new skills and knowledge. The design includes the production of job aids or other materials and systems to support on-the-job application.

❑

Manager Involvement

Participants and managers meet following the course. Guidelines are provided for that meeting. On-going manager involvement is facilitated.

❑

Accountability

Processes are in place to periodically remind participants of their obligations, hold them accountable for progress, and recognize superior effort and accomplishment.

❑

Process Management

There are a process and systems in place to allow learning professionals to monitor, support, and manage the learning transfer process.

PHASE IV—Achieve

CH002.indd 105

❑

Recognition

Significant improvements and accomplishments are recognized in a meaningful way.

❑

Assessment

An end point is defined weeks or months after the instruction. There is a plan in place to assess achievement and participants know what it is. Significant improvement and accomplishments are recognized.

6/18/10 3:44:48 PM

106


Action Points For Learning Leaders • Review the programs for which your group is responsible to ensure that the designs truly encompass the complete experience from the learner’s point of view. • Use the “staple yourself to the learner” exercise, or a similar approach, to affirm that all four phases of learning and all six Ds have been included in the planning. • Be vigilant for mixed messages—where what is taught in the program and what is practiced in the business are inconsistent or where one phase does not support another. Such inconsistencies discourage participants from trying to transfer their knowledge and, if glaring, lead to cynicism. • Emphasize to general management the importance of their support in all four phases of the program. Explain how partnerships between line management and training and between learners and their managers are necessary to achieve optimal return on investment from learning and development programs. • Pay particular attention to Phase III plans. This period has been traditionally ignored and offers the greatest opportunities for improvement. • Be sure there is a long-enough period between instruction and assessment to allow participants to demonstrate improvement or achieve significant results. For Line Leaders • Ask learning and development to show the links between the instructional design and the business requirements for all-important programs. • Review them to be sure that the business objectives, the kinds of behavioral improvements expected, and the learning exercises link logically to one another. • If such linkages cannot be drawn or are suspect, the learning plan should be reconsidered. • Ask line managers what they are doing to ensure that development programs are reinforced so they “stick.” • Devise systems to hold line managers accountable for their roles in obtaining results from learning and development. • Measure and reward managers’ active engagement in the process.

CH002.indd 106

6/18/10 3:44:48 PM


107

• Ensure that “the video matches the audio.” That is, be certain that what management says, what managers do, and what the system rewards are in alignment. If not, you are wasting time and money in training. • Make sure that there is a process to assess, recognize, and, ideally, reward and celebrate achievements that result from learning well applied. • Make the necessary changes in the environment to support what is being taught and to maximize the impact of learning and development.

CH002.indd 107

6/18/10 3:44:49 PM

CH002.indd 108

6/18/10 3:44:49 PM

DELIVER FOR APPLICATION The real question is what can you do on Monday? What can you do Monday six weeks from now, six months from now? —LEO BURKE

T

he value delivered by a corporate educational program is proportional to the amount of new knowledge and skills that are applied to the work of the organization. Unused knowledge is like unplanted seed; it will never bear fruit. Therefore, the job of corporate learning and development is not merely to impart knowledge, but to be sure that it is put to work so that its benefits can be harvested. A key factor that influences application of knowledge is the way in which it is taught. Hence, the Third Discipline of creating breakthrough learning programs is to structure them so that they deliver for application. That is, so that they employ instructional methods that help bridge the learning-doing gap, honor adult learning principles of relevance and utility, and provide adequate time and guidance for goal setting and application planning. In this chapter we focus on those aspects of learning delivery that enhance the application of new knowledge and skills to the work of the individual and organization. Our goal is to complement more extensive works on instructional design by calling attention to the issues we frequently encounter in corporate learning and development programs despite the availability of well-honed instructional design models. Topics covered include how to: • Narrow the learning-doing gap • Motivate learners • Make relevance clear 109

CH003.indd 109

6/17/10 3:25:43 PM

110


• • • • • • •

Improve perceived utility Provide know-how Make learning memorable Prepare and plan for transfer Communicate what happened A checklist for D3 Action points for learning and line leaders

Narrow the Learning-Doing Gap There is always a gap between the learning environment and the environment in which knowledge and skills must be put to work. Differences include time pressures, novelty of the problems, magnitude of the consequences, complexity of the situations, distractions, ability to recall, and many others. The challenge for corporate educators is to help learners successfully bridge the gap between knowledge acquisition in a learning program and knowledge application on the job (see Figure D3.1). The size of the gulf between learning and doing is influenced by the way in which new skills and actions are taught. Relevant examples, active learning, simulations, practice sessions, and other application-focused delivery techniques narrow the gap, making it easier for participants to connect learning to their FIGURE D3.1. THE LEARNING ORGANIZATION’S CHALLENGE IS TO DELIVER TRAINING IN WAYS THAT BRIDGE LEARNING TO DOING

CH003.indd 110

6/17/10 3:25:46 PM

Deliver for Application

111

work. Passive listening, endless PowerPoint slides, poorly designed e-learning, and assessments focused solely on recall widen the gulf, with the result that fewer participants make the transition and achieve results (see Figure D3.2). Medina (2008) recently reviewed the literature on brains and learning and in his engaging book, Brain Rules, propounded twelve “rules” about how the brain functions. He summed up his findings this way: “What do these studies show, viewed as a whole? Mostly this: If you wanted to create an educational environment that was directly opposed to what the brain is good at doing, you would probably design something like a classroom” (p. 5). For participants to generate business results by applying what they learn in a corporate educational program, three conditions must be met: • They must perceive value for the effort required. • They must have learned how. • They must be able to retrieve the relevant information and strategies. The goal of the Third Discipline, Deliver for Application, is to be sure that participants leave Phase II of learning programs with the motivation and ability to apply their new knowledge. Methods to achieve these outcomes are discussed below.

FIGURE D3.2. THE WIDER THE GAP BETWEEN LEARNING AND DOING, THE FEWER PARTICIPANTS MAKE THE LEAP AND ACHIEVE RESULTS

Learning

Doing Work

Copyright © 2010 Fort Hill Company. Used with permission.

CH003.indd 111

6/17/10 3:25:47 PM

112


Motivate Learners Motivation to learn is a sine qua non for effective education; it’s vital to learning through coursework and it is vital to continued learning through application on the job. And what motivates most adults to learn is the expectation of future benefits that they value—in other words, “What’s in it for me?” Program participants will be more motivated to learn and, ultimately, more motivated to apply what they learn, if they are Motivation is vital to learning. convinced that doing so will produce an outcome that they personally value. The specific benefits that each individual values is a matter of personal preference. From a very early age, human beings exhibit a strong intrinsic motivation to learn (National Research Council, 2000); mastery of a concept or skill is its own reward. In the context of corporate training programs, however, most employees are motivated to learn and use skills that will make their work demonstrably easier or faster, improve their working environment, and increase their chances for promotion, recognition, and so forth. As Richard O’Leary, director of Human Resources and Diversity for Science and Technology at Corning, explained to us: “A design criterion should be that the program feels integrated with how you get results. Make sure you are designing learning that people have a hunger for because it helps them solve what they think is the problem” (personal communication, 2004). Victor Vroom of the Yale School of Management developed the Expectancy Model as a way to explain what motivates employees (Vroom, 1964/1995). Vroom proposed that a person is motivated when he or she believes that effort, performance, and outcomes are linked and when the outcome is of personal value to him or her (see Figure D3.3). One element is expectancy—the strength of the belief that making an effort will produce better performance. In terms of training and development, the more I believe that learning the material and using it on the job will improve my performance, the stronger my motivation will be to make the effort to do so. For example, a participant will be more motivated to attend training and modify her sales approach if she expects that it will help her achieve significantly higher sales than if she believes that it will make only a small, incremental difference. The second element in Vroom’s motivation model is instrumentality—that is, the perception of how closely linked (instrumental) improved performance is to realizing an outcome or reward. Obviously, I will be more motivated to try to improve my performance if I believe performance is instrumental in winning a promotion than if I am convinced that promotion depends entirely on politics.

CH003.indd 112

6/17/10 3:25:48 PM


113

FIGURE D3.3. MOTIVATION DEPENDS ON THE BELIEFS THAT EFFORT WILL IMPROVE PERFORMANCE AND THAT PERFORMANCE WILL LEAD TO REWARD AS WELL AS THE PERSONAL VALUE ASCRIBED TO THE REWARD

Motivation

Valence I value the reward

Effort

Expectancy

Instrumentality

I believe this will improve my performance

I believe improved performance will be rewarded

Performance

Reward

Finally, valence is the relative worth that each individual places on the expected outcome. Other things being equal, the more a person values the outcome, the greater his or her motivation will be to learn the skills and behaviors necessary to achieve it. It is important to note here that intrinsic rewards—for example, the personal satisfaction of solving a problem or achieving mastery—may be valued as highly as, or even more highly than, extrinsic rewards (Pink, 2009). Thus, motivation is the product of three factors: how much the person values the potential outcome, how closely he believes the outcome is tied to performance, and how much he believes that the effort required will produce improvement. In general, the greater the strength of each element, the greater the overall motivation will be. We say “in general” because if any one element is completely missing—for example, if I place no value on the outcome, or I don’t believe I am capable of improving my performance—then I will not be motivated, even if two out of the three conditions are met. Maximizing participants’ motivation to learn, then, requires satisfying three conditions: • Addressing learning programs to outcomes that people value • Persuading them that the recommended approach is tightly coupled to improved performance • Convincing them that their efforts will be rewarded

CH003.indd 113

6/17/10 3:25:48 PM

114


While learning professionals largely control the first two conditions, rewards and recognition are the province of the participants’ direct supervisors. That is why in D1 we recommended raising the question: “Besides training, what else needs to be in place to achieve these results?” It is much easier to motivate people to learn and apply their learning when they value the outcomes from doing so and there is a process to ensure that successful efforts are identified, recognized, and in some way rewarded. Motivation is especially important in corporate training and development programs because Meaningful change inevitably such programs are, in the end, change initiainvolves some discomfort. tives. They are intended to help people change from the way they currently do their jobs to an approach that is more productive for themselves and for the company. Meaningful change inevitably involves some discomfort and requires sustained effort. Compelling answers to the “What’s in it for me?” (WIIFM) question are needed to motivate people to put forth the required effort. WIIFM has been called the “universal radio station” because everyone listens to it (see Figure D3.4). When FIGURE D3.4. LEARNERS WANT TO KNOW WHAT’S IN IT FOR THEM

© Robert Weber/Condé Nast Publications/www.cartoonbank.com. Used with permission.

CH003.indd 114

6/17/10 3:25:49 PM


115

people understand “what’s in it for me,” they are more receptive to new ideas, more motivated to learn, and more willing to soldier through the discomfort that accompanies any significant change. Therefore, the first task of delivering for application is to be explicit about the value that participants can expect if they attend the program and apply its teachings. The promised benefits need to be believable and achievable—a task made easier if the objectives have been clearly linked to business priorities in the first place. Participants will want to attend the program if both its benefit and relevance are clear. Beverly Kaye, founder of Career Systems International and a thought leader in the area of employee development and retention, put it this way: To ensure that people get the maximum value from learning and development, we need to engage their hearts as well as their heads. We design and deliver every program so that when participants walk out the door they feel empowered and prepared to take action with a “can-do” attitude. We intentionally work to get participants’ adrenalin going. We engage their hearts so they have a passion to apply what they learn. They leave with an understanding that no one can take charge of their development and job satisfaction but themselves. So learning about taking charge of your own development is more than a cognitive exercise.

Make Relevance Clear For adults, relevance and motivation to learn are closely linked. A key principle of adult learning is that learners want to know why they are being asked to learn something—how the material relates to them, their role, their division, and their business—before they are willing to learn and then use it (Exhibit D3.1). “Research indicates that the need to know affects motivation to learn, learning outcomes, and post-training motivation to use learning” (Knowles, Holton, & Swanson, 2005, p. 201). It is not surprising, therefore, that perceived lack of relevance is one of the major barriers to the transfer of learning (Kirwan, 2009; Newstrom, 1986). Jim Trinka is the technical training director for the U.S. Federal Aviation Administration and co-author of A Legacy of 21st Century Leadership: A Guide for Creating a Climate of Leadership Throughout Your Organization. When we asked him why such a high percentage of his trainees utilize what they learn, he told us: I think it’s the ease with which we can connect what’s in the training program to what they’re going to do on the job. It’s easy to make those connections, so I like to call it connecting the dots. It’s easy for an instructor to connect the dots

CH003.indd 115

6/17/10 3:25:50 PM

116


from what the information and content in the course is to what the controller or technician is going to have to do every day on the job. And because that connection is easy, there is a great belief among the developmental or student controllers that they are actually going to use the information they are getting. Everything that we teach them, they will have to know in their careers. We don’t teach things that don’t have any applicability. We keep all that out. So it’s all 100 percent usable content. That’s why you get 100 percent transfer.

Exhibit D3.1 Some Relevant Principles of Adult Education Many of the principles of “delivering for application” reflect the principles of adult learning as summarized by Knowles, Holton, and Swanson in their classic The Adult Learner (2005). Human beings continue learning throughout their lives. Adult learners, however, have specific needs and preferences that must be addressed for maximum effectiveness. Specifically: • Adults are practical and place high value on relevancy. • In a work setting, they may not be interested in knowledge for its own sake. • Adults want to see a reason for learning something. They want to know, “What’s in it for me?” • The relevance of theories, concepts, examples, and exercises needs to be immediately apparent; concrete examples of application help. • Adults are goal-oriented. • They prefer a problem-centric rather than content-centric approach. • They want to know how the class will help them attain their personal goals. • Adults learn from experience. • Include opportunities (simulations, role play, practice sessions) to apply their new knowledge and practice skills in a safe environment. • Provide feedback that reinforces the correct actions and provides insight into missteps. • Adult learners need to be shown respect. • Adults bring with them a rich tapestry of experiences and knowledge that should be honored and drawn upon. • Treat learners as equals and encourage them to share their wisdom and opinions in class. • Never ridicule or “talk down to” participants. • Adults are autonomous and self-directed. • They need to be actively involved in the learning process, assuming responsibility for group discussions, presentations, and so forth. • Instructors of adults are most effective when they act as facilitators—guiding participants to their own knowledge and conclusions—rather than supplying them with facts.

CH003.indd 116

6/17/10 3:25:50 PM


117

Further evidence for the importance of relevance comes from the analysis of tens of thousands of post-program evaluations at Hewlett-Packard. Peggy Parskey, who was then global learning process manager, worked closely with Michael Ross of Performance Challenges Corporation to build a model using several multivariate techniques (including multiple regression and cluster analysis). The model captures higher-order reaction constructs, predicts overall assessment from three specific metrics, and presents the data in a powerful and easily interpreted way (see Figure D3.5). The strongest predictor of a program’s overall perceived quality was one facet of each of the following factors: achieved objectives (learning experience), recommend to peers (learner impact), and enhance job performance (utility). FIGURE D3.5. IN HEWLETT-PACKARD’S DATA, THREE FACTORS—UTILITY, LEARNING EXPERIENCE, AND LEARNER IMPACT—PREDICT THE OVERALL ASSESSMENT OF VALUE Training techniques

Instructor style, method, pace

Learning experience

Recommend to peers Instruction satisfaction

Achieved objectives

Learner Impact

Index average

Content satisfaction

Logistics satisfaction

Utility

Skill/concept relevance

Enhance job/role performance

Motivated to apply

KEY BSC measures

Predicts overall quality

Primary indicator

Copyright © 2003 Hewlett-Packard. Used with permission.

CH003.indd 117

6/17/10 3:25:51 PM

118


Three ratings contributed to the utility score: • The relevance of the skills and concepts • The extent to which learners expected the program to enhance their performance • Whether the learners were motivated to apply what they had learned Hewlett-Packard tracked the utility scores for each course offering in real time. Any program that scored low on perceived utility was targeted for special attention. Low perceived utility, of course, does not necessarily mean that the program was not useful or relevant. But it does mean that somehow the relevance and applicability of the material were not effectively communicated to the participants, or that they were not the right audience, or that there was some other problem. The rationale for singling out courses with low utility ratings for special attention is that if participants leave a program with a poor opinion of its utility, it is very unlikely that they will make the effort to transfer the learning or produce positive results on the job (Parskey, personal communication, 2005). In other words, if the course content is not relevant as perceived by the participants, then they won’t be motivated to learn or use it. Other aspects of the program—instruction, venue, learning exercises—no matter how brilliantly executed, cannot compensate for a lack of perceived utility.

Improve Perceived Utility To improve the perceived utility of a program—and hence the motivation for learning and learning transfer—be explicit about the business needs that the program is designed to address (D1) and use business-relevant examples to illustrate the connections between the content and the work of the company. This requires that instructors understand the business needs in depth. What are the key business drivers? How does the company actually generate value? What are the major challenges it faces from competitors, a changing market, regulation, internal inefficiencies, and so forth? What is its vision? What are the expectations of its leaders? A firm grasp of these core business issues greatly enhances an instructor’s effectiveness and ability to make the content relevant for participants. It is also essential for his or her credibility. The importance of credibility explains why using senior executives as faculty is one of the fastest-growing trends in executive education (Bolt, 2005, p. 11): they have instant credibility and a deep grasp of the issues. While internal instructors should, in theory, have a deeper grasp of the company’s specific business challenges than external facilitators, it is not always

CH003.indd 118

6/17/10 3:25:52 PM


119

possible or practical to use only internal trainers. External instructors and experts can introduce fresh insights, new perspectives, and new approaches that are important to keep a company from becoming too insular. But it is impossible for instructors from outside the company to be conversant with the details of the multifarious product lines, operating divisions, and business units. One creative solution to this dilemma is to team-teach by pairing external content experts with internal managers who can provide the business context and company-specific examples. This approach achieves the best of both: deep content expertise as well as credible examples of work situations that are immediately recognizable and relevant to the participants (Connolly & Burnett, 2003). Another rich source of relevant examples is the participants themselves: “for many kinds of learning, the richest resources for learning reside in the adult learners themselves” (Knowles, Holton, & Swanson, 2005, p. 66). Consistent with the principles of adult learning, the key is to honor and build on the experience participants bring to the program. Design exercises that draw upon the experiences of the group members to generate examples that are immediately relevant and applicable to their work environment.

Provide Know-How For learning to produce performance improvement, participants have to know more than just new information; they have to know how to use it in their work. Delivering for application means using instructional approaches and exercises that require participants to practice the kind of thinking and actions they will need on the job. For example, if the objective is to improve the ability of the participants to handle objections, then they need to be given the opportunity to For learning to produce practice their skills during the program improvement, participants and get feedback on their performance. have to know how to use it. It is not sufficient for the instructor to just talk about it. Marc Lalande, president of Your Learning GPS, illustrated the point this way: “I was a ski instructor for fifteen years. Teaching skiing in the classroom with PowerPoint was never an option.” Yet, in today’s time-pressed business climate, interactive exercises and practice are often the first things jettisoned because they “take too much time.” Lalande feels strongly that such reasoning represents false economy. Indeed, when he was head of training for a pharmaceutical company and was asked to shorten a program, he would tell the sponsors: “You can eliminate anything you want from the

CH003.indd 119

6/17/10 3:25:52 PM

120


agenda except the time for practice and role play. If you remove that, you might as well cancel the program” (Lalande, personal communication, 2010). Link Instruction, Behaviors, and Business Outcomes Delivering for application requires forging a clear and logical value chain that links each learning experience to the on-the-job performance required, and the required performance to the ultimate business outcomes. Brinkerhoff and Gill (1994) developed the “impact map” as a way to think about and illustrate the links between learning and business results. We prefer the term “value chain” as it is more familiar to business leaders. Michael Porter defined a value chain as the sequence of value-adding activities by which a company creates competitive advantage (Porter, 1985). The value chain for learning illustrates the sequence by which learning contributes to business success. A simple three-link chain is shown in Figure D3.6. The requisite skills and behaviors are identified in D1 by the needs-analysis and first two quadrants of the Outcomes Planning Wheel. The task for instructional designers, then, is to select the instructional strategies that will optimize the ultimate business outcomes. That seems painfully obvious. And yet, if you were to actually map the value chain of many corporate training programs, they are as illogical as Lalande’s remark about teaching skiing by PowerPoint; there is often an obvious disconnect between the stated purpose of the instruction and the methodology employed (Table D3.1). There may be a place for an orientation lecture, but ultimately, people have to hit the slopes to learn to ski, just as they need to practice new skills to succeed in business.

FIGURE D3.6. SIMPLIFIED VALUE CHAIN LINKING LEARNING ACTIVITIES TO BUSINESS OUTCOMES

Desired Business Outcome

CH003.indd 120

Skills and Behaviors Required

Learning Modalities

6/17/10 3:25:53 PM


TABLE D3.1.

121

A FACETIOUS EXAMPLE OF A DISCONNECT BETWEEN METHOD AND OBJECTIVE

Outcome Sought


Learning Modalities

Safe and enjoyable downhill skiing for beginners

Ability to start, stop, and turn effectively

Have students sit passively listening to an illustrated lecture

Safely enter and leave chair lift Correctly identify trail and difficulty markers Obey safety rules Be aware of and courteous to other skiers And so forth

A useful check on the degree to which the program has been designed to “deliver for application,” then, is to complete a value chain for the proposed course, looking for congruence between objectives and methods. Remember to include all four phases of learning in the learning modalities column; not just the formal instruction period (Table D3.2). Note that value chains grow increasingly more detailed as they progress from the high-level business objectives to specific behaviors to learning strategies. Any given outcome usually requires multiple skills and behaviors, each of which typically requires several different instructional methods and opportunities to learn and practice. For planning purposes, it may be helpful to insert columns on either side of the behaviors to allow a more detailed breakdown of the outcomes and requisite skills and actions (see Table D2.3). An obvious question is: Is it worth the time and effort to map the linkages in this way? Our answer is unequivocally, “Yes”; the ultimate success of a training program depends on getting the instructional strategy right. Being able to clearly explain how a given exercise relates to the business purpose helps participants understand its relevance and, by so doing, increases their motivation to learn and apply its lessons. Margolis and Bell (1986) presented a compelling case for introducing every learning exercise with the rationale from the learner’s perspective (see Case in Point D3.1). The learning and performance consulting group at Farm Credit Canada— Canada’s leading agricultural lender—has extended the use of value chain mapping in two important and innovative ways (Bartlett, personal communication, 2010): • They include a performance map in the summary letter that is sent to a business sponsor as follow-up to a discussion about a potential training opportunity. The map helps to summarize and confirm the links between the core strategies,

CH003.indd 121

6/17/10 3:25:54 PM

122


the business scorecard, the desired behavioral outcomes, and the recommended approach to training. • Performance maps are also included in the descriptions of courses in the Field Development Program to make it clear to participants and their managers how the course relates to the organization’s key strategies and performance scorecard. Mapping the congruence between requisite behaviors and learning modality helps avoid the overwhelming temptation to start with the method (for example, a podcast) and try to make the content fit, or see what slides are already on file and work backward from there. As consultant and author John Izzo facetiously remarked: “It’s easier to find a new audience than to write a new talk.” To avoid the latter temptation, UBS Bank instituted a “PowerPoint Free Zone” rule in its leadership development program for managing directors. Not even senior TABLE D3.2. PORTION OF A VALUE CHAIN LINKING LEARNING AND OUTCOMES FOR A SALES MANAGEMENT PROGRAM Business Outcome Sought


Learning Modalities

Increased sales as a result of more effective sales management

Correctly identify a sales representative’s development stage and use most effective situational leadership style to manage

Preparatory reading about situational leadership Instructor-led discussion of model Guided practice identifying stages in videos On-the-job practice and reflection Self-assessment on use of model

Effective use of the situation, behavior, impact coaching model

Preparatory reading about model Instructor-led discussion of model Practice giving feedback to peers in program, observed by expert coach On-the-job practice and reflection, with access to coach Assessment of coaching effectiveness by direct reports

CH003.indd 122

6/17/10 3:25:54 PM


123

Case in Point D3.1 Tell Me Why Corporate training exercises are (or at least should be) designed with the end in mind— that is, how the exercises relate to building the requisite skills to achieve the business goals. But while that linkage is clear to the instructional designers, it is often lost on the participants. If they are unable to make the link between the exercise and their job responsibilities, they will not be motivated to take it seriously, participate, and learn from it. A significant contributor to the disconnect is the way in which exercises are typically introduced, whether in the classroom or online. The tendency is to go immediately to the “how” without explaining the “why,” what Margolis and Bell (1986) call the “administrative approach.” A typical example in the classroom would sound something like this: “In the next ten minutes, I am going to break you into small groups. . . .” or in e-learning: “In each of the following scenarios, pick the best next step.” The problem with introducing exercises in this way is that participants begin thinking immediately about whether they will have enough time and whether they like or hate this kind of activity, rather than about its purpose and potential payoff; the thread to the purpose is lost and so are the participants. Margolis and Bell recommend that every exercise be introduced in a four-step process (pp. 61–70): 1. 2. 3. 4.

Explain the rationale. Explain the task. Define the context. Explain what is to be reported.

“This sequence follows the logic of learning and the logic of motivation. . . . The introduction/rationale is a statement that answers a fundamental question for the learner: ‘Why should I enter into this task or experience?’ . . . The rationale [should be] always stated from the learner’s perspective, not the trainer’s or the organization’s perspective” (pp. 62–63). An effective explanation of the task (step 2) is constructed so that the participants produce a product that aids their learning. The explanation should always include an action verb such as identify, list, rank, solve, and so forth, and usually a phrase that defines the quantity and quality desired, such as the five most important. Margolis and Bell explain the third step as follows (p. 65): “The definition of context explains how learners will accomplish the task. The context for the task involves three elements: • The size of the working unit (individuals or subgroups); • The composition of the subgroups, if they are used; and • The amount of time allotted for completing the task.” (continued )

CH003.indd 123

6/17/10 3:25:55 PM

124


Case in Point D3.1 Tell Me Why (continued) The fourth and final step—explain what has to be reported—is simply a matter of telling participants what they are responsible for reporting to the group as a whole once their subgroup has completed its task; it need not include administrative details, which have a tendency to confuse rather than clarify. Reporting out continues and enhances learning by allowing participants to review and summarize their discussions and by giving the instructor the opportunity to comment, embellish, and amplify key concepts and principles. “The cardinal principle to remember is to give the rationale for the activity before describing the task, how it is to be done, and what will be reported. This approach puts into practice a fundamental precept of adult learning: ‘Adults need to know why they need to learn something, before undertaking to learn it’ ” (Knowles, Holton, & Swanson, 2005, p. 74).

executives were allowed to bring their canned presentations. Although initially there was trepidation on the part of the presenters, both they and participants agreed that interaction was much greater and the outcome was superior to typical corporate presentations. In general, the closer the instructional environment mimics the actual work environment, the easier the content is to recall and apply (reviewed by Thalheimer, 2007). For this reason, Eli Lilly, for example, maintains a teaching facility for Good Manufacturing Practices complete with its own warehouse (for practicing forklift driving and safety) and a production line (for teaching and practicing sterile technique, cleaning procedures, and so forth). Some Examples The key point is that there is no one best method of instruction. Debates about the value of classroom versus e-learning versus virtual or other forms of instruction are meaningless without specifying what needs to be learned and what performance is required. As the National Research Council’s report How People Learn noted: “Asking which teaching technique is best is analogous to asking which tool is best—a hammer, a screwdriver, a knife, or pliers. In teaching as in carpentry, the selection of tools depends on the task at hand” (p. 22). Learning professionals should be “method agnostic.” That is, we should recommend the method, duration, and structure only after the business objectives and performance goals (D1) are clear. We do not have the space here, nor is it our intent, to write a treatise on instructional methods and their application. Numerous texts are available that

CH003.indd 124

6/17/10 3:25:55 PM


125

cover these topics in depth, for example Gagné, Briggs, and Wager (2004) and Smith and Ragan (2005). Instead, we have chosen to provide a few current examples and some caveats from our experience with corporate training programs. Skill Practice When the business objective of training requires that participants

become proficient in a particular skill, there is no substitute for practice with feedback. That requires some sort of simulated environment such as a role-play exercise, computer simulation, mock interview, or the like. Even then, a training program can only help participants reach a basic level of proficiency; true mastery requires ongoing practice with assessment and feedback on their performance on the job. For most people, having to demonstrate a task or technique in front of an audience or instructor is an uncomfortable—and therefore unpopular—exercise. So, while such skill practice is valuable, perhaps even indispensable for learning, it is not likely to generate positive reactions. One of the dangers of relying solely on learner reaction data (so-called Level 1 measurement) for evaluating programs is the evidence that participants tend to rate comfortable, easy, and ineffective teaching methods higher than they rate challenging, instructive, and effective ones (see Case in Point D6.2). One technique to ensure relevance and transferability, as well as providing time for practice, is by using “real play.” Participants bring real issues to the program and these are then used as the scenarios for role play and practice (see Case in Point D3.2). Simulations The best-known examples of delivering learning to maximize

application are flight simulators. First developed by E. A. Link to train pilots to

Case in Point D3.2 Real Play Versus Role Play An example of the application of D3 at a leading biopharmaceutical company is the concept of “real play” instead of role play. The senior training manager responsible for management development explains: “ The goal of ’ real play ’ is to reduce ’ death by PowerPoint ’ and increase the application of the feedback skills taught in our sales leadership program. Participants bring real personnel issues they are facing to the session. During the program, they rehearse applying the course principles to these real problems. The response to real play has been very positive. Participants report that the training is very relevant and that they feel much better prepared to apply the skills they have learned.”

CH003.indd 125

6/17/10 3:25:55 PM

126


fly by instruments, they were vital in accelerating the training of pilots during the Second World War. Current simulators are so sophisticated that commercial pilots can qualify to fly new models entirely in the trainers; they can also lose their licenses if they violate FAA rules while flying the simulators during recertification. In many ways, the simulators are actually superior to actual flight experience because they can be programmed to mimic any number of critical conditions that would be impossible or too dangerous to practice in a real airplane (engine fires, icing, landing with jammed landing gear, and so on). Of course, such simulators cost tens of millions of dollars, but they indicate what is possible in terms of matching the learning situation to the conditions in which the learning must be applied. As such, there is growing interest in and application of simulations to business training. A welldesigned simulation requires the same kind of Simulations are powerful, thinking (analysis, synthesis, pattern recognition, engaging, dynamic, and and so forth) and decision making that needs to effective. be applied to real business issues. Simulations compress time, allowing learners to complete business cycles in hours rather than months or years, so that they can see the longterm effects of their decisions. Simulations require active engagement and decision making, as opposed to merely passive listening, which boosts the retention of the lessons learned. Active involvement is one of the most frequently cited factors in making learning memorable. Akerman, Ekelund, and Parisi (2005) summarized the advantages of business simulations as follows: • Simulations increase executive business acumen and financial literacy. • Simulations build competency, alignment, and commitment around complex corporate strategies, business models, and initiatives. • Simulations are a powerful, engaging, dynamic, and effective way to reach executive audiences (p. 28). The challenge is to find simulations that teach skills applicable to real work and not just “gamesmanship.” Simulations can also be helpful in simplifying situations to highlight the effects of specific actions. To be effective, simulations need to be context-specific enough for participants to easily transfer the learning to their actual work roles; if participants cannot see the connection between the simulation and their business challenges, it will fail to produce the desired results. For this reason, many companies are now having custom simulations created to address their unique business challenges and educational objectives.

CH003.indd 126

6/17/10 3:25:56 PM


127

Regardless of the specific simulation or technology employed, a skillfully facilitated debriefing is essential to draw out the lessons learned from the exercise and illustrate their application to the real work of the participants and enterprise. “The goals are two: to make sure the participants clearly understand the relationships between their simulated decisions and market changes on the business result, and also to clarify the key learning points from the simulation” (Akerman, Ekelund, & Parisi, 2005, p. 35). Finally, action plans and support for learning transfer are required to ensure that the learning from the simulation is put to work in a way that leads to business value. Case Studies Case studies are a popular approach to executive education, but

they also can be used effectively in a broad range of other corporate training initiatives. Case studies can help draw out principles and illustrate the impact of decisions in the context of real problems. Like simulations, they have the advantage of requiring active engagement, thought, and decision making. Case studies are most effective when the case is presented and participants are given time to work in teams to come up with recommendations before the “rest of the story” is revealed and discussed. The same caveats apply to case studies as to simulations. That is, they must be realistic and similar enough to the participants’ actual job needs that participants can see how to apply the lessons to their work. As with simulations, much of the value derived from a case study depends on the skill of the facilitator in drawing out the lessons and applications during the debriefing. Using several different case studies to illustrate the same principle positively affects learning transfer. Nadler, Thompson, and Van Boven (2003), for instance, found that participants who were taught negotiation principles using several analogous cases were three times more likely to transfer their knowledge than those taught using only a single context. Games Sivasailam Thiagarajan (Thiagi) is probably the best-known and strongest proponent of using games to teach, arguing that engagement is essential to learning and that games produce a higher level of engagement than most other modes of instruction. Thiagi is careful to distinguish between “fun” and “engagement.” The goal of games in instructional design is to enhance engagement, not “fun” as such (ASTD, 2000). Games aren’t a good medium for delivering information, but they can help participants develop memorable mental models that enable them to better respond to complex situations in the real world that entail similar patterns. While much of traditional instructional design focuses on delivering content effectively, instructional game design focuses on the behaviors the learners need and the information required to perform them.

CH003.indd 127

6/17/10 3:25:56 PM

128


Thiagi makes the point that designing games is relatively easy; designing effective games is hard. The same caveats apply to games as to simulations: they need to be relevant to the business situation; they need to provide insight into the problem or issue; they need to be simple enough to highlight a specific lesson or consequence of an action, and yet not be so simplistic that participants dismiss them. Both games and simulations can be delivered effectively online as well as in person. Continuing advances in computer technology make it possible to create highly sophisticated and interactive online games and simulations. Unless planned and managed carefully, however, they can quickly degenerate into style over substance and actually obscure the main teaching points. As Judith Blair and Nancy Maresh of BrainWorks like to say, “Content is queen and context is king.” That is especially true when games are used to teach. e-Learning e-Learning, like classroom learning, encompasses a wide range of approaches and techniques that vary from highly engaging and instructive to mind-numbingly dull and ineffective. The technical capabilities of e-learning have the potential to make it especially effective for teaching people certain types of skills, as in the case of the simulations discussed above. But everything depends on the actual execution. Much of e-learning has failed to fulfill the hype of a few years ago—when pundits predicted that classrooms would disappear entirely—precisely because many programs have ignored the principles of relevance, utility, transferability, and practice. Simply converting a content-heavy, passive classroom learning experience to an online course does not improve its efficacy. While the periodic “quizzes” and self-checks typical of e-learning programs do require retrieval from memory, and therefore help foster retention, most fail to provide meaningful formative evaluation to help the learner improve his or her skills. That’s because the vast majority of the questions require only simple recall, whereas application to most jobs requires much higher-level cognitive and psychomotor skills (Shrock & Coscarelli, 2007). One other caveat regarding e-learning is worth mentioning. That relates to the issue of cognitive load. Cognitive load theory (Sweller, 1994) and numerous research studies illustrate that learning declines as the information processing demands on working memory increase (Clark, Nguyen, & Sweller, 2006). Instructional software and technology are now so powerful that it is easy to overwhelm the learner with too many simultaneous inputs (flash video, narration, text, and graphics all at once) so that understanding is actually impaired rather than enhanced. Action Learning Action learning would seem to be ideally suited to delivering for

application because it takes place in the context of real work. Its relevance and

CH003.indd 128

6/17/10 3:25:57 PM


129

utility should be immediately apparent and facilitate the transfer of the lessons learned to new challenges on the job. Smith and O’Neil (2003a, 2003b) summarize the key elements of action learning as follows: • Participants tackle real problems in small, stable learning groups. • The problems addressed are relevant to the participants’ workplace realities. • The groups meet periodically over time and participants take action between the meetings. • A collaborative learning process is followed that is based on mutual exploration, reflection, questioning, and constructive challenge. Numerous studies have demonstrated that, managed well, action learning can produce significant performance improvement. In part, as Kirwan (2009) points out, that is because “there is a large degree of overlap between the learning situation and the application situation, and so many of the barriers to transfer that might otherwise be in place are already dealt with” (p. 32). The downside of action learning is that, unless it is carefully managed, it quickly degenerates into simply another task in which learning takes a back seat to just getting the job done. Bolt expressed the concern that “the definition of action learning is getting broader and looser, with anything identified as projectoriented, experiential learning, or related to business problems being defined as action learning” (2005, p. 12). The learning component needs to be planned and executed with the same level of attention to detail as other forms of instruction. In particular, effective action learning requires guided reflection on lessons learned and facilitation that helps participants generalize from their specific problems to more broadly applicable principles and approaches.

Make It Memorable Retrieval of information from memory is essential at multiple points in the process of responding to a stimulus (Figure D3.7). Thus, the ability to recall the insights and skills gleaned from training is prerequisite to their adaptation and application. An important aspect of the Third Discipline, therefore, is to utilize instructional methods that enhance long-term memory and subsequent retrieval. This is especially important in preparing people for situations that they may not encounter for some time, such as difficult personnel issues, crisis management, leadership challenges and so forth. Such topics need to be presented in ways that are particularly memorable—so that they come immediately to mind much later during a crucial moment.

CH003.indd 129

6/17/10 3:25:57 PM

130


FIGURE D3.7.

GENERAL MODEL FOR THE ROLE OF MEMORY IN DIRECTING ACTION

Search memory for similar stimuli

Become aware of situation; direct attention to it

Retrieve relevant experiences, rules, and approaches from memory

Analyze and characterize situation

Retrieve the psychomotor skills needed to execute

Formulate plan of action

The research on human memory and learning is extensive—see, for example, reviews by Anderson (2010) and Medina (2008). New discoveries relevant to the craft of workplace learning continue to be made. In this section, we call attention to several findings that have specific bearing on improving people’s ability to recall what they have learned so that they are able to then apply it. These include: • • • • • •

Attention is essential, but limited Emotions matter Stories stick Repetition, especially at intervals, aids retention Connections count Learning requires reflection

Attention Is Essential, But Limited “The more attention the brain pays to a given stimulus, the more elaborately the information will be encoded—and retained” (Medina, 2008, p. 74). The problem that creates for corporate learning programs, however, is several-fold. First, humans can attend to only a very limited number of inputs at any given time. Second, attention is short-lived and easily diverted. Third, people don’t pay attention to boring things.

CH003.indd 130

6/17/10 3:25:57 PM


131

Multiple lines of evidence show that people can truly attend to only one or two input streams simultaneously. The issue appears to be a bottleneck in the processing of sensory input, rather than limits on the input as such (Anderson, 2010, p. 63). You know from your own experience that when you are truly attending an interesting conversation with someone at a crowded reception, you are able to “tune out” the other conversations occurring around you, even though many are clearly audible. Another example familiar to many learning professionals is the classic work of Simons and Chabris (1999). They showed that when people were asked to closely attend to a specific action in a video, over 50 percent completely failed to notice a man in a gorilla suit who walked into the center of the action and beat on his chest. The relevance to corporate learning is that participants cannot truly attend to the program if they are simultaneously reading their email or text messaging. The common practice of attempting to “multi-task” during webinars reduces the effectiveness of both the learning and the task, since there are demonstrable moments of mental “black-out” as attention shifts from one task to the other (Medina, 2008, p. 87). Compounding the problem of the bottleneck in the processing of input is the fact that attention is easily diverted. This is a special problem for knowledge workers, who need to be in what Csikszentmihalyi (1990) termed “the flow” to do their best work. The trouble is that it is not easy to get into “the flow.” It takes, on average, fifteen minutes for a knowledge worker to start working at maximum productivity. And it is very easy to be knocked out of the flow. Any number of distractions—phone calls, instant messages, email alerts, and so on—are sufficient to divert attention. Taking just one minute to answer a text message in effect costs sixteen minutes of high-productivity work. The same principle applies to learning. Frequent and unnecessary interruptions—including self-generated interruptions (“I’ ll just quickly check my inbox”)—disrupt the intense concentration needed to truly master a new topic or skill. In one study, for example, a single instance of a ringing cell phone demonstrably reduced the amount learned (McDonald, Wiczorek, & Walker, 2004). The point is that we need to minimize disruptions to maximize learning, and we need to educate learners on the cost of divided attention. What we attend to is affected by many factors, interest in particular. We find it easier to pay attention to things that interest us. Likewise, humans also are better able to remember items that they find personally interesting. Each of us knows someone who has a passion for some particular subject, sport, or hobby. Such people are able to recall information about their favorite topic—dates, names, statistics, and so forth—at a level of detail unimaginable to those who do not share the same passion. Their interests fuel their learning. The challenge for

CH003.indd 131

6/17/10 3:25:58 PM

132


corporate education is to create the same sort of passion for learning by making the relevance of the subject matter so compelling that it captures participants’ interest and, in turn, heightens their ability to attend to, absorb, and retain new learning (see Figure D3.8). Finally, the brain needs a break now and then. It is very difficult to truly attend to anything The brain needs a break for long periods. Medina (2008) feels strongly now and then. that the most common error in teaching is the tendency to overstuff students with information, with too little time allotted to “connecting the dots.” He recommends breaking all instruction into ten-minute segments because most learners’ attention begins to wander after ten or fifteen minutes. He suggested giving audiences a break from FIGURE D3.8.

LEARNING NEEDS TO BE RETAINED TO BE USEFUL

© Frank Cotham/Condé Nast Publications/www.cartoonbank.com. Used with permission.

CH003.indd 132

6/17/10 3:25:59 PM


133

the fire hose of information every ten minutes or so with some sort of “hook” that will get through to their executive control and recapture attention. Effective “hooks” have two characteristics: they are relevant to the topic and they evoke some sort of emotion—laughter, anxiety, disbelief, surprise, and so forth. Well-chosen stories and anecdotes can be especially effective. The point is that people have to first pay attention before they can learn and remember. So a key task of instruction is to capture, hold, and periodically recapture attention. Emotions Matter Why does Medina recommend that the periodic attention “hooks” involve some sort of emotion? Because emotions are extremely powerful at capturing attention and creating long-lasting memories. This is again something you know from your own experience; memories of traumatic or especially emotional events persist for years and are easily recalled, even when you would prefer to forget them. It turns out that all emotions—joy, sorrow, surprise—aid memory and recall. The reasons are beyond the scope of this book, but the practical application is that engaging participants’ emotions as well as their reason will help them remember information. That is why a surprising outcome to a simulation, an engaging game, or an amusing anecdote that underscores a point are powerful teaching devices. Stories Stick “Stories are easier to remember,” writes Daniel Pink in A Whole New Mind, “because, in many ways, stories are how we remember” (Pink, 2006, p. 101). In their superlative book, Influencer, about effecting lasting change, Patterson and colleagues (2008) underscore the power of stories this way: “A well-told narrative provides concrete and vivid detail rather than terse summaries and unclear conclusions. It changes people’s view of how the world works because it presents a plausible, touching, and memorable flow of cause and effect” (p. 59, emphasis added). Information is remembered more rapidly and in more detail when it is first presented as a story; it is also more persuasive (Martin & Power, 1982). The same information presented in story form is, surprisingly, considered also more believable than “just the facts.” Researchers presented MBA students with the same information in three different formats—verbal descriptions with facts and figures, charts and tables, and as a story involving a little old winemaker. “To the researchers’ surprise, when tested several weeks later, not only did those who heard the story recall more of the detail than the other two groups did (that

CH003.indd 133

6/17/10 3:26:00 PM

134


was predicted), but they also found the story more credible” (Patterson, Grenny, Maxfield, McMillan, & Switzler, 2008, p. 60). The finding was a surprise because most business people are acculturated to believe that hard-nosed analytical thinking with facts and data is what drives business. Obviously, good business decisions do require facts and data and sound analysis. But facts and data alone are not memorable, nor do they suffice to change behavior, which is what, at heart, most training initiatives seek to do. Writing in The Leader’s Guide to Storytelling, Stephen Denning explains: “At a time when corporate survival often requires transformational change, leadership involves inspiring people to act in unfamiliar and unwelcome ways. Mind numbing cascades of numbers or daze-inducing PowerPoint slides won’t achieve this goal. Even logical arguments for making the needed changes won’t do the trick. But effective storytelling often does” (Denning, 2005, p. 5). What’s the point? To make the message of training both compelling and memorable, encourage leaders and instructors to use stories to illustrate key lessons. The power of stories is the core concept behind 50Lessons, a company that specializes in capturing leaders’ stories in short, topical, memorable vignettes. Repetition Aids Retention The value of repetition to aid later recall has been known since Ebbinghaus’s classic studies in the late nineteenth century that differentiated long- and shortterm memory and showed how repetition could slow the rate of forgetting (Ebbinghaus, 1913). Revisiting the same topic at intervals enables learners to store information in a manner that facilitates long-term retrieval and makes the information more resistant to forgetting than a single presentation or non-spaced repetitions (Figure D3.9). What are the practical implications for corporate training and development? Returning to the same topic several times will increase its later recall, especially if these repetitions occur at intervals—for example, in both the Phase I preparation and again at various points of the Phase II instruction. In multi-day programs, starting each morning with a review of the prior day will help cement lessons learned. Charleen Allen, director of global learning and development at Baker Hughes, provides an example: “Reviewing each morning was handled differently than the usual review. We asked participants what one thing from the previous day resonated with them. By the time everyone shared, you had a review with the advantage of self-reflection included. By the third or fourth day of the program, these were deep, heart-felt revelations. I have never seen a better way to anchor the learning points to individuals emotionally” (Allen, 2008).

CH003.indd 134

6/17/10 3:26:00 PM


135

FIGURE D3.9. LEARNING AND FORGETTING WITH SPACED LEARNING

During Learning 100 80

Remembering on the Job

Spacing slows learning slightly But spacing makes learning more resistant to forgetting

60 40 20 0 During learning events, memory of information is strengthened

As soon as the learning event ends, forgetting begins Spaced learning Non-spaced, all-at-once learning

From Thalheimer, 2006. Copyright © 2006, Work-Learning Research. Used with permission.

For e-learning programs, shorter modules taken over a period of days or weeks are likely to produce more durable memory than those that can be completed in a single session. Finally, requiring participants to submit periodic progress reports post-training (see D4) and providing performance support tools (see D5) help ensure long-term, sustainable memories and skills (Figure D3.10). Jim Trinka of the Federal Aviation Administration, where lives depend on getting it right all the time, every time, explains: “The key to our training is repetition. There is no shortcut to repetition and practice. And so we repeat ourselves a lot and we practice over and over and over. Repetition is the key to competency for us” (personal communication, 2009). It’s also important to use multiple examples that illustrate the application of a principle, technique, or skill in different contexts. People are better able to abstract the relevant features of concepts when they are taught in multiple contexts rather than in just one (Gick & Holyoak, 1983). Using multiple examples helps learners generalize the knowledge and thus facilitates its transfer and application to novel situations (National Research Council, 2000).

CH003.indd 135

6/17/10 3:26:01 PM

136


FIGURE D3.10.

LEARNING AND FORGETTING WITH SPACED REPETITIONS ON THE JOB

During Learning 100 80

Remembering on the Job

Learning Curve

60 40 20

Forgetting Curve

0 Spaced learning on the job turns the forgetting curve into a learning-and-forgetting curve, improving memory. From Thalheimer, 2006. Copyright © 2006, Work-Learning Research. Used with permission.

Connections Count Usable knowledge is much more than mere lists of disconnected facts. Experts’ knowledge is connected to and organized around important, unifying concepts (National Research Council, 2000). Humans are better able to remember new information when it is linked to existing ideas, patterns, and knowledge already stored in long-term memory (Buzan & Buzan, 1993). Concepts that extend or enrich existing knowledge are much easier to recall—and therefore use—days, weeks, or months later than those that are free-standing or disconnected. The richer and more relevant the network of connections between new and existing knowledge, the easier it is to retrieve and use the new information appropriately. Elaborative rehearsal—thinking about or talking to others about an event (see Case in Point D3.3)—greatly strengthens its connections and the ability to recall it (Medina, 2008). A key difference between experts and novices is the richness of their mental schemas. “Schemas enable experts to categorize problems and consequently solve them. Novices, not possessing schemes, . . . have no alternative other than

CH003.indd 136

6/17/10 3:26:02 PM


137

Case in Point D3.3 Elaborative Rehearsal Glenn Hughes is director of global learning for KLA-Tencor, the world’s leading supplier of process control and yield management solutions for the semiconductor and related microelectronics industries. When we asked Glenn about what learning strategies have the highest payoff, he thought immediately of elaborative rehearsal. “I realized that that’s probably my preferred learning tool. People have often remarked at my memory and how I can remember the names of Japanese temples I went to twenty years ago. I can remember many movies, line for line and script for script. I remember courses, events, and conversations in much the same way, and I could never really understand why until I read about elaborative rehearsal. And then I realized that the first thing I do when I walk out of a learning experience is I go and explain it to someone. “I literally walk out, I come home and I tell my wife, ‘Hey! I was just at this class today, here’s what I learned.’ And bang, bang, bang, I go through it. That weekend, when I call my dad I tell him, ‘This is really an interesting class,’ and then I tell him about slide:ology or the 6Ds or whatever I was just learning about. I’ll come into work and I’ll get out my pen and I’ll diagram it on the board and tell my colleagues what it was I just learned. “And people challenge me and ask me questions; they force me to be clear about my thinking. They bring their own applications. And of course, the whole while I’m telling them about it, I’m tapping it into experiences that the two of us have in common. So I’ll say, ‘Hey, you know, remember three months ago when we were talking to this leader and he said . . . Well, I think I might have found the answer.’ “That’s the sort of web I weave within, say, forty-eight hours of any critical learning experience. I weave this web that involves a lot of other people and even different media and tools, how it relates one idea to another. And what’s interesting is from that network of people, the number of application points multiplies exponentially, because they all see ways that it can be used too. And so, then it becomes very easy to take something or apply it and have some reinforcement in the environment. “So that’s probably the most important thing I do when I come out of a learning experience. It is the action I take—the fact that I go through this whole elaborative rehearsal process.”

to engage in general search techniques” (Cooper, 1990). Thus, helping participants see the relationships between new topics and what they already know strengthens the staying power of what they have just learned and improves their ability to apply it. But it takes time and effort for participants to forge their own connections between the new material, what they already know, and how they might use it. We recommend having the facilitator encourage people to “connect the dots.” A useful exercise is to stop periodically and ask participants to write down their insights and ideas for application in a learning journal, action planning sheet, or other similar device. An example is shown in Figure D3.11.

CH003.indd 137

6/17/10 3:26:02 PM

138


FIGURE D3.11. AN EXAMPLE OF A WORKSHEET TO HELP PARTICIPANTS LINK NEW WITH EXISTING KNOWLEDGE

Connections and Applications Worksheet A key concept from this session is:

How it reinforces things I already knew:

How I can use it in my work:

It causes me to reconsider:

Related program concepts include:

What the benefits will be:

Copyright © 2010 Fort Hill Company. Used with permission.

The act of writing down ideas for application (rather than just being asked to think casually about them) is important; it requires deeper thought and begins the process of personal ownership. Since each person’s “mind map” is unique, the connections that a person creates himself or herself are the most powerful. It is nevertheless valuable to ask a few participants to share their insights and ideas for application. Publicly stating a plan deepens the owner’s commitment. It also gives other participants additional ideas for their own application and, as a side benefit, enriches the instructor’s reservoir of relevant examples. Learning Requires Reflection Forum Corporation convened a panel of leading corporate educators and executive coaches to formulate principles to guide corporate education. In their report, Principles of Workplace Learning, the panel concluded that a best practice was alternating periods of action and reflection. The problem is that “when we are faced with little time and an urgent need for results, we sacrifice reflection for action. But reflection is the motor that propels the cycle—without it, learning slows or stops completely” (Atkinson & Davis, 2003, p. 25).

CH003.indd 138

6/17/10 3:26:03 PM


139

Setting aside time for reflection, however, runs counter to the pressure to reduce training time and also to an instructor’s natural zeal to transmit as much information as possible; the temptation is to fill every waking moment of a program with activity and instruction. This “sausage stuffer” approach to corporate education, by eliminating time for reflection and practice, is counterproductive. Reflection is essential to build the mental schemata that differentiate experts from novices. Prompting Reflection is the motor— guided reflection during learning (Phase II) as well without it, learning stops. as transfer (Phase III) is important to the elaborative rehearsal that makes learning sticky and useful. Leo Burke, director of executive education for the University of Notre Dame, believes that the emphasis on values and on reflection are key to the success of Notre Dame’s innovative Integral Leadership program (see Case in Point D3.4).

Case in Point D3.4 Reflection in Leadership Education The University of Notre Dame’s Integral Leadership program is a comprehensive approach to leadership development that incorporates individual and collective perspectives as well as internal and external points of view. According to Leo Burke (personal communication, 2005), associate dean and director of executive education, its success is a result of “the holistic nature of the program. In particular, we have built in a more substantive reflective dimension than one would normally find in leadership programs.” Prior to attending the five-day intense, residential portion of the program, participants submit a strategic business issue, threat, problem, or opportunity and complete a customized 360-degree assessment. During the week-long session, participants use a structured problem-solving and journaling process that requires them to explore their values, their organization’s values, and societal or more market-related factors, and then look for areas of alignment and misalignment. “To really get a grasp of that, participants need time to reflect in this exploration process, so we build that in each day,” says Burke. “In addition, there are coaching sessions that also give people pause for reflection as well. “We relate all of this back to their business issue. So let’s not talk about changed behavior in the abstract, let’s talk about it with relation to the issue that you have selected. What are you going to do differently with regard to this issue that you were not going to do prior to coming into this program? And that becomes the litmus test for whether people are really serious about change. One of the key themes we emphasize is that you have a choice in terms of your response to the situation; it is your responsibility.” Is it working? In follow-up telephone interviews, 95 percent of the participants were extremely satisfied and many rated it as the best leadership program they had ever attended. Most important, they were able to cite specific examples of how they had used what they had learned to make significant improvements to their businesses and personal lives.

CH003.indd 139

6/17/10 3:26:03 PM

140


The most effective programs, in our experience, are those that stop after each key learning segment and give participants time to reflect and make connections. Agilent considers time for reflection one of the most important parts of every exercise in their Managing at Agilent course and provides participants with four questions to help guide the process (Girone & Cage, 2009): • • • •

I hadn’t realized before that . . . I hadn’t considered before that . . . I hadn’t thought before that . . . I hadn’t noticed before that . . .

Other useful reflection questions include: “How could you use what we have just covered to improve performance in your area of responsibility?” and “What specific actions could you take to put this learning to use?” These not only help participants relate the learning to their specific circumstances, but also reinforce the importance and expectation for practical application. Learning professionals need to protect time for reflection against the urge to stuff in more content. As Fred Harburg, writing about corporate training and development, put it: “If there is not time for reflection, there is almost no chance for improvement” (2004, p. 21).

Prepare for Transfer The final aspect of delivering for application is to structure the learning experience in such a way that the participants are prepared and looking forward to using what they learned. We have already discussed the importance of setting expectations for transfer as part of Phase I and communicating the plan and timeline for assessing achievement (Phase IV). Preparing for transfer also includes goal setting, action planning, and getting ready for re-entry. Goal Setting If the program has been successful in motivating employees to want to learn, and the delivery has taught them how, then the third precondition for generating results is having opportunities to practice their new knowledge and skills while they are still fresh. An important first step in that process is to be sure that each participant has specific goals for learning transfer and application. Many programs already include a goal-setting exercise in which participants are asked to set specific objectives to apply what they have learned. Ideally, the process should begin in Phase I. For example, building on Brinkerhoff ’s concept

CH003.indd 140

6/17/10 3:26:04 PM


141

of “learner intentionality” (Brinkerhoff & Apking, 2001, p. 88), Plastipak’s High Impact Learning Job Aid for Supervisors provides step-by-step guidance for preparing the learner that includes asking the associates’ thoughts about the business goals, key results, on-the-job behaviors, and key skills they need the most; the outcome is an agreement between the manager and associate on the learning focus and a personal learning “roadmap.” Associates are then expected to bring their learning roadmaps to the training along with their other required preparatory work. Similarly, Phase I for Pfizer’s Advance Transition program includes having each participant meet with her or his manager to identify top priorities. Chubb facilitates a meeting between participants and managers (see Case in Point D5.2). In our experience, however, such programs are exceptions. Goal setting is usually left until the end of Phase II and is frequently weak in otherwise superlative programs, resulting in poorly articulated, half-hearted, often trivial goals that no one takes seriously. Such inadequate attention to setting objectives for transfer and application can seriously compromise a program’s full potential. Participants’ learning objectives need to be treated with the same seriousness as their business objectives, because they are, in effect, business objectives, the whole rationale of training being to support the greater business goals. Goals Learning objectives need to for using learning should be planned, be treated with the same executed, tracked, and rewarded with the seriousness as business same degree of attention as other perforobjectives. mance metrics. Unfortunately, in most organizations, learning transfer objectives are still seen as distinct from business objectives and of lesser importance. The problem usually begins with their creation. The most common problems with writing learning transfer objectives and recommended solutions are presented in Table D3.3. Each is discussed below. Problem 1: Not Enough Time For obvious reasons, final goal setting cannot be done until all the main topics have been covered. Accordingly, the goal-setting and application-planning exercise is usually scheduled at the very end of the course. The first cause of poor transfer and application goals is failure to schedule enough time for this segment. It takes significant time and effort to develop a good set of objectives and an implementation plan. This is as true for learning transfer objectives as it is for other business objectives. The short time scheduled for goal setting and application planning is often further compromised. Educational programs tend to fall behind schedule;

CH003.indd 141

6/17/10 3:26:04 PM

142


TABLE D3.3. COMMON PROBLEMS WITH GOAL SETTING AND RECOMMENDED SOLUTIONS Problem

Solutions

1. Not enough time

1.1 1.2 1.3 1.4

2. No quality control

2.1 Review the goals 2.2 Send the goals to the manager

3. Writing strong goals is hard

3.1 Provide a structure 3.2 Provide examples 3.3 Have participants share their goals

Start earlier Allocate enough time to begin with Stay on schedule Provide preset goals for application

facilitators are, understandably, reluctant to cut off a particularly fruitful or interesting discussion just to stay on schedule. When subject-matter experts are the presenters, they find it hard to resist the urge to elaborate (something we struggle with in our own workshops) and often go over their allotted time. As a result, some of the time set aside for goal setting and action planning is used to cover the last few topics. In addition, some participants inevitably leave early, especially in open-enrollment programs. They may miss the planning exercise entirely, and their parting farewells disrupt the group, further consuming the limited time available. The time constraints imposed on corporate training and development are real and likely to intensify; managers are increasingly asking for shorter courses to reduce employees’ time out of the office. Under such pressures, the temptation is to eliminate goal setting and action planning as part of the curriculum and expect participants to complete them later. Experience proves this to be a poor tradeoff. Every time we have acceded to requests to allow participants to set their objectives after the course, the results have been unsatisfactory; fewer than half ever do, despite repeated requests. Participants’ day-to-day work simply overwhelms even the best intentions. To receive the maximum benefit from training and development experiences, participants should not be allowed to leave the formal coursework without clear and compelling goals for application. Anything less impedes transfer and increases the amount of “learning scrap.” Solution 1.1: Start Earlier Thinking about learning transfer objectives is most

effective when it begins in Phase I because it fosters learning intentionality, which has multiple benefits in terms of motivation, commitment, and value derived (Brinkerhoff & Apking, 2001). Participants should be encouraged to meet with their

CH003.indd 142

6/17/10 3:26:05 PM


143

managers to discuss what business goals the learning will advance; what behaviors (performance of tasks) need to be strengthened, changed, added, or deleted; and why these are important to both individual performance and business success. As we will discuss in greater detail in D5, both managers and participants should be provided with guidelines for that discussion and provided with easy-to-use forms and processes. Suggested outlines, worksheets, and examples are available in Getting Your Money’s Worth from Training and Development (Jefferson, Pollock, & Wick, 2009); a learning management system (LMS)–compatible version is also available. Start the goal-setting process earlier in the course itself. Build in short, specific times for reflection after each major topic or segment. Encourage learners to think about what they have just learned and write down their ideas for how they might apply their new knowledge and skills to improve their performance and that of their unit. The time spent in such short, focused periods of reflection is recaptured by making the final objective-setting process more efficient; participants can review the objectives they set in Phase I and the ideas they captured as the learning progressed, then select the most valuable to define as the specific targets of their post-course follow-through. Another way to reduce the class time required for setting goals is to have participants create a first draft as a homework assignment on the night before the last session. That way, class time can be devoted to discussion and refinement rather than to initial conceptualization. Solution 1.2: Allocate Enough Time to Begin With The second part of a solution

is to be sure that enough time is allocated for goal setting and action planning to begin with. How much is enough? That depends on the nature, length, and complexity of the course, whether the goals are individual or team-based, and how much preliminary work (reflecting on potential application) has been done. For an in-depth course, an hour is needed to generate high-quality goals and initial action plans, more if they are team-based. The best test for whether enough time has been allotted is for the designers to try to complete the actual exercise to see how long it takes and the quality of the work they produce. Providing too little time sends the wrong signal. It undermines the importance of goal setting, since it suggests that even the instructional designers did not take it seriously or really expect it to be done. Solution 1.3: Stay on Time Staying on time is hard—we know—but it is vital.

Having enough time set aside in the agenda is of no value unless it is respected. Allowing other sessions to run over so that goal setting and action planning are compromised sends the wrong message: it implies that content is more important than application, when quite the opposite is true in terms of business results.

CH003.indd 143

6/17/10 3:26:05 PM

144


If portions of the course chronically run overtime, it suggests a flaw in the fundamental design: too little time was allocated originally for the material or exercises. The solution is to consciously revise the program plan, not to skip goal setting. The design team needs to reexamine and adjust the number of topics and the time allotments as part of a continuous improvement effort. It also may be useful to appoint an impartial timekeeper to remind the facilitators of approaching stop times. Solution 1.4: Preset Goals for Application In theory, participants should have

greater ownership and commitment to goals they set themselves, especially for leadership or other personal development programs. But if the program has a clear and relatively narrow focus, or if management has specific expectations for post-course actions and results, then it may be both more efficient and more efficacious to simply let participants know what those are rather than having them try to re-create what the designers had in mind. That’s what Charleen Allen, director of global learning and development at Baker Hughes, did for the Sales Organization Leadership Development (SOLD) program. Because there were very specific things participants needed to do to drive a culture of effective account ownership following training, Charleen developed three goals that were common for all participants rather than having them create their own. “This helped us drive a common culture and common skill sets based on the course objectives,” she explained (Allen, 2008). A web-based learning transfer management system (Friday5s) was used to ensure that participants followed through on these objectives. As a result, Charleen and her team were able to document positive outcomes in the areas of the greatest importance to the business. Research has shown that both assigned and participant-specified goals are efficacious in helping maintain new behaviors post-training (Wexley & Baldwin, 1986). Whichever approach is used, the most important thing is to make expectations and timelines clear. Problem 2: No Quality Control Even if enough time is allocated to goal setting and action planning, it isn’t worth much unless the process generates meaningful objectives (1) that are related to the business needs the course was designed to address, and (2) that the participants are truly committed to accomplishing. It is surprising, given the importance of setting strong goals, that so few programs include any kind of quality-assurance process. Each year, we process more than fifty thousand objectives from a wide range of corporate training and development programs. We are surprised by how many

CH003.indd 144

6/17/10 3:26:06 PM


145

are weak or bear no apparent relationship to the business or the program. A few real examples will suffice: • “ Lose ten pounds ” (after a three - day, $ 4,000 leadership program at Gettysburg). • “Read a book on management” (following a multi-day leadership development program for high-potential candidates). • “Be a better listener.” Such objectives are unlikely to produce meaningful transfer or business impact. If they were to become known to management as representative of the output of training, then the program’s efficacy (and likely its future) would be called into question. Either the program failed to help participants understand how to apply the material to the work of the firm or it allowed them to make a mockery of the objective-setting exercise. Whatever the case may be, low-quality objectives reflect badly on the program, the instructors, and the participants; they represent a serious breakdown in the learning process. Solution 2.1: Review the Goals One of the first and simplest steps to improve the quality of post-course objectives is for learning professionals to actually review them (or at least a sample of them). Surprisingly, this is not common. Indeed, most learning organizations have no idea what kinds of objectives participants are setting, nor, for that manner, do their managers (see discussion below). Imagine how absurd it would sound if the same process—or, rather, lack of process—were applied to other business functions: if the sales department had no idea what an individual salesperson’s targets were, or if the research department did not know what its scientists were working toward. No business enterprise could succeed if it allowed its managers to set their own business goals and then file them away without ever reviewing them with anyone. But that is precisely what happens with most learning and development objectives. Small wonder most participants do not take the process seriously. Copies of participants’ learning transfer objectives can be obtained in any number of ways. Goal forms can be collected, copied, and returned before the end of the program. Carbonless NCR forms could be used for writing goals and a copy left with the instructor. If privacy is an issue, names could be omitted from the copies, since the purpose is to analyze the overall goal quality and alignment, rather than specific individuals’ objectives. We personally favor electronic learning transfer management systems because once the goals are in electronic form they can be utilized for communication (see discussion below) as well as analysis.

CH003.indd 145

6/17/10 3:26:06 PM

146


However they are obtained, the participants’ goals (or for very large programs, a sample of the goals) should be reviewed by learning professionals to determine their quality and whether they are in alignment with the business objectives of the program. Basarab (in press) recommends establishing criteria and a success rate for acceptable goals in advance and then reviewing the goals from each iteration of the program to ensure that they meet the quality standards. The core concept is that intention precedes adoption, which is prerequisite to results (Figure D3.12). If you don’t start with the right intention, then the odds of getting the desired results are low. If the If you don’t start with the right percent of acceptable goals falls below the preintention, the odds of getting determined success rate, then the program needs results are low. to be revisited and adjusted since it is failing to produce the desired intent. Electronic learning transfer management systems (which are discussed in more detail in D4) allow learning professionals to efficiently review both the quality and the distribution of a very large number of goals (Figure D3.13). Knowing the distribution of participants’ objectives allows the learning organization to adjust emphasis as necessary to ensure objectives are congruent with the most important business needs. A striking example of why this is important is a client who had planned and conducted a several-day program on change management. When the learning team reviewed the participants’ objectives, however, they discovered that fewer than one in ten were actually concerned with change management. Clearly there was a disconnect: either the program’s message was not getting through or the program did not address the most pressing issues perceived by participants. Adjustments were made to the program’s focus and presentation. Had learning and development not reviewed the participants’ transfer objectives, they might have continued offering the same program, oblivious to the misalignment.

FIGURE D3.12. THE PREDICTIVE EVALUATION MODEL, WHICH STRESSES THE IMPORTANCE OF GOALS (INTENT) New skills, knowledge, and beliefs from training

INTENTION (goals)

ADOPTION (actions, transfer)

IMPACT on performance

Copyright © 2010 V.A.L.E. Consulting. Used with permission.

CH003.indd 146

6/17/10 3:26:06 PM


147

FIGURE D3.13. ELECTRONIC LEARNING TRANSFER MANAGEMENT SYSTEMS FACILITATE REVIEWING THE DISTRIBUTION AND QUALITY OF A LARGE NUMBER OF GOALS

Distribution of 1,455 Participants’ Managing@Agilent Goals Situational Leadership 29%

Improving Team Performance 19%

Other 1% Performance Planning and Development 9% Giving and Receiving Feedback 13%

Business Knowledge 4%

Setting Clear Priorities and Expectations 25%

From Roche, Wick, and Stewart, 2005. Used with permission.

Solution 2.2: Send the Goals to Managers The second solution to making sure that learning transfer objectives are treated more seriously is to send a copy of each participant’s goals to his or her manager. Just as business objectives must be discussed with one’s boss and mutually agreed on, so should learning objectives. Yet discussions of learning objectives between supervisors and their direct reports happen far less frequently than either management or learning directors suppose, despite clear evidence of the importance and benefit of such discussions (Brinkerhoff & Montesino, 1995; Gregoire, Propp, & Poertner, 1998). The magnitude of the problem was evident in a study we did in collaboration with a leading technology company. The course represented a significant

CH003.indd 147

6/17/10 3:26:07 PM

148


investment for the department and the participants. The sponsoring department was billed $2,500 per participant, each of whom spent a full five days out of the office. Participants were encouraged to discuss their objectives with their managers. Yet when we polled the participants’ managers three months after the program, fewer than half reported that they knew what their direct reports’ post-program objectives were (Figure D3.14). It is hard to see how managers can provide the needed support, coaching, reinforcement, and opportunities to use new knowledge and skills when they have no idea what their direct reports are trying to accomplish. To address the issue, the client instituted an electronic learning transfer management system that automatically sent each manager a copy of his or her direct reports’ goals. Thereafter, 100 percent of the managers reported that they were aware of their direct reports’ objectives. More importantly, the number of discussions between managers and their reports about the program more than doubled, the amount of effort spent on learning transfer increased, and the managers’ ratings of post-program improvement also increased. The other salutary effect of sending managers a copy of their direct reports’ goals is that the quality of the goals improves. If participants know that their objectives will be sent to their managers, they put a lot more quality thought into their objectives than when they know the objectives are just going to be put on the shelf. The first time that Hewlett-Packard introduced participants to the idea that their goals would be sent to their managers, there was a stunned silence. After an FIGURE D3.14. MANAGERS’ AWARENESS OF THEIR DIRECT REPORTS’ LEARNING TRANSFER OBJECTIVES BEFORE AND AFTER INSTITUTING A TRANSFER MANAGEMENT SYSTEM

% of Managers

100% 80% 60% 40% 20% 0% Before

CH003.indd 148

After

6/17/10 3:26:08 PM


149

appropriate pause, the learning director quipped, “I guess that changes the way you will write them, doesn’t it?” Based on our experiences of the past ten years, we now recommend to all our clients that they send a copy of each participant’s goals to his or her manager and that this be done automatically, rather than left to individual initiative. We have found that exhorting participants to share their goals with their managers is not as effective as simply sending them. Obviously, this is more easily accomplished with an electronic transfer management system, but it can also be done by making an extra copy of paper-based forms. In the rare instances in which a developmental goal is truly personal or is related to an issue involving the manager, it may make sense to allow individuals to designate one (but not all) goals as “private” (non-shared). Because managers can provide (or deny) opportunities to put new knowledge to work, they need to be informed not only of the overall business objectives of the training, but also their direct reports’ personal objectives. Absent such knowledge, managers are less able to create opportunities and coach, with the result that much of the knowledge will be lost and the program may fail to deliver the desired results. If you send managers their direct reports’ goals, it is helpful to also provide them guidance on how to evaluate the goals and how to coach for optimal performance. A suggested rubric and process are provided in Getting Your Money’s Worth from Training and Development for Managers (Jefferson, Pollock, & Wick, 2009, p. 37). Problem 3: Writing Strong Goals Is Hard A third reason that learning transfer and application goals are often weak is that writing good objectives is more difficult than generally supposed. Course designers usually assume that because most participants have experience with management by objectives, they know how to write good objectives. Our analysis indicates otherwise; indeed, some of the weakest objectives are from the most senior managers. Going over the SMART acronym in class does not seem to make a great deal of difference. A number of the programs in our database included a discussion or handout on SMART goals (specific, measurable, achievable, relevant, and time-bound—or some variation thereof). There is no evidence that the goals from these groups are consistently better than those in which SMART was not discussed. It is not clear whether the root cause is that participants have heard about SMART so often that they find it mind-numbing and tune out, or whether not enough time or guidance was allocated to assess the goals against the SMART criteria.

CH003.indd 149

6/17/10 3:26:09 PM

150


Whatever the cause, a high percentage of the goals set following learning and development programs are not specific, quantifiable, business-related, or timebound. Such goals do not reflect well on the course or the participants and are unlikely to lead to meaningful effort and results. Given the importance of “intent,” improving the quality of the objectives set in corporate education programs will improve learning transfer, business impact, and return on investment. Solution 3.1: Provide a Structure In our experience, giving participants a template to follow has been much more effective than presenting SMART or other models for goal setting. The template should guide the participants to specify what they will do, by when, why, and how they will know. (See Exhibit D3.2.) Such a template is a simple performance support tool that enables your participants to craft goals that meet the SMART criteria without having to spend time on another ineffective discussion of the model. This is similar to the structured essay concept in academic education. Traditional essay questions present students with a topic to discuss but provide little or no guidance on how to do so. Each student approaches the topic in a different way. As a result, it is difficult for the instructor to know whether a student failed to discuss a specific topic because he thought it was unimportant or because he knew nothing about it and so avoided mentioning it. The solution is to give students an outline of the topics to discuss, thus structuring the essay for them and ensuring greater uniformity and completeness. Providing a template for learning transfer goals achieves the same benefit; it gives form to the participants’ objectives and ensures that each contains the key elements of a strong and actionable objective: a specific target, time frame, rationale, and measures of success. Solution 3.2: Provide Examples Providing well-thought-out examples of suitable

objectives is another simple but effective means to improve the quality of learning transfer goals. It is surprising that so many programs fail to do so. Perhaps it is simply an oversight. Perhaps it is because writing really good examples is difficult. Perhaps it is because too many existing programs skipped the discipline of defining expected business outcomes in the first place. Whatever the reason, failure to provide exemplars adversely affects outcomes. If the desired business outcomes are truly well understood, then it is relatively easy to write examples of well-articulated goals. A good test for business alignment is the ease with which line and learning leaders can agree on the criteria for acceptable participant goals. There is, of course, some danger in presenting examples. Some participants may follow them too slavishly and not “own” the goal in the same way that they do when they generate it for themselves. We feel, however, that the benefits of

CH003.indd 150

6/17/10 3:26:10 PM


151

Exhibit D3.2 An Example of a Structured Goal-Setting Form GOALS FOR APPLYING LEARNING Instructions: Select one to no more than three things you want to accomplish using the new knowledge and skills you acquired in the program. Goal 1 In the next _____________ (weeks/months), I will: ___________________________________ _____________________________________________. [Be specific about what you intend to accomplish.] The benefit to me and the organization will be: ____________________________________ _______________________________________________________________________________. [Explain the value.] Evidence of my progress and success will include_____________________________________ ________________________________________________________________________________ _________________________. [Define the indicators of progress and measures of success.] Goal 2 In the next _____________ (weeks/months), I will: ___________________________________ ______________________________________________. [Be specific about what you intend to accomplish.] The benefit to me and the organization will be: _____________________________________ _______________________________________________________________________________. [Explain the value.] Evidence of my progress and success will include: ___________________________________ ________________________________________________________________________________ _________________________. [Define the indicators of progress and measures of success.]

CH003.indd 151

6/17/10 3:26:10 PM

152


providing examples and of being clear about expectations far outweigh this risk. Moreover, if participants know that they will be held accountable for following through and results on the objectives they submit, they will be less inclined to simply copy an example—especially one with an appropriate degree of stretch. Solution 3.3: Have People Share Their Goals Providing time for participants to

work in pairs or learning teams and share their goals has a number of beneficial results. First, they can help one another improve their objectives. In formulating goals, it is sometimes useful just to be able to “think out loud” with a colleague. Second, sharing your goals with another person who has attended the same program is a good check on whether they are clear and understandable. A useful exercise is to have each participant describe what his or her situation will be like in three to six months. How will she have used what she learned? What will she have accomplished? What will be different and better? Such questions help participants create a vision of a new and better reality. The importance of envisioning improved performance is now widely recognized in sports psychology (Loehr, 2007). Athletes are encouraged to picture themselves making the perfect swing, executing a flawless maneuver, or accepting the gold medal. The ability to envision great performance increases the probability of realizing it. Telling a positive story also has positive effects on personal and business success (Dweck, 2006; Loehr, 2007). Having participants envision their own improved performance by applying what they have learned is a first step toward achieving it. Finally, having participants share their objectives improves the probability that they will be achieved. “Social psychologists learned long ago that if you make a commitment and then share it with friends, you’re far more likely to follow through than if you simply make your commitment to yourself ” (Patterson, Grenny, Maxfield, McMillan, & Switzler, 2008, p. 152). Encourage learning partners to test each other’s commitment to achieving their objectives. Is what they have written really something they want to accomplish because they can see the value and payoff, or are they just going through the motions? Little will be achieved without commitment and a vision of a better future. If the program is well designed and executed, participants should be able to define objectives about which they can honestly say: “This is something I really want to accomplish.”

Plan for Transfer Reiterating the expectation for transfer, explaining the process, and providing time and a structure for developing action plans all contribute to effective learning transfer. How well the facilitator explains and manages the process impacts the willingness of participants to follow through.

CH003.indd 152

6/17/10 3:26:10 PM


153

Good Facilitation We were surprised at how large an impact the facilitator has on whether participants follow through on their learning transfer goals. The evidence comes from analysis of a large-scale learning intervention at a global technology company. We noticed that some classes had a much higher rate of participation on post-training follow-through than others. To try to understand the extent to which the facilitator influenced post-course application, we analyzed the number of participants who completed their first post-course update by the instructor who led the session. (We reasoned that the facilitators had the most influence on whether or not the participants logged their first post-course update; thereafter, other factors come into play.) We included only instructors who had taught at least four classes involving a total of one hundred or more participants. The results were surprising (Figure D3.15). There was as much as a two-fold difference in post-course follow-up between instructors who, theoretically, had taught the same program, with the same content, to the same level employees.

FIGURE D3.15. IMPACT OF DIFFERENT FACILITATORS ON POST-COURSE APPLICATION

First Post-Training Update Percent 87%

A

85%

B

83%

INSTRUCTOR

C

81%

D

78%

E

74%

F

72%

G

66%

H

58%

I 42%

J 0%

CH003.indd 153

20%

40%

60%

80%

100%

6/17/10 3:26:11 PM

154


Clearly, some instructors embraced the importance of post-course learning transfer and presented it more persuasively than others. They took the exercise seriously and spoke to the importance of follow-through with passion and conviction. Others were less concerned about learning transfer or less knowledgeable about the process and so gave it short shrift and lukewarm endorsement. Not surprisingly, participants in those classes made much less post-course effort. The take-home lesson for learning leaders and program designers is that the way in which facilitators present the need for learning transfer and the process for its planning and execution has a significant impact on post-course actions by participants, and subsequently, on business results. It is advisable, therefore, to be sure that all instructors embrace the importance of learning transfer and that they are prepared to present the material convincingly and compellingly. Action Planning The acronym for goals without plans is GWOP. To get the greatest value from training and development, participants should leave a course with clear and committed objectives. But they need more than GWOP to succeed. It is therefore worth investing some time to help participants think through what they will need to do to accomplish their objectives. Our bias is to keep the action planning high-level and process-oriented rather than excessively detailed. In part, this is a matter of time. Most programs fail to budget sufficient time for setting goals, let alone for detailed action planning. We have reviewed proHelp participants think through gram agendas that called for participants to set what they will need. goals and complete a complex multi-page action planner in twenty minutes—an impossible assignment. We feel it is more valuable for participants to develop a clear view of the first action they need to take than to rush to superficially fill in a detailed planner. We are also convinced that many of the transfer objectives, especially for management and leadership programs, do not lend themselves to overly specific project plans. Since they often involve interpersonal skills, opportunities must be seized when they arise; the plans are more directional than specific. That having been said, at the very least, participants should leave Phase II of the program with concrete plans for how they will communicate their intentions to their managers and associates (see “Communicate What Happened,” following). A fun, efficient, and productive exercise for generating ideas for application is an adaptation of what Marshall Goldsmith calls “feedforward” (Goldsmith, 2002). The goal of the exercise is to focus on future actions and to garner ideas from the collective wisdom of the group.

CH003.indd 154

6/17/10 3:26:12 PM


155

Have all the participants stand and pair off. Have one member of each pair succinctly describe one of his objectives, then ask for feedforward—one or two suggestions for future actions that might help him achieve a positive change. The focus should be on the future (feedforward) as opposed to the past (feedback). Even if participants have worked together in the past, they should avoid giving feedback and provide only ideas and suggestions for the future. Those asking for feedforward should listen attentively and take notes, but they should not comment on the suggestions in any way, even to make positive judgmental statements. They should simply thank their colleague for the ideas and then trade roles. As soon as a pair has exchanged goals and ideas, each person should find a new partner and repeat the process. The process of giving and receiving feedforward should only take about two minutes. In fifteen minutes, participants can receive five to eight ideas of specific things they can do to apply their learning and improve their performance. Every time we use this exercise, the energy level in the room goes up dramatically; participants tell us that they derive real value from the exchange and that they wish we had allotted more time for it.

Communicate What Happened Participants have an exceptional, but short-lived, communication opportunity when they return to work. Their manager, assistants, co-workers, and direct reports are aware that they have participated in a learning and development program. Some contributed 360-degree feedback; some had to take on extra work to “cover” for the participant. All are interested in what happened. The first question from most will be something like, “So what did you learn?” or “What happened?” Such an opening provides the participant with an exceptional opportunity to signal his or her intentions, recruit allies, share a key insight, and thank those who provided feedback or carried on the work of the department. As in all communication, the message has to be planned and rehearsed for maximum impact. We strongly encourage giving participants time and help preparing to answer these predictable questions. Well-thought-out answers reinforce the learning (elaborative rehearsal), underscore the value of the program, and begin the process of application. Weak, muddled, or otherwise ill-prepared answers will raise questions about the value of attendance. Therefore, it is in the best interest of both the attendees and the learning organization to give participants a few minutes to prepare elevator speeches. The concept of elevator speech was developed years ago at the Xerox Corporation. The idea was that if you stepped into an elevator with one of the company’s senior managers and she said to you, “So tell me about this idea of yours,” you

CH003.indd 155

6/17/10 3:26:12 PM

156


Exhibit D3.3 A Job Aid for Creating an Elevator Speech Instructions : Answer the four questions below. Practice your elevator speech with a partner. 1. The most important/striking/insightful/valuable [pick one] thing I learned was: ____________________________________________________________________________ 2. I am going to capitalize on this learning by: _____________________________________ ____________________________________________________________________________ 3. The benefit for our organization will be: _________________________________________ ____________________________________________________________________________ 4. I’ll need the following help to make this happen: ________________________________ ____________________________________________________________________________

needed to be able to articulate the key concepts and benefits in the time it took to ride between the first and fourth floors. If you could, you greatly improved the idea’s chance for success. If you could not, you squandered a golden opportunity to advance both your concept and your career. Provide participants with an outline (see Exhibit D3.3) and time to write out and practice a one- to two-minute reply to the “What happened?” question. Encourage them to think through the key points they want to make and what will help them achieve their goals. Ideally, have them practice their elevator speeches in pairs and give each other feedback. The act of stating what they will do and its value will reinforce their commitment and improve the probability of learning transfer. Providing a few minutes to craft and practice the message will pay ample rewards for the participants, the program, and the company.

Check the Process Effective practice of D3 facilitates application of new skills and knowledge by motivating participants to learn, making the relevance and utility of the program clear, providing know-how and practice, making the learning memorable, and preparing participants to transfer their learning back to their jobs. The extent to which learning delivery achieves these objectives determines the extent to which the learning contributes to business results.

CH003.indd 156

6/17/10 3:26:13 PM


157

But how do you know whether the process is working? We recommend building in-process checks along the way, rather than simply waiting for the final evaluation. This is analogous to using in-process checks in manufacturing or business systems to ensure that critical sub-processes are working as expected. Three checks are of particular importance for learning programs: perceptions of relevance, goal quality, and first actions. We made the case for the importance of relevance and utility to adult learners at the beginning of this chapter. Given its impact on motivation to learn and apply, it is important to know whether participants perceive the program’s utility. At the end of Phase II, ask them to rate the degree to which they are motivated to use what they learned, understand how to apply it, and see its relevance (Exhibit D3.4). The answers to these questions are much more useful indicators than typical reaction data, which are often little more than ratings of enjoyment. It really does not matter whether participants “enjoyed” the program; what matters is whether they are convinced that what they learned is useful, that they are motivated to use the course content in their work, and that they know how and when to apply new capabilities. If the utility ratings are low, steps must be taken to identify and address the source of the process failure. If the utility ratings are high, that’s good news—it suggests that the process is working as planned to this point. Unfortunately, it does not itself predict high levels of learning transfer because of myriad other factors in the post-course environment that influence transfer (see D4: Drive Learning Transfer). That is why driving learning transfer is a discipline unto itself in converting learning to business results.

Exhibit D3.4 Suggested Questions for Gauging Participants’ Perceptions of a Program’s Utility Rate the extent to which you agree or disagree with the following statements: 1. The learning was directly relevant to my job. Strongly disagree

Disagree

Neutral

Agree

Strongly agree

Neutral

Agree

Strongly agree

2. I feel well-prepared to use it. Strongly disagree

Disagree

3. Using what I learned will improve my performance. Strongly disagree

Disagree

Neutral

Agree

Strongly agree

Agree

Strongly agree

4. I am motivated to put the learning to work. Strongly disagree

CH003.indd 157

Disagree

Neutral

6/17/10 3:26:13 PM

158


The second key in-process check for D3 is a periodic review of the quality and focus of participants’ goals for application to be certain they are in alignment with the program’s objectives. That’s because it is essential to get the intent right so that effective action and application follow. Finally, it is good practice to check the rate of first post-course action. As with the other in-process checks, the rate has more negative than positive predictive value. That is, a high rate of taking immediate post-course action suggests, but does not guarantee, a high rate of transfer and ultimately results. On the other hand, if participants do not even take the first step, then that strongly suggests that there is a problem with the course, facilitation, or transfer environment that will preclude realization of results. Low rates of follow-through on first action steps should be investigated for the root causes and the issues addressed. If you are using an electronic learning transfer management system, the first response rate will be tracked automatically. If you are still using a manual system, you can obtain a sense of the extent of first actions by, for example, polling participants a week or ten days after the course work with a simple one- or two-question survey, such as: “Have you met with your manager to discuss your application goals?” or “Have you completed the first action you planned during the program?” The survey itself will serve as a useful reminder of the need for follow through (see D4); the response rate to the survey will be indicative of the amount of post-course engagement.

Summary In this chapter, we have emphasized the importance of designing learning delivery to support application; that is, ensuring that every element of the learning experience is selected and constructed in a way that supports the ultimate objective: improved performance through participants’ use of their new knowledge and skills. We stressed the importance of perceived relevance and utility. Program designers and instructors need to answer the WIIFM (“What’s in it for me?”) question for participants because the more readily participants appreciate the relevance and usefulness of what they are learning, the more motivated they will be to learn it, and the more likely they will be to use it. Each topic and exercise should map to a chain of value that links learning to requisite job actions and desired business results. The links in the value chain should be explicit and shared with participants and their managers. Selecting instructional methods that coincide with the manner in which the learning will be used facilitates transfer and provides the know-how and practice needed for application. Finally, participants

CH003.indd 158

6/17/10 3:26:14 PM


159

should set or be assigned challenging goals for learning application to accelerate their personal improvement and business success. Use the checklist for D3 to check your own training program and to help you improve the transferability of training and development. Aim to include as many of the factors that favor learning transfer and application as possible. CHECKLIST FOR D3 ❑

Motivate

Program descriptions, materials, and instruction answer the what’s-in-it-for-me (WIIFM) question for participants.

❑

Relate

The links between the program content and current business needs and job responsibilities are clearly stated and reiterated for each major exercise or topic.

❑

Connect

Participants are encouraged to connect new learning to past experience to enhance retention and applicability.

❑

Demonstrate

Relevant examples of successful application are used throughout to show what “good” looks like.

❑

Share

Adult learning principles are honored by encouraging sharing of best practices and learners’ experiences.

❑

Practice

The design includes sufficient time for practice with feedback.

❑

Support

“Job aids” are provided to support transfer and on-the-job application.

❑

Monitor

Participants’ perceptions of the program’s relevance and utility are solicited, tracked, and acted upon.

❑

Review

Utility scores and participants’ goals for application are analyzed to make certain they reflect the program’s objectives.

Action Points For Learning Leaders • Ask two fundamental questions about each program component: • Will the value be obvious to participants? • Will they know how and when to use what they learn? • The better and more thoroughly these two conditions can be satisfied, the more likely that the education will be applied and pay returns to the company. • Review the objectives that participants set for themselves. • Are they well-crafted? • Do they reflect the business objectives for which the program was created?

CH003.indd 159

6/17/10 3:26:14 PM

160


• If participants’ objectives were to be sent to senior management, would they reflect well on the program and the learning organization? • If the answer to any of these questions is no, take corrective action. • Include questions on the end-of-course evaluations that assess the extent to which participants understand the relevance, utility, and value of what they learned and whether or not they feel confident that they know how to use it in their work. • If any of their responses are suboptimal, do a root-cause analysis to determine why. • Take corrective action. For Line Leaders • Review the learning and development programs that affect your area of responsibility. • Look for logical links among the business objectives, the kinds of cognitive and behavioral skills required to achieve them, and the learning exercises being used. • Send programs back for redesign if this linkage is not clear. • Ask to see the objectives that participants set for Phase III (on-the-job transfer and application). • Are these in line with your expectations? • Are they relevant to business needs? • Are the goals clear and specific with the appropriate amount of stretch? • Check with the managers who report to you. • Are they aware of their direct reports’ Phase III objectives? If not, why not? • Do they feel an important obligation to provide support and opportunities for putting the learning to work? If they do not, that is a serious management issue that you must address if your investment in learning and development is going to yield results. • Review end-of-program evaluations. • Do they ask participants about the program’s utility and whether or not they are motivated and confident to apply their learning? The answers to these latter questions are more important than whether they enjoyed the instructors and venue. • If these questions are not currently part of the evaluation, ask that they be added. • If they are, and the results are suboptimal, ask for an action plan to correct the situation. • Request reports on the end-of-program utility, motivation, and confidence ratings for each of the programs you sponsor.

CH003.indd 160

6/17/10 3:26:16 PM

DRIVE LEARNING TRANSFER To change behavior and get the results you want, you need structure, support, and accountability. —KEN BLANCHARD, KNOW CAN DO!

L

earning and development programs create value only when the new knowledge and skills they impart are transferred out of the learning environment and put to use in the work of the individual and firm. If learning is never used in a way that improves how things are done, then it is learning scrap—a waste of time and resources. That is why the Fourth Discipline of breakthrough learning is to drive learning transfer. We chose the word drive deliberately. The companies that are most effective at turning learning into results do so by implementing systems and processes to impel learning transfer and application. They invest energy to actively move the process forward and monitor progress, as they do with every other important business process, rather than passively leaving it to chance, individual initiative, or a miracle (Figure I.3). That learning transfer is critical has been recognized for decades. As far back as the 1950s, Mosel pointed to “mounting evidence that shows that very often the training makes little or no difference in job behavior” (Mosel, 1957, p. 56). Thirty years later, Baldwin and Ford reviewed the literature and concluded: “There is growing recognition of a ‘transfer problem’ in organizational training today. It is estimated that while American industries spend up to $100 billion on training and development, it is estimated that not more than 10 percent of those expenditures actually result in transfer to the job. . . . Researchers have similarly concluded that much of the training conducted in organizations fails to transfer to the work setting” (Baldwin & Ford, 1988). Despite such long-standing and clear articulation of the 161

CH004.indd 161

6/18/10 3:46:08 PM

162


FIGURE D4.1.

THE LEARNING TRANSFER PROBLEM

© Leo Cullum/Condé Nast Publications/www.cartoonbank.com. Used with permission.

problem, the failure of learning transfer continues to be the “elephant in the room” for training and development today (see Figure D4.1). When asked directly, both learning leaders and line managers acknowledge that a substantial amount of training goes unused (see “A Pernicious Problem” following). There needs to be a much more open dialogue about the issue and concerted efforts to resolve it. The learning transfer problem persists for two main reasons. The first is a lack of shared responsibility. Managers perceive training as the sole responsibility of the training department; they do not appreciate the essential role they have in ensuring its application. The learning organization and As Jim Trinka explained: “The typical manager line management co-own the believes that training or learning or whatever you success or failure of training. want to call it, is important. But they think: ‘Okay, that’s really important. But obviously I’m too busy to do it myself, so I’m going to outsource that job to the training department. Now it’s their job, not necessarily my job’” (Trinka, personal communication, 2009). Most learning professionals, likewise, perceive on-the-job transfer as outside their sphere of influence and accountability. They think: “My job is to be sure

CH004.indd 162

6/18/10 3:46:09 PM

Drive Learning Transfer

163

that program participants get the right content in the most effective way. But I have no control over what happens afterward; that’s the manager’s job.” In reality, the learning organization and line management co-own the success or failure of training. Unless both work together to actively drive learning transfer, it won’t happen. The initiative will fail and they will be like two people in a boat that has sunk, arguing about whose end leaked the most (Figure D4.2). That having been said, we believe that learning professionals need to take the lead in solving the learning transfer problem because they have the most to gain. Here is why: If performance fails to improve after training, the conclusion, from management’s perspective, is: “The training failed.” The truth is more likely that the training itself was a success, but the transfer failed. That doesn’t matter. The likely outcome is that the program will be reduced or eliminated. From the company’s point of view, “training”—which really means the whole learning process of preparation, instruction, transfer, and assessment—either succeeded or it didn’t. For learning professionals to try to narrow the definition of “training” to mean only the instructional event, and then argue that training succeeded even though performance remained unchanged, is akin to arguing that an operation

FIGURE D4.2. WHEN TRAINING FAILS TO PRODUCE RESULTS, EVERYONE LOSES, EXCEPT THE COMPETITORS (THE SHARKS)

© 2008 Fort Hill Company, used with permission.

CH004.indd 163

6/18/10 3:46:10 PM

164


was a success even though the patient died. Training is a success only if performance improves. Everyone benefits when learning organizations partner with line managers to ensure that learning is transferred and applied. And, because transfer is the weakest link in the learning value chain for most organizations, it represents the single greatest opportunity for improvement. Indeed, truly meaningful gains in training’s contribution are impossible unless the learning transfer challenge is met through shared responsibility. The second reason that the learning transfer problem persists has been a lack of systems and processes that could be effectively applied to large-scale corporate learning programs. To achieve a step change in transfer rates will require new approaches and new tools—as the saying goes: “Insanity is doing the same thing over and over again but expecting different results.” This chapter is about a new approach to learning transfer. Topics include: • • • • • • • • • • • •

A pernicious problem Learning scrap Root causes Why a great learning experience is not enough How to get to Carnegie Hall The transfer climate Overcoming inertia Follow-through breakthrough Better return on learning Implementing learning transfer management A checklist for D4 Action points for learning and line leaders

A Pernicious Problem Our determination to solve the learning transfer problem began more than a decade ago as the result of a project we did for a major chemical company. Together with a colleague, Dr. Bruce Reed, we were asked to evaluate the impact of the company’s flagship development program. Approximately six hundred managers had attended the program over the course of a year. Traditional endof-course evaluations were excellent and the program had won an award from the American Society for Training and Development. These plaudits notwithstanding, management wanted to understand how the program was being applied and

CH004.indd 164

6/18/10 3:46:10 PM


165

the impact it was having. We were engaged to conduct follow-up interviews with participants. We asked three questions: • What stands out for you about the program? • What action(s) have you taken to improve performance? • What help or support do you need? What we found was troubling. Only 15 percent (one out of every seven) of the participants we interviewed could give an example of improvement that resulted from an on-the-job action they had taken to apply what they had learned. They reported no particular need for help or support because they had felt no particular requirement to use what they had learned. In their minds, they had fulfilled their obligations simply by attending the class. They had no expectation for anything beyond that. Indeed, for many, our interview was the only follow-up of any kind they received. Senior management had very different expectations, however. They certainly expected more than ninety out of the six hundred managers to apply what they had learned in the program. We realized what a terrible waste of time and money it had been to train the other 510 managers, who achieved nothing more than if they had never been given the opportunity. We wondered whether these results were peculiar to this particular program or whether they were typical of a pernicious and widespread problem. So we began asking learning professionals: “After a typical corporate training program, what percent of participants apply what they have learned well enough and long enough to improve their performance?” The results should be a major wake-up call for our profession. The vast majority of workplace learning professionals still estimate that less than 20 percent of corporate training actually leads to improved performance. The results in Figure D4.3, collected at the 2008 ASTD International Conference, are typical. We have now asked the question of more than one thousand learning professionals, as well as a smaller number of line managers, all with similar results. In other words, the failure of most training and development to actually deliver improved performance is the biggest dirty little non-secret in the training industry. Other studies suggest that our findings are the rule rather than the exception. For example, when Kirwan (2009) asked participants to rate their behavior change following a course using a five-point scale, the average rating was 1.67 (p. 144). Berk (2008), based on thousands of self-reports across a wide range of companies, estimated the transfer rate at 40 percent. Even if the actual rate is nearer to Saks’ (2002) estimate of 50 percent than the often-cited 10 percent (Georgenson, 1982), it still represents a very low yield for a very expensive process.

CH004.indd 165

6/18/10 3:46:11 PM

166


FIGURE D4.3. ESTIMATES OF EFFECTIVE LEARNING TRANSFER BY 126 WORKPLACE LEARNING PROFESSIONALS

% of Respondents

35% 30% 25% 20% 15% 10% 5%

0% >4

0% 36

–4

5% –3 31

0% –3 26

5% –2

% 21

20 16 –

15 % 11 –

10 % 6–

0–

5%

0%

Estimate of Learning Transfer From Wick, Pollock, & Jefferson (2009). Used with permission.

Learning Scrap A major concern for companies in manufacturing is the cost of scrap. Manufacturing scrap is product that cannot be sold because it does not meet the customer’s specifications or expectations. We coined the term “learning scrap” to refer to learning that goes unused—learning that, therefore, fails to meet the customers’ expectations for improved performance. Manufacturing scrap has high costs—not only the cost of the raw materials and labor invested in producing unsaleable products but also the opportunity cost of tying up resources that could have been put to more productive use. In addition, there is the very real cost associated with losing customers when products fail to meet their expectations. For these reasons, manufacturing companies have worked relentlessly to drive down the scrap-rate toward the goal of six sigma quality— generating only one failure (defective part or service failure) in every 300,000.

CH004.indd 166

6/18/10 3:46:12 PM


167

Just like manufacturing scrap, learning scrap also has a high cost. There are the direct costs for instructional designers, trainers, travel, materials, and so forth, as well as the lost opportunity costs of having people spend time learning things Learning scrap also has a they cannot or will not use. There is also high cost. the very real cost of customer dissatisfaction when the departments that invest in training observe no subsequent improvement. In other words, the failure of learning transfer costs companies billions of dollars a year in wasted expenditures and probably several times that in lost opportunities. No company could stay in business if its other processes were as inefficient as most learning and development programs seem to be. To illustrate the point, we ask learning professionals in our workshops what their reaction would be if they gave FedEx one hundred packages to deliver and only fifteen arrived as expected. Obviously, FedEx would be out of business “overnight.” Knowing that meeting customer expectations for reliable delivery is paramount to success, companies in the package delivery business continue to pour creative energy, technology, and money into efforts to relentlessly improve their processes; their goal is to reduce the number of failed deliveries to as close to zero as possible. By comparison, training and development departments have done relatively little to improve their success rates in the fifty years since Mosel called attention to the problem. For their own survival, and for the good of the enterprises they serve, learning organizations need to have a much greater sense of urgency about reducing the amount of time and resources wasted through ineffective transfer. They, too, need to invest time, creativity, and technology to improve the process and reduce scrap, just as their colleagues in manufacturing and service industries do.

Root Causes Fundamental to any process improvement effort is the need to understand “root causes,” that is, to develop a deep level of understanding about what is causing the problem, as opposed to just focusing on the symptoms. As a first step in that direction, we built on an exercise originally developed by Rob Brinkerhoff and asked learning professionals where they think the process of converting learning to improved performance most often breaks down (Exhibit D4.1). We encourage you to jot down your own estimates before reading further.

CH004.indd 167

6/18/10 3:46:12 PM

168


Exhibit D4.1 Where the Learning-to-Improvement Process Breaks Down When training and development fail to deliver the expected improvement on the job, where do the breakdowns occur (by percent)? A. B. C. D.

Training was not the right solution to begin with ________% The preparation of the participants and course ________% The training itself ________% The post-training learning transfer period ________%

The typical range of responses is shown in Exhibit D4.2. There is strong consensus that the most frequent causes—more than 50 percent—of so-called “training failures” actually occur in the post-training period, back on the job. These are really failures of learning transfer. That should not be a surprise since, historically, the post-training period has received comparatively little attention compared to the efforts exerted to improve the design and delivery of instruction itself. There is also a strong consensus that breakdowns in the process of converting learning into business results occur least often in the coursework itself (10 percent or fewer). Again, that should not be surprising; substantial research and effort have gone into improving instruction and instructional design over the past thirty years. A review of the agendas of the major training and development conferences reveals this bias: whereas the vast majority of the seminars and workshops are concerned with improving the design, delivery, and assessment of instruction, fewer than 10 percent (and often none) are concerned with improving the actual application on the job.

Exhibit D4.2 Range of Responses to Where the Learning Process Breaks Down When training and development fail to deliver the expected improvement on the job, where do the breakdowns occur (by percent)? A. B. C. D.

CH004.indd 168

Training was not the right solution to begin with: 10 to 25% The preparation of the participants and course: 10 to 40% The training itself: 5 to 15% The post-training learning transfer period: 50 to 75%

6/18/10 3:46:13 PM


169

Several implications are immediately apparent: 1. The greatest opportunity to improve the output of the overall process is to improve the post-training learning transfer and application period, since that is where most of the breakdowns occur. 2. Further improvements in designing and delivering instruction itself will make a minimal contribution to improving the overall success rate, since that step is already highly effective. 3. Training programs that try to solve performance gaps due to causes other than lack of knowledge or skills will fail no matter how good the preparation, instruction, and follow-through are. Learning to avoid such quixotic missions (see D1) will improve the overall performance of the learning organization. 4. The remaining failures result from inadequate preparation. Inadequate preparation of the course, participants, or managers not only causes failures directly, but also contributes to failure to transfer. A manager’s lack of engagement prior to training influences a participant’s willingness to followthrough and put the learning into practice afterward (see D5). Therefore, engaging participants and managers more prior to the training itself has to be part of the overall solution to improving the success rate of training and development.

Further Analysis An important tool in process improvement is the cause-and-effect diagram, also called a “fishbone diagram” because when completed it resembles the skeleton of a fish. Its value is that “it focuses on the causes of variability. Getting people to think about and identify the causes of variability is a valuable education and will help future performance” (Scherkenbach, 1988, p. 106). Figure D4.4 shows a cause-and-effect diagram of the main causes of failure of learning transfer. The arrow forming the “backbone of the fish” points toward the effect; the “ribs” or rays pointing toward it are the contributing causes and contributing factors. To help identify contributing factors, it’s useful to label the main rays with the 6Ms: Machine, Man, Measurement, Management, Materials, and Method, or the 4Ps: People, Policies, Procedures, and Plant as “thought-starters.” These do not necessarily all contribute to a process failure, nor are they the only possible categories, but they are useful headings to guide exploration.

CH004.indd 169

6/18/10 3:46:13 PM

170


FIGURE D4.4. CAUSE-AND-EFFECT DIAGRAM FOR MANY OF THE FACTORS THAT CONTRIBUTE TO THE FAILURE OF LEARNING TRANSFER Measurement no tracking or metrics

Management

no accountability

no quality assessment of goals or outcomes

lack of automation no visibility into process

“too busy” unclear or conflicting priorities no recognition or reward

weak or no goals past experience low expectations poor motivation

Learning Transfer Failure

lack of manager engagement

no reminders

no opportunity to apply

no support

no or limited feedback peer pressure

Systems

Participant

no follow-up no action plan

Environment

Procedures

Several things are apparent from even a cursory review of the diagram: 1. Multiple factors contribute to the poor rate of learning transfer. Therefore, the solution to the learning transfer problem will have to be multifaceted. 2. In most companies, there is no real process for managing learning transfer. One of the first steps these companies need to take is to define a core process that can be subsequently improved. 3. Since all of the people involved—from participants, to managers, to managers of managers, as well as learning professionals—have expectations based on past experience, it will be necessary to reset those expectations. Our recommendation to take a process approach to improving learning transfer does not assume that developing people will ever be as predictable as manufacturing a product or delivering a package. But as McLagan put it: “Process excellence has to be a major focus of any organizational improvement intervention—including the design of learning for work” (McLagan, 2003,

CH004.indd 170

6/18/10 3:46:14 PM


171

p. 54). There is ample evidence that taking a process approach to learning transfer can improve outcomes, reduce scrap, and increase the benefit of learning for the organization (Brinkerhoff & Apking, 2001; Park & Jacobs, 2008; Phillips & Broad, 1997; Wick, Pollock, Jefferson, & Flanagan, 2006). In an increasingly competitive economy, no company can afford the high cost of doing nothing about the learning transfer problem.

A Great Learning Experience Is Not Enough Most corporate learning initiatives still rely on the “big bang” theory. That is, they hope that by creating a compelling enough event, they will impart sufficient energy and momentum to carry the participants through the hard work of turning learning into results. A big bang was how Jules Verne proposed to get his astronauts to the moon in his 1865 science fiction masterpiece From the Earth to the Moon: he shot them into space from an oversized cannon. Since then, we’ve learned that no cannon ever built has generated the force required to propel even a small object into space. It is simply not possible to generate sufficient impulse in one big bang to accelerate an object to the escape velocity it needs to break free of gravity. Sooner or later, air resistance and gravity take over, the projectile loses momentum, and it falls back to earth (Figure D4.5). It wasn’t until the advent of modern rockets, which are capable of providing sustained propulsion over a long period, that it was possible to finally achieve escape velocity, tear free of gravity, and get into orbit (Sputnik in 1957). The problem for training and development is analogous. No matter how much impetus the FIGURE D4.5. THE PROJECTILES OF THE MOST POWERFUL CANNONS EVER BUILT ALWAYS FALL BACK TO EARTH

THU D!

CH004.indd 171

6/18/10 3:46:14 PM

172


FIGURE D4.6.

UNLESS ENERGY IS EXERTED TO DRIVE TRANSFER, CHANGE EFFORTS LOSE MOMENTUM

Proficient

Competence

For learners to become truly proficient, learning must continue on the job.

Minimal Without reinforcement, performance rapidly returns to precourse levels.

Course

On-the-job application and continued learning since training

program imparts initially, resistance to change and the weight of old habits cause the participant to lose momentum and fall back into pre-training routine. Just like a rocket, a continuing input of energy over time is needed to help participants achieve “escape velocity” (Figure D4.6). Why is so much energy required to prevent recidivism? John Izzo, consultant and best-selling author, shared the following explanation with us from a new book he is writing on why it is so hard for organizations and people to change: Learning is like sledding down a hill on a fresh field of snow. On your first try, you could take any number of paths. But the next time you go down, the sled tends to follow the path you have already established. The more times you go down, the deeper you wear a particular groove and the harder it is to go a different way. That is very much how the brain works: the more you perform a particular action, the more it becomes automatic and the harder to change. That’s essential for survival; we’d be paralyzed if we had to think through every single action all the time. But it does mean that long-established habits—whether how to do a certain task or how to react to another person—take real effort to change. It might even be, as in the sled analogy, that you have to start from a different place to avoid falling back in the rut (Izzo, personal communication, 2009).

CH004.indd 172

6/18/10 3:46:15 PM


173

We know now, from recent advances in neuroscience, that there are actually physical changes in the brain that reflect often-used pathways (Deutschman, 2005). Professional violinists, for example, have a much larger portion of their Great learning alone is brains dedicated to their left (fingering) insufficient to produce real hand than to the right (bow) hand. Once change. habits are hardwired into the structure of the brain itself, it takes time and sustained effort to change them. Thus, a great learning experience can be critical to get participants started in the right direction from a new place, but great learning alone is insufficient to produce real change because adopting new and more productive behaviors requires much more.

How to Get to Carnegie Hall There is an old saw about the tourist asking the policeman in New York, “How do I get to Carnegie Hall?” to which the officer replies: “Practice, practice, practice.” It turns out that the policeman is right. Recent research suggests that in every field of human endeavor—from chess, to business, to sports, to the performing arts—what really separates world-class performers from everybody else is the amount that they practice. When we think of top performers—Jack Welch in business, Lance Armstrong in bicycling, Mozart in music—we tend to ascribe their success to “natural talent.” The implication is that we could never be as good at something as they are because we just weren’t born with “the right stuff.” Not so, according to a thorough review of the evidence by Geoff Colvin in Talent Is Overrated (2008). It turns out that “natural talent” is a lot less important than people think. When researchers from around the world gathered to discuss “The Acquisition of Expert Performance,” they concluded that what differentiated experts from the less-accomplished in a wide array of fields was the amount of practice (Ericsson, Krampe, & Tesch-Romer, 1993). The more practice, the better. In other words, Mozart became Mozart by working furiously hard and, starting when he was four years old, by practicing his art more than anyone else. But not just any practice will do. The research shows that a special type of practice is needed — something the researchers have termed “ deliberate practice.” Deliberate practice is repetition that focuses on technique as much as results. It requires intense concentration, feedback on performance, and taking time for reflection: What contributed to a positive outcome and should be continued? What detracted from the desired outcome and should be reduced or avoided?

CH004.indd 173

6/18/10 3:46:16 PM

174


Colvin (2006), writing in Fortune, summarized the findings this way: “The evidence, scientific as well as anecdotal, seems overwhelmingly in favor of deliberate practice as the source of great performance. ” Daniel Coyle ’s study of “talent hotspots,” The Talent Code (2009), comes to a similar conclusion. Deliberate practice (which he terms “deep practice”) is prerequisite to great performance in any field of human endeavor. “Deep practice is built on a paradox: struggling in certain targeted ways—operating at the edges of your ability, where you make mistakes—makes you smarter” (p. 18). Gladwell (2008), quoting John Lennon, argues that the Beatles became great only after playing up to eight hours a night, seven nights a week in a Hamburg club (p. 49). By the time of their first hit in 1964, they had already performed live some twelve hundred times. So, while training, job assignments, and self-study can catalyze learning, there are no shortcuts to becoming highly skilled in any field of endeavor—it takes practice and lots of it. It should come as no surprise, then, that so much of whether training “works”—that is, whether it improves performance—depends on whether or not people actively practice their new skills in the immediate posttraining period. That is why facilitating deliberate practice is one of the most effective ways to increase the return on the training investment.

The Transfer Climate Writing about executive education in Harvard Business Review, Professor Herminia Ibarra of INSEAD noted that “the personal learning catalyzed by a top-notch program can be tremendous. . . . The problem, my research suggests, is what happens when a manager comes back to the day-to-day routine of the office” (Ibarra, 2004). In other words, whether or not learning generates meaningful performance improvement is strongly affected by the “transfer climate.” Direct evidence of the power of the transfer climate to influence the results of training comes from a study conducted by American Express (2007). The original goal of the study was to compare three different kinds of instruction: instructor-led, blended, and e-learning only. The outcome measures of interest were (1) assessments completed by manager, learner, and direct reports three months after the learning event and (2) improved business results such as cycle time, conversion rates, revenue impact, forecast accuracy, sales, customer satisfaction, and handling time—depending on the specific manager’s responsibilities. Managers who attended the program were categorized as either “ high improvement leaders” or “low improvement leaders” based on the results. The differences were dramatic. The direct reports of “high improvement” leaders increased productivity by an average of 42 percent, while the productivity of the direct reports of “low improvement” leaders increased only 16 percent.

CH004.indd 174

6/18/10 3:46:16 PM


175

As it turned out, the instructional method as such was not the key factor; both high- and low-impact leaders were found in groups taught by any one of the three methods. So something else must have accounted for the difference. To gauge the impact of the transfer climate, American Express included questions about the transfer climate in the post-program survey. They identified three factors that consistently predicted whether a participant would be in the “low improvement” or “high improvement” category. Specifically, they found that participants who achieved high improvement were • Four times more likely to have had a meeting with their manager to discuss how to apply the training • Almost twice as likely to perceive that their manager supported and endorsed the specific training • More than twice as likely to expect to be recognized or rewarded for the training-related behavior change [supporting Vroom’s expectancy model discussed earlier in this book] The study authors concluded: • “The true impact of a training program will best be predicted by the work environment participants return to after the event.” • “A highly supportive transfer climate needs to be present in order for a solid and lasting transfer of the information back to the job.” • “[T]hese climate factors can quite literally make or break a company’s training investments.” • “[T]he importance of understanding and creating a high transfer climate should swiftly move to the forefront of any training initiative or strategy.” Numerous other studies have come to a similar conclusion: The transfer climate to which an employee returns after training has a profound impact (see reviews by Burke & Hutchins, 2007, and Saks & Belcourt, 2006). But what constitutes the transfer climate and determines whether it is conducive or toxic to learning transfer? Holton, Bates, and Ruona (2000) developed a “Generalized Learning Transfer System Inventory” to help gauge the state of the transfer climate and whether it was likely to have a positive, reinforcing effect on learning, or whether it was likely to be counterproductive. It has also proven useful as a diagnostic tool to direct learning transfer system change (Holton, 2003). The inventory has subsequently been validated in a number of contexts and countries. As its name implies, the Learning Transfer System Inventory recognizes the systems nature of the learning transfer environment, that is, that learning

CH004.indd 175

6/18/10 3:46:17 PM

176


FIGURE D4.7.

THREE MAIN CLUSTERS OF FACTORS THAT AFFECT LEARNING TRANSFER Motivation to Use

Ability to Use

Learning

Work Environment

LEARNING TRANSFER

RESULTS

transfer is affected by complex interactions among a large number of variables. Three main clusters of factors influence the probability of transfer: ability to use; motivation to use; and the work environment (Figure D4.7). These three clusters influence transfer activity directly as well as indirectly through their interactions with each other. In the Learning Transfer System Inventory, four factors are considered to contribute to the ability of trainees to apply new learning on the job. First, the employee must have the personal capacity (time, energy, and mental space) to make changes in his or her work. Second, the employee’s work situation must afford him or her opportunities (tasks and resources) to apply the new skills and knowledge. Third, the employee needs to perceive that what was taught is relevant, valid, and applicable (has content validity). Finally, the ability to apply is influenced by the extent to which the program was delivered with application in mind (see D3). As we discussed in D3, three factors influence the motivation to use: the extent to which employees believe that applying the new skills will improve their performance; the extent to which they believe that improved performance will be recognized and rewarded; and the value they ascribe to the outcome (Figure D3.3). Two secondary factors also contribute to motivation. The first is the employee’s personal confidence in his or her ability to change and improve (“self-efficacy” or what Dweck [2006] refers to as “mindset”). Obviously, employees who have a strong internal belief in their ability to change and grow are more motivated and more likely to try to apply new learning than those who feel they are victims of their environment, genome,

CH004.indd 176

6/18/10 3:46:18 PM


177

and the like. Finally, motivation is also influenced by the extent to which the learners were prepared to participate in the training to begin with. The third cluster of factors that determine the transfer climate concern the work environment. Numerous research studies over many years confirm that the work environment influences learning transfer (see review by Gilley & Hoekstra, 2003, for example). Work environment factors include the influence of the manager, the peer group, and the reward system. In the inventory, managers’ influence is further divided into the amount of feedback and coaching they provide, the extent to which they are perceived to support and reinforce the use of learning, and, conversely, the extent to which they are perceived not to be negative about the training or discourage its use. Peer groups exert their influence through their general openness or resistance to change and the extent to which they support or discourage efforts to apply new skills and knowledge. The reward system has two elements: the belief that applying the new skills will lead to positive personal outcomes and the corresponding belief that not using what was taught will lead to negative personal consequences (see Figure D4.8).

FIGURE D4.8. ALL SIXTEEN FACTORS OF THE LEARNING TRANSFER SYSTEM INVENTORY • Readiness for learning • Confidence in ability to change • Personal capacity • Opportunity • Training design • Content validity

Motivation to Use

Ability to Use

Learning

CH004.indd 177

MANAGER • Positive attitude • Encourages use • Provides feedback PEERS • Open to change • Encourage use REWARDS • Positive for use • Negative for non-use

• Believe use will improve performance • Believe performance will be recognized • Value ascribed to outcome

Work Environment

LEARNING TRANSFER

RESULTS

6/18/10 3:46:18 PM

178


It is important to note that in the foregoing description, words like “belief ” and “perception” are used, reinforcing a theme that runs throughout our book: People act on their perceptions and beliefs, not necessarily on objective reality. Therefore, if an employee perceives that his or her manager is negative about the training, the employee will be less likely to try to use it, even if, in reality, the manager is supportive, but too distracted by other duties to make that clear. An important starting point, then, for People act on their perceptions increasing learning transfer is to make an assessand beliefs, not necessarily on ment of the current transfer climate as perceived by objective reality. the participants. The full Learning Transfer System Inventory contains ninety-nine items. Holton published a shorter “audit” version in Improving Learning Transfer in Organizations (2003, pp. 73–76). We developed the self-assessment in Exhibit D4.3 to help learning professionals assess the transfer climate in their organizations and identify areas for improvement. For each of the items on the Transfer Climate Scorecard shown in Exhibit D4.3, rate your post-program environment from very unfavorable/unhealthy (–3) to very favorable/ very healthy (3). In general, the more of the factors that are positive, and the more strongly positive they are, the greater the probability of transfer (Holton, 2003, p. 68). Not all of the factors have equal impact. Some, such as manager involvement, exert greater influence than others, although this varies depending on the overall company culture and the nature of the training and work. Given the difficulty of effecting lasting change (discussed in the next section), improving the learning transfer climate—and therefore the efficacy of training—will require a systemic approach that optimizes multiple factors; a “single-shot” solution (as we learned the hard way early in our work on learning transfer) won’t be sufficient.

Overcoming Inertia Corporate learning programs are ultimately change initiatives. But human beings and human organizations are remarkably resistant to change, even in the face of what should be a “burning platform.” Change Is Hard You would think that people would change if their lives depended on it. A lifethreatening illness ought to be the ultimate incentive for change. And yet, among people who have had coronary bypass surgery, only 10 percent succeed in changing

CH004.indd 178

6/18/10 3:46:19 PM


179

Exhibit D4.3 Self-Assessment of Transfer Climate

CH004.indd 179

Factor

Description

Perceived Utility

Participants returning to work believe that they will be able to utilize their new skills and knowledge and that they will perform more effectively when they do.

Opportunity

Individuals have opportunities to apply their new skills and knowledge on the job. They are provided with the resources they need to do so (time, tasks, assistance, materials, people, and so on).

Expectations/ Rewards

Participants believe that they are expected to use their new skills and knowledge and that they will receive positive recognition for doing so. They also expect negative consequences of not using what they learned. The organization links development, performance, and recognition; tracks progress; and rewards improved performance.

Feedback/ Coaching

Participants receive constructive input, assistance, and coaching from managers, colleagues, and others when they attempt to use what they have learned.

Managerial Engagement

Managers actively support the use of new skills and knowledge. They discuss performance expectations both before and after training, help identify opportunities to apply new skills, set relevant goals, provide feedback, and help work through difficulties.

Work Group Impact

Participants’ co-workers encourage them to apply new skills and knowledge. They display patience with the difficulties of mastering new approaches. They are willing to accept new approaches and do not force conformity to existing norms.

Personal Experience

Individuals experience positive outcomes from using what they have learned, for example: improved productivity; increased job satisfaction; additional respect; recognition, advancement, or reward.

Rating

6/18/10 3:46:21 PM

180


their lifestyles in ways that reduce the risk of restenosis and a potentially fatal heart attack (Deutschman, 2005). Even when the choice is “change or die,” many people are unable to do so. We all know from personal experience how hard it is to keep resolutions to get more exercise, lose weight, take more vacation, and so forth. Part of the explanation may be that people are genetically wired to not change too quickly. Families, organizations, and societies can only function if the people in them behave in reasonably predictable ways. Since being part of a group improves the chances of survival, people who behaved in a consistent manner over time had an advantage; indeed, erratic or unpredictable behavior by a member of a group causes severe stress on an organization and its members. Over time, being predictable was favored over being mercurial. Change is also hard because, while the brain is an extraordinary organ, its conscious processing capacity is actually quite limited. Therefore, over time, frequently repeated actions are reduced to habits that can be performed “without thinking”— that is, without a lot of conscious thought. Most adults, for example, are able to think of other things while performing the complex task of driving a car. The disadvantage is that once specific patterns of behavior are reduced to habit and automaticity, it becomes very difficult to change them; the tendency—at both the personal and organizational level—is to quickly revert to old habits, especially under stress. In other words, there is a tremendous amount of organizational and personal inertia resisting change (Figure D4.9). And, in accordance with Newton’s First Law of Motion, “an object at rest remains at rest unless acted upon by an external

FIGURE D4.9.

Change initiative

PEOPLE AND ORGANIZATIONS HAVE GREAT INERTIA THAT RESISTS CHANGE

nd al a al n o s Per ization an Org nertia I

ved pro

Per

e

anc

m for

Im

CH004.indd 180

6/18/10 3:46:21 PM


181

force.” Things will continue as they are until there is a sufficient outside force to overcome the inertia. The challenge for training and development organizations, then, is to find the right combination of factors that provide sufficient power to the engine of change to move the organization to a new level (Figure D4.10). Overcoming resistance to change requires a multi-pronged approach that includes people (learners, managers, peers) as well as systems and processes to sustain learning transfer and create new norms. Essential Elements The literature on learning transfer and our work with hundreds of training and development programs over the past ten years indicate that an effective learning transfer management process has six elements: • • • • • •

A schedule of events for the post-course period Reminders Accountability Feedback and coaching Performance support Finish line FIGURE D4.10. A MULTI-PRONGED APPROACH IS NEEDED TO OVERCOME INERTIA AND RESISTANCE TO CHANGE

, ple o e s s, P se em roces t s Sy nd P a

nd al a nal n so tio Per niza a a ti Org Iner

ved pro

Per

e

anc

m for

Im

CH004.indd 181

6/18/10 3:46:23 PM

182


Schedule of Events The first prerequisite for a more effective approach to driv-

ing learning transfer is to have a clear schedule of events post-instruction. Just having some definitive activities, such as assignments, reports, teleconferences, or other touch-points scheduled for Phase III helps shift the paradigm away from “the end of class equals the end of learning and responsibility” and strengthens the “expectation” aspect of the transfer climate. Spacing post-course activities out over time also takes advantage of the spacing effect of learning, resulting in more durable and accessible knowledge and skills. As discussed in D2, the schedule of post-course events should be part of the overall program design and should be included as an integral part of the program agenda (Figure D2.5). The various kinds of useful post-course activities and their relationship to the learning transfer model are discussed below. To realize the benefit of the spacing effect, schedule a series of activities over several weeks, each of which will cause the participants to recall and revisit what they learned. Something as simple as sending an application tip, a relevant article synopsis, or an example of successful application can help bring some aspect of the training back to mind. Post-course assignments that require active engagement and thinking by the participant are more effective than simply pushing additional information to them. Examples include sharing learning with others, submitting progress reports, and participation in teleconferences, blogs, quizzes, or contests. After a workshop on the 6Ds, a safety trainer at General Electric implemented a contest to reinforce learning from one of his programs. Every week for several weeks after the training, he posed a question to the participants based on some aspect of what they had covered. There was a small token prize and recognition of the winner at the end. The contest caused all the participants to revisit what they learned each time a new contest question arrived. The main incentive was being recognized as the “best in class.” The contest also had the benefit of providing useful feedback—reinforcing the knowledge of those who knew the correct answers and correcting misconceptions of those who submitted incorrect ones. A number of companies require, as a pre-condition for attending a program or conference, that the attendee share a synopsis of the key topics and the “lessons learned” with colleagues when they return. The benefits include broader dissemination of learning Reconvening the group can be and—importantly—strong reinforcement for the a powerful stimulus to revisit person required to prepare and teach what they and apply learning. learned through the neurophysiologic process of “elaborative rehearsal” (see Case in Point D3.3). An even more effective learning transfer strategy is to require participants to submit periodic progress reports on their efforts to implement what they have learned. This is analogous to the kind of progress reporting expected of

CH004.indd 182

6/18/10 3:46:24 PM


183

individuals and project teams on other business initiatives. Scheduling a series of required post-course progress reports underscores the expectation that people will apply their new knowledge and that they will make progress toward the goals they set (or were assigned) from the program. Indeed, to ask people to set goals and then not require some sort of progress report makes a mockery of the entire goal-setting process. If time and budget permit, reconvening the group also can be a powerful stimulus to revisit and apply the initial learning, especially if the session begins with a report-out from participants of the progress they have made, the obstacles they have encountered, and what they learned in working through them. Where a physical reconvene is not possible, a teleconference or web meeting can be used to good effect. A number of companies, including Chubb Insurance, Sony Electronics, and others, use a teleconference as the capstone event for key development programs. As follow-up to Six Disciplines Workshops, we have successfully used web-conferences in which each participant is given a few minutes to report on his or her accomplishments. The results (success stories) shared by participants are often outstanding and powerful affirmation of what can be accomplished through diligent application of new learning. For teleconferences to be effective, however, they must have a specific agenda that includes active participation by members and some form of accountability. Organizations have been disappointed by low participation in post-training teleconferences and webinars that were just refreshers or additional instruction. That result is not surprising, given the known drivers of change in the learning transfer model. Reminders In addition to a schedule of specific post-course activities, participants have to be reminded periodically of their obligations and the importance of applying what they learned. A fundamental impediment to learning transfer is the “outof-sight, out-of-mind” problem. Employees are constantly reminded of their other obligations through required reports, balanced scorecards, financial informaMcDonald’s doesn’t tell you tion systems, and conversations with they sell hamburgers just once. their managers. In contrast, reminders about following through on their learning transfer objectives are rare or non-existent. For the great majority of programs, participants never hear about learning again after the end of class—until they receive a solicitation to sign up for the next course. The problem relates to what marketers call “share-of-mind.” A consultant and colleague, Janet Rechtman, pointed out that “McDonald’s doesn’t tell you they sell hamburgers just once.” McDonald’s is, without a doubt, already one of

CH004.indd 183

6/18/10 3:46:24 PM

184


the best-known and most-recognized brands in the world, and yet they continue to spend hundreds of millions of dollars on advertising annually. Why? Because they know that if they were to stop repeating the message, they could quickly lose shareof-mind to competitors whose message consumers would encounter more often. And in most businesses, share-of-mind equals share-of-wallet. So McDonald’s invests enormous amounts of creative energy and resources to stay top-of-mind. Learning organizations are not directly interested in a share of the participants’ wallets, but they do need a share of their time for learning transfer to be effective. In the absence of reminders, commitments to follow through on learning lose “share-of-mind.” In the face of so many messages about so many other commitments, the urgency of educational objectives rapidly declines until they are forgotten completely. A common practice in leadership programs years ago was to mail participants copies of their objectives several months after the program. The authors’ personal experience is that, in the absence of any interim contact from the learning organization, all this practice accomplished was to reinforce how little had actually been done. It was a reminder, but it was too little and too late. In their classic book on marketing, Positioning: The Battle for Your Mind, Ries and Trout (2001) emphasize that the same message must be repeated many times to get through the clutter of competing ideas. In the clutter of competing priorities, the message about the importance of transfer and application will be lost if communication stops as soon as the course is over. The original insight that led to the development of learning transfer management systems was our realization that one of the few things that was up-to-date on our computers was the anti-virus software. And that was because the software automatically alerted us when it needed to be updated. It was the active reminder that prompted us to take action, not horror stories from colleagues whose computers had been infected. As the pace of business has accelerated and schedules have become more and more crowded, people increasingly rely on electronic calendars and automated reminders for important appointments, phone calls, project deadlines, and so forth. Otherwise, it is too easy to become involved in one thing and completely forget some other obligation. Post-course reminders can take any number of forms—mail, email, telephone calls, automated calendar items, or any combination thereof. The most important thing is that there are reminders so that the objective is not forgotten. Levinson and Greider (1998) developed a simple device they called the MotivAider, which was worn like a beeper and did nothing more than vibrate on a set schedule. Nevertheless, that simple, periodic reminder proved remarkably effective in helping people follow through on a wide variety of goals. Levinson and Greider defined the

CH004.indd 184

6/18/10 3:46:25 PM


185

two key attributes of an effective reminder system as “(1) it must reliably get attention and (2) it must occur often enough to serve as a useful clue” (1998, p. 173). Despite the email overload we all experience nowadays, email reminders still work. In a study of over two thousand employees in five workplaces in Canada, Plotnikoff and colleagues (2005) tested the efficacy of email reminders for helping to change behavior related to exercise and nutrition. Compared to the control group, the employees who received weekly email reminders for twelve weeks showed improvements in both physical activity and healthy eating. A similar study at Kaiser Permanente, funded by the U.S. Centers for Disease Control and Prevention, found that employees who received weekly email reminders with small, practical suggestions for improving their health significantly improved their lifestyle habits compared to those who did not receive the messages (Pallarito, 2009)—further evidence of the power of periodic reminders by email. Accountability Reminders are all well and good, and they have demonstrable

efficacy in their own right. But consistent with the learning transfer system model, they are much more effective when they are backed by clear accountability for action. Participants exert more effort to apply their learning when they expect to be held accountable, rewarded for using their new learning and skills, and critiqued if they fail to do so (Figure D4.8). Holding participants accountable for making effective use of companysponsored training is simply good business practice; in any well-run business, employees are responsible for making good use of their time and other company Holding participants resources. Since training and development accountable is simply good programs represent an investment, they business. are not unlike taking out a loan. When you borrow money from a bank, you take on an obligation to repay it (with interest). There is a set schedule for repayment; the bank does not hesitate to remind you if you fall behind. If you fail to repay the investment, it becomes much more difficult to borrow money in the future. Although training programs are rarely explicit about it, participants in corporate-sponsored educational programs incur similar obligations. Management expects its training investment to be repaid through greater productivity and effectiveness (Figure D2.8). When employees are given the opportunity to participate in learning programs, they should be informed that they are, in effect, entering into a contract in which they will be held accountable for using what they have learned. Those who are unwilling to make the required “payments” through application (which is in their own best interests anyway) should be considered poor candidates for further educational investments.

CH004.indd 185

6/18/10 3:46:25 PM

186


Echols (2005) makes the interesting point that companies and employees are actually co-investors in training and development. Even when the company bears 100 percent of the financial cost of the education, “without the concurrent employee investment of the human elements of their time, motivation, and emotional energy, the dollar investment is a wasted economic expenditure with very low potential of producing economic benefit to the company” (p. 22). He goes on to say that securing co-investor status—shared accountability for the results—between employees and the company is prerequisite to earning a return on human capital investments. Being accountable is defined as “having the obligation to report or justify something; responsible; answerable ” ( Webster ’s College Dictionary , 2001). Accountability implies a system of follow-up, a person or agency to answer to, the means and schedule for reporting, and a standard against which to gauge performance. Participants’ first accountability should be to themselves. They have the most to gain. They have already invested their time in attending a program. They can enhance or reduce the likelihood that they will be given greater responsibility and authority in the future. They owe it to themselves to make good on the investment by extracting as much value as possible from all four phases of the learning experience. For most people, achieving one’s own standards is validating and fulfilling and often a more powerful motivator than external rewards (Csikszentmihalyi, 1990). One needs only to reflect on the amount of time, money, and energy people devote to trying to lower their golf scores to see the truth of that statement. The second level of accountability should be to their teams—the people who work with them and for them. We strongly encourage participants to “go public” with their goals—to share them with their direct reports and colleagues when they return from an educational program. There is strong evidence that people who share their goals with others are more likely to achieve them (Patterson, Grenny, Maxfield, McMillan, & Switzler, 2008). Making learning transfer goals public not only helps raise the level of accountability but also engages others as sources of ideas and encouragement. Goldsmith (1996) found a direct correlation between the degree to which participants followed up with their co-workers and the amount of improvement those co-workers perceived (p. 233). Finally, to maximize learning transfer, program participants must feel accountable to their managers. Managers should require their subordinates to develop clear plans for applying what they learned, act on them, and report the progress and results achieved. It is in the managers’ best interest to do so, since his or her career depends in part on a proven ability to develop others. Managers also have a fiduciary responsibility to ensure that corporate resources are used to further the goals of the organization.

CH004.indd 186

6/18/10 3:46:26 PM


187

As the above dictionary definition suggests, being accountable means being answerable and reporting outcomes. That implies a process for reporting and a system for tracking and accounting. Historically, that has been a stumbling block for creating more accountability for learning transfer; it was difficult or impossible to keep all the relevant parties informed, engaged, and accountable using manual methods. That is why we consider the development of computer-based transfer management systems a true breakthrough in corporate training and development (see Follow-Through Breakthrough, page 189). Feedback and Coaching In our review of the research about what it takes to become good at anything (page 173), we cited the numerous studies of human expertise, all of which stress the importance of feedback and coaching. It is virtually impossible to improve any skill without some form of feedback. Imagine trying to learn archery wearing a blindfold. If you could not see where the arrows landed, and no one would tell you, you could take ten thousand shots and, in the absence of feedback, still never become any better. People need to know what is working and what is not working in order to reinforce positive actions and correct ineffective or negative ones. In the absence of feedback, employees are unable to maximize their strengths or modify counterproductive or downright destructive behaviors. For these reasons, finding ways to enhance feedback and coaching is an important aspect of improving the transfer climate. We will discuss ways to achieve this in greater detail in D5. Suffice it to say at this point that learning organizations can accelerate mastery of new skills and behaviors by making sure that effective feedback is available and encouraging its use. Performance Support A closely related

concept is the availability of support Motivation to transfer and assistance in the post - training learning is influenced by the period. Recall that motivation to use participant’s confidence. new knowledge and skills is influenced by the participant’s confidence in her or his ability to do so successfully. Practicing D3, delivering instruction in ways that facilitate application, helps build this confidence. Providing ongoing performance support in Phase III adds to it. Even if the program has been well designed to deliver for application, and even if participants leave confident in their ability to utilize what they have learned, they are bound to encounter unexpected situations, impediments, or lapses of memory as they begin to try to apply the program material and principles to their work. Their confidence and motivation will be sustained if they can

CH004.indd 187

6/18/10 3:46:26 PM

188


find help and answers quickly. If they encounter a problem, however, and have no place to turn for guidance, then they are likely to quickly abandon the effort. How to deploy effective performance support is discussed in Chapter D5. The point here is that it is an essential element of the learning transfer management system. Finish Line In D2, Design the Complete Experience, we suggested that the fourth

and final phase of learning should be assessment of achievement—a summing up of what had been accomplished as a result of attending the training and applying it—a goal line to drive for. That’s because most people, especially those who are attracted to business as a career, have a powerful, intrinsic drive to succeed. We also argued that learning organizations cannot succeed unless the “conditions of satisfaction” are defined in advance. The same is true for program participants. Consequences, rewards, and recognition drive motivation and impact learning transfer in multiple ways, both directly and indirectly. For participants to be motivated to use what they have learned, they have to know where the finish line is and what success means. Therefore, having a definitive point at which achievement will be reviewed and recognized contributes to a positive transfer climate. For this reason, many progressive companies build “reporting out” sessions into their programs, several weeks to months post-instruction. In the action learning program at OSRAM, for example, learning teams had to present their results and recommendations to members of the company’s executive committee. Participants needed no prompting to understand the value of being able to demonstrate high-quality work to that audience. At Honeywell, participants in the strategic marketing program are divided into project teams that have clearly defined reporting requirements. As Rod Magee, former CLO at Honeywell, explained to us: “At the end of the program, each team has to define deliverables and an action plan for the next ninety days. Rather than letting them go off and assume they will do it, we keep them accountable by having scheduled teleconference updates. The teams know when they leave the program that at thirty, sixty, and ninety days they will have to report to management. So there is clear accountability after the program. The first call at thirty days is usually with the head of marketing for their business or the head of business strategy. The goal is to reinforce, validate, or challenge the team’s plans and progress. At sixty days, the president of the business usually joins the call. At ninety days, the team has to report its success against its promised deliverables. In each of the calls, the team is joined and supported by the coach they had during the program” (Magee, personal communication, 2005).

CH004.indd 188

6/18/10 3:46:27 PM


189

At the Chubb Group of Insurance Companies, participants do not receive credit for completing the program until they have participated in a teleconference in which they report what they have accomplished with what they learned (Amaxopoulos, personal communication, 2009). At the “graduation” exercises from the Center for Leadership at Children’s Healthcare of Atlanta, participants post their accomplishments on the Wall of Success and share their success stories with an audience that includes the senior leadership team (Mohl, 2008). Ten weeks after the Leadership Forum at Sony Electronics, participants are required to present and defend an estimate of the value they created by applying their learning on a conference call that includes one of the executives on Sony’s Talent Management Council (Grawey, 2005). The common thread in these approaches is that participants know in advance that there is a specific time and forum in which they will be required to account for what they have accomplished. The forums include persons of importance to the participants to maximize the motivational factors related to recognition and outcomes. Having clear expectations and a known point in time at which they will be assessed helps drive learning transfer.

Follow-Through Breakthrough Most of the above principles for optimizing learning transfer have been known for a long time. But the complexity, cost, time, and difficulty of implementing them posed significant barriers for learning organizations. The breakthrough was The breakthrough was the the advent of new technology: computer advent of new technology. databases, electronic mail, and universal Internet access that finally made it possible to efficiently facilitate and monitor the learning transfer process for large numbers of participants simultaneously. We introduced the first online learning transfer management system, Friday5s, in 1999. It was subsequently named a “Training Product of the Year” by the independent Training Media Review. Since then we have continued to research and refine the concept of computer-based learning transfer management; our most recent system is called ResultsEngine to underscore our belief that, in the end, training must deliver results and requires an engine to help drive them. To date, more than 100,000 participants, in thousands of programs, in hundreds of companies around the world have used these systems. By and large, companies have been able to demonstrate that adding a transfer management system to a learning

CH004.indd 189

6/18/10 3:46:27 PM

190


or development program enhances participants’ efforts to use what they learned, facilitates interactions with their managers, accelerates performance improvement, and increases the return on investment in the program (see Case in Point D4.1). Not every application has been a success, however. In keeping with the systems nature of the learning transfer climate, firms that introduced transfer management systems into a hostile climate did not see the expected uptick in results. Indeed, it was the variability of success that led to our recognition of the Six Disciplines of Breakthrough Learning. We discovered that, while a learning transfer management system can be a powerful catalyst for application, it cannot

Case in Point D4.1 Making Safety Training Stick BST, Behavioral Science Technology—a world leader in safety consulting—uses a customized learning transfer management system developed in partnership with the Fort Hill Company. Kristen Bell, vice president, research and development, explains: “The followthrough system is an integral part of our practice. It helps us reach our clients’ managers and supervisors by providing an efficient means to deliver coaching and feedback on their safety leadership. “ We have hundreds of success stories captured in the system. In one example, a manager used the system to track his own progress in providing consistent feedback and support to employees in the use of safe work practices. In one entry, he described observing employees collaborating to minimize risk exposure as he walked through the facility. The employees were washing a very large piece of equipment and while one removed standing water, two others were maneuvering the fifty-foot hose. On seeing this, the manager stopped to tell the employees and their direct supervisor how pleased he was to see them working together safely. To the employees, this kind of recognition was unprecedented. “In another work area, this same manager saw an employee putting himself in harm’s way. In his update, the manager described immediately stopping the employee due to the risks present. The manager expressed appreciation that the employee was trying to complete a job but explained that his personal safety came first. The manager asked the worker to get assistance from a co-worker for the time being but committed to working on a safe alternative for the employee to use in the future. Again, this kind of positive managerial support for safety had been rare in this organization. As part of a larger effort to improve safety the manager recorded these and other examples so that he could receive feedback from his coach and refine his own skills. “Through hundreds of examples like these, the learning transfer management system helps us make what was once an intangible result into a measurable outcome for our clients.”

CH004.indd 190

6/18/10 3:46:28 PM


191

compensate for lack of managerial engagement, poor delivery, or weak business links. The magnitude of the ultimate business benefit of training and development is the product of the complete learning experience; it is limited by the weakest link in the chain. The discussion that follows summarizes what we have learned over the past ten years about the essential elements of an effective learning transfer management system and its deployment for maximum effect. While we draw primarily on our own experience and system—since we know it best—the principles are broadly applicable to all efforts to enhance learning transfer. We offer suggestions on what to look for in a transfer management system, since there is some confusion between learning management systems (LMSs)—which support the administrative aspects of learning and many e-learning platforms—and what we are calling learning transfer management systems—which are specifically designed to enhance transfer and application post-training. Caveat emptor: Some learning management systems purport to have transfer support modules, but many are very rudimentary and incomplete or require extensive administrative overhead. Components A comprehensive learning transfer management system includes eight intercommunicating components that are linked to a common database: a scheduler, a communications module, a flexible update engine, a coaching loop, an online guidance system, a collaborative-learning capability, an administrative and security subsystem, and a management information system (see Figure D4.11). Scheduler The scheduler is the master timekeeper for Phase III and Phase IV learning activities. The scheduler needs to be flexible enough to accommodate a wide range of programs and objectives. For example, updates every two weeks strike the right balance between frequency and overkill for most training and development programs. However, monthly updates are more appropriate for six- or twelve-month personal development plans. The system should support variations in both number and timing of events based on the complete learning experience design. The scheduler also tracks and triggers other aspects of the complete experience such as notification of managers, final assessment questionnaires, and other items. Communications Subsystem The communications subsystem is notified by the scheduler when something must be communicated regarding a particular program. The communications module then selects, personalizes, and sends appropriate communications to the relevant parties. These can include reminders

CH004.indd 191

6/18/10 3:46:28 PM

192


FIGURE D4.11.

KEY COMPONENTS OF AN EFFECTIVE LEARNING TRANSFER MANAGEMENT SYSTEM

Scheduler Management Information System

Administration and Security

Communications Subsystem

DATABASE

Collaborative Learning

Flexible Update Engine

Feedback Loop Online Guidance

to participants, as well as to their managers, coaches, or other stakeholders. Participants are reminded to update their progress, reflect to stimulate continued learning, and plan actions that further their progress. Facilitators or program directors can be reminded to check on the group’s progress; managers can be sent copies of their direct reports’ objectives and their final reports; program sponsors can be sent links to a dashboard of the group’s progress; and so forth. The communications system needs to be “intelligent” so that, for example, it only sends follow-up reminders to those who have not completed assignments and so that it generates personalized messages based on the participants’ goals, managers, and coaches. The learning organization should be able to specify the timing of messages and customize any communication. Although email reminders alone have been shown to encourage followthrough, they are only a small part of the overall process.

CH004.indd 192

6/18/10 3:46:29 PM


193

Flexible Update Engine The update engine provides most of the motive power. The communications system prompts participants to update their progress and provides a link to a web application in which they are asked to stop, reflect, and record what they have done and what they have learned in the process of practicing their new capabilities (Figure D4.12).

FIGURE D4.12. A TYPICAL POST-COURSE UPDATE FOR PROGRAM PARTICIPANTS


CH004.indd 193

6/18/10 3:46:29 PM

194


We have found that three update questions are applicable to most learning programs: • What have you done to make progress toward your goal? • How do you rate the progress you have made? • What are you going to do next? Short and deceptively simple, these three questions reinforce the expectation for the participants to take action, prompt reflection on how much progress they are making, and underscore the expectation for continued action. The update engine has to be sufficiently flexible to support a wide variety of programs and learning objectives. It should be possible to readily alter the prompting questions and data collected. The system should be capable of gathering both qualitative and quantitative data and support rating scales, predefined selections Given that time pressures (one or many), numerical entry, and free - text are impediments, the entries. It should be able to ask different quesupdate engine needs to tions at various points during the transfer process, be fast and easy. as well as different questions for different types of objectives (leadership, business, communication, sales, and so on). Finally, the system should be able to support relevant Phase IV questions that help participants and their managers assess achievement and lay the foundation for the “success case” approach to program evaluation (see D6). Given that time pressures are key impediments to learning transfer, the update engine needs to be fast and easy to use. It should not require any special training or software; controls and navigation should be straightforward and intuitive. Coaching Loop Given the importance of feedback in developing expertise and

driving learning transfer, a learning transfer management system should facilitate feedback and coaching. We have designed our systems so that participants are presented with the option of requesting feedback as soon as they complete their updates. They are able to decide from whom they wish to solicit input—their managers, executive coaches, peers, or other mentors (anyone of their choosing). The feedback request includes a link that allows the coach or manager to view the participant’s objectives, actions, progress, and challenges and to respond in context. Feedback provided by a coach or manager is sent to the participant by email and added to his or her private online data so that it is available for later reference.

CH004.indd 194

6/18/10 3:46:31 PM


195

The ability to respond through the system improves the efficiency of coaching by obviating the need for both parties to be available at the same time; coaches are able to review the participants’ progress and to provide input asynchronously. This is especially helpful for widely dispersed groups of participants. According to John Bakhos, president and CEO of Grid International, “Grid is a culture change process, which starts with the change in an individual’s behavior. Follow through is essential to help individuals achieve this change. The learning transfer management system, ResultsEngine, has helped us work virtually and efficiently with our clients to provide coaching on the job and the follow-up needed to ensure that change is longterm and constant” (Bakhos, personal communication, 2010). Using this system, an instructor in a global leadership program at the London Business School was able to single-handedly coach seventy participants on four different continents. A sometimes-expressed concern is that the ability to provide written feedback might be used by some managers to avoid face-to-face meetings. In theory, the number of such meetings—which are powerful catalysts for learning transfer— might decline. This has not proven to be the case. Quite the contrary. When we compared the number of interactions between managers and direct reports in a program before and after implementing a learning transfer management system, we found that adding the system significantly increased interactions. Of the forty-four participants in each group, the number who had any discussion with their managers increased from 65 percent to 96 percent after the learning transfer management system was added to the program (without any other changes), and the number who had multiple interactions increased from 25 percent to 68 percent (Figure D4.13). Using the system to support individual development plans produced a similar increase. Online Guidance As we will discuss in greater detail in the next chapter (D5),

there is an exceptional “teachable moment” when participants are asked to plan their next steps. They are, at that specific instant, especially receptive to suggestions on how they can improve a given skill or competency. Therefore, we feel that a comprehensive learning transfer management system should include just-in-time guidance that functions as a sort of learning GPS, providing contextsensitive advice for continuing development activities. Collaborative Learning There is growing recognition of the power of social

networking and interest in effectively harnessing it to support collaborative learning. Peer support is one of the factors in the learning transfer model (Figure D4.8) and so creating and sustaining a “community of learners” can be a powerful contributor to the efficacy of learning transfer. A learning transfer management

CH004.indd 195

6/18/10 3:46:32 PM

196


FIGURE D4.13. MANY MORE PARTICIPANTS HAD MULTIPLE DISCUSSIONS WITH THEIR MANAGERS AFTER IMPLEMENTING A LEARNING TRANSFER MANAGEMENT SYSTEM

80% Percent of Participants

70% 60% 50% 40% 30% 20% 10% 0% With Support System

Without Support System

system can facilitate informal learning among participants by allowing them to see others’ reflections and updates and encouraging ongoing communication and peer coaching. Fellow participants and peers have immediate credibility because they have shared the same experience and face similar challenges. As one of the participants in a program for senior executives in nursing wrote: “I learned the most from scrolling through the thoughtful responses of others.” Professor Beta Mannix of Cornell’s Johnson Graduate School of Management said: “It’s just amazing to me how much information the peer groups share with each other, from small [tips] on how to handle difficult employees, to strategic initiatives, to tracking down an article” (Wick, 2003). At the same time, the system must be sophisticated enough not to divulge private information, such as personal feedback. Ideally, it should be possible to turn off sharing for specific kinds of goals or programs.

CH004.indd 196

6/18/10 3:46:32 PM


197

We have also found that “co-opetition”—a blend of cooperation and competition in the group—adds a mild level of accountability that spurs transfer efforts. So we provide a bar graph, visible to all the members of a class, that plots the relative progress reported by each participant (Figure D4.14). Since most people like to be seen as leaders rather than laggards, the graph is an incentive to continue to strive to make progress. At the same time, it helps identify those who are making good progress and could be consulted for advice (cooperative learning). FIGURE D4.14. A BAR GRAPH OF REPORTED PROGRESS ENCOURAGES CO-OPETITION


CH004.indd 197

6/18/10 3:46:33 PM

198


Administrative and Security Subsystem A learning transfer management

system, like any IT system, requires an administrative “back end” that allows designated administrators to set up groups, enroll or delete participants, create teams, manage permission levels, and so forth. Participants’ access should be passwordprotected and the site limited to members of their cohort; the security system must be sufficiently robust to prevent “hacking” or other unauthorized access. Since most learning organizations are already “lean,” with limited administrative support, we believe that an effective system should impose minimal additional administrative burdens on the learning organization and that in-depth support should be provided by the supplier. Management Information System The real power of learning transfer

management systems is the insights that they provide learning and development professionals into Phase III activities and progress—a learning phase that one of our clients characterized as having been historically a “black hole.” Jayne Johnson of GE pointed out that without some sort of transfer support system, most learning organizations “have no idea” what learners do once they leave a course. Program directors, managers, and facilitators should be provided with a “dashboard” of the key metrics, such as participation rates, goal distribution, progress indicators, and achievements. The system should provide the ability to quickly “drill down,” sort, and analyze the details for individuals, groups, and company-wide programs. An example of a learning leader’s dashboard is shown in Figure D4.15. Analogous to financial management information systems, the learning transfer management system dashboard should facilitate “management by exception.” It should make it easy to identify and recognize (recognition being a key driver of motivation) those making real progress. It should also highlight those who are not following through for special attention (recall that negative consequences for not applying the learning are part of an effective transfer climate). The dashboard should provide early warning for programs that are “off track” to permit timely intervention—the essence of management. Program designers should be able to efficiently review the quality and distribution of learning transfer goals to gauge the extent to which these match the overall course objectives. Learning officers, program directors, instructors, and others should be able to mine the data, generate reports to support continuous improvement, and document the business impact of the initiative (D6). The system should also allow selected professionals to monitor the quality and quantity of feedback being provided, because feedback and coaching have such a profound impact on the learning transfer climate.

CH004.indd 198

6/18/10 3:46:34 PM


199

Finally, the last update of any cycle should complete Phase IV of the learning process by asking participants to assess their achievements and, ideally, having managers triangulate the results. The transfer management system should make the results easily accessible to support program evaluation as well as continuous improvement (see D6). FIGURE D4.15. AN EXAMPLE OF A LEARNING LEADER’S DASHBOARD IN A LEARNING TRANSFER MANAGEMENT SYSTEM


CH004.indd 199

6/18/10 3:46:35 PM

200


Better Return on Learning While well-designed learning transfer management systems are easy to implement and relatively inexpensive—especially when compared to the true cost of learning scrap—they nevertheless incur costs of time Learning transfer management and money that must, like training itself, more than pay for themselves in improved performance. systems more than pay for We have now worked with hundreds of compathemselves in improved nies, partners, and business schools to implement performance. support for learning transfer following both courses and individual development plans (see Case in Point D4.2). In most cases, the results have been a substantive improvement in the transfer climate and outcomes. As the examples below illustrate, management of the Phase III learning transfer process generates greater effort to apply learning, more rapid progress, improved application, and increased return on investment from essentially the same learning investment. Both companies and participants have benefited.

Case-Control Studies One of the earliest studies we did to assess the impact of learning transfer management was done in conjunction with Sun Microsystems’ Sun University (Sun U). We selected the Sun New Manager Course because of its importance to Sun University’s efforts to create competitive advantage through human capital development. The program’s intent was to introduce newly appointed managers to key concepts of management and leadership and to Sun’s expectations for team leaders and managers. Sun U leadership wanted to test the value of adding learning transfer support to this already successful program. We used a case-control design. Four sessions of the Sun New Manager Course were taught to eighty-eight participants (twenty-two per group) with the same materials and curriculum by the same instructor. In two sessions involving a total of forty-four participants, the Friday5s learning transfer management system was added to Phase III. The other two sessions were conducted as usual (without specific attention to follow-through). Approximately three months after the class, all the participants and their managers were polled anonymously via a web-based survey. Results The addition of the learning transfer management system increased

managers’ awareness of their direct reports’ learning transfer goals. Managers of those in the learning transfer support group were significantly more aware of

CH004.indd 200

6/18/10 3:46:36 PM


201

Case in Point D4.2 Don’t Confuse Learning with Changing Behavior ASK Europe, plc. helps organizations improve workplace performance through organizational development, management and leadership development, and executive coaching. Established in 1994, ASK’s high-impact learning interventions have created value for organizations across the world. Since 2004, ASK has been a leader in utilizing the best practices in transfer and application to ensure new learning and behaviors are not merely acquired, but become embedded within individuals’ and organizations’ daily practices. According to managing director, Robert Terry: “Habits are easy; change is uncomfortable. Learning is a necessary but insufficient precondition for improving workplace performance. The existence of the ’transfer issue’ was first formally acknowledged by Mosel in 1957. Since then researchers and academics have expanded greatly our understanding of the ’transfer problem’ and its possible solutions, but the level of take-up of that new knowledge in the practitioner community has been disappointing. Training without transfer limits the extent to which new skills are applied in the workplace and impairs their sustainability. That is why we have worked to include the Six Disciplines and support for learning transfer in all our work.” For example, ASK has been working with a global financial services organization to develop its senior leadership since 2006. Between November 2006 and December 2009, 135 senior managers completed the program, including a rigorous follow-through process using ASK’s learning transfer support system (ASK Elephant) to support, monitor, and evaluate the outcomes. Of these, all 135 reported that they were “more effective” or “much more effective” after the program. Two-thirds also reported that the business impact of the program had been “significant” or “very significant.” The organization has grown rapidly during this period, including several successful acquisitions; the program is credited with helping to create a single dynamic leadership community among the senior managers of the parent and acquired companies. “We have found that having a system for follow-through to development programs provides opportunities for continuing learning, creates a useful environment within which learners can provide mutual support and encouragement, and emphasizes the importance of time in changing workplace behavior,” said Terry.

their direct reports’ learning transfer objectives than those in the control group. In the absence of the follow-through system, less than half (40 percent) of the managers claimed they were aware of their direct reports’ goals. In contrast, 100 percent of the managers of the follow-through group reported being aware of their direct reports’ goals (see Figure D3.14). As shown in Figure D4.13, participants in the sessions that included the learning transfer management system were much more likely to discuss the course and

CH004.indd 201

6/18/10 3:46:37 PM

202


their efforts to apply it with their managers. Managers also reported observing more post-course effort in the learning transfer support group, compared to the group in which transfer was left to individual initiative. While participants and their managers in all four groups felt that the program had contributed to improved managerial effectiveness, the improvement ratings by both participants and their managers were higher in the transfer management systems groups. These results support Baldwin and Ford’s (1988) contention that transfer of learning depends on more than the course content and its presentation. In this instance, the program agenda and content as well as the instructor were the same for all four groups. The immediate end-ofclass evaluations were virtually identical for the Transfer of learning depends groups with and without transfer support. Yet, on more than the course three months later, the change in effectiveness was content and its presentation. greater for those in the learning transfer management system group. In other words, it is possible to increase the effectiveness of a program by adding learning transfer support, without changing the program itself. Comments from the participants, the instructor, and course coordinator also underscored the value of a transfer management system: • “Best thing was the reminder that I had specific goals and needed to work on them every week.” (Program participant) • “I enjoyed the instructor feedback. I truly believe [my] skills were molded by this course.” (Program participant) • “Overall, the best part was reading how others were doing. It’s good to know others are in the same position you are and are dealing with the same problems you are.” (Program participant) • “Holds people accountable, tracks participant actions, and provides evidence of results. Positive impacts back on the job have been clear across the board with participants who committed to the process. Triggers prompting participants to update are critical.” (Instructor) • “The system was most helpful in supporting course participants to take postcourse action. The impact of their actions was easily captured and can readily be used as Level 3 evaluation data and evidence. As a course manager, this process really helps in documenting and assessing the real-world effectiveness of the course and its design.” (Course manager) Positive results were also observed in a comparative study conducted by Laurie Cusic at Wyeth Pharmaceuticals (Cusic, 2009). Two groups were compared: sales

CH004.indd 202

6/18/10 3:46:37 PM


203

managers who had attended a Transformational Coaching for Managers program with and without the addition of the ResultsEngine learning transfer management system. Overall, the program significantly improved the coaching skills of managers and appeared to have contributed to a higher level of sales goal attainment for the company. The learning transfer support tool proved to be effective in maintaining participants’ engagement and connection to the learning, which seemed to be a contributing factor to improved sales goal attainment the following quarter. The value of a learning transfer management system to improve execution of individual development plans was tested in a Fortune 50 company. The company had implemented a process to support the development of the behaviors it expected of its leaders. The process included a customized 360-degree feedback instrument, personal coaches, and the creation of individual development plans for each leader. To test the value of adding a learning transfer management system, two groups of director-level managers were compared. All managers completed the 360-degree feedback process, were assigned coaches, and were encouraged to discuss their goals and efforts with their managers. In addition, one of the two groups was provided with access to a learning transfer management system called DevelopmentEngine, which they used to track their progress every three weeks for fifteen weeks. At the end of that period, all the participants were surveyed and the results analyzed. Adding the learning transfer management system contributed positively to the development and coaching process. It significantly increased the amount of interaction that participants had with their managers and coaches (Figure D4.16). Every participant in the learning transfer support group indicated that using DevelopmentEngine had prompted them to have more discussions with their coach or manager and that it had helped them stay focused on their continued development. The participants all agreed that the support system had facilitated their progress and highly rated both its value and ease of use. Over 80 percent wanted DevelopmentEngine made available to their direct reports. Their free-text entries supported its value: • “An excellent prompt system. Brings discipline to the process.” • “Keeps you on track to carry out the actions agreed, otherwise they would be forgotten.” • “Forced you to stay focused on the commitments given in response to the 360degree feedback.” • “I would not hesitate to use it again and would strongly commend its continued use. Equally, it did not pose a significant time challenge.”

CH004.indd 203

6/18/10 3:46:38 PM

204


FIGURE D4.16. A LEARNING TRANSFER MANAGEMENT SYSTEM ALSO INCREASES INTERACTIONS WITH BOTH COACHES AND MANAGERS REGARDING INDIVIDUAL DEVELOPMENT PLANS

10 8 6 4 2 0 Avg. interactions with manager Control

Avg. interactions with coach DevelopmentEngine

Reducing Waste at Hewlett-Packard The results of the Dynamic Leadership Program at Hewlett-Packard were reported in the Journal of Organizational Excellence (Connolly & Burnett, 2003). The authors identified six factors that were key to the documented success of the program: • • • • • •

Strong connection with business imperatives Commitment to creating business value Use of program tools and concepts by the project team itself Fast action and rapid adjustment Partnership between content experts and line managers Reinforcement and measurement

The design team recognized that a two-day event was insufficient to produce the desired change. They decided to extend the learning period for ten weeks by incorporating learning transfer support as a core component of the program. The Phase III process of the Dynamic Leadership Program is shown in Figure D4.17. The authors concluded: “ The post - workshop support system created a reinforcing cycle of follow-up action, coaching, feedback, and measurement to assure that participants were learning and manifesting the desired new

CH004.indd 204

6/18/10 3:46:38 PM


205

FIGURE D4.17. LEARNING TRANSFER SUPPORT PROCESS USED IN H-P’S DYNAMIC LEADERSHIP PROGRAM

Course

Participants learn new skills and set objectives

Alignment

Objectives sent to their managers for discussion

Reminder

Participants reminded by email to update progress

x5

Coaching

Update

Participants update their progress in Friday5s

Boss, peers, or instructors provide online advice/counsel

Follow-Through Process

Ask for Advice

Copy sent to coach or manager for feedback

Learn More

Learning continues by reviewing others’ progress

Documented Results

User input documents impact and provides data to improve next offering Source: Connolly and Burnett, 2003. Used with permission.

CH004.indd 205

6/18/10 3:46:39 PM

206


behaviors and that the new behaviors were producing the desired business value” (Connolly & Burnett, 2003). Increased Return on Investment Direct evidence of the economic impact of learning transfer management comes from a study at Pfizer—the world’s largest pharmaceutical company and the first firm ever named Best Training Company by Training magazine two years in a row (Schettler, 2003). An important factor in Pfizer’s top ranking was its rigorous approach to evaluating training effectiveness. Pfizer’s Learning Center maintained a Department of Measurement, Evaluation, and Strategic Analysis (MESA) specifically for this purpose. Approximately six months after the learning transfer management system was implemented, Pfizer’s MESA group conducted an in-depth return on investment (ROI) analysis of the leadership program using the methodology recommended by Phillips (2003). Managers who had attended the course were asked to quantify the benefits in financial terms and indicate both their degree of confidence in the estimate as well as the percentage of the benefit that could be directly ascribed to the program. Over 140 responses were available for analysis. A conservative estimate of the actual benefit was obtained by multiplying: Estimate of financial value percent confidence percent attributed to program Participants overwhelmingly endorsed the value of the program in helping them improve their managerial and leadership skills. The overall ROI was estimated to be 150 percent. In addition, numerous other benefits were cited that could not be directly dollarized. Since some of the respondents had attended the program before the learning transfer management system was implemented, it was possible to compare the reported ROI with and without the system. Adding learning transfer support increased the return on the same training investment. Managers who had attended sessions Adding learning transfer with transfer support reported dollarized bensupport increased the efits nearly 50 percent greater than those who return on the same training had attended sessions without follow-through— investment. about $14,000 additional value per participant (Trainor, 2004). A positive impact on business outcomes was also seen at UPS (see Case in Point D4.3). These results further underscore the power of learning transfer management systems to increase the return on training. They demonstrate that

CH004.indd 206

6/18/10 3:46:39 PM


207

Case in Point D4.3 Extra Value from Follow-Through UPS is the world’s largest package delivery company and a leading global provider of specialized transportation and logistics services. Training is essential to maintain UPS’s exceptional level of service and efficiency. To test the effect of soft skills training on business outcomes, UPS tracked the results of over 1,500 managers who attended its Building Relationships workshop. The results were compared to fifty-two managers who attended the initial three workshops that included the Friday5s learning transfer management system. The goals of the workshop are to provide managers with skills and techniques to help them improve the work environment, handle objections, and deal with issues in a constructive way to motivate their employees. The relationship training alone resulted in improvement of key efficiency metrics—demonstrating that soft skills training can produce demonstrable business outcomes. Those groups that had the added support for learning transfer produced greater improvement in cost savings, performance, and safety. The difference translated into substantial additional savings for the company. When asked about the value of the training and transfer process twelve weeks after the workshop, one of the participants responded: “priceless.” He explained: “My employees trust me and my management team even more and they believe that we are going to do what it is we say that we are going to do. My attendance has improved and so has my employee retention.” “Our goal of linking business relationships to business results was achieved sooner as a result of applying the Friday5s learning transfer management system. The key to success was the active coaching performed by our managers,” said Jerry Frasso, UPS employee relations manager.

the increased effort, management discussions, and follow-through observed in the Sun study are converted to measurable productivity gains in the workplace. According to Van Potter, senior learning consultant at Fidelity Investments, “We want evidence that learning impacts thinking, behavior, and results . . . and that it leads to a shift that makes the business more successful. With managed follow-through, we have the data to quantify our results and put a dollar figure on the [ROI]” (Wick, 2003). Insights Into Development Laura Santana (2009) used information from the Center for Creative Leadership’s learning transfer management system (Friday5s) to answer some key questions about development and dispel some common myths. She selected for study 248 individuals who had attended the Center’s Leadership Development Program

CH004.indd 207

6/18/10 3:46:40 PM

208


(LDP) and who had indicated through their Friday5s reports that they had completed one or more of their personal development goals. What she found was evidence of both “horizontal” development—integration of new knowledge and skills into existing frameworks—as well as “vertical” development, which requires re-organizing existing knowledge and beliefs to make sense of new information and deal with ambiguity. She was also able to dispel the myth that development affects only the individual; there was clear evidence of learning transfer that had interpersonal, team, and organizational impact. Indeed, even though the focus of the LDP program is on individuals, more than half the reported impact extended well beyond the participants. Santana concluded: “We have seen how a five-day plus online follow-through initiative leads to development of human capital, social capital, collective intelligence, and expanded consciousness through vertical development” (p. 208).

Implementing Learning Transfer Management Well-designed learning transfer management systems can be implemented quickly and easily. Implementing them effectively, however, requires making the commitment, selecting a system, putting it in place, and “learning on the fly,” that is, improving and perfecting the process over time. The first step is making the commitment. We hope that by now you are persuaded that learning transfer support offers exciting opportunities to enhance the output of already excellent programs and that you are ready to commit to doing more to support Phase III learning. We hope that you are equally persuaded that, to be effective, learning transfer support needs to be part of an integrated process based on execution of the 6Ds. Issues that need to be considered include: how the concept will be introduced to participants; how facilitators will be prepared to explain the process and its payoff; how support and active engagement of managers will be secured; who in the learning organization will champion the process, check on progress, recommend action; and so forth. A credible supplier of learning transfer management systems should be able to provide advice and consulting on these and other implementation matters. The second step is to select a system. While some aspects of learning support could be implemented using manual systems, aspects of an LMS, or possibly an electronic evaluation system, achieving a breakthrough in learning transfer requires implementing a thoroughgoing purpose-built solution. A key decision, as with any information management system, is “make or buy.” The advantage of building your own custom system is that it can be specifically tailored to the needs, culture, and objectives of the organization. The disadvantages include the time required to design, build, test, and maintain the system.

CH004.indd 208

6/18/10 3:46:40 PM


209

Cost savings of “build it yourself ” systems often prove illusory when the true costs of design, programming, and ongoing maintenance are taken into account. The advantages of commercial systems are (1) they represent the accumulated best practices of experience from many different companies and (2) providers are driven by competition to continually upgrade and improve them. If the decision is taken to employ a commercial solution, the challenge becomes one of selecting the best system and vendor. The usual questions of track record, experience, and reputation apply. The selection process should include vetting the provider by consulting with respected current users. In addition, the choice has to be made between a full-service provider, who will manage many of the administrative functions (such as customizing the application, enrolling participants, entering goals, providing ongoing support, help desk, and so forth) and a “do-it-yourself ” approach in which these functions must be performed by your own learning and development staff. Once the system and target program have been selected, it is time to “just do it”—to put learning transfer support into place and begin the process of continuously learning and improving the use of this powerful new tool of breakthrough learning. Finally, whether additional value was realized as a result of adding learning transfer support needs to be evaluated. It’s important that the criteria for evaluation reflect the business outcomes for which the program was created (see D6), not whether the participants “liked” the process. Few people “like” being held accountable for delivering on their commitments, preparing performance reports, and so on, but these activities are essential to ensure that the company “gets its money’s worth” from training and development, as they are for other investments.

Summary For at least fifty years, improving learning transfer has been recognized as key to converting more of learning into business outcomes. No matter how superb the instruction, e-learning modules, simulations, or other learning experiences, unless new knowledge and skills are transferred to and used on the job, they are nothing more than costly learning scrap. In today’s competitive environment, no company can afford to produce scrap in any aspect of its operations, be they manufacturing defects, service failures, or unused learning. The true cost of doing nothing to improve learning transfer is more than any company can afford. The root causes of learning scrap are numerous and originate both within and outside the learning organization. Therefore, the solution requires a thoroughgoing approach and cooperation between the learning organization and line management. Improving learning transfer (reducing learning scrap) begins with program design and

CH004.indd 209

6/18/10 3:46:41 PM

210


Phase I preparation, continues through instruction, and most importantly, includes systems and processes to support and drive deliberate practice in the workplace. In well-managed companies, systems are in place to ensure that business objectives are implemented, progress is monitored, and achievement is rewarded. To maximize the value of training and development, companies must implement similar systems to ensure that learning transfer objectives are similarly implemented, monitored, and rewarded. Best practice learning organizations drive learning transfer by actively managing the follow-through process. They use systems and procedures to ensure that participants put their learning to work by setting expectations, issuing reminders, ensuring accountability, and providing support. Companies that have implemented learning transfer management systems have experienced significantly higher levels of post-course effort, achievement, and return on investment for a modest incremental investment. The advent of efficient and effective learning transfer support systems represents a true breakthrough in corporate education and the single greatest opportunity to increase its value. Use the checklist for D4 to help ensure that you have a robust plan to ensure learning transfer and application to sustain the value of the learning experience.

CHECKLIST FOR D4

CH004.indd 210

✓ ❑

Element

Criterion

❑

Goals

Participants set, or are provided with, strong stretch-goals that require learning transfer and application to achieve.

❑

Reminders

Learning is kept top-of-mind through periodic reminders about the program’s content, the participant’s personal goals and objectives, and the need to continue to practice new knowledge and skills.

❑

Accountability for Managers

Managers are reminded of the program’s objectives and informed of their direct reports’ personal goals for application.

❑

Accountability for Participants

Learners’ objectives and progress are made public—at least to their cohort of learners and their managers— similar to business goals and progress.

❑

New Finish Line

There is a mechanism and pre-determined reporting schedule to underscore the need for action and reflection that includes a defined endpoint and method to assess achievement.

❑

Recognition

Appropriate recognition is provided for those who make great progress and/or complete their objectives.

6/18/10 3:46:42 PM


211

Action Points For Learning Leaders • Answer the following questions for each of the key programs that your group delivers: • What is your organization doing to drive learning transfer in Phase III? • Do you know what the participants’ learning transfer objectives are? • Do you actively remind and support them? • Are managers actively involved in supporting learning transfer? • Do you have systems in place to manage follow-through, transfer, and application? • Based on your answers, develop a plan to take much greater ownership of what happens during Phase III, since you will be held accountable for the results. • Explore methods and systems to help support learning transfer. • Have a candid discussion with senior management about how the participants’ managers can enhance or destroy the value of learning. • Work with management to ensure that learning transfer occurs. It requires a team effort. Learning and development cannot single-handedly drive learning transfer, neither can it eschew its responsibility because, as Eldridge Cleaver put it: “you’re either part of the solution or you’re part of the problem.” For Line Leaders • Reflect on your own experiences in learning and development programs. Were you expected to follow through and generate a return on the company’s investment? Or was the last day of class treated as though it were the finish line? • Interview employees in your organization who have recently attended programs or who have had direct reports attend programs. • Are developmental objectives taken seriously in your unit or not? • Is there a culture of execution or a culture of indifference? • Interview the managers of participants in recent programs. • Were they aware of the program’s business objectives? • Did they know what their direct reports’ personal objectives were? • Did they hold their direct reports accountable for demonstrating a return on the educational investment? • If you discover that developmental objectives are afforded “second-class citizenship ” and are frequently ignored by both participants and their

CH004.indd 211

6/18/10 3:46:42 PM

212


managers, you are wasting time and money on training. Work with the learning organization to address the problem. • Exert leadership to ensure that program participants are held accountable for following through and transferring their learning to the work of the firm in a way that improves their personal performance and the business’s results. • Make sure that there is some form of recognition for those who make significant progress on their developmental objectives; learning agility is a key attribute of future leaders.

CH004.indd 212

6/18/10 3:46:43 PM

DEPLOY PERFORMANCE SUPPORT The problem, my research suggests, is what happens when a manager comes back to the day-to-day routine of the office. —HERMINIA IBARRA

I

n the previous chapter, we discussed the importance of having a system of accountability to achieve a breakthrough in the amount of learning that is applied on the job. In this chapter, we explore the other half of the equation: providing post-training performance support to help participants bridge the learningdoing gap. The best results are obtained when both accountability and support are present and in balance (Kirkpatrick & Kirkpatrick, 2005). Breakthrough learning and development organizations ensure that participants have the resources they need to succeed in putting new learning to work. In the preface to his book, The First 90 Days: Critical Success Strategies for New Leaders at All Levels, Michael Watkins (2003) explains what piqued his interest in the topic. “I was struck by how few companies invested in helping their precious leadership assets succeed during transitions—arguably the most critical junctures in their careers. Why did companies leave their people to sink or swim? What would it be worth to companies if managers entering critical new positions could take charge faster?” (p. xii). The same questions pertain to corporate education. What would it be worth to companies if employees applied new learning to their work faster and more effectively? Why do so many companies leave learners to sink or swim instead of investing to help them bridge the gap between the classroom and the workplace? Having already invested time, effort, and dollars to impart new skills and knowledge,

213

CH005.indd 213

6/17/10 3:29:27 PM

214


why don’t more companies make the modest incremental investment to ensure application? Our experience across hundreds of initiatives confirms that when learning and development programs do include ongoing support for learners—especially during the first few crucial weeks of the transfer phase—they enjoy a greater return on their educational investments. That is why the Fifth Discipline is to provide performance support for learning transfer. Topics in this chapter include: • • • • • •

Analogy to customer and product support Materials Systems People A checklist for D5 Action points for learning and line leaders

Analogy to Customer and Product Support We recently purchased a new refrigerator, not a terribly complicated or difficult product to use. Nevertheless, the manufacturer included a detailed user’s manual and troubleshooting guide, access to an online help site, a toll-free product support number, and even live online chat with a specialist. Obviously, the manufacturer understood that customer satisfaction is the result of the “ whole product ” experience—which includes the quality of support as well as the quality of the product itself (see Figure D5.1). Customer satisfaction is a leading indicator of future success because a satisfied customer is inclined to purchase additional products or services; he or she will also recommend the brand to family and friends. A dissatisfied customer, on the other hand, will not only refuse to buy more products or services, but is likely to voice dissatisfaction to anyone who will listen. So keeping customers happy is essential for long-term success. But few things frustrate customers more than being unable to figure out how to make something work or having a problem and being unable to obtain a clear and timely answer. That’s why successful companies strive to provide exceptional customer support. We realized there is a strong analogy to corporate education. A sponsor’s satisfaction with a training and development program depends on whether or not performance actually improves. A participant’s satisfaction depends on the complete learning experience which, as we pointed out in D2, includes much more than just the classroom or e-learning. Ultimately, customers’ satisfaction with training depends on whether or not the program delivers on its promise of helping

CH005.indd 214

6/17/10 3:29:28 PM

Deploy Performance Support

215

FIGURE D5.1. CUSTOMER SATISFACTION DEPENDS ON THE WHOLE PRODUCT EXPERIENCE, WHICH INCLUDES AVAILABILITY AND QUALITY OF SUPPORT

Customer support

Product quality

Customer satisfaction

Results vs. expectations

Ease of use

participants improve their performance and achieve personally important goals. Achievement requires being able to successfully utilize what they learned, which, in part, depends on whether they can get assistance if they need it. Corporate training and development programs aim to teach people to do things in new and better ways. Given that, we should not be surprised that learners sometimes experience difficulties when they first begin to try out the new approaches on their own. Everything taught in corporate training and development programs is a great deal more complicated than plugging in a refrigerator, The goal of corporate training and yet the corresponding support is typiand development is to teach cally an order of magnitude less, if there people to do things in new is any at all. As with consumer product and better ways. satisfaction, a satisfied learner—and his or her manager—will be inclined to pursue more training and recommend the program to others. Dissatisfied learners—and, in particular, dissatisfied managers who failed to see any improvement as a result of the investment—are likely to voice their dissatisfaction widely and loudly. Such negative publicity undermines support for training and development and makes

CH005.indd 215

6/17/10 3:29:30 PM

216


it even more difficult for the learning organization to fulfill its mission in the future. So it is in everyone’s best interest—the learners’, the company’s, and the learning organization’s—to provide effective support for performance, especially during the critical learning transfer phase. But what does effective performance support for learning transfer look like and what options are available? Rossett and Schafer (2007) defined performance support as: “A helper in life and work . . . a repository for information, processes, and perspectives that inform and guide planning and action” (p. 2). Key to the definition are the concepts that performance support needs to be readily available in time and place and that it must contain an easily accessible repository of relevant information, processes, and wisdom to inform planning and guide action. Performance support is especially useful when people are trying to master a new skill because it reduces cognitive load, allowing novices to concentrate shortterm (working) memory on performing the task, rather than trying to simultaneously remember it (Clark, Nguyen, & Sweller, 2006). As Clark (1986) wrote: Designers should encourage learners to use working memory to process information, not to store it. For example, as learners first practice a new procedure, give them access to clear, written, summary steps for reference so that all working memory can be directed toward executing the procedure. The use of job aids, in the form of a written procedure table . . . can be especially powerful for this purpose. With enough repetitions of the task, it will become automatic and bypass working memory. Then the job aid will become unnecessary (p. 19).

A broad range of approaches and media can fulfill the definition of performance support: an experienced co-worker, a credit-card-sized job aid, a computerbased or cell phone application, even a posted sticky note—anyone or anything that is able to deliver the needed information, procedure, or suggestion where and when it is needed. Performance support can be classified into three general types: materials, systems, and people, although technology is beginning to blur the boundaries. The most effective programs use some combination of all three.

Materials Various kinds of printed material—from step-by-step procedures, to checklists, to wall posters, to magnets—can be useful job aids and a form of performance support. For example, one of us has carried a wallet card with the four steps of

CH005.indd 216

6/17/10 3:29:30 PM


217

opening a sales call for years; he still uses it to refresh his memory before starting an important meeting. The general purpose of a job aid is as an external memory storage device to help the user recall specific steps in a procedure, specific information needed to perform a task, and so forth, without relying on (notoriously fallible) human memory. For that reason, job aids are especially useful when: • • • •

Trying to master a new procedure or skill Performing an infrequently used procedure Many steps are involved There is no room for error (as in a pre-flight checklist or operating room procedure)

The reason that products come with owner’s manuals is that manufacturers know that even though a feature may have been demonstrated in the showroom, the consumer probably won’t remember how to use it days or weeks later. And if consumers can’t figure out how to use something, they won’t use it, and they will be unhappy with their purchase. So manufacturers invest significant time If companies provide guides and energy putting together and testing for refrigerators, shouldn’t product manuals to make sure that they we do the same for learning are easy to use and that they answer the programs? most frequently asked questions. A welldesigned user’s manual is not merely a compendium of product information; it anticipates the kind of difficulties a user might encounter and provides practical advice on what to do. More and more, such users’ guides are built into the device itself to ensure that they are immediately available where and when they are needed. If companies provide guides to using everything from tractors to software to copiers—even refrigerators—shouldn’t we do the same for learning programs? Participants in corporate education programs often struggle to translate what they learned into concrete actions once they return to their jobs. What seemed clear during the classroom exercise is less clear in the hurly-burly of day-to-day work. As Allison Rossett, professor of educational technology at San Diego State University, explains: “The literature is pretty clear that most people are not particularly good as independent learners or performers. They don’t know when they need another example; they don’t know when they are ready to test. So we try to pack it all in ‘class,’ but then they go out and things get crazy. And they don’t know how to do what they are supposed to do. If they could do it in class, it doesn’t at all mean they’re going to be able to do it out there on the job, or that they will elect to

CH005.indd 217

6/17/10 3:29:31 PM

218


do it” (Rossett, personal communication, 2009). That is why participants benefit from guides that remind them how to apply course principles, jog their memories, provide checklists for procedures, or suggest ways to approach a problem. There are ample opportunities to create inexpensive and effective job aids for corporate training and development programs. Job aids should be part of the design checklist—an integral part of planning the complete learning experience. Instructional designers should ask themselves: “In performing this skill or procedure on the job, especially the first few times, what might be hard to remember?” and then design a job aid or other support tool to help participants overcome that barrier to performance. To be effective a job aid needs to be • • • • •

Readily available at the time and place it is needed Simple Quick Relevant Specific (no extraneous information)

Program binders aren’t job aids. They are designed to support instruction, rather than provide performance support for transfer and application. They tend to be big and cumbersome, and it is difficult to locate specific topics or suggestions for action in them. That may be why most of the program participants we interviewed said they rarely refer to their program binders once the course is over. Job aids need to be specifically designed to meet the criteria above if they are to be effective in supporting tasks and skill application. Past participants are a good source of effective job aids: What are they using to help them apply what they learned? Some will have created clever mnemonics or “cheat sheets” that contain key information to help them remember and perform better. Since these have already been “field-tested” for utility, they are likely to be effective for other participants as well. Examples of simple but effective job aids include: • Wallet cards, Post-it Notes, magnets, etc., with the key steps in a procedure or approach • A pre-call planning checklist for representatives • Posters with strong, simple, memorable graphics of a particular topic • A list of questions to help an employee reflect on his or her performance • A small notebook with tabbed pages for key procedures • A cube with the one of the 6Ds on each face • A rubric to help managers evaluate learning transfer goals (Jefferson, Pollock, & Wick, 2009, p. 37)

CH005.indd 218

6/17/10 3:29:31 PM


219

A principal value of job aids is that they make expectations explicit and they reinforce them at the time of action. Allison Rossett explains: There are various forms of delivery, but clarity of expectation is to me the most powerful. The neuroscience literature and the new things that are coming out now suggest that if it’s really clear to people what is expected of them, they will go there. Let me give you a very simple example. I was doing some work for IBM a few years back when they were trying to increase customer focus. On the back of their nametags, they had six or seven things that they should think about before they go to meet with customers, just a reminder. Great! I think it’s great. It’s the most simple delivery, but that’s what I mean when I say you’ve got to deliver closer to where it’s needed (Rossett, personal communication, 2009).

In short, the value delivered by learning programs of any type can be enhanced by the addition of well designed and use - tested job aids that deliver simple, relevant, specific guidance on what is required to do the job well.

The value delivered by learning programs can be enhanced by job aids.

Systems The most exciting developments in post-training performance support have been the result of advances in technology. Entirely new forms of performance support are now possible. These new systems provide richer, more portable, more specific, and more personal guidance than ever before. A now-familiar example is the revolution that has occurred in getting directions. Until a few years ago, if you wanted to go from one place to another, you bought a map of the area and worked out your own route. Alternatively, you asked someone knowledgeable about the area for directions and copied them down. That worked pretty well until you took a wrong turn, were forced to detour, or the roads had changed. Then trying to get back on the right highway could be a nightmare. Now, as a result of GPS and computerized maps that include almost every road in every country, you can put in where you want to go and get step-bystep instructions for the route to take. If you deviate from the original plan, either intentionally or unintentionally, the system will rapidly recalculate what you need to do to get back on the right path.

CH005.indd 219

6/17/10 3:29:32 PM

220


Using the metaphor of personal development as a journey, we should aim for performance support tools that would perform an analogous function: allow participants to specify their destination and then guide them in selecting the best routes, including alternatives and re-routing if they get off track. The ability to use technology to inform and guide planning and action is especially well-suited to situations in which: • Performance depends on access to a very large body of information (knowing every road in the country, for example) • The information or procedures change frequently (processing insurance claims or checking for drug interactions) • The information needs to be personalized (such as to meet your personal financial or developmental goals) Electronic Performance Support More and more companies are using the Internet to provide first-line customer and product support. Online user guides can be more comprehensive, more readily updated, more easily expanded, and more quickly searched than printed versions; they are continuously available to virtually everyone through the World Wide Web. Moreover, the beauty of hypertext transfer protocol (http) and modern search engines is that they allow users to follow their unique approach to the topic; many different starting points and paths can link to the same resource (see Figure D5.2). Making performance support for corporate education available online provides similar advantages. It also makes it possible to offer customized content for specific learning objectives and to employ company-specific terms and concepts. Realizing the full potential of electronic performance support systems, however, requires more than simply offering electronic copies of books and binders. As with job aids, online performance support systems need to be purpose-built to be effective. Rossett and Schafer (2007) defined eight criteria for great performance management systems (Exhibit D5.1). Brief explanations, application to learning transfer, and examples are provided below. 1. Support Achievement of Important Business Objectives As we have stressed

throughout our discussion of the Six Disciplines, learning transfer objectives are (or at least should be) important business objectives. Programs should be designed from the onset to support business needs and prioritized by their relative contribution to the mission of the organization. When this is done, then the post-training

CH005.indd 220

6/17/10 3:29:32 PM


221

FIGURE D5.2. ELECTRONIC PERFORMANCE SUPPORT SYSTEMS ALLOW USERS TO FIND RELEVANT INFORMATION FROM DIFFERENT POINTS OF VIEW

Information about applying the concept of situational leadership

Exhibit D5.1 Eight Principles of Great Performance Support Systems Great performance management systems: 1. 2. 3. 4. 5. 6. 7. 8.

Support achievement of important business objectives. Help users define, track, and achieve goals. Focus on what really differentiates superior performance. Recognize and deliver the help that people need. Help people collaborate. Use the language of the work and worker. Provide what is needed, no more, no less. Help people act smarter.

From Rossett and Schafer, 2007, pages 155–162. Used with permission.

CH005.indd 221

6/17/10 3:29:33 PM

222


performance support system will help the organization achieve its ultimate business objectives by helping people achieve their learning transfer objectives. A firm, causal link to business achievement is essential because any kind of performance support necessarily entails some cost. That cost, like the cost of the training itself, needs to be repaid through improved performance. The higher the business value of the outcomes, the easier it is to justify the investment. Early on, many companies invested substantial sums to buy extensive content sites, only to be disappointed by how infrequently they were used. The problem is that such sites were only weakly linked to business and development objectives. Time pressures Adults require a need to know in today’s business climate preclude people from before they will seek out “browsing” content sites, however rich and interinformation. esting they might be. Employees turn to performance support systems when they have a specific question, a problem to solve, or task to complete related to their work. Even superlative support systems will languish if they are not part of an overall process of transfer and application. This is one more example of a core principle of andragogy (adult education): adults require a “need to know” before they will seek out information and learn (Knowles, Holton, & Swanson, 2005, p. 64). When it assists in achieving real business goals, the cost of a targeted performance support system is usually repaid many times over. IBM, for example, calculated that in a single year their manager portal delivered $14 million in time savings (Rossett & Schafer, 2007, p. 155); Pfizer realized an almost 50 percent greater return from its leadership program by adding transfer and performance support (Trainor, 2004). 2. Help Users Define, Track, and Achieve Goals Learning and development

contribute to improved overall organizational performance by improving individual performance (Figure D5.3). Thus, to fulfill the first criterion of a great performance support system (achievement of important business objectives), the system must support the more granular definition of helping individuals define, track, and achieve personal goals. That is why, for example, each individual’s goals—what she or he is trying to accomplish by applying new skills and knowledge—is the fundamental organizing principle of the learning transfer systems we have built (Figure D5.4). As in the GPS example above, once the destination is defined, the system is able to provide specific directions. 3. Focus on what Really Differentiates Superior Performance The third criterion

for great performance support is that it truly presents “best practices.” This

CH005.indd 222

6/17/10 3:29:34 PM


223

FIGURE D5.3. TRAINING AND DEVELOPMENT CONTRIBUTE TO ORGANIZATIONAL PERFORMANCE BY IMPROVING INDIVIDUAL PERFORMANCE

Training and Development

Learning Transfer and Performance Support

Improved Individual Performance







Improved Organizational Performance

principle requires learning professionals to research the best practices for a given task or skill and then build them into both the training and the performance support system. Ideally, the “vital behaviors” have already been identified as a result of the needs analysis and outcomes definition process. They may also be available from published research on leadership, selling, coaching, and so forth. For maximum impact, the performance support system must be consistent with, and reinforce, the methods, approaches, and processes taught during the coursework. Presenting multiple models, conflicting concepts, and different terms for the same thing in the performance support system can confuse learners, especially novices. For this reason, Grid International, a global leadership, management skill, and team-building training company, has built over a hundred specific suggestions into its learning transfer management system to reinforce the specific principles and methods taught in its seminars. Cannon’s Blue Wing Consulting has done the same to help participants apply the principles of Fierce Conversations. GlaxoSmithKline’s Leadership Edge program uses a learning transfer management system to send participants links to Senn Delaney’s reinforcement of learning tools to help refresh understanding and support application of key concepts. Scotwork, an international negotiation skills training company, developed a library of HelpMe items with specific suggestions to help participants get a better deal, handle a grievance, break a deadlock, and so on. And the Center for Creative Leadership has published forty-six short “Ideas into Action” guides to help participants convert learning into action and results.

CH005.indd 223

6/17/10 3:29:34 PM

224


FIGURE D5.4. EXAMPLE OF A LEARNING TRANSFER MANAGEMENT SYSTEM ORGANIZED TO SUPPORT ACHIEVEMENT OF INDIVIDUAL GOALS


One of the most exciting aspects of electronic performance support systems— as opposed to printed job aids, manuals, and guides—is the potential for them to be continuously improved and updated. Just as companies and individuals must continue to learn and adapt to stay competitive, the performance support system also needs to evolve and improve over time as conditions change and new best practices emerge. Some of the most useful suggestions to include in the support system are those that have their origin in the company itself— things that have proven

CH005.indd 224

6/17/10 3:29:35 PM


225

successful for other employees in the company’s peculiar culture and environment. Companies that identify such best practices, capture them in a form that can be fed back into the system, and then rapidly disseminate them have a decided competitive advantage. An example of the potential of this approach comes from the Store Managers Learning Forums at Home Depot. To help drive the performance of its stores, Home Depot embarked on an ambitious program to educate all of its 1,600plus store managers. To ensure that the program had impact, each manager was expected to report progress over a three-month period in an electronic learning transfer management system (Friday5s). At the end of three months, each manager was asked: “What is the one change you have made that has had the biggest impact on making your store run better?” Hundreds of examples of specific, successful changes were submitted. Importantly, each had proven effective in actual practice—they weren’t just abstract theory. That meant that they had immediate credibility, since they were generated and tested in real stores by real store managers accountable for the bottom line. Such an approach, repeated over time, would produce an invaluable repository of company-specific knowledge about best practices. Another example is Aperian Global’s GlobeSmart web tool, which provides detailed information on how to most Participants value help on how effectively communicate, manage employto apply course principles. ees, transfer technology or skills, and improve relationships with customers and suppliers in countries around the world (Aperian Global, 2010). The system continues to get smarter by inviting executives who have experience working in particular cultures to contribute ideas and insights. These are checked, edited, and added to the database, so the system continues to grow increasingly rich, deep, and specific. Learning organizations can increase their value by putting in place mechanisms to find out and record what really produces superior post-training performance for the employees they teach so that they are able to disseminate it to the next generation. 4. Recognize and Deliver the Help That People Need Participants in corporate

education programs value help on how to apply course principles in the conduct of their day-to-day activities. They need the equivalent of an owner’s manual for their new capabilities that explains how and when to use them effectively, analogous to the way that a product comes with instructions on how to install a new printer cartridge or set up a wireless network.

CH005.indd 225

6/17/10 3:29:37 PM

226


A number of texts have been published to fulfill this need, such as FYI: For Your Improvement (Lombardo & Eichinger, 2009); Successful Manager’s Handbook (Gebelein & colleagues, 2004); Essential Manager’s Manual (Heller & Hindle, 1998); and many others. The most useful of these provide specific suggestions on how to improve specific competencies. For example, The Essential Manager’s Manual provides this advice on running team meetings: Making team meetings effective is a major test of leadership skills. The key to holding a productive meeting is to actively involve everybody in the proceedings. • Ensure that team members understand the purpose of each meeting and what is expected of them. • Change the chairperson at each meeting to involve everybody. • Try to delegate as much as possible to other members of the team. • Distribute agendas in advance of the meeting to give your team time to prepare (p. 390).

The popularity of such guides attests to the need they fill. A limitation of published guides, however, is that they must necessarily be generic. That runs counter to the current trend in which more and more companies are defining their own competency models and terminology to emphasize specific attributes they consider important to their culture and corporate strategy. Truly embedding these models in the corporate culture requires consistent use of terms and concepts over time. Learners can be confused when they are provided materials, instruments, and guides that use different terminology for the same concepts covered in the coursework, none of which may exactly match the corporate credo. A welter of conflicting terms and unaligned concepts creates suspicion about whether the company is truly committed to the avowed model or whether it is yet another “trend du jour.” To maximize reinforcement and utility, all the program materials, including the post-course user guides, should employ a consistent set of terms and concepts— something that is obviously easier to accomplish with electronic support systems. People remember solutions best and value them most when they are learned in the context of an immediate and pressing problem. The ideal “teachable moment” to reinforce a technique, skill, or principle from corporate learning and development is when a relevant problem is encountered in the conduct of work. That’s why small, portable job aids are used much more frequently than large procedural manuals. They can go where the need is. The ideal performance support system offers the learner ideas just in time—at the moment and in the context when he or she needs them.

CH005.indd 226

6/17/10 3:29:37 PM


227

Income tax preparation software, such as TurboTax, illustrates this principle in action. Each form and schedule contains links to the relevant instructions and expert advice. Based on where the user is when he or she asks for help, the links take the user directly to the pertinent section of the tax code, rather than forcing him or her to search the whole morass. Another example is the electronic version of the Physicians’ Desk Reference. A physician can now carry a version on his or her PDA and refer to it any time there is a question about a drug dose, indication, or interaction, rather than having to go to the library and drag a several-hundred-page tome off the shelf. A third example would be a call-planner job aid built into account management software. Downloaded to a sales representative’s laptop, it is easy to review at the moment it is most effective: immediately prior to a call. The possibilities for such specific, just-in-time performance support systems are exciting and limited only by our creativity and interest in creating them. As with developing job aids, the key to designing effective performance support for learning transfer is to consider where there is likely to be a “need to know.” Look for places in which memory or knowledge may be incomplete or unreLook for places in which liable, especially for novices. Make the memory or knowledge may be performance support system (job aids, incomplete or unreliable. electronic systems, and people) available then and there; these are the times and places where additional information will be most valuable and employed most often (Clark, Nguyen, & Sweller, 2006). 5. Help People Collaborate Great performance support systems help people

collaborate to harness more of the tacit knowledge of the organization. Or, as former Hewlett-Packard CEO Lou Platt allegedly quipped: “If only HP knew what HP knows, we would be three times more productive.” A learning transfer management system can help keep a cohort of learners connected after a shared learning experience and make it easy for participants to reach out to one another as they work to implement new skills and knowledge. The system we designed, for example, calls the user’s attention to other participants who have the same kinds of goals and are reporting good progress (see Figure D5.5). Two trends in organizations make fostering collaboration electronically both more feasible and more important. The first is the increasing number of young workers who have grown up with digital social media and who are used to communicating and collaborating digitally. The second is the increasing number of people who work from their homes at least part of the time, making traditional

CH005.indd 227

6/17/10 3:29:38 PM

228


FIGURE D5.5.

AN EXAMPLE OF A SYSTEM TO HELP PEOPLE COLLABORATE


forms of connecting and collaborating more difficult. It is clear that digital social networking will play an increasingly important role in both learning and learning transfer in the future. The challenge for learning and development organizations is to utilize these media effectively to support multiple forms of learning and, at the same time, not to be deceived into thinking that they are a magic bullet that miraculously solves all training and development challenges. 6. Use the Language of the Work and Worker As we will touch on again in D6

on documenting results, it is essential that learning professionals speak the language of the business. This is especially vital in performance support systems; they need to provide advice and information in the terms and in the ways that they are actually used on the job. To fulfill this criterion, have items in the performance support systems reviewed not only by subject-matter experts, but by employees who actually do the jobs involved, to ensure that the items are clear, straightforward, and understandable. Ideally, performance support systems should include a feedback mechanism similar to Wikipedia, where users can flag unclear, confusing, or otherwise defective items so that the system becomes self-correcting and continuously better. 7. Provide What Is Needed, No More, No Less Lack of time is the most com-

mon reason that participants give for not doing more to transfer and practice new

CH005.indd 228

6/17/10 3:29:38 PM


229

skills after training. Email, cell phones, Support for learning transfer voice mail, faxes, memoranda, and other communications pour in. Time runs out. has to be simple, specific, easy to use, and fast. Speed matters. A system that is intended to support learning transfer and performance has to be simple, specific, easy to use, and fast. If it takes too long to find the relevant information or to glean its key points, the system won’t be used. To meet these demanding criteria, focus on utility, not glitz or splash. Avoid complex layouts, flashy but non-value-adding graphics, and complex navigation. Avoid the temptation to dazzle the eye while ignoring the brain. The most important aspect of any online system is functionality, not graphical design (Flanders & Willis, 1996). Likewise, since the audience for the system is employees and managers who want to locate relevant information quickly, speed of access and relevance of the information are more important than the size of the database as such. For example, a search of the Internet for “how to delegate” returned over six million links. “How to delegate effectively” returned 400,000. (Apparently it is easier to delegate than to do so effectively!) In both cases, some of the links were useful, but most were a waste of time. The sheer number is so overwhelming as to discourage all but the most dedicated searcher. The fact is, most people rarely look beyond the first page returned by a search engine. The problem with simply relying on the World Wide Web for performance support is that a portion of the advice that turns up is bad or bogus. It may also run counter to your organization’s principles and procedures. The World Wide Web is an amazing repository of information, but it is a lousy performance support tool. In developing or purchasing a post-training performance support system, remember that people read material on computer screens much differently than they do information on paper. For example, only 16 percent of users read text on a screen word by word (Nielsen, 1997). Instead, they scan the material to rapidly pick out key concepts, and they avoid long, dense passages typical of books and articles. Users of electronic information systems prefer succinct, clear writing, frequent paragraph breaks, bullets, and other condensation techniques. It is therefore important to rewrite material for presentation on the web specifically for that purpose by using “scannable” text: • Highlighted keywords (hypertext links serve as one form of highlighting; typeface variations and color are others) • Meaningful subheadings (not “clever” ones) • Bulleted lists

CH005.indd 229

6/17/10 3:29:39 PM

230


• One idea per paragraph (users will skip over any additional ideas if they are not caught by the first few words in the paragraph) • The inverted pyramid style, starting with the conclusion • Half the word count (or less) than conventional writing (Nielsen, 1997) Researchers at Sun Microsystems found that they could double the usability of information on the Web when it was rewritten according to these guidelines (Nielsen, Schemenaur, & Fox, n.d.). Improvements were seen in all key metrics: • • • •

Reduced task time Fewer errors Greater content retention Higher user satisfaction

Brevity, however, carries the attendant risk of superficiality. A savvy employee will be skeptical of unsupported one-liners of “things to do” or guides that reduce the complex tasks of management to superficial sound bites. The solution to the speed/depth paradox is to use the capabilities of electronic media to present “topline” information succinctly, with hypertext links to more in-depth treatment and references. This allows the users to retain control; they can quickly review a large amount of summary information but also have the option to pursue knowledge in depth in areas of special interest. An example of these concepts in practice is the GuideMe feature we developed for inclusion in the learning transfer management system, ResultsEngine. When activated by the participant, the system selects relevant suggestions based on the kind of objective the user is pursuing. A list of succinct suggestions for action is presented, all of which have bearing on the kind of skill or competency the participant is trying to improve. Adjacent to each suggestion is a “Tell Me More” link for users who want more explanation or deeper understanding. Clicking the “Tell Me More” link opens a window to more in-depth information, explanation, references, and so forth (see Figure D5.6). 8. Help People Act Smarter Finally, a great performance support system helps people do the right thing—even if they cannot remember exactly how themselves—by providing specific guidance for action. The Third Discipline (deliver for application) applies as much to the performance support system as it does to the course itself. A learning transfer management system should provide concrete, actionable suggestions—practical steps people can take to change their behavior in ways that will deliver better results. For a time-challenged employee looking to improve

CH005.indd 230

6/17/10 3:29:40 PM


231

FIGURE D5.6. AN EXAMPLE OF GUIDANCE THAT ILLUSTRATES ACTION ORIENTATION AS WELL AS THE ABILITY TO VIEW DETAILS


performance, nothing is more frustrating than to invest the time seeking guidance, only to receive vague and general platitudes such as “Be more strategic.” In the systems we have developed, for example, we insist that each recommendation be written as a positive statement in first-person future tense starting with: “I will. . . . ” This forces our recommendation writers to think about how specific principles can be applied or how specific skills can be practiced, not merely contemplated. With continuing advances in technical capabilities and portability, electronic performance support systems will become increasingly pervasive and sophisticated. Workplace learning professionals should use their expertise in learning and human performance to design and apply performance support systems that will enhance continued learning and learning transfer in the post-course (Phase III) period.

People People are the third and critical source of support for learning transfer. People play a complementary and crucial role in supporting learning transfer; they provide uniquely human capabilities such as empathy, wisdom, feedback, encouragement,

CH005.indd 231

6/17/10 3:29:40 PM

232


collaborative problem solving, and motivation in a way that no system can fully replace. Hence, devoting time, planning, energy, and creativity to harnessing human support systems is an important part of D5 and the complete learning experience. Many different people can play a role in supporting learning transfer—everyone from managers to the participant’s friends or partners. Four—managers, peers, instructors, and coaches—have special impact and are discussed in detail below. Managers Managers represent the most influential, and most underutilized, resource available to effect application and ensure that education produces results. Broad and Newstrom (1992) found that managerial involvement before and after the program were ranked, respectively, the first and third most potent learning transfer strategies. Yet Managers represent the involvement by the manager after the course was most influential, and most the least frequently used of the nine role-time combiunderutilized, resource. nations they studied. They concluded: “Managers do not consistently and powerfully support the transfer of training in the work environment. We believe that this represents a fundamental problem, and also a substantial opportunity for improvement” (p. 53, emphasis in the original). That statement appears to be as true today as it was when it was published eighteen years ago. A recent study at Pfizer illustrates how great an impact managers have. To assess the effectiveness of its core leadership development program, Pfizer conducted repeat 360-degree assessments of participants several months after the program and compared them to the pre-program results. Program participants were also asked to indicate to what degree their managers had been actively engaged in their post-training development. The results showed clearly that training worked — participants showed statistically significant gains on all five of the five most-frequent development plan items—provided their managers were actively involved during the post-course period. Participants in the same program whose managers were not actively engaged showed no performance improvement or made much smaller gains than those who enjoyed managerial support. “Those coached by their supervisor clearly had greater change over time” (Kontra, Trainor, & Wick, 2007). In other words, the same program, with the same content, and the same kind of participants yielded different results depending on whether or not the manager was engaged. Pfizer’s study confirmed the previously mentioned results of the study at American Express (2007), which concluded, “The key criteria for a high transfer climate includes having a manager who clearly communicates endorsement and

CH005.indd 232

6/17/10 3:29:42 PM


233

support for the training and sets goals and expectations before the employee initiates the learning event; who follows up with the participant after the event to discuss what was learned; and who recognizes and rewards improved leadership behavior” (p. 11). Many other studies (for example, Saks & Belcourt, 2006) confirm the importance of managerial engagement and support. That managers should have such a profound impact on learning transfer is not surprising, given that a person’s direct supervisor holds the keys to salary increases, promotions, and advancement. Employees therefore pay close attention to signals (intentional or unintentional) from their managers. Given the influence exerted by managers, it is impossible to achieve a high level of learning transfer without managerial engagement; managers are an integral and crucial part of the total system for helping employees apply learning and convert it into business results, provided they are willing and able to do so effectively (see Figure D5.7). FIGURE D5.7. MANAGERS MAY BENEFIT FROM GUIDANCE ON HOW TO GIVE EFFECTIVE FEEDBACK

© James Stevenson/Condé Nast Publications/www.cartoonbank.com.

CH005.indd 233

6/17/10 3:29:43 PM

234


That’s why Agilent CEO Bill Sullivan wanted to start cascading Agilent’s new, business-focused, applied learning experience down from the top, starting with the company’s one hundred general managers. He told his chief learning officer, Teresa Roche: “Let’s make sure the soil is fertile so that when people come out of their experience, they have a manager and a set of colleagues who are getting what they are talking about and we can have better transfer and application” (quoted in Prokopeak, 2009). Range of Manager Engagement Kirkpatrick (1998) proposed that a manager’s

response to training falls along a continuum (Table D5.1). At the most destructive end of the range are managers who actively prevent their subordinates from using what they have learned. The most egregious example we have ever heard was told to us by an oil company executive in Canada. His company had sent him to a six-month executive program at Harvard at substantial cost—not only in tuition, but also in his salary, travel, lodging, and so forth. The day he returned to work, his manager called him in to his office and said: “Welcome back. I don’t want to hear any of that c**p you learned at Harvard.” That is, of course, an extreme example, but there are managers in every company who actively block the use of new approaches by telling their subordinates, in effect, that what they learned in training is “not the way we do it around here.” Such complete contradiction between managers and training suggests a failure to define real business needs and garner management support in the design phase. It also suggests a breakdown in buy-in of the management chain of command. If senior management supports a program, mid-level and front-line managers should feel obligated to try to make it work. Nearly as damaging are managers who discourage using new methods or approaches. Such situations are a recipe for disaster; they squander resources TABLE D5.1. Manager’s Action Reinforcement

CH005.indd 234

RANGE OF MANAGERS’ RESPONSES TO EMPLOYEES USING NEW CAPABILITIES

Prevent Application

Discourage Application

Neutral

Encourage Use

Require Use

Negative Reinforcement

Positive Reinforcement

Effect on Learning

Counteracts benefits of learning and development; reduces return on investment

Enhances benefits of learning and development; increases ROI

Effect on Employee

Confuses employees; devalues learning and discourages self-development

Encourages employee to continue learning; reinforces value of selfdevelopment

6/17/10 3:29:44 PM


235

and leave employees confused and frustrated: “If I am not supposed to use this stuff, why did you waste my time making me learn it?” The most telling example in our personal experience comes from a discussion with a major healthcare firm. We were trying to make the point about the importance of management support. We said: “Managers have a profound influence on the transfer of learning. If people attend a program and get enthused about using what they learned only to get a negative reaction from their boss when they try, the value of the program is dramatically reduced. In ten minutes, a manager can undo a week’s worth of education.” Our client replied: “It is much worse than that. In fifteen seconds, a manager can undermine a year’s worth of work.” They went on to explain that, in one of their divisions, sales training had worked with sales management for a full year to move the sales force toward a more consultative selling style. But no one had made sure that the division general manager was fully on board and informed. So when he was asked in front of the entire sales force what he thought of the approach, he replied: “Oh, that’s just what they teach in training.” In one remark, he undermined months of effort. Kirkpatrick placed “neutral” or “indifferent” in the center of his scale. We disagree. We believe that indifference is negative. If a person attends an educational program and is excited about applying what he or she learned, but is told by the boss in so many words, “I really don’t care one way or the Indifference is negative. other,” most will understand that to mean, “You have better things to do.” Managerial indifference is not neutral; it is destructive and expensive. On the positive side of the ledger are managers who encourage their direct reports to apply what they have learned. Such active encouragement is an important contributor to a positive transfer climate (Figure D4.8) and important for an organization to get its money’s worth from training and development. At a minimum, managers should be expected to encourage the use of new learning; ideally, they should require it as “the way we do business here.” The latter is exceptionally powerful in effecting organizational change (see Case in Point D5.1). Learning and development do not take place in a vacuum. The environment to which learners return—especially the messages that managers convey (overtly and covertly)—must be actively managed to support transfer and application. Motivating Managers to Coach For a long time, we puzzled over the question of why managers do not do more to encourage the application of learning by their direct reports. It is clearly in their personal best interest to have the employees in their departments become more effective and more efficient and capable of

CH005.indd 235

6/17/10 3:29:44 PM

236


Case in Point D5.1 Maximizing the Value of Marketing Training When Jorge Valls accepted the leadership of SmithKline Beecham Animal Health, he identified an urgent need to improve the quality of marketing and marketing plans. He contracted with the Impact Planning Group to conduct an intensive marketing workshop. He included not only the marketing department, but all managers, true to his belief that marketing is everyone’s responsibility, because every department contributes positively or negatively to the customer’s perception of the firm and brand. The training was excellent, but what made it effective was that at the conclusion of the training, Valls announced a non-negotiable requirement: All future marketing plans were to be prepared in accordance with the principles that had just been taught. He required that the training be utilized and he backed up what he said by his actions. All subsequent reviews were conducted in accordance with the agreed-on principles. He would reject out-of-hand any proposal that did not follow the guidelines. The result was that the quality of operational plans improved immediately, as did the quality of the discussions among managers, departments, and the leadership team because everyone shared common concepts and vocabulary. Within months, the improved quality of planning and marketing were evident in both top-line and bottom-line growth, even in the absence of new products. Had Valls only encouraged the use of the new methodology or left it to individual discretion, improvement would have been much less and far slower, with some departments embracing the new while others clung to the old way of doing things. The company would have realized a much smaller return on the investment in training, if any.

handling additional responsibilities. Moreover, the company has already made the investment and the department has already committed the employee’s time. Why not make a small incremental investment in coaching to gain a significant return in enhanced performance? Our clients really helped us understand the impediments as well as practical and workable solutions to address them. Managers do not coach to maximize transfer of training because they aren’t sure how, they are not convinced of the value, and they don’t have time (which really means it is not a priority). Four conditions, then, must be satisfied to motivate managers to support their direct reports in the application of learning and development. Managers must: 1. Believe that there is value in doing so. 2. Feel confident that they can.

CH005.indd 236

6/17/10 3:29:44 PM


237

3. Know what the program and their direct reports are trying to accomplish. 4. Be held accountable for providing support; it must be an expected part of their jobs. See Value As busy as managers are these days, they have to be convinced that

investing their time in the short run to coach their direct reports will be repaid in the long run by greater effectiveness. That requires creating a compelling Managers have to be answer to the WIIFM (What’s in it for convinced that investing their me?) question for managers. Learning time will be repaid. professionals need to educate supervisors on the impact they have on whether or not training pays dividends to their department. Show them the evidence. A fascinating side-effect of the Pfizer study was that when the data on the impact of managerial engagement were shared with managers, the amount of coaching increased; managers could see its value. Share the research findings with senior management as well (see Case in Point D6.6). They need to be persuaded to make follow through on educational opportunities a key responsibility for all managers. Show them the incremental value that manager support adds to learning and development programs. Calculate the cost of the learning scrap that results when managers fail to support implementation. Encourage senior leaders to make support for employee development part of annual reviews and other assessments of managerial effectiveness. The goal is to create a company culture in which both learners and their managers accept their shared responsibility for maximizing the value of educational programs. Have Confi dence To provide effective coaching, managers must feel confident of

their ability to do so. No manager wants to be embarrassed by appearing uninformed or unskilled. Managers who are unsure how best to coach to training, or who are unclear about what was covered in a program, solve the problem by simply avoiding any discussions of the content or its application. For managers to confidently provide post-course mentoring, they need to understand what was covered in the program, feel that they have the requisite coaching skills, and have a defined process to follow. Just as participants benefit from job aids and performance support tools that help them fulfill their post-training obligations, so do managers. Provide them with a specific process to follow and guidelines for maximizing learning transfer as Plastipak Academy, Chubb Insurance, and other progressive learning organizations do. These companies have discovered that while no manager is likely to call human resources to request

CH005.indd 237

6/17/10 3:29:45 PM

238


assistance with post-training coaching, if you simply provide it, many will use it and to good effect. Geoff Rip, president of ChangeLever International, a learning transfer consulting firm, feels so strongly about the benefits of managerial support that he holds a special course for managers in advance of training their subordinates. The program for managers focuses on how they can and should use their influence, directly and indirectly, to ensure transfer and extend learning, thereby maximizing the benefits of the upcoming training. According to Rip, “Since leadership is fundamentally about influence, the training is positioned as an important leadership skill. Managers often have a narrow perspective on coaching and overlook many non-coaching strategies they can use to influence transfer and the action learning cycle” (Rip, personal communication, 2010). At Centocor Inc., for example, managers are brought together for a refresher session on maximizing the value of 360-degree feedback at the same time their direct reports are receiving the results of their feedback. Time is scheduled immediately afterward for one-on-one meetings while the material is top of mind for both parties. Lisa Bell, manager of the North American Learning Center for Holcim, held day-long “impact booster” sessions for managers of participants in the company’s Building Leader Performance Program, which she believes were a key factor in its success (Bell, 2008). Bell felt so strongly about the value of these sessions that she resisted pressure to shorten them. “Initially, one of our biggest concerns was that managers would never give up their precious time to participate in the ‘extra’ steps we asked of them. And, now, lo and behold, they themselves have asked for more” (p. 191). When it is impossible or impractical to provide specific training for managers, provide them Provide easy-to-use forms, a practical guide with advice on how to maximize step-by-step processes, and the benefits of their reports’ education. Make the examples. guide concise, practical, and action-oriented, following the guidelines for performance support systems discussed earlier. Provide easy-to-use forms, step-by-step processes, and examples. Communicate the availability of the guide and make it easily accessible in both print and online versions. At the request of our clients, we developed such a guide—a workbook called Getting Your Money’s Worth from Training and Development (Jefferson, Pollock, & Wick, 2009). We persuaded the publisher to print it as two books in one: one side for the manager and the flip side for the participant, to ensure each knows what advice the other has received. A completed example from the workbook is given in Figure D5.8.

CH005.indd 238

6/17/10 3:29:46 PM


FIGURE D5.8.

239

A COMPLETED EXAMPLE OF A FORM TO FACILITATE PARTICIPANT-MANAGER DIALOGUE

From Jefferson, Pollock, and Wick, 2009. Used with permission.

In partnership with Option Six Inc., we subsequently completed an online learning and performance support version called “How to Get Your Money’s Worth from Training” (Jefferson, Pollock, & Wick, 2010) that is compatible with learning management systems. Like the print version, there are separate but integrated programs for the manager and participant. The online version makes use of animated avatars to guide users as well as interactive forms to facilitate communication between the manager and participant (Figure D5.9). Know the Objectives For managers to become actively and meaningfully engaged in

the learning transfer process, they need to know what the program’s objectives are and what their direct reports are specifically trying to achieve. Ideally, managers should attend the program prior to or with their direct reports. Unfortunately, this is often impractical, especially as reporting relationships change and new

CH005.indd 239

6/17/10 3:29:46 PM

240


FIGURE D5.9. AN EXAMPLE OF AN ONLINE LEARNING AND PERFORMANCE SUPPORT PROGRAM TO ENHANCE MANAGERIAL ENGAGEMENT IN LEARNING TRANSFER


managers are promoted. Moreover, courses continue to evolve in concert with the needs of the business, so that even if a manager previously attended the program, new material may have been introduced. Thus, there is an ongoing need to keep managers informed of program objectives, content, and expectations. When an employee enrolls in a course, send the manager a short synopsis of the business needs the program is designed to address, the topics covered, and the desired outcomes, as a learning leader with GE in Europe does. Be sure communications are succinct and efficient; managers experience even greater demands on their time than the participants themselves. A brief introductory email with links to more in-depth information and suggestions on the company’s intranet is more likely to be read than a lengthy course description. For key courses, schedule a brief pre-course teleconference with participants and managers, as Bill Amaxopoulos does at Chubb Insurance (see Case in Point D5.2). Encourage managers and their direct reports to meet prior to the program to identify the most important learning opportunities. Give them a simple and effective process to follow, as Diane Hinton does at Plastipak, Lisa Bell at Holcim, and

CH005.indd 240

6/17/10 3:29:47 PM


241

Case in Point D5.2 Truly Engaging Managers Vasilios (Bill) Amaxopoulos is the leadership program manager for the Chubb Group of Insurance Companies and a tireless champion of continuous improvement. He shared with us his journey to more fully engage managers over the past several years in one of Chubb’s hallmark programs, the Leadership Development Seminar. “We have some wonderful leadership programs at Chubb, but we continue to raise the bar every year. We wanted to truly be able to say we saw some great behavior change and that our programs are a terrific investment of time for managers. So I started to say, ‘Maybe we can’t control the environment people come from or return to, but we owe it to them to have a major influence over that environment to increase their likelihood for success.’ “I had always copied managers of participants on all pre-work notes and would make a comment or two asking them to help their direct report succeed after the program. We also asked the manager and participant to sign a ‘learning contract’ prior to attending the session, but we noticed that more and more people would come to our sessions and say, ‘Oh, I didn’t really get a chance to speak with my manager. Is that okay?’ “Now the process is that the managers need to attend a Phase I webcast with their direct report, who is the participant, four to six weeks before our program. In the webcast, we lay out the specific roles they can play before, during, and after every session. I let them know that it’s okay if they haven’t been coaching and preparing their direct reports before and after programs in the past because they may not have known better, but it’s not okay to continue in that fashion. Because if we all don’t have some skin in the game, we’re not going to get the best return on our investment for this learning event. “The other part of the message is that during this webcast we’re suggesting several very easy and specific things to do before the session, some easy things to do during the session, and some easy things to do afterward. The webinar itself takes forty-five minutes, but we’ve asked them to book at least seventy-five to ninety minutes. We’ve created a situation where when the webcast ends, the manager and participant can immediately start a coaching conversation. They discuss expectations, development needs, follow other tips offered during the webcast, and conclude by putting some dates on their calendars for additional pre- and post-course chats. The good thing is the participants know their managers have some accountability and the managers know their participants have accountability. Getting that message simultaneously is really powerful. “I still send emails, but the main messages come through in a ‘Phase I webcast for participants AND their manager’ four to six weeks before a program and ‘Phase III webcast’ thirteen to fifteen weeks after a program. Some managers have gone out of their way to let me know how informative these messages were, that they never realized how important their role actually was. Overall, the message has been well received. “After the program, we also engage them in the Friday5s [learning transfer support] process, making sure that the manager is also being asked for feedback on at least two to (continued)

CH005.indd 241

6/17/10 3:29:49 PM

242


Case in Point D5.2 Truly Engaging Managers (continued) three of the eight participant updates over a six-month period. After the program, anywhere from thirteen to fifteen weeks, there’s a Phase III webcast for participants, as well as additional tips and follow-up that we do with their managers. “The Leadership Development Seminar has been around for fifteen years. People tend to make comments like ‘This is the best leadership program I’ve taken at Chubb or anywhere else in my career’ and we’re very proud of that. In the last few years, though, I’ve had several managers say to me, ‘When I took the LDS years ago, I thought that it was the best program I’d ever taken and I still do. But now that I’m a manager and have the tools and accountability for helping my direct reports be successful after they attend, I see how much more powerful it is now than it has ever been. I can really see the value of these follow-through tools.’ “The other thing that makes me really, really happy is when I see managers providing feedback to their participants post-program like: ‘I’ve noticed that when you run your meetings you’ve done this or that differently’ and ‘When you came back, you said you were going to try to take more of a big-picture approach; I’ve noticed that you have a much different approach now and that you talk strategically and look at things more broadly than you used to before the class.’ When you have managers acknowledging in their comments that they’ve noticed behavior change, we’ve hit a home run because we all know how difficult behavior change is.”

Bill Amaxopoulos at Chubb. Consider using commercially available workbooks or online programs to streamline the process. Encourage participants to have a post-program discussion with their managers regarding their specific objectives for application. Increase the probability that this will occur by collecting and sending a copy of each participant’s learning transfer objectives to his or her manager using an electronic learning transfer management system or other means. Finally, make sure that managers are aware of what their direct reports did (or did not) accomplish as a result of the training. This can be done in a number of ways, such as including the manager in a final session (in person or virtually) in which each participant reviews his or her goals, accomplishments, and “lessons learned.” Electronic learning transfer management systems can be programmed to automatically send each participant’s manager a summary of the participant’s accomplishments as well as a brief survey to validate, from the manager’s perspective, that valuable progress was made. Such automated triangulation of results can help provide credible and compelling evidence of the program’s efficacy (see Figure D5.10).

CH005.indd 242

6/17/10 3:29:49 PM


243

FIGURE D5.10. PAIRED RESPONSES FROM 134 PARTICIPANTS AND THEIR MANAGERS FIVE MONTHS AFTER A DEVELOPMENT PROGRAM WITH LEARNING TRANSFER SUPPORT

Manager Ratings

Participant vs. Manager Ratings

5 4 3 2

82% of manager and participant pairs BOTH agreed or strongly agreed that the program improved effectiveness

1 1

2 3 4 Participant Ratings

5

The program and follow-through increased the participant's effectiveness. 1=Strongly Disagree 2=Disagree 3=Don’t Know/No Opinion 4=Agree 5=Strongly Agree

Be Accountable for Coaching The most common excuse managers give for why they

don’t coach more to maximize training’s value is that they “don’t have time.” What that really means is that it is not a high enough priority to make the cutoff between what will be done and what won’t be done. The late management guru Peter Drucker observed more than thirty years ago that managers had become so busy that they could no longer accomplish all of their tasks—they did not even have time for all the important tasks; they had to concentrate only on the most important. Obviously, managers won’t make spending time providing feedback a priority if they are not convinced of its value. There are ample data that show how posttraining managerial engagement creates additional value for the participant and the company. But what about for the manager him- or herself ? In the bestmanaged companies, a proven ability to develop people is prerequisite to further promotion. Managers should be held accountable for developing their direct reports as a key performance metric.

CH005.indd 243

6/17/10 3:29:50 PM

244


Learning and organizational development professionals need to help senior managers understand their role in getting the greatest value from training and development. They need to provide a balance of support and accountability for participants’ managers, just as the Help senior managers managers need to for program participants. Programs understand their role. are more successful when there is active oversight by the managers of participants’ managers. In AstraZeneca’s Breakthrough Coaching program, for example, the learning organization used an electronic learning transfer management system to help area sales managers identify which of their regional managers were doing a particularly good (or poor) job of coaching their direct reports. Calling attention to these outliers during regional discussions helped recognize and reinforce positive coaching behaviors and bolster the efforts of less active coaches. The support for and tracking of manager involvement were important contributors to the program’s success. Managers are the most influential, but underutilized, resource to improve learning transfer and thus the results delivered by corporate education. To fill this role, managers need information about the course and their direct reports’ objectives, coaching skills, and guidance on how they can best contribute to the process. Breakthrough learning and development organizations supply these (see Case in Point D5.3). Senior management needs to make support for learning transfer a

Case in Point D5.3 Ongoing Support at ADP With over half a million clients, ADP is one of the world’s largest providers of business outsourcing solutions. ADP’s National Account Services has publicly stated its total commitment to providing world-class service. According to Dianne Keefer, senior director of learning and development, “Every interaction between our associates and our clients matters.” Training is vital to achieving that goal, but as Keefer notes, “Training is more than just an event; it takes commitment, demonstrated support from leadership and management, and a compelling business case to drive change.” It also takes ongoing reinforcement and visible engagement of managers to maintain the momentum and convince associates that this is not just another “program du jour.” To ensure that associates continue to practice and refine their skills, Dianne’s group provides supervisors with complete scripts and guides for holding quarterly “huddles”—short group discussions that reinforce and extend the principles and practice of world-class service. Keefer credits ADP’s systemic approach to training and support—including these periodic huddles—with contributing to a continuing improvement in client satisfaction (Keefer, 2010).

CH005.indd 244

6/17/10 3:29:51 PM


245

part of every manager’s job—one that is monitored, recognized, and rewarded. Their importance notwithstanding, managers are not the only source of support, however, nor should they be relied upon exclusively. Instructors, peers, and other coaches can also be part of the post-training support team as discussed below. Instructors and Facilitators Another important source of support for learning transfer is the learning and development department itself. Participants often express the desire to have ongoing contact with the faculty. When we surveyed participants three months after a management development program that provided ongoing support from the instructor, participants put a very high value on continuing access to the instructor during the learning transfer phase. A Trusted Resource That makes sense, as Teresa Roche of Agilent Technologies

explains: “ Facilitators are selected for their superior knowledge and teaching ability. During the program, participants come to value the facilitator’s knowledge, opinions, and advice. Yet, historically, teaching ended when the class ended; communication was cut off. As a result, there was no support for learning transfer from the facilitators—the very people with the greatest insight into the material and whose opinion the learners value most” (Roche & Wick, 2005, p. 6). Instructors are a trusted resource for participants. Finding efficient ways to make them available to support learning transfer—and providing them the time and accountability to do so—will contribute to a healthy learning transfer climate. Facilitators’ New Role To address this issue, Agilent redefined the role of facili-

tators so that their responsibilities extended beyond the last day of class and into the learning transfer period. That is a significant departure from the prevailing paradigm, which limits the role of educators to delivering the course. It requires a broader conceptualization of the facilitator’s role and a reallocation of resources. The new role for facilitators is to lend their expertise throughout all four phases of the learning process, rather than in just the classroom or virtual sessions. It supports Broad and Newstrom’s contention that learning professionals should redefine their role from “strictly trainers/presenters to facilitators of behavioral change on the job” (1992, p. 113) and the Robinsons’ concept of workplace learning professionals as “performance consultants” (2008, p. 1). Roche and Wick (2005) put it this way: “Facilitators must move from the ‘sage on the stage’ to the ‘guide by the side,’ from facilitator of learning to facilitator of performance” (p. 13).

CH005.indd 245

6/17/10 3:29:51 PM

246


The problem with asking facilitators to take on this role is that most learning organizations have barely enough staff to manage what is already on their agendas. A facilitator who completes a course one afternoon is typically scheduled to start another the following morning. When are they supposed to have time to follow up with Move from the “sage on participants of previous sessions? The question the stage” to the “guide by that training departments have to ask is whether the side.” it is the best use of resources to continue to pour so much effort into a system that has such low yield—training one thousand people to improve the performance of a couple of hundred—or whether it makes more sense to reallocate resources, human and financial, and invest them in those parts of the process most likely to improve results. In other words, having facilitators teach less and guide more will probably generate greater value than continuing to grind out so much learning scrap from every session. Bob Sachs, vice president of learning and development at Kaiser Permanente, agrees: “It’s not just about how many programs we have. We are continuing to work on the idea that doing a lot of programs is less important than doing fewer programs with greater impact. In order to do that, you need to change the role of the instructors so that they are not just responsible for the classroom delivery, but they are actually following a cohort of people into the follow-through stage. That means that you’re going to have instructors who do fewer programs but with more impact” (Sachs, personal communication, 2009). It is important to note that you cannot just add coaching onto the facilitators’ other responsibilities and expect to be successful; you have to lighten their teaching schedule to provide time for performance support and give them tools to make the process efficient. You can begin the change by asking your instructors to simply contact each participant two to three weeks after the course to ask, “How are things going in your effort to transfer your new skills?” It will be enlightening for the instructors and a reminder stimulus for the participants. Recognizing that facilitators’ time is valuable and limited, use technology to make the process efficient as well as effective. Electronic transfer management systems provide facilitators with a “dashboard” that shows them how the group as a whole is doing and which individuals will benefit from personal attention. Peers: Learning Communities A third potent but underutilized resource for ongoing support is the other participants in the program. When Linda Sharkey was director of leadership development at General Electric, she noted the value of peer-to-peer coaching in GE’s

CH005.indd 246

6/17/10 3:29:52 PM


247

renowned leadership development programs: “Because all of our leadership teams participated in the 360-degree-feedback survey process, they now work as teams to help each other improve. When the leadership teams share their development needs with each other and use the coaching model, they often find three things: (1) they have similar issues, (2) they get great improvement suggestions from each other, and (3) they get support from each other to improve” (2003, p. 73). All sorts of training programs can take advantage of the collective knowledge and experience of the participants by encouraging shared learning throughout the entire learning process. Peer support relationships are especially powerful in the learning transfer (Phase III) period. Etienne Wenger, who studies communities of practice, explained why they are so powerful: “There is something about hearing the words of someone who is a peer that makes the relevance of the knowledge that you get very immediate. So for me that is the fundamental value proposition of a peer-to-peer network” (quoted in Dulworth & Forcillo, 2005, p. 111). The following examples attest to the value that participants place on maintaining their learning network: • Participants in one of Cornell’s open-enrollment management programs organized a class reunion. • Participants in a session at the Center for Creative Leadership created their own website to help stay connected. • Fellows in one of the Kellogg national leadership programs have continued to meet annually for twenty years. Learning and development organizations can help build and sustain the learning community by reconvening groups, either in person or virtually. Kirwan (2009) recommends that the learning organization set up the learning commuParticipants’ actions don’t nities before participants leave the course, always match their good rather than leaving networking to chance, intentions! noting that “participants’ actions don’t always match their good intentions! ” (p. 60). An important component of Babcock and Wilcox’s Leader-to-Leader program was the establishment of peer coaching teams who met throughout the year-long process to support one another. (See Case in Point D6.3.) Gary Jusela, who has led learning organizations at Boeing, Cisco Systems, and Home Depot, says: I am a huge believer in learning events that are designed as multistage events. I think multistage is always better and more impactful than one-shot. If you

CH005.indd 247

6/17/10 3:29:52 PM

248


can have learning practice and reflection built in, plus coming back for some disciplined collective reflection, it really increases the odds of getting real transferability. Learning from experience happens by thinking about the experience itself: what worked and what did not. What I love is bringing people back and having them reflect on their experiences in small groups and then also share some of that collectively in the larger room. What people discover is that they are not so alone, or they are not so weird. Everybody struggles with these things and they can learn from each other and get some tips about how to overcome some of the most perplexing struggles (Jusela, personal communication, 2004).

Although peer-to-peer support can take many forms, we have found computerbased collaborative systems especially valuable in open-enrollment or other programs in which participants disperse to different locations. Helping participants stay in touch can be as simple as sharing email addresses or as robust as a fully integrated system that allows participants to view one another’s progress reports and exchange feedback. Bob Sachs, vice president of learning and development at Kaiser Permanente, believes that informal and peer-to-peer learning will be increasingly important complements to structured learning and the next frontier for learning organizations. “So I’ve been talking to my instructional design folks about how designing the complete experience changes as a result of that perspective. And that what goes into the complete experience now are all the things that Bersin would put in the informal learning category as well as what goes in the formal learning category. We need to begin to think about designing our learning with that full spectrum of formal and informal learning. We have to look at it from an enterprise perspective, not just from a program perspective” (Sachs, personal communication, 2009). Critical Mass How co-workers react to an employee’s efforts to apply new learning is a key contributor to the learning transfer environment (Figure D4.8). The amount of learning that gets applied is enhanced when co-workers have received the same training simultaneously, creating critical mass (Kirwan, 2009). Training everyone in the same work unit or department simultaneously creates an environment in which learners can provide mutual support and reinforce each other’s use of the new language, concepts, and behaviors. Creating critical mass and a similarly trained group of co-workers is especially important if the training is part of a larger change initiative. If possible, train intact teams together, including the group’s leader or manager. Targeting specific departments, business units, or working groups and rapidly training a significant percent of the employees in them has a greater chance of success than the “shotgun”

CH005.indd 248

6/17/10 3:29:52 PM


249

approach of one here, one there. When A lot of people need to be only a small number of individuals has learning similar things, all at been trained, the freshly minted “evanthe same time. gelists” for the new approach are under tremendous pressure to conform to the old way of doing things when they return to work. If they are “lone voices crying in the wilderness,” fewer will be successful in applying what they learned. A group of disciples has a better chance of keeping the faith. “If you are looking for a substantial upshift in company performance, a lot of people need to be learning similar things, all at the same time” (Bordonaro, 2005, p. 163). Designated Coaches Designated coaches, typically professionals, are another potential source of performance support. Professional mentors can help managers master the art of leadership in the same way that professional coaches help musicians, actors, athletes, and other performers master their arts. When Daniel Coyle (2009) studied “talent hotspots” around the world—places that produced a disproportionate number of outstanding performers—he always found “master coaches,” men and women who had a talent and love for helping others achieve their personal best. According to Mary Jane Knudson, vice president of human resources at Fidelity Investments, “Nearly every major corporation—and progressive smaller firms as well—identify executive coaching as one of their critical executive and leadership development activities” (2005, p. 40). Professional coaches can be invaluable in helping participants maximize the value of formal training, 360degree feedback, and on-the-job learning experiences. Having a coach is in itself an incentive to follow through, practice, and reflect. However, using executive-level coaches is cost prohibitive for most learning and development programs. Moreover, participants in new-manager programs, for example, don’t need high-priced executive coaches, but they do need access to mentors more experienced and more knowledgeable than themselves. If outside instructors are used in the program, consider retaining them to provide support for learning transfer. Other sources of coaches include managers, internal development specialists, retirees, and HR generalists. Goldsmith and Morgan (2004) compared the results of development programs in eight different companies. Some used paid external coaches; others used internal coaches. Both internal and external coaches added value. Goldsmith and Morgan concluded: “Coaching can be a great complement to training. Leaders can clearly benefit from coaching, but it does not have to be done by external coaches. ” Some programs we have worked with have used recent program

CH005.indd 249

6/17/10 3:29:53 PM

250


graduates as mentors, a process that enriches and deepens the knowledge of both the teacher and student. The business outsourcing giant, ADP Inc., created an innovative program to certify internal coaches. Although participation in the program was voluntary and required a significant amount of the participant’s own time (without compensation), there was a waiting list of people wanting to become internal coaches. If you make the opportunity available, you are likely to find significant willingness to coach and mentor; most people are genuinely interested and pleased to help others. Learning transfer support technologies enables new forms of coaching that foster interaction and, at the same time, reduce the time commitment for the providers. For one thing, the time wasted playing telephone tag is eliminated. Second, when coaching is part of an online support system, coaches can review the participants’ objectives, most recent activities, successes, issues, and insights before the interchange. This is not only more efficient and accurate than a verbal retelling, but it also allows the coach more time to reflect and formulate the most helpful advice or questions—rather than having to respond off the cuff in real time. Third, online coaching leaves a record in the database. This is helpful to the learners in that they can refer to it again later. It also is helpful to the coach because over time he or she can build a library of well-articulated coaching responses to recurring issues. And it is helpful to learning leaders because they are able to monitor both the quantity and quality of the advice being provided. This is especially important when paid coaches are employed. Given the cost of professional coaching, we have been surprised by how many learning organizations invest substantial sums in executive coaching, but have no system to monitor the quality or value received. More and more companies are implementing learning transfer management systems to help them facilitate and monitor the amount and quality of the coaching being delivered. Feedback Sustains Learning Transfer Facilitating feedback is important not only to enhance the learning transfer climate (Figure D4.8) but also to sustain employee commitment and enthusiasm in general. Ken Blanchard has complained that the only way most people know that they are doing Simple acknowledgment of a good job is that “no one’s yelled at me lately” efforts is a powerful incentive. (Blanchard, 2004). Simple acknowledgment of participants’ efforts to transfer and apply their learning, especially from their own managers, is a powerful incentive to continue those efforts.

CH005.indd 250

6/17/10 3:29:53 PM


251

Direct evidence for how powerfully feedback impacts learning transfer efforts is illustrated by work we did with an international technology company. We reviewed the records from the learning transfer management system for more than five thousand employees who had participated in a company-wide skills program. We compared the behavior of those who requested and received feedback with that of those who requested feedback but received none. The difference was dramatic. The group that received feedback in response to their requests completed, on average, twice as many subsequent progress updates as those who asked for feedback but received none (Figure D5.11). This makes sense. If employees ask their managers for assistance with learning transfer and their manager provides it, that sends a clear signal that what they are doing is important to their manager and worth their time. Conversely, if they ask for feedback and the request is ignored, that sends an equally clear signal (intentional or unintentional) that their manager does not value the effort and that they should spend their time on other things. The conclusion is that learning organizations that want to see more of their efforts generate real business results need to pay attention to—and find ways to facilitate—the amount and quality of post-course coaching and feedback. It is FIGURE D5.11. EFFECT OF FEEDBACK. PARTICIPANTS WHO RECEIVED FEEDBACK COMPLETED TWICE AS MANY PROGRESS UPDATES AS THOSE WHO DID NOT

Number of Updates Completed

3.5 3.0 2.5 2.0 1.5 1.0 0.5 0 Received feedback

CH005.indd 251

Did not receive

6/17/10 3:29:54 PM

252


important that all the signals an employee receives regarding the importance of learning and development are consistent and mutually reinforcing. Practicing the Fifth Discipline means investing in materials, systems, people, and processes to provide, monitor, and continuously improve performance support for learning transfer.

Summary The Fifth Discipline that characterizes breakthrough training and development programs is that they deploy active, ongoing performance support during Phase III of learning, after the traditional “course” is over. By providing support in addition to accountability, breakthrough programs increase the learner’s probability of success, extend the learning period, and accelerate learning transfer leading to improved performance. The best corporate education programs approach Phase III support with the same rigor and customer focus as the best consumer product companies approach customer service. They recognize that the quality of post-training support affects training’s overall success, just as the quality of product and customer support affects a company’s success. Highly effective programs provide “owner’s manuals,” job aids, and online support systems to help employees apply course principles and practices. They engage managers throughout the process, and back them up with learning and coaching professionals as needed. They facilitate collaborative learning through social networking and provide efficient, online guidance and support systems. Companies that manage the “whole product” by investing a portion of their learning and development resources in Phase III support enjoy a greater return on their investment than companies that limit their thinking and investment to courses and classrooms. Use the checklist for D5 to ensure that you have a robust plan to provide the post-course coaching and performance support necessary to maximize the value of the learning experience.

CHECKLIST FOR D5

CH005.indd 252

✓ ❑

Element

Criterion

❑

Content

Course materials are organized and indexed in a way that makes them useful for later reference as well as for instruction.

❑

Content

Learners are provided with job aids, on-line materials, reference works, and so forth to reinforce course principles and support application during the transfer period.

6/17/10 3:29:55 PM


253

❑

Collaboration

Continuing peer-to-peer learning and sharing after the formal instruction period is encouraged and facilitated.

❑

Coaching

Learners are provided easy and efficient ways to engage their managers, subject matter experts, instructors, or other advisors during the transfer and application process.

❑

Supervisor Support

Managers are encouraged to provide coaching and are provided with relevant background materials and easy-to-use coaching guides to maximize their probability of success.

❑

Senior Management Support

Senior management acknowledges the criticality of managerial support by recognizing and rewarding managers who do a superior job of developing their direct reports.

Action Points For Learning Leaders • Think more holistically about your programs. • Where is the most value created? Where is it being allowed to dissipate? • What are the leverage points, especially in Phase III, where there is potential to accelerate value creation? • Compare where you are spending resources to where the most value is created. Are they proportional? • Interview a sample of participants three months after a learning or development program. • Find out where they have achieved success and what factors supported their efforts. Maximize these in subsequent program iterations. • Identify the barriers that impeded their progress or worked against learning transfer and application. Were they able to get the help and information they needed? • Review the materials (job aids, online information systems, and so forth) that are available to participants to support learning transfer. Are they as good as the support your company provides to its external customers? • Use your findings to create performance support systems that will maximize the successful transfer of learning to work. Include in the plan all three kinds of support: materials, systems, and people. For Line Leaders • Review the plans of proposed and ongoing programs at your company. • Are there adequate systems and resources to provide support for learning transfer?

CH005.indd 253

6/17/10 3:29:55 PM

254


• If not, leave no doubt about the importance you place on active support for learning transfer by rejecting any proposal in which the learning transfer process is not addressed. • At the same time, review your own actions and those of your team. • Is what you do consistent with what you say about the importance of a return on training and development? • Do you and your managers model the behaviors you expect of others? • Do you hold your team accountable for maximizing the value of what they and what their subordinates learn? • Do you recognize and reward managers who do a superior job of developing their direct reports? • If the answer to any of the last points is no, put your own house in order. Otherwise, employees will sense the disconnect between your words and your actions. Both your credibility and the effectiveness of training initiatives will suffer.

CH005.indd 254

6/17/10 3:29:56 PM

DOCUMENT RESULTS You, your leaders, and your investors are interested in learning only insofar as it improves performance and gets business results. —MICHAEL DULWORTH AND FRANK BORDONARO

T

he bottom line on learning is: Did it make a difference? Has it helped the organization achieve its objectives? That’s because the only justifiable reason to continue to invest time and money in training and development programs is if they are making a positive difference to the success of the organization. For that reason, the Sixth Discipline is to document results in a relevant, credible, and compelling way in order to justify continued investment and support continuous improvement. In this chapter, we discuss:

• • • • • • •

Why document results Root causes of poor evaluation Guiding principles A six-step process A caveat A checklist for D6 Action points for learning and line leaders

Why Document Results? Business leaders have a fiduciary obligation to use an organization’s assets responsibly—that is, in the way that maximizes the enterprise’s ability to fulfill its mission and achieve its goals. “Investment is about value creation in the future. 255

CH006.indd 255

6/17/10 3:30:56 PM

256


Investing is about committing resources now to create future value (Echols, 2008a, p. 61, emphasis in the original). That necessarily requires making decisions about asset allocation: How much of the available resources should be dedicated to marketing, how much to sales, to research, manufacturing, infrastructure, human capital, and so forth? What distribution best balances short-term realities with long-term opportunities? What mix of investments will create the greatest long-term value for shareholders, employees, and customers? Getting the allocation right—or wrong—has profound implications for the future of the company and its employees. There is no escaping the necessity of making such choices. Even in the best of times, there are always more good ideas for spending money than there is money to spend. When the economy slows, the decisions get even more difficult and crucial. Learning Must Add Value Learning and development initiatives consume time and money. Therefore, like it or not, learning competes with other departments, needs, and opportunities for corporate resources. Business leaders have to make hard choices among, for example, funding a promising new product idea to fuel future growth, hiring more sales representatives to improve the top line, investing in technology that will cut costs, or providing a training program to improve managerial effectiveness. All of the proLearning competes with posals have merit, but it is rarely possible to fund other departments, needs, all of them in a given year. So choices have to be and opportunities for made; leaders have to weigh and balance the stracorporate resources. tegic importance, relative contribution, and probability of success of all these disparate initiatives, winnowing them into those that will be funded, partially funded, and eliminated (Figure D6.1). Moreover, business leaders know that whatever they decide, their choices will be second-guessed by more senior leaders and, ultimately, shareholders; they need to be able to defend their choices. Savvy department heads understand the difficult choices senior leaders have to make and the pressures they are under. So they build the strongest possible case for their plans and budgets, buttressing their arguments with examples of past success, market research that suggests a favorable outcome, and projections of the impact. To compete successfully for time, money, and attention, learning organizations must put together equally compelling proposals. And to accomplish that, they need credible, compelling, and relevant evidence of results. Some learning and talent development professionals find the idea of competing for funds in this way offensive. In Idalene Kesner’s Harvard Business Review Case

CH006.indd 256

6/17/10 3:30:58 PM

Document Results

257

FIGURE D6.1. MANAGEMENT MUST DECIDE WHICH INITIATIVES TO FUND AND WHICH TO REDUCE, ELIMINATE, OR POSTPONE

Good Ideas to Grow the Business Ma

Sales

nu

Cust

fac

tur

ome

r Ser

ing

vice

t ct men u d p Pro velo e D gy nolo Tech

Marketing

Training

Management Filter (decision making)

Rejected

Postponed

Partly Funded

Fully Funded

“Leadership Development: Perk or Priority?” the learning director complains: “I hate it when people make those types of comparisons. First, we are talking about people in my case. That’s different from calculating the payback from a machine” (2003, p. 31). She is arguing that, because they deal with people issues, training and human resources should be exempt from having to support their case in the way that other functions do. We agree that demonstrating the value of learning is different from calculating the payback from a machine. Indeed, we think that trying to apply the same

CH006.indd 257

6/17/10 3:30:58 PM

258


methodology uncritically to both is a mistake. But there is no exemption from having to prove worth. Since the learning organization consumes resources that could be put to use elsewhere, it must make a compelling case for the benefit that will be generated. The argument for continuing to invest in learning and development has to be especially well articulated in times of fiscal constraints; it must project tangible benefits of significant importance to the goals of the organization. Requests for funding that are based on vague generalities or because “development is the right thing to do” are likely to be reduced or eliminated. Senior leaders want credible, compelling, and relevant data to support a decision to commit resources to training in preference to other investment opportunities. And they want a great deal more of it than most learning organizations currently provide. Business executives want learning organizations to spend far more time on business linkage and Leaders want credible, measurement of results than they do today (Dilworth compelling, and & Redding, 1999). A recent survey of Fortune 500 relevant data. CEOs found that the most important data senior leaders want from learning and development is evidence of business impact (Phillips & Phillips, 2009), yet this is the outcome least often measured (American Society for Training and Development, 2009). To find out what CEOs really want from training and development, Rothwell and colleagues interviewed more than one hundred chief executive officers. They concluded: “In keeping with a trend toward increased accountability in all organizational activities, CEOs want information that the organization is gaining value for its investments in performance improvement interventions. That trend is not likely to abate” (Rothwell, Lindholm, & Wallick, 2003, p. 218). Although in some organizations management has not yet required measures of training effectiveness, it is a mistake to wait until management demands that you document results because, as Daniel Tobin (1998) cautions: “If you wait until the CEO asks for an ROI study to . . . try to demonstrate how your training group adds value to your company, it is too late—the CEO has already decided to greatly reduce your budget or to eliminate the training group altogether.” A well-established record of adding value is the best defense for the learning and development budget in times of economic restraint. Begin to document and communicate results now to establish a strong track record of contribution before it is called into question. Continuous Improvement The second reason to document results is to drive continuous improvement. In today’s highly competitive, rapidly changing global economy, no organization can

CH006.indd 258

6/17/10 3:30:59 PM

Document Results

259

afford to stand still. If your organization is not improving while your competitors are, you are falling behind. The practice of continuous improvement originated in manufacturing, where it has contributed to extraordinary increases in quality and, simultaneously, reductions in cost. The principles and methods have subsequently been applied to other business processes with similar success; Jack Welch ascribes a significant portion of GE’s exceptional success to its aggressive six sigma improvement program (Welch, 2005). The power of a continuous cycle of improvement is enormous. In his bestselling book, Good to Great, Jim Collins (2001) likened the process of building a great organization to turning an enormous flywheel: “Good to great comes about by a cumulative process—step by step, action by action, decision by decision, turn by turn of the flywheel—that adds up to sustained and spectacular results” (p. 165). For training and development to remain relevant, competitive, and worthy of investment, learning organizations need to achieve continuous improvement to the same extent that other business functions have. Model for Improvement Langley and colleagues (1996) proposed a general model for improvement based on three core questions:

• What are we trying to accomplish? • How will we know? • What are our options? The importance of the first question—What are we trying to accomplish?— is obvious. The surest route to failure is to not understand where success lies. Improving learning—making it more efficient and effective—starts with D1, defining the business outcomes that training’s customers are striving to accomplish. The second question—How will we know?—is at the heart of D6, documenting results. Without some sort of tracking and evaluation system, it is impossible to know whether the current program is effective and whether any changes have been positive, negative, or superfluous. The third question—What are our options?—challenges learning organizations to come up with ideas for making good programs even better. The underlying assumption is that no matter how good a current process is, there is always opportunity for improvement. That concept is at the heart of kaizen, the continuous improvement philosophy that figured so prominently in Toyota’s spectacular rise to dominance (Liker, 2004). PDSA Cycle The Plan–Do–Study–Act (PDSA) cycle (Figure D6.2) was developed

by Dr. W. Edwards Deming, considered by many to be the father of the modern

CH006.indd 259

6/17/10 3:30:59 PM

260


FIGURE D6.2. THE DEMING PLAN—DO—STUDY— ACT QUALITY CYCLE

Study the results, act accordingly to improve the next cycle

AC T

O

U

N

T

S

Measure and analyze the effects

Plan test or change

PL A

DY

D

Implement the plan

quality movement. It is a simple but powerful tool that should be applied to efforts to continuously improve learning. The PDSA cycle is a never-ending process of planning improvements, implementing them, measuring the impact, and then acting on the results to initiate the next cycle. Astute readers will notice the parallels between the PDSA cycle and the four phases of learning discussed in D2 and see why we added Phase IV (assess achievement). Each cycle needs a check step to evaluate progress and inform subsequent action. Six sigma uses the related DMAIC (Define, Measure, Analyze, Improve, and Control) cycle (Islam, 2006). In both, measurement is central. Effectiveness and efficiency can only be improved by rigorously assessing the quality of the current output, impleEvaluation is the principal tool menting ideas for improvement, measuring the impact, and repeating the cycle. “[W]e see evaluthat learning leaders can use ation as the principal tool that learning leaders to strengthen capability. can use to accomplish this mission: building and

CH006.indd 260

6/17/10 3:31:00 PM

Document Results

261

strengthening learning capability so that organizations reap continuously better results from their learning investments” (Brinkerhoff & Apking, 2001, p. 165). An example of these principles in practice is Humana’s leadership development program, with its clear objectives, action plans, and documented results that were used to drive improvement on subsequent cycles (see Case in Point D6.1). Continuous improvement depends on feeding accurate assessment of the outcomes back into the system. Process improvement can be likened to hitting targets with artillery. The first few shells are “ranging shots.” After each, a comparison is made between the target and where the shell actually landed. The results are

Case in Point D6.1 Transformation at Humana, Inc. Ray Vigil, CLO for Humana, Inc., helped lead a critical transformation of leadership development at Humana. He and his team began with the end in mind—what leaders needed to be able to do differently and better to achieve the organizational vision and transform the organization and industry. Vigil also understood the need to “close the loop,” to put in place systems to measure and rigorously evaluate the outcomes of the program to show that it was achieving its objectives and to support continuous improvement. As he explains: “We targeted the top 150 leaders in the business and proposed to take them through a business simulator that required them to make decisions about investments in the consumer-centric strategy and what they really meant in practice. “At the end of each session, we asked them to identify an action plan that included the measurable outcomes that would occur if they were to implement the plan. After three or four months we went back and did a survey to ask them which of these action plans they had implemented and whether they had been able to identify any value. “We had savings come in on the order of $7 million. So we were able to justify the cost of the program within just a few months. There were also additional savings that were going to take longer to realize; so, in total, we thought that we would achieve at least twice as much once all the returns came in. “We really focus on taking the learning and applying it in action. We expect participants to pick a couple of things that they are accountable for, make changes, and measure the results. We let them know that we want to follow up, to check and see how they have done. “We did some after-action reviews to complete the cycle and used what we had learned to propose a second iteration. While reaction to the first simulator was very positive, participants feel the second one is even better. Their comments indicate that it is really accomplishing what we wanted: to have them see our business and the challenges we face in a very different way, to know what to do about it, and to change behavior.”

CH006.indd 261

6/17/10 3:31:00 PM

262


used to adjust the aim before the next shot is taken. With each cycle of aim, fire, measure, and assess, the accuracy improves. Eventually, every shot is squarely on target, but the cycle of checking and adjusting must continue to make sure they stay on target as conditions change. Now imagine the results of a gunnery crew that has been given the finest artillery piece, plenty of ammunition, and strong management support, but which is so busy shooting that they never bother to find out where their shots land. They carefully track the number of shots, cost per shot, shots per hour, and so forth, but they have no feedback on whether the shots are on target or a mile wide. With each shot, they alter the controls in the direction they think will help, but since they have no reliable information on whether or not it improves the outcome, they only occasionally hit a target. When they do, the shot cannot be repeated, because they don’t know which factors made it successful. In the absence of trustworthy outcome data, they are unable to improve their performance. Sadly, many corporate educators are in a similar situation. They know what the objectives are, they are adequately funded and can adjust their programs, but they either have no way of knowing the relevant results—or worse—they have only misleading “ersatz measures” that have little to do with hitting the target. Will Thalheimer argues that the lack of reliable measurement has impeded our professional development: “The fact is that we receive very little valid feedback about how we are doing as learning-and-performance professionals. Our impoverished feedback loops leave us in the dark. We simply don’t receive good enough feedback to improve our performance” (Thalheimer, 2008).

Evaluation Challenge Despite dozens of books on the subject and innumerable seminars on how to measure learning impact, it remains a source of considerable consternation for many learning organizations. More than a third are dissatisfied with their evaluation efforts (Anderson, 2010; American Society for Training and Development, 2009). In a study by the Wharton School of Business, learning executives ranked “how to measure and communicate value” as their top challenge (Betoff, 2007). We believe there are three root causes: • The first is confusion between internal process metrics and actual results. • The second is focusing too quickly on method or technology, without sufficient analysis of the audience, environment, purpose, and politics of the evaluation. • The third is a self-fulfilling prophecy that evaluation is too hard and too costly and doesn’t provide believable results anyway.

CH006.indd 262

6/17/10 3:31:01 PM

Document Results

263

Business Results Versus Process Metrics To be clear from the outset: Practicing the Sixth Discipline means documenting results, not just training metrics. By results we mean: outcomes that are of interest to the business and to which training contributed in a material way. This definition specifically excludes many of the metrics routinely collected and reported by learning organizations, such as number of participants, endof-course evaluations, self-ratings, alignment, knowledge tests, completion rates, cost of delivery, and so forth. That is not to say that such measures are not useful or even vital for the management of the training process and department. For Results are outcomes of example, although reaction ratings have interest to the business. almost no correlation with actual results (Alliger, Tannenbaum, Bennett, Traver, & Shotland, 1997), they may still be an important internal process check. As one CLO told us, “If people hate the pilot, I’m not going to roll it out.” Likewise, an end - of - course knowledge test may be important to meet a regulatory requirement, or to ensure that the material can at least be recalled, but learning as such (Level 2 assessment) is almost never the real objective; learning is simply an intermediate milestone on the path to improved performance (Figure D6.3). Interestingly, the U.S. Army Center for Lessons Learned doesn’t even consider that something has been learned, unless it results in a change in behavior (Darling & Parry, 2001).

FIGURE D6.3. MEASURES OF KNOWLEDGE AND SKILL ACQUISITION ARE ALMOST NEVER THE REAL MEASURES OF INTEREST TO THE BUSINESS

Timeline from Learning to Results Training or other structured learning

Acquisition of new skills and knowledge

Transfer learning to work

Intermediate, “in-process” check data

CH006.indd 263

Improved performance / business outcomes

Measures that really matter to the business

6/17/10 3:31:01 PM

264


For example, if the raison d’être for the program is to decrease the number of accidents by reducing unsafe work practices, then proving that trainees learned the proper procedures doesn’t answer the question: “ Did the training produce the desired result?” Knowing the correct procedure is not enough because “knowledge of what needs to be done frequently fails to result in action or behavior consistent with that knowledge” (Pfeffer & Sutton, 2000, p. 4). If the training does not result in safer work practices and fewer accidents, whatever learning may have occurred is irrelevant. (We realize that there are instances in heavily regulated industries in which proof of attendance or a test of knowledge is all that is required. A business objective in such cases to fulfill regulatory requirements or mitigate adverse legal judgments, so that evidence of satisfactory course completion may meet the customers’ needs, even in the absence of any evidence of impact on the job.) The point we are making here is that, while internal learning metrics such as participant satisfaction, alignment with the business, perceived utility, or even the amount learned may be essential to monitor the process, they should never be confused with—or presented as—results, because they are almost never the outcomes that are of interest to the business. The analogy is to the intermediate checks in a manufacturing process, which are essential to ensure that the individual components or sub-assemblies are working properly. But these internal measures are not what customers care about; their only concern is that the completed product performs to their expectations. That 99 percent of the in-process checks were satisfactory is irrelevant if the final product fails to perform as expected. Since companies invest in training and development with the goal of improving performance, better performance is the expected “product” of training. Expectations might include employees who do their jobs more efficiently or effectively; close a performance gap; launch a new product successSponsors want to see evidence fully; minimize risk; or improve customer satisfacof improved performance, not tion. The only measures, then, that are of interest to the business are those that answer the question: internal training metrics. “Did the training contribute to improved performance as planned?” In other words, did it produce a “return on the expectations?” (Kirkpatrick & Kirkpatrick, 2009). The fact that the participants completed the course, or rated it highly, or even learned a lot, is irrelevant if the initiative did not improve performance in the targeted areas. Much of the confusion about evaluating learning impact would be resolved by more clearly distinguishing between internal measures needed to manage the learning function and actual results, that is, the outcomes that matter to business management. When we define D6 as “document results,” we are referring only to the latter measures (Table D6.1). Thus, the first question that should be asked of any proposed measurement process—whether internally developed, recommended by a consultant, or

CH006.indd 264

6/17/10 3:31:02 PM

Document Results

265

TABLE D6.1. EXAMPLES OF INTERNAL LEARNING PROCESS METRICS VERSUS BUSINESS OUTCOMES Internal Learning Process Metrics

Business Outcomes

• • • • • • • • • •

• Documented increases in positive behaviors • Increased productivity • Higher quality/fewer errors • Improved customer satisfaction • Greater employee engagement • Reduced accidents and downtime • Shorter time to productivity • Better work product • Lower costs of production • Increased sales effectiveness

Number of participants Courses taught E-learning programs developed Number of courses completed Hours of instruction Costs per program, participant, or hour Post-test scores Satisfaction ratings Utility ratings Perceived alignment

promoted by a vendor—is: “Does this actually measure an outcome for which the program was created?” If the answer is “no” or “not exactly,” then the measure is not a “result” and should not be represented as such. We find that focusing on “outcomes that are of interest to the business” is a simpler and more effective approach than talking about “levels” of evaluation or CBA, ROI, IOL, IOB, or other alphabet soup introduced into the discussion in recent years. For some programs, the definition of success is a change in behavior, for others it is improved perception by stakeholders, and for others it is a quantitative measure of impact. It’s situational, like leadership. The critical thing is to measure what matters to the customer in a way that leads to informed action. Ends Versus Means In D1, Define Business Outcomes, we defined one of the traps for training as the “means versus ends” problem (page 54). Many learning organizations are caught by a similar pitfall when it comes to evaluation, namely, picking the process before clearly defining the purpose. A great deal of the debate about evaluation seems to quickly devolve into so-and-so’s method versus someone else’s method, about whether there are four levels versus five or six or none. Ultimately, of course, to execute an evaluation one must have a method, but the means should serve the end, not vice versa. Our concern is that that method seems to have achieved the ascendancy. Our antidote, described below, is to return to some fundamental guiding principles and let form follow function.

CH006.indd 265

6/17/10 3:31:02 PM

266


Self-Fulfilling Prophesy In her highly acclaimed book, Mindset (2006), Stanford University psychologist Carol Dweck cites hundreds of studies that demonstrate—across fields as diverse as business, school, sports, leadership, and even love—how powerfully a person’s mindset influences the outcome. People with what Dweck calls a “fixed mindset” believe that abilities are fixed; as a result, there are many things they simply cannot do or learn. People with a “growth mindset,” on the other hand, believe that humans have an almost infinite capacity for growth and, as such, they are convinced that virtually anything can be learned and mastered given sufficient interest and effort. A series of experiments illustrates how mindset plays out in corporate training and development. In one study (Kray & Haselhuhn, 2007), half the students in a negotiation skills class were given a paper to read called “Negotiation Ability, Like Plaster, Is Pretty Stable over Time.” The other half read a paper called, “Negotiation Ability Is Changeable and Can Be Developed.” The students were then given a choice of two negotiation tasks, one of which would allow them to show off what they already knew, but in which they would not learn anything new. The other task was more difficult; they were likely to become confused and make mistakes, but they would learn some useful negotiation skills. Whereas 88 percent of students who read the “ability can be developed” paper chose the “let’s make mistakes and learn” task, only half the students who read the “ability is fixed” paper did so. When the actual negotiation ability was evaluated in a subsequent study, those who believed they could learn and improve came out the clear winners: “The people who had learned the growth mindset persevered through the rough spots and stalemates to gain more favorable outcomes (Dweck, 2006, p. 138). In other words, whether students believed they could improve their negotiation skills or whether they believed they could not improve—they were right. What does that have to do with evaluation of learning? The 2009 American Society for Training and Development (ASTD) evaluation study makes an interesting observation: “We should note, however, that there might be a self-fulfilling prophecy problem here. Companies that simply give up on evaluation because they can’t ‘prove’ it has results might well be sabotaging their own learning effectiveness. After all, data from There might be a self-fulfilling this study shows that there’s a significant negative prophecy problem here. correlation between the extent to which respondents see this as a barrier and their own learning evaluation effectiveness” (American Society for Training and Development, 2009, p. 25). In other words, as learning professionals, we may have talked and read and written so much about how difficult it is to do evaluation well, that we have come to believe it. We may have created a collective fixed mindset that inhibits us from even trying, experimenting, learning, and growing our evaluation abilities.

CH006.indd 266

6/17/10 3:31:02 PM

Document Results

267

As we articulate below, we have a growth mindset about evaluation. We believe that evaluation skills can be learned and that there are innumerable opportunities to improve current practices. We are convinced that every learning organization can begin to measure meaningful results. Sure, we will make some mistakes along the way, but that is how we will learn and grow as a profession. Evaluation is not that difficult if you begin with some basic principles, apply common sense, and ask for help when you need it.

Guiding Principles The following core principles simplify the task of deciding how best to assess the results of learning programs. An effective evaluation is • • • •

Relevant Credible Compelling Efficient

Efficiency must always be considered There is nothing as useless as last, because efficiency only matters if doing efficiently that which the other three principles are satisfied. In should not be done at all. other words, measuring the wrong thing quickly is not efficient. Or, as Drucker put it: “There is nothing so useless as doing efficiently that which should not be done at all.” These four criteria should be the touchstones for any plan to assess training outcomes. They separate much of the wheat from the chaff and provide guidance for making choices, even though they do not in themselves define the specific questions, timing, data collection, or analytical techniques that should be employed. Given the vast range of topics covered in corporate training, the range of skills taught, objectives pursued, and impact expected, there is simply no one approach that can be used to evaluate all training programs. Or, as H.L. Mencken put it: “For every complex problem there is an answer that is clear, simple, and wrong.” While we do not believe that it is possible to reduce the evaluation of training to a simple “plug and play” solution, we are convinced that the principled approach outlined below will dramatically improve the credibility, reliability, and usefulness of documenting results. Effective Evaluation We define an effective evaluation as one that provides sound evidence to support informed decisions that are in the best interest of the organization. An evaluation isn’t effective if

CH006.indd 267

6/17/10 3:31:03 PM

268


it isn’t sound, if it doesn’t inform, if it fails to influence, or if it leads to erroneous and detrimental decisions. Effective evaluations always achieve a dual purpose: they prove (to an appropriate level of certainty) that the program did or did not achieve the desired outcomes and they provide insights to improve subsequent courses or iterations; they provide what Bersin calls “actionable information” (2008b, p. 13). To achieve these ends, the evaluation of a training program must be relevant, credible, compelling, and—once the first three are met—efficient. Relevant The first criterion of an effective evaluation is that what is measured is relevant. Relevance has two aspects: the measures must be relevant to the program’s objectives and they must be deemed relevant by the customer. Relevant to Program Objectives The first aspect of relevance is that what is measured ought to reflect the program’s objectives. That seems so obvious that we are almost embarrassed to mention it, were it not for the frequency with which this principle is violated. The most common violation is presenting positive-reaction data as if they were evidence that the program was a success. Positive reaction would be an appropriate outcome measure if the sole objective of the program were to show people a good time. We are overstating the point for effect, but training and development programs are not funded What is measured for entertainment value; they are funded in order to ought to reflect the solve business issues, seize opportunities, improve program’s objectives. service, enhance efficiency, or in some other way enhance the productivity of the trainees. The relevant measures, therefore, are whether or not the training program and transfer environment achieved those ends, not whether the program itself was enjoyable or even informative. Reaction data, especially perceptions of utility, may be useful internal process measures (see page 157), but they are not relevant measures of program effectiveness. Of greater concern is that optimizing for a positive reaction may actually sub-optimize the real outcomes (Case in Point D6.2). Examples So what measures are relevant? Some illustrations by way of analogy may help: If your company developed a new appliance that the engineers claimed used only half as much electricity as the competitor’s, it would be obvious how to substantiate the claim: measure and compare the electricity consumed by the two models. If you want to claim that your customer service is better than anyone else’s, the appropriate measure is obvious: You need to ask customers to rate your service and your competitors’. If you are in the pharmaceutical business, and you want to make the claim that you have developed a drug that treats influenza, it is obvious that you will need to treat a lot of people who have influenza and collect

CH006.indd 268

6/17/10 3:31:03 PM

Document Results

269

Case in Point D6.2 Better Reaction Scores May Not Be Better A European technology company was having serious concerns about one of its instructors. The instructor consistently received poor ratings on an end-of-course questionnaire that asked, in essence, “How do you feel about the trainer?” and “Do you think he or she was effective?” So they asked Neil Rackham, best-selling author of SPIN Selling, for his advice. When he looked into the problem more deeply, the results were startling. When the outcome measure was learning gains for students, the poorly rated trainer was actually among the best on staff. “In the end,” said Rackham, “Level I smile sheets had given management the exact wrong impression” (quoted by Boehle, 2006). Rackham’s story was corroborated by the work of Roger Chevalier at Century 21 Real Estate. Chevalier and his team tracked graduates of each course based on business results (number of listings, sales, and commissions generated post-training). He found that a trainer who was rated in the bottom third of all trainers by his students on Level I satisfaction surveys was found to be one of the most effective in terms of how his students performed during the first three months after they graduated. According to Chevalier: “There turned out to be very little correlation between Level I evaluations and how well people actually did when they reached the field” (quoted by Boehle, 2006). Why the discrepancy? Because when the criterion is on-the-job performance, the kinds of things that make an instructor effective (requiring role play, challenging participants to think, giving candid feedback) do not necessarily make him or her popular. Conversely, actions that boost reaction ratings do not necessarily improve learning or application. A training company related the following example: They had entered into a contract with a client in which they earned a bonus payment if their reaction scores exceeded a certain threshold. They were earning the bonus consistently, but, to their credit, they told their client that they felt that the contract needed to be renegotiated. They admitted to the client that they were diluting the program content and lessening the rigor of instruction in order to drive up reaction scores and earn the smile-sheet-based bonus. The point is that “what you measure is what you get.” But getting the highest possible reaction ratings does not necessarily equate to getting maximum possible value from training.

relevant measures like duration of illness, number of complications, symptom severity, and so forth. The specific methodology that you would employ and the specific parameters you would measure are different in each of the above cases, but the principle is the same: You have to measure what you claim. You can’t just say, “We know it works.” Nor is it enough to ask people whether they think their appliance uses less electricity or whether they think the medicine will help them get well more quickly. In the former case, you might end up in court with your competitor; in the latter, you would never get the drug approved.

CH006.indd 269

6/17/10 3:31:04 PM

270


What is the analogy to training? Fundamentally, training makes two claims: 1. Employees who attend a training program will perform better than if they had not attended. 2. The resulting benefit to the organization will outweigh the cost. As in the analogies above, the appropriate measures and methods of collecting them depend on the nature of the claim being made. For training, that means the specific business outcomes that the program was designed to achieve—what people were supposed to do better and differently—and the impact that was supposed to produce. Again, the principle is that you have to measure things that are directly related to what you claim. If the program was developed in order to improve some aspect of customer service, you can’t claim that it was effective unless you measure changes in that aspect of customer service. If the program was supposed to improve safety, then you need to measure changes in safety practices or outcome parameters in the areas addressed by the program. If it is a program to enhance leadership effectiveness, then you have to find and collect measures of leadership effectiveness that are considered relevant and credible in your organization (see Case in Point D6.3). Relevant to the Customer “Voice of the Customer” is a fundamental precept of six sigma

quality improvement methodology (Islam, 2006, p. 19). The concept is that, in the end, it is the customer who decides whether a particular product or service met expectations. It doesn’t The customers decide whether matter how good a company thinks it is; what mator not their expectations ters is whether the customers agree that what the were met. company delivers satisfies their wants and needs, and therefore, that they are willing to pay for it. The same principle applies to program evaluation. It does not matter how good or how compelling the training department thinks its evaluation is; what matters is whether it meets the customers’ needs and wants—whether it is relevant to them. For learning organizations, “customers” means those who make the budget decisions, since they decide whether or not to “buy” what training offers. Understanding the “voice of the customer” is prerequisite to designing both effective training and effective evaluation, which is why we placed so much emphasis on a consultative approach to D1, defining the business outcomes. The best time to listen to the “voice of the customer”—to have meaningful dialogue about what constitutes success and credible evidence—is before the program is created. If that has not happened, the second best time is before the evaluation is conducted. Nothing is more tragic, more wasteful of resources, or more

CH006.indd 270

6/17/10 3:31:04 PM

Document Results

271

Case in Point D6.3 Did It Work? Babcock & Wilcox, a venerable supplier of solutions to the energy industry, faced a challenge: 65 percent of its senior leadership was eligible to retire in the next five to ten years. That made it critical to develop the next generation of leaders before much of the tacit knowledge and experience of the current leadership was lost. The solution was the Leader to Leader program developed in collaboration with Dave Schrader of LeadingWork. This innovative eight-month-long leadership journey included: • A series of four workshops using 360-degree feedback, lectures, small group work, and participative exercises on leading, coaching, and developing new leaders • Ongoing peer coaching teams to reinforce accountability and support for results • Action - learning projects to apply concepts and skills to the participants ’ regular assignments • An online performance support system (DevelopmentEngine) to accelerate learning transfer and ensure that results were delivered Given the magnitude of the investment and strategic importance to the company, the senior team was keenly interested in knowing whether the approach was working and sought feedback from participants and from the participants’ managers and direct reports. They reasoned that if the program were actually improving leadership effectiveness, those who worked with and for the leaders should notice the difference and be able to cite specific instances of change. Data were collected by survey, interview, and review of the DevelopmentEngine entries. The results were outstanding. Although many of the participants in the program had been in managerial positions for years, nevertheless: • 62 percent were rated as more effective leaders after the program by their direct reports • 43 percent were rated as more effective at developing their own reports • 100 percent rated themselves as more effective overall In addition, numerous “success stories” provided specific examples of how participants were using their new learning and skills to improve their personal performance and the performance of their department or area. The survey results provided relevant, credible, and compelling evidence of the program’s value to the senior team, which subsequently directed that it be continued and expanded (Sturm & Schrader, 2007).

damaging to training’s reputation than to conduct an extensive evaluation only to find out after the fact that the approach was not what the customer wanted or valued (see Case in Point D6.4). Listening to the voice of the customer does not mean, however, that training and development professionals should blindly follow any customer dictate or

CH006.indd 271

6/17/10 3:31:05 PM

272


Case in Point D6.4 Voice of the Customer . . . Too Late A large company in the automotive industry spent over $100,000 to evaluate the ROI of one of its programs. The results were very impressive, suggesting a many-fold return. The training department was elated . . . that is, until they presented the results to the management team. The chief financial officer took one look at the data and said, “This is not how I define ROI at all; these data are worthless.” The point of the story is not whether the specific methodology they used was right or wrong in the abstract. The point is that it was not the right approach in their situation. Before embarking on the study, they should have listened to the voice of the customer and confirmed with the management team that the approach they proposed and the results that would be measured—good, bad, or indifferent—would be viewed as relevant and credible. The voice of the customer will always make itself heard; it is just a question of when. Far better to listen to the customer early, before you commit the time, funds, and effort to conduct a study, than afterward, when it is too late.

conduct an evaluation in a suboptimal or invalid manner. Indeed, a key part of being a trusted advisor is to bring your expertise and professionalism to bear and to have a point of view (Maister, Green, & The voice of the customer will Galford, 2000, p. 198). Dispassionately discuss the strengths and weaknesses of a suggested approach always make itself heard. in a dialogue with the key stakeholder(s); propose alternatives if appropriate. Summing Up Relevance Relevance is the sine qua non of an effective evaluation. To ensure that your evaluation is relevant, always listen to the voice of your customer when designing assessments and never impose language or methodology that the customer does not consider pertinent. Thus, what to measure to evaluate training’s impact is straightforward: what is relevant to the customer and reflects on what the program promised. Credible The second criterion for an effective evaluation is that the data, anal-

ysis, and conclusions be considered credible—trustworthy or believable—by the intended audience. A credible evaluation has five attributes: • • • • •

CH006.indd 272

Intelligible Reasonable Fair Rigorous From a credible source

6/17/10 3:31:05 PM

Document Results

273

Intelligible An evaluation has to be understandable by the intended audience to

be credible. People do not credit something they do not understand; no one likes to feel he or she is being “snowed.” If people do not understand the way the evaluation was conducted or explained, they will tend to discredit it. The phrase “intended audience” is key. What may be appropriate—or even required—for credibility in an academic journal may be unintelligible and counter-productive in a business setting. To ensure intelligibility: • Use the most straightforward approach that answers the question. • Avoid overly complex designs or arcane analytical techniques unless absolutely necessary. • Employ terms and concepts familiar to business leaders and eschew learning jargon or technical terms of your art. Reasonable In the evaluation literature, this is also known as face validity. In the

vernacular, it is known as the “sniff test.” Presented with the results of an evaluation, a business leader quickly forms an opinion about its “reasonableness.” Does the approach seem a reasonable way to answer the question? Did it make sense to gather these data, in these ways, from these sources? Do the conclusions follow logically from the findings? Do the claims seem too good to be true? The last question points to an interesting paradox: the better your results, the higher the level of skepticism and the greater the burden of proof required before they will be accepted as “true.” One of the paradoxes of learning ROI calculations is that they are so much better than typical business results (often claiming returns of several hundred percent) that they strain credibility with business leaders. Fair Business leaders are trained to be analytical and skeptical. They will be looking for anything that smacks of intentional bias or misleading claims. They will ask themselves: Was the evaluation even-handed, or was the study obviously biased to make the learning organization look good? A balanced and unbiased evaluation boosts credibility; an analysis that is perceived as “rigged” destroys it and undermines its effectiveness. Rigorous The evaluation must be sufficiently rigorous to be credible. Business leaders are accustomed to a high degree of rigor in the financial analyses, strategic plans, market research, and so forth that they review. They will reject out-of-hand an analysis that is superficial or amateurish. Our use of “sufficiently,” however, should alert the reader that this is another relative attribute, one that depends on the audience and culture, and for which, therefore, it is impossible to set absolute criteria.

CH006.indd 273

6/17/10 3:31:06 PM

274


Why not always strive for the highest level of rigor, then? Because evaluation itself takes time and costs money. Indeed, lack of time and resources are frequent excuses for not documenting the results of training and development more often. Therefore, evaluation should be done only to the level of rigor necessary to draw reasonably valid conclusions and satisfy the target audience. Not every study can or should be done Evaluation should be done only to doctoral thesis standards. Carly Fiorina favors to the level of rigor necessary the phrase “perfect enough” (Anders, 2003). Rob to draw valid conclusions. Brinkerhoff suggests we apply the legal standard: “beyond a reasonable doubt” (Brinkerhoff, 2006, p. 9). In other words, it is not necessary to prove the case for training’s effectiveness beyond any doubt, but only to the level of rigor necessary to give management confidence in its decisions. The 2009 ASTD study advises: “Companies should not try to prove that a learning experience affects results but rather show that, given the preponderance of evidence, it very likely does.” Absolute proof is probably never possible to achieve, even in the “hard” sciences. From a Credible Source Last, the credibility of an evaluation is strongly influenced by the reputation of its source. Or, as the Kouzes-Posner First Law of Leadership puts it: “If you don’t believe the messenger, you won’t believe the message ” (Kouzes & Posner, If you don’t believe the 2007, p. 38). In D2, we discussed the startling messenger, you won’t believe degree to which participants’ preconceptions of an the message. instructor influences their opinion of the instruction (page 72). The same effect occurs when managers review reports and proposals. Other things being equal, a report from a source known to be trustworthy will be considered more credible than the same report from an unknown or previously unreliable source. Hence, another important reason to document real results is to build and maintain the learning organization’s credibility and authority. Gaining a reputation for credibility takes time. Maintaining it requires constant attention: “Credibility is one of the hardest attributes to earn. And it is the most fragile of human qualities. It is earned minute by minute, hour by hour, month by month, and year by year. But it can be lost in very short order . . . ” (Kouzes & Posner, 1990, p. 24). To earn a place at the table, a voice in strategic discussions, and the resources needed to achieve its mission, the learning function must consistently deliver credible evidence of its impact. Summing Up Credibility To be effective, a program evaluation—the documentation

of its results—must first and foremost be credible. To be credible, it must be

CH006.indd 274

6/17/10 3:31:06 PM

Document Results

275

understandable by the target audience; contain no obvious bias or deceptive practices; appear reasonable (have face validity); be rigorous in proportion to the magnitude and import of the decision to be made; and come from a respected source. Compelling The third attribute of a truly effective evaluation is that it makes a compelling case for a particular course of action—for example, to continue, expand, revise, or discontinue a learning initiative. Even if your evaluation is relevant and credible, thereby fulfilling the first two criteria, it could still fail if it is not sufficiently compelling. So what makes a compelling case? At a minimum it is

• Memorable • Impactful • Concise Memorable Your key conclusions need to be memorable if they are to persuade

decision-makers to take your recommended course of action. You want to be sure that the key messages come to mind at the crucial moment of decision. But manag- Most reports from development ers, like the rest of us, are constantly bom- are too long, too dull, and just barded by hundreds of (often conflicting) plain uninteresting. messages. Most are promptly forgotten; only a few stick. You need to make sure your message stands out and is remembered above the background noise. In their best-selling book, Made to Stick, Chip and Dan Heath (2008) set out to answer the question: “Why do some messages stick, while others die?” After reviewing a host of “sticky” messages—from urban legends, to Aesop’s fables, to science analogies—they conclude that memorable messages have six attributes (p. 16); they are • • • • • •

Simple Unexpected Concrete Credible Emotional In story form

Those attributes are in sharp contrast to typical corporate presentations and reports, which tend to be complex, predictable, abstract, boring, and story-free. Having invested the time and energy to collect, compile, and analyze reams of data, everyone’s natural impulse is to put together a big, detailed report, print

CH006.indd 275

6/17/10 3:31:06 PM

276


and bind it, and circulate it to senior managers. That is a huge mistake. Few will read it, and fewer still will remember the key messages. Sullivan put it bluntly: “Most reports from development are too long, too dull, and just plain uninteresting” (Sullivan, 2005, p. 282). To create a compelling evaluation: 1. Simplify. Make sure there is a simple, clear, and unambiguous recommendation that follows from the analysis. 2. Surprise. Find an unexpected element or angle, if there is one, or present the information in an unexpected way. Use the latter approach with caution, however; there is a fine line between presenting information in an interesting and unexpected manner and being perceived as “gimmicky” or too clever to be taken seriously. Where that line lies depends on your company’s culture; some are much more tolerant of novel approaches than others. 3. Use stories to make the results concrete, emotionally interesting, and memorable. A story of a surprising result simultaneously satisfies two criteria for stickiness—unexpected and in story form. Even if you have solid quantitative measures, include a few select stories to make them more memorable (see Case in Point D6.5).

Case in Point D6.5 Too Many Numbers; Not Enough Stories We made the mistake of being too formal and too scientific in one of our early consulting engagements. We had been asked to help a company assess the impact of a major training/ change management initiative. To help gauge the results, we asked participants to supply examples of the results—if any—that they had achieved by utilizing what they had been taught. We collected hundreds of rich, detailed, specific, and concrete examples of ways in which the training had helped accelerate processes, eliminate waste, delight customers, and so forth. With the help of the finance department, we were able to assign credible dollar values to these and to compare the results to the costs of the program. The ROI was impressive. So we prepared what we thought were a series of powerful charts, tables, and slides. But we failed to include the stories. The presentation was eventually made to the board of directors of this Fortune 50 company, and they were positively impressed. But it is unlikely that any of them remembered a single graph or statistic the following day. Had we included a few of the remarkable stories of success, however, those stories would probably still be being told now, years later. Such is the power of stories. The lesson is that, while stories alone are not a substitute for quantitative analysis, they are the leavening that transforms an eminently forgettable presentation into a memorable one.

CH006.indd 276

6/17/10 3:31:07 PM

Document Results

277

There is a tendency in business to dismiss stories as fluff, anecdote, and insufficiently serious to be included in reports or presentations to management (Denning, 2005). That is a serious mistake. It flies in the face of what is known about what makes things memorable (see D3). The identification, collection, validation, and use of stories “worthy of the telling” is the core of the Success Case Evaluation Method for documenting results and “telling training’s story” in a memorable and compelling way (Brinkerhoff, 2003, p. 19). The bottom line is that, regardless of what other data collection and analysis methods you use, look for opportunities to include illustrative stories to make the message memorable. Impactful A compelling case for training makes an impact on those who read or

hear it. Include a clear “call to action”: what you want the audience to do as a result. Impact is a matter of both substance and style. Substance is prerequisite; the analysis needs to demonstrate an outcome that is large enough and strategic enough to capture people’s attention. To make a compelling case, you first have to have the data, then have to show that the consequences of pursuing (or failing to pursue) your recommended course of action will have significant impact on those making the decision. Style is about how you convey the message. You have to do so in a way that compels people to take action (see Case in Point D6.6). That means communicating to them in their language about the things they care most about. Don’t undersell your findings. Think about what matters most to the target audience and emphasize those aspects (see “Selling the Sizzle” below). It is a tragedy to

Case in Point D6.6 Data Are Compelling Steelcase University’s mission is to “promote optimal individual and organizational performance, leading to business success.” Recognizing how vital managerial support is to achieving that mission, Steelcase U has begun tracking the level of managerial engagement (pre- and post-course discussions between managers and program participants) and including those metrics in reports to management. Even more creatively, they post the results on the university’s Learning Portal, where they are visible to all managers and employees to help drive awareness, recognition, and behavior change. According to Faye Richardson-Green, director of global learning and development: “Senior management at Steelcase has always recognized the importance of managerial support for learning. Having current data on the actual levels of engagement, however, has really helped us to raise their awareness and interest in managing the process.”

CH006.indd 277

6/17/10 3:31:07 PM

278


have great results and then to communicate them in a manner that fails to have impact. Concise Finally, compelling analyses and recommendations are short and to the

point. People are more likely to be swayed by a short, sharp analysis than by a long and convoluted argument. Be sure your chain of reasoning is easy to follow and include only as much detail as is necessary to make the case. Summing Up Compelling The third attribute of an effective evaluation is that it is compelling—that is, it is sufficiently persuasive to cause your target audience to take your recommended course of action. Regardless of the exact methodology of your evaluation, be sure your report of the analysis is memorable, impactful, and to the point. Efficient Finally, an evaluation has to be efficient—but only if it already meets

the criteria of relevant, credible, and compelling. Efficiency is important because evaluation itself consumes time and resources. The challenge is to produce the needed information at the lowest possible cost. Efficiency is also vital to ensure that the results of the evaluation are available quickly enough to be useful. “Part of the success of any evaluation effort is the timeliness of the evaluations findings. . . . Sometimes [delivery dates] are related to the budgeting cycle, a deadline for a request for funding, a production and delivery launch, or a ‘need to know’ before taking other actions. When evaluations miss these deadlines, their findings may be of limited use” (Russ-Eft & Preskill, 2009, p. 29). Use efficient methods of data collection and “leading indicators” of effectiveness (see below) to ensure that results are available in time to inform decisions about continuing, enlarging, eliminating, or revamping programs. The cost and time required to gather data can often be reduced by using metrics that are already being collected as part of normal business operations, individual assessments, or as part of driving learning transfer. If additional aspects of performance have to be assessed, look for ways to automate data collection. For large programs, use a random sub-sample of the whole population; trying to include every participant drives up cost but, beyond a certain point, adds no additional insight. In the drive for efficiency, however, be careful not to sacrifice too much accuracy or relevance. Tradeoffs between rigor and efficiency are inevitable in the design of an evaluation in order to stay within time and budget constraints. Such choices, however, should be made thoughtfully, conscious of what is being lost and gained. If the number of compromises required becomes so great that the credibility and relevance of the study will be destroyed, then it may be better not to pursue the

CH006.indd 278

6/17/10 3:31:08 PM

Document Results

279

evaluation. Easily accessible, but invalid, measures can lead to decisions that are not in the best interests of the company or its employees (see Case in Point D6.2). The advent of relatively simple, inexpensive, online surveys has greatly increased the efficiency with which information can be collected from a large number of people. Unfortunately, they have also contributed to a decline in validity. In part this is because surveys are misused and in part because many are poorly constructed or administered. For some program objectives, for example improving the perceived quality of service, opinion is an appropriate measure. For others, it is a cheap but inaccurate proxy. Asking people to rate how much they think they learned, for example, is not the same as actually measuring the amount of learning that took place (Level 2 How the survey is constructed assessment in learning jargon); it should profoundly affects its validity not be represented as such. Asking peoand reliability. ple how valuable they expect the result of training will be is not the same as evaluating how valuable the results actually were. We are reminded of the cartoon of the artist standing in front of the blank canvas and proclaiming: “Ah, it is a true masterpiece that I am about to paint.” Keep in mind that how the survey is constructed—wording of questions, length of survey, choice of rating scales, and so forth—profoundly affect its validity, reliability, and completion rate (Babbie, 2010). Get expert assistance if you are not well-versed in questionnaire design. Table D6.2 summarizes the attributes of an effective evaluation. TABLE D6.2.

CH006.indd 279

ATTRIBUTES OF AN EFFECTIVE EVALUATION

Relevant

• To the course objectives (desired business outcomes) • To the customer

Credible

• • • • •

Compelling

• Memorable • Impactful • Concise

Efficient

• Meets the first three criteria • Makes good use of time and money

Intelligible Reasonable Fair Rigorous From a credible source

6/17/10 3:31:08 PM

280


The Six-Step Process of Evaluation There are six steps to D6, document results (Figure D6.4): 1. 2. 3. 4. 5. 6.

Confirm outcomes that matter. Design the details. Collect and analyze the data. Report the findings. Sell the sizzle. Implement improvements.

1. Confirm Outcomes That Matter: The D1–D6 Connection Deciding what to measure is the single most important decision in the process of documenting and evaluating results. Ideally, this should have been part of the D1 discussion with the sponsor in which the business outcomes and definition of success were explored and agreed to. If you used the Outcomes Planning Wheel (Figure D6.5) in D1, then in D6 this step consists simply of reviewing your summary of the discussion and, if the program has evolved, reconfirming the measures, timing, and data sources. Whether it is part of the D1 discussion, or part of designing the evaluation for an existing course, picking the right parameters to measure includes:

FIGURE D6.4.

CH006.indd 280

A SIX-STEP PROCESS FOR EVALUATING RESULTS

1. Confirm the outcomes that really matter

2. Design the details

3. Collect and analyze the data

4. Report the findings

5. Sell the sizzle

6. Implement improvements

6/17/10 3:31:08 PM

Document Results

281

FIGURE D6.5. OUTCOMES PLANNING WHEEL

4.

1.

What are the specific criteria of success?

What business need(s) will be met?

3.

4 1 3 2

Who or what could confirm these changes?

2. What will participants do differently and better?


• Understanding what metrics actually matter to the stakeholders • Making sure that training ⫹ learning transfer are likely to change them • Reducing to the “critical few” Use the flow chart in Figure D6.6 to help you. Understand What Metrics Actually Matter to Stakeholders Effective evalua-

tions start with the voice of the customer—a thorough understanding of the outcomes that really matter to the people paying the bills. You should not proceed with any measurement scheme until this question is answered; to do so is to court

CH006.indd 281

6/17/10 3:31:09 PM

282


FIGURE D6.6. DECISION TREE FOR SELECTING PARAMETERS TO MEASURE

Stop, start over

NO Do we know, precisely, what measures matter to the customer? YES

Pick another parameter and confirm with customer

NO Is training + support likely to significantly change these measures? YES

Reduce number being tracked to increase speed and efficiency

NO Have we identified the “critical few” plus some insurance? YES

Decide when to measure

disaster (see Case in Point D6.4). The success of all your efforts hinges on getting the outcomes right. To understand what really matters to your constituency requires dialogue. You cannot rely on measurement gurus, vendors, or textbooks to tell you which metrics matter most; they do not know your customer, your program, or your culture. Whether you use the Outcomes Planning Wheel, the GAPS! methodology (Robinson & Robinson, 2008), a six sigma tool like quality functional deployment (QFD) (Islam, 2006), or some other method, the key is to be systematic in your approach so that you understand how the program sponsors define success and what they will consider as credible evidence that it has been achieved. In our experience, these are rarely the learning management metrics most often tracked. In some ways, the advent of computer-based learning management systems has made matters worse, since they accelerate the collection and reporting of activity and cost data, but rarely results. In the ASTD study (2009), the second most commonly noted barrier to effective evaluation was the lack of useful evaluation data as part of the company’s learning management system (LMS). “This may signal that, as companies rely more heavily on software solutions to create and deliver learning, they become dependent on those systems to provide metrics. If their system does have the proper reporting functions, it may not be robust enough to provide the data they need for effective evaluation” (p. 25). The seductiveness of measures such as the number of instructional hours, courses, number of trainees, and expenditures is hard to resist, although in reality

CH006.indd 282

6/17/10 3:31:10 PM

Document Results

283

they only quantify the size of the investment, not whether it produced any benefit. While such measures may be essential to the management of the learning organization (see “Business Results Versus Process Metrics”), they are rarely the outcomes for which the program was created or that sponsors are willing to pay for. According to Phillips and Stone (2002): “Because costs can easily be identified, they are communicated in all types of ways, such as the cost of the program, cost per employee, cost per contact hour, and cost compared to industry standard. While these methods may be helpful for efficiency comparisons, they have no relationship to results” (p. 202). Indeed, in the absence of true outcome measures, it is impossible to say whether a company is investing too much or too little in training and whether it is getting value for its investment. “If you are spending just $1 on training but getting no business value in return, then you are overspending” (van Adelsberg & Trolley, 1999, p. 75). To address the shortcomings of their learning management systems for program evaluation, more and more companies are turning to purpose-built training measurement systems, such as Metrics That Matter by Knowledge Advisors, Inc. Such systems greatly facilitate gathering and analyzing post-training data, but they are only as good as the relevance and credibility of the data they collect. For example, some companies continue to invest significant resources to efficiently collect and slice and dice end-of-course evaluations, even though CEOs ranked CEOs ranked end-of-course them dead last among eight possible meaevaluations dead last. sures in terms of the data they want to see (Phillips & Phillips, 2009). Regardless of how the metrics are collected, the success or failure of the evaluation depends on collecting the right data—credible evidence of the outcomes the business values. Make Sure That Training ⴙ Transfer Is Likely to Produce Meaningful Change An

important check-step when selecting metrics by which to evaluate training programs is to be realistic about whether the learning and transfer are likely to cause measurable changes in those metrics. The business sponsor might be very clear about how he or she wants to see training’s impact measured, but these may not be the best measures—either because they take too long to become manifest or because the impact of training will be completely obscured by other changes in the business or environment. For example, the ultimate business goal of a training program on consultative selling is to increase sales. It seems logical to conclude, therefore, that the most relevant and credible measure of the effectiveness of the sales training would be to track sales figures. Theoretically, yes. But in a dynamic market, the effects of the training may be overshadowed by a new product introduction (by the company

CH006.indd 283

6/17/10 3:31:10 PM

284


or by a competitor), an uptick or downturn in the economy, seasonality, or other non-training causes. The longer the sales cycle, the greater the impact of such unrelated causes will be. While it may be possible to isolate the effect of training through control groups, multivariate analysis, or other methods (Echols, 2008b), it is usually simpler and more informative to choose a different parameter—one that is more closely related to the training itself (see Leading Indicators below). Reduce to the Critical Few For any intervention, a very large number of things

could be measured. Many can be eliminated because they fail the criterion of relevance or credibility, but you may still be left with a fairly large number of potential measures. It is essential, for reasons of cost, time, and ease of understanding, to reduce the universe of things that could be measured to the “ critical few ” that More is not necessarily better. will be measured. More is not necessarily better. As Charles Jennings, global head of learning at Reuters, explains: “If you focus on very detailed metrics, you could end up in a situation where you are perfecting the irrelevant and caught up in the complexity of the process. You have to look at the effort versus the value you are getting out of it” (quoted in Redford, 2007). The more parameters you track, the more it costs (in time as well as money), the more you have to explain, and the more likely you will encounter something that you cannot explain. The real message can easily be lost in the welter of measures of marginal value. Remember that a clear, crisp, straightforward analysis is more likely to be compelling than a complex, long, convoluted one. On the other hand, you do not want to “put all your eggs into one basket” by measuring only one parameter. Unexpected problems with data collection or analysis can occur in the course of any study, so it is essential to have complementary or alternative measures in place. Also include among “the critical few” some questions and data points that will allow you to improve the program in subsequent iterations. For example, have participants rank which aspects of the program proved the most helpful in actual practice or indicate the obstacles they encountered in the application phase. Continuous improvement is essential to stay competitive. Design the evaluation so it serves to both prove and improve learning’s value. The need to establish a “chain of evidence” is a common theme in the literature on evaluating corporate training. This is typically taken to mean the need to measure each of Kirkpatrick’s four levels sequentially—first demonstrating positive reaction, then that learning took place, then that behavior changed, before finally assessing results (Kirkpatrick & Kirkpatrick, 2009). We respectfully disagree. While a “chain of evidence” makes perfect sense in theory, no business leader in our

CH006.indd 284

6/17/10 3:31:11 PM

Document Results

285

experience has ever required it. If the analysis shows that those who were trained achieved the desired outcomes to a greater degree than those who were not trained, managers take for granted that the participants must have learned something in the training; they don’t need a separate Level 2 analysis. Indeed, Holton (1996) has argued that the four-level model is really a taxonomy rather than an evaluation model, because it implies causal relationships that have not been demonstrated by research. Given the limited time and resources available for evaluation, we recommend against expending them on measures that have little importance to the business and poor predictive value. Forget the dogma; focus evaluation efforts on the critical few relevant outcomes and measure them directly and credibly. 2. Design the Details of the Evaluation Once you have selected the critical few measures to evaluate that really matter (to the customer), it is time to roll up your sleeves and design the details of the evaluation process. That includes deciding when to collect the data, recognizing leading indicators, choosing comparators, selecting appropriate data collection techniques, and planning the analysis, each of which is elaborated on below. Decide When to Collect Data Deciding when to collect the data is almost as

important as deciding what to collect. Since D6 is about documenting businessrelevant results, that virtually excludes any data collected at the end of the period of instruction, since the participants will not yet have done anything that would generate results. Participants need time to transfer their new knowledge and skills to their jobs and apply them long enough to improve performance. Relevant results Deciding when is almost as can only be collected after enough time important as deciding what. has elapsed for this to occur. For some types of training—such as customer service or computer system skills—demonstrable improvement might be manifest within days. For others—such as strategic selling, management, or leadership training—it might be weeks or months before the desired effects can be documented. Kirwan (2009) notes that delaying evaluation also helps give the learners perspective: “Evaluation of a training event will be more rational and considered at some distance from the event, when not caught up in the event itself ” (p.134). The general principle is to assess the results at the earliest point at which the outcomes of interest are expected to be evident. The sooner the impact can be assessed, the more useful the data are for continuous program improvement and deciding whether to expand or contract the program’s scale. The greatest challenge

CH006.indd 285

6/17/10 3:31:12 PM

286


is in evaluating programs where the desired changes take a long time to implement and produce results. If you attempt to assess the impact of such “long-tail” programs too soon, you will underestimate their true worth. On the other hand, the longer you wait, the more factors influence the results and the more difficult it becomes to gauge the actual impact of the training (see Figure D6.7). Look for Leading Indicators For programs that require a long time before results

are evident, look for leading indicators. The concept of leading indicators stems from Kaplan and Norton’s groundbreaking work on the “Balanced Scorecard” (Kaplan & Norton, 1992). The balanced scorecard has been widely accepted as a better way of tracking corporate performance than traditional financial statements because it includes both the traditional financial measures—which reflect actions already taken—as well as forward-looking measures of operational activities— such as customer satisfaction and products in the pipeline—that are leading indicators of future performance. Even if a company is doing well today, an empty

FIGURE D6.7.

TIMING AND EVALUATION

fa ct or s in g on -tr ai n

Magnitude of Impact

Evaluate too late and the benefit of training may be overwhelmed

Evaluate too soon and you underestimate true value

s ct fe ef

n of

e iv at l u m r d transfe Cu ing an n i a tr of cts e f ef ve i t la mu Cu

Time Since Training

CH006.indd 286

6/17/10 3:31:12 PM

Document Results

287

product pipeline or a growing proportion of disgruntled customers predicts a less than rosy future. By analogy, if you can demonstrate that people are actively utilizing the more effective approaches they learned in a training program, that is a leading indicator that the desired outcomes will be attained, even if the ultimate results may not be apparent for some time. In most cases, the leading indicator of training’s success is the adoption of the vital behaviors that were identified in the second quadrant of the Outcomes Planning Wheel. The business need, for example, might be to increase sales. A specific training initiative might contribute to this goal by teaching sales representatives to ask more probing questions before launching into their sales pitches. An early indicator of success, then, would be an increase in the number and quality of the questions they ask, compared to the norm prior to training. A significant increase in the desired behaviors would be observable before actual sales numbers changed and would be less subject to non-training factors such as seasonality, pricing promotions, and competitors’ actions. Often, management already accepts the correlation between certain behaviors and business results, in which case they willing to define training success as clear evidence of behavior change. David Brennan, CEO of AstraZeneca, explains it this way: Obviously, people want to quantify business results. While that is an important measure, the kinds of things that are much more measurable are the quality of the behavioral outcomes of the programs. If we believe that coaching is an important part of the performance management process, and we put coaching programs in place, then what we want to measure is the quality of the coaching—not as perceived by the coaches, but by the people being coached. If we say that demonstrating behaviors about the team’s passion for winning is important to us, then what we have to do is ping the environment to see whether or not those behaviors are being demonstrated by people and that they are being reinforced by management. There are other, harder measures you can put in place . . . but in terms of the operating environment of the organization, evaluation needs to be much more focused on the behavioral outcomes that you’re driving for (Brennan, personal communication, 2004).

Time series designs are especially useful in evaluation (Brethower, 2009). By capturing periodic “snapshots” of what is happening post-training, online learning transfer management systems provide early indicators of the extent to which participants are applying what they learned as well as examples of the positive effects being noticed by their teams, peers, managers, and coaches. Organizations that use learning transfer management systems, such as Kaiser

CH006.indd 287

6/17/10 3:31:13 PM

288


Permanente, find it valuable to be able to track learning transfer efforts as early indicators that training is having the desired effect. “ Just as learning management systems have made it possible to track courses, enrollment, and attendance for Evaluation needs to be much thousands of employees and programs, followmore focused on behavioral through management systems have made it outcomes. possible to track learning transfer and application for thousands of participants, programs, coaches, and managers. . . . Prior to the implementation of the follow-through management system, L&D [Learning and Development] had no insight into what happened following training and development. There was no way to easily identify success cases or the failures of implementation to drive continuous improvement” (Chai, 2009). Choose Comparators Making the claim that training improved performance begs the question: “Compared to what?” Is the claim that people’s performance was better after training than before? Or is the claim that those who attended the training achieved better results than those who did not? One way or another, there is always a comparison, expressed or implied. So part of designing (or contracting for) a program evaluation is to decide what the results from the trainees will be compared to. This decision is important, because any time you make comparisons, you may be challenged to show that the comparison is valid and, therefore, that your conclusions are justified. The credibility—and therefore the effectiveness—of the evaluation depends on the comparisons you make being viewed as relevant and fair. There are two basic types of comparison: historical and contemporaneous. A typical historical comparison would be to measure each person’s performance before and again after training. An example would be Pfizer’s comparison of 360-degree feedback results before and after their leadership programs (Kontra, Trainor, & Wick, 2007). Another approach is to ask observers to compare someone’s current performance to his or her past performance. This approach was used by Goldsmith and Morgan (2004) to demonstrate the importance of follow-up. Because each person serves as his or her own “control,” this design avoids many sources of bias that can arise when comparing two different groups of people. It has high face validity: If the great majority of trainees do better, or are rated better, after training, on the face of it that makes a pretty compelling argument that the training added value. On the other hand, the approach is not perfect. People’s performance tends to improve with experience anyway, so that some or all of the improvement in the

CH006.indd 288

6/17/10 3:31:13 PM

Document Results

289

second observation might simply reflect the experience curve, rather than the effect of the training as such. Likewise, approaches that ask people to compare someone’s present (post-training) to their past (pre-training) performance rely on human memory, which is demonstrably fallible. Another challenge to historical compariThere is always a comparison, sons is that other, unrelated changes in the expressed or implied. environment are likely to have occurred between the pre- and post-training assessments. Such changes could contribute (positively or negatively) to any observed change in performance—especially if the interval between measures has been several weeks or months. All that having been said, before-after comparisons are generally a good choice that is well understood and accepted by business leaders. Contemporaneous case-control evaluations obviate many of the concerns about historical comparisons. A case-control study is the classic experimental laboratory approach of comparing an experimental group (in this case, people who received training) to a control (untrained) group. An example would be Laurie Cusic’s (2009) study of the impact of coaching and follow through on sales leadership effectiveness training at Wyeth. Ideally, subjects should be assigned at random to one group or the other to avoid any systematic differences (bias) between the populations being compared. Randomization is rarely practical in corporate training programs, however. It is probably unnecessary for practical purposes, provided learning professionals apply common sense to avoid the most egregious sources of bias. The question to ask yourself when choosing comparison groups for a case-control study is: “Are there significant pre-existing differences between the groups that would influence the outcome?” It might seem reasonable, for example, to compare the results at one location, where a new approach is being piloted, to a “control” location in which training had not been implemented. But since other, potentially significant differences probably exist between the sites, it may be hard to be sure that any differences measured were, in fact, due to training. It would be better to compare groups of employees within the same environment—for example, those who have already attended training and those who have not yet gone—or to take pre- and post-measurements at both locations and compare the rates of change. A few of the other obvious sources of bias to avoid include: • Selection bias—including only those more likely to produce a positive response, for example, asking only those who did well to rate the training program or including only “high-potential” employees in the trained group. • Questionnaire or interviewer bias—setting up data collection forms or processes that “lead the witness” by making it easier to answer in the affirmative. • Response bias—always a concern with surveys, especially if the response rate is low. When only a small number of the people surveyed respond, it raises the

CH006.indd 289

6/17/10 3:31:14 PM

290


concern that they may not represent the group as a whole, but only those who were especially positive or who had an ax to grind. • Reporter bias—if employees think that their identities might somehow become known and that negative responses could result in reprisals (even if this is not true), they tend to “sugar coat” their responses or select “politically correct” answers. In the same way, unsubstantiated self-reports tend to be overly optimistic about the magnitude of improvement or contribution. They can be made more credible by “triangulating” the results by using a second method of assessment or data from another source, such as their manager (Kirwan, 2009, p. 132). Consult a textbook on evaluation (for example, Russ-Eft & Preskill, 2009) or on experimental design (Ryan, 2007) for a more thorough discussion. The key point is that for an evaluation to be credible, any comparisons must be perceived as legitimate. When bias cannot be completely eliminated, acknowledge it and its potential impact on the conclusions. Norms and Benchmarks Sometimes it is informative to compare your company’s

results to those of other companies or your industry as a whole. Online data collection systems have facilitated such cross-company benchmarking and comparisons. It is important, however, to note that norms and benchmarks are not the same, although they are often confused or used interchangeably. Benchmarking is comparing one’s own performance to that of a recognized industry leader or “best practice.” Norms are averages or typical values for the industry as a whole. Whether it makes sense to compare your organization’s training results to industry benchmarks or industry norms depends on much the same criteria as does the choice of internal comparators: • • • •

Are the measures relevant to the sponsors and course objectives? Are the data credible? Are the other courses, programs, and companies truly comparable? Does the comparison lead to better decision making?

For example, benchmarking the time and cost of developing an hour of elearning against the most efficient companies out there makes sense—assuming the complexity of the topics and the quality of the output are similar. It might not make sense, however, to compare your company’s learner satisfaction ratings against industry norms, since optimizing satisfaction ratings does not necessarily optimize results (see Case in Point D6.2). Having the highest satisfaction ratings in your industry does not mean that you are necessarily the most effective at turning learning investments into business results.

CH006.indd 290

6/17/10 3:31:14 PM

Document Results

291

Isolating the Effects of Training Much has been written about the importance of isolating the effect of training. To make the claim that training made a difference, it is important to try to “isolate” the effect of training from concurrent effects by means of comparison groups or data as discussed above. Using this approach and multivariate analysis, for example, Bellevue University’s Human Capital Lab was able to isolate the effect of training for car salespeople from all the other influences (size of dealership, location, experience, and so forth) and calculate the impact of training in terms of the number of additional vehicles sold (Echols, 2008b). While it is important to try to control for other influences to the extent possible (by design, statistical analysis, or both), two important points need to be made: (1) in complex, real-world environments, it is probably impossible to completely isolate the effect of any one variable and (2) it is seldom necessary for practical decision making. We agree with Brinkerhoff (2006) that trying to calculate the percent of an improvement that should be ascribed to training is an exercise in futility. That’s because, as we stressed in D4 and D5, the effects of training can never be disambiguated from the transfer climate. The two are inextricably bound to one another. The effects of training and The transfer climate—and in particular the the transfer climate are participant’s immediate supervisor—has a inseparable. profound impact on whether or not the program produces business results. When training fails to produce results, it is more often a reflection of an inadequate transfer environment than inadequate instruction as such. It follows that, when training succeeds, it is the result of training and support for follow through; you can never completely separate the two. Trying to calculate percent contribution may be a meaningless exercise. By analogy: If you were to fly from New York to San Francisco, how much of your success in reaching your destination was due to the pilot? What percent did the engines contribute? How about the wheels? Trying to answer these questions provides no insight in how to run an airline. The fact is, the pilot, the engines, and the wheels are all 100 percent responsible. Had any one been missing, you would not have gotten off the ground; each is essential to the system that got you there. It is the same for training. The preparation, the on-the-job application, and the performance support are all part of the system needed to achieve the result. In carefully controlled trials, it may be possible to isolate the effects of the “training system” through multivariate statistical techniques, as in the car salespeople study above. And it may also be possible to assess the relative impact of parts of

CH006.indd 291

6/17/10 3:31:14 PM

292


the system—like managerial engagement—as in the American Express (2007) or Saks and Belcourt (2006) study to help drive continuous improvement. The common practice, however, of having participants estimate the percent of improvement that was due to the training suggests that they would have achieved the remaining amount even without the training. Yet that is not what you find if you compare them to a control group of people who did not attend the training. Moreover, the reliability of such estimates has never been rigorously demonstrated; indeed, it is a very difficult question to answer. Try it yourself: How much of your personal success has been due to learning to read? If you had not learned to read, you would not have your current job at all. So learning to read was essential for your success, but any calculation beyond that adds no clarity. In our view, an evaluation needs to show that the training was essential (participants point to it as critical to their success or those without it did not achieve the same results), but attempting to dissect the contribution beyond that threatens the credibility and usefulness of the whole evaluation. Summing Up Metrics and Comparators Such considerations are not merely academic. The purpose of evaluation is to support informed decision making. The choice of the wrong metrics or comparators can lead to erroneous conclusions and, therefore, detrimental decisions. An evaluation that overestimates the true value of a program would lead to a wasteful decision to roll out or expand a marginally effective initiative. Perhaps even more tragic would be to scale back or eliminate a program because a poorly designed evaluation underestimated its true value. The challenge is to find the right level of rigor for the magnitude of the investment and the strategic importance of the initiative. You are not trying to conduct Nobel Prize–winning research. On the other hand, there are real consequences to the business and indiThe purpose of evaluation is to support informed decision viduals of getting it wrong. Use the checklist at the end of this chapter to help evaluate the proposed making. design. Select Data Collection Techniques How will you get the data on the groups and metrics you’ve selected? As always, “The devil is in the details.” The parameters you have chosen to measure affect the evaluation’s relevance; how you measure them affects its credibility. Thus, you need to be sure that the client agrees not only with what you propose to evaluate, but also how you propose to gather the information. “An evaluation plan functions as a contract between the evaluator(s) (whether they’re internal or external) and the organization, and is the document that guides them through the evaluation process” (Russ-Eft & Preskill, 2009, p. 142).

CH006.indd 292

6/17/10 3:31:15 PM

Document Results

293

For example, the sponsor might agree that a certain change in behavior constitutes success for a particular program. But there are a number of different ways that a behavior can be documented. Will self-reports (“I am doing that much more”) suffice? Is a supervisor’s confirmation required? Or is it necessary to have a third party observe and count the behaviors? Similarly, it may be that the sponsor wants to see analysis of the financial impact. How rigorous does that analysis need to be? Will estimates from the participants suffice, or must the analysis come from the finance department? The “correct” answers to these questions are situational; they depend on what is feasible with the resources available and what is acceptable to the target audience. Look before you leap (see Case in Point D6.4). There are five basic kinds of data you can collect: business metrics, observations, estimates, opinions, and stories. Once you are clear about to which category the outcomes you want to measure belong, then the ways in which to collect them follow, since there are a limited number of methods for each (Table D6.3).

TABLE D6.3.

CH006.indd 293

KINDS OF EVIDENCE AND DATA COLLECTION METHODS

Kind of Data

Examples

Collection Methods

Business Metrics

• • • • •

Sales Production figures Quality Index Lost Days Turnover

• Extract from business systems • Add tracking system if not in place already

Observations

• • • • •

Use of proper procedure Telephone etiquette Sales technique Coaching technique Interpersonal skills

• • • • •

Estimates

• Time saved • Number of times used • Financial benefit

• Surveys • Interviews

Opinions

• • • • •

• • • •

Stories

• Success stories • Critical incident • Examples

Quality of service Leadership efficacy Quality of presentation Amount of improvement Value of program

Survey of observers Direct observation (overt or covert) Recordings Demonstrations/Role play Self-assessment

Surveys Interviews Focus groups Learning transfer management systems

• Surveys • Interviews • Learning transfer management systems

6/17/10 3:31:15 PM

294


Business Metrics Business metrics are data that the company routinely collects as

part of its ongoing operations. These include everything from sales (items and dollars) to number of errors or cost of scrap, manufacturing costs, lead times, out of stocks, shrink, forecast accuracy, and so on. Indeed, most companies are awash in more data than is ever fully analyzed and used. If an existing business metric meets the criterion of a relevant outcome for the program—and one that won’t be obscured by other factors—by all means use it. Business metrics have the advantage of being fully credible and relevant to the business. And since they are already being collected, there is no additional cost to gather them. If you plan to use business metrics as an outcome measure, just be sure that you can gain access to them at the level of detail you need (individual participant). You’ll need that degree of specificity in order to make before-after or traineduntrained comparisons. Meet with IT, or finance, or whoever owns the data, and explain what you need, for whom, and covering what period. If confidentiality is a problem, it may be possible to circumvent the issue by encoding the data—that is, by substituting random numbers for names. Participants’ names are not needed for the analysis as long as you know which set of data represent the post-training and which the pre-training or control group. Observations For training programs in which the outcome of interest is a change in

on-the-job behavior and actions, or in which this is an important leading indicator, the most relevant and credible data are direct observations. Depending on the level of rigor required, these can range from self-observations on the low end to performance checklists completed by trained observers on the high end. (See Case in Point D6.7.) A mid-level of rigor will usually suffice, such as asking a customer “Were you greeted when you came into the store?” or “Did the salesperson ask if you had any other questions or concerns?” Other examples would include asking managers, peers, or direct reports about specific behaviors. A variety of formats can be used, including counts, rating scales, or requests for specific examples. “It’s all about changing behaviors and driving business results; that’s the only justification for trainThe real question is not ing,” says CallSource founder Jerry Feldman. “For that whether employees liked the you need to have high-quality, relevant data. The real training, but how it benefited question is not whether employees liked the training— the business. or even whether they learned something from it. You need to show how it benefited the business.” For the data to be valid, evaluators need to have had the opportunity to personally observe the behavior, they need to clearly understand what you are asking, and the timing needs to be such that they can remember accurately.

CH006.indd 294

6/17/10 3:31:16 PM

Document Results

295

Case in Point D6.7 Using Training to Drive Business Results Southern California-based CallSource, Inc. provides an outstanding example of driving results through training and measurement. Using a platform that combines targeted, ondemand training with expert analysis, CallSource is able to help clients substantially increase incremental sales—without any additional investment in marketing or advertising. The key, according to CallSource Founder and 2009 Entrepreneur of the Year Jerry Feldman, isn’t just training—it’s the continuous evaluation and redeployment of that training to achieve specific, measurable results. “Most organizations have some method of tracking employee performance as it relates to sales,” says Feldman. “Some of those organizations even go on to target training based on the data they collect. But far too few actually track the business results of that training, make the necessary adjustments, and then start the cycle over.” An integrated marketing, sales, training, and management company, CallSource is uniquely structured to bridge this gap. The process begins with a comprehensive analysis of employee performance against set business metrics, followed by training targeted at the specific employee behaviors that will improve those metrics. CallSource rigorously evaluates training results by monitoring and evaluating on-the-phone performance and productivity, then uses the data to both prove its effectiveness and to drive continuous improvement. Continuous training maintains the improvement month-to-month while the employee base gets ongoing performance evaluation and rankings. How does this process work in the real world? Take a company that depends on telephone inquiries. “When the average representative handles ten calls,” explains Feldman, “he or she generates four appointments, which ultimately lead to one sale. When we train that representative to move the ratio to five appointments and two sales from every ten calls, we’ve effectively doubled his or her revenues, with no additional marketing expense.” Because the representatives know that their performance will be assessed—and because they see how increased sales are reflected in their commissions—they work diligently to use their training. By evaluating and tracking actual on-the-job performance and outcomes, CallSource is able to show the client compelling evidence of real behavioral change and, simultaneously, how these changes improve yield and productivity. CallSource has created a learning management system that creates a personal scorecard for every trainee, showing his or her scores over time, areas of strength, areas in need of improvement, and even the specific training modules that would help the most. The resulting “performance culture” keeps everyone focused on continuous improvement. According to Sam Klein, general manager, NOARUS Auto Group and a CallSource client, “Our employees now critique each other so they can all become better on the phone.” Using financial data supplied by its customer companies, advisors at CallSource are able to calculate the increased revenue and cost savings generated as a result of training, which are inevitably many times the cost of the system. The advisors then work with company leaders to target and forecast the next level of profitability.

CH006.indd 295

6/17/10 3:31:16 PM

296


Observational data is usually obtained by survey, but it could also be done by interview or by providing observers with checklists to be completed and returned. Estimates Estimates are widely used in business for planning purposes: “How long

do you think it will take?” “How many do you think you will need?” “What is your sales estimate for next quarter?” Estimates are sometimes co-opted for use in evaluation, such as: “How many times did you use something you learned from the program?” “How much do you think it was worth?” Because they are estimates, they necessarily are less credible than business metrics or direct observations. To have any validity, the respondents who provide the data need to have sufficient knowledge and expertise to provide a reasonable estimate. A claims processor can probably reliably estimate how long it takes her to process a claim. On the other hand, her estimate of how much money the company will save as a result of a training program in claims processing is not likely to have much credibility to Finance, since she has no knowledge of the true costs involved or the principles of activity-based costing. If you decide to use estimates as one of the outcome measures in your evaluation, be sure your client agrees (preferably in D1). Then design a questionnaire or interview guide that asks people to provide estimates that are appropriate to their purview and expertise. Opinions Opinions, or perceptions, would seem to have the lowest credibility

of all. Yet there are times when opinions are both the most credible and most critical outcomes to measure. Remember that people buy products based on their perception of value. Hence customers’ opinions are leading indicators of their willingness Sometimes opinions are the to use your services again or commend them to most important and most others. You can use surveys or interviews to colcredible outcomes. lect opinion data, typically through some sort of rating scale. For example, you could ask people to rate, on a score of 1 to 10, how likely they are to recommend the product or service to a colleague or friend. Indeed, the answers to that question, and the Net Promoter Score derived from it, have been proposed as the single most important predictor of future growth (Reichheld, 2003). Many leading companies, for example, General Electric, have begun tracking the Net Promoter Score as a key performance metric (General Electric, 2008). In such companies, therefore, demonstrating that a learning program helped increase the net promoter score for a service or function would be strong evidence of success.

CH006.indd 296

6/17/10 3:31:17 PM

Document Results

297

Similarly, people’s decisions to stay or leave a company are strongly influenced by their opinions of the person to whom they report. Given that, changes in the opinions of direct reports before and after their manager attends a development program would be evidence of a business-relevant outcome, since improvement in their perception of their boss is strongly correlated with retention. Or, as the saying goes: “People don’t leave the company; they leave their boss.” David Campbell of the Center for Creative Leadership puts it this way: “The opinions of those being led is relevant to the evaluation of the leader.” For that reason, many companies use improvement in 360-degree assessments or employee engagement surveys as a measure of the effectiveness of their leadership and management development programs. Alternatively, you could simply ask the direct reports of a sample of attendees and a group of control managers to rate their managers’ effectiveness now compared to, say, three months ago. Opinions are relevant outcomes when they are good predictors of future action, as in the above samples. But not all opinions are. The problem with endof-course evaluations—which are a form of opinion poll—is that they are poor predictors of learning transfer and use. Because reaction scores are easy and inexpensive to collect in large numbers and because they can be computed to several (not necessarily significant) digits, they impute a predictive value far beyond what research has shown (see Case in Point D6.8). Opinion data can also be collected through a structured interview, but nowadays they are usually collected by an online survey for convenience and cost savings. Almost anyone can easily put together, administer, and tabulate an online survey using one of any number of inexpensive commercial services. But there is a difference between being able to construct a survey and being able to construct a good and valid survey. Many that we receive violate the most basic principles of survey design. There is a well-established science and discipline to creating surveys that generate valid and reliable data. Poorly designed surveys can lead to erroneous conclusions. There is a difference between Obtain professional advice if you are not being able to construct a an expert in survey design yourself, or survey and being able to consult one of the many specific texts on construct a valid survey. the subject, such as Babbie (2010). Expert Opinion A special subcategory of opinion measures is expert opinion— assessment of an outcome by someone “skilled in the art.” Expert opinion can be an appropriate measure for training programs that seek to improve the work product, for example, courses on presentation skills, advanced programming techniques for software engineers, strategic planning simulations, and so forth.

CH006.indd 297

6/17/10 3:31:17 PM

298


Case in Point D6.8 Predictive Value of Reaction Data End-of-course reactions are the most widely and commonly collected opinion data. The implication is that a course with higher scores is a “better” course than a course with lower scores. By extension, a trainer, training department, or training provider that consistently achieves scores higher than the norm is assumed to be more effective. That relationship holds true in the entertainment or hospitality business: Movies that viewers like better are more valuable to the business because they earn higher box office receipts. Unfortunately, that relationship does not hold true when training is viewed from a business perspective. Participants’ immediate reactions to a program do not predict whether they will use what they have learned or whether it will produce any value for the business. As it turns out, positive end-of-course evaluations say little about how much actual learning took place, whether the learning was relevant to the organizational goals, and whether it will be usefully applied on the job. Several studies have investigated the correlation. Dixon (1990), for example, found no significant correlation between post-course test scores and participants’ perception of the program’s relevance, their estimation of amount learned, enjoyment, or instructor’s skill. Alliger and colleagues (1997) analyzed thirty-four previous studies and found very poor correlation between reaction data, objective measures of learning, and on-the-job learning transfer. More recently, Ruona and colleagues (2002) studied the relationship between learner reactions and learning transfer by using the Learner Transfer Systems Inventory. They concluded that “this study supports the position that reaction measures have limited use in evaluating the outcomes of learning and development.” That should not be surprising, given the numerous other factors that impact the learning transfer and value creation process (Holton, 1996). Opinions can be useful measures of training impact, provided they are proxies for future behavior that benefits the company. Unfortunately, Level 1 reaction data fail in this regard.

The expected outcomes for these three examples would be better presentations, higher quality code, and more robust strategic plans, respectively. But only people skilled in the art of presentations, software, and strategic planning would be in a position to judge whether the program had actually improved the quality of the output and therefore had delivered the desired results. Experts don’t necessarily have to be outside consultants—although they could be. The main requirement is that the “judges” have sufficient experience and expertise to render a credible opinion of the relative quality before and after training. When the desired outcome is an improved work product, then the measurement strategy is to find a person (or an objective rating system) who can render an expert opinion on the quality of the output.

CH006.indd 298

6/17/10 3:31:17 PM

Document Results

299

An example would be the “Checkpoints to Mastery” system that Securian Financial Group developed in cooperation with Frank Sarr of Training Implementation Services to assess the performance of newly hired advisors. After ten weeks of training and on-the-job performance support, new advisors have to demonstrate their mastery not only of the subject matter but also of consulting skills in a live interaction with a team of experienced managers who assess their performance against a set of defined performance criteria. Stories A fifth category that, to some extent, cuts across the others is stories—

case examples in narrative form about what participants accomplished following participation in a learning or developmental opportunity. There is growing recognition of the power of stories to illuminate, educate, motivate, and create lasting impressions in business (Denning, 2005). Stories are at the heart of the Success Case Evaluation Method developed by Rob Brinkerhoff. The process is straightforward. At an appropriate time after training (depending on its nature and objectives), the participants are polled. They are asked whether they have used specific aspects of the program and to rate their success from “none” to “clear success with measurable outcomes” (Brinkerhoff, 2003, p. 102). A sample of those who claim they have had real success is interviewed to gather the details, confirm the claims, and, if appropriate, document or estimate the financial impact. A sample of those who reported no success is also interviewed to understand what prevented them from enjoying achievements similar to their classmates. The resulting success stories are then used to demonstrate “beyond a reasonable doubt” that the training plus the right transfer support can and does produce valuable business outcomes. Because the success cases are in story form and contain specific details, they are both memorable and compelling. Comparing the value created by successful participants to the number who reported no success helps management appreciate the magnitude of the learning scrap—the amount of potential value being “left on the table” when learning transfer fails. The survey results and interviews from those who were not successful in applying their training provide valuable insights into the impediments they encountered. The vast majority of these typically concern the transfer climate and are issues within management’s control—such as a lack of a supervisor’s support or misalignment of incentives. Importantly, this approach allows you to present the obstacles in the participant’s own words, which is a compelling way to get management’s attention. Together with ability to illustrate the magnitude of the potential value that went unrealized, the Success Case Evaluation Method helps motivate senior managers to take the actions necessary to improve support for learning transfer.

CH006.indd 299

6/17/10 3:31:18 PM

300


A critical caveat, however, about this approach: You must always independently confirm the cases that you plan to use as exemplars. If the participant reports that she used the training to land an important new account, check the sales records or contact the client. If another claims he saved a key employee who was about Always independently confirm to quit, check with the employee. Why? Because the cases that you plan to use nothing will destroy your credibility more quickly as exemplars. than presenting an example of success that later turns out to be untrue or greatly exaggerated. We have automated aspects of the success case approach using the ResultsEngine learning transfer management system. At the end of the learning transfer period, participants are asked to what extent they have achieved results and to briefly describe them. In every program, we have been able to identify rich and illustrative stories of the kind of impact the right learning program and environment can have. (See Case in Point D6.9.) Planning the Analysis The final check on the design is to anticipate the analysis, that is, to think through how the data will be collated, encoded if necessary, “crunched,” and summarized, as well as what statistics (if any) will be used. An important consideration is to be sure that you will have enough data points to draw valid conclusions. In general, the more subjects that can be included in the study, the better. There are two reasons: (1) the larger the size of the groups studied, the more likely they are to be truly comparable and (2) the larger the group size, the easier it is to confidently detect differences that can be attributed to the training. The smaller the number of participants studied, the greater the probability of making both type I and type II errors—that is, finding an apparent effect when there is none (type I) or failing to demonstrate a benefit when there really is one (type II). Even when the difference between two groups seems large, it may still not be “significant” statistically because, if the groups are small, the difference is just as likely to be due to chance as to some actual effect of the training. There are statistical methods to determine the number of subjects needed to detect a significant difference of a particular size. Consult with your company’s statisticians or external consultants in advance to determine whether the design of the evaluation is likely to have enough power to detect a difference. Rigor How much data you need and how rigorous the analysis must be depend

on what it will take for the conclusions to be considered credible and actionable. The level of rigor that is required to have a paper published in, say, Science or the Harvard Business Review is very high and well-known. The rigor typically required to adequately evaluate a training program is less, but how much less depends

CH006.indd 300

6/17/10 3:31:18 PM

Document Results

301

Case in Point D6.9 Success Stories A leading manufacturing company conducts an intensive two-day program to introduce managers to its unique approach to management. To ensure that the program leads to new behaviors and results, participants are expected to report progress on their implementation every two weeks for ten weeks in an online learning transfer management system (ResultsEngine). On their final update, they are asked to provide examples of what they have accomplished. Below are a few of the many examples of success stories that demonstrate the kind of impact that the learning and transfer is producing. “By applying the principles we learned, we were able to put together a first-class meeting with a new distributor that opened up the potential of $1 million in new business in this account. We achieved this by enrolling all of our decision-making people, so everyone has a stake in the success.” “During a shop steward/shift leader safety commitment meeting, an operator was so focused on monetary reward as the only incentive that he was blind to alternatives. When I provided other types of incentives and asked him how he felt about nonfinancial rewards, he had an entirely different point of view and actually said that receiving a ’thank you’ is extremely rewarding to him. Many other people indicated that they too were getting a better appreciation of how important non-monetary recognition can be. It was one of the most rewarding meetings from a learning perspective for me personally. We have a long way to go, but we now have the right tools to get there.” “The time it usually takes to create these equipment strategies can be as much as ten weeks. The template process discovered by my team members will easily cut this in half. I don’t have dollar values to apply to this at this time, but it is apparent that 50 percent time reduction in the application of this process is a long-term value opportunity that will generate significant cost savings.” “I would say the impact is significant across my entire sales team. Best practices have been shared, and this will continue. The new review process and improved communication will lead to increased revenue. I estimate the impact in my area will be about $2 million in revenue for the next twelve months. This is approximately $100K per territory, which equates to about two good sales per sales professional. This would generate $1.25 million in incremental margin.” “One of my direct reports has begun taking on more responsibilities. I believe this to be a direct result of the feedback sessions I had with him, along with how I have changed the way I share knowledge. Since I began focusing on communicating the intent of our vision as well as the principles behind our goals, he has stepped up to the plate and has taken a more active role in the department.”

CH006.indd 301

6/17/10 3:31:19 PM

302


on a variety of factors. These include the culture of the company (some are far more analytical than others); the temperament of the decision makers (some are detail-oriented; others are big-picture); the magnitude and strategic importance of the initiative (the bigger and more important, the more rigor is required); and the politics surrounding the program. Politics may seem like an odd concept to introduce into a discussion of documenting the results of training programs. But evaluation is inherently political or, as one wag put it (Patton, 1997, p. 352): “Evaluation is NOT political only when the following conditions apply: • • • •

No one cares about the program. No one knows about the program. No money is at stake. No power or authority is at stake.”

In other words, all evaluation is political, since there is always money, power, reputation, and authority at stake. The bigger these issues are in the organization, the more political the results will be, and correspondingly, the All evaluation is political. more rigorous the data collection, analysis, and reporting must be. Finalize the Plan The final step in the design is to create a project timeline and plan that includes all the key activities (when surveys and reminders will be sent, followup interviews, and so on). If you are new to evaluation, or are using an approach for the first time, it would be smart to have an expert review your plan. There may be ways to streamline the process or strengthen the analysis. Anticipating the analysis and how the results will eventually be used is important; there is no point in collecting the data if you do not have a plan for what you are going to do with it.

3. Collect and Analyze the Data Once the design has been agreed to and reviewed, it is time to execute. Someone needs to be assigned the responsibility to manage the project and execute the plan, because even a relatively simple design—for example, gathering opinions from internal customers—has a fair number of moving parts. If no dedicated resource for project management is available, you’ll have to manage the project yourself or contract it out. In either case, don’t underestimate the importance of execution; poor execution is the most common cause of failure in business (Bossidy & Charan, 2002, p. 5).

CH006.indd 302

6/17/10 3:31:19 PM

Document Results

303

Gather the Data If you are using a newly developed questionnaire or data

collection system, it is a good idea to pilot it first to be sure the questions are understandable and the answers are being correctly recorded. If you are using interviews, then you must take precautions to ensure consistency and impartiality, which are discussed in texts on qualitative research. Other than that, gathering the data is mainly a matter of periodically monitoring the incoming information to be sure that response rates are adequate and that the systems are working. Of course, few investigators can resist performing some initial analyses to see if trends are developing and to get an “early read” on the outcome. Analyze the Data Once the data have been collected, the most exciting phase

begins: analyzing the data to see whether the program really delivered on its promise. The essential process is comparing one set of results (for example, posttraining) to another (for example, pre-training or untrained). Online evaluation systems, such as Metrics That Matter, include cross-tabulation capabilities to assist with such comparisons. If the results are qualitative, then it is important to be balanced in the definition of themes and selection of examples. If the data are quantitative (counts, rating scales, dollars, and so forth), then the analysis needs to include some degree of statistical analysis to confirm that the results are not just random variation. The mention of statistics brings to mind the phrase popularized by Mark Twain: “There are lies, damned lies, and statistics.” Many people in business are not comfortable with statistical analysis and are not persuaded by statistical arguYou have an obligation to ments. Nevertheless, you need to know report the results—good, bad, how much reliance you can place on any or indifferent. differences observed and whether they reach the level of “statistical significance.” That’s important to how you position the results, whether or not you choose to include the statistical analysis in your final report (see “Selling the Sizzle” later in the chapter). 4. Report the Findings to Management Once you have the results—good, bad, or indifferent—you have an obligation to report them and to do so in a manner that leads to informed decision making. A thorough evaluation that rigorously documents superb results is of no value if no one knows about it. In our view, there are two different but important uses for the results. The first is to make a formal report to management; the second is in marketing

CH006.indd 303

6/17/10 3:31:19 PM

304


communications to all the key stakeholders (what we refer to as “selling the sizzle,” covered in the next section). Having consumed time, money, and attention to deliver training and development programs, learning organizations have a responsibility to report back to the sponsors what they received in return. As investors, sponsors are looking for a critical analysis of the value that was generated. They want to see relevant, credible, and compelling evidence that their investment is paying off in order to justify a decision to continue to invest in the initiative. And, no matter how good the results may be, they will also want to know how you plan to do even better in the future. You never get to rest on your laurels for long in business. Key Aspects of the Communications The primary (but not exclusive) commu-

nication to leadership will be some sort of formal report. It should be a concise, fact-filled, no-nonsense analysis written in the language of business. To maximize “understandability”—and therefore credibility and impact— use terms and concepts familiar to business leaders and eschew learning jargon. For example, most learning professionals are familiar with Kirkpatrick’s model and will know what you mean when you say “we conducted a Level 3 analysis.” Conversely, few business leaders know (or care) about Kirkpatrick’s taxonomy. As Sullivan (2005) puts it: “History has proven that managers will not learn your language or shift to your focus, so it is you who must adapt” (p. 283). In other words, use plain or business language, not educational jargon. Say “We implemented a computer-based examination to see whether the participants had learned the key concepts,” rather than “We conducted a Level 2 analysis.” It is much more important to be understood than to flaunt erudition. A similar caveat applies to the use of statistics. Most analyses will require some level of statistical analysis to show that the results are due to more than random chance. However, knowledge (or abhorrence) of statistics varies widely among business leaders. Don’t try to dazzle them with complex analyses unless they are absolutely essenNever present anything tial (see the section on “rigor”). If the nature of that you cannot confidently the analysis demands unusual or arcane methods, explain. succinctly explain them and the rationale for their use. Important: Be sure you understand them; never present anything that you cannot confidently explain—someone in the audience may be a black belt. Nothing undermines your credibility like being unable to explain something that you, yourself, are presenting. Finally, get to the point quickly. Business leaders tend to be “drivers”; they have neither the time nor the patience to wade through pages of text or reams of slide printouts trying to find the actionable information. Executives and senior leaders,

CH006.indd 304

6/17/10 3:31:20 PM

Document Results

305

in particular, are extremely busy; they have no tolerance for long preambles, convoluted messages, educational claptrap, and poorly organized materials. If they cannot get to the meat of the matter in the first paragraph or two, they are likely to set the report aside or discard it. Always make the first page an executive summary. This is crucial; it is all that many senior managers will read. State the key findings and recommendations clearly, concisely, and unambiguously in the executive summary. Write it in straightforward declarative sentences and keep it to one page or less. Summarize the business need that was addressed, the program’s objectives, what was done, how it was evaluated, the results, and the recommendations for the future. You are not writing a novel; don’t keep the reader in suspense. The executive summary is the most difficult and time-consuming part of the report to write. As the great French mathematician and philosopher Blaise Pascal wrote: “I have made this letter longer than usual only because I have not the time to make it shorter.” Write the executive summary last. The challenge is to make sure that it contains all the key facts and recommendations and is still brief and to the point. The rest of the report should provide the details that support the executive summary, including the design of the complete learning experience, the evaluation methodology, data tables, success stories, and analyses necessary to support the conclusions. Business managers deal in numbers; present quantitative information in well-constructed, clear tables and graphs. Be sure the report reflects the peculiar culture of your organization and management team; for some, too much data is a “turn off ”; for others, too little data undermines credibility. Include high-level summaries, tables, and graphs in the body of the report; relegate detailed analyses, tables, and so forth to the appendices. There is a trend in many organizations to use printouts of slides (PowerPoint and others) in lieu of written reports. We agree with Edward Tufte, professor emeritus of political science, computer science, and statistics, and senior critic, graphic design at Yale, that this is a mistake. Because of the limitations that slides impose, “the PowerPoint style routinely disrupts, dominates, and trivializes content. Thus PowerPoint presentations too often resemble a school play—very loud, very slow, and very simple” (Tufte, 2003). If your corporate culture requires slides as the primary communication vehicle, be sure to also distribute a one-page executive summary and, preferably, a well-constructed, full report. Logic and Organization A great evaluation can be ruined by a poorly constructed report. Make sure the report is organized in a way that makes it easy to follow so that the reader comes to the same conclusions that you have. The conduct

CH006.indd 305

6/17/10 3:31:20 PM

306


of the study needs to be logically consistent with the question being posed. The conclusions need to follow logically from the data and claim no more than what the data will support. If the report has been prepared by or with the help of a consultant, be sure to vet it so that it matches the communication preferences of your intended audience. Acknowledge the contribution of managers and others outside the learning organization and be forthright about any limitations of the evaluation or conclusions. Explain your “lessons learned” and plans to make subsequent programs even more effective. Although a formal report to senior leadership is necessary, it is insufficient. A common and costly error is to equate distributing a report with communicating results. To make sure that your message is heard among all the competing noise, it must be reinforced. If possible, ask to give a short presentation of the results in person. We emphasize short, given the time constraints senior managers face. Get to the point quickly; use only as much time and as many slides as absolutely necessary; deliver the message succinctly, and finish in less than the allotted time. Try to anticipate the most likely questions or objections and be prepared to answer them (with back-up slides if necessary). If there are no questions, resist the urge to keep talking; heed the advice given to salespeople and “don’t buy it back.” 5. Sell the Sizzle Evaluation results are vital to building learning’s brand. The concept of a “brand” for learning may seem odd, but a recent whitepaper from Corporate University Xchange put it well: Everything has a brand, and the learning organization is no exception. This is because brand is made up of opinions, and it is impossible to prevent people from having opinions. If a learning leader stopped employees, managers, or senior executives in the hallway and asked them their opinion of the learning organization, there is no doubt that they would have one” (Dresner & Lehman, 2009).

Because the perception that people have of learning’s “brand” shapes their willingness to commit money, time, and effort to learning programs, it is important to know what learning’s brand is in your organization and to actively manage it. Documenting the results of training’s impact is one of the best ways to build and sustain the learning organization’s reputation . . . but only if you market them

Everything has a brand, and the learning organization is no exception.

CH006.indd 306

6/17/10 3:31:21 PM

Document Results

307

well. You have to sell the sizzle, or as Kevin Wilde, chief learning officer at General Mills, puts it: “You have to market what you are doing, demonstrate value, and know what matters” (Wilde, personal communication, 2004). If you have great results, you have to actively promote them; “you can’t be subtle or naïve if you want managers to pay attention to your metrics” (Sullivan, 2005, p. 282). The M Word Professionals, in general, have been acculturated to think of mar-

keting as a slightly dishonorable or tawdry endeavor, a prostitution of values and ethics that is beneath them. They believe that the value of their work should speak for itself. That is a very naïve and dangerous frame of mind because, as Sue Todd of Corporate University Xchange put it: “Your learning organization has a brand, whether you choose to manage it or not” (Todd, 2009). Eschewing marketing means letting your reputation drift. Making no effort to communicate the value that learning has delivered could lead to a learning brand that is perceived as having no value. “A brand is a promise to a customer,” says Chris Quinn, president of Imprint Learning Solutions. “A brand is something that exists in the mind of the customer that summarizes a product’s attributes, benefits, and value. Brands matter. And because brands matter, they have to be managed. Strategic marketing is an integrated set of targeted activities and communications that positively influence the perception of value” (Quinn, 2009). The key is the “perception of value,” since people make decisions based on their perception of value, not necessarily the true value, which they may not know or fully understand. Therefore, it is in the learning organization’s—as well as the company’s—best interests to be sure that the business leadership correctly perceives training’s value. Our experience suggests that learning organizations will benefit greatly by becoming more adept, and investing more energy, in commuA brand is a promise to a nicating their value to key audiences— customer. in other words, by getting better at marketing. Segment and Target The first step in strategic marketing is to segment and target customers. Segmentation means dividing them into groups with similar wants and needs. Targeting means focusing communication efforts on the most important segments. With respect to reporting results, ask yourself:

• Who are learning and development’s most important audiences? • Which messages need to be communicated to whom?

CH006.indd 307

6/17/10 3:31:21 PM

308


Clearly, those who make the decisions to fund corporate education constitute a critical customer segment. As discussed above, the message that has to be delivered to this group is that the time and money for training was well spent, that it yielded a good return in terms of business-relevant outcomes. Learning’s brand promise is that future investment can be expected to yield similar results and that the learning and development organization is serious about continually improving its output. Some savvy CLOs specifically pursue awards as a brand-building strategy. If they are successful, they make sure that the success is widely communicated, reasoning that continuing support is more likely for an “award-winning” department. A recent poll of CEOs (Philips & Phillips, 2009) revealed that they give credence to such awards as external validation of worth. Although many awards still depend too heavily on input or activity measures, evaluation is becoming an increasingly important criterion. The ISPI submission form for an “outstanding human resources intervention award,” for example, contains numerous questions about how the program was evaluated, because: “Evaluation is about measuring the efficiency and effectiveness of what you did, how you did it, and the degree to which the solution produced the desired results so you can compare the cost incurred to the benefits gained” (ISPI, 2010). A second key customer segment for communications is those who participated in the evaluation. They need to know that their efforts were appreciated, that the information is being used constructively, and that they made an important contribution to the success of the company. A third target is employees who have not yet participated in the program. They are potential Tailor the messages to each consumers; communication to this group should audience’s specific interests, make them want to attend and to do so with needs, and preferences. positive expectations. Other targets for communication may include shareholders to illustrate how the company invests in building future capabilities and the corporate education community to attract outstanding professionals to your team and foster interchange of best practices. Once you have identified the key audience segments and their proclivities, you need to tailor the messages and select communication vehicles that are most effective for each. Repetition, Repetition, Repetition Advertisers understand that no matter how compelling the message, it takes repetition to build a brand and create “share of mind.” Companies that are good at marketing repeat the same core message over and over in a variety of media (magazines, television, direct mail, and so forth).

CH006.indd 308

6/17/10 3:31:21 PM

Document Results

309

Effective CEOs use a similar approach. They emphasize a few core themes over and over until everyone in the company gets the message. Although each individual communication is tailored to the medium and audience, they all underscore the same few core messages and themes. Learning and development organizations need to do likewise. To be sure that “the message gets through,” communicate the results of learning and development initiatives multiple times, in different settings and formats. Tailor each to the target audience, but always emphasize the few core brand attributes you want to “stick.” Suggestions for doing so follow. Participants and Their Managers The participants in the program are naturally

interested in the results of their efforts. They will be curious about the outcome of any evaluation in which they participated, as well as what other members of their cohort achieved. Those who made a concerted effort to use what they learned and, as a result, achieved something of significance crave recognition. Help ensure they receive it. Whenever you recognize participants’ achievements, do not forget the critical role played by their managers in the transfer and application of learning. Be certain to acknowledge managers’ contributions to the success of the initiative and encourage senior managers to do likewise. It’s not just about the training department. An interesting paradox is that the more of the credit you share, the more you will receive. Key Aspects of the Communications The first objective is to thank those who participated in the evaluation (either as subjects or as controls) for their time. Time is perhaps the most precious asset for any employee these days; whatever time participants spent assisting in the evaluation should be regarded as a gift. If you want people to participate in future evaluations, you need to thank them for their participation and provide a brief synopsis of the findings. In particular, acknowledge their suggestions for improvement and, if possible, describe the changes that will be made in future programs as a result of their input. The second objective of communications to participants is to reinforce the value of the program’s principles by highlighting success stories. One way to accomplish this goal is through “human interest” stories in company publications that report notable accomplishments of individuals or teams. “The heart of the documentation that I think is really critical is stories, not just metrics or numbers,” says Richard Leider, founder and chairman of the Inventure Group. Describing a particularly successful learning organization, he explained: “They do a lot of work to build leadership stories and communicate them in all kinds of formats so the story and the language are out there everywhere” (Leider, personal communication, 2004).

CH006.indd 309

6/17/10 3:31:22 PM

310


Similarly, Jim O’Hern, when he was director of leadership development at Honeywell, used success stories to recognize participants who completed their follow through and also to energize new program entrants. He compiled examples of successful application from the learning transfer management system (ResultsEngine), including positive feedback from coaches and managers, and shared them with participants in subsequent programs. In this way, he simultaneously recognized superior performance of prior participants, illustrated what is possible when program principles are applied, and motivated current participants by indicating that they could be selected as avatars for subsequent sessions (O’Hern, personal communication, 2005). Communication Vehicles If participants have access to email, an electronic “thank you” letter and brief synopsis of the results may suffice to meet the first objective. Given the email overload most managers now experience, a paper letter and oneor two-page summary may actually receive greater notice. If the program design includes reconvening the group, use it as an occasion to publicly acknowledge examples of superior performance. Take advantage of as many communication vehicles as possible, such as newsletters, internal publications, and the company website. The editors of such communications are always in need of stories that are both interesting and illustrative of company values and goals. Qualcomm’s Learning Center, for example, actively feeds sucTake advantage of as many cess stories to their corporate communications communication vehicles as department. If you use a computer-based learnpossible. ing transfer management system or the successcase approach, they are excellent ways to identify individuals and teams who have achieved significant results. Bring them to the attention of the communications department, which can then develop a complete story through follow-up interviews. Having such success stories reported in company publications enhances the perceived value of learning and contributes to building learning’s brand. Employees who are featured in such articles will be motivated to even greater accomplishments; future participants will strive to be among those so recognized. Moreover, having others talk about the value of learning is more valuable than messages from the learning organization itself; or, as the saying goes: “A dollar’s worth of public relations is worth ten dollars of advertising.” Get others to tell your story whenever you can. Non-Participants Employees who have not attended the program are a third key

target for communication. The objectives are to create active interest in attending,

CH006.indd 310

6/17/10 3:31:22 PM

Document Results

311

show how the company is investing in its human capital, motivate others by recognizing superior performance, and encourage managers to support employee development. Key Aspects of the Communications Articles that describe the program and illustrate the value participants have derived support these objectives. Human interest stories like those discussed previously will be the most compelling, especially if they include quotes, photos, and examples from individual participants, their managers, and teams. Consider awards or other recognition to motivate participation and learning transfer, but be careful to validate claims of success. Public recognition of accomplishments that later prove to be exaggerated or unsubstantiated can discredit the whole program.

People want recognition for Communication Vehicles The main vehicles work well done. are the internal employee publications, company blogs and social networking sites, electronic communications, and team and company meetings. Do not underestimate the power of public recognition; it is a very powerful motivator. “Over and over, research has told us that money is not the major key to keeping good people. When employees across the country answered the question, ‘What kept you?’ few had dollars in their top three reasons. People want recognition for work well done” (Kaye & Jordan-Evans, 2008, p. 182). Other Stakeholders Promoting corporate education’s success in building new

competencies and competitiveness contributes positively to a company’s reputation. Four external audiences to consider are customers, prospective employees, shareholders, and the corporate education community. Key Aspects of the Communications A company’s overall reputation is one of the factors that customers consider when they make a buying decision. They are more likely to buy, and to pay a premium, from a company that they perceive as high-quality and progressive with good prospects for the future. Communicating examples of how the company is improving its quality and performance through training and development helps reinforce a positive perception. Similarly, a company’s reputation as an employer affects its ability to recruit employees. Companies that are perceived as willing to invest in their employees have an easier time attracting good candidates and a lower cost of talent acquisition. Younger employees in particular—the so-called Millennials—want to work for companies that are willing to invest in their development (Phillips & Torres, 2008). News stories that illustrate how the company invests in employee development

CH006.indd 311

6/17/10 3:31:22 PM

312


will enhance its reputation as an employer. If you have a great story to tell, work with the public relations department to get it the placement it deserves. Shareholders are most concerned about a company ’s future prospects. Conveying tangible evidence that the company is building capabilities in its workforce helps bolster shareholder confidence in the company’s continued prosperity. Include concrete examples and financial analyses; savvy investors have become inured to “spin” without substance. Learning awards received by the company may carry some merit with this audience as well. Finally, it is important for corporate learning professionals to communicate their successes (and failures) to their peers. Not only does doing so increase the learning unit’s reputation—helping it to attract the best and brightest practitioners in the field—but it will also help create a network for the open exchange of ideas and best practices that transcends corporate boundaries. Such networks have real economic benefit and substantial ROI in their own right (Dulworth & Forcillo, 2005). Communication Vehicles Success stories about how learning programs are adding

value are great fodder for communications to customers (newsletters, websites, and other publications). They may also be of interest to trade publications that are read by customers and prospective employees. Trade publications have the added benefit of being “independent,” in contrast to house organs. Work with your communications and public relations department to secure placement. If you have evidence that training has contributed to key business outcomes, suggest that it be included in the company’s annual report. Competition for space there is intense, but if the program truly enhanced shareholder value, it may merit mention. For example, Honeywell’s chairman, David Cote, chose to recognize the company’s strategic marketing initiative in the company’s annual report to shareholders: “The focus on functional excellence and introduction of the Strategic Marketing Course is starting to pay off. While the benefits may seem obvious, few industrial companies are truly effective here” (Honeywell, 2004, p. 3). To have any chance of such recognition, however, you must have well-documented results relevant to the business and its shareholders. If the program were truly innovative and delivered exceptional results, the story might even be picked up by one of the leading business publications, such as The Wall Street Journal. The public relations impact would be huge not only for the learning organization but for the company as a whole. To even aspire to such recognition, however, requires something that is truly newsworthy with welldocumented business impact; participant reaction data will not suffice. Venues for sharing the results with others in the learning community include conferences, learning and development periodicals, workshops, and networks. Apply to present your findings at one of the growing number of conferences for learning professionals.

CH006.indd 312

6/17/10 3:31:23 PM

Document Results

313

Summing Up Selling We feel strongly that learning and development organiza-

tions are often undervalued because, historically, they have not done enough to measure their impact and then market the results. As a consequence, there is a tremendous opportunity to enhance learning’s brand and increase the “share of mind” about the value that well-designed and well-executed programs create. We strongly encourage you to use the ideas in this chapter to “sell the sizzle”—position and communicate the value of learning and development, and create a positive brand for learning. 6. Implement Improvements The last—and critical—step of the process is to implement improvements. No matter how spectacular the results were, no matter how compelling the success stories, or how efficient the learning transfer, there is always room for improvement. The continual search for, and action on, improvement opportunities— kaizen—is what differentiates truly outstanding organizations from the merely good, in much the same way that outstanding individual performance is the product of deliberate practice and constructive feedback. Most of us prefer to read positive comments and revel in high-scoring items. It is tempting to skip over, dismiss, or rationalize negative responses. But the real opportunities for growth are in the lower-scoring areas and the critical comments. Recall that to develop true expertise requires seeking out constructive feedback (Ericsson, Prietula & Cokely, 2007). Every program should be reviewed periodically with a view to developing a specific plan of action for improvement. For new programs, the first review should be as soon as data are available to find out what is working and what needs to be fixed. Ongoing programs should be reviewed quarterly or semi-annually, depending on their frequency, and even one-off programs (specific initiatives that will not be repeated) should be reviewed to inform future programs. A leading biotechnology company, for example, holds “lessons learned” sessions after each major learning initiative that include marketing and sales managers as well as learning professionals, with the goal of capturing the lessons of the experience to improve future cycles. Approach The following approach is useful in developing a continuous improve-

ment action plan. First, review the data looking for low scoring areas, negative comments, and suggestions for improvement and then decide: 1. Is there a trend, or are these isolated individual comments? You will never be able to meet the needs or preferences of every individual learner. Concentrate your efforts on those issues that were scored low by a significant number of participants or that were mentioned repeatedly.

CH006.indd 313

6/17/10 3:31:24 PM

314


2. How big is the payoff ? Not everything that can be improved, should be improved. Look for the areas that have the greatest potential to improve the business impact, not just learner satisfaction. Be sure to look beyond the traditional boundaries of training and development. For example, if manager engagement scores are low, improvement there is likely to produce far greater value than marginal improvements in instructional methodology or materials. 3. What is the root cause? Having identified the areas with the greatest potential for payback, look beyond the superficial. If some participants feel that what they learned was of low utility, what’s behind that? Is it really the content, or was it the selection process? Were these the wrong participants, or perhaps the right participants but at the wrong time? Be sure you know what the real problem is before you start devising solutions. 4. What is required to fix it? Once you know which areas, if improved, would add the greatest value, it is time to develop an action plan to address them. Don’t feel you have to “reinvent the wheel.” Read the literature and talk to other learning professionals who have tackled similar issues. 5. Which shall we tackle first? Finally, pick one or two areas to tackle first, weighing the benefit of improvement against the cost and time required to solve the issue. Look for what is known as “low-hanging fruit” in the process improvement literature, that is, opportunities with significant payback that are relatively easy or inexpensive to fix. 6. Decide how you will know, and repeat. As part of the improvement plan, decide how you will know whether you have, indeed, improved the situation—that is, what you will measure and when. Repeat the cycle endlessly to build, stepby-step, a competitive advantage for your learning organization and for your company as a whole. Go Public In the same way that individuals are more likely to achieve their goals

if they share them with others, the learning organization is more likely to achieve its goals if it publicly announces the areas it intends to improve. Presenting a balanced appraisal (we did this well; here is where we can improve) and a specific plan for improvement will enhance your credibility and standing in the organization.

A Caveat Much of the discussion of measurement in training and development seems to have a subtext to the effect that: “If only we could reliably measure outcomes, then all our problems would be solved.” That is a faint hope. One assumes that the evaluation of a well-designed initiative will be positive, of course. But there is

CH006.indd 314

6/17/10 3:31:24 PM

Document Results

315

always the possibility that the evaluation will show that the program produced no demonstrable benefit or that the benefit was too small to justify its cost, especially if learning transfer is poor. The time to decide what to do with negative findings is before the evaluation is begun. Even before putting the plan into action, answer the question: “Suppose the evaluation is not favorable, what will we do?” Once the data have been collected, they cannot be “buried” or ignored. You have both a moral and management responsibility to report negative as well as positive findings and to make a recommendation consistent with the data—to fix the program or to kill it. Negative results are still valuable, and the possibility that the outcome won’t be favorable ought not to deter learning organizations from asking the hard questions. As Kevin Wilde, CLO of General Mills, explains: “Some studies did not pan out, some did. But unless I am asking, I do not know exactly where the value is. By doing this kind of work, I am interacting with the CEO in a very business-like way that he expects out of all the other business leaders— getting results, producing insights. Some things work out, some things do not. You have to have the courage to ask and figure it out” (Wilde, personal communication, 2004). Our point is that documenting results should not be undertaken lightly, nor should you assume that the results will always support your claims. Evaluation is a sharp tool that cuts both ways. When we were asked to speak about measurement Used carelessly, evaluation can at the Center for Naval Leadership, we do a lot of damage. recommended that evaluation be handled like high explosives—very carefully. Used properly, evaluation will give a tremendous boost to the credibility and respect of the learning organization. Used carelessly, or directed at the wrong outcomes, it can explode in your face and do lot of damage. Likewise, it may be a mistake to evaluate a program until the other five disciplines (clear business objectives, a complete experience, application-focused delivery, learning transfer management, and performance support) have been addressed. In particular, evaluation of a program that leaves learning transfer to chance is likely to be disappointing and to reveal a lot of learning scrap. Simply measuring results does not itself change the equation. Take steps to improve the complete learning experience before jumping into evaluation to increase the probability of demonstrating the results the business is looking for. Finally, remember that documenting results is never the end; it is simply the beginning of the next cycle. What was learned in D6 from one program is the starting point for D1 of building the next, even better and more effective, program (see Figure D6.8).

CH006.indd 315

6/17/10 3:31:25 PM

316


FIGURE D6.8.

USE THE RESULTS OF D6 TO CREATE A CYCLE OF CONTINUOUS IMPROVEMENT

Summary The last of the Six Disciplines, document results, is critical to demonstrate the value of all the effort and investment that have gone before. Rigorous assessment of outcomes is essential to establish learning’s credibility and value, justify continued investment in learning and development, and support its continuous improvement. Effective evaluations are planned as part of the overall program design so that they are relevant, credible, and compelling to the target audience and efficient to execute. Planning involves agreeing on the parameters to be tracked, as well as the timing, data sources, comparators, and the definition of success. The design should be driven by the customer’s needs, the nature of the program, and the intended business outcomes, rather than theoretical models of evaluation. It should assess outcomes of interest to the business and not confuse learning management metrics with business results.

CH006.indd 316

6/17/10 3:31:25 PM

Document Results

317

Data analysis needs to be balanced—reporting successes as well as opportunities for improvement—and should include stories to help make the key conclusions memorable. The final report should be concise and to the point, with the key conclusions, recommendations, and plans for further improvement clearly set out in the executive summary. Finally, the results need to be actively marketed—communicated broadly and effectively through a variety of media—to all the relevant stakeholders. Only when these criteria are fulfilled does learning and development achieve its full promise and enjoy a well-deserved reputation for adding value. Use the checklist for D6 to help ensure that you have a robust plan to document results, prove the value of the learning experience, and improve subsequent iterations. CHECKLIST FOR D6

CH006.indd 317

✓ ❑

Aspect

Criterion

❑

Agreement

The way in which the program will be evaluated has been discussed and agreed upon with the program’s sponsor(s) in advance.

❑

Key Indicators

A small number of the key indicators that the program is expected to impact have been identified and agreed to by both the sponsor and the training and development organization.

❑

Leading Indicators

The earliest indicators that the program is working have been identified. A plan is in place to use these as in-process checks to drive improvement during the roll-out.

❑

Improvement Opportunities

The data that will be collected explicitly seek out information that will help identify opportunities to improve subsequent programs.

❑

Data Sources

The sources of the data that will be used in the evaluation have been identified; their availability has been confirmed.

❑

Data Collection

A plan is in place to gather needed data that are not already collected routinely.

❑

Comparators

Consideration has been given to how the effect of training will be isolated from other causes and what the post-training results will be compared to.

❑

Confounders

The factors most likely to confound (obscure or invalidate) the effects have been considered. There is a plan to control for them.

❑

Review

The evaluation plan has been reviewed by someone “skilled in the art” for validity and reliability.

❑

Presentation Plan

How the data will be reported and presented has been considered.

❑

Marketing

The key audiences for the results have been identified and there is a communications plan for each.

6/17/10 3:31:26 PM

318


Action Points For Learning Leaders • Review the evidence you currently have to illustrate that learning and development contributes to business success. • Can you make a compelling case for the economic value added that learning and development provides? • Can you convincingly demonstrate why reducing the investment would hurt the company’s long-term performance? • If the review identifies deficiencies, begin at once to rectify the situation. • Ensure that the plans for evaluation and the definition of success are part of every “contract” (explicit or implicit) that you accept for delivering learning and development programs. • Be proactive. Begin to build multiple lines of evidence of value. If you wait until you are asked to do an ROI study, it will be too late. • Market the value. • Get help from the marketing department. • If you have a great story to tell about the value you create, you cannot be shy and hope that people will notice. • If you believe it, you have to sell it. For Line Leaders • Review the current measures used in your business to assess the value of learning and development initiatives. • Are you satisfied? • Are they as rigorous as the criteria used to assess other investments of similar magnitude? • Require that every plan for a learning and development initiative includes a section that discusses the criteria for success and the plan for their evaluation. • Provide learning leaders with access to experts on evaluation and communications or external consultants as necessary. • Finally, save training’s budget presentations—particularly the sections on promised benefits—and require an evaluation of whether or not these were achieved as prerequisite to any subsequent budget request.

CH006.indd 318

6/17/10 3:31:26 PM

CODA Unless objectives are converted into action, they are not objectives; they are dreams. —PETER DRUCKER

Throughout this book, we have emphasized four key themes: • Training and development initiatives are strategic investments that a company makes in its human capital. They are as important to a company’s future as the investments it makes in research, new product development, sales and marketing, and acquisitions. • Training and development can produce significant returns and competitive advantage, provided they are planned, delivered, and managed in a systematic and disciplined way. • The 6Ds (see Figure C.1), when practiced in concert, produce a breakthrough in the proportion of learning that is converted into value and business results. • The third phase of the learning process—the transfer of learning to the work of the individual and firm—offers the greatest opportunity for transformational change.

Reprise: The Six Disciplines of Breakthrough Learning Define Business Outcomes The First Discipline is essential to ensure that resources are directed to the most important problems, to gain clarity about what needs to be delivered, and to define how results ultimately will be measured. It requires an open dialogue 319

CODA.indd 319

6/17/10 3:33:36 PM

320


FIGURE C.1. THE SIX DISCIPLINES OF BREAKTHROUGH LEARNING

6Ds Mind Map


between line management and learning leaders and a needs-assessment process that links business goals, the behaviors and performance needed to achieve them, and the learning required. Design the Complete Experience The Second Discipline serves as a reminder that the participant’s learning experience is much broader than that which occurs during the traditional period of instruction. It begins beforehand and continues afterward. It is influenced by

CODA.indd 320

6/17/10 3:33:37 PM

Coda

321

many organizational and environmental factors, most especially the participant’s direct supervisor. The discipline of designing the complete experience requires thinking holistically and systematically about learning. It challenges learning organizations to actively plan and influence elements outside the historical scope of learning and development. Deliver for Application Practicing the Third Discipline means focusing constant and vigilant attention on the ultimate goal: helping individuals improve their performance and hence that of the organization as a whole. It requires utilizing instructional methods that help bridge the learning-doing gap, stimulating the kind of thinking that will be needed on the job and providing practice with feedback. It means providing context, making relevance and utility clear, and helping participants continually come back to the question of how they can use what they are learning to advantage. Drive Learning Transfer The Fourth Discipline ensures that what is taught is used in ways that support the business aims. Driving learning transfer requires close collaboration between line management and the learning organization to ensure that participants are held accountable for using what they have learned. It requires a new mindset about what it means to successfully complete a program, and new tools to manage the process. Deploy Performance Support The Fifth Discipline is to provide the performance support that participants need to be successful as they strive to transfer their learning to their work. It requires marshalling an array of resources: job aids; the participants’ managers, peers, and coaches; instructors; and online information systems in order to provide a strongly supportive environment for learning transfer and application. It requires willingness to redeploy resources from pure instruction to instruction plus support and to think about information systems in a new way. Document Results The Sixth Discipline is both the end and the beginning of the cycle. It is the discipline of measuring, in a relevant, credible, and compelling way, the extent to which the program delivered on the promise defined in D1. Documenting

CODA.indd 321

6/17/10 3:33:38 PM

322


results and communicating them widely and well are essential to build a strong brand for learning, justify continued investment, and support continuous improvement.

Putting It All Together Implementing the 6Ds has helped many learning organizations increase the amplitude of the results they deliver and lower the cost of achieving them. A brief outline for getting started is presented at the end of this chapter (see Exhibit C.1). While practicing any one discipline adds value, the Six Disciplines work best when operating in concert. The chain is only as strong as its weakest link. The results are diminished when one discipline is missing or weak, no matter how well the others are executed. Continuous improvement of training and development depends on creating a never- ending cycle of identifying the weakest link, strengthening it, measuring the impact, and repeating the process. Learning is vital to staying competitive. Let us strive to make our contributions ever more effective.

The Journey So Far Since the original publication of The Six Disciplines of Breakthrough Learning, we have been delighted by the number of organizations that have experienced transformational changes, achieved outstanding results, and been honored for what they have accomplished by applying the 6Ds. Two final examples are given below, Cases in Point C.1 and C.2. We wish there were room to include more. The credit in all such cases belongs to the forward-looking and courageous learning leaders who were willing to abandon the safe harbor of the status quo and champion change into uncharted waters. If we helped show the path to where some of the buried treasure lies in training and development, we are genuinely pleased. Over the past five years, our readers and clients have presented us with fresh challenges and insights and encouraged us to learn along with them. This second edition is the result.

CODA.indd 322

6/17/10 3:33:40 PM

Coda

323

Case in Point C.1 Award-Winning Program Ontario Lottery and Gaming’s Leading the Way program was selected for a 2010 Award of Excellence by the International Society for Performance Improvement (ISPI). Criteria for the award include a systematic approach to the problem identification, needs analysis, design of the intervention, execution, monitoring, and assessment. Ontario Lottery and Gaming (OLG) is a provincial agency with a fiduciary responsibility for strong fiscal management. In partnership with the Rotman School of Management and Fort Hill Company, OLG expanded its strategy to heighten its focus on results with the introduction of a leadership development program, Leading the Way (Ontario Lottery and Gaming Corporation, personal communication, 2010). The goals of the program were to: • • • •

Engender a performance-based culture for leaders and managers. Build a culture of accountability with respect to employee development. Expand the repertoire of leadership skills, knowledge, and competencies. Create consistency in the language and application of leadership skills, knowledge, and competencies across the organization.

In designing the program, the learning organization took a systems approach that allowed the learning team to: • • • • • •

Prioritize the areas of greatest performance need. Integrate the new Leadership Principles developed by the CEO. Map the learning objectives for each module to OLG’s competency behaviors. Identify multiple barriers and issues in advance. Identify key leverage points and work collaboratively with them. Gain advocates for the program and increased buy-in with business leaders.

According to Priscilla Fraser, senior manager, training and development, “Keys to the success of the program included designing the complete experience and deploying active support. Both are critical in order to set expectations for participants, managers, and senior leaders. Each has a distinct role to play towards shared success. When you design the complete experience, these roles are identified and aligned which helps ensure involvement and active support throughout the entire intervention.” Other success factors included: • A focus on application (“Ideas I Can Use”) • Learning Transfer Support System: ResultsEngine was used to support and drive the six-month follow through of learning to business results, followed by integration into performance objectives (continued)

CODA.indd 323

6/17/10 3:33:40 PM

324


Case in Point C.1 Award-Winning Program (continued) • Performance support tools such as job aids and insights emails • Feedback and coaching: • A “coach to” classroom session for senior executives • Coaching tutorial for participants’ managers • Online coaching through the follow-through support system • Feedback scorecards and recognition for follow through cascaded from senior management • Senior OLG leadership stayed involved in the process, communicated expectations before, during, and after instructional intervention, and cascaded feedback to the group on how they were doing against the expected performance. Results to date have been positive. Five months after completing the course both participants and their managers were asked to rate the change in their effectiveness. • Ninety percent of managers agreed or strongly agreed that the program improved the performance of the participants. • Ninety percent of attendees agreed or strongly agreed that the program improved their effectiveness. • There was a very good correlation between managers’ and participants’ ratings. • Eighty - two percent of the time BOTH agreed or strongly agreed that it improved effectiveness. For example: “In accomplishing this goal you have been a major contributor to our Employee Engagement. The dedication that you have put forth with this objective shows that you care about this room, our employees, and the success of our management team.” Karen Zidenberg, program application and measurement lead, offers this advice to other learning leaders: “Engage your business partners to be committed and involved every step of the way; be sure you know what they think success looks like. Find out what is not working quickly. Request feedback and use the information to continually improve.”

Case in Point C.2 Revolutionizing Results at Securian Securian Financial Group is one of America’s largest providers of financial security for individuals and businesses in the form of insurance, retirement plans, and investments. When Chris Jenkins took over leadership of the learning function, he faced daunting challenges: the four-year retention rate for advisors was well below industry norms and it took more

CODA.indd 324

6/17/10 3:33:40 PM

Coda

325

than a year to get a new advisor “client-ready.” Together, these two issues were costing Securian’s member firms (agencies) hundreds of thousands of dollars annually. Four years later, what Chris and his colleagues had accomplished was extraordinary: • The time to client readiness for new associates was reduced from eighteen months to ninety days. • Advisors trained under the new system outperformed those trained in the old way by 100 percent. • First-year retention rates increased by more than 50 percent. • Program costs for the member firms were cut by 68 percent. How did they achieve such remarkable results? First, they persuaded management that to achieve meaningfully different results, they had to significantly change the way they approached training. Second, they rigorously applied the Six Disciplines, with special emphasis on Phase III of the learning process. With respect to doing things differently, Jenkins says: “Do not underestimate the effort required to achieve a cultural shift from old methods of training—meaning a talking head downloading information—to learning that actually changes behaviors and delivers results” (Jenkins, personal communication, 2010). Surprisingly, much of the resistance came from management, even from managers who faced some of the greatest challenges with retention. “Prepare for ‘we have always done it that way . . . they expect it,’” advises Jenkins. To find a champion, Jenkins and his colleagues identified a firm that had some of the greatest financial challenges and persuaded the manager that if he wanted a different result, he had to try something new. Since he was in the most pain and had the most to gain, he eventually came on board and is now a firm champion. The new approach had three key elements: • A complete learning experience that begins with a (scripted) discussion about the process and expectations during the hiring process and continues through a rigorous certification process ten weeks later. • The accountability for coaching and support resides in the local office where the advisor works. The central learning team does the “heavy lifting” by providing turnkey processes, scripts, and so forth—even rehearsing with the advisor’s managers—but the local leadership owns the responsibility to make sure that learning is practiced and applied. • There is a well-defined, rigorous assessment—the Checkpoints to Mastery—after ten weeks of training and on-the-job coaching. New advisors must show that they can discuss, explain, and demonstrate effectively in a live interview/examination with senior managers. According to Jenkins, “The Six Disciplines gave us a framework to approach training in a new and more effective way. The proof is in the results Securian member firms were able to achieve. As a result of this experience, I’ve come to the conclusion that if there is no commitment to all three phases of learning, you shouldn’t do the program. It will be a lot cheaper and about as effective to just send people the bagels.”

CODA.indd 325

6/17/10 3:33:41 PM

326


The Future Sir Isaac Newton, one of the greatest scientists of all time, said: “If I have seen further than others, it is by standing on the shoulders of giants,” acknowledging the contributions of previous thinkers to his own success. In the same way, the ideas that we have presented here also build on the shoulders of giants: other learning and development researchers, professional colleagues, learning leaders, program participants, our clients, and our employees. We invite you to apply and build on the concepts and disciplines we have presented to create breakthroughs of your own, add your wisdom to the accumulating knowledge, and share your insights with others. We look forward to hearing about them.

Exhibit C.1 A Brief Guide to Getting Started with the 6Ds Define Business Outcomes 1. Select either a new program or a critically important one. 2. Interview business leaders regarding their needs and expectations for post-program results using the Outcomes Planning Wheel; agree on the definition of success. 3. Triangulate the results by reading business plans, conducting additional interviews, and gathering additional data to complete the needs-assessment. 4. Create a value chain that shows the links between the business needs, the required knowledge and skills, and the proposed learning experiences. 5. Confirm your analysis with business leaders and discuss what else needs to be put in place to maximize the probability of success. Design the Complete Experience 1. Complete the 6D Scorecard (see Exhibit I.1 on page 16) for a new or existing program to identify opportunities for improvement. 2. Review the proposed instructional design and make sure it is in alignment with the required skills and on-the-job actions and that it addresses all four phases of learning. 3. Facilitate learner-manager discussions before and after the course. 4. “Staple yourself to the learner” as a final check. Walk through all the planned activities from the learner’s point of view to ensure that they are comprehensive and mutually reinforcing.

CODA.indd 326

6/17/10 3:33:41 PM

Coda

327

Deliver for Application 1. Begin each topic and exercise with the business rationale to answer the WIIFM (“What’s in it for me?”) question for participants. 2. Make sure the instructional method aligns with the knowledge and skills needed on the job. 3. Have learners practice new skills and behaviors in simulations that mimic the business environment. 4. Give participants time after each major topic to reflect on what they have learned and to think about how they can use it in their work. Drive Learning Transfer 1. 2. 3. 4.

Set clear timelines for reporting out progress and results. Remind participants periodically of the need to apply what they learned. Identify success cases and recognize superior achievement. Pilot-test the value of adding an online learning transfer management system. Deploy Performance Support

1. Engage managers. Inform them of their roles, and provide a straightforward process, clear guidelines, and the information they need to support learning transfer. 2. Promote collaboration through learning buddies, teams, shared accountability, online collaboration, and so on. 3. Provide job aids, online content, and other forms of performance support. 4. Reallocate some resources from pure instruction to providing post-instructional performance support. Document Results 1. Pick a strategically important new or existing program. 2. Define or confirm the sponsor’s definition of “success” and what he or she would consider relevant, credible, and compelling evidence. 3. Reduce to the critical few and collect the data as efficiently as possible. 4. Analyze the results conservatively. 5. Identify success cases and use them to illustrate the program’s value in reports and to future participants. 6. Report the outcomes in a clear, concise, and compelling manner. Provide a high-quality, one-page executive summary. 7. Market the results.

CODA.indd 327

6/17/10 3:33:42 PM

CODA.indd 328

6/17/10 3:33:43 PM

CLOSING THOUGHTS: LEARNING IS THE MASTER SKILL

A

“

re leaders born or made?” It’s a question we are asked nearly every time we give a speech or conduct a workshop. Our answer? We’ve never met a leader who wasn’t born! We’ve also never met an accountant, artist, athlete, engineer, lawyer, physician, writer, or zoologist who wasn’t born. We’re all born. That’s a given. It’s what you do with what you have before you die that makes the difference. Yet, as straightforward as that sounds, not everyone believes it. An insidious myth persists in our technologically advanced world that leadership—and many other abilities, for that matter—is reserved for only a lucky few. So let’s get something straight from the start. Leadership is not preordained. It is not a gene, and it is not a trait. There is no hard evidence to support the assertion that leadership is imprinted in the DNA of only some individuals and that the rest of us missed out and are doomed to be clueless. We’ve told this truth more times than we can count over the last thirty years, from our first book, The Leadership Challenge, to our newest, The Truth About Leadership. Leadership, like all skills worth having, can be learned—and must be practiced. The truth is that the best leaders are the best learners. Leadership is an observable pattern of practices and behaviors and a definable set of skills and 329

CLOSING.indd 329

6/17/10 3:32:19 PM

330


abilities. Skills can be learned, and when we track the progress of people who participate in leadership development programs we observe that they improve over time (Posner, 2009a). They learn to be better leaders as long as they engage in activities that help them learn. The same thing is true for the other roles people play in organizations—and in life. But here’s the rub. While leadership can be learned, not everyone learns it, and not all those who learn it master it. Why? Because to master something you have to have an intense desire to excel, you have to have a strong belief that you can learn new skills and abilities, and you have to be willing to devote yourself to continuous learning and deliberate practice. In The Six Disciplines of Breakthrough Learning, Cal Wick, Roy Pollock, and Andy Jefferson have done an extraordinary job of spelling out in great detail the things you need to do in order to transfer training and development into business results. We have followed their advice, and, full disclosure, we have incorporated their ResultsEngine into The Leadership Challenge Workshop. We know their process works. What we want to offer in these closing thoughts is our observations about what will keep these results coming not only after a program, but throughout your career and your life.

The truth is that the best leaders are the best learners.

Learning Is the Master Skill Over the years we’ve conducted a series of empirical studies to find out whether leaders could be differentiated by the range and depth of learning tactics they employ. We’ve wanted to know whether the way in which leaders learned played a role in how effective they were in leading. The results have been most intriguing. First, we found that leadership can be learned in a variety of ways. It can be learned through active experimentation, observation of others, study in the classroom or reading books, or simply reflecting on one’s own and others’ experiences (Posner, 2009b; Posner & Brown, 2001). Certain styles contribute to more effectiveness in some practices, but there is no one best style for learning everything there is to know. The style was not the thing. What was more important was the extent to which individuals engaged in whatever style worked for them. Those leaders who were more engaged in each of their learning styles, regardless of what their styles were, scored higher on our measures of leadership practices. The best leaders turned out to be the best learners. This same observation can be applied to any skill set you can name. Learning comes first. When people are predisposed to be curious and want to learn something new, they are much more likely to get better at it than those who don’t become fully engaged.

CLOSING.indd 330

6/17/10 3:32:20 PM

Closing Thoughts

331

Learning is the master skill. When you fully engage in learning—when you throw yourself whole-heartedly into experimenting, reflecting, reading, or getting coaching—you are going to experience the thrill of improvement and the taste of success. More is more when it comes to learning. Deliberate Practice Is Required There’s another pervasive myth that has captivated the training and development world over the last few years. It’s the myth of talent, and some have come to accept it as the new gospel. If only we search far and wide enough and long and hard enough, we’ll be able to identify the right person for the right job at the right time. No training required; just find the right person. Well, good luck with that. Florida State University professor and noted authority on expertise, K. Anders Ericsson, makes this point: Until most individuals recognize that sustained training and effort is a prerequisite for reaching expert levels of performance, they will continue to misattribute lesser achievement to the lack of natural gifts, and will thus fail to reach their own potential (Ericsson, 2006, p. 699).

Anders and his colleagues have found, over the twenty-five years of their research, that raw talent is not all there is to becoming a top performer. It doesn’t matter whether it’s in sports, music, medicine, computer programming, mathematics, or other fields; talent is not the key that unlocks excellence. What truly differentiates the expert performers from the good performers is hours of practice (Colvin, 2008; Coyle, 2009; Gladwell, 2008). You’ve got to work at becoming the best, and it sure doesn’t happen over a weekend. If you want a rough metric of what it takes to achieve the highest level of expertise, the estimate is about 10,000 hours of practice over a period of ten years (Ericsson, 2006, p. 692). That’s about 2.7 hours a day, every day, for ten years! These days you also hear a lot about how you should ignore your weaknesses, or you should find someone else who’s good at what you aren’t and partner with him or her. While it may be decent operational advice, that message is not consistent with what those who study expertise have found. Researchers have shown, across a variety of occupations and professions, that only by working at what you can’t do can you expect to become the expert you aspire to be (Ericsson, Prietula, & Cokely, 2007). If you want to be the best you can be, you will have to attend to your weaknesses. You can’t delegate or assign to others those skills you aren’t good at. If you do, you’ll only get as good as your weakest skill. While you may never get as

CLOSING.indd 331

6/17/10 3:32:20 PM

332


good as someone else at everything you do, by constantly practicing, practicing, and practicing you can improve. And you’ll also gain an appreciation for why persistence is another one of those attributes that differentiates the best from the also-rans. So here’s the mantra for the twenty-first century learner: No matter how good I am I can always get better. The secret to success is persistent, unrelenting, dedicated, and deliberate practice. That’s what discipline is. And it’s altogether fitting that Cal, Roy, and Andy should choose that word to use in the title of this book. Discipline comes from the Latin word meaning “teaching” and “instruction.” When you apply these disciplines you become a student, a learner, a disciple, someone who is dedicated to continuously improving one’s abilities and character. We urge you to take to heart the disciplines in this book. When you define, design, deliver, drive, deploy, and document, you’ll see the results that come from disciplined work. And you’ll gain a renewed appreciation for why some individuals develop mastery and others remain amateurs. Those who attain the highest levels of performance understand that learning is the master skill and that learning does not stop at the end of a course. In fact, it never stops. Learning is a lifelong journey that permits you to stay open to opportunities to turn your workplace into a practice field and every experience into a chance to grow. June 2010

CLOSING.indd 332

JIM KOUZES AND BARRY POSNER Santa Clara, California

6/17/10 3:32:21 PM

REFERENCES

Abrashoff, D. (2002). It’s your ship: Management techniques from the best damn ship in the Navy. New York: Grand Central Publishing. Akerman, J., Ekelund, H., & Parisi, D. (2005). Using business simulations for executive development. In J. F. Bolt (Ed.), The future of executive development (pp. 25–39). San Francisco: Executive Development Associates, Inc. Allen, C. (2008, October). Power messaging and SOLD. Paper presented at the Fort Hill Best Practices Summit, Mendenhall, Pennsylvania. Alliger, G., Tannenbaum, S., Bennett, W., Jr., Traver, H., & Shotland, A. (1997). A metaanalysis of the relations among training criteria. Personnel Psychology, 50(2), 341–358. American Educational Research Association, American Psychological Association, National Council on Measurement in Education. (1996). Standards for Educational and Psychological Testing. Washington, DC: American Psychological Association. American Express. (2007). The real ROI of leadership development: Comparing classroom vs. online vs. blended delivery. Retrieved April 2010 from www.ninthhouse.com/papers/ AmEx_RealROI.pdf. American Society for Training and Development. (2000, November/December). Playing with the rules: An interview with Thiagi. Alexandria, VA: ASTD. Retrieved April 2010 from www.thiagi.com/about-thiagi-astd-magalog.html. American Society for Training and Development. (2009). Value of evaluation: Making training evaluations more effective. Alexandria, VA: ASTD. Anders, G. (2003). Perfect enough: Carly Fiorina and the reinvention of Hewlett-Packard. New York: Penguin. Anderson, J. (2010). Cognitive psychology and its implications (7th ed.). New York: Worth. Aperian Gobal. (2010). GlobeSmart features. Retrieved April 2010 from www.globesmart .com/about_globesmart.cfm?content⫽11. 333

REFERENCE.indd 333

6/17/10 3:44:07 PM

334


Atkinson, T., & Davis, J. (2003). Principles of workplace learning: Insights and tools for performance improvement. Boston: Forum Corp. Babbie, E. (2010). The practice of social research (12th ed.). Belmont, CA: Wadsworth. Bahrick, H., & Hall, L. (2005). The importance of retrieval failures to long-term retention: A metacognitive explanation of the spacing effect. Journal of Memory and Language, 52(4), 566–577. Baldwin, T., & Danielson, C. (2000). Building a learning strategy at the top: Interviews with ten of America’s CLOs. Business Horizons, 43(6), 5–14. Baldwin, T., & Ford, J. (1988). Transfer of training: A review and directions for future research. Personnel Psychology, 15, 63–105. Basarab, D. (in press). Predictive evaluation: Maximize the value of training on your business. San Francisco: Berrett-Koehler. Bell, L. (2008). Raising expectations for concrete results: Leadership development at Holcim. In T. Mooney & R. Brinkerhoff (Eds.), Courageous training: Bold actions for business results (pp. 175–193). San Francisco: Berrett-Koehler. Bennis, W. (n.d.). Human capital is the basis for competitive advantage. Retrieved April 2010 from www.50lessons.com/viewlesson.asp?l⫽487. Berk, J. (2008). The manager’s responsibility for employee learning. Chief Learning Officer, 7(7), 46–48. Bersin, J. (2008a). Leadership development in 2008. Chief Learning Officer, 7(2), 18. Bersin, J. (2008b). The training measurement book: Best practices, proven methodologies, and practical approaches. San Francisco: Pfeiffer. Betoff, E. (2007, September). Profile of the chief learning officer: Executive program in work-based learning leadership. Paper presented at the Fort Hill Best Practices Summit, Mendenhall, Pennsylvania. Bingham, T., & Galagan, P. (2008, November). No small change. T ⫹ D, 62(11), 32–37. Blanchard, K. (2004). Foreword. In S. Blanchard & M. Homan, Leverage your best, ditch the rest: The coaching secrets top executives depend on (pp. ix–xii). New York: HarperCollins. Blanchard, K., Meyer, P., & Ruhe, D. (2007). Know can do! Put your know-how into action. San Francisco: Berrett-Koehler. Blee, B., Bonito, J., & Tucker, R. (2005). Pfizer Inc. In L. Carter, M. Sobol, P. Harkins, D. Giber, & M. Tarquino (Eds.), Best practices in leading the global workforce: How the best companies ensure success throughout their workforce (pp. 249–288). Burlington, MA: Linkage Press. Bloom, B., Englehart, M., Furst, E., Hill, W., & Krathwohl, D. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain. New York: Longmans Green. Boehle, S. (2006). Are you too nice to train? Training, 43(8), 16–22. Bolt, J. (2005). Mapping the future of executive development: Forces, trends, and implications. In J. F. Bolt (Ed.), The future of executive development (pp. 3–21). San Francisco: Executive Development Associates, Inc. Bordonaro, F. (2005). What to do. In M. Dulworth & F. Bordonaro (Eds.), Corporate learning: Proven and practical guides for building a sustainable learning strategy (pp. 123–232). San Francisco: Pfeiffer. Bossidy, L., & Charan, R. (2002). Execution: The discipline of getting things Done. New York: Crown Business. Boston, J. S., Allred, S., & Cappy, C. (2009). McKesson: Leaders teaching leaders— Accelerating the development of high potentials. In D. Giber, S. Lam, M. Goldsmith, &

REFERENCE.indd 334

6/17/10 3:44:07 PM

References

335

J. Bourke (Eds.), Linkage Inc.’s best practices in leadership development handbook (pp. 171–192). San Francisco: Pfeiffer. Boudreau, J. (2010). Retooling HR: Using proven business tools to make better decisions about talent. Boston: Harvard Business Press. Brafman, O., & Brafman, R. (2008). Sway: The irresistible pull of irrational behavior. New York: Doubleday. Brethower, D. (2009). It isn’t magic, it’s science. Performance Improvement, 48(10), 18–24. Brinkerhoff, R. (2003). The success case method: Find out quickly what’s working and what’s not. San Francisco: Berrett-Koehler. Brinkerhoff, R. (2006). Telling training’s story: Using the success case method to improve learning and performance. San Francisco: Berrett-Koehler. Brinkerhoff, R., & Apking, A. M. (2001). High impact learning: Strategies for leveraging business results from training. New York: Basic Books. Brinkerhoff, R., & Gill, S. (1994). The learning alliance: Systems thinking in human resources development. San Francisco: Jossey-Bass. Brinkerhoff, R., & Montesino, M. (1995). Partnerships for learning transfer: Lessons from a corporate study. Human Resource Development Quarterly, 6(3), 263–274. Broad, M. (2005). Beyond transfer of training: Engaging systems to improve performance. San Francisco: Pfeiffer. Broad, M., & Newstrom, J. (1992). Transfer of training: Action-packed strategies to ensure high payoff from training investments. Cambridge, MA: Perseus Books. Burke, L., & Hutchins, H. (2007). Training transfer: An integrative literature review. Human Resource Development Review, 6, 263. Buzan, T., & Buzan, B. (1993). The mind map: Radiant thinking. London: BBC Books. Campbell, D. (1974). If you don’t know where you’re going, you’ll probably end up somewhere else. Valencia, CA: Tabor. Carter, L., Ulrich, D., & Goldsmith, M. (Eds.). (2005). Best practices in leadership development and organization change: How the best companies ensure meaningful change and sustainable leadership. San Francisco: Pfeiffer. Chai, S. (2009). Small changes result in big improvements. Chief Learning Officer, 8(1), 48–49. Charan, R., Drotter, S., & Noel, J. (2001). The leadership pipeline: How to build the leadershippowered company. San Francisco: Jossey-Bass. Charlton, K., & Osterweil, C. (2005, Autumn). Measuring return on investment in executive education: A quest to meet client needs or pursuit of the Holy Grail? 360˚—The Ashridge Journal, pp. 6–13. Christensen, C., & Raynor, M. (2003). The innovator’s solution: Creating and sustaining successful growth. Boston: Harvard Business School Press. Clark, R. (1986). Defining the D in ISD. Part I: Task-general instruction methods. Performance and Instruction, 25(3), 17–21. Clark, R., Nguyen, F., & Sweller, J. (2006). Efficiency in learning: Evidence-based guidelines to manage cognitive load. San Francisco: Pfeiffer. Clark, T., & Gottfredson, C. (2008). In search of learning agility: Assessing progress from 1957 to 2008. TRClark LLC. Retrieved April 2010 from www.trclarkglobal.com/pdf/ TRCLARK-in_search_of_learning_agility-2008.pdf. Collins, J. (2001). Good to great: Why some companies make the leap…and others don’t. New York: HarperCollins. Colvin, G. (2006). What it takes to be great. Fortune, 154(9), 88–96.

REFERENCE.indd 335

6/17/10 3:44:08 PM

336


Colvin, G. (2008). Talent is overrated: What really separates world-class performers from everybody else. New York: Penguin. Connolly, M., & Burnett, S. (2003). Hewlett-Packard takes the waste out of leadership. Journal of Organizational Excellence, 22(4), 49–59. Connolly, M., & Rianoshek, R. (2002). The communication catalyst: The fast (but not stupid) track to value for customers, investors, and employees. Chicago: Dearborn Trade Publishing. Cooper, G. (1990). Cognitive load theory as an aid for instructional design. Australian Journal of Educational Technology, 6(2), 108–113. Covey, S. (2004). The 7 habits of highly effective people: Powerful lessons in personal change (2nd ed.). New York: Simon & Schuster. Coyle, D. (2009). The talent code. Greatness isn’t born. It’s grown. Here’s how. New York, Bantam Dell. Cromwell, S., & Kolb, J. (2004). An examination of work-environment support factors affecting transfer of supervisory skills training to the workplace. Human Resource Development Quarterly, 15(4), 449–471. Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. New York: Harper & Row. Cusic, L. (2009, October). A control-test case study: Business impact of coaching and followthrough support. Paper presented at the Fort Hill Best Practices Summit, Mendenhall, Pennsylvania. Danielson, C., & Wiggenhorn, W. (2003) The strategic challenge for transfer: Chief learning officers speak out. In E. Holton III & T. Baldwin (Eds.), Improving learning transfer in organization (pp. 16–38). San Francisco: Jossey-Bass. Darling, M., & Parry, C. (2001). After-action reviews: Linking reflection and planning in a learning practice. Reflections, 3(2), 64–72. Denning, S. (2005). The leader’s guide to storytelling: Mastering the art and discipline of business narrative. San Francisco: Jossey-Bass. Deutschman, A. (2005, May). Making change. Fast Company, pp. 52–62. Dilworth, R., & Redding, J. (1999). Bridging gaps: An update from the ASTD research committee. Human Resource Development Quarterly, 10(3), 199–202. Dixon, N. (1990). The relationship between trainee responses on participation reaction forms and posttest scores. Human Resource Development Quarterly, 1, 129–137. Dresner, M., & Lehman, L. (2009). The astounding value of learning brand: Learning brand is the learning organization’s most valuable intangible asset. Retrieved April 2010 from http://documents.corpu.com/research/CorpU_Astounding_Value_of_Learning_ Brand.pdf. Drucker, P. (1954). The practice of management. New York: Harper & Row. Drucker, P. (1974). Management: Tasks, responsibilities, practices. New York: Harper & Row. Dugdale, K., & Lambert, D. (2007). Smarter selling: Next generation strategies to meet your buyer’s needs—every time. Harlow, UK: Pearson Education. Dulworth, M., & Bordonaro, F. (Eds.). (2005). Corporate learning: Proven and practical guides for building a sustainable learning strategy. San Francisco: Pfeiffer. Dulworth, M., & Forcillo, J. (2005). Achieving the developmental value of peer-to-peer networks. In M. Dulworth & F. Bordonaro (Eds.), Corporate learning: Proven and practical guides for building a sustainable learning strategy (pp. 107–121). San Francisco: Pfeiffer. Dweck, C. (2006). Mindset: The new psychology of success: New York: Ballantine Books. Dyer, D., Dalzell, F., & Olegario, R. (2004). Rising tide: Lessons from 165 years of brand building at Procter & Gamble. Boston: Harvard Business School Publishing.

REFERENCE.indd 336

6/17/10 3:44:09 PM

References

337

Ebbinghaus, H. (1913). Memory. A contribution to experimental psychology. New York: Teachers College, Columbia University. Echols, M. (2005). ROI on human capital investment (2nd ed.). Arlington, TX: Tapestry Press. Echols, M. (2008a). Creating value with human capital investment. Wyomissing, PA: Tapestry Press. Echols, M. (2008b). Why is business impact important? Chief Learning Officer, 7(4), 12. Ericsson, K. (2006). The influence of experience and deliberate practice on the development of superior expert performance. In K. A. Ericsson, N. Charness, P. Feltovich, & R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 683–704). New York: Cambridge University Press. Ericsson, K., Charness, N., Feltovich, P., & Hoffman, R. (2006). The Cambridge handbook of expertise and expert performance. Cambridge, UK: Cambridge University Press. Ericsson, K., Krampe, R., & Tesch-Romer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363–406. Ericsson, K., Prietula, M., & Cokely, E. (2007). The making of an expert. Harvard Business Review, 85(7/8), 114–121. Feldstein, H., & Boothman, T. (1997). Success factors in technology training. In J. J. Phillips & M. L. Broad (Eds.), Transferring learning to the workplace (pp. 19–33). Alexandria, VA: ASTD. Flanders, V., & Willis, M. (1996). Web pages that suck: Learn good design by looking at bad design. San Francisco: Sybex. Gagné, R., Briggs, L., & Wager, W. (2004). Principles of instructional design (4th ed.). Belmont, CA: Wadsworth/Thompson Learning. Gebelein, S., Nelson-Neuhaus, K., Skube, C., Lee, D., Stevens, L., Davis, B., & Hellervik, L. (Eds.). (2004). Successful manager’s handbook: Develop yourself, coach others (7th ed.). Minneapolis: Personnel Decisions Int’l. General Electric Company. (2008). GE citizenship report 2007–2008: Investing and delivering in citizenship. Retrieved April 2010 from www.ge.com/files_citizenship/pdf/ GE_07_08_Citizenship_Report.pdf. Georgenson, D. (1982). The problem of transfer calls for partnership. Training and Development Journal, 36(10), 75–78. Gick, M., & Holyoak, K. (1983). Schema induction and analogical transfer. Cognitive Psychology 15(1), 1–38. Gilley, J., & Hoekstra, E. (2003). Creating a climate for learning transfer. In E. Holton, III, & T. Baldwin (Eds.), Improving learning transfer in organizations (pp. 271–303). San Francisco: Jossey-Bass. Girone, M., & Cage, P. (2009, October 7–8). Redesigning for enhanced learning transfer. Paper presented at the Fort Hill Best Practices Summit, Mendenhall, Pennsylvania. Gladwell, M. (2008). Outliers: The story of success. New York: Little, Brown. Goldsmith, M. (1996). Ask, learn, follow up, and grow. In F. Hesselbein, M. Goldsmith, & R. Beckhard (Eds.), The leader of the future: New visions, strategies, and practices for the next era (pp. 227–237). San Francisco: Jossey-Bass. Goldsmith, M. (2002, Summer). Try feedforward instead of feedback. Leader to Leader, 25, 11–14. Goldsmith, M., & Morgan, H. (2004, Fall). Leadership is a contact sport: The follow-up factor in management development. Strategy⫹Buisness, 36, 71–79. Grawey, J. (2005, May). Sony electronics talent and organizational development. Paper presented at Fort Hill Company’s 2005 Best Practices Summit, Mendenhall, Pennsylvania.

REFERENCE.indd 337

6/17/10 3:44:09 PM

338


Gregoire, T., Propp, J., & Poertner, J. (1998). The supervisor’s role in the transfer of training. Administration in Social Work, 22(1), 1–18. Hammonds, K. (2005, August). Why we hate HR. Fast Company, pp. 40–47. Harburg, F. (2004). They’re buying holes, not shovels. Chief Learning Officer, 3(3), 21. Heath, C., & Heath, D. (2008). Made to stick: Why some ideas survive and others die. New York: Random House. Heller, R., & Hindle, T. (1998). Essential manager’s manual. New York: DK Publishing. Holton III, E. (1996). The flawed four-level evaluation model. Human Resources Development Quarterly. 7(1), 5–21. Holton III, E. (2003). What’s really wrong: Diagnosis for learning transfer system change. In E. Holton III & T. Baldwin (Eds.), Improving learning transfer in organizations (pp. 59–79). San Francisco: Jossey-Bass. Holton III, E., Bates, R., & Ruona, W. (2000). Development of a generalized learning transfer system inventory. Human Resource Development Quarterly, 11(4), 333–360. Honeywell Corporation. (2004). Annual report. Morristown, NJ: Author. Ibarra, H. (2004, February). Breakthrough ideas for 2004. Harvard Business Review, pp. 13–32. International Society for Performance Improvement. (n.d.). What is human performance technology? Retrieved April 2010 from www.ispi.org/content.aspx?id⫽54. International Society for Performance Improvement. (2002). Performance Technology Standards. Silver Spring, MD: ISPI. International Society for Performance Improvement. (2010). Award submission guidelines. Retrieved April 2010 from www.ispi.org/content.aspx?id⫽954. Islam, K. (2006). Developing and measuring training the 6 sigma way: A business approach to training and development. San Francisco: Pfeiffer. Jefferson, A., Pollock, R., & Wick, C. (2009). Getting your money’s worth from training and development: A guide to breakthrough learning for participants. San Francisco: Pfeiffer. Jefferson, A., Pollock, R., & Wick, C. (2010). How to get your money’s worth from training and development [eLearning program]. Wilmington, DE: Fort Hill Company and Option 6. Kaplan, R., & Norton, D. (1992). The balanced scorecard—Measures that drive performance. Harvard Business Review, 70(1), 71–79. Kaye, B., & Jordan-Evans, S. (2008). Love ‘em or lose ‘em: Getting good people to stay (4th ed.). San Francisco: Berrett-Koehler. Keefer, D. (2010, March 4). Driving organizational change: ADP’s approach to create a world class service culture. Paper presented at the Knowledge Advisors Analytics Conference, National Harbor, Maryland. Kelley, H. (1950). The warm-cold variable in first impressions of persons. Journal of Personality, 18(4), 431–439. Kesner, I. (2003, May). Leadership development: Perk or priority? Harvard Business Review, pp. 29–38. Kirkpatrick, D. (1998). Evaluating training programs: The four levels (2nd ed.). San Francisco: Berrett-Koehler. Kirkpatrick, D., & Kirkpatrick, J. (2005). Transferring learning to behavior: Using the four levels to improve performance. San Francisco: Berrett-Koehler. Kirkpatrick, J., & Kirkpatrick, W. (2009). Kirkpatrick then and now: A strong foundation for the future. St. Louis, MO: Kirkpatrick Partners, LLC.

REFERENCE.indd 338

6/17/10 3:44:10 PM

References

339

Kirwan, C. (2009). Improving learning transfer: A guide to getting more out of what you put into your training. Surrey, England: Gower. Knowles, M., Holton III, E., & Swanson, R. (2005). The adult learner: The definitive classic in adult education and human resource development (6th ed.). Burlington, MA: Elsevier. Knudson, M. (2005). Executive coaching. In J. Bolt (Ed.), The future of executive development (pp. 40–53). San Francisco: Executive Development Associates. Kontra, S., Trainor, D., & Wick, C. (2007, September 12). Leadership development at Pfizer: What happens after class. Retrieved January 2010 from www.corpu.com. Kouzes, J., & Posner, B. (1990). The leadership challenge. San Francisco: Jossey-Bass. Kouzes, J., & Posner, B. (2007). The leadership challenge (4th ed.). San Francisco: Jossey-Bass. Kray, L., & Haselhuhn, M. (2007). Implicit negotiation beliefs and performance: Experimental and longitudinal evidence. Journal of Personality & Social Psychology, 93(1), 49–64. Kuhn, T. (1962). The structure of scientific revolutions. London: The University of Chicago Press, Ltd. Langley, G., Nolan, K., Nolan, T., Norman, C., & Provost, L. (1996). The improvement guide: A practical approach to enhancing organizational performance. San Francisco: Jossey-Bass. Levinson, S., & Greider, P. (1998). Following through: A revolutionary new model for finishing whatever you start. New York: Kensington Books. Liker, J. (2004). The Toyota way. New York: McGraw-Hill. Loehr, J. (2007). The power of story. New York: The Free Press. Lombardo, M., & Eichinger, R. (2009). FYI: For your improvement (5th ed.). Minneapolis, MN: Lominger International. Mager, R., & Pipe, P. (1997). Analyzing performance problems, or You really oughta wanna (3rd ed.). Atlanta, GA: CEP Press. Maister, D., Green, C., & Galford, R. (2000). The trusted advisor. New York: The Free Press. Mankins, M., & Steele, R. (2005, July/August). Turning strategy into great performance. Harvard Business Review, pp. 65–72. Margolis, F., & Bell, C. (1986). Instructing for results. San Diego, CA: University Associates. Martin, J., & Power, M. (1982). Organizational stories: More vivid and more persuasive than quantitative data. In B. Staw (Ed.), Psychological foundations of organizational behavior (pp. 161–168). Glenview, IL: Scott Foresman. McDonald, D., Wiczorek, M., & Walker, C. (2004). Factors affecting learning during health education sessions. Clinical Nursing Research, 13(2), 156–167. McLagan, P. (2003). New organizational forces affecting learning transfer. In E. Holton III & T. Baldwin (Eds.), Improving learning transfer in organizations (pp. 39–56). San Francisco: Jossey-Bass. Medina, J. (2008). Brain rules. Seattle, WA: Pear Press. Mohl, L. (2007). Leadership development with impact: The definition of Done? Paper presented at the Fort Hill Best Practices Summit, Mendenhall, Pennsylvania. Mohl, L. (2008). Diary of a CLO: Executive development at Children’s Healthcare of Atlanta. In T. Mooney & R. Brinkerhoff (Eds.), Courageous training: Bold actions for business results (pp. 139–158). San Francisco: Berrett-Koehler. Mosel, J. (1957). Why training programs fail to carry over. Personnel, 34(3), 56–64. Nadler, J., Thompson, L., & Van Boven, L. (2003). Learning negotiation skills: Four models of knowledge creation and transfer. Management Science, 49(4), 529–540. National Research Council. (2000). How people learn: Brain, mind, experience and school. Washington, DC: National Academy Press.

REFERENCE.indd 339

6/17/10 3:44:10 PM

340


Newstrom, J. W. (1986). Leveraging management development through the management of transfer. Journal of Management Development, 5(5), 33–45. Nielsen, J. (1997). How users read on the web. Retrieved April 2010 from www.useit.com/ alertbox/9710a.html. Nielsen, J., Schemenaur, P., & Fox, J. (n.d.). Writing for the web. Retrieved April 2010 from www.sun.com/980713/webwriting/. Paine, N. (2003, April). Presentation at the UNICOM Conference, Ashridge, England. Pallarito, K. (2009). E-mailing your way to healthier habits. Health News. Retrieved April 2010 from www.healthfinder.gov/news/newsstory.aspx?docid⫽627207. Park, Y., & Jacobs, R. (2008, February 20–24). Transfer of training: Interventions to facilitate transfer of training based on time and role perspective. Paper presented at the Academy of Human Resource Development International Research Conference in the Americas, Panama City, Florida. Patterson, K., Grenny, J., Maxfield, D., McMillan, R., & Switzler, A. (2008). Influencer: The power to change anything. New York: McGraw-Hill. Patton, M. (1997). Utilization-focused evaluation: The new century text. Newbury Park, CA: Sage. Peterson, B., & Nielson, G. (2009). Fake work: Why people are working harder than ever but accomplishing less and how to fix the problem. New York: Simon Spotlight Entertainment. Pfeffer, J., & Sutton, R. (2000). The knowing-doing gap: How smart companies turn knowledge into action. Boston: Harvard Business School Publishing. Phillips, C., & Torres, C. (2008). The inside scoop on what spurs millennial hires. Advertising Age, 79(35), 56. Phillips, J. (2003). Return on investment in training and performance improvement programs (2nd ed.). New York: Butterworth Heinemann. Phillips, J., & Broad, M. (Eds.). (1997). Transferring learning to the workplace. Alexandria, VA: ASTD. Phillips, J., & Phillips, P. (2002, September). 11 reasons why training & development fails . . . and what you can do about it. Training, pp. 78–85. Phillips, J., & Phillips, P. (2009). The real reasons we don’t evaluate. Chief Learning Officer, 8(6), 18–23. Phillips, J., & Stone, R. (2002). How to measure training results: A practical guide to tracking the six key indicators. New York: McGraw-Hill. Pink, D. (2006). A whole new mind. New York: Riverhead Books. Pink, D. (2009). Drive: The surprising truth about what motivates us. New York: Penguin Group. Plotnikoff, R., McCargar, L., Wilson, P., & Loucaides, C. (2005). Efficacy of an e-mail intervention for the promotion of physical activity and nutrition behavior in the workplace context. American Journal of Health Promotion, 19(6), 422–429. Porter, M. (1985). Competitive advantage: Creating and sustaining superior performance. New York: The Free Press. Posner, B. (2009a). A longitudinal study examining changes in students’ leadership behavior. Journal of College Student Development, 50(5), 551–563. Posner, B. (2009b). Understanding the learning tactics of college students and their relationship to leadership. Leadership & Organization Development Journal, 30(4), 386–395. Posner, B., & Brown, L. (2001). Exploring the relationship between learning and leadership. Leadership & Organization Development Journal, 22(6), 274–280. Prokopeak, M. (2009). Passion and precision. Chief Learning Officer, 8(6), 26–29.

REFERENCE.indd 340

6/17/10 3:44:11 PM

References

341

Quinn, C. (2009). Branding: Marketing Foundations Suite. Online course. Durham, NC: Imprint Learning Solutions. Redford, K. (2007, June). What’s the point of ROI? Training & Coaching Today, pp. 12–13. Reichheld, F. (2003, December). One number you need to grow. Harvard Business Review. Richardson-Green, F. (2010, March 4). How Steelcase is evolving its talent measurement. Paper presented at the Knowledge Advisors Analytics Conference, National Harbor, Maryland. Ries, A., & Trout, J. (2001). Positioning: The battle for your mind. New York: McGraw-Hill. Robinson, D., & Robinson, J. (1995). Performance consulting: Moving beyond training. San Francisco: Berrett-Koehler. Robinson, D., & Robinson, J. (2008). Performance consulting: A practical guide for HR and learning professionals. San Francisco: Berrett-Koehler. Robison, J. (2008, May 8). Turning around employee turnover. Gallup Management Journal. Retrieved April 2010 from http://gmj.gallup.com/content/106912/turning-aroundyour-turnover-problem.aspx#1. Roche, T., & Wick, C. (2005). Agilent Technologies. In L. Carter, M. Sobol, P. Harkins, D. Giber, & M. Tarquinio (Eds.), Best practices in leading the global workforce: How the best global companies ensure success throughout their workforce (pp. 1–23). Burlington, MA: Linkage Press/ Linkage, Inc. Roche, T., Wick, C., & Stewart, M. (2005). Innovation in learning: Agilent Technologies thinks outside the box. Journal of Organizational Excellence, 24(4), 45–53. Rossett, A., & Schafer, L. (2007). Job aids and performance support: Moving from knowledge in the classroom to knowledge everywhere. San Francisco: Pfeiffer. Rothwell, W., Lindholm, J., & Wallick, W. (2003). What CEOs expect from corporate training: Building workplace learning and performance initiatives that advance organizational goals. New York: AMACOM. Ruona, W., Leimbach, M., Holton III, E., & Bates, R. (2002). The relationship between learner utility reactions and predicted learning transfer among trainees. International Journal of Training and Development, 6(4), 218–228. Russ-Eft, D., & Preskill, H. (2009). Evaluation in organizations: A systematic approach to enhancing learning, performance, and change (2nd ed.). New York: Basic Books. Ryan, T. (2007). Modern experimental design. Hoboken, NJ: John Wiley & Sons. Saks, A. (2002). So what is a good transfer of training estimate? A reply to Fitzpatrick. Industrial-Organizational Psychologist, 39(3), 29–30. Saks, A., & Belcourt, M. (2006). An investigation of training activities and transfer of training in organizations. Human Resources Management, 45(4), 629–648. Santana, L. (2009). Making the value of development visible: A sequential mixed methodology study of the integral impact of post-classroom leader and leadership development. Unpublished doctoral dissertation, Leadership and Change Program, Antioch University, Yellow Springs, Ohio. Saslow, S. (2005). Executive education best practices. In M. Dulworth & F. Bordonaro (Eds.), Corporate learning: Proven and practical guidelines for building a sustainable learning strategy. San Francisco: Pfeiffer. Schaffer, R., & Thomson, H. (1992, January/February). Successful change programs begin with results. Harvard Business Review, pp. 2–11. Scherkenbach, W. (1988). The Deming route to quality and productivity: Road maps and roadblocks. Rockland, MD: Mercury Press.

REFERENCE.indd 341

6/17/10 3:44:11 PM

342


Schettler, J. (2003). The 2003 training top 100: Top five profile & ranking: Pfizer. Training, 40(3), 18–68. Senge, P. (1990). The fifth discipline: The art and practice of the learning organization. New York: Doubleday. Shapiro, B., Rangan, V., & Sviokla, J. (1992). Staple yourself to an order. Harvard Business Review, 70(4), 113–122. Sharkey, L. (2003). Leveraging HR: How to develop leaders in “real time.” In M. Effron, R. Gandossy, & M. Goldsmith (Eds.), Human resources in the 21st century (pp. 67–78). San Francisco: Jossey-Bass. Shrock, S., & Coscarelli, W. (2007). Measuring learning—Evaluating level II assessments within the eLearning Guild. In S. Wexler (Ed.), Measuring success: Aligning learning success with business success (pp. 155–164). Santa Rosa, CA: The eLearning Guild. Simons, D., & Chabris, C. (1999). Gorillas in our midst: Sustained inattentional blindness for dynamic events. Perception, 28(9), 1059–1074. Smith, P., & O’Neil, J. (2003a). A review of the action learning literature 1994–2000. Part 1: Bibliography and comments. Journal of Workplace Learning, 15(2), 63–69. Smith, P., & O’Neil, J. (2003b). A review of the action learning literature 1994–2000. Part 2: Signposts into the literature. Journal of Workplace Learning, 15(4), 154–166. Smith, P. L., & Ragan, T. (2005). Instructional design (3rd ed.). Hoboken, NJ: John Wiley & Sons. Smith, R. (2008, November). Aligning learning with business strategy. T ⫹ D, 62(11), 40–43. Stolovitch, H., & Keeps, E. (2004). Training ain’t performance. Alexandria, VA: ASTD Press. Sturm, R., & Schrader, D. (2007, October). Leader to leader. Paper presented at the Fort Hill Best Practices Summit, Mendenhall, Pennsylvania. Sullivan, J. (2005). Measuring the impact of executive development. In J. F. Bolt (Ed.), The future of executive development (pp. 260–284). New York: Executive Development Associates, Inc. Swanson, R. (2003). Transfer is just a symptom. The neglect of front end analysis. In E. Holton III & T. Baldwin (Eds.), Improving learning transfer in organizations (pp. 119–137). San Francisco: Jossey-Bass. Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. Learning and Instruction, 4, 295–312. Thalheimer, W. (2006). Spacing learning events over time: What the research says. Retrieved January 2010 from www.work-learning.com/catalog/. Thalheimer, W. (2007, April). Measuring learning results: Creating fair and valid assessments by considering findings from fundamental learning research. Retrieved April 2010 from www .work-learning.com/catalog/DocumentDownloadPages/DL_MeasuringLearningR.htm. Thalheimer, W. (2008). We are professionals, aren’t we? What drives our performance? In M. Allen (Ed.), Michael Allen’s 2008 e-learning annual (pp. 325–337). San Francisco: Pfeiffer. Tharenou, P. (2001). The relationship of training motivation to participation in training and development. Journal of Occupational and Organizational Psychology, 74(5), 599–621. Thomson, H. (1992, January/February). Successful change programs begin with results. Harvard Business Review, pp. 2–11. Tobin, D. (1998). The fallacy of ROI calculations. Retrieved April 2010 from www.tobincls. com/fallacy.htm. Todd, S. (2009, October). Branding learning and development. Paper presented at the Fort Hill Best Practices Summit, Mendenhall, Pennsylvania.

REFERENCE.indd 342

6/17/10 3:44:12 PM

References

343

Trainor, D. (2004, February). Using metrics to deliver business impact. Presentation at The Conference Board’s 2004 Enterprise Learning Strategies Conference, New York City. Tufte, E. (2003). The cognitive style of PowerPoint. Cheshire, CT: Graphics Press. van Adelsberg, D., & Trolley, E. (1999). Running training like a business: Delivering unmistakable value. San Francisco: Berrett-Koehler. Vanthournout, D., Olson, K., Ceisel, J., White, A., Waddington, T., Barfield, T., Desai, S., & Mindrum, C. (2006). Return on learning: Training for high performance at Accenture. Chicago: Agate. Vroom, V. (1995). Work and motivation (classic reprint). San Francisco: Jossey-Bass. (Originally published in 1964). Wall, S., & White, E. (1997). Building Saturn’s organization-wide transfer support model: Saturn Corporation. In J. J. Phillips & M. L. Broad (Eds.), Transferring learning to the workplace (pp. 165–187). Alexandria, VA: ASTD. Watkins, M. (2003). The first 90 days: Critical success strategies for new leaders at all levels. Boston: Harvard Business School Press. Webster’s college dictionary. (2001). New York: Random House. Welch, J. (2005). Winning. New York: HarperCollins. Wexley, K., & Baldwin, T. (1986). Post-training strategies for facilitating learning transfer: An empirical exploration. Academy of Management Journal, 29(3), 503–520. Wick, C. (2003). Going beyond the finish line. Training and Development, 58(7), 17–18. Wick, C., Pollock, R., & Jefferson, A. (2008). The six disciplines workshop [Workbook]. Wilmington, DE: Fort Hill Company. Wick, C., Pollock, R., & Jefferson, A. (2009). The new finish line for learning. T ⫹ D, 63(7), 64–69. Wick, C., Pollock, R., Jefferson, A., & Flanagan, R. (2006). The six disciplines of breakthrough learning: How to turn learning and development into business results. San Francisco: Pfeiffer. Zenger, J., Folkman, J., & Sherwin, R. (2005). The promise of phase 3. Training and Development, 59(1), 30–35.

REFERENCE.indd 343

6/17/10 3:44:12 PM

REFERENCE.indd 344

6/17/10 3:44:12 PM

INDEX

Page references followed by t indicate a table; followed by fig indicate an illustrated figure; followed by t indicate a table.

A Abrashoff, M., 57 Accountability: co-opetition impact on participant, 197fig; learning transfer role of, 185–187; of managers for coaching, 243; training and development, 10fig–11 Action learning, 128–129 Action planning, 154–155 ADDIE model, 23, 34 ADP Inc., 244, 250 Adult education principles, 116e Advanced Transition Program (Pfizer), 68, 84, 141 Agilent Technologies, 55, 67, 245 Akerman, J., 126, 127 Allen, C., 134, 144 Alliger, G., 263, 298 Amaxopoulos, B., 83, 95, 240, 241, 242 American Educational Research Association, 26 American Express, 77, 92, 174–175, 232–233, 292 American Psychological Association, 26 Anders, G., 274 Anderson, J., 130, 131, 262 Aperian Global’s GlobeSmart web tool, 25 Apking, A. M., 30, 88, 141, 142, 171, 261

Ashridge Business School, 30 ASK Europe, plc., 201 Assessment: Generalized Learning Transfer System Inventory tool for, 175–176fig, 177fig, 178, 298; as learning, 98–101; making expectations explicit through, 96–97fig; nature of the, 99–101; reasons for, 96; Self-Assessment of Transfer Climate, 178, 179e; setting clear goal line through, 97–98. See also Evaluation ASTD (American Society for Training and Development), 65, 127, 164, 258, 262, 266 ASTD evaluation study (2009), 266, 274, 282 ASTD International Conference, 165 AstraZeneca, 244, 287 Atkinson, T., 138

B Babbie, E., 279, 297 Babcock & Wilcox, 247, 271 Bahrick, H., 93 Baker Hughes, 144 Bakhos, J., 195 Baldwin, T., 25, 144, 202 Bartlett, R., 121

345

index.indd 345

6/17/10 3:41:32 PM

346 Bates, R., 68, 175 BBC, 61 The Beatles, 174 Belcourt, M., 66, 175, 292 Bell, C., 123 Bell, K., 190 Bell, L., 238, 240 Bellevue University’s Human Capital Lab, 291 Bennett, W., Jr., 263 Bennis, W., 19 Berk, J., 165 Bersin, J., 47, 59, 78 Best Practice Institute, 78 Betoff, E., 262 Bias sources, 289–290 Bingham, T., 21 Blair, J., 128 Blanchard, K., 70, 161, 250 Blee, B., 87 Bloom, B., 99, 100 Blue Wing Consulting (Cannon), 223 Boehie, S., 269 Bolt, J., 118 Bonito, J., 87 Boothman, T., 77 Bordonaro, F., 26, 249, 255 Bossidy, L., 80, 302 Boudreau, J., 33 Brafman, O., 73 Brafman, R., 73 Brain Rules (Medina), 111 BrainWorks, 128 Brand, 307 Breakthrough Coaching program (AstraZeneca), 244 Brennan, D., 287 Brethower, D., 287 Briggs, L., 125 Brinkerhoff, R., 30, 77, 88, 120, 140, 141, 142, 147, 167, 171, 261, 274, 277, 291, 299 Broad, M., 49, 66, 67, 68, 70, 77, 171, 232, 245 Brown, L., 330 BST (Behavioral Science Technology), 190 Burke, L., 109, 139, 175 Burnett, S., 24, 53, 119, 204, 206 Business knowledge: connections required for, 136–138; do your market research, 36–37; evaluating measures of, 263fig; “explicit” and “tacit,” 36; need for, 31–32; reflection component of, 138–140; self-test of, 32e; translating needs into outcomes, 37–41; understanding the current situation, 35–36; worksheet linking new and existing, 138fig. See also Information; Learning

index.indd 346

Index Business models, 33 Business needs: creating co-ownership, 45–46; enforcing prioritization, 46–47; interview guidelines for discovering, 38e; Outcomes Planning Wheel use of, 37, 38e, 40fig–45, 57, 120, 280–281fig; recognizing and delivering help for individual, 225–227; translated into desired outcomes, 37, 40–41 Business objectives: COS (conditions of satisfaction) as, 23; effective training programs aligned with, 22fig; electronic performance support of, 220, 222; facetious example of disconnect between method and, 121t; manager’s awareness of learning transfer related to, 239–245; picking the right, 31–41. See also Goals; Learning objectives Business outcome measures: identifying specific, 44–45; importance of taking, 25–26, 28; verifiable outcomes of relevance as, 29–31 Business outcomes: beginning with the end in mind for, 20–24; comparing learning objectives and, 28t–29; definition of, 30; documenting major categories of post-learning, 44t; examples of internal learning process metrics vs., 265t; examples of learning objectives vs., 29t; expressed in business terms, 24–31; factors affecting medical, 27fig; linking learning activities to, 120fig–124; pathways to improving, 33fig; using training to drive, 295; translating needs into desired, 37–41; value chain between Phase II exercise and, 89t. See also Define business outcomes (D1) Business strategy: creating learning transfer climate, 42, 174–178; Deloitte U.S. Firms’ Six Strategic Conclusions on, 85; learning as a, 21, 58; for managerial engagement, 241–242; Sun Microsystems on usability of information, 230; USS Benfold strategy replacing ineffective training, 57 Buzan, B., 136 Buzan, T., 136 BYN Mellon, 59

C Cage, P., 140 CallSource, Inc., 295 Campbell, D., 23, 297 Cannon’s Blue Wing Consulting, 223 Career Systems International, 115 Carter, L., 78 Case in Point: ASK Europe, plc. on learning vs. changing behavior, 201; Babcock & Wilcox evaluation of training, 271; BBC lack of training ROI, 61; better reaction scores may not be better, 269; COS (conditions of satisfaction), 23; detailed roadmap to unclear destination, 48; elaborative rehearsal for learning, 137; evaluation success

6/17/10 3:41:33 PM

Index stories, 301; failing to include stories, 276; failure of management support for training, 79; Gap, Inc. Merchant University, 53; GE (General Electric) use of Six Ds, 2; including compelling data in evaluation, 277; Ingersoll Rand’s use of learning as business strategy, 21; Kaiser Permanente and use of the 6Ds, 14; leadership development with impact, 101–102; Leading the Way program, 323–324; learning to effect change, 39; linking learning activities to business outcomes, 123–124; listening too late to voice of the customer, 272; making safety training stick, 190; maximizing value of marketing training, 236; ongoing support at ADP, 244; Plastipak Packaging linking learning and strategy, 58; predictive value of reaction data, 298; real play versus role play, 125; reflection in leadership education at Notre Dame, 139; resetting the finish line, 83; Securian Financial Group and Six Disciplines results, 324–325; Six Strategic Conclusions (Deloitte U.S. Firm), 85; transformation at Humana, Inc., 261; truly engaging managers, 241–242; UPS on extra value from follow-through, 207; USS Benfold strategy replacing ineffective training, 57 Case studies, 127 Center for Creative Leadership, 207, 223, 297 Center for Leadership, 101–102, 189, 247 Center for Naval Leadership, 315 Centocor, Inc., 238 Chabris, C., 131 Chai, S., 14, 288 Change: challenge of, 178–181; essential elements for, 181–189; learning versus behavior, 201; making sure that training produces meaningful, 283–284; multipronged approach to achieving, 181fig; reaching critical mass of, 248–249; resistance to, 180fig ChangeLever International, 91, 238 Charan, R., 80, 84, 302 Charlton, K., 30 Charness, N., 93 Checklists: D1 (define business outcomes), 62; D2 (design the complete experience), 104–106; D3 (deliver for application), 159; D4 (drive learning transfer), 210; D5 (deploy performance support), 252–253; D6 (document results), 317; Phase I (prepare), 104; Phase II (learn), 105; Phase III (transfer), 105; Phase IV (achieve), 105–106 Chevalier, R., 269 Chief Learning Officer magazine, 65 Children’s Healthcare, 101–102 Christensen, C., 35 Chubb Group of Insurance Companies, 83, 95, 141, 183, 189, 237, 240

index.indd 347

347 Clark, R., 128, 216, 227 Clark, T., 65 Co-opetition, 197fig Coaches: designated, 249–250; improving transfer climate through feedback of, 187; learning support of designated, 249–250; motivating managers to be, 235–245 Coaching: accountability of managers for, 243; increasing individual development plan role with, 204fig; LMS (learning management system) application of, 194–195 Cokely, E., 93, 99, 313, 331 Collaborative learning: electronically fostering, 227–228fig; facilitating, 195–197 Collins, J., 259 Colvin, G., 93, 173, 174, 331 Communication: electronic performance support systems using work, 228; A Job Aid for Creating an Elevator Speech, 155–156e; LMS (learning management systems), 192–193; ParticipantManager Dialogue form for, 239fig; by participants on training experience, 155–156; reporting evaluation findings, 304–305; targeting your marketing, 307–308 Communication vehicles, 310, 311, 312 Community of learners, 195–197, 246–249 Compelling evaluation, 275–278, 279t Complete learning experience: components required for the, 7fig; four-phase process of, 68–101; a new paradigm for, 67t–68; participant satisfaction as dependent on, 214–216. See also Design the complete experience Connolly, M., 54, 119, 204, 206 Continuous improvement, 258–259 Cooper, G., 137 Cornell’s Johnson Graduate School of Management, 196 Corning, 112 Corporate University Xchange, 32, 306 COS (conditions of satisfaction), 23 Coscarelli, W., 99, 128 Cote, D., 312 “The course is the thing” paradigm, 81–82 Covey, S., 20 Coyle, D., 93, 249 Credible evaluation, 272–275, 279t Critical mass concept, 248–249 Cromwell, S., 92 Csikszentmihalyi, M., 131, 186 Cusic, L., 203, 289 Customer satisfaction: future success dependent on, 214; whole product experience increasing, 215fig

6/17/10 3:41:33 PM

348 Customers: brand as promise to, 307; satisfaction of, 214, 215fig; “voice of the customer,” 270–272

D Dalzell, F., 19 Danielson, C., 25 Darling, M., 263 Dashboard (learning management system), 198–199fig Data analysis: planning the, 300, 302; process of, 302–303 Data collection: bias sources in, 289–290; choosing comparators, 288–292; deciding when to conduct, 285–286; kinds of evidence and methods of, 293t; looking for leading indicators, 286–288; selecting techniques for, 292–300 Data collection evidence: business metrics, 293t, 294; estimates, 293t, 296; listed, 293t; observations, 293t, 294, 296; opinions, 293t, 296–299; stories, 293t, 299–300 Davis, J., 138 Define business outcomes (D1): avoiding training traps, 47–59; begin with the end in mind for, 20–24; brief guide to getting started with, 326fig; Checklist for, 62; express outcomes in business terms, 24–31; as First Discipline, 3–5, 13fig; manage expectations and, 59–61; pick the right problems, 31–41; reprise of, 319–320; timeline from learning to results, 12fig, 263fig. See also Business outcomes Deliberate practice, 331–332 Deliver for application (D3): brief guide to getting started with, 327e; check the process, 156–158; Checklist for, 159; communicate what happened, 155–156e; improve perceived utility, 118–119; make it memorable, 129–140; make relevance clear, 115–118; motivate learners, 112–115; narrow the learning-doing gap, 110fig–111fig; overview of, 8fig–9, 13fig, 109–110; plan for transfer, 152–155; prepare for transfer, 140–152; provide know-how, 119–129; reprise of, 321 Deloitte LLP, 24, 84 Deloitte U.S. Firms’ Six Strategic Conclusions, 85 Deming, W. E., 65 Denning, S., 134, 277, 299 Deploy performance support (D5): analogy to customer and product support, 214–216; brief guide to getting started with, 327e; Checklist for, 252–253; materials, 216–219; overview of, 10fig–11, 13fig, 213–214; people role in, 231–252; reprise of, 321; systems role in, 219–231fig Design the complete experience (D2): brief guide to getting started with, 326e; Checklist for, 104–106; as encompassing more than instruction, 7fig; four-phase

index.indd 348

Index process for, 68–101; many factors influence outcomes, 66fig; a new paradigm for, 67–68; overview of, 5–8, 13fig; participant satisfaction related to, 214–216; reprise of, 320–321; stable yourself to the learner, 101–103fig. See also Complete learning experience; Instructional design models Designated coaches, 249–250 Deupree, R., 19 Deutschman, A., 173, 180 Development. See Training/development programs DevelopmentEngine system, 203–204 Dilworth, R., 258 Dixon, N., 298 DMAIC (Define, Measure, Analyze, Improve, and Control), 260–261 Document results (D6): attributes of effective evaluation, 279t; benefits of, 255–262; brief guide to getting started with, 327e; caveat to evaluation, 314–315; Checklist for, 317; evaluation challenge and, 262–267; guiding principles for, 267–279; major categories of post-learning outcomes and, 44t; overview of, 11–12, 13fig, 255; reprise of, 321–322; six-step process of evaluation, 280fig–314; timeline from learning to results, 12fig, 263fig Dresner, M., 306 Drive learning transfer (D4): better return on learning, 200–208; brief guide to getting started with, 327e; Checklist for, 210; creating transfer climate, 42, 174– 178, 179e; follow-through breakthrough, 189–199fig; Generalized Learning Transfer System Inventory, 175–176fig, 177fig, 178; a great learning experience is not enough, 171fig–173; implementing learning transfer management, 208–209; learning scrap for, 166–167; overcoming inertia, 178, 180fig–189; overview of, 9–10, 13fig, 161–164; as pernicious problem, 164–165; practicing to get into “Carnegie Hall,” 173–174; reprise of, 321; root causes of transfer failure, 167–171; as weak link in training, 9fig. See also Learning transfer Drotter, S., 84 Drucker, P., 19, 24, 25, 243, 319 Dugdale, K., 75 Dulworth, M., 247, 255, 312 Dweck, C., 152, 176, 266 Dyer, D., 19 Dynamic Leadership Program (Hewlett-Packard), 204–205fig

E e-Learning, 128 E.A. Link, 125–126 Ebbinghaus, H., 134

6/17/10 3:41:33 PM

Index Echols, M., 186, 291 Effective evaluation, 267, 279t Eichinger, R., 226 Ekelund, H., 126, 127 eLearning Guild, 65 Electronic performance support systems: business objectives achieved through, 220–231; description of, 220; eight principles of great, 221e; finding relevant information using, 221fig; focusing on superior performance differentiates, 222–225; fostering collaboration through, 227–228fig; to help users define, track, and achieve goals, 222; helping people act smarter, 230–231; using language of the work and workers, 228; providing what is needed, 228–230; recognizing/delivering help that people need, 225–227; supporting achievement of business objectives, 220, 222. See also Learning transfer management system Eli Lilly, 124 Employees: Millennials, 311; post-course learning activities for, 182–183; range of managers’ responses to new capabilities of, 234t; reaching critical mass through new learning applied by, 248–249. See also Participants Englehart, M., 100 Ericsson, K. A., 93, 99, 173, 313, 331 “Escape velocity,” 172fig Essential Manager’s Handbook (Heller & Hindle), 226 Evaluation: bias in, 289–290; caveat on, 314–315; challenges of, 262–267; definition and attributes of effective, 267–279t; isolating effects of training, 291– 292; Kirkpatrick’s model of, 304; Outcomes Planning Wheel review during, 280–281fig; participant and manager roles in, 309–310; timing of, 286fig. See also Assessment Evaluation attributes: compelling, 275–278; credible, 272–275; effective, 267, 279t; efficient, 278–279; relevant, 268–272; summary of, 279t Evaluation challenges: business results versus process metrics, 263–265; ends versus means, 265; selffulfilling prophesy, 266–267 Evaluation process: 1: confirm outcomes that matter, 280–285; 2: design the details of the evaluation, 285–302; 3: collect and analyze the data, 302–303; 4: report the findings to management, 303–306; 5: sell the sizzle, 306–313; 6: implement improvements, 313–314 Exemplary Practice Award for Business Alignment, 32 Expectancy Model, 112–113fig Expectations: assessment creating explicit, 96–97fig; improve the invitation to learn to influence, 76; learning outcomes influenced by, 72–74fig; managing

index.indd 349

349 performance improvement, 4fig; performance relative to, 81fig; program timeline showing learning that resets, 82fig; raising learning, 75–76; training as raising performance, 80fig Expert Opinion, 297–299 Explicit business knowledge, 36

F Facilitators, 245–246 “Fake work,” 47 Farm Credit Canada, 121–122 FDA (Federal and Drug Administration), 22, 26 Federal Aviation Administration, 135 Feedback: learning transfer role of, 187; learning transfer sustained through, 250–252; performance improvement through, 99; 360-feedback survey process for, 247, 288 “Feedforward,” 154–155 Feldman, J., 295 Feldstein, H., 77 Feltovich, P., 93 Fidelity Investments, 249 Fifth Discipline (D5). See Deploy performance support (D5) The finish line, 188–189 Fiorina, C., 274 The First 90 Days: Critical Success Strategies for New Leaders at All Levels (Watkins), 213 First Discipline (D1). See Define business outcomes (D1) “Fixed mindset,” 266 Flanagan, R., 171 Flanders, V., 229 “The flow,” 131 Folkman, J., 9, 69 Follow-through breakthrough: description of, 189–191; example of learning leader’s dashboard for, 199fig; learning management system for, 191–199 Forcillo, J., 247, 312 Ford, J., 202 Forgetting: spaced learning and learning and, 135fig; spaced repetitions and learning and, 136fig Fortune magazine, 174 Forum Corporation, 138 4Ps (People, Policies, Procedures, and Plant), 169 Fourth Discipline (D4). See Drive learning transfer (4D) Fox, J., 230 Fraser, P., 325 Frasso, J., 207 Friday5s system, 87–88, 189, 200, 207, 225, 241–242 From the Earth to the Moon (Verne), 171 Furst, E., 100 FYI: For Your Improvement (Lombardo & Eichinger), 226

6/17/10 3:41:34 PM

350

G Gagné, R., 125 Galagan, P., 21 Games, 127–128 Gap, Inc., 53 Gebelein, S., 226 General Electric (GE), 182, 246–247, 259, 296 General Mills, 307, 315 Generalized Learning Transfer System Inventory, 175–176fig, 177fig, 178, 298 Georgenson, D., 165 Getting Your Money’s worth from Training and Development (Jefferson, Pollock, & wick), 143, 149, 238 Gick, M., 135 Gill, S., 88, 120 Gilley, J., 177 Gilson, P., 70 Girone, M., 140 Gladwell, M., 93, 174, 331 Glasbergen, R., 52 GlaxoSmithKline’s Leadership Edge program, 223 GlobeSmart web tool (Aperian Global), 25 Goals: assessments setting clear, 97–98; electronic learning transfer management systems reviewing, 147fig; electronic performance support to define, track, and achieve, 222; GWOP (goals without plans), 154; learning transfer, 140–152; learning transfer management system to achieve individual, 224fig; picking the right, 31–41; predictive evaluation model on importance of, 146fig; as prerequisite for success, 21–24; SMART, 149–150. See also Business objectives Goldsmith, M., 78, 154, 186, 249, 288 Good to Great (Collins), 259 Gottfredson, C., 65 Grawey, J., 189 Gregoire, T., 147 Greider, P., 184 Grenny, J., 34, 134, 152 Grid International, 195, 223 “Growth mindset,” 266 GWOP (goals without plans), 154

H Hall, L., 93 Hammonds, K., 54 Harburg, F., 4 Harter, J. K., 3 Harvard Business Review, 102, 174, 256, 300 Haselhuhn, M., 266 Heath, C., 275 Heath, D., 275

index.indd 350

Index Heller, R., 226 Henkel, H., 21 Hewlett-Packard, 117–118, 148, 204–205fig, 227 “High-Impact Learning and You” (Plastipak Academy), 87 Hill, W., 100 Hindle, T., 226 Hinton, D., 32, 87, 240 Hoekstra, E., 177 Hoffman, R., 93 Holcim’s North American Learning Center, 238 Holton III, E., 68, 90, 115, 119, 124, 175, 178, 222, 285, 298 Holyoak, K., 135 Home Depot: resetting the finish line by, 83; Store Managers Learning Forums at, 225 Honeywell, 188, 312 How People Learn report, 124 How to Get Your Money’s Worth from Training (online support), 239 Hughes, B., 134 Hughes, G., 137 Human Performance Technology (ISPI), 23 Human Resources and Diversity for Science and Technology (Corning), 112 Humana, Inc., 39, 261 Hutchins, H., 175

I Ibarra, H., 174, 213 IBM, 222 Impact map, 120 Improving Learning Transfer in Organizations (Holton), 178 Influencer (Patterson and colleagues), 132–133 Information: memory retrieval of, 129–140; providing know-how, 119–129; World Wide Web source of, 229. See also Business knowledge; Learning Ingersoll Rand, 21 Innovator’s Solution (Christensen & Raynor), 35 INSEAD, 174 Instructional design models: identifying key KSAs, 34; identifying learning objectives, 23. See also Design the complete experience Instructional methods study (American Express), 174–175 Instructors: new facilitator role of, 245–246; as trusted learning resource, 245 Integral Leadership program (Notre Dame), 139 Integrated Leadership Curriculum (Sony), 68 International Society for Performance (ISPI), 23, 59, 68, 308 Inventure Group, 309

6/17/10 3:41:34 PM

Index Islam, K., 270, 282 Izzo, J., 122, 172

J Jacobs, R., 171 Jefferson, A., 20, 69, 79, 80, 81, 143, 149, 166, 171, 218, 238, 239, 330 Jenkins, C., 85, 324, 325 Jennings, C., 30, 284 A Job Aid for Creating an Elevator Speech, 156e Job aids, 218–219 Johnson Graduate School of Management (Cornell), 196 Jordan-Evans, S., 311 Journal of Organizational Excellence, 67, 204 Jusela, G., 247–248

K Kaiser Permanente, 14, 185, 246, 248 Kaplan, R., 286 Kaye, B., 115, 311 Keefer, D., 244 Keeps, E., 49, 51 Kelley, H., 72 Kesner, I., 256–257 Kirkpatrick, D., 60, 213, 234, 235, 264, 284, 304 Kirkpatrick, J., 60, 213, 264, 284 Kirwan, C., 115, 129, 165, 247, 248, 290, 385 KLA-Tencor, 137 Klein, S., 295 Know Can Do! (Blanchard), 161 Knowledge Advisors, 283 Knowledge. See Business knowledge Knowles, M., 90, 115, 119, 124, 222 Knudson, M. J., 249 Kolb, J., 92 Kontra, S., 84, 92, 232, 288 Kouzes, J., 77, 274, 332 Kouzes-Posner First Law of Leadership, 274 Krampe, R., 93, 173 Krathwohl, D., 100 Kray, L., 266 Kuhn, T., 67

L Lalande, M., 119, 120 Lambert, D., 75 Langley, G., 259 Leader to Leader program, 271 The Leader’s Guide to Storytelling (Denning), 134 Leaders Teaching Leaders Program, 76 The Leadership Challenge Workshop, 330

index.indd 351

351 “Leadership Development: Perk or Priority?” (Kesner), 257 Leadership Development Program (LDP), 207–208 Leadership Edge program (GlaxoSmithKline), 223 Leadership Essentials and Leading for Performance (SCB), 68 Leadership Forum (Sony Electronics), 189 Leadership Institute (USC), 19 The Leadership Pipeline (Charan, Drotter, & Noel), 84 Leading Edge course (Pfizer), 84 Leading the Way program, 323–324 LeadingWork, 271 Learner community, 195–197, 246–249 “Learner intentionality,” 141 Learning: assessment as, 98–101; as a business strategy, 21, 58; changing behavior versus, 201; collaborative, 195–197, 227–228fig; to effect change, 39; enforcing prioritization of, 46–47; four-phase phases of, 68–101; gap between doing and, 8fig, 111fig–112fig; Hewlett-Packard’s data on factors of, 117fig–118; learning and forgetting with spaced, 135fig; as the master skill, 330–331; motivation as vital to, 112–115; Phase II of, 88–90; providing know-how for, 119–129; reflection required for, 138–140; relevant principles of adult education, 116e; timeline between results and, 12fig, 263fig; value must be added by, 256–258. See also Business knowledge; Information; Memory retrieval; Participants Learning activities: action learning, 128–129; case studies, 127; e-Learning, 128; elaborative rehearsal, 137; games, 127–128; post-course, 182–183; real play versus role play, 125; simulations, 125–127; skill practice, 125; value chain by linking business outcomes to, 120fig–124 Learning Center (Pfizer), 84, 206 Learning four-phase process: 1: prepare, 72–88; 2: learn, 88–90; 3: transfer, 90–96; 4: achieve, 96–101; overview of, 68–72 Learning management systems (LMS): components of an effective, 191–199fig; dashboard of, 198–199fig; description of, 191 Learning objectives: comparing business outcomes and, 28t–29; examples of business objectives vs., 29t. See also Business objectives Learning organizations: enforcing prioritization in, 46–47; training success/failure co-owned by line management and, 162–164 Learning scrap: description of, 166; high costs of, 167 Learning transfer: “elephant in the room” problem of, 162fig; estimates of effective, 166fig; expectations influencing outcomes of, 72–74fig; feedback to sustain, 250–252; Fourth Discipline on driving, 9–10;

6/17/10 3:41:35 PM

352 Learning transfer (continued ) helping to create a positive climate for, 42; many factors influencing outcomes of, 66fig; “out-of-sight, out-of-mind” problem of, 183–185; Phase III on, 90–95; planning for, 152–155; preparing for, 140–152; reaching the finish line, 188–189; role-time transfer matrix for, 67t; root causes of failure, 167–171; self-efficacy factor in, 176; timeline showing expectations reset by, 82fig; as weak link in training/development programs, 9; work environment as factor in, 177. See also Drive learning transfer (D4); Participants; Phase III (transfer) Learning transfer climate: Generalized Learning Transfer System Inventory to assess, 175–176fig, 177fig, 178; self-assessment of, 179e; strategies for creating a, 42, 174–178; Transfer Climate Scorecard on, 178, 179e Learning transfer failure: cause-and-effect diagram tool for analysis of, 169–171; examining the causes of, 167–169; learning-to-improvement breakdown and, 168e Learning transfer goals: example of structured goalsetting form, 151e; importance of setting, 140–141; no quality control problem/solutions, 144–149; not enough time problem/solutions, 141–144; summary of, 142t; writing strong goals problem/solutions, 149–150, 152 Learning transfer management system: DevelopmentEngine, 203–204; Friday5s, 87–88, 189, 200, 207, 225, 241–242; implementing the, 208–209; increasing management role in individual development plans, 204fig; LMS (learning management systems) for, 191–199fig; organized to achieve individual goals, 224fig; ResultsEngine, 189, 195, 203, 230, 301, 310, 330; ROI (return on investment) increased by, 200–208. See also Electronic performance support systems Learning transfer plan: action planning, 154–155; on facilitators impact on post-course application, 153fig; good facilitation, 153–154 Learning Transfer System Inventory tool, 175–176fig, 177fig, 178, 298 A Legacy of 21st Century Leadership: A Guide for Creating a Climate of Leadership Throughout Your Organization (Trinka & Wallace), 115 Lehman, L., 306 Leider, R., 23, 309 Lennon, J., 174 Levinson, S., 184 Liker, J., 259 Lindholm, J., 258

index.indd 352

Index Loehr, J., 152 Lombardo, M., 226

M McDonald, D., 131 McDonald’s, 183–184 McKesson, 76 McLagan, P., 170 McMillan, R., 34, 134, 152, 186 Made to Stick (Heath & Heath), 275 Magee, R., 188 Mager, R., 49, 97 Manage expectations, 59–61 Management: decisions on which initiatives to fund, 257fig; expectations for performance improvement by training, 4fig; reporting evaluation findings to, 303–304; secure training support from, 76–79; training success/failure co-owned by learning organization and line, 162–164 Managerial engagement: awareness of learning transfer objectives for, 239–247; awareness of meeting transfer objectives by, 148fig; Chubb Group’s strategies for, 241–242; example of online learning and performance support for, 240fig; ParticipantManager Dialogue form, 239fig; sending learning transfer goals for, 147–149 Managers: demanding accountability from subordinates, 186–187; evaluation role of, 309–310; increasing individual development plan role by, 204fig; learning support by, 232–245; Pfizer’s study on learning impact by, 232, 237; Phase 1 involvement of, 77e; range of responses to employees’ new capabilities by, 234t; training ratings by participants and, 243fig Mannix, B., 196 Maresh, N., 128 Margolis, F., 123 Market research, 36–37 Marketing: importance of, 307; repetition of message for effective, 308–309; segmenting and targeting your, 307–308 Martin, J., 133 Materials: job aids, 218–219; training support through, 216–219 Maxfield, D., 34, 134, 152, 186 Measures/metrics: business outcome, 25–26, 28–31, 44–45; decision tree for selecting parameters to, 282fig; evaluating knowledge, 263fig; Outcomes Planning Wheel, 41–42, 281fig; skill acquisition, 263fig; summing up comparators and, 292; understanding what matters to stakeholders, 281–283 Medina, J., 111, 130, 131, 132, 136

6/17/10 3:41:35 PM

Index Memory retrieval: attention and, 130–132; connections used to facilitate, 136–138fig; emotions role in, 132; as essential to training, 129–130; model showing role of memory in directing action, 130fig; repetition as aiding, 134–136fig; stories used to facilitate, 132–134. See also Learning Mencken, H. L., 267 Merchant University (Gap, Inc.), 53 Metrics That Matter, 283, 303 Millennials, 311 Mindset (Dweck), 266 MIT learning experiment, 72–73 Mohl, L., 101–102, 189 Montesino, M., 77, 147 Morgan, H., 249, 288 Mosel, J., 45, 161 MotivAider, 184–185 Motivation/motivating: Expectancy Model on, 112–113fig; learning and required, 112–115; managers to coach, 235–245; post-course reminders as, 183–185; post-course support role in maintaining, 187–188; WIIFM (“What’s in it for me?”), 158, 237. See also Participants Mozart, W. A., 173

N Nadler, J., 127 National Council on Measurement in Education, 26 National Research Council, 124, 135, 136 Needs. See Business needs Net Promoter Score, 296 Newstrom, J., 49, 67, 68, 77, 115, 232, 245 Newton, Sir I., 326 Newton’s First Law of Motion, 180–181 Nguyen, F., 128, 216, 227 Nielsen, G., 47 Nielsen, J., 229, 230 NOARUS Auto Group, 295 Noel, J., 84 North American Learning Center (Holcim), 238 Norton, D., 286 Notre Dame, 139

O Objectives. See Business objectives; Learning objectives O’Hern, J., 310 O’Leary, R., 112 Olegario, R., 19 O’Neil, J., 129 Ontario Lottery and Gaming (OLG), 323–324 Opinions, 293t, 296–299

index.indd 353

353 Organizations: learning, 46–47, 162–164; training ROI for, 29–30, 61, 94fig, 200–208; training value chain to, 89t, 120fig–124, 122t, 223fig OSRAM, 188 Osterweil, C., 30 Outcomes Planning Wheel: creating value chain using the, 120; description of, 37, 57; evaluation reviewing process of, 280–281fig; four key questions to, 41–45; illustrated diagram of, 40fig; interview guidelines for discovering needs, 38e Outcomes Planning Wheel questions: 1: what business needs will be met?, 41–42, 281fig; 2: what will participants do differently and better?, 42, 281fig; 3: how can we confirm these changes?, 42–44, 281fig; 4: what are the specific measures of success?, 44–45, 281fig Outliers (Gladwell), 93 Overcoming inertia: challenge of, 178, 180–181; essential elements for, 181–189; multi-pronged approach to, 181fig; Newton’s First Law of Motion and, 180–181

P Paine, N., 61 Pallarito, K., 185 Parisi, D., 126, 127 Park, Y., 171 Parry, C., 263 Parskey, P., 117, 118 Participant-Manager Dialogue form, 239fig Participants: accountability of, 185–187, 197fig; collaborative learning by, 195–197, 227–228fig; communicating their training experience, 155–156; community of learners, 195–197, 246–249; evaluation role of, 309–310; having them share their learning goals, 152; improve perceived utility of program to, 118–119; improving the learning invitation to, 76; learning outcomes influenced by expectations of, 72–74fig; making relevance of training clear to, 115–118; paired training ratings from managers and, 243fig; providing know-how to, 119–129; questions for gauging perceptions of, 157e; reaching the finish line, 188–189; self-efficacy of, 176; training satisfaction of, 214–216; WIIFM (“What’s in it for me?”), 158, 237; worksheet linking new and existing knowledge for, 138fig. See also Employees; Learning; Learning transfer; Motivation/motivating Pascal, B., 305 Patterson, K., 34, 132, 134, 152, 186 Patton, M., 302 PDSA (Plan-Do-Study-Act) cycle, 259–260fig Peer support relationships, 195–197, 246–249

6/17/10 3:41:35 PM

354 People support: designated coaches, 249–250; importance to learning, 231–232; instructors and facilitators, 245–246; managers, 232–245; peers of learning communities, 246–249 “Perception of value,” 307 Performance: causes of problems solved without training, 50e; Fifth Discipline as deploying support for, 10fig–11; gap between learning and, 8fig; relative to expectations, 81fig Performance Challenges Corporation, 117 Performance Consulting (Robinson & Robinson), 24 Performance improvement: evaluation step of implementing, 313–314; feedback for, 99; four phases of learning necessary for, 71fig, 72–101; management expectations on training for, 4fig; motivation dependent on beliefs that effort will result in, 113fig; post-course support for, 187–188; providing knowhow for, 119–129; training as raising expectations for, 80fig Performance support: analogy of customer/product support to, 214–216; materials for, 216–219; overview of training and development, 10fig–11, 13fig, 213–214; people role in, 231–252; systems role in, 219–231fig Peterson, B, 47 Pfeffer, J., 264 Pfizer: adding transfer and performance support, 222; Advanced Transition Program, 68, 84, 141; Learning Center, 84, 206; MESA group’s ROI study at, 206; post-course environment created by, 92; study on impact of managers by, 232, 237; 360-degree feedback use by, 288 Phase I (prepare): Checklist for, 104; expectations influencing outcomes, 72–74fig; get everyone on the same page, 85–88; get the right people into the room, 82–85; improve the invitation, 76; manager’s involvement in, 77e; raise expectations, 75–76; redefine the finish line, 80–82; secure management support, 76–79; strengthening phase 1, 74–88 Phase II (learn): Checklist for, 105; critical issues of, 88; strengthening, 89–90; value chain between business outcomes and, 89t Phase III (transfer): accept responsibility for, 95; adopt new systems and approaches, 95; Checklist for, 105; impediments to training transfer, 92t; overview of, 90–91; providing performance support, 96; real work beginning after course ends, 91–93; strengthening, 93–95; treat training as a process, 94; as weakest link in most learning programs, 91fig. See also Learning transfer

index.indd 354

Index Phase IV (achieve): assessment as learning, 98–101; assessment makes expectations explicit, 96–97fig; assessment setting clear goal line, 97–98; reasons for assessment during, 96 Phillips, C., 311 Phillips, J., 49, 92, 171, 206, 258, 283, 308 Phillips, R., 49, 92, 258, 283, 308 Physicians’ Desk Reference, 227 Pink, D., 71, 98, 113, 133 Pipe, P., 49, 97 Plastipak Academy, 32, 87, 237 Plastipak Packaging, 58 Plastipak’s High Impact Learning Job Aid for Supervisors, 141 Platt, L., 227 Poertner, J., 147 Pogo (comic character), 55 Pollock, R., 20, 69, 79, 80, 81, 143, 149, 166, 171, 218, 238, 239, 330 Porter, M., 120 Positioning: The Battle for Your Mind (Ries and Trout), 184 Posner, B., 77, 274, 330, 332 Post-course learning activities, 182–183 Potter, V., 207 Power, M., 133 Practicing: deliberate practice, 331–332; learning, 173–174 Prepare. See Phase 1 (prepare) Preskill, H., 278, 292 Prietula, M., 93, 99, 313, 331 Principles of Workplace Learning report, 138 Procter & Gamble, 19 Product support, 214–216 Prokopeak, M., 55, 234 Propp, J., 147

Q Qualcomm’s Learning Center, 310 Questionnaire or interview bias, 289 Quinn, C., 307

R Rackham, N., 269 Rangan, V., 102 Raynor, M., 35 “Real play,” 125 Rechtman, J., 183 Redding, J., 258 Redford, K., 30, 284 Reed, B., 164 Reichheld, F., 296

6/17/10 3:41:36 PM

Index Relevant evaluation, 268–272, 279t Repetition: learning and forgetting with spaced, 136fig; memory retrieval facilitated by, 134–135 Reporter bias, 290 Research Council, 112 Response bias, 289–290 ResultsEngine system, 189, 195, 203, 230, 301, 310, 330 Retrieval practice, 98 Reuters, 30 Rianoshek, R., 54 Richardson-Green, F., 277 Ries, A., 184 Rip, G., 91, 238 Robinson, D., 24, 42, 57, 245 Robinson, J., 24, 42, 57, 245 Robison, J., 3 Roche, T., 55, 67, 234, 245, 246 ROI (return on investment): BBC lack of training, 61; defining outcomes in terms other than, 29–30; learning must add value and, 256–258; learning transfer management to increase, 200–208; maximizing marketing training, 236; offsetting training cost by improved results for, 94fig. See also Value chain Role play, 125 Role-time transfer matrix, 67t Ross, M., 117 Rossett, A., 216, 217–218, 219, 221, 222 Rothwell, W., 258 Ruona, W., 68, 175, 298 Russ-Eft, D., 278, 292

S Sachs, R., 14, 246, 248 Saks, A., 66, 165, 175, 292 Sales Organization Leadership Development (SOLD) program, 144 San Diego State University, 217 Santana, L., 207 Sarr, F., 299 Saslow, S., 47 Schafer, L., 216, 221, 222 Schaffer, R., 55 Schemenaur, P., 230 Scherkenbach, W., 169 Schrader, D., 271 Science (journal), 300 Scotwork, 223 Second Discipline (D2). See Design the complete experience (D2) Securian Financial Group, 299, 324–325

index.indd 355

355 Segmentation, 307–308 Selection bias, 289 Self-Assessment of Transfer Climate, 178, 179e Self-efficacy, 176 Self-fulfilling prophesy, 266–267 Senge, P., 68 Shapiro, D. A., 23, 102 “Share-of-mind,” 183–184 Sharkey, L., 246–247 Sherwin, R., 9, 69 Shotland, A., 263 Shrock, S., 99, 128 Simons, D., 131 Simulations, 125–127 Singos, M., 87 The Six Ds: brief guide to getting started with, 326e–327e; evaluation confirming that outcomes matter, 280–285; GE (General Electric) use of, 2; illustrated diagram of, 2fig, 13fig; introducing D1 (define business outcomes), 3–5; introducing D2 (design the complete experience), 5–8; introducing D3 (deliver for application), 8–9; introducing D4 (drive learning transfer), 9–10; introducing D5 (deploy performance support), 10fig–11; introducing D6 (document results), 11–12; introduction to each of, 1–13; mind map of, 320fig; reprise of, 319–322. See also specific discipline Six sigma, 260–261, 270 Six Strategic Conclusions (Deloitte U.S. Firm), 85 6Ms (Machine, Man, Measurement, Management, Materials, and Method), 169 Sixth Discipline (D6). See Document results (D6) Skill acquisition measures, 263fig SMART goals, 149–150 Smith, P., 129 Smith, R., 21 SmithKline Beecham Animal Health, 236 Sony Electronics: Integrated Leadership Curriculum, 68; Leadership Forum at, 189; Talent Management Council, 59; teleconference used at, 183 SPIN Selling (Rackham), 269 Sponsors: beginning process of educating, 45–46; business measures valued by, 30; satisfaction with training and development by, 214–216 Sputnik (1957), 171 Standard Chartered Bank, 68, 87–88 “Standards for Educational and Psychological Testing” (NCME), 26 Steelcase University, 277 Stewart, M., 67 Stolovitch, H., 49, 51

6/17/10 3:41:36 PM

356 Stone, R., 283 Store Managers Learning Forums (Home Depot), 225 Storytelling: data collection evidence from, 293t, 299–300; evaluation dissemination using, 276; evaluation success stories, 301; memory retention through, 132–134 Strategy. See Business strategy The Structure of Scientific Revolutions (Kuhn), 67 Sturm, R., 271 Success Case Evaluation Method, 299 Successful Manager’s Handbook (Gebelein & colleagues), 226 Sullivan, B., 234 Sullivan, J., 276, 304 Sun Microsystems: Sun University (Sun U), 200–203; usability of information strategies used at, 230 Support. See Performance support Sutton, R., 264 Sviokla, J., 102 Swanson, R., 29, 35, 90, 115, 119, 124, 222 Sway (Brafman & Brafman), 73 Sweller, J., 128, 216, 227 Swiss Army Brands, Inc., 70 Switzler, A., 34, 134, 152, 186 Systems. See Electronic performance support

T Tacit business knowledge, 36 The Talent Code (Coyle), 93, 174 Talent Is Overrated (Colvin), 93, 173 Talent Management Council (Sony), 59 Tannenbaum, S., 263 Targeting, 307–308 Taxonomy: of cognitive domains, 100t; for educational objectives, 99 TDQAS model, 34 “Tell Me More” link, 230, 231fig Terry, R., 201 Tesch-Romer, C., 93, 173 Thalheimer, W., 98, 136, 262 Tharenou, P., 73 Thiagi (Sivasailam thiagarajan), 127 Third Discipline (D3). See Deliver for application (D3) Thompson, L., 127 Thomson, H., 55 360-feedback process, 247, 288 Timeline from learning to results, 12fig, 263fig Tobin, D., 258 Todd, S., 307 Torres, C., 311 Training: BBC lack of ROI from, 61; causes of poor performance solved without, 50e; “the

index.indd 356

Index course is the thing” paradigm of, 81–82; driving business results using, 295; evaluation isolation of effects of, 291–292; everyone loses in case of failure of, 163fig; making relevant clear to participants, 115–118; mistaken as cure-all, 48, 51; producing meaningful change through, 283–284; securing support from managers for, 76–79 Training Implementation Services, 299 Training magazine, 206 Training Media Review, 189 Training traps: confusing means and ends belief as, 54–56; importance of avoiding, 47–48; inadequate input as, 56–57, 59; laudable intent as, 56; training as cure-all belief as, 48, 51; we need a program belief as, 51–54 Training/development programs: accountability and support as adding value to, 10fig; being better than nothing, 60; business objectives as part of effective, 22fig; business results factors other than, 27fig; contributing to organization by improving individual performance, 223fig; designing the complete experience and, 4–8; improve perceived utility of, 118–119; learning management systems (LMS) for, 191–199fig; learning transfer weakness in most, 9fig–10; management expectations for performance improvement by, 4fig; offsetting cost by improved productivity and results, 94fig; questions for gauging participant perceptions of, 157e; timeline showing learning that resets expectations, 82fig Trainor, D., 84, 92, 206, 222, 232, 288 Transfer Climate Scorecard, 178, 179e Transfer. See Learning transfer Traver, H., 263 Trinka, J., 115–116, 135, 162 Trolley, E., 1, 283 Trout, J., 184 Tucker, R., 87 Tufte, E., 305 Twain, M.., 303

U UBS Bank, 76, 122 Ulrich, D., 54, 78 United Parcel Services (UPS), 167, 206 University of Notre Dame, 139 U.S. Army Center for Lessons Learned, 263 U.S. Bank Wealth Management Group, 85 U.S. Centers for Disease Control, 185 USC’s Leadership Institute, 19 USS Benfold, 57

6/17/10 3:41:37 PM

Index

Valls, J., 236 Value chain: between Phase II exercise and business outcomes, 89t; example of sales management program, 122t; improving individual performance, 223fig; linking learning activities to business outcomes, 120fig–124; maximizing marketing training, 236; Outcomes Planning Wheel used to create, 120. See also ROI (return on investment) Value perceptions, 307 Van Adelsberg, D., 1, 283 Van Boven, L., 127 Verne, J., 171 Verschuren, A., 83 Vigil, R., 39, 261 Voice of the customer, 270–272 Vroom, V., 112

Wexley, K., 144 What CEOs Expect from Corporate Training (Rothwell, Lindholm, & Wallick), 31 Whistle While You Work (Leider & Shapiro), 23 White, E., 31 A Whole New Mind (Pink), 132 Wick, C., 20, 67, 69, 79, 80, 81, 84, 92, 143, 149, 166, 171, 196, 207, 218, 232, 238, 239, 245, 246, 288, 330 Wiczorek, M., 131 Wiggenhorn, W., 25 WIIFM (“What’s in it for me?”), 158, 237 Wilde, K., 307, 315 Willis, M., 229 Work environment, 177 World Wide Web, 229 Wyeth Pharmaceuticals, 203, 289

W

X

Wager, W., 125 Walker, C., 131 Wall, S., 31 The Wall Street Journal, 312 Wallick, W., 258 Watkins, M., 213 Welch, J., 25, 52, 173, 259 Wenger, E., 247

Xerox Corporation, 155–156

V

index.indd 357

357

Y Yale School of Management, 112 Young, W. C., 58

Z Zenger, J., 9, 68

6/17/10 3:41:37 PM

index.indd 358

6/17/10 3:41:37 PM

ABOUT THE AUTHORS

Calhoun Wick is nationally recognized for his work on improving the performance of people and organizations. Cal was named “Thought Leader of the Year” by ISA, the Association of Learning Providers, in 2006. In 2007, Cal received the first-ever Neon Elephant Award. According to Work-Learning Research President Will Thalheimer, “The Neon Elephant Award is awarded to a person, team, or organization exemplifying enlightenment, integrity, and innovation in the field of workplace learning and performance.” Cal earned his master’s degree as an Alfred P. Sloan Fellow at MIT’s Sloan School of Management. He is founder and chairman of the Fort Hill Company and co-author of Getting Your Money’s Worth from Training and Development. His goal is nothing less that to transform the training and development industry. Roy Pollock, DVM, Ph.D., is chief learning officer of the Fort Hill Company. Roy is a popular speaker who has a passion for helping individuals and teams succeed. Prior to joining Fort Hill, Roy served as vice president, Global Strategic Product Development, for SmithKline Beecham Animal Health; vice president, Companion Animal Division, for Pfizer; president of IDEXX Informatics; and president of VetConnect Systems, Inc. Roy received his B.A. from Williams College and his Doctor of Veterinary Medicine and Ph.D. degrees from Cornell University. He has received numerous awards, including the Ralston-Purina Research Award and the American Animal 359

AUTHOR.indd 359

6/17/10 3:20:07 PM

360

About the Authors

Hospital Association’s Veterinarian of the Year. He is a fellow of the Kellogg Foundation National Leadership Program and teaches instructional design, leadership, and marketing at Swamp College in Trumansburg, New York. Andrew Jefferson, J.D., is chief executive officer of the Fort Hill Company and co-author of Getting Your Money’s Worth from Training and Development. He excels in helping companies maximize the value they realize from their investments in learning and development. Andy is an accomplished executive with deep line-management expertise, as well as experience in strategic planning, sales and marketing, productivity, and technology development in both large and small corporate environments. Prior to joining Fort Hill, Andy served as the chief executive officer of Vital Home Services and chief operating officer and general counsel of AmeriStar Technologies, Inc. Andy is a graduate of the University of Delaware and graduated Phi Kappa Phi with honors from the Widener University School of Law, where he currently serves on the school’s Board of Overseers.

About Fort Hill Fort Hill is a consulting, training, and learning technology company that focuses exclusively on helping organizations and individuals put learning to work and prove and improve its impact. We are committed to the proposition that learning creates competitive advantage for individuals and organizations, provided it is properly targeted, supported, and applied. Fort Hill identified the The 6Ds: The Six Disciplines of Breakthrough Learning and developed the first online learning transfer management system. Its followthrough tools have since been used by more than 100,000 participants in over forty-eight countries. Fort Hill also provides advice and consultation on best practices of program design, execution, measurement, follow through, and marketing to help organizations enhance the beneficial impact of their learning and development efforts. Additional information is available at www.forthillcompany.com.

AUTHOR.indd 360

6/17/10 3:20:09 PM

The page represents a continuation of the Copyright page. The following publications have been excerpted with the permission of the publisher: John Wiley & Sons, Inc. The Leadership Challenge (4th ed.), 2007, by J. Kouzes and B. Posner. Defining the D in ISD. Part I: Task-general instruction methods. Performance and Instruction, 25(3), 17–21, 1986, by R. Clark. Getting your money’s worth from training and development: A guide to breakthrough learning for participants, 2009, by A. Jefferson, R. Pollock, and C. Wick. Job aids and performance support: Moving from knowledge in the classroom to knowledge everywhere, 2007, by A. Rossett and L. Schafer. The International Society for Performance Improvement, www.ispi.org, ©2010, All rights reserved. 2010, Award submission guidelines. Undated, What is human performance technology? 2002, Performance technology standards. Berrett-Koehler Publishers, Inc., San Francisco, CA. All rights reserved. www.bkconnection.com. Courageous training: Bold actions for business results, ©2008, by T. Mooney and R. Brinkerhoff (Eds.). Know can do!: Put your know-how into action, ©2007, by K. Blanchard, P. Meyer, and D. Ruhe. Performance consulting: A practical guide for HR and learning professionals, ©2008, by D. Robinson and J. Robinson. Running training like a business: Delivering unmistakable value, ©1999, by D. van Adelsberg and E. Trolley. Courtesy of Agilent Technologies, Inc. Redesigning for enhanced learning transfer. Paper presented October 2009 at the Fort Hill Best Practices Summit, Mendenhall, Pennsylvania, by M. Girone and P. Cage. All illustrations, unless otherwise noted, are copyrighted by Fort Hill Company and are used here with permission. ResultsEngine, DevelopmentEngine, Friday5s, GuideMe and Follow-Through Tools are registered trademarks of the Fort Hill Company.

badvert.indd 1

6/17/10 6:19:29 PM

Pfeiffer Publications Guide This guide is designed to familiarize you with the various types of Pfeiffer publications. The formats section describes the various types of products that we publish; the methodologies section describes the many different ways that content might be provided within a product. We also provide a list of the topic areas in which we publish.

FORMATS In addition to its extensive book-publishing program, Pfeiffer offers content in an array of formats, from fieldbooks for the practitioner to complete, ready-to-use training packages that support group learning.

FIELDBOOK Designed to provide information and guidance to practitioners in the midst of action. Most fieldbooks are companions to another, sometimes earlier, work, from which its ideas are derived; the fieldbook makes practical what was theoretical in the original text. Fieldbooks can certainly be read from cover to cover. More likely, though, you’ll find yourself bouncing around following a particular theme, or dipping in as the mood, and the situation, dictate. HANDBOOK A contributed volume of work on a single topic, comprising an eclectic mix of ideas, case studies, and best practices sourced by practitioners and experts in the field. An editor or team of editors usually is appointed to seek out contributors and to evaluate content for relevance to the topic. Think of a handbook not as a ready-to-eat meal, but as a cookbook of ingredients that enables you to create the most fitting experience for the occasion.

RESOURCE M aterials designed to support group learning. They come in many forms: a complete, ready-to-use exercise (such as a game); a comprehensive resource on one topic (such as conflict management) containing a variety of methods and approaches; or a collection of likeminded activities (such as icebreakers) on multiple subjects and situations. TRAINING PACKAGE An entire, ready-to-use learning program that focuses on a particular topic or skill. All packages comprise a guide for the facilitator/trainer and a workbook for the participants. Some packages are supported with additional media—such as video—or learning aids, instruments, or other devices to help participants understand concepts or practice and develop skills. • Facilitator/trainer’s guide Contains an introduction to the program, advice on how to organize and facilitate the learning event, and step-by-step instructor notes. The guide also contains copies of presentation materials—handouts, presentations, and overhead designs, for example—used in the program.

badvert.indd 2

6/17/10 6:19:31 PM

• Participant’s workbook Contains exercises and reading materials that support the learning goal and serves as a valuable reference and support guide for participants in the weeks and months that follow the learning event. Typically, each participant will require his or her own workbook.

ELECTRONIC CD-ROMs and web-based products transform static Pfeiffer content into dynamic, interactive experiences. Designed to take advantage of the searchability, automation, and ease-of-use that technology provides, our e-products bring convenience and immediate accessibility to your workspace.

METHODOLOGIES CASE STUDY A presentation, in narrative form, of an actual event that has occurred inside an organization. Case studies are not prescriptive, nor are they used to prove a point; they are designed to develop critical analysis and decision-making skills. A case study has a specific time frame, specifies a sequence of events, is narrative in structure, and contains a plot structure— an issue (what should be/have been done?). Use case studies when the goal is to enable participants to apply previously learned theories to the circumstances in the case, decide what is pertinent, identify the real issues, decide what should have been done, and develop a plan of action.

ENERGIZER A short activity that develops readiness for the next session or learning event. Energizers are most commonly used after a break or lunch to stimulate or refocus the group. Many involve some form of physical activity, so they are a useful way to counter post-lunch lethargy. Other uses include transitioning from one topic to another, where “mental” distancing is important. EXPERIENTIAL LEARNING ACTIVITY (ELA) A facilitator-led intervention that moves participants through the learning cycle from experience to application (also known as a Structured Experience). ELAs are carefully thought-out designs in which there is a definite learning purpose and intended outcome. Each step—everything that participants do during the activity— facilitates the accomplishment of the stated goal. Each ELA includes complete instructions for facilitating the intervention and a clear statement of goals, suggested group size and timing, materials required, an explanation of the process, and, where appropriate, possible variations to the activity. (For more detail on Experiential Learning Activities, see the Introduction to the Reference Guide to Handbooks and Annuals, 1999 edition, Pfeiffer, San Francisco.)

badvert.indd 3

6/17/10 6:19:51 PM

GAME A group activity that has the purpose of fostering team spirit and togetherness in addition to the achievement of a pre-stated goal. Usually contrived—undertaking a desert expedition, for example—this type of learning method offers an engaging means for participants to demonstrate and practice business and interpersonal skills. Games are effective for team building and personal development mainly because the goal is subordinate to the process—the means through which participants reach decisions, collaborate, communicate, and generate trust and understanding. Games often engage teams in “friendly” competition.

ICEBREAKER A (usually) short activity designed to help participants overcome initial anxiety in a training session and/or to acquaint the participants with one another. An icebreaker can be a fun activity or can be tied to specific topics or training goals. While a useful tool in itself, the icebreaker comes into its own in situations where tension or resistance exists within a group. INSTRUMENT A device used to assess, appraise, evaluate, describe, classify, and summarize various aspects of human behavior. The term used to describe an instrument depends primarily on its format and purpose. These terms include survey, questionnaire, inventory, diagnostic, survey, and poll. Some uses of instruments include providing instrumental feedback to group members, studying here-and-now processes or functioning within a group, manipulating group composition, and evaluating outcomes of training and other interventions. Instruments are popular in the training and HR field because, in general, more growth can occur if an individual is provided with a method for focusing specifically on his or her own behavior. Instruments also are used to obtain information that will serve as a basis for change and to assist in workforce planning efforts. Paper-and-pencil tests still dominate the instrument landscape with a typical package comprising a facilitator’s guide, which offers advice on administering the instrument and interpreting the collected data, and an initial set of instruments. Additional instruments are available separately. Pfeiffer, though, is investing heavily in e-instruments. Electronic instrumentation provides effortless distribution and, for larger groups particularly, offers advantages over paperand-pencil tests in the time it takes to analyze data and provide feedback. LECTURETTE A short talk that provides an explanation of a principle, model, or process that is pertinent to the participants’ current learning needs. A lecturette is intended to establish a common language bond between the trainer and the participants by providing a mutual frame of reference. Use a lecturette as an introduction to a group activity or event, as an interjection during an event, or as a handout.

badvert.indd 4

6/17/10 6:19:53 PM

MODEL A graphic depiction of a system or process and the relationship among its elements. Models provide a frame of reference and something more tangible, and more easily remembered, than a verbal explanation. They also give participants something to “go on,” enabling them to track their own progress as they experience the dynamics, processes, and relationships being depicted in the model. ROLE PLAY A technique in which people assume a role in a situation/scenario: a customer service rep in an angry-customer exchange, for example. The way in which the role is approached is then discussed and feedback is offered. The role play is often repeated using a different approach and/or incorporating changes made based on feedback received. In other words, role playing is a spontaneous interaction involving realistic behavior under artificial (and safe) conditions. SIMULATION A methodology for understanding the interrelationships among components of a system or process. Simulations differ from games in that they test or use a model that depicts or mirrors some aspect of reality in form, if not necessarily in content. Learning occurs by studying the effects of change on one or more factors of the model. Simulations are commonly used to test hypotheses about what happens in a system—often referred to as “what if?” analysis—or to examine best-case/worst-case scenarios.

THEORY A presentation of an idea from a conjectural perspective. Theories are useful because they encourage us to examine behavior and phenomena through a different lens.

TOPICS The twin goals of providing effective and practical solutions for workforce training and organization development and meeting the educational needs of training and human resource professionals shape Pfeiffer’s publishing program. Core topics include the following: Leadership & Management Communication & Presentation Coaching & Mentoring Training & Development E-Learning Teams & Collaboration OD & Strategic Planning Human Resources Consulting

badvert.indd 5

6/17/10 6:19:53 PM

What will you find on pfeiffer.com? • The best in workplace performance solutions for training and HR professionals • Downloadable training tools, exercises, and content • Web-exclusive offers • Training tips, articles, and news • Seamless on-line ordering • Author guidelines, information on becoming a Pfeiffer Partner, and much more


badvert.indd 6

6/17/10 6:19:54 PM



Praise for the Second Edition of

The Six Disciplines of Breakthrough Learning “No other book in the last decade has been more important for the workplace learning field. The second edition is even better, incorporating new wisdom learned in the crucible of real-world practice.” —WILL THALHEIMER, PH.D., president, Work-Learning Research, Inc.

From left to right: Roy Pollock, Calhoun Wick & Andrew Jefferson Photo by Terence Roberts

THE AUTHORS Calhoun Wick is the founder and chairman of the Fort Hill Company. He is recognized nationally as a consultant, educator, and researcher on improving the performance of managers and organizations. Roy Pollock serves as chief learning officer for Fort Hill. Andrew Jefferson is president and CEO of Fort Hill. They authored the award-winning first edition of The Six Disciplines of Breakthrough Learning as well as Getting Your Money’s Worth from Training and Development. The Fort Hill Company specializes in helping companies earn a better return on their investments in training and development. Over the past ten years, Fort Hill’s 6Ds methodology and learning transfer management systems have helped many of the world’s leading companies ensure that learning gets transferred and results improve. The company is based in Wilmington, Delaware.

“The Six Disciplines brings together many of the most important principles of corporate training in an easy-to-understand, highly visual format. Readers will find this book filled with examples, models, and practical tools you can use to create high-impact learning solutions in your own organization.” —JOSH BERSIN, president and chief executive officer, Bersin & Associates “Wick, Pollock, and Jefferson have a well-proven formula for moving learning from an academic exercise to business results. Their 6Ds offer all leaders specific and concrete things that they can do to turn learning into results.” —DAVE ULRICH, professor, Ross School of Business, University of Michigan, and partner, The RBL Group “I introduced The Six Disciplines to GE’s Global Learning Council and I am very glad that I did. The Six Disciplines gives us a common language and a common process that work very well in GE.” —JAYNE JOHNSON, director, Global Leadership Development, General Electric “The 6Ds model provided a powerful framework for designing high-impact learning programs that helped to shape our approach to learning and to align our learning leaders across the organization. The second edition adds insights and tools that make this edition even more relevant, practical, and valuable.” —ROBERT SACHS, Ph.D., vice president, Learning and Development, Kaiser-Permanente “Embracing the Six Disciplines has allowed us to develop truly innovative high-impact leadership programs that have changed the trajectory of our firm.” —CAROL BONETT, vice president, leadership development officer, Morgan Stanley Smith Barney LLC

ORGANIZATIONAL DEVELOPMENT

The SIX DISCIPLINES of Breakthrough Learning

“Simply put, this book is a critical read for anyone who wants to ensure that the time and money spent in development produces results.” — TERESA ROCHE, vice president and chief learning officer, Agilent Technologies

SIX DISCIPLINES The

Wick Pollock Jefferson

2nd

Edition


SIX DISCIPLINES The


When the first edition of The Six Disciplines of Breakthrough Learning was published, it quickly became the new standard for companies who were serious about increasing the return on their investment in learning and development. This thoroughly revised, updated, and expanded edition of The Six Disciplines of Breakthrough Learning reveals how training, OD, and HR professionals can enhance their contributions to the success of their organizations. The book is filled with practical, proven-to-work guidelines and examples that show clearly how to design, deliver, and document comprehensive learning experiences that improve leadership and management, sales, customer service, marketing, and other business functions. The authors—experts in the field of learning and development as well as business— explain the theories and principles that undergird the 6Ds approach and provide practical methods, tools, roadmaps, and checklists for bridging the “learning-doing” gap. The book’s guidelines are illustrated with dozens of case studies from successful companies on the cutting edge of results-driven educational performance.

2nd

Edition

How to Turn Training and Development into Business Results


Calhoun Wick, Roy Pollock, and Andrew Jefferson

The 6Ds encompass the entire process of converting learning into improved performance, from pre-course communications to the impact of the learning transfer climate. Designed to be used by a broad spectrum of professionals, the book’s knowledge can be applied to virtually any learning and development program. Most important, The Six Disciplines of Breakthrough Learning focuses on business and performance outcomes—the new “finish line” for learning and development.

The Six Disciplines of Breakthrough Learning: Second Edition; How to Turn Training and Development into Business Results (Pfeiffer Essential Resources for Training and HR Professionals)

The Six Disciplines of Breakthrough Learning: How to Turn Training and Development Into Business Results

Engaging Learning: Designing e-Learning Simulation Games (Pfeiffer Essential Resources for Training and HR Professionals)

Tough Training Topics: A Presenter's Survival Guide (Pfeiffer Essential Resources for Training and HR Professionals)

Leveraging the Impact of 360-degree Feedback (Pfeiffer Essential Resources for Training and HR Professionals)

The Facilitator Excellence Handbook (Pfeiffer Essential Resources for Training and HR Professionals)

101 Ways to Improve Customer Service: Training, Tools, Tips, and Techniques (Pfeiffer Essential Resources for Training and HR Professionals)

Diversity Mosaic Participant Workbook: Developing Cultural Competence (Pfeiffer Essential Resources for Training and HR Professionals)

E-Learning Solutions on a Shoestring: Help for the Chronically Underfunded Trainer (Pfeiffer Essential Resources for Training and HR Professionals)

The New Virtual Classroom: Evidence-based Guidelines for Synchronous e-Learning (Pfeiffer Essential Resources for Training and HR Professionals)

Graphics for Learning: Proven Guidelines for Planning, Designing, and Evaluating Visuals in Training Materials (Pfeiffer Essential Resources for Training and HR Professionals)

Developing and Measuring Training the Six Sigma Way: A Business Approach to Training and Development

I'll Take Learning for 500: Using Game Shows to Engage, Motivate, and Train (Pfeiffer Essential Resources for Training and HR Professionals)

Global Training and Development (Training & Development)

HR for Small Business: An Essential Guide for Managers, Human Resources Professionals, and Small Business Owners, Second Edition

Lunch and Learn: Creative and Easy-to-Use Activities for Teams and Work Groups (Pfeiffer Essential Resources for Training and HR Professionals)

Courageous Training: Bold Actions for Business Results

Best of The eLearning Guild's Learning Solutions: Top Articles from the eMagazine's First Five Years (Pfeiffer Essential Resources for Training and HR Professionals)

The Next Generation of Corporate Universities: Innovative Approaches for Developing People and Expanding Organizational Capabilities (Pfeiffer Essential Resources for Training and HR Professionals)

The Ten-Minute Trainer: 150 Ways to Teach it Quick and Make it Stick! (Pfeiffer Essential Resources for Training and HR Professionals)

Turning Training into Learning: How to Design and Deliver Programs that Get Results

Retreats That Work: Everything You Need to Know About Planning and Leading Great Offsites (Pfeiffer Essential Resources for Training and HR Professionals)

Human Resource Development: Learning and Training for Individuals and Organizations

Simulations for Assessment, Training, and Development

The Six Disciplines of Breakthrough Learning: Second Edition; How to Turn Training and Development into Business Results (Pfeiffer Essential Resources for Training and HR Professionals)

The Six Disciplines of Breakthrough Learning: How to Turn Training and Development Into Business Results

Engaging Learning: Designing e-Learning Simulation Games (Pfeiffer Essential Resources for Training and HR Professionals)

Tough Training Topics: A Presenter's Survival Guide (Pfeiffer Essential Resources for Training and HR Professionals)

Leveraging the Impact of 360-degree Feedback (Pfeiffer Essential Resources for Training and HR Professionals)

The Facilitator Excellence Handbook (Pfeiffer Essential Resources for Training and HR Professionals)

101 Ways to Improve Customer Service: Training, Tools, Tips, and Techniques (Pfeiffer Essential Resources for Training and HR Professionals)

Diversity Mosaic Participant Workbook: Developing Cultural Competence (Pfeiffer Essential Resources for Training and HR Professionals)

E-Learning Solutions on a Shoestring: Help for the Chronically Underfunded Trainer (Pfeiffer Essential Resources for Training and HR Professionals)

The New Virtual Classroom: Evidence-based Guidelines for Synchronous e-Learning (Pfeiffer Essential Resources for Training and HR Professionals)

Graphics for Learning: Proven Guidelines for Planning, Designing, and Evaluating Visuals in Training Materials (Pfeiffer Essential Resources for Training and HR Professionals)

Developing and Measuring Training the Six Sigma Way: A Business Approach to Training and Development

I'll Take Learning for 500: Using Game Shows to Engage, Motivate, and Train (Pfeiffer Essential Resources for Training and HR Professionals)

Global Training and Development (Training & Development)

HR for Small Business: An Essential Guide for Managers, Human Resources Professionals, and Small Business Owners, Second Edition

Lunch and Learn: Creative and Easy-to-Use Activities for Teams and Work Groups (Pfeiffer Essential Resources for Training and HR Professionals)

Courageous Training: Bold Actions for Business Results

Best of The eLearning Guild's Learning Solutions: Top Articles from the eMagazine's First Five Years (Pfeiffer Essential Resources for Training and HR Professionals)

The Next Generation of Corporate Universities: Innovative Approaches for Developing People and Expanding Organizational Capabilities (Pfeiffer Essential Resources for Training and HR Professionals)

The Ten-Minute Trainer: 150 Ways to Teach it Quick and Make it Stick! (Pfeiffer Essential Resources for Training and HR Professionals)

Turning Training into Learning: How to Design and Deliver Programs that Get Results

Retreats That Work: Everything You Need to Know About Planning and Leading Great Offsites (Pfeiffer Essential Resources for Training and HR Professionals)

Human Resource Development: Learning and Training for Individuals and Organizations

Simulations for Assessment, Training, and Development

Recommend Documents