Evaluating Career Education and Guidance

Evaluating Career Education and Guidance offers a complete basis for evaluation judgements and decisions. It is an inval...

Author: James Athanasou

34 downloads 1194 Views 1MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Evaluating Career Education and Guidance offers a complete basis for evaluation judgements and decisions. It is an invaluable guide for anyone who needs to decide about the merit or worth of career programs or services. The book is aimed at a wide audience of practitioners, researchers and policy makers. It provides a useful reference and guide for anyone who is required to evaluate a career initiative.

Australian Council for Educational Research

Evaluating Career Education and Guidance

The ECCOES framework— Ethics, Coverage, Costs, Objectives, Effects, and Stakeholders—emphasises the perspectives of key stakeholders in any evaluation. It deals with fundamental issues such as the ethics of a program, the cost-benefits of a service and the extent of coverage of any new initiative. Readers are shown the importance of determining whether a program achieves its objectives and its relative effectiveness. Both qualitative and quantitative approaches are discussed.

James Athanasou

Career education or guidance programs vary in their nature and purpose; each operates in a different context, time and space. Through the use of a synthetic and holistic approach, Evaluating Career Education and Guidance describes a framework developed by the author for judging the merit or worth of career programs, policies, services or initiatives.

James Athanasou

Evaluating Career Education and G uidance

James A Athanasou

EVALUATING CAREER EDUCATION AND G UIDANCE ACER Press

To the Estia Foundation of Australia for disabled children

First published 2007 by ACER Press Australian Council for Educational Research Ltd 19 Prospect Hill Road, Camberwell, Victoria, 3124 Copyright © James Athanasou 2007 All rights reserved. Except under the conditions described in the Copyright Act 1968 of Australia and subsequent amendments, no part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the written permission of the publishers. Edited by Carolyn Glascodine Cover and text design by Divine Design Typeset by Baseline Design Printed by Shannon Books Cover photograph by Purestock/©Getty Images National Library of Australia Cataloguing-in-Publication data: Athanasou, James A. Evaluating career education and guidance. Bibliography. Includes index. ISBN 9780864317971. 1. Vocational education - Evaluation. 2. Vocational guidance - Evaluation. I. Title.

370.113 Visit our website: www.acerpress.com.au

Contents Preface

v

Chapter one What is evaluation?

1

Chapter two What is the background of evaluation?

11

Chapter three Is there a framework for evaluating career education and guidance?

21

Chapter four What information do I need for educational evaluations?

35

Chapter five How do I assess program coverage?

47

Chapter six How do I analyse program costs?

53

Chapter seven How do I describe a program?

63

Chapter eight How do I determine effectiveness?

81

Chapter nine How do I make judgements in educational evaluations?

93

Further reading

101

References

102

About the author

105

Index

106

Foreward

Preface Welcome to the evaluation of career education or career guidance. This is a multifaceted endeavour that addresses real questions about the value of our career programs, policies or services. In this text I examine the application of evaluation to the fields of career education, career guidance, career development, as well as employment programs and services. A general framework for evaluation called the ECCOES model (ethics, costs, coverage, objectives, effects and stakeholders) is offered to you. It is designed to help you make a judgement about the value of a program, policy or service in which you are involved. Although evaluation involves investigation, it differs from research. It is an activity that involves the application of critical thinking to the solution of social, political and administrative problems. More often than not it takes place under conditions, timeframes and restraints that are less than ideal. Right from the outset, I assume that you have expertise in a variety of research methods or that you are prepared to gain the necessary expertise. It is not an onerous task. The idea for this book arose from the need to summarise some ideas on evaluation for my students and other readers. I have tried to emphasise a holistic and synthetic approach to evaluation rather than provide you with a smorgasbord of ideas and methods. It is not a handbook of evaluation methods but a guide to judgement. I hope that you find this guide useful when you come to evaluate careers programs. The content of this book comes from my classes to Masters’ students at the University of Technology, Sydney and some earlier writings on educational evaluation. The career development program at Edith Cowan University has assisted and supported me in the preparation of this manuscript. Readers should feel free to contact me and to make any comments or to point out any errors or omissions. This book developed out of my lecture notes over many years and wherever possible I have tried to provide a reference for my ideas and sources but in some cases I may have omitted specific acknowledgements. I would be grateful if this is drawn to my attention. I wish you well in your efforts to evaluate career education and guidance. James A Athanasou University of Technology, Sydney

Chapter one

What is evaluation?

Most readers would agree that evaluation is an important part of managing that has a place in every field of human activity. It is especially relevant in career education or career guidance in order to ensure the following: the needs of participants are being satisfied; or there are benefits to the community; or the guidance approach is efficient; or the most ethical methods are being used; or that the rights of all interested parties are being maintained. Evaluation is one step to improving career guidance. Its purpose is to provide the information that will help us to make judgements about the appropriateness, effectiveness and efficiency of a program. Many professionals are merely interested in knowing the tricks of the trade in evaluating the contribution of a program to their organisation. Others may want to know the most appropriate ways of justifying their programs, services or policies, especially in times of staff cut-backs or other financial restrictions on services. Some might feel a personal need to evaluate in their field of practice. All of these are worthwhile reasons and linked directly to the purpose of evaluation as the systematic process of investigating the merit or worth of a program, policy, service or initiative in career education or career guidance. So, when I talk about evaluation I mean just the quality and the value. I really mean to ask: ‘Is what I am doing positive or negative, effective or ineffective, worthwhile or not worthwhile, appropriate or not appropriate, ethical or not so ethical, and better or worse than an alternative?’


Evaluation encompasses the word ‘value’. It is sometimes confused with research or inquiry because it certainly involves a variety of research methods, but unlike pure research it has an applied and policy emphasis. The greatest benefit of evaluation is that it helps you to think critically about what you are doing and maybe to look a little more objectively at your programs or services. The skills associated with evaluation are as relevant to your personal life as they are to government policy or community programs – it is really about thinking critically and rationally in order to do what is best. The word ‘evaluate’ comes from the French valuer, meaning to value; and also from the Latin e meaning from and valuere, to be strong and to be of worth. The common-sense meaning of the word is to determine the worth of something or to appraise, in the sense of calculating an amount. In social and educational contexts, evaluation has the formal meaning of determining the value of a program or service, policy or initiative. The United States Joint Committee on Standards for Educational Evaluation defined it as ‘the systematic merit or worth of some object’ (Joint Committee, 1981, 1.12). Evaluation is a natural response to improving any initiative. While informal evaluations occur frequently in the lives of individuals, evaluation as a discipline has achieved prominence as a formal exercise in the area of examining the impact of public programs, policies and initiatives. It is an example of the deeply held view that ‘the methodology of science can be harnessed for the improvement and effective management of social affairs’ (Norris, 1990, p. 11). Some of the basic ideas that supported the practice of evaluation have been summarised in the following way: (a) that research is more authoritative or trustworthy than individuals’ judgements; and (b) that the knowledge of experts will lead to better decisions. This ethos has been described as: a pervasive belief that well-conducted evaluations could, and should, constitute the single most important factor in the rendering of educational decisions. (Popham, 1993, p. 5)

What is evaluation?

The need to evaluate outcomes I have always argued that no program is complete without some formal or informal evaluation. Sometimes claims are made about the effectiveness of a technique and busy practitioners adopt this because it seems to make sense and is considered suitable for their context. There is nothing wrong with implementing reforms but it may also be worthwhile to determine the effectiveness of such approaches prior to wide-scale applications. You need to know whether the approaches you are using are more effective than other approaches and/or whether they are providing some unintended or placebo effect for participants (e.g., social enjoyment, escape from work, morale, group cohesion). You may have concerns about how long-lasting are the benefits of any service or, on the other hand, your programs may be designed to bring about longer-term changes. At the very least you should be able to justify to yourself the ethical aspect as well as the cost benefits (in both dollar, environmental, human and social terms) together with the effectiveness and efficiency of your efforts.

Career education and guidance programs Now this book is unashamedly about career education and guidance programs because this is the area in which I have practised since 1978 but I have worked and taught in other areas of evaluation and the ideas are widely applicable to other educational, social and business areas. The framework that I will outline can be transported across contexts. The evaluation of career education and guidance comes under the broad umbrella of educational evaluation. Some readers with a background in other disciplines, say psychology, might also classify career guidance under the heading of counselling psychology and that is fine for me. The principles are the same. In referring to career education and guidance I would include the following: group counselling, computer-assisted guidance, one-to-one counselling services, career advisory services, telephone career information and


advice, employment services, career information, recruitment programs, supported employment programs, human resource development programs, outplacement services, rehabilitation programs, vocational assessment services, and vocational guidance. You can see that I will take a broad view.

Evaluating the scope of programs or services Evaluations can go beyond looking at outcomes to considering whether the aims of the program are worthwhile. Figure 1.1 summarises the general scope of program evaluation. For instance, you may have an effective and cost-efficient guidance service but one which is not relevant to the aims of the organisation. It is all very well to achieve an objective, but what if the objective is not really that important. For this reason, I encourage evaluators to consider the needs and views of all stakeholders in their judgements and analyses. It is unlikely, however, that any evaluation will address all the needs of groups with a vested interest in a program.

Purposes of evaluations • to improve • for decisions • as a response to controversy • to compare

Evaluations

• to judge

The objects of evaluations • programs • policies

• as a management requirement

• initiatives

• to determine effectiveness • to assess quality • to determine performance

Figure 1.1. The general scope of educational evaluations (Athanasou, 2000)

What is evaluation?

In addition to considering whether outcomes were achieved, whether the aims are appropriate and whether needs are fulfilled, evaluators are also encouraged to consider by whom the results of their evaluations will be used. Will the evaluation be conducted with some cost and effort only to have the results gather dust on some desk or find obscurity in the ‘too hard’ section of a filing cabinet? Career practitioners are encouraged to have regard for their evaluation audience, the sponsors of the program but especially the types of information needed by the decision makers. And a final point, sometimes a formal evaluation of a program or policy is not required because the answer is obvious.

Summary It is not the purpose of this chapter to outline all possible evaluation techniques. In fact there are more than 50 techniques, evaluation philosophies or approaches from which you could choose. There is even an encyclopaedia of educational evaluation. Some approaches are more suited to careers practice than others; some are suited to education, commerce, industry or government; others are useful for program evaluations; and some are related directly to the community. I think that all of them can contribute to your thinking and that they have added to the different ways in which you can go about evaluating services. In particular, as an educator I am proud of the fact that the evaluation of programs is a product of the early work in education, such as the curriculum work of Tyler (1950). In Australia and the United States of America, the evaluation of some health, education and welfare programs is now legislated. Unfortunately many evaluations are piecemeal and wishy-washy efforts to substantiate a service. We can do much better than this and we have available a wide range of research, statistical, economic and educational approaches to assist in evaluations. The conscientious professional can set in place a program of evaluation starting with the collection of routinely available data. My recommendation is that you focus on the qualitative or descriptive analyses of your efforts. If


these are conducted continuously and taken from various vantage points in your organisation, then you will have an effective means of substantiating your claims about a program. You will also have the necessary ammunition to make reforms. Ideally evaluation is an informative and ongoing process designed to improve human initiatives. While program evaluations are not straightforward, they are not really that difficult to implement. I encourage you to develop skills in this area in order to be able to determine what works and what is not helpful to you. The benefits for you are that evaluations can: • identify needs to be fulfilled • help you choose from different options • enable you to follow the progress of a program • allow you to monitor the personal development of participants • enable you to critically analyse proposals and ideas • focus your efforts on outcomes and criteria • ensure that ethical criteria are satisfied • identify the costs and benefits of a program for you. The next chapter introduces some details of the background of evaluation and its historical development. It offers an introduction to the field. I hope that you find this background of some interest. -oOo-

REVIEW QUESTIONS Here are some questions for you to consider. Do not overlook them the way I do and rush on to the next chapter. They are meant to complement your reading and allow you the chance to apply some of the ideas you have encountered. 1 Why should program evaluation be applied routinely in a careers context? 2 Should evaluation always involve comparisons? Can you think of arguments in favour of, or opposed to, comparative evaluation in a program, policy or service in which you are involved or are likely to be involved?

What is evaluation?

3 A joint committee on standards for evaluation defined it as ‘the systematic merit or worth of some object’ (Joint Committee, 1981, 1.12). Can this definition be applied to your workplace? 4 Evaluation can serve a variety of functions:

(a) a formative function for the improvement of a program or activity, or

(b) a summative function for accountability or accreditation.

If you have not come across these terms, then the following brief definitions may be helpful: formative refers to evaluations of activities that are ongoing and still capable of modification; summative refers to the evaluations of completed activities. Considering the above functions, what would be the most important purposes for evaluation in your work context? What do you think should be the main reason for an evaluation in a program, project or service in which you have an interest?


CASE STUDY Here is a case study for you to consider. It describes a program of online support for students. Read the abstract and then consider some issues. Please note that the abstract is only a brief summary and may not include all the details of the full paper. Work Integrated Learning (previously known as Cooperative Education) is a valued component in undergraduate degrees delivered through the Business Portfolio at RMIT in Victoria. After completing two years at university, students move into workplaces as paid employees for a year before returning to RMIT to complete their degree. There have been recent changes in the funding of work industry placements with the Higher Education Support Act 2003 specifying that higher education providers must ‘provide support’ in order to charge a tuition fee for the placement year. As a result, the concept of support has widened from preparatory job-seeking, resumé writing and interview skills to encompass the student experience while on placement and to assist transition back to the university environment. Online resources have been developed to provide support for completing the required assessment tasks in a workplace setting. Source: Rizzetti, J. & Smith, S. (2005), WIL-ling and able: Online support for students moving between work integrated learning experience and the university. In Searle, J., Beven, F. & Roebuck, D. (Eds.), Vocational learning:Transitions, partnerships and sustainable futures (Vol. 2), pp. 124–130, Centre for Learning Research, Griffith University.

What is evaluation?

My questions to you are: • Is this a program, policy, service or initiative? • Is it important to evaluate Work Integrated Learning? • Is it important to evaluate online support for Work Integrated Learning? • What is the key evaluation question to be answered? • What factors would need to be considered in an evaluation? • How could Work Integrated Learning have been designed so that evaluation was ‘built-in’? • How could the online support for Work Integrated Learning be designed so that the evaluation is ‘built-in’? • What is your view of Work Integrated Learning? • From what you have read and can infer, what is your view of the online support for Work Integrated Learning? • Is online support for Work Integrated Learning just a cynical effort to maintain federal funds? • Is the description too specific to an institution? • Are there lessons that can be generalised to other forms of cooperative education and career education? • What evidence would be necessary to suggest that Work Integrated Learning was not worthwhile? • What evidence, if any, is provided? • What evidence would be sufficient for you to say it is worthwhile?

C h a p t e r t wo

What is the background of evaluation?

Evaluation is a common experience for program providers and is now a component of the funding for many human services. For instance, a careers teacher in a country college may want to compare a newly developed, selfdirected career development program for students with previous career education instruction. An adult migrant education service dealing with people from a professional background may be asked to justify the effectiveness of its work placement program. A trainer for the supported employment placement of persons with a developmental disability wants to determine the costs and benefits of direct instruction on job performance. A local employment network may want to review the scope and influence of its community program in a region. A firm of consultants wants to document the value of its computerassisted guidance program for senior high school students using case studies. A government department calls for a report on the effects of a labour market program. All of these scenarios have something in common. There is a need to evaluate programs in order to determine whether they are worthwhile. How should you go about evaluating such programs? What issues should you consider? Are there standard techniques for evaluations? Are there any guidelines in designing an evaluation?

11


This chapter1 provides some answers by sketching a historical overview of some of the issues in educational evaluation. The development of educational evaluation as a field of practice is outlined and a survey of some evaluation approaches is provided (based on Popham, 1993, Chapter 2). I hope that you come away with the idea that evaluation approaches are continually evolving and that many of your own concerns about how to evaluate have already been addressed by the earlier developments in this field. Before considering the background and development of evaluation, it may help to clarify the use of some terms. New program evaluation standards have been developed and replace the 1981 standards (Joint Committee, 1994) and for the sake of consistency, I would like to use the definitions provided by this Joint Committee in the latest standards (1994, p. 3). Evaluation: the systematic investigation of the worth or merit of an object Object: the object of evaluation, namely educational and training programs, projects and materials Program: educational activities that are provided on a continuing basis Project: educational activities that are provided for a defined period of time Client: the evaluation contractor Stakeholder: individuals or groups that may be involved in or affected by a program evaluation Meta-evaluation: an evaluation of an evaluation.

1

An earlier version of this chapter appeared in Athanasou, J. A. (1995). Issues in the evaluation of adult education. In G. Foley (Ed.), Understanding adult education and training (pp. 75–88), Sydney: Allen & Unwin, and Athanasou, J. A. (2000), Evaluating adult education and training. In G. Foley (Ed.), Understanding adult education and training (2nd ed.), (pp. 81–93). Sydney: Allen & Unwin.

12


The background of evaluation as a discipline The early history of formal program provision was not characterised by the need for evaluations as you know them today. Prior to this century there were social ideas of what was best in education. I imagine that there were fairly fixed, common-sense and a priori views about how people responded to programs. These social ideas and common-sense views together with political beliefs influenced the development of services in education and welfare. Accordingly, there was little demand for formal evaluations since pre-determined values and perspectives largely influenced the delivery of educational services. The theory and practice of evaluation emerged in the United States of America, mainly in response to the development of educational initiatives. For example, many existing ideas on education were challenged by educational reformers such as John Dewey (1916). Educational evaluation has now developed to the point where it is a field of study in its own right.

Objectives and goals as a basis for evaluation A change in the concept of evaluation from a priori ideas about what was good or best was influenced by Ralph Tyler with his comparison of progressive versus conventional schooling. Tyler was invited to direct the evaluation staff of the Eight Year Study (1932–40), which arose from concerns that some universities were not admitting high school graduates from progressive schools. These were schools that were developing student-centred education more closely linked to the interests of their students, without regard to standardised curricula. Tyler was among the first to apply evaluation to programs rather than to the mere assessment of people. At the time, this was a significant development in evaluation approaches. A program would now be judged successful according to the extent to which the objectives had been met. Tyler (1950) stated these objectives in observable terms and developed measures or collected data in order to make recommendations concerning the program goals.

13


The objectives model of evaluation offered a rational approach to program evaluation. While it was suited to some programs with very specific outcomes, it was unsuitable as an all-purpose evaluation model. Michael Scriven, who is a philosopher of evaluation, emphasised the importance of rational (scientific) judgement in his paper The Methodology of Evaluation. Scriven (1967) argued that another failing of the objectives model was that there were no procedures for judging the worth of the goals themselves: ‘… it is obvious that if the goals aren’t worth achieving then it is uninteresting how well they are achieved’. He distinguished the formative and summative roles of educational evaluation. A formative evaluation is conducted during a program in order to bring about changes, if required; a summative evaluation summarises evidence and occurs mainly at the conclusion of a program.

Research and educational experiments as a basis for evaluation In the United States of America, a major impetus for the role of evaluation was the Elementary and Secondary Education Act (1965). This was among the first pieces of legislation that required the compulsory evaluation of projects, programs and instructional materials. The concern was whether local agencies utilised funds properly and at that time evaluation was based on research methods and educational testing (see Norris, 1990). The favoured approach to evaluation was some type of comparison such as the educational testing of an experimental and a control group. It reflected an empirical approach based on methods that could ideally be replicated by others. Evaluation projects were treated as social experiments with tests of hypotheses but this type of design was not always capable of dealing with the complexity and specificity of learning. As a result, many of these evaluations were largely unhelpful. These mandatory requirements created a demand for evaluation that was far in excess of supply. It led eventually to dissatisfaction with so-called

14


scientific attempts to evaluate broad educational programs. In 1967, Robert Stake indicated the disparity between the purpose of the legislation and the inability of research or testing to produce the desired evaluation outcomes (Stake, 1967a, b). The field of evaluation diversified beyond educational research to such an extent that it is now difficult to characterise. Evaluation is an applied area of methodology that draws inter alia upon disciplines such as psychology, psychometrics, sociology, epidemiology, demography, ethnography, economics and business disciplines. There are some similarities between research and evaluation. They both use disciplined inquiry, assessment methods and data analysis. There are also distinctions between research and evaluation. These centre on the purpose of the activity. The focus of research is the discovery of phenomena for their own sake whereas evaluation focuses on the application of knowledge to decisions, programs and policies. The generalisability of research and evaluation findings also differs. In evaluation studies, there is no intention that results will necessarily generalise to other situations. The importance of the establishing of value or worth has been noted as a key feature of evaluation especially when compared to pure research. One aspect not often emphasised is the extent to which researchers work within abstract constructs and theoretical frameworks whereas evaluation has a more pragmatic focus.

Judgement as a basis for evaluation Judgemental models of evaluation were the next phase after the objectives, the educational testing and educational research orientations. These contributed to the further development of evaluation and moved it on from the objectives-based or effectiveness approaches to focus on merit and worth, while recognising that the outcomes of programs were multi-dimensional. In a 1963 paper Course improvement through evaluation, Cronbach argued that evaluation should deal less with program comparison than with the extent to which a program promoted its desired consequences. Cronbach made

15


other points which are still relevant to evaluators, namely: (a) that there are links between evaluation and decision making; (b) that evaluation performs a number of functions such as curriculum improvement or the review of policies; (c) that follow-up or longitudinal studies are extremely useful in determining the lasting effects of educational programs; (d) that learner performance and/or experimental studies should not be the only focus and method of evaluation; and (e) that measurement approaches to evaluation should be broadened to include the sampling of specific instances and matrix sampling that avoids the need to assess every outcome with every person. Other perspectives on evaluation built on Cronbach’s insights (see Scriven, 1991). A form of goal-free evaluation was advocated (see Popham, 1993, Chapter 3 for a summary). In this approach the evaluator recorded the major effects of a program, which could then be compared with the original purpose of the program. It was a significant learner-centred innovation that acknowledged the unplanned results of education. Performance data were gathered to establish the merit or worth of educational activities, and the evaluator was responsible for judging the merit of the goals. The strength of this approach was that it encouraged educators to systematically investigate educational processes and to critically examine the value of educational goals.

User-oriented approaches as the basis for evaluations In an article ‘The countenance of educational evaluation’, Stake (1967a) went on to address the important question of what kind of evidence an evaluator should collect. This was a user-oriented approach. It involved (a) the description of a program; (b) reporting the description to groups; (c) obtaining and analysing their judgements; and (d) reporting the analysed judgements back to the audiences. User-oriented approaches can be said to reflect the beginning of more descriptive approaches to evaluations.

16


Decisions as the basis for evaluation Decision-management approaches to evaluation offered a third dimension of evaluation in addition to objectives-based and judgemental approaches. To Stufflebeam (1971) the evaluation of a program was linked to decision making about four dimensions of the program: Context (information about the aims, desired outcomes), Input (the resources and materials), Process (procedures and strategies used) and Product (the achievements). This approach encouraged educators to systematically generate criteria by which educational programs could be evaluated: context criteria for planning; input criteria for programming; process criteria for implementation; and product criteria for decisions. The value of this model lay in its recognition of the need for holistic evaluation. A program needed to be evaluated in its entirety and through its different phases. Objectives-based, judgemental and decision-making models are only some of the evaluation approaches but they were important in the development of evaluation as a discipline. They give you an indication that previous generations of evaluators have considered program objectives; they have tried comparisons between programs; and they have analysed programs from the perspectives of decision makers or users but no single approach per se has been sufficient for educational evaluations. Although these approaches contributed to an awareness of the merit of programs, they did not always provide the comprehensive basis required for program evaluation.

Summary There are now many ways in which educational evaluations can be approached, including: • objectives-based evaluation (Tyler, 1950) • evaluation for decision making (Cronbach, 1963) • context-input-process-product (Stufflebeam, 1971) • community impact of a program (Stake, 1967a, b) • cost-benefit analysis (Levin, 1983)

17


• evaluation as a judgement of worth and merit (Scriven, 1967) • outcome or output-oriented models (Kirkpatrick, 1975) • illuminative approaches (Parlett & Dearden, 1977). Such a large number of evaluation models is a positive sign in the sense that it reflects the diversity of the contexts in which we work and the variety of views we hold about education. The complexity of career issues and their uniqueness is such that cookbook approaches to evaluation are not always helpful. It is doubtful that one approach is sufficient for all circumstances and a general strategy is required. Experience in evaluations and research quickly reveals that these activities can be highly subjective, political and ideological. The orientation taken in this text is towards an integrated and multidimensional view of evaluation where information from various sources is combined. This approach is more likely than others to capture the complexity of career education processes and guidance services. Another aspect that is relevant is the fact that many evaluation approaches seem to consider issues of effectiveness, usefulness and efficiency from the perspective of an administrator or an organisation rather than from the perspective of the consumers of services. The standards which an educator uses to judge a program may emphasise completely different aspects in the values or the merit of some policy, program or initiative. -oOo-

REVIEW QUESTIONS 1 What major elements of the different models would you emphasise in your evaluation work? 2 Describe/define these models of evaluation briefly:

(a) educational research model of evaluation

(b) objectives-based evaluation

(c) goal-free evaluation

(d) judgemental model of evaluation.

18


CASE STUDY Here is another case study for you to consider. It describes a career education program for gifted students. Read the abstract and then consider some issues. Please note that the abstract is only a brief summary and may not include all the details of the full paper. This career education program was developed for gifted high school students in a New South Wales selective high school. This program replaced a previous approach, which was demand-responsive, generic in approach, and not aligned with student needs. A needs analysis of the school and formative evaluations (through a survey of participants, participant observation, stakeholder input and outcome reviews) were undertaken to ensure a successful program re-development. These evaluations resulted in the following changes: earlier and more flexible career awareness opportunities; increased scope for values clarification; inclusion of psychological, psychocreative and social elements; integration of career education and student welfare; lifelong learning emphases; aspiration enhancement for particular students; strategy employment for females; and an expansion in community learning opportunities. The implications for school personnel, students and parents are considered with the aim to deliver a ‘best practice’ career education program for gifted high school students. Source: Boyd, G., Hemmings, B., & Braggett, E. (2000). The development of a career education program for gifted high school students. Paper presented at the Australian Association for Research in Education. http://www.aare.edu.au/00pap/hem00464.htm. Retrieved 16 November 2006.

19


My questions to you are: • What is your view of the career education program? • From what you have read and can infer, what is your view of the responsive and generic approach? • Is the description too specific to an institution? • Are there lessons that can be generalised to other forms of career education? • What evidence would be necessary to suggest that the responsive and generic approach was not worthwhile? • What evidence, if any, is provided? • What evidence would be sufficient for you to say it is or is not worthwhile?

20

Chapter three

Is there a framework for evaluating career education and guidance?

Although people pride themselves on the ability to act rationally, it is not always the case that social programs or educational services are reviewed adequately. As far back as 1987, Fitz-Gibbon and Morris highlighted the neglect of evaluation. They referred to a review of 2000 projects in which there was no acceptable evidence regarding project success or failure (Fitz-Gibbon & Morris, 1987, p. 12). Unfortunately, there is a common view that almost any educational program is beneficial. Well it might be but it will be more certain when you determine standards for the merit or worth of a program. It is not the case that people are reluctant to evaluate but what are some of the barriers? Some of the reasons have centred on a lack of standards, the difficulty in measuring learning, insufficient staff, difficulties in quantifying or measuring, as well as a lack of expertise in evaluation methods (Morris, 1994). In this chapter, I outline a general framework for evaluating educational programs.1 1

An earlier version of this chapter appeared originally in Athanasou, J. A., 1997, ‘Testing competency in a high tech learning environment’, International Quality & Productivity Centre (ed.) Technology-Assisted Learning Conference, Sydney and also in Athanasou, J. A. (1998a), A framework for evaluating the effectiveness of technology-assisted learning, Industrial and Commercial Training 30(3), 96–103, as well as Athanasou, J. A. (2000), Evaluating adult education and training. In G. Foley (Ed.), Understanding adult education and training (2nd ed.), (pp. 81–93). Sydney: Allen & Unwin.

21


Evaluation as a way of deciding worth and merit The introduction of any program or service reflects a personal, social and organisational investment that needs to be reviewed. How can one be a professional if there is no built-in monitoring or some form of program evaluation? Without this, you can end up with sloppy thinking, with poor judgements about the efficacy of services and below average performance. Evaluation can serve as a useful component of all programs and services, ideally in a formative sense to make amendments and improvements as well as a general monitoring role. In a field such as career education or career guidance, which has a proud tradition of theory and practice, there could be nothing more consistent with professionalism than to mandate the evaluation of major programs and services. As you saw in the previous chapter, evaluation was developed largely by educators and has an applied emphasis. It has been refined over many years to become a separate discipline and area of study in its own right, growing from developments in social science research methodologies and the healthy respect that we have for evidence and facts. For the purposes of this section I am concerned with public, structured, formal and summative evaluations and I shall focus on the key steps for evaluating a program or service, mainly in the area of career education and guidance. The aim of this section is to outline a holistic or synthetic approach that you can use to evaluate educational programs (see Figure 3.1).

22


Ethics

Stakeholders

Coverage

Effects

Costs

Objectives

Figure 3.1. The ECCOES model as a holistic evaluation framework for career education and guidance I do not dwell on the specific plans or procedures (i.e., methodology) in this section but focus on six key issues. Some of the examples that I am using will come from various fields of education or training but the ideas can readily be transferred to all areas of learning within career education and guidance.

Steps in an evaluation of a career education or career guidance program Let me now direct your attention to the criteria that I think should be used in an evaluation. An attempt has been made to provide you with a framework and a structure for your decision making. Six steps are summarised by the ECCOES acronym (ethics, costs, coverage, objectives, effects and stakeholders) and I have called this a holistic approach to evaluation. It is up to you, however, to decide how to combine the answers to these six pieces of information in deciding upon the merit or worthiness of any program or service.

23


Step 1 – Is the program or service ethical? This is the first and foremost question prior to any program evaluation. It takes into account questions such as: • ethics/morality • trust • conflicts of interest • social and political implications • legality or vested interests • any harmful ecological issues • any impact on privacy or confidentiality • any abuse of privilege • whether human ethical guidelines have been satisfied. You may wish to question the ways in which programs are applied or the rationale for services or policies. The latter should probably involve valid social, economic, vocational or pedagogic reasons. To my mind the issue of ethics is paramount; there can be no merit or worth in any program or service if ethical aspects are transgressed. I would not presume to tell you what is appropriate but your own professional ethics and conscience might serve as a starting point.

Step 2 – To what extent do the programs or services cover those who are most in need? The second issue here is the coverage of the program (e.g., coverage efficiency, specificity and sensitivity). While it is important to establish that large numbers of people are capable of being served, it is also important to relate it to questions of: • the incidence of the educational or career need • the prevalence of the educational or career need • the sensitivity of the program • the specificity of the career education or career guidance • the attendance and completion rates for any course. The issue of program coverage is dealt with in greater detail in Chapter 5.

24


Step 3 – What are the costs, benefits and utilities of the program or service? Most education and guidance programs are labour-intensive in delivery. My initial estimates of the general break-even cost level of face-to-face career guidance (without additional resources) was around A$85 in 2006 per person per hour including on-costs but excluding overheads and capital costs. This can be compared with more flexible forms of delivery where the bulk of the costs are transferred to the production stages rather than to delivery. It is a helpful first step to take into account all the likely costs associated with the production and delivery of any service. These include salaries, on-costs, participant wages, rental and all the other overheads that would be required if the program was to be developed or offered independently or externally. Many development and delivery costs are buried or cross-subsidised and the real expenses are hardly ever considered. See Chapter 6 for a discussion of cost issues.

Step 4 – Did the program or service achieve its key objective(s)? The focus of this question is to concentrate attention on the key objectives of your program. To what extent were you able to achieve these outcomes? Here I am focusing on the gross outcomes or effects of the program. A peculiar difficulty in evaluations is separating out the short- from the long-term effects of career education. You may have the experience that participants evaluate some programs positively but the program has few longterm effects while other programs might not be evaluated well in the shortterm but have significant long-term effects. Moreover, the placebo effect of any program should not be overlooked. A multi-dimensional approach that offers a straightforward, pragmatic and initial basis for an educational evaluation is Kirkpatrick’s hierarchy of evaluation (Kirkpatrick, 1996). This is a comprehensive approach to evaluation that has been around for many years and which is popular in adult education and training. It focuses on four levels of program objectives and is illustrated

25


in Figure 3.2. It is especially suited to training contexts where specific outputs are of interest right from the outset. It is a pity, therefore, that the four levels of program outcome are not used more often.

Results

Behaviour Learning Reaction

Figure 3.2. An evaluation hierarchy (Athanasou, 2000) 1 Reaction – program participants’ estimates of satisfaction provide an immediate level of evaluation 2 Learning – the extent of learning (i.e., skill acquisition, attitude change) that has been achieved is also of interest 3 Behaviour – the extent to which learning has generalised to the work situation or there has been a transfer of skills is assessed at this level through follow-up after a program 4 Results – the wider impact of a program in the community or the organisation is observed at this level in the hierarchy. Most people undertake the evaluations of program objectives only at Level 1 through the use of end-of-course or follow-up questionnaires. This is helpful but is only part of the picture. At the very least, the questionnaires need to be standardised, comparable with other databases of results, anonymous and processed independently. My opinion is that all course participants should be assessed at Level 1. Depending on the size of the group, course participants could evaluate either (a) the quality of the delivery; (b) the curriculum; or (c) the overall course. Further random samples should be taken to assess the

26


effect of training at Level 2 (say 80 per cent of course participants over time where numbers are less than 300), with smaller group samples at Levels 3 and 4. In conducting these surveys I recommend the benefits of matrix sampling (i.e., asking different questions to individuals and combining results for the group) rather than asking every person every question. The simplicity of the Kirkpatrick model masks some real complexities in the relationships between reaction, learning, behaviour and results. You should also be aware that there are documented criticisms of this approach relating to the validity of the hierarchy and the relationship between the levels (e.g., Tamkin, Yarnall & Kerrin, 2002) but it has served as an influential catalyst for training evaluation.

Step 5 – What is the net effect of the program or service? A challenge for any evaluator is to determine the effectiveness of his or her program or service and to contrast this with comparable programs. This means that you need to be able to describe the impact of a program, its significance and the size of any learning effect. Firstly, the net outcome must be established. You also might want to analyse the differential impact that a program is having for various groups and some analytical framework is helpful. I would refer you to almost any textbook on research designs for some help in selecting the most appropriate method of analysis or design. It is difficult to make any statement about value, however, without a benchmark or some basis for comparison. This is because program evaluation is largely a relative process.

Step 6 – To what extent have the perspectives or interests of all stakeholders been considered and met? Programs and services may inadvertently overlook the interests or needs of users. Consulting clients and obtaining feedback makes you aware of factors that you may not have considered important. Stakeholder perspectives are many and varied (e.g., participants, staff, management, community, government) and some of the questions that you may wish to consider could include:

27


• Does the program, policy or initiative meet the needs of users? • Are intended outcomes useful? • How do people feel about the program? • Have all the key stakeholders been consulted? • What are the perceived costs and benefits of the program? • Have all the stakeholders been treated fairly? • What is important to stakeholders in any evaluation findings of the program or service?

Summary These six questions comprise ethics, coverage, cost, objectives, effect and stakeholder issues. The approach is holistic. These issues synthesise the educational evaluation process for me and I hope that they also provide a concise but comprehensive framework within which you can operate. I think that they cover the most important issues when you are ready to undertake an evaluation of any program or service. The six aspects of the ECCOES model and some of the criteria are summarised in the checklist in Table 3.1. These steps, however, do not provide an automatic answer. The results form the basis for a decision about the worthiness of the program or service. With all the information from these six steps, you then have to decide whether the program has merit. In some cases a program may be costly but very effective; or it may be effective in the short-term and wellliked but may not be meeting long-term needs; or it may be less than perfectly ethical but have wide coverage and popularity; and so on. You will bring to the evidence a set of priorities, which must ultimately influence any decision. That is fine and it is exactly the way it should be – there is no right answer. Some criteria will be more relevant to you than others but all need to be weighed appropriately for your decision. Such decisions involve complex judgements about each career program or service. Maybe the best that you can hope for is to provide information that confirms your ideas about the potential of a program or that refutes claims about the value of a program. In this way evaluations will add to our store of knowledge about what works and what does not work in education and training and how well it works. 28


Table 3.1. Checklist for holistic evaluation of programs Steps and criteria – ✓or ✗ Step 1 – Is the program or service ethical? • • • • • • • • •

ethics/morality trust conflicts of interest social and political implications legality or vested interests any harmful ecological issues any impact on privacy or confidentiality any abuse of privilege whether human ethical guidelines have been satisfied.

Step 2 – To what extent does the program or service cover those who are most in need? • • • • •

the incidence of the educational or career need the prevalence of the educational or career need the sensitivity of the program the specificity of the career education or career guidance the attendance and completion rates

Step 3 – What are the costs, benefits and utilities of the program or service? • • • •

cost feasibility cost-effectiveness cost utility cost benefit

Step 4 – Did the program or service achieve its key objective(s)? • • • •

reaction learning behaviour results

Step 5 – What is the net effect of the program or service? • comparison with other programs • determine statistical effect size (if relevant)

Step 6 – To what extent have the perspectives or interests of all stakeholders been considered and met? • • • • • • •

does the program, policy or initiative meet the needs of users? are the intended outcomes useful? how do people feel about the program? have all the key stakeholders been consulted? what are the perceived costs and benefits of the program? have all the stakeholders been treated fairly? what is important to stakeholders in any evaluation findings of the program or service?

Evaluating Career Education and Guidance © James Athanasou 2007

29


To return to the perceived barriers for evaluation, firstly it is clear that evaluation does require expertise but it can be acquired. It is not overly complex and is an inter-disciplinary field in which I would encourage your greater involvement (e.g., through your own profession or specialist organisations such as the Australasian Evaluation Society). Secondly, many of the difficulties in evaluation, such as the lack of standards or the difficulty in isolating behaviours often relate to the ways in which programs were established and conceived at the outset. You may need to set your own standards and define the behaviours that you consider most important. Thirdly, the difficulties in measurement or assessment can be overcome by having well-stated and specific learning outcomes which are few in number. Qualitative outcomes can be described using case studies that yield valuable descriptions. Fourthly, the issue of insufficient staff might be considered from the viewpoint of building evaluation into every service component from the outset. The easiest evaluations to conduct are those in organisations that keep excellent records or have set up mechanisms for program monitoring. It is not possible to make career education or career guidance perfect but over time it is possible to reduce errors through evaluations and to improve quality. -oOo-

REVIEW QUESTIONS 1 Early approaches to evaluation focused only on the results or outcomes. The scope of evaluation studies has been extended in recent years to goals, design, implementation process, outcomes (such as Stufflebeam’s (1971) CIPP model). What kinds of information do you think should be collected in an evaluation of a program or service in which you are involved? 2 The achievement of important goals is one possible basis for evaluation criteria. Other criteria include needs of actual and potential clients, ideals, social values, known standards set by experts, or the quality of alternatives. What criteria should be used to judge the merit and worth of a program or service?

30


3 Show how an evaluator might select criteria for educational programs that are easily assessed but miss the central point of the program. 4 List the stakeholder groups for an educational program in which you might have an interest. 5 Consider information that might reflect the quality of an educational program. Arrange the information in two columns (a) factors that might show the program is effective and (b) factors that might show the program is not effective. 6 Consider the Kirkpatrick approach to the evaluation of training. Indicate its strengths and weaknesses for use in career programs and services.

31


CASE STUDY Assume that you have been asked to evaluate the careers service at a university. The unit helps students who need assistance with career planning and job-seeking. Many of them are young students with little experience of tertiary study and its requirements. The service operates as an individual self-help centre offering one-to-one counselling with some small group programs covering common issues. The unit relies on self-referrals and advertising to encourage students to attend. The staff of the unit comprise a full-time Director, a full-time careers counsellor and three support staff. The Registrar has been concerned about the viability of such programs that are not considered as core areas of activity. The Registrar has asked for an evaluation and analysis of the operations of the careers service. Some of the initial concerns are: • What kinds of students are using the services? • Do they really need the service? • Do students in-need differ from others? • What is the drop-out rate from the counselling, career planning and job-seeking programs? • How are the services being delivered? In your first discussion with the Director you have discovered that some information is available, such as basic demographic information on participants and attendance records. You have been

32


given permission to interview the Director, counsellors, support staff and students. What questions would you want answered? Which questions deal with service delivery? Is it more important to focus on inputs, processes or outcomes? Outline the structure of a report that you would provide to the Registrar.

33

Chapter four

What information do I need for educational evaluations?

Any evaluation or problem to be solved is socially defined and can be viewed from different perspectives. For instance, job-finding difficulties in unemployed adults might be identified as due to the following: (a) poor knowledge of job-search processes, or (b) the result of a disability; or (c) a deficiency in schooling; or (d) as a wider symptom of socioeconomic disadvantage; or (e) as a result of structural problems in the labour market. Depending on the vantage point that is assumed, then this will have implications for the evaluation process of a labour market program. Identifying the program, policy, product or initiative to be evaluated is a political type of process. It begins even before the client for the evaluation and the evaluator meet. This reflects the notion that both bring to a situation their own preconceptions and values. Sometimes in the process of an evaluation you will come to see more clearly the dimensions of the problem and you may often realise that the client’s perspective might need to be revised in the light of the data you have collected. This is the advantage of an approach to evaluation where the data allow you to answer questions and revise your ideas. You should realise that others would disagree with my view, saying that the whole process is subjective and already constructed within an ideology.

35


In practice, there will be some important pieces of information that an evaluator will need to know about any program. For instance, a key aspect of any evaluation is to be very clear about the main questions to be answered in the evaluation. These should be framed as simple unequivocal questions. The questions should be such that they could be understood by a layperson. I have listed some details that you might want to obtain in order to set the parameters for your evaluation. These details help you to frame the evaluation question. The parameters have been categorised under two headings and can encompass: The nature of the program • the description of the program • details of program participants • an outline of services provided • the program goals, and • the number of sites involved. The purpose of the evaluation • the client’s concerns • the underlying question(s) to be answered by the evaluation • the overt reasons for the evaluation • the need for an outside evaluator • the constraints on the evaluation • the groups likely to have a stake in the outcome of the evaluation, and • the resource commitment to the evaluation. The purpose of this section is to indicate ways in which an evaluator can approach a project. The emphasis is on evaluating an established program. I emphasise formal evaluations of an external nature but much of what I have to say also relates to other contexts. External evaluations are helpful where there is a need to prove a point to others and to have this done by persons for whom there is no conflict of interest in the outcomes. Internal evaluations might be conducted on your own initiative or as an internal requirement. In those cases many of the steps that I describe below and also some of the questions that I list will be quite redundant. 36


Where you are seeking to fine-tune established programs you may be aware already of the context but you may still wish to obtain background information or comparative data. In all cases, an understanding of the program, and knowing the expectations of the client are fundamental for the progress of an evaluation.

Collecting and analysing information A first step is to review the information that is publicly available on the organisation, program or policy that you are about to evaluate. This background information helps you to establish a perspective. You then develop some tentative hypotheses, which can be revised in the light of additional information. Use available data from annual reports, or from official publications to help you establish the dimensions of the problem. In searching published data be aware of any trends and ensure that you examine historical data series where they are available. Agency data and files can also be a useful source of information. For instance, an agency may record the training provided to clients and the shortterm employment outcomes. A search of published articles through online databases (e.g., ERIC, Proquest, Academic ASAP) can also provide additional insights but there is no substitute for pre-existing expertise in the field to be evaluated. I am not convinced that there are all-round evaluators; my opinion is that expertise in a field is invaluable and can be supplemented by knowledge of evaluation methods. If you are planning an evaluation in an area in which you are not familiar and/or one in which there is no source of data or publications, then it may be necessary to interview key persons, such as branch heads in a bureaucracy, program managers or directors of services. They will be able to alert you to key issues and their vested interests. A disadvantage of relying only on key persons is the potential bias that might be introduced. In other words, the validity of the information that you obtain might be compromised. These processes also enable you to define the stakeholders and the target groups for the policy or program. In the evaluation of training programs 37


for unemployed persons it may be necessary to divide the target population into those with varying lengths of unemployment since there is a qualitative difference in the experiences of short-term, long-term and very long-term unemployed. There will also be scope at the planning stage for you to consider how the evaluation might be conducted or how it is required to be conducted. All this presumes that evaluation is proceeding under ideal conditions. My experience has been that clients, stakeholders and available resources can place constraints on an evaluation. There are many possible extraneous influences. A steering committee for a project may not be able to meet at the required time to give approval for the next stage and this can seriously delay progress in a project. Stakeholders with vested political interests can lobby for their perspectives to be included in any contracted evaluation. In many ways the evaluator is a type of social program detective, unearthing evidence and making sense of clues. Some programs will be easier to evaluate than others. Some will have specific and clearly stated objectives while others will have been implemented in an ad hoc way. In my experience there has nearly always been some vested interest. The comments I made refer mainly to established services rather than proposed programs. I am using programs and services to refer to any object of evaluation such as a careers initiative, an educational product, an organisation, a policy or a program. A further qualification that I would like to make is that the process of preparing for the evaluation may need to operate within a tendering and submission process so that many of these desirable steps may not be possible for an evaluator to undertake until after the approval for the evaluation to proceed has been granted. The evaluator often has to make a submission so they can then be shortlisted from the set of tenderers and in this submission the evaluator indicates the proposed methodology on the basis of limited information. This may pre-empt many of the steps I have described or they may be included in a tender as initial steps in the formal evaluation process. The details of the evaluation will depend on your approach and it is recommended that the evaluator seek answers to a range of questions. Three overarching questions have been recommended (Stecher & Davis, 1987, 38


p. 44): what is the program?; why is it being evaluated?; and what constraints limit the evaluation? These three questions are a useful start to providing you with information about the size of the problem or the key issue, the types of clients served by a program and the social context in which it occurs. The aim is to generate a clear statement for the evaluation. What is the key question? In many ways it can be helpful to focus the evaluation by phrasing it in terms of a specific question(s) that can be answered ‘yes’ or ‘no’ (e.g., Has this system of instruction enhanced job acquisition?). This is the same aspect that I teach to doctoral students about to embark on a research thesis – the phrasing of the question is a critical step, as it will focus your thinking and investigation. The analysis in the following sections is based on Stecher and Davis (1987, Chapter 3), Owen and Rogers (1999) and Herman, Morris and Fitz-Gibbon (1987).

What is the program? This first question covers basic familiarity with the details of the program. There are a number of sub-questions about the clients, the goals and the processes (see Table 4.1 for a classification of some questions adapted to career education and guidance). All of these provide some insight into the extent and size of the career problem being addressed by a particular program, policy or initiative. By asking these questions you start to sort out what are the important facts for your evaluation of the program. You come to understand some of the factors influencing the evaluation problem. These questions are especially useful when official and published information is lacking, when agency records are inadequate or inaccessible to you or there is little previous research upon which to rely. There is no substitute, however, for experience and knowledge in a field and if an evaluator has these qualities, then they know which questions to ask. This range of questions can be asked of a wide range of people from those who designed or developed the program, from those who have to implement it as well as those who are most affected by it in the community. You can ask 39


these questions of people within an organisation as well as of people who are outside the organisation. Some of the groups you may need to contact include the following: government departments; bureaucrats at all levels; managers; client groups; staff; or community organisations. The questions being asked centre on the goals and objectives for the program. It is helpful to clarify these objectives when assessing the potential impact of any service. Sometimes selective interviews or small pilot studies can be helpful to give you a better understanding of the dimensions of a problem. Throughout this process a type of hypothesis generating and testing process is beginning to take shape. You are also seeking to determine the size and extent of the problem addressed by the program or service as well as the resource implications and the characteristics of the clientele. Some specific evaluation benchmarks have been listed by Rossi, Freeman and Lipsey (1999, Chapter 3). To summarise: Incidence: the flow of new clients or the number of new people with a problem (e.g., new recruits, newly retrenched) Prevalence: the rate of occurrence of a problem (e.g., rate of unemployment, extent of persons with lower than average literacy skills) At risk: the potential group of persons who might be likely to encounter a problem (e.g., social and economic disadvantages leading to poor completion rates) Sensitivity of a program: the adequacy of the criteria used to identify the appropriate persons covered by a program (e.g., sensitivity of a screening test to identify persons with learning disabilities) Specificity of a program: the adequacy of the criteria used to exclude or reject those unlikely to benefit from a program (e.g., the value of a selection test to diagnose those unlikely to benefit from training). Taken together, these five benchmarks help to establish the quantitative dimensions of the career issue or problem. In asking all of these questions, it soon becomes apparent that different groups will have perceived the issues and the career problem quite differently. 40


Why is the program being evaluated? The answers to this question help the evaluator establish his or her role and client expectations. In most cases evaluators aim to provide evidence of accountability or to justify the existence of a program. The questions summarised in Table 4.1 indicate whether the evaluator needs to focus on the coverage of the program, the services it provides, its impact, efficiency and/ or financial aspects for particular groups. These basic criteria are the ones upon which most people will base their decisions, either rightly or wrongly, for or against a program.

What constraints limit the evaluation? The evaluator also needs to know to what extent there are constraints that limit the process (e.g., deadlines, budgets). Some potential questions that are relevant to career education and guidance are also summarised in Table 4.1 on page 42.

41


Table 4.1. Evaluation questions for career education and guidance

What is the program? Clients • Details of the program participants and non-participants • Ostensible reasons for involvement in the program or service

Goals • The scope of the program’s goals and objectives • An indication of the desired outcomes

Process • A description of the career education or guidance program • An outline of the specific services provided • The underlying career theory or guidance rationale for the program • The schedule of activities

Organisation • Information on the staffing and the sites for program delivery • Details of funding • Information on organisation and funding • Background on any changes in the program or its organisation • The nature of the information that is publicly available about the program

Why is the program being evaluated? Immediate purposes • The reasons for the evaluation of the program or the service • Expectations about the evaluation process and the outcome • Indication of any special concerns about the program or the service

Audiences for the evaluation • Type of information that will be of most value to stakeholders • Type of information that will be of most value to decision makers

What constraints limit the evaluation? Budget • Details of the funding (if any) available for the evaluation • The extent of staffing or other support (if any) available for the evaluation

Schedule • Details of the final and interim reporting dates • Availability of information • The extent to which the findings of the evaluation are confidential • The extent of access to the individuals, organisations and sites related to the program

Source: Adapted from Stecher & Davis (1987, Chapter 3), Owen & Rogers (1999) and Herman, Morris, & Fitz-Gibbon (1987). Evaluating Career Education and Guidance © James Athanasou 2007

42


Summary Some fundamental steps in the planning stage are now summarised below in Table 4.2. Table 4.2. Proposed steps in planning for evaluations of programs Proposed steps in planning for evaluations of programs 1 Check background information and details. 2 Specify the question the evaluation will seek to answer. 3 Identify the aims and objectives of the program. 4 Ensure you are familiar with the content of the program. 5 Describe the stakeholders and client groups. 6 Consult key decision makers. 7 Consider how the program ought to be evaluated. 8 Prepare submission or tender or the evaluation plan.

It is not necessary to follow each of these steps religiously or in a specific sequence. They offer a roadmap and broad guide as to the direction in which your evaluation planning might travel. With this background and information you would now be in a position to address any ethical concerns. I do not write about these matters in a separate section because I find them difficult to describe in advance. Nor am I sure that the topic benefits greatly from a lengthy description. Ethical concerns can be quite straightforward where they reflect matters of conscience and what is right or wrong. At this point you should have enough information to decide about the ethics or morality of the program or service, the social and political implications of your evaluation, whether there are issues of legality in what you are doing, whether you have a vested interest in some outcome, as well as broader issues relating to your own values. Once again, I would suggest that your personal conscience serves as a guide. If you have an intuition that your involvement in a program or your evaluation of it has ethical problems then my recommendation is that it is best to walk away from it. -oOo-

43


REVIEW QUESTIONS 1 Evaluations can have more than one client. These may have different needs. Who should be served by an evaluation? 2 Document your answers to the following three questions for a program, service, policy or initiative in which you are involved or are likely to be involved.

(a) What is the program?

(b) Why should the program be evaluated?

(c) What constraints would limit any evaluation?

3 How do answers to some of the following questions about a program or service affect your evaluation?

(a) What are the components of the service?

(b) How was the service provided?

(c) What was the frequency and duration of the service?

(d) How many people are being serviced by the program?

4 Do you agree that people who are personally interested and involved in an evaluation are more likely to use evaluation findings? 5 Under what conditions would a needs assessment be threatening to an organisation?

44


CASE STUDY The career education quality framework was designed as a useful document for internal evaluation of a career education program. The Careers Education Quality Framework is a ‘guide for those who guide’. It provides teachers, careers specialists, school leaders and school communities with a practical tool for self-assessing Careers Education and developing action plans for improvement … The Framework is a development tool for a school’s Careers Education program. It is most effectively used when stakeholders work through it together and jointly produce an Action Plan for the coming 12 months. This group could include parents, school staff and members of the community (p. 2). This document can be downloaded from http://www.dest.gov.au/directory/ceqfwholebook.pdf. Source: Department of Education, Science and Training, 1999. Note that some aspects of the Careers Education Quality Framework have been superseded by later developments. Both the career and work preparation outcomes framework and the Australian Blueprint for Career Development also contain ideas for evaluating career education programs.

Indicate how this Careers Education Quality Framework approach might be applied in your work context and to what extent it covers the relevant issues for an evaluation.

45

CHAPTER FIVE

How do I assess program coverage?

Questions always arise in my mind whenever someone boasts about their career education program or service. Before I even ask about impact I often want specific answers to questions about the nature of the program, the clientele and the coverage of the service. Answers to questions about the coverage of a program require meaningful information about its constituents. Some of the questions have already been answered in the planning stage of the evaluation but some more may remain to be answered by further investigations. In the last chapter I considered how you go about determining the focus of an evaluation. This included obtaining some details about the scope of a program and its operations. This chapter emphasises data on a program’s coverage. These data are required to describe a program’s operations so that its implementation can be monitored or some aspects of its coverage can be assessed. An evaluation of a program or service cannot proceed in the absence of data and observations. These can range from quasi-standardised personal reflections to systematic analyses of existing records in an organisation (see the continuum in Figure 5.1 on page 48).

47

5813-05-Chapter 5.indd 47

16/02/2007 11:57:30 AM


Personal reflection

Observation

Data collection

Management information system

Figure 5.1. A continuum of information for evaluation (Athanasou, 2000)

How many people are being serviced by the program? Participation in a program is a convenient index of the performance of a program. The question to ask is: ‘How many people participated in the program?’ One of the first factors that needs to be considered in any analysis of the answer to the question includes the extent to which the program is voluntary in nature. If it is voluntary then you get an idea about how acceptable the service is in proportion to the target population. A second factor that needs to be considered is whether the program is servicing only a select group such as a minority, or only those able to access a service, or those who are aware of a service rather than the entire population. For instance, a labour market program might service only those who were most likely to be employed in any instance rather than those who are at the highest risk of being long-term unemployed. Programs and services by their nature often develop selectivity over time. Different groups may come to participate at varying levels with inequities developing in meeting the original aims. You will need to be aware of this factor and assess whether there is any bias in the delivery of a service. Such selectivity or bias occurs when client selection is not directed towards the target population. A service can be operating profitably and to everyone’s satisfaction but also providing programs to those who are not in need. The first pieces of information required to support any claim about a service are to determine aspects of the target population. You need to know: • the size of the target population (e.g., those of non-English speaking background in a local government area)

48


16/02/2007 11:57:31 AM


• the proportion of people to whom the service is delivered • the proportions of people in need and/or not in need to whom the service is distributed. Two aspects of a program’s operations need to be discerned with respect to coverage. The first is what proportion of those in need are being serviced. The second is how many of those who are being serviced are really in need. people in need who are serviced by a program Program coverage = total number of people who are in need The coverage of a program gives you an indication of the extent to which the service is meeting needs. It is an index of participation. The coverage of the population is important because a program or service may only be reaching a minority of the eligible participants. An important political question to be answered is whether it should be discontinued if it is not reaching a significant proportion of the population. A vocational guidance service claimed that it served 50,000 per year but counted services and not clients; moreover it did not consider the clientele as a proportion of the population of some 750,000. There was no analysis of whether the service was really needed by clientele or whether other groups were excluded. It is important to distinguish both the target population and the entire population for the purposes of program coverage. The following formula has been suggested. people in need who Extent of needs serviced correctly

are serviced by a program = total number of people serviced by a program

49


16/02/2007 11:57:31 AM


This leads to considering a measure for whether particular groups within a community are being serviced. Rossi, Freeman and Lipsey (1999, p. 211) have suggested that the efficiency of coverage can be calculated by this formula: Number in Coverage efficiency = 100 ×

need served ____________

Number not _

in need served _____________

Total number

Total number

in need

served

Source: Rossi, Freeman & Lipsey (1999, p. 211)

Example: Population

Program

50 in need

20 in need

150 not in need

30 not in need

Coverage efficiency =

20 30 – = 0.4 – 0.6 = - 0.2 or –20% 50 50

Negative values occur when inappropriate groups are being serviced. The conceptual basis for this index can be difficult to interpret because there are no common units in the formula. It should be used with caution as the index will be subject to fluctuation depending on the proportions involved and may not always apply to particular groups. Finally, I would always want to know how many people continue in a program. Attrition rates for many social and educational programs are very high and some official sources are embarrassed to provide data on drop-outs from programs.

Sources of information Where do you obtain the information for these calculations? Obviously program records are important. If these do not exist (and you should be

50


16/02/2007 11:57:31 AM


asking why they are non-existent), then you may need to establish a record system for the duration of the evaluation. If a record system is established for the evaluation, then there are many practical considerations about the quality of the data obtained. For this reason, it may be necessary to collect only a minimum amount of data, (a) so that agency staff is not opposed to the procedures established, and (b) also to ensure that aspects of client privacy and confidentiality are maintained. When programs and services are diverse in nature and/or it is difficult to obtain information directly from an organisation, then you may need to consider community surveys as a source of information. Such surveys can encompass participation rates as an index of awareness; drop-out rates as an index of dissatisfaction; and failures as an index of program effectiveness. This brief section has emphasised the initial analysis of program operations. It acts as a basis for the next step in an evaluation. In some instances the information obtained at this stage may already be sufficient to answer the evaluation questions. For instance, a program may not be achieving efficient coverage, may have high drop-out rates during its operation and poor success rates after program completion. Accordingly any further analysis may be redundant. -oOo-

REVIEW QUESTIONS 1 Describe and analyse the coverage and efficiency of coverage of a program or service in which you are involved or likely to be involved. Comment on your findings. 2 How does program monitoring fit into the function of program evaluation? 3 How does monitoring relate to formative and summative evaluation? 4 What types of program evaluation questions can be answered using an information system?

51


16/02/2007 11:57:31 AM


CASE STUDY 1 The Career Horizons Program is located at a large Mid-western university [in the USA]. It provides at-risk students going into seventh grade with an intensive summer program combining career classes with exposure to a college campus and recreational team-building activities to enhance students’ career selfconfidence and vocational considerations. Source: O’Brien, K., Dukstein, R. D., Jackson, S. L., Tomlinson, M. J., & Kamatuka, N. A. (1999). Broadening career horizons for students in at-risk environments. The Career Development Quarterly, 47, 215–228.

What program coverage questions would you raise in conjunction with this program for students who are considered at risk? 2 This table shows the pattern of services requested from a university careers service. Reasons for attendance Per cent Job search 28 Interview skills 4 Resumé writing 13 Course issues 13 Course change 8 Career direction 28 Prospective student 3 Industry placement 1 Other 2 What program coverage questions would you ask about this service?

52


16/02/2007 11:57:32 AM

Chapter six

How do I analyse program costs?

Efficiency in career education and guidance programs is linked to their costeffectiveness and/or the cost of the benefits that they provide. This is the way these terms are defined in the second edition of the Program Evaluation Standards (Joint Committee, 1994): Cost: the total value, social and monetary of all the human and physical resources used to carry out the program (p. 77) Benefits: the value of all the results and outcomes (p. 77) Cost-effectiveness: the extent to which one program, project or instructional material produces equal or better results than competitors’ programs that cost about the same amount of time, effort and resources; or the extent to which an object produces the same results as competitors’ but is less costly (p. 204).

How can you determine the efficiency of a program or service? This aspect of many programs is not always determined and in many cases it is overlooked. My experience is that some large-scale programs or services are 53


well below standard in their level of efficiency. What do I mean by this? Firstly, the cost of the benefits (human, economic, ethical, social) is greater than comparable programs. Secondly, the effectiveness might be much less than what is stated. Certainly, long-term effectiveness is rarely established. This does not mean that career education and guidance should be measured only in terms of cost. Nor does it mean that effectiveness or efficiency is the only criterion. Education and guidance are not production lines. Sometimes less effective approaches might suit a community or costly methods might have long-term advantages for some individuals. This is fine provided you know about it. You should also consider the social costs and efficiency of any program. At the present time it is not clear to me how this could be done in a way that would have overall consensus. What I am saying is that you should know at least what the costs and benefits are and whether the efficiency is as high as is claimed. You should know which program produces the same impact for a given level of expenditure. This chapter covers issues that relate financial costs to program results. The ideas are derived from business contexts but also relate to any reasonable evaluation where financial expenditures are involved.

Efficiency Efficiency is a relative concept. There is no maximum level of efficiency so standards need to be established. The standard would involve best practice in career education or career guidance or some other yardstick. You should also realise that efficiency will depend on resources, equipment, size of program and quality of management. In this chapter I look at costs and efficiency as a static concept. In part this is misleading since it fails to consider the development of a program and the fact that these indices change over the natural history of a program. The advantage of measuring efficiency is that it makes you think about a program’s advantages and disadvantages. It disciplines your evaluation of claimed benefits. You can undertake an efficiency analysis in the planning of a program or after a service has been offered for some time. 54


It is also relevant to determine costs and benefits (financial, personal, community and social) prior to establishing programs. Too often governments establish programs and services and people then build their careers and working lives around these services, only to have them withdrawn at a later stage. Both clients and agency staff are greatly dislocated by such policy inefficiencies. It is difficult to recall any major human services that have not been subject to policy whims or changes. Secondly, some excellent policies are often implemented in less than effective ways. Prior analysis of likely efficiency is relevant where services cannot be easily withdrawn or where capital investments are involved. Post-program costbenefit analyses are relevant in order to examine the effectiveness of a program or to compare the efficiency of a service or to record its impact. I realise that converting outcomes to money terms is not straightforward and that not all inputs into a program can be costed. Effectiveness and efficiency, however, do not always require you to estimate financial costs. There are human, social and material outcomes and costs that can be considered. You could compare training in terms of the time required to learn; or you could look at the efficiency of an educational service in terms of long-term social effects on individuals; or you could estimate the time required to implement a new system of instruction.

How to calculate costs Costs can be expressed in monetary values but they can also be compared in terms of alternatives. They can be calculated in resources required or in terms of the value or benefit derived. Costs include labour, rent, facilities, materials, capital equipment, resources and services. You may need to add any costs undertaken by participants in an employment context (e.g., textbooks, uniforms, travel, tools). There are different methods of cost analysis including (a) cost feasibility; (b) cost-effectiveness; (c) cost benefit; and (d) cost utility.

55


Cost feasibility = cost of program – funds or resources available Cost effectiveness = cost of program or the effect of one program compared with another program on some common measure Cost feasibility: Cost feasibility answers the question of whether I have sufficient funds available. It is useful for eliminating options that are not viable. If the costs of three employment training programs are $1248, $4589 and $2512 per participant and the budget available is $1.8m, then only the first and third alternatives are feasible if you want to train at least 600 people. Cost-effectiveness: Cost-effectiveness is useful when programs or services have the same outcomes or goals. There must be a common measure for costeffectiveness. Effectiveness allows you to rank programs and services in terms of their costs for achieving the same outcomes. A hypothetical example for calculating effectiveness is shown in Table 6.1. Table 6.1. Example showing how to calculate cost-effectiveness Job-finding services

Additional cost to college

No. of cases 2007

Effectiveness per person helped

Direct instruction Individual counselling Supported placement No treatment

$88,000 $92,000 $71,000 na

47 32 18 7

$1872 $2875 $3944 –

Cost benefit: Cost-benefit analysis allows you to compare different alternatives when their inputs and outcomes are measured in dollar terms. If money values have not been assigned to inputs and outputs, then these must be determined in some fashion. The cost-benefit analysis requires you to subtract the cost from the benefit. It can also be expressed as a ratio. A hypothetical and simplified example that compares the effectiveness of two programs for an organisational career development program is shown in Table 6.2. Note the larger net benefits for one program over another but also that the benefit to cost ratio of one over the other is markedly different. This highlights one weakness of the costbenefit approach.

56


Table 6.2. Example showing how to calculate cost benefit for a corporate career development program

Computer-based recruitment service

Career information publication

Benefits Costs Net benefits Benefit–cost ratio

$5.6m $0.8m $4.8m 7.0

$3.4m $0.3m $2.9m 11.3

Discounting: Career education and guidance are considered to have shortterm as well as long-term benefits. In order to estimate the present value of future benefits, you can discount these future benefits. For instance, a program may claim to provide future benefits of $200,000 per year to a corporation and over a five-year period it is stated that around $1 million will be saved. This amount should be reduced to its present value. In order to do this, you discount it by any amount you choose. If you allow for an inflation rate of around 5 per cent over the period, then the value in present dollars would be equivalent to $865,893. Table 6.3. Example showing how to calculate present value Present value of an amount = Amount/(1 + r)t Example: Year 1 $200,000 ÷ (1 + 0.05)1 = $190,476 Year 2 $200,000 ÷ (1 + 0.05)2 = $181,405 Year 3 $200,000 ÷ (1 + 0.05)3 = $172,767 Year 4 $200,000 ÷ (1 + 0.05)4 = $164,540 Year 5 $200,000 ÷ (1 + 0.05)5 = $156,705 Where t = period of time

Cost utility: Programs and services can be analysed in terms of their costs and monetary benefits but they might also vary in their usefulness and in other values. To undertake a utility analysis, a program or service must be rated in terms of its utility. A scale of values can be determined, say from 1 to 5: 5 – Very high utility (top 10%) 4 – High utility (next 20%) 3 – Average utility (middle 40%) 2 – Low utility (next 20%) 1 – Very low utility (bottom 10%)

57


Note: I prefer to take the logarithm of these numbers as the units from 1 to 5 are not equal; the spacing between the logarithms is for me a better indication of real differences; for example: log(5) = .69, log(4) = .60, log(3) = .47, log(2) = .3, log(1) = 0. I can now apply cost-utility analysis to the previous two examples (see Table 6.3). In the case of the job-finding services (Table 6.4), the high value placed on the individual counselling means that this program will be favoured in a cost-utility analysis even though it is not as cost-effective. Similarly, in the case of the corporate career development program (Table 6.5), the greater utility is for career information over computer-based recruitment. In many instances the cost-utility approach provides you with contradictory findings because educational phenomena cannot always be weighted and costed in the same way as other inputs and outputs. Table 6.4. Example showing how to calculate cost utility of job-finding services Program

Direct instruction

Individual counselling

Supported placement

No treatment

Additional cost No. of cases Effectiveness per person helped Utility to college policy Cost-utility

$88,000 47

$92,000 32

$71,000 18

nil 7

$1872 2 $44,000

$2875 4 $23,000

$3944 3 $23,666

– 1 na

.6 $38,333

.47 $50,353

0 na

Using logarithms of utilities Utility to college policy Cost-utility

.3 $29,333

58


Table 6.5. Example showing how to calculate cost utility of corporate career development

Computer-based

Career information

Benefits Costs Net benefits Benefit–cost ratio Utility Cost-utility

$5.6m $0.8m $4.8m 7.0 3 $266,666

$3.4m $0.3m $2.9m 11.3 5 $60,000

.47 $1,702,127

.69 $434,782

Using logarithms of utilities Utility Cost-utility

-oOo-

REVIEW QUESTIONS 1 Which cost-analysis procedure do you believe offers the most useful basis for analysis: (a) career education programs; (b) career counselling services; (c) computer-assisted guidance? 2 What would you identify as some of the costs associated with the provision of a careers service at a secondary college? 3 What would you identify as the benefits of (a) an employment training program for the long-term unemployed; (b) an interactive multimedia career education package; (c) an associate diploma course in career development at a regional technical college; (d) a job training program for long-term unemployed? 4 For a service or program in which you are involved (or plan to be involved or interested), undertake an analysis of the costs and benefits. Apply those cost-analysis procedures that you believe offer the most useful basis for analysis. 5 Compare cost benefit and cost-effectiveness in terms of the decisions that might be made for a career education or guidance program in which you are involved or interested.

59


CASE STUDY 1 Consider the table below, which is reproduced from ‘Restoring Full Employment’. In your opinion, which of the labour market programs described is (a) most cost effective; (b) has the greatest cost benefit; and (c) appears to have the greatest cost utility? Justify your response and/or indicate what further information is required to make a conclusion. Outcomes and net costs of labour market programs 1992–93 Program

Proportion of Proportion of Net unit cost Net cost to long-term clients assisted of assistance each positive unemployed with a positive outcome targeted by outcome the program % %

JOBSTART

62.0

58.9

$1,519

$2,572

Job Clubs

58.0

47.0

$625

$1,460

Skillshare

58.2

41.8

$747

$1,791

JOBTRAIN

67.4

36.4

$1,213

$3,226

Special intervention

65.1

40.7

$1,088

$2,572

100.0

44.0

$9,055

$20,768

53.7

73.0

$7,218

$9,888

JOBSKILLS Self-employment assistance

Notes: A positive outcome is defined as unsubsidised employment some three months after assistance ceases. Net costs take account of direct offsets and take into account that some job-seekers would have found jobs without assistance. Source: © Commonwealth of Australia, reproduced by permission from Committee on Employment Opportunities, Restoring Full Employment, Canberra, AGPS, 1993.

60


2 Now consider the estimated return in Social Security spending and the tax paid for each dollar of program spending which is listed in the following table. Which of the three programs (JOBTRAIN, JOBSTART, or JOBSKILLS) would you support as being most cost-efficient, cost-effective and having the greatest cost utility?

Source of saving for each dollar spent

JOBSTART wage subsidies (cents)

JOBTRAIN courses (cents)

JOBSKILLS work experience (cents)

Social security savings during the life of the program

26

47

26

Social security savings after the end of the program

30

1

5

Tax paid by participants during the life of the program

4

< 1

6

Tax savings after the end of the program

4

< 1

Evaluating Career Education and Guidance

Career Guidance in Context

Critical Reflections on Career Education and Guidance: Promoting Social Justice

International Handbook of Career Guidance (Springer International Handbooks of Education)

Career in Music Education

Evaluating the measurement uncertainty: fundamentals and practical guidance

Rethinking Careers Education and Guidance: Theory, Policy and Practice

Evaluating Bilingual Education: A Canadian Case Study

Evaluating Teacher Quality in Higher Education

Contextualizing Inclusive Education: Evaluating Old and New International Perspectives

Screenwriting For Dummies (For Dummies (Career Education))

Phonics for Dummies (For Dummies (Career Education))

Phonics for Dummies (For Dummies (Career Education))

ASVAB For Dummies (For Dummies (Career Education))

The Guidance

The Guidance

The Guidance

Education and Career Choice: A New Model of Decision Making

Best Career and Education Web Sites: A Quick Guide to Online Job Search (Best Career & Education Websites)

Risk Management: Concepts and Guidance

Principles of Guidance and Counselling

Evaluating Development Programmes and Projects

Missile Guidance and Control Systems

Sight and Sensibility: Evaluating Pictures

Evaluating Evaluators: An Evaluation of Education in Germany

Evaluating and Reviewing Children's Books

Workplace and Career

Career Choice and Development

Cosmetology Career Starter (Career Starters)

Evaluating Career Education and Guidance

Career Guidance in Context

Critical Reflections on Career Education and Guidance: Promoting Social Justice

International Handbook of Career Guidance (Springer International Handbooks of Education)

Career in Music Education

Evaluating the measurement uncertainty: fundamentals and practical guidance

Rethinking Careers Education and Guidance: Theory, Policy and Practice

Evaluating Bilingual Education: A Canadian Case Study

Evaluating Teacher Quality in Higher Education

Contextualizing Inclusive Education: Evaluating Old and New International Perspectives

Screenwriting For Dummies (For Dummies (Career Education))

Phonics for Dummies (For Dummies (Career Education))

Phonics for Dummies (For Dummies (Career Education))

ASVAB For Dummies (For Dummies (Career Education))

The Guidance

The Guidance

The Guidance

Education and Career Choice: A New Model of Decision Making

Best Career and Education Web Sites: A Quick Guide to Online Job Search (Best Career & Education Websites)

Risk Management: Concepts and Guidance

Principles of Guidance and Counselling

Evaluating Development Programmes and Projects

Missile Guidance and Control Systems

Sight and Sensibility: Evaluating Pictures

Evaluating Evaluators: An Evaluation of Education in Germany

Evaluating and Reviewing Children's Books

Workplace and Career

Career Choice and Development

Cosmetology Career Starter (Career Starters)

Recommend Documents