Principles of Performance-based Evaluation
This chapter establishes and examines six principles that guide every high-quality evaluation. Evaluators must (1) ask the right questions. To do this, they must (2) understand the distinction between means and ends and (3) examine the goals of the organization as a function of its valid needs. (4) Needs should be validated using a top-down external approach. (5) Evaluation standards should be based on a useful and real strategic vision. (6) The set of questions derived from the first five principles drives the evaluation study.
Evaluation seeks the answer to the general question, What worked and what didn’t? It should also ask at the same time, Of what value is this, and what does it contribute? Successful evaluation hinges on how the questions are framed in the context of results-oriented programs that serve an organizational and societal purpose. It also seeks to determine whether the results obtained are of value to the client organization and the society it serves.
The results and consequences of evaluation are not arbitrary. It is important not only to see whether we get the results we hope for but also to ensure that what we hope for is reflected in the means selected and that the ultimate value was added by a program, project, solution, or organizational system to our shared society. All of these are important.
Societal and organizational needs are met through organizational efforts focused on value-added outcomes. The primary focus of performance evaluation is to determine that the goals and objectives of programs are valid, as well as effective and efficient, in producing desired results internal and external to the program and organization.
Planning for legitimate programs starts by identifying desired organizational impacts on the organization’s clients and market as well as on society—needs, not wants—and moves into the organization to derive strategic, tactical, and operational objectives. Thus, evaluators should look at the alignment of programs, organizational objectives, and desired external impact. All must be clearly aligned and linked. These ideas are essential threads that must be woven into the fabric of every evaluation.
Words and concepts are rigorously defined here, for evaluation should be used to measurably improve what organizations use, do, produce, and deliver. Undefined ideas and hasty selection of methods will not serve anyone well. Thus, we must be thoughtful and reflective as the pragmatic evaluation landscape is created. The journey will be worth the planning time and attention.
The following set of principles, based on Kaufman, Guerra, and Platt (2006), will be helpful to define and determine what worked and what did not.
https://scholar.flatworldknowledge.com/books/27617/portolesedias_1.0-ch11_s02/read
https://ebookcentral-proquest-com.ezproxy.snhu.edu/lib/snhu-ebooks/reader.action?ppg=30&docID=876641&tm=1535061883493 (PAGES 17-30)
https://www-proquest-com.ezproxy.snhu.edu/docview/1037692128?accountid=3783
OVERVIEW OF CLASSIC EVALUATION MODELS
Many worthwhile models that will not be covered here can certainly be applied (and in fact already may have been) in any number of contexts, including performance improvement. Before examining the evaluation models covered in the upcoming chapters, this chapter mentions some worthy evaluation approaches without studying them in detail.
Objective-Based Evaluation: Tyler (1949)
Over the course of Ralph Tyler’s more than sixty-year career, he influenced education at all levels, including curriculum, testing, evaluation, and educational policy. Directly and indirectly, he influenced many noteworthy developments such as objective-referenced testing, objective-based program evaluation, mastery learning, achievement test construction, item banking, the taxonomic classification of educational outcomes, and cooperative test development (Madaus & Stuffleman, 1989). Objective-based evaluation describes whether students have met their goals, with the results informing how to handle a new instructional strategy (revise, adopt, or reject). One noted weakness of this approach is that the evaluator may overlook unexpected outcomes or benefits of instruction beyond the original goals.
Consumer-Oriented Evaluation: Scriven (1967)
This approach is focused on meeting consumer needs and societal ideals more than achieving the developer’s objectives for a given program. Scriven made a major contribution to this approach with his distinction between formative and summative evaluation. He proposed that summative evaluation enables administrators to decide whether the entire finished curriculum, refined by the use of formative evaluation, represents a sufficiently significant advance on the available alternatives to justify the expense of adoption by a school system (Fitzpatrick et al., 2004). Scriven proposed a set of seminal criteria for evaluating any education product, titling it a Key Evaluation Checklist (Scriven, 1991, 2002). He continues to revise this checklist and uses it as part of a data-reduction process, where large amounts of data are obtained and assessed and then synthesized in an overall judgment of value (Stufflebeam & Shinkfield, 2007). Scriven’s checklist also addresses meta-evaluation.
Discrepancy Model of Evaluation: Provus (1971)
For Provus, preestablished objectives were the basis for evaluation; however, he also emphasized the importance of providing data about the consistency of (or discrepancy between) what was planned and what was actually executed. The model has four basic phases: establishing objectives, collecting evidence of compliance with the standards, identifying any discrepancies between preestablished objectives and what was accomplished, and identifying and starting corrective actions. This model lends itself to a self-evaluation framework and a systematic approach to improvement. This model is the theoretical foundation for the Impact Evaluation Process (Guerra-López, 2007a).
Goal-Free Evaluation: Scriven (1974)
This approach compensates for inherent weaknesses in a goals-oriented approach by providing an unbiased perspective of ongoing events. Here, the evaluator purposely remains uninformed of the program’s predetermined goals and looks for all effects of a program regardless of its developer’s intended objectives. If a program is meeting its intended purpose, the evaluation should confirm that. In addition, the evaluator will be more likely to find unanticipated effects that goal-based evaluators could miss because of the specificity of their search. Stufflebeam and Shinkfield (2007) indicate that goal-free evaluation provides important supplementary information, expands the sources of evaluative information, is effective for finding unexpected information, is cost-efficient, and is welcomed by clients.
Responsive/Client-Centered Evaluation: Stake (1975)
Based on Ralph Tyler’s conception that evaluators should compare observed outcomes to intended outcomes, Stake’s approach expanded evaluation focus to include background, process, standards, judgments, and outcomes. Although his philosophy of evaluation first appeared in the literature in 1967, his formal proposal of this approach, named responsive evaluation, was not published until 1975. In this work, he broke away from Tyler’s perspective of examining whether intentions had been realized, instead assuming that intentions would change and calling for continuing communication between evaluator and stakeholders for the purposes of discovering, investigating, and addressing important issues (Stufflebeam & Shinkfield, 2007).
Utilization-Focused Evaluation: Patton (1997)
Closely linked to decision-oriented theory, this approach to evaluation is concerned with designing evaluations that inform decision making. Although many authors have contributed to this approach, Stufflebeam and Shinkfield (2007) credit Michael Patton as the most prominent figure (1980, 1984, 1997, 2003). In his 1997 book, Utilization-Focused Evaluation, Patton fully articulated his approach to evaluation. Here he defined a utilization-focused program evaluation as one done “for and with specified intended primary users for specific, intended uses” (p. 23).
SELECTING A MODEL
There is no one best model. The utility of evaluation models, as with any other type of model or tool, depends entirely on the situation at hand. What works in one organization does not necessarily work in another, and what worked yesterday will not necessarily work in the same way today. Stufflebeam and Webster (1980) analyzed thirteen alternative evaluation approaches in terms of their adherence to the definition of an educational evaluation: one that is designed and conducted to assist some audience to judge and improve the worth of an educational endeavor. Their analysis resulted in three categories of evaluation studies: politically oriented, or pseudo-evaluations; question oriented, or quasi-evaluations; and values oriented, or true evaluations. They provided strengths and weaknesses for each in order to offer evaluators a variety of frameworks and perspectives.
The most important part of choosing a responsive model is that we clearly identify the requirements of the situation and use that as the criterion for selecting an evaluation model. Popham (1975) has said that comparing evaluation approaches in order to select the best model is usually a waste of time. He argues that instead of focusing on similarities and differences, evaluators should become sufficiently proficient with the evaluation models to decide which is the most appropriate for the situation. Moreover, he has argued for an eclectic approach: selectively drawing from various models the concepts that are most helpful. Following Popham’s advice, we might find it useful to borrow the most relevant bits and pieces of various models to form an evaluation plan. This approach can be taken for every evaluation. Cronbach (1982) also supports this view and states that evaluation design must start from a blank slate at each new undertaking, addressing the countless decisions to be made in each situation.
CONCEPTUALIZING A USEFUL EVALUATION THAT FITS THE SITUATION
Nevo (1981) revised a list of questions initially proposed by Stufflebeam (1974) in an effort to conceptualize evaluation and its key dimensions. The revised list remains applicable today:
- How is evaluation defined?
- What are the functions of evaluation?
- What are the objects of evaluation?
- What kinds of information should be collected regarding each object?
- What criteria should be used to judge the merit or worth of an evaluated object?
- Who should be served by the evaluation?
- What is the process of doing an evaluation?
- What methods of inquiry should be used in evaluation?
- Who should do evaluation?
- By what standards should evaluation be judged?
Some reflective questions will help formulate a model for a specific evaluation. Note that this list is not exhaustive but rather is a starting point:
- What are the characteristics of the evaluation task?
- What is the object of evaluation (the evaluand)?
- At what stage of conception (planning, design, development, implementation, maintenance, evaluation) is the evaluand?
- What are the limitations and constraints of the evaluation effort?
- Which evaluation models best lend themselves to address these characteristics?
- What are the pros and cons of each model with regard to the evaluation task?
- Is blending these approaches to create a unique model for this situation more feasible and responsive to the evaluation task than using one of them as it stands?
You may find it useful to keep these questions in mind as you read through the following chapters.
https://scholar.flatworldknowledge.com/books/27617/portolesedias_1.0-ch11_s03/read
https://scholar.flatworldknowledge.com/books/27617/portolesedias_1.0-ch10_s01/read
https://www-proquest-com.ezproxy.snhu.edu/docview/216364942?accountid=3783
https://learn.snhu.edu/d2l/le/content/854335/viewContent/14578818/View
Performance Evaluation Sheet
This document contains performance evaluation forms for three employees. Replace the bracketed text with specific information.