Are you lookingforwaysto evaluatetrainingeffectivenessof a program or course? If so, you may already know that there area number ofapproaches you can take. In this post, we’ll look at threeof themost widely usedmethods of evaluating training effectiveness and how to use them.
The three approaches are:
TheKirkpatrickTaxonomy
The PhillipsROIModel
TheCIPP Evaluation Model
1. The Kirkpatrick Taxonomy
The Kirkpatrick Taxonomy is perhapsthe most widely used method of evaluatingtrainingeffectiveness. Developed by Don Kirkpatrick in the 1950s,this frameworkoffers a four-level strategy that anyone can use to evaluate the effectiveness of any training course or program.
The four levels are:
- Level 1: Reaction
- Level 2: Learning
- Level 3: Behavior
- Level 4: Results
Here’s how each level works:
Level 1: Reaction
At this level, you gauge howtheparticipantsreactedor responded to the training.Asking the participants to complete ashort survey will help you identify whetherthe conditions for learning were present.
Level 2: Learning
The second stage is togaugewhat theparticipants learned from the training. Most commonly, shortquizzes or practical tests are used to assess this; one before the training, and one afterward.
Level 3: Behavior
The third stage takes place a while after the training. Using various assessment methods, you try to assesswhether the course participants put what theylearnedinto practice on-the-job.To assessthis, you may ask participants to complete self-assessments orasktheir supervisorto formally assess them.
Level 4: Results
Lastly, you need to evaluate whether the training metthe stakeholders’ expectations.In most companies or organizations, the stakeholders are usually the management or executives who decided to implement the trainingin the first place.The goal is to determine the return on theseexpectations, known as ROE(Return on Expectations).
Limitations of the Kirkpatrick Taxonomy
Anecdotally, most people have heard of Kirkpatrick and fall into one oftwocamps:
You may use the term ‘Kirkpatrick’ but not follow the full taxonomy.Your organization may not require you to use anythingmore than simple ‘Smile’ feedback sheets after each training session.
You may have read and agreed with the taxonomy, but have little idea about how to apply it beyond basic Level One feedback forms given out after training.
Whichever camp you fall into, there are undeniably some limitations to the Kirkpatrick taxonomy. For starters, the Kirkpatrick taxonomy is often referred to as a ‘model’ or ‘theory’ whenin reality it’slargely ascientific.
A more stinging criticism is thatthe Kirkpatrick approach gathers little data that helpsactually improve training.For example, if a watchmakerruns a training program designed to decrease customerwait times for repairs, yet thewait times don’t decrease afterward, the Kirkpatricktaxonomyonly tells us that the training‘didn’twork’; it doesn’t help to improve the training.
The main limitation, and most common criticism of the Kirkpatricktaxonomy, is that there’s little evidence to support the idea of linear causality –ie. a favorable learning response at level 1 will result inbetter learning outcomes at Level2, orimprovedon-the-job performances in level 3.Research has foundno causal linkbetween the first two levels, even though the Kirkpatrick’spresentationas a taxonomy suggests that there would be.
Thefourthmain limitation of the Kirkpatrick model is that it measures learning in terms of return on stakeholder expectations(ROE). Some firms are looking for a traditional return on investment evaluation where the cost of the training is set against thebenefitsthat itdeliveredfor the company.
How to use the Kirkpatrick taxonomy effectively
Infairness to Don Kirkpatrick, he addressed many of the limitations laid out above. For accuracy, he suggests working backwardsthrough his four levels during the design phase of any training program. This approach helps ensure that an organization decides which outcomes it wants to address first, and then designs or develops the training accordingly.
This approach was pioneered by the lateGrant WigginsandJay McTighein their book, Understanding by Design (UbD®).The UbD® framework is used by educators across the world whendesigning courses and content units. The Kirkpatrick taxonomy is best applied in this fashion, so that the stakeholders or management begin with the outcomes in mind.
Here’s how to applythe Kirkpatrick modeleffectively according to the UbD® principles:
Decidewhat business results you are targeting;ie. the results.
Determinewhether the training matches the stakeholder’s expectations.
Identify whaton-the-job behavior or performance changes you would need to look for to prove that the trainees had met the end results.
Define the learning objectives that will develop the on-the-jobbehavior.
Decidehow to deliver the necessary instruction in an engaging and appealing way.
As you can see, the UbD® framework is a helpful way of implementingthe Kirkpatricktaxonomy. Decide what results you want to see first, and then plan what you’ll need to include in the training in order to get there.
2.The Phillips ROI Methodology
Thesecondmethod for evaluating training effectivenessthat we’ll discussis the Phillips ROI Methodology. When Jack Phillips published his own work on training evaluation in 1980, the Kirkpatricktaxonomywas already well establishedas the dominant training evaluation model. However,Phillips wanted toaddress several of the shortcomings he saw in the Kirkpatricktaxonomy. His ROI methodology is best thought ofasan expanded version of Kirkpatrick’s taxonomy.
The Phillips ROI Methodology has five levels:
Level 1: Reaction
In common with the Kirkpatrick taxonomy,the Phillips methodology evaluates theparticipants’reaction.
Level 2: Learning
The second level evaluates whether learning took place.
Level 3: Application and Implementation
The originalKirkpatricktaxonomyevaluated behavior in the workplace to see whether the learning translated into on-the-job training. Phillips expanded this level to cover both application and implementation. This addresses one of the centralcriticismsof the Kirkpatrick taxonomy:that it doesn’t gather enough data to helpimprovetraining.
Phillips’ methodology makes it far easier to seewhytrainingdoes or doesn’ttranslate intoworkplace changes. If thereisa problem, did it lie with the application or the implementation? For example,was thelearning applied incorrectly? Orwasthe on-the-job trainingimplemented ineffectively?
Level 4: Impact
While thefourth level of theKirkpatricktaxonomyfocusespurelyon results,Phillips’ methodology is much broader.His level 4 –Impact–helpsidentify whether factors other than training were responsible for delivering the outcomes.
Level 5: Return on investment (ROI)
Unlike the Kirkpatrick taxonomy that simplymeasurestrainingresults again stakeholder expectations(ROE), Phillips’ methodology contains a fifth level specifically for measuring‘return on investment’, ROI. This level uses cost-benefit analysis to determine the value of training programs. It helps companies measure whether the money they invested in the training has produced measurable results.
Here’s how it works:
You gather business data from before, during and after the training and look for quantifiable factors, such as process improvements, productivity improvements, or increased profits, depending on the nature of the training. You then compare the cost of the training with the value that it provided. This gives you an indication of the value of the training to the company’s bottom line.
3.TheCIPP evaluation model
The thirdapproach to evaluating training effectiveness thatwe’ll discuss is the CIPP model, developed in the 1960s byDanielStufflebeam. Often referred to as theStufflebeammodel,CIPP is an acronymfor the following four areasof evaluation:
Context
Input
Process
Product
CIPPevaluates these areas when judging the value of a program. Unlike the Kirkpatrick Model and the Phillips ROI Methodology, CIPP is less about proving what you did and more about improving what you’re doing.Arguably, this makesit more useful for businessesand organizations.
The CIPP model was later expanded to includeSustainability, Effectiveness, and Transportability. It offers a decision-centered approach to the evaluation of programs.
CIPP model: How it works
The CIPP model aims to link evaluation with the decision-making that goes into running atraining course orprogram. Each of the above four aspects of evaluation – context, input, process,and product – are used provide ananalyticbasis for making the decisions that go into a program. These decisions can be groups into five main areas:
Planning
Structuring
Implementing
Reviewing
Revising
By examining each of these areas through the four aspects of evaluations, the decision-makers behind anytrainingcourse have a logical framework for making decisions.The CIPP model makes it easier for businesses to answer four main questions:
1. What do weneedto do?
This involves looking at the concerns, needs, attitudes and perceptions of the business and involves collecting and analyzing assessment data. This process helps businesses decide upon their most pressing goals. In other words, which area(s)do they need to prioritize? What should their objectives be?
2.How should weapproach training?
The next step is tobegin researchingsuccessful training materials or programs todecide onthe best approach. Shouldyouhire external experts to deliver the training? Or developyourown in-houseapproach? This step helps identify the right resourcesthat a company needs to meetit*objectives, and the steps they’ll need to take to get there.
3. Are we on the right track?
This stage involves continually monitoring the training program andfine-tuning its direction. By looking at factors such as the delivery, staff morale, arising conflicts and the results of testingdata, businesses can see how well the training is being implemented. This stage helps identify problems as they arise, giving the training providers time to fix or addressany concerns.
4. How successful was the program?
The final stage is to measure the outcomes of the training and compare them to the expected outcomes. This helps businesses determine the value of the training. Should it be continued? Modified? Or discontinued?
How CIPP works in the different stages of evaluation
Unlike the Kirkpatrick taxonomy or the Phillips ROImethodology, the CIPP model lets stakeholdersor decision-makers evaluate a training program before, during and after it has finished. This provides an opportunity to tailor the training to the specific needs of the participants, fine-tune it while it is being implemented, and assess its impact after it wraps up.
The four aspects of the evaluation – context, input, process, and product – can be applied as both formative evaluations (before the training) and summative assessments (after the training).
Using CIPP as formative assessment
To use the four areas of evaluation before the program, these are the types of questions you’d ask:
Context:Whatdo we need to do?
Input:How should we approach training?
Process:Are we on the right track?
Product:Does this program have a successful track record?
These questions help improve the quality of the training provided and ensure that the stakeholders’ goals are met.
Using CIPP assummativeassessment
CIPP can also be usedas a form ofsummativeassessment, to identify what went right and wrong in a training course or program. Here are some questions you may ask:
Context:Did the training address our needs?
Input:Was the training well designed?
Process:Did the training stay on the right track?Why? Why not?
Product:How successful was the program in meeting our goals?
The CIPP model is a helpful way of determining the right type of training that a business or organization needs and how best to implement and monitor it.
To learn more about theCIPPmodel, check outMichigan University’s evaluation center. They have checklists and anonline bibliography ofliteraturerelated to the CIPP model.
Conclusion
As you’ve seen, the three most widely used training evaluation models – the Kirkpatrick taxonomy, the Phillips’ ROI Methodology,and theStufflebeamModel – all have their own unique advantages and disadvantages. Finding the right one for your organization will depend on your budget and the time and resources you have available.
If you’re keen to find ways of evaluating the effectiveness of your course, download our free white paper here.
Related posts:
- 10 training effectiveness survey questions to ask
- How to measure training effectiveness in 4 simple steps
- Four good reasons to evaluate training effectiveness
- Are you spending millions on training without knowing its true effect?
- Why Measuring Training Effectiveness will Soon Become Standard
