Training is an expensive business. US organisations alone pour a staggering $101.8 billion into training annually. That’s a sum three times Iceland’s entire GDP. How can we make sure this is money well spent? The answer lies in effective training evaluation.
Training evaluation is the process of determining whether our learning initiatives are delivering meaningful change or merely consuming resources.
Unfortunately, according to this LinkedIn Workplace Learning Report, only 8% of CEOs say they see any impact from their organisational learning programmes. Even fewer (4%) said they saw a clear return on investment (ROI).
Perhaps this is because most evaluation processes focus on learner satisfaction, rather than real results. According to the CIPD, only 12% of learning practitioners evaluate their training’s wider impact on their organisation and society as a whole.
Thankfully, training evaluation helps you to shine a spotlight on your learning programme’s impact. This article will explore four evaluation methods, address common challenges, and provide practical implementation tips. Let’s get started!
What is Training Evaluation?
Training evaluation is a systematic process of assessing the effectiveness and efficiency of a training programme. To achieve this, you’ll need to collect and analyse data to understand whether your training has successfully accomplished its goals.
This data-driven approach empowers you to optimise resource allocation and make informed decisions about your training initiatives. In fact, data-driven organisations are 3x more likely to report significant improvements in decision-making.
Ultimately, it arms you with the knowledge you need to improve employee performance and organisational outcomes.
Why is Training Evaluation Important?
Imagine attempting to fly a plane without any dials or instruments. You’d have no way to measure altitude, speed, or direction. As a result, you’d have to fall back on your senses, which can be misleading, especially in challenging conditions.
The same is true when it comes to your training. You might believe it’s effective based on a limited selection of participant feedback or your own intuition, but without concrete data, you’re essentially flying blind.
Here are just a few of the reasons why training evaluation is essential:
- Measuring Effectiveness: If you’re not evaluating your training approach, you have no way of knowing whether it’s successfully meeting its objectives. In other words, you have no way of knowing which parts are working, and what needs to be refined further.
- Improving Your Approach: Training evaluation naturally surfaces what’s working well and what needs improvement. This provides you with the insights you need to create better optimised learning experiences and helps you to allocate your time and resources more efficiently.
- Demonstrating ROI: Demonstrating the tangible impact of your learning initiatives is a key challenge for L&D professionals. Thankfully, by showcasing the direct impact of your training programmes on business outcomes, it will become easier to secure budget allocation in the future.
- Better Decisions: Data-driven decision making is essential for optimising your training initiatives. By harnessing actionable insights, learning professionals can confidently align their training programmes with organisational objectives.
- Enhancing Performance: Training programmes aim to change behaviour and improve performance. Evaluation helps you to understand whether your employees are applying new skills on the job and identify areas where additional support may be required.
Still need convincing? A 2012 study of 150 training professionals revealed that the frequency of training evaluation is ‘positively related’ to knowledge transfer. In other words, the more you evaluate your training, the better learning outcomes you’ll see.
Key Evaluation Methods
Your choice of evaluation methods will largely depend on your specific training programme, its goals, and your available resources. In the majority of cases, you’ll use the following approaches to acquire an understanding of your training’s effectiveness.
- Pre- and post-tests: Tests and assessments help you to compare participant knowledge or skills before and after training delivery.
- Surveys and feedback forms: Feedback is the breakfast of champions. Collecting this information helps you to assess participant satisfaction.
- Performance appraisals: Appraisals can help you to assess changes in employee performance following training delivery.
- Observation: Similar to appraisals, observation can help you to understand whether participants are applying new skills in real-world settings.
- ROI analysis: A comprehensive return on investment analysis helps you to calculate the financial benefits of your training initiative.
- Focus groups and interviews: Sitting down with your learners helps you to collect qualitative data on participant experiences and perceptions.
Need a helping hand? Learning management systems and learning apps typically incorporate assessment, survey, and reporting tools, providing valuable data on learner engagement and performance.
Beyond these methods, there are several models of evaluation that you can use to assess the effectiveness of your training programme. In this article, we’ll explore four of the most popular approaches.
The Kirkpatrick Model
Introduced by Donald L. Kirkpatrick in 1959, the Kirkpatrick Model remains the gold standard for assessing training effectiveness. Its structured framework, evaluating training across four key levels, has proven to be remarkably durable.
- Reaction: This level measures participants’ satisfaction with their training. It focuses on their opinions and feedback about the training experience, including the instructor, materials, and overall delivery.
- Learning: By level two, the focus shifts to assessing whether participants acquired the intended knowledge, skills, and attitudes as a result of their training. This requires measuring the increase in knowledge or skill level compared to before the training intervention.
- Behaviour: Level three focuses on evaluating the extent to which participants apply what they’ve learned on the job. As such, it seeks to assess observable changes in behaviour and performance.
- Results: The final level determines the impact of training on organisational performance. This typically requires measuring the overall return on investment of your training by assessing its contribution to business objectives.
This structured approach helps organisations to gather a comprehensive understanding of the effectiveness of their training programmes. As a result, they will be in a better position to make data-driven decisions to improve future initiatives.
The Phillips ROI Model
The Phillips ROI Model is an extension of the Kirkpatrick Model created by Jack Phillips in the early 2000s. It specifically focuses on quantifying the financial return on investment of training programmes.
As such, it incorporates the original four levels of evaluation (reaction, learning, behaviour, and results) and uses them as the basis for a fifth level: return on investment.
As you might expect, this level focuses on converting the impact of your training into monetary terms and comparing it to the overall cost of the programme. Here are the key steps involved in calculating ROI:
- Identify Metrics: Start by determining the key performance indicators (KPIs) that are impacted by your training.
- Measure Improvement: Quantify the improvement in these KPIs following your training.
- Calculate Value: Assign a monetary value to this increase in performance. Ensure this is clearly stated and justified.
- Determine Costs: Collate all expenses associated with your training programme.
- Establish ROI: Divide your net monetary benefit by the total cost of your training programme. This will determine your overall return on investment.
For example, if a sales training programme generated an additional $50,000 in sales revenue and the total training cost was $10,000, the resulting ROI would be 5:1. Still unsure how to calculate ROI? Then check out our comprehensive guide.
Kaufman’s Model of Evaluation
Like Jack Phillips’ ROI Model, Kaufman’s Model is an extension and refinement of the Kirkpatrick Model. It simultaneously offers a more granular and wider-reaching approach to evaluating the success of your training programme.
For instance, Kaufman refines Kirkpatrick’s Level 1 (Reaction) by splitting it into two distinct components:
- Input: Evaluating the resources and materials used within your training programme.
- Process: Assessing the delivery and facilitation of your training programme.
Kaufman also sought to establish a broader perspective by adding a fifth ‘Mega-level’. This level examines the impact of your training on society as a whole. In other words, it goes beyond your organisation to consider external stakeholders.
Unfortunately, it’s often costly and impractical to gather data at this ‘Mega-level’. With that said, the value of Kaufman’s Model of Evaluation is the comprehensive view it provides of training effectiveness. It really does leave no stone unturned.
The CIRO Model
Finally, we have the CIRO Model, a framework that’s typically used to evaluate the effectiveness of management training programmes. This approach was developed by Peter Warr, Michael Bird, and Neil Rackham back in 1970.
Unlike the Kirkpatrick Model, which mostly focuses on learner outcomes, the CIRO Model emphasises organisational context and business objectives. Both models, however, utilise a four-step evaluation framework. Here’s how CIRO breaks down:
- Context: This step involves analysing your organisational environment and identifying specific training needs. In essence, you are establishing the ‘problem’ that your training is aiming to solve.
- Input: The second step focuses on the design, development, and delivery of your training programme in order to meet these specific needs. This includes your resources, materials, and facilitators.
- Reaction: By stage three, you’re ready to measure your learners’ satisfaction with their training. This is similar to Level 1 of the Kirkpatrick Model.
- Output: Finally, at stage four, you’re ready to assess the immediate outcomes of your training. Typically, this will cover increased knowledge, skills, or attitudes. This is similar to Level 2 of the Kirkpatrick Model.
Unlike the Kirkpatrick Model, which probes further into behaviour change and overall programme impact, CIRO primarily focuses on the initial stages of training. This means it offers an approach to evaluation that’s easier to implement but less comprehensive.
A Note On Evaluation Models
As you can see, most evaluation models build upon the foundational work of Donald Kirkpatrick. The optimal approach to evaluating training effectiveness ultimately depends on the specific goals, resources, and context of your organisation.
This may require combining models, or only reaching certain levels of evaluation. After all, this can be a costly and time-consuming process. It’s important to consider resource constraints and your desired depth of analysis when making these decisions.
As a guide, Jack Phillips suggests:
- All training programmes require Level 1 (Reaction) evaluation.
- About 90% of training programmes require Level 2 (Learning) evaluation.
- About 30% of training programmes require Level 3 (Behaviour) evaluation.
- 10-20% of training programmes require Level 4 (Results) evaluation.
- And only 5-10% of training programmes require Level 5 (ROI) evaluation.
As you might imagine, this final set of programmes are usually the most expensive initiatives, with the largest audiences. You’ll need to determine whether you can justify an evaluation investment on this scale.
Challenges in Training Evaluation
Unfortunately, many training evaluations fall short of their potential. Accurately assessing training impact is often hindered by a range of challenges. In this next section, we’ll explore these obstacles and offer some potential solutions.
- Time Constraints: There’s no getting around it. Conducting a full and thorough evaluation requires significant time and resources, which are in short supply in many organisations. We recommend adjusting the depth of your evaluation in line with the budget of your learning programme.
- Data Collection: Gathering accurate and reliable data for training evaluation can be challenging. This is especially true for behavioural and results-oriented metrics. To overcome this difficulty, we recommend building evaluation into your training implementation plan, to help normalise data gathering.
- Isolating Impact: Determining the precise impact of training on employee performance can be challenging due to the influence of other factors. Thankfully, implementing a control group allows for a comparative analysis, helping to isolate the effects of your training programme.
- Intangible Benefits: Quantifying the impact of training on intangible factors like employee morale or customer satisfaction can be difficult. While precise monetary values may be elusive, qualitative feedback and benchmarking can provide valuable insights into these areas.
- Long-Term Evaluation: The impact of your training could have a long tail, which makes it challenging to determine the appropriate time frame for evaluation. To address this, consider embedding data collection and analysis into your L&D processes for ongoing monitoring and evaluation.
Of course, overcoming these challenges requires careful planning, dedicated resources, and a commitment to continuous improvement in evaluation practices. But, with the right mindset, you can unlock a wealth of actionable insights.
Best Practices in Training Evaluation
To maximise the impact of your training programmes, let’s explore some evaluation best practice. These strategies will streamline your data collection process and help you achieve optimal results. Let’s dive in!
- Maintain Alignment: Always align your evaluation methods with your overall training objectives. After all, you’re going to have a lot of data to comb through. This laser-like focus prevents distractions and ensures you’re measuring what truly matters.
- Plan Properly: Effective evaluation begins with meticulous planning. This isn’t something you can freewheel. A well-structured evaluation process requires upfront consideration of data collection methods and analysis procedures.
- Involve Stakeholders: Ultimately, your goal is to deliver actionable insights to key stakeholders. You should collaborate with senior leadership to define critical data points and create compelling visualisations that effectively communicate your findings.
- Master Your Data: Collect both qualitative and quantitative data to gain a comprehensive understanding of your training programme. Remember you can use an LMS or analytics tools to streamline data collection and analysis.
- Communicate Effectively: Don’t let valuable insights go to waste by putting all this data into a black box. Share your findings with stakeholders in a clear and concise manner so they can be used to inform data-driven decision-making.
- Close the Loop: Remember, this data is a means to an end, not the end itself. Once you’ve analysed your evaluation results, implement necessary changes to enhance your future training programmes. Then go back and measure the impact of these changes!
Final Words
There we have it. Whilst evaluation might seem like an additional burden placed upon busy learning professionals, it’s actually an investment in the future of your organisation.
With a variety of evaluation methods at your disposal, the key lies in crafting a robust data collection plan. Remember, depth isn’t always necessary. Instead, you should seek to tailor your evaluation to the specific goals of your training programme.
By harnessing the power of evaluation, you can transform your training programmes from cost centres into strategic assets that drive organisational success. It’s time to step into the light and showcase your true value. Good luck!
Thank you for reading. Evaluation is one of many important steps in an organisational learning programme. Get the full breakdown in ‘The L&D Professional’s Handbook’. Download it now.
