Measuring the return on investment (ROI) of any major purchase is essential in all corporate settings. After all, identifying the value and quantifying the benefits of your investment helps you to justify the costs. But L&D teams still struggle to measure and demonstrate the ROI and value of their eLearning interventions.
Online learning holds several benefits, and most organisations use it as a significant part of their training delivery. But now that it’s so commonplace, the focus is shifting to determining the impact and ROI of online learning interventions.
In fact, it’s increasingly essential to measure (and prove!) that your eLearning programme provides value, improves organisational outcomes and results in sufficient ROI. Given today’s uncertain economic climate, it’s all too easy for low impact initiatives to end up on the chopping board.
With this in mind, let’s explore what eLearning ROI is and why and how you should measure it.
What Is ROI?
Return on investment, or ROI, is the ultimate measure of the success of your investment. ROI calculation takes the benefit or return of your investment and divides it by the cost or investment. As such, it shows the ratio of what you put in and get back as a result.
You can calculate your return on investment by following this formula:
In other words, ROI is the positive financial outcome your training programme achieves, divided by the amount you spent on the investment.
When you multiply this by 100, you’ll see your ROI as a percentage. Although ROI can be expressed in several ways, it’s usually presented either as a percentage or cost/benefit ratio.
A positive ROI means that your investment has paid off, while a negative ROI indicates losses. For example, if your ROI is 62%, this means that every pound you have invested has returned an additional 62 pennies. Well done!
ROI calculations allow you to convert both the benefits and costs of your investment into a single monetary value. This gives you a clear comparison and helps you to evaluate the effectiveness of your investment.
If you can establish a positive return on investment, you’re in a good position to effectively communicate your success and win more support for future initiatives. After all, this is a concept that’s understood by individuals across job roles and levels, from financial experts to executives and managers.
What Is eLearning ROI?
Just like any other investment, a successful online learning initiative needs to demonstrate value for money. As such, eLearning ROI is similar to traditional ROI. The only difference is that in this case you are focusing on determining the monetary benefits of your online training programme.
Calculating the ROI of your online learning programme involves comparing the costs of designing and rolling out your training courses with the benefits produced by your digital training. You can determine if your programme has been successful when you can demonstrate that the value and benefits outweigh the costs.
Monetary benefit derives from, for example:
- Increased customer satisfaction.
- Increased productivity.
- Improved revenue.
- Decreased costs.
Costs, on the other hand, typically include:
- Course development.
- Instructors.
- Online learning platform set up.
It’s beneficial to measure your eLearning ROI on multiple occasions. In particular, you’ll want to establish this before and after your training courses have been delivered. This gives you the projected and actual ROI of your training course.
The Importance of Measuring Your eLearning ROI
Calculating the ROI of your investment is critical to the operation of any enterprise. After all, it allows you to evaluate your performance and determine if your initiative has been profitable or not.
Segmenting your ROI calculations also helps you to determine how individual parts of your operations are performing. Combining and comparing these figures helps you and your business leaders to draw an accurate picture of the overall health of your organisation.
In the context of online learning, it’s essential to measure your eLearning ROI to establish the value of your training interventions. By successfully measuring and tracking your online learning ROI, your business can identify aspects of the initiative that may need refining. And, of course, it helps you to justify the use of a digital training approach to your entire organisation.
Challenges L&D Professionals Face
Sounds relatively straightforward, right? And it can be if your online training has a tangible goal that can be measured financially. If you’re well organised, your costs are fairly easy to define. It’s simple to multiply and divide numbers when it comes to abstract examples.
But determining the true value of your training programme is far trickier. In fact, finding out how much you earned by assigning your employees a certain course can be quite a challenge. This is especially true if the training unit is not directly related to a skill that has a direct monetary value.
To establish the success of your training programme, you will need to find a way to apply a monetary value to all learning outcomes. This may include hard to quantify outcomes such as customer satisfaction, increased compliance and reduced risk. If that sounds difficult, it’s because it is.
Without this additional data, your eLearning ROI metric won’t tell the whole story. As LinkedIn’s 2021 Workplace Learning Report puts it: “Quantifying the value of learning is not straightforward. Learning leaders have to rely on a wide spectrum of measures to gauge success and identify areas to improve.”
Prevailing Issues
With this in mind, it’s not a surprise that many learning and development professionals have faced challenges in delivering success metrics that their executives appreciate and understand. In fact, 81% of L&D teams say that they experience barriers when evaluating their initiatives. The most common problems are other pressing business priorities, lack of time, lack of funding and poor quality of data collected.
In fact, according to Emerald Works’ Report, just 8% of L&D professionals calculate the ROI of their learning programmes. Yes, you read that right! Fewer than 1 in 10 organisations actively measure the return on their training investment.
But all is not lost! 2021 Learning and Skills at Work report by Chartered Institute of Personnel and Development (CIPD) highlights that 76% of L&D teams evaluate the impact of their training initiatives in some way. This reveals that organisations are able to track business impact in abstract terms, but collecting data to state it in ROI terms is much more difficult.
Risks
Not evaluating and measuring the ROI of your training programmes is risky. According to Training Magazine’s Training Industry Report, US companies spend $87.6 billion on training initiatives per year. It’s important that this expenditure is being put to good use.
Considering all that money spent, your business leaders will want to know if they are getting a solid return on their investment. As such, it’s no surprise that 98% of L&D teams consider programme evaluation a priority.
To combat these challenges, L&D professionals should use an established methodology that includes various performance metrics. And we’re ready to help you do exactly that!
How to Measure eLearning ROI?
As explored, calculating the ROI of your online learning initiative is one of the best ways to showcase its value. Your business leaders are typically more interested in numbers, and presenting your training’s ROI is the best way to speak their language. As such, you need to have a method in place to measure the success of your digital training programme.
There is no official model to measure online learning success. However, two analysis methods have become highly popular.
Let’s have a look!
The Kirkpatrick Model of Training Evaluation
Kirkpatrick Model was developed in the 1950s by Donald Kirkpatrick, a professor and training specialist. The model features four levels and covers everything from learners’ reactions to training results.
The model became popular due to its convenience. And to this day, it is used by learning and development professionals around the world to measure the effectiveness of their training interventions. His model evaluates the impact of training by assessing (1) reaction, (2) learning, (3) behaviour and (4) results.
Let’s take a closer look at the four levels of the Kirkpatrick Model of Training Evaluation to determine how you can use it to evaluate your training.
Level 1 – Reaction
The first level focuses on how your learners are responding to their online learning. This helps you to understand how learners feel about their training. This level is essential, as it alerts you to any issues with your training programme. This allows you to clear up any confusion, or refine your initiative as necessary. You can also identify how effective they consider your training to be.
You can find out, for instance:
- How well your learners are participating.
- If your training material is relevant to your learners.
- How satisfied your learners or instructors are with their digital training solution.
You should determine and measure this reaction by gathering feedback from your learners. You are able to do so by using your learning management system to push out post-training questionnaires or surveys. Using parameters like the following will help you to achieve this:
- The number of days it takes your learners to complete a learning course.
- The projected data of dropouts from the training programme.
- Course content ratings and reviews.
- Logins to the learning platform.
- Learner progression.
The first level is popular with training professionals as it’s relatively easy to administer and measure. Furthermore, it provides you with immediate information about how valuable learners find their training interventions.
However, the Reaction Level alone is not an accurate measure of the effectiveness of your training interventions. After all, surveys are rarely in-depth enough to indicate the extent to which participants have internalised their training. Your learners may love their training materials, without getting very much from them. Nor does it offer direct insights into how your organisation is benefitting from the training programme.
And that’s why you need to level up to more accurately evaluate the effectiveness.
Level 2 – Learning
The second level determines what your course participants learned during their online learning experience. Here, you evaluate the increase in knowledge and skills. You can evaluate this through assignments, tests, quizzes or knowledge demonstrations.
The second level is also essential as it helps L&D professionals determine if they have met their training objectives. You’ll also be able to identify any knowledge or skills gaps, which helps you improve your course content.
You can use your learning management system (LMS) to measure effectiveness. For example, Growth Engineering LMS offers a robust reporting suite that helps you to measure your eLearning ROI. You can quickly run reports on learners or content using filters and custom settings. Filtering the results by question helps you to identify any knowledge gaps.
While Level 2 gives you bucket loads of useful information, it’s not substantiated by hard facts. Therefore, you cannot rely exclusively on the second level to measure the effectiveness of your training programme.
Level 3 – Behaviour
Kirkpatrick’s third level aims to determine the continuity between learning and practice by assessing how learners apply their new knowledge in the workplace. As such, this level is used after learners complete a course, to measure how their behaviour has changed as a result of them enrolling in the training programme.
This level is extremely important as behaviour change is the ultimate goal of every training initiative. It validates the investments you have made to provide training. After all, if your learners don’t change their behaviour, you won’t affect any change across your business. And if nothing changes, you won’t be able to improve performance or any other relevant metrics.
However, behaviour change can be difficult to measure and assess. In addition, your learners might struggle in applying their new skills in real-life situations. If your learners don’t demonstrate their newly acquired skills or knowledge, they might need knowledge reinforcement.
Traditionally, this would have been measured by managers and supervisors, who evaluate learner’s progress and behaviour change. Now, technology can be used to assess whether a learner can apply their knowledge to perform tasks. And by making sure your KPIs align with your aim to change behaviour, it suddenly becomes much easier to analyse!
In a nutshell, the third level helps you to establish whether your training has inspired your learners to take action. If not, it gives you an opportunity to reinforce key information that encourages the right behaviours.
Level 4 – Impact
Kirkpatrick’s final level measures the business results that occur because of your training. Typically, this is measured through increased productivity. However, you can also assess reduced costs, improved quality, enhanced efficiency, increased sales or employee retention.
Level four should allow you to clearly determine if you have achieved your desired learning objectives by implementing training. As such, the aim is to produce evidence that your training has had a measurable effect on your organisation’s performance.
Arguably, this is the most important level when it comes to assuring your stakeholders that the investment has been worthwhile. However, the challenge is to identify what results are specifically connected to your training interventions and how to quantify your impact.
Due to these limitations, the Kirkpatrick Model of Training Evaluation has been expanded by various authors. While Kirkpatrick’s 4-Level model remains the most renowned performance evaluation approach, it’s not without issues. In fact, the methodology has faced vast critique, which has led to new models being born.
Phillips ROI Model
Jack Phillips was one of those critics. He refined Kirkpatrick’s model and presented the Phillips ROI Model, a 5-Level evaluation methodology. Phillips added Level 5 (ROI), which gives stakeholders the hard numbers they are craving.
Phillips ROI Model is now best known as an expanded version of the Kirkpatrick Model. In today’s context, adding Phillips’ ROI calculation as the fifth level makes the framework even more beneficial and relevant. Let’s have a look at how it helps you to measure your eLearning ROI!
Level 5 – ROI
Meet the fifth level of learning evaluation. Phillips Model added Level 5 to help learning professionals determine the return on their training investment. As such, this level focuses on converting the results from the last step into pure financial terms.
By determining the business impact of your learning programme at Kirkpatrick’s Level 4, you can then translate it into monetary value. You can then compare it with your overall programme costs to measure the ROI of your eLearning training programme. These costs include, for instance, programme development and delivery and the opportunity cost that arises when you take learners off the proverbial shop floor.
You can look at your training evaluation as a chain reaction:
- Learners react to their training (Level 1).
- Learners pick up new skills and knowledge (Level 2).
- They then apply these skills on the job (Level 3).
- This application drives business impact (Level 4).
- And this impact can be communicated in ROI terms (Level 5).
When you evaluate the results at each level, it’s easier to identify the cause of negative ROI. For instance, did your learners not learn the information they needed to at Level 2?
Phillips’ Evaluation Process In Practice
As established, calculating eLearning ROI isn’t easy. But by following Phillip’s process, it becomes much easier to understand. Let’s take the following step-by-step process as an example:
Step 1. Collecting Pre-Programme Data
Collecting data before your training takes place will give you a baseline measure for comparing metrics before and after training.
Step 2. Collecting Post-Programme Data
The next step involves collecting data after your training is complete. You can do this, for example, by gathering data from existing records or conducting follow-up surveys.
Step 3. Isolating the Effect of the Training Programme
You need to be able to identify the root cause of these results. Did your training programme bring them about, or could they have another source? To determine this, you should focus on all the different factors that may have contributed to the performance improvement.
Other potential sources include bonus programmes, marketing material or seasonal effects. You have a few tools you can use to help isolate the impact of your training programme. For instance, you could utilise control groups (we find this to be particularly effective), trend line analysis or apply appropriate forecasting methods.
Step 4. Converting Data to Monetary Benefits
The fourth step involves converting your data to monetary values. As such, you need to put a price tag on each data item that is connected to your training programme. Make sure to do this for both hard objective data and soft subjective data.
However, you can set aside estimating these kinds of benefits for now! You will focus on them in Step 7.
Step 5. Tabulating Programme Costs
Once you have collected data, you need to be able to compare it to your programme costs. Calculate the costs, including expenditure on design and development, instructors, materials, your online learning platform and administration.
Step 6. Calculating the Return
You then have all the values you need to measure your eLearning ROI. Using the previously presented return on investment formula will give you your ROI percentage.
Step 7. Identifying Intangible Measures
The last step focuses on identifying the intangible, non-monetary benefits that are associated with your online learning programme. These benefits are difficult or too costly to quantify.
Typical intangible variables include employee and customer satisfaction, team effectiveness and employee retention.
Common Criticisms of the Phillips ROI Model
Even though the Phillips ROI Model has gained a reputation as the enhanced version of the Kirkpatrick model, it’s not without its detractors. The most common critique touches on the timing and usefulness of the model.
1. Timing
Phillips ROI Model only calculates ROI after the training programme has been delivered. As such, if you calculate that your training initiative costs more to deliver than the value it has provided, it’s too late to make any changes.
However, Phillips responded to this issue by writing another piece with his wife, Patti Phillips. It’s called ‘The Consultants Guide to Results-Driven Business Proposals: How to Write Proposals that Forecast Impact and ROI‘ and it presents forecasting as a solution to this problem.
By following his guide and forecasting ‘ROI estimates’, you can plan a successful learning programme. You already have plenty of information at hand that helps you make more informed decisions. You can, for example, determine business goals and budgets and revise your plans before making any final decisions.
2. Usefulness
Another common criticism is that knowing the ROI of a training course offers little practical benefit to your organisation. Whilst it’s nice to know, it’s not essential. You can already identify if your programme has been successful by using the first four levels of evaluation. As such, the value of level five has come into question. Is it worth the time and costs involved in accurately calculating your training’s ROI?
Jack Phillips has responded to this criticism by noting that ROI calculations are not necessary for all training programmes or courses.
Phillips recommends the following approach:
- Evaluating all training programmes at Level 1.
- Measuring 90% of programmes at Level 2.
- Then evaluating 30% of programmes at Level 3.
- Focusing efforts on evaluating 10-20% of programmes at Level 4.
- And finally, only evaluating 5-10% of programmes at Level 5.
As such, not all training programmes need to reach Level 5. If your training programme is small and simplistic, it’s enough to only reach Level 1 or Level 2. But if you’ve made a significant investment into your training, you may wish to extend your evaluation up to Level 4 and Level 5 to measure the ROI of your eLearning programme.
Determining the Right Model
As explored, the Kirkpatrick Model comes with issues. And while Phillips ROI Model is perhaps the most commonly cited expanded version of Kirkpatrick’s evaluation methodology, it’s not the only one.
Ever since the model was created back in the 1950s, it has been modified, extended, tinkered and revised by successive generations of L&D theorists and practitioners. To name a few, the following have refined and added aspects to the model over the years:
- The CIRO approach by Warr et al. (1970)
- Hamblin (1974)
- Bushnell (1990)
- Sleezer et al. (1992)
- Fitz-Enz (1994)
- Kaufman (1994)
- Bernthal (1995)
- The Indiana University approach by Molenda et al. (1996)
- The KPMT model by Kearns and Miller (1997)
The extent of different models and versions emphasises that one ROI model is not necessarily meant for every training intervention. Luckily, you have plenty to choose from.
The wide selection of choices may feel a little baffling at first, but don’t fret. There is a clear principle that can guide you in your choice of which variant to use. After all, many of these variants have a specific focus that makes them more suited to particular aims.
In the end, it all comes down to your training objectives and what you are trying to achieve by evaluating your training programme.
Let’s have a look at few examples of how this works in practice:
1. Proved ROI
If your main aim is to measure your monetary savings, tangible profits and financially quantifiable improvements in productivity based on your eLearning programme, Phillips ROI Model is your best bet.
As explored, Phillips added a fifth Level to the Kirkpatrick Model, providing us all with a methodology for calculating the financial impact of our training interventions. However, this method can be time-consuming and expensive. As such, he recommends using this model selectively.
2. Improved Training
When your main driver is establishing whether a training programme is working or not, Brinkerhoff’s Success Case Method (SCM) is your friend. The method focuses on identifying the most and least successful cases within your training. This could include, for instance, who scored well and who failed. It then uses qualitative methods to go in-depth into the reasons behind this success or failure.
You can use SCM on its own or combined with other methodologies. Either way, it should be your go-to model when you want to analyse improvements in the quality and effectiveness of your training programme.
3. Successful Learning Transfer
If your focus is on determining how successful your learning transfer has been, you can use the Learning-Transfer Evaluation Model. LTEM focuses on learning transfer and has eight tiers:
- Attendance
- Activity
- Learner perceptions
- Knowledge
- Decision-making competence
- Task competence
- Transfer
- Effect of transfer
LTEM differentiates between knowledge, decision-making and task competence. This helps you to target more meaningful learning outcomes.
Will Thalheimer (the mastermind behind this model) notes that these requirements are more ‘robust’ than those presented to us by the Kirkpatrick evaluation model. As such, they provide us with more appropriate ‘guideposts’ and a deeper insight into the learning process.
Final Words
It’s essential to ensure your online training provides value for money. However, determining the benefits, value and return on investment is not always as easy as it sounds. But it doesn’t have to be hard!
Today we have looked at some commonly used evaluation models you can use to measure the ROI of your eLearning interventions. These give you an excellent starting point when it comes to determining whether your online learning programme has been worth the investment. While they require some effort and resources, finding the right measures and methodologies for your organisation will pay off.
Check out our guidebook, ‘How to Win Your Training Budget’, to supercharge your training evaluation. You’ll find out how to drive meaningful ROI from your training programmes!
