What happened to American Donald Kirkpatrick is quite rare. Half a century ago, he proposed a conceptual approach to evaluating training programs that has become classic. Any evaluator today simply must know the four levels of evaluation “according to Kirkpatrick.”
It is interesting that there is still controversy surrounding the approach proposed by Kirkpatrick, but his training evaluation model remains one of the basic ones, and his books are the most cited. The classic is still alive today, although it has already retired from active business. He had a brilliant career; wrote several books that became bestsellers and brought him worldwide fame; was the president of one of the most authoritative professional associations of trainers and consultants in the world - the American Research and Development Society (ARSD). Today, Donald Kirkpatrick continues to engage in social work and maintains contacts with the professional community. He continues to publish articles and give lectures and seminars at major forums.
So, what did Kirkpatrick suggest?..
Kirkpatrick views assessment as an integral part of the training cycle, which includes 10 stages :
- Determining needs.
- Setting goals.
- Definition of subject content.
- Selection of training participants.
- Formation of an optimal schedule.
- Selection of appropriate premises.
- Selection of appropriate teachers.
- Preparation of audiovisual media.
- Program coordination.
- Program evaluation.
- Justify the existence of a training department by showing how this department contributes to achieving the goals and objectives of the organization.
- Decide whether to continue or terminate the training program.
- -Get information on how to improve the training program in the future.
Kirkpatrick himself believes that in most cases the assessment is carried out in order to understand how to increase the effectiveness of training, what ways can it be improved. In this regard, it is proposed to answer the following 8 questions:
- To what extent does the training content meet the needs of the participants?
- Is the choice of teacher (Kirkpatrick uses the word “leader” here, which has a completely different meaning in Russian) optimal?
- Does the teacher use the most effective methods to maintain participants' interest, impart knowledge, and develop skills and attitudes?
- Are the training conditions satisfactory?
- Are participants comfortable with the class schedule?
- Do audiovisual aids improve communication and maintain participant interest?
- Was the program coordination satisfactory?
- What else can be done to improve the program?
Note that all questions except the first and last are formulated by the author as closed (requiring “yes” or “no” answers). From the point of view of formulating task questions for assessment, this form of questions is not always good. However, we adhere to the author's text.
Kirkpatrick believes that in most cases, evaluation is limited to the use of post-training questionnaires - studying the immediate reaction of participants to the training. He calls these questionnaires “smile-sheets,” meaning that participants most often use the questionnaires to express gratitude. A more complex and in-depth assessment is not carried out because
- it is not considered urgent or important,
- no one knows how to carry it out,
- management doesn't require this
- people feel safe and do not see the need to “dig” deeper,
- there are many things that are more important to them or that they prefer to do.
Four levels Four levels, according to Kirkpatrick, determine the sequence of assessment of training (training). He writes: "Each level is important and affects the next level. As you move from level to level, the assessment process becomes more difficult and time-consuming, but it also provides more valuable information. No level can be skipped simply because to concentrate on what the coach considers most important" (it should be noted that many experts do not agree with this statement by Kirkpatrick). Here are the famous four levels according to the author:
- Level 1 – Reaction
- Level 2 – Learning
- Level 3 – Behavior
- Level 4 – Results
Reaction
Evaluation at this level determines how program participants respond to the program. Kirkpatrick himself calls it customer satisfaction assessment. When training is conducted in-house, the response of participants is not always interpreted as customer satisfaction. The fact is that participation in such trainings is mandatory. People simply have no choice. The company's management determines the need for this training and obliges employees to take part in it. It would seem that in this case, we need to talk about the reaction of management. Kirkpatrick emphasizes that in this case too the reaction of the participants is a very important criterion for the success of the training for at least two reasons .
- Firstly, people one way or another share their impressions of the training with their management, and this information goes higher. Consequently, it influences decisions about continuing training.
- Secondly, if participants do not respond positively, they will not be motivated to learn. According to Kirkpatrick, a positive reaction does not guarantee the successful development of new knowledge, skills and abilities. A negative reaction to training almost certainly means a decrease in the likelihood of learning.
Learning
Learning is determined as changing attitudes, improving knowledge and improving the skills of participants as a result of their completion of the training program. Kirkpatrick argues that changes in participants' behavior as a result of training are only possible when learning occurs (attitudes change, knowledge improves, or skills improve).
Behavior
At this level, an assessment is made of the extent to which the participants' behavior has changed as a result of training. Kirkpartick points out that the lack of changes in participants' behavior does not mean that the training was ineffective. Situations are possible when the reaction to the training was positive, learning occurred, but the behavior of the participants did not change in the future, since the necessary conditions for this were not met. Therefore, the lack of change in the behavior of participants after the training cannot be a reason for making a decision to terminate the program. Kirkpartick recommends that in these cases, in addition to assessing the reaction and learning, check the presence of the following conditions :
- Participants' desire to change behavior.
- Participants have knowledge of what and how to do.
- The presence of an appropriate socio-psychological climate.
- Reward participants for behavior change.
Speaking about the socio-psychological climate, Kirkpatrick refers primarily to the immediate supervisors of the training participants. He identifies five types of “climate”: prohibitive, discouraging, neutral, supportive, demanding. The manager’s position, accordingly, changes from a prohibition on changing behavior to a requirement to change behavior after the end of the training. Kirkpatrick believes that the only way to create a positive climate is to involve leaders in curriculum development.
results
Outcomes include changes that occurred as participants completed the training. As examples of results, Kirkpatrick cites increased productivity, improved quality, decreased accidents, increased sales, and decreased employee turnover.
Kirkpatrick insists that results should not be measured in money. He believes the changes listed above could, in turn, lead to increased profits. Kirkpatrick writes: "I laugh when I hear that professional trainers must be able to demonstrate the benefit to the client in terms of return on investment in training. I think the same thing about the relationship between training programs and profit. Just imagine all the factors that influence profits! And you can add them to the list of factors influencing return on investment."
According to Kirkpatrick, assessment at this level is the most difficult and expensive. Here are some practical tips that can help you evaluate your results:
- if possible, use a control group (no training)
- carry out the assessment after some time so that the results become noticeable,
- conduct pre- and post-program assessments (if possible),
- conduct the assessment several times during the program,
- compare the value of the information that can be obtained through the assessment and the cost of obtaining this information (the author believes that conducting an assessment at level 4 is not always advisable due to its high cost).
How Kirkpatrick's Book is Organized Kirkpatrick, D. L. (1998). Evaluating training programs: the four levels. San Francisco: Berrett-Koehler Publishers, Inc.
The book is divided into two unequal parts. The first is a description of the 4-level model, theory (about 70 pages). The second part (about 200 pages) is a detailed analysis of several case studies and examples of tools used to conduct assessments at different levels.
Based on materials from the Process Consulting company newsletter (processconsulting.ru/bulletin1.pdf)
Term type: Definition
Term: Kirkpatrick model
Alternative terms: -
Short Description: Kirkpatrick model - a four-level model for assessing the effectiveness of training
Long Description:
Kirkpatrick model- four-level model for assessing the effectiveness of training:
Reaction (emotional level)
Mastery (level of knowledge)
Behavior (skill level)
Result
Reaction (emotional level)- emotional assessment of learning by participants . On the one hand, a positive reaction as such is not of great importance, since “liked training” does not at all mean “effective training.” It may not change the participants in any way. On the other hand, this factor should not be underestimated: satisfaction with the training is the interest, attention and positive attitude of the participants, necessary for successful training.
Mastery (level of knowledge) shows what knowledge, skills, techniques and methods have been acquired by the training participants. Acquiring new knowledge is an important, but insufficient result of training. To gain knowledge, it is not necessary to undergo training - just read a book or attend a lecture. In addition, the acquisition of new knowledge in itself, without the ability and desire to apply it, often does not affect the effectiveness of a person’s work. A more valuable result of the training may be that the participants receive “realizations,” so-called “insights,” when the training participant receives some knowledge not from the trainer, but as a result of analyzing the experience gained during an exercise.
Behavior (skill level) - changing the behavior of the participant in a work situation, applying the acquired skills in work conditions. Behavior change shows that participants are putting into practice the knowledge and skills acquired during the training. This is a critical performance indicator, since the main goal of the training is to improve business performance by improving the behavior of participants. After all, new knowledge and skills are useless if they are not applied.
Result- Measurable results are identified (for example, improved quality of service, reduced waste, etc.). Companies usually have data on key, integral business indicators, such as sales volume, profit, costs. It is clear that the training provided is not the only factor that influences these indicators; there are many other factors, both external and internal. And it is quite difficult to accurately measure the contribution of training, for example, to increasing sales volume, especially if it was training on leadership skills or self-organization skills. The costs of researching this indicator can be many times higher than the cost of the trainings themselves, so this level of assessment is recommended only in the case of long-term and expensive programs.
Without assessing the effectiveness of the learning process, it is impossible to build a training and development system that provides the necessary business results. Unfortunately, the potential of the most common assessment model - D. Kirkpatrick's - has not yet been fully used by practitioners. And most of our HR managers are not at all familiar with the features of its latest version.
In 1954, Donald Kirkpatrick defended his PhD thesis at the University of Wisconsin (USA) on the topic “Evaluating the Effectiveness of Program Management.” He proposed a short formula to describe the learning cycle: reaction - learning - behavior - results. Dividing the learning process into stages helped explain how to ensure that new skills are applied in the workplace, without which the desired results cannot be achieved. In addition, practitioners received tools to evaluate the effectiveness of each stage of training. In 1959, D. Kirkpatrick wrote a series of articles for the ASTD Journal*, in which he clearly formulated criteria for all four levels of assessment ( table).
Four levels of learning assessment by D. Kirkpatrick
|
Levels |
What is being assessed |
Key questions |
|
Level 1: |
How participants react to a learning event | Did the participants enjoy the learning process? What do they plan to do with their new knowledge and skills? |
|
Level 2: |
To what extent participants have acquired knowledge, skills and formed the necessary relationships upon completion of the training event | What skills, knowledge, attitudes have changed after training? How significant are these changes? |
|
Level 3: |
How participants apply what they learned during training in the workplace | Did participants change their behavior in the workplace after the training? |
|
Level 4: |
To what extent have the intended results been achieved as a result of the training? | Do changes in member behavior have a positive impact on the organization? |
Kirkpatrick's early papers stimulated further research into the evaluation of training effectiveness (mainly at Levels 1 and 2). In the 1970s, Kirkpatrick's four levels were already widely used by many organizations around the world, over time they were formalized into a holistic evaluation model (Four Levels TM Evaluation Model) and adopted as standard for assessing professional learning. Throughout the 1980s, many different assessment methods and tools were developed, but practitioners' attention remained focused on levels 1 and 2.
It was not until 2005 that Donald Kirkpatrick proposed a Level 3 (behavior) assessment tool, which he described in his book Transferring Learning to Behavior, co-authored with his son Dr. James D. Kirkpatrick. Ensuring the effective application of learning results in real activities (transfer of knowledge and skills) continues to be one of the most important tasks for many training organizations today.
|
Changes in recent years:
|
In 2006, with the third edition of Evaluating Training Programs, Kirkpatrick greatly expanded the scope of the four-level model. Its focus is now on Level 4 (Outcomes), so the model can be used to evaluate the effectiveness of both training programs and the change management process, and in addition - to demonstrate the business value of the training function as a whole. What is important, the algorithm for working with the model has been revised: it is now proposed to start the assessment “from the top” - from level 4, and then consistently move “down” - to less complex levels ( rice. 1). According to the author, this will allow training specialists to focus their efforts on achieving planned business results and supporting the required behavior.
Click image for a larger view
Rice. 1. Kirkpatrick's Modified Model: "The End is the Beginning"
In 2007, Donald and Jim Kirkpatrick developed their ideas in Implementing the Four Levels, in which they offered practitioners a fundamentally new approach: formation of a “chain of evidence”(chain of evidence) for managers. This algorithm allows learning professionals to gather compelling evidence to help show executives how value is being created for the business.
Algorithm "chain of evidence":
- focuses managers' attention on levels 3 and 4, where it is about the impact of training on achieving planned results and creating business value;
- helps to use the four levels model to measure the result obtained, and at all stages, starting from the moment the training initiative arises.
This is its fundamental difference from the traditional approach, which largely evaluates the effectiveness learning process(since it is focused on levels 1 and 2, managers' attention is mainly focused on training costs).
According to the developers of the new concept, in order to guarantee results, it is still necessary before the start of training define:
- Expected results;
- key indicators;
- measurement and evaluation methods.
And that's not all the changes! The authors added another fifth level of assessment to the model (Level 5: “Return on Investment”). Assessing the effectiveness of training at this level allows you to get an answer to a key business question: “Was it worth spending money on training”?
|
Strengths of the Kirkpatrick Model(from the experts' point of view):
Model limitations:
|
The Kirkpatrick model has been working successfully for over 50 years. But significant changes in methods of management and business organization required rethinking the role of the learning function In the organisation. At the same time, as the Kirkpatricks point out, many training professionals are still unwilling to accept responsibility for anything beyond the design and implementation of training programs (especially the results that trained people demonstrate in the workplace).
Continuing to creatively develop his approach, Jim Kirkpatrick, now co-authored with his wife Wendy (Wendy Kayser Kirkpatrick), wrote the book Training on Trial: How Workplace Learning Must Reinvent Itself to Remain Relevant. In fact, this is a “guide to change” for corporate trainers and training company specialists. Experts strongly recommend that practitioners in times of crisis:
- rethink your role;
- actively expand the areas of your competence by gaining experience in business units;
- increase your influence beyond the classroom - providing solutions that bring tangible (measurable) business results.
But so far, change in the heads of internal and external trainers is happening very slowly: many of them are still trying to prove their indispensability using indicators of “attendance level” and “evaluation data” (level 1), as well as “test results” (level 2). At the same time, benchmarking studies convincingly demonstrate that those who start at level 1 very rarely reach the highest levels of assessment.
The authors believe that this persistence of training professionals clearly illustrates the persistence of the myth that simply participating in a training event automatically improves business results (Level 4). But today stakeholders regard such “naivety” as evidence of inadequacy, which literally provokes them to ask questions: “How much does this pleasure cost? What benefits does it bring?
Jim and Wendy Kirkpatrick believe that the reputation of the training function in the eyes of managers is very serious. To improve the situation, training professionals are no longer content with “just” developing new programs and training methods; today they must change the way they operate, learn how to create real business value and convincingly demonstrate the importance of these values.
The authors propose a new principle that will help to radically reconsider the approach to planning and developing training programs: “The end is the beginning!” As a starting point, they advise coaches to discuss with managers:
1) expectations of business leaders;
2) their understanding of success (what the desired outcome “will look like”);
3) what observable and measurable indicators of success will be used to evaluate the outcome (level 4).
The result of the work at this stage should be a set of HR metrics, on the basis of which a new indicator is calculated: return on expectations, ROE (similar to ROI - return on investment). Training professionals should then discuss with line managers which behaviors are critical to achieving the intended results (Level 3). Only then can they begin to design training activities (Levels 1 and 2) that will provide the critical employee behaviors necessary to improve employee effectiveness in the workplace.
Only on this basis can we ensure:
- effectively applying new skills and consolidating behavioral changes in the workplace (Level 3);
- achieving business goals (level 4);
- implementation of tasks in the field of talent management (level 4).
Thus, Level 4 metrics become the cornerstone in determining the scope of responsibility of all stakeholders. In turn, a clear delineation of areas of responsibility is a basic condition for the successful implementation of the business partnership model.
Another improvement of D. Kirkpatrick is the development of a new “Model of Goal Achievement.” A special place in it is occupied by “drivers” (or “amplifiers”). Drivers are all processes and systems ( rice. 2), which:
- strengthen actions (behavior);
- control procedures (monitoring);
- Reward demonstrating critical behavior in the workplace.
Rice. 2. Drivers (“amplifiers”) - encourage or prevent the consolidation of new behavioral patterns
Effective performance of work tasks is supported by the combined impact of the behavior assessment process (Level 3) and drivers. Without this support, only about 15% of new knowledge, skills and attitudes are successfully put into practice, reducing the value of learning to business.
Kirkpatrick's new Achievement Model also allows practitioners to go beyond the actual training. It helps people develop plans to achieve goals that include specific activities/actions (and performance measurement at all four levels). Step-by-step planning, a clear understanding of the end goals/results, close monitoring of the implementation of plan activities, and objective evaluation of each stage of the process increase the likelihood of success.
In addition, the Kirkpatrick model provides training consultants and consulting service providers with a methodology and tools so that they can convincingly demonstrate to their clients what results will be obtained from the implementation of proposed training and development programs.
New developments from the Kirkpatrick family give HR professionals effective tools that allow them to truly become business partners, clearly demonstrate the power of business partnerships between coaches, line managers and top managers. (Questionnaire by D. Kirkpatrick “Business Partnership”, see application).
|
Application Business Partnership Questionnaire (Kirkpatrick SM) Rating scale:
Interpretation of results To assess how well the department (specialist) responsible for training meets the interests of the business, use the scale:
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
A simplistic view of learning is widely accepted among learning professionals and managers. This approach has given rise to a number of myths that steadily circulate among managers: Myth 1:“Anyone who has ever learned anything is an expert in learning.” In fact:
Myth 2:“Experts can identify training needs.” In fact:
Myth 3:“Response to learning = learning.” In fact:
Myth 4:“Learning directly (by itself) leads to behavior change.” In fact:
|
Why evaluate learning?
In the article “Techniques for Evaluating Training Programs,” Don Kirkpatrick wrote, “I would encourage training directors to be proactive in evaluating training programs before the day of reckoning comes.” What's surprising is that the ASTD Journal article was written in November 1959! Over the past half century, the thesis that the learning process, its costs and results should be evaluated, it would seem, has become a banality. But despite this, staged training assessments, unfortunately, have not changed the “modus vivendi” of corporate trainers.
- improvement of training programs;
- increasing the efficiency of employee training;
- demonstrating the value of the training function to senior managers.
Traditionally, the efforts of training professionals have focused on the most obvious results: the emotional response of participants after completing a training program. Unfortunately, many existing practices are inadequate and need improvement.
|
To analyze participants' opinions of the training program (Level 1: Reaction), trainers developed a variety of tools. They are called differently: “comment questionnaires”, “reaction questionnaires”, “smile sheets” or “happiness sheets”, etc. What should be the form that will allow you to get maximum information and at the same time require a minimum of resources? We offer some tips.
You can start with negative or positive reviews - it doesn’t matter.
Why is it necessary to develop a special form? Firstly, we are talking about clients (it does not matter, internal or external), so it is important that they are satisfied with the program. Some colleagues, managers or future participants in training programs will probably ask those who have already completed the training: “What did you think of this course?” It's hard to imagine. What damage will be done to the learning function if people give negative feedback about the experience gained: “I just wasted my time!”, “Nothing that would be useful in work” or “Give up this if you can”... Secondly, most likely, negative rumors will reach one of the top managers. The consequences of this can be tragic for the training department: most likely, the manager will not bother to analyze the general opinion about the course, but will conclude that it is ineffective (with all the ensuing consequences). Therefore, asking for the opinions of the participants is important, first of all, for the trainers themselves! Excerpt from the book “Four Steps to Successful Training,” Donald L. Kirkpatrick and James D. Kirkpatrick, M. - HAR Media, 2008. |
With the growing complexity of business conditions (increasing competition, the increasing role of the human factor, financial crisis, etc.), enterprises are increasingly striving to reduce costs and finance activities that produce predictable results and the highest return. Ultimately:
- Top managers require evaluation of the effectiveness of training at higher levels - Level 3: “Behavior” and Level 4: “Results”.
- Many companies abandon training programs whose value cannot be justified from a financial point of view (and, unfortunately, fire training specialists “until the situation improves”).
This suggests that today it is no longer individual trainers/training companies that are in serious danger - in many cases the need for the training function itself is being questioned! By developing a “rescue strategy,” the Kirkpatricks suggest that coaches urgently change their understanding of the business world and master new roles in the organization. The most important conditions for change:
- identify business needs;
- achieve practical (!), not just outstanding (!), but exceeding expectations (!) results;
- measure (in numbers) your contribution to the success and development of the company and convincingly demonstrate it to all key stakeholders.
The most important thing for trainers is to learn to act within a “project framework”, as is common in other functions: to provide an economic justification for obtaining funds (investments) for projects, to plan and budget their implementation in advance.
__________
* ASTD Journal- Journal of the American Society for Training & Development (ASTD)
As has been mentioned several times above, the problem of assessing the effectiveness of training still remains relevant for many companies involved in the development of their personnel.
It is interesting that there is still controversy surrounding the approach proposed by Kirkpatrick; back in the late 1950s, he formulated the now famous 4 levels of performance assessment, but his training evaluation model remains one of the basic ones, and his books are the most cited. Kirkpatrick views assessment as an integral part of the training delivery cycle, which includes 10 stages:
1. Determination of needs.
2. Setting goals.
3. Definition of subject content.
4. Selection of training participants.
5. Formation of an optimal schedule.
6. Selection of appropriate premises.
7. Selection of appropriate teachers.
8. Preparation of audiovisual media.
9. Program coordination.
10. Program evaluation.
Kirkpatrick believes that in most cases, evaluation is limited to the use of post-training questionnaires - studying the immediate reaction of trainees to the training. He calls these questionnaires “smile-sheets,” meaning that participants most often use the questionnaires to express gratitude.
Kirkpatrick's four levels define the sequence in which learning assessments are conducted. He writes: “Each level is important and affects the next level. As you move from level to level, the assessment process becomes more difficult and time-consuming, but provides more valuable information."
· Level 1 - Reaction
· Level 2 - Learning
· Level 3 - Behavior
· Level 4 - Results
1. Reaction
Evaluation at this level determines how program participants respond to the program. Kirkpatrick himself calls this a customer satisfaction score. When training is conducted in-house, the response of participants is not always interpreted as customer satisfaction. Often management determines the need to participate in an educational program. Kirkpatrick emphasizes that in this case, the reaction of the participants is a very important criterion for the success of the training, for at least two reasons.
· Firstly, people one way or another share their impressions of the training with their management, and this information goes higher. Consequently, it influences decisions about continuing training.
· Secondly, if participants do not respond positively, they will not be motivated to learn. A positive reaction does not guarantee the successful development of new knowledge, skills and abilities. A negative reaction to training almost certainly means a decrease in the likelihood of learning.
2. Learning
Learning is defined as the change in attitudes, improvement in knowledge and improvement in skills of participants as a result of their completion of a training program. Changing the behavior of participants as a result of training is possible only when learning occurs (attitudes change, knowledge improves, or skills improve).
3. Behavior
At this level, an assessment is made of the extent to which the participants' behavior has changed as a result of training. Kirkpartick points out that the lack of change in participants' behavior does not mean the training was ineffective. Situations are possible when the reaction to the training was positive, learning occurred, but the behavior of the participants did not change in the future, since the necessary conditions for this were not met. Therefore, the lack of change in the behavior of participants after the training cannot be a reason for making a decision to terminate the program. In these cases, in addition to assessing reaction and learning, it is recommended to check for the following conditions:
· Participants' desire to change behavior.
· Participants have knowledge of what and how to do.
· Availability of an appropriate socio-psychological climate.
· Reward participants for changing behavior.
Speaking about the socio-psychological climate, Kirkpatrick refers primarily to the immediate supervisors of the training participants. He identifies five types of “climate”: prohibitive, discouraging, neutral, supportive, demanding. The manager’s position, accordingly, changes from a prohibition on changing behavior to a requirement to change behavior after the end of the training. Kirkpatrick believes that the only way to create a positive climate is to involve leaders in curriculum development.
4. Results
Outcomes include changes that occurred as participants completed the training. As examples of results, Kirkpatrick cites increased productivity, improved quality, decreased accidents, increased sales, and decreased employee turnover. Results should not be measured in money.
According to Kirkpatrick, assessment at this level is the most difficult and expensive. Here are some practical tips that can help you evaluate your results:
· if possible, use a control group (those who did not receive training),
· carry out the assessment after some time so that the results become noticeable,
Conduct pre- and post-program assessments (if possible),
Conduct the assessment several times during the program,
· compare the value of the information that can be obtained through the assessment and the cost of obtaining this information (the author believes that conducting an assessment at level 4 is not always advisable due to its high cost).
The fundamental disadvantages of this approach include the fact that the selected information blocks are not interconnected; in addition, there are no feedback mechanisms to manage the process of generating and transferring knowledge. Other difficulties are associated with the undeveloped procedural level of information collection and its processing. Consequently:
1) generalized numerical characteristics do not objectively reflect states and relationships in the measured empirical system;
2) the measurement techniques used do not have the properties of validity, sensitivity and reliability;
3) the influence of external factors is not taken into account;
4) it is not clear how to isolate the “net effect” of training.”
1This article is devoted to a comparative analysis of various models of personnel training effectiveness, including the most common models: Kirkpatrick, Berne, Stufflebeam and Phillips. The models in question were studied on the basis of secondary information, including a study carried out by the British company Embrion, as well as a study by the American company ASTD from a qualitative point of view: the composition, features, limitations and advantages with disadvantages of each of them were analyzed. The comparison results were verified using an expert survey carried out by the authors regarding the accuracy and prevalence of each approach to assessing the effectiveness of training. Based on the results obtained, conclusions were drawn regarding the applicability of each model in the practice of Russian companies. In addition, the authors offer some recommendations regarding the use of these approaches to obtain greater economic benefits.
Philipps model
Bern model
Stufflebeam model
Kirkpatrick model
staff development
training effectiveness assessment
1. Kirkpatrick D.L., Kirkpatrick D.D. “Four steps to successful training: a practical guide to assessing the effectiveness of training” Moscow 2008. pp. 50-51.
2. Mironov V. Assessment of training according to Kirkpatrick: the test of time [Electronic resource]. – Access mode: http://www.top-personal.ru/issue.html?2185 (date of access: -01.12.14).
3. Oparina N.N. Assessing the effectiveness of training and development of top managers [Electronic resource]. – Access mode: http://istina.msu.ru/media/publications/articles/e4f/7fc/422942/Otsenka_effektivnosti.pdf (date of access: 11/10/14).
4. Skiba E. Topic March 06: assessing the effectiveness of training. Digest on nine main assessment models [Electronic resource]. – Access mode: http://www.trainings.ru/library/articles/?id=6328 (access date: 12/5/14).
5. Casebourne I., Downes A. Evaluation of learning – a family affair? [Electronic resource]. – Access mode: http://epiclearninggroup.com/uk/files/2013/10/WP_Evaluation_of_-learning.pdf (access date: 11/22/14).
6. Passmore J., Velez M.J. SOAP-M: A training evaluation model for HR // Industrial & Commercial Training, 2012. Vol.6, No. 44. P. 315-326.
7. Topno N. Evaluation of Training and Development: An Analysis of Various Models // Journal of Business and Management, 2012. Vol. 5, No. 2. P. 16-22.
In modern conditions of globalization and fierce competition in national and international markets, one of the factors for increasing competitiveness is improving the quality of the organization’s personnel. However, the total amount of investment in personnel retraining, or various trainings aimed at developing the qualities of an individual employee necessary for the company, cannot serve as an accurate assessment of the effect obtained. Thus, a significant part of the problem is the question of how to evaluate the effectiveness of investments in personnel development. For this purpose, separate models for assessing the effectiveness of training were developed, which mainly used the Kirkpatrick approach.
Kirkpatrick model
Without assessing the effectiveness of training, it is impossible to build a training and development system that provides the necessary business results. In 1954, Donald Kirkpatrick defended his PhD thesis at the University of Wisconsin (USA) on the topic “Evaluating the Effectiveness of Program Management.” He proposed a short formula to describe the learning cycle: reaction - learning - behavior - results (see Figure 1). This scheme is necessary for effective training of company personnel, as well as for obtaining the necessary business results.
Fig.1 Kirkpatrick model
The first level of learning is reaction. Feedback lets you know what clients think about the program, based on the responses, the trainer needs to make some changes to the program to improve it. The first level assessment, “Reaction”, is very important, the trainer needs to know the participants’ opinions about his program, and the listeners must also be sure that the trainer is not indifferent to their opinion.
Second level: acquired knowledge. It’s good if the participant is satisfied with the training, but this does not mean that they learned anything. To assess the knowledge gained, tests are used to test knowledge of the material studied, and skill checklists are also used. Testing is as follows: some time after completing the course, participants are invited to a conversation with an experienced colleague or supervisor to find out what knowledge they have learned from the training.
Third level: behavior. Kirkpatrick identifies this level as the most important and difficult. It is at this level that an assessment is made of how the participants’ behavior has changed as a result of training, and to what extent the acquired knowledge and skills are applied in the workplace.
Level four: results.
The main thing in assessing the fourth level is to imagine the end result. The trainer's task is to realize the expectations of stakeholders, i.e. payback.
Donald Kirkpatrick's method shows us how to turn the training process into an effective business tool, and integrate individual training into an organization, making training specifically corporate. It explains in detail how to plan, how
set goals, how to argue with managers for the need for training, and, very importantly, how to eliminate unnecessary training if it does not solve the stated problems.
In practice, however, not all four levels are used, but most often only two, in descending order of importance (based on a study conducted by ASTD - American Society for Training and Development) (see Fig. 2).
Fig. 2 Organizations' use of levels of the Kirkpatrick model
Thus, assessing the effectiveness of the learning process must be approached from the perspective of financial management and quantify the results of such investment. A solution to this problem was proposed in 1997 by Jack Phillips.
Model by Jack Phillips
In the early 90s, the famous HR expert Jack Phillips developed a methodology for assessing the effectiveness of investments in employee training. In his work, published in 1975, Kirkpatrick expressed an unexpected thought: “Under no circumstances should you think of translating the fourth level (business result) into money. It’s not necessary, you can’t.” And in 1991, another person, Jack Phillips, not only stated that this was possible, but also proposed a calculation algorithm. To the four levels already mentioned, another one has been added - return on investment, or calculation of the ratio of the profit from the project and the costs of it.
This indicator allows you to evaluate the effectiveness of training. Phillips' methods work well in companies with regular management.
ROI helps to obtain a number of benefits, in particular for company managers: assess the financial effectiveness of investments in employees; obtain a clear and reliable tool for determining the effectiveness of HR activities; make the “human factor” and its impact on the company’s business results measurable. First of all, an ROI model is necessary to calculate the return on investment in human resources and personnel programs, when management invests in people, they need to understand what the return on this investment is.
Stufflebeam model (CIPP)
The Stufflebeam abbreviation (see Fig. 3) is deciphered as follows:
Rice. 3 Stufflebeam model (CIPP)
The model allows you to evaluate both the results and the process of learning and development. Therefore, this model can be successfully applied for long-term modular training programs for top managers.
Bern model (CIRO)
Similar to the Stufflebeam model is the CIPO scheme (see Fig. 4), developed by Bern. The model includes the following assessment steps:
Rice. 4 Bern model (CIRO)
This model can also be used to assess the effectiveness of training for top managers, especially in the context of launching modular long-term programs.
Advantages and Disadvantages of Effective Teaching Models
|
Advantages |
Flaws |
||
|
Kirkpatrick |
The presence of four levels of assessment: by the reaction of participants, by the results of final control, by changes in production behavior and by changes in the company’s business indicators. Possibility of choice to evaluate the effectiveness of top manager training. |
The difficulty of measuring the third level, the unwillingness of top managers to undergo the assessment procedure or work with coaches. Refusal to evaluate based on financial indicators. |
|
|
Development of the Kirkpatrick model. Introduction of the fifth level of training and development effectiveness assessment. The ability to use a financial return on investment in training and development. |
Calculation of return on investment is possible only if full-fledged management financial accounting is maintained in the organization. It is mainly used to calculate the effectiveness of those programs that are lengthy, expensive and complex. |
||
|
Stufflebeam |
The ability to evaluate both the process and the results of training and development in close connection with the company's goals. |
Unclear methods and procedures for assessing the results of training and development of top managers. |
|
|
Development of the Stufflebeam and partly Kirkpatrick model. Setting goals, identifying opportunities, identifying participants’ opinions on training and development, evaluating results. |
The difficulty of maintaining a constant procedure for assessing the effectiveness of training and development of top managers, application for long-term programs. |
||
The table provides a comparison of the most popular models and methods for assessing the effectiveness of personnel development and training in terms of advantages and disadvantages, including financial, calculation complexity, and so on.
It is worth noting that today there is no ideal model of effective training that would include all the necessary assessment parameters. Each of the presented models has its pros and cons, which are clearly presented above.
The authors conducted a survey of 7 experts in the field of HR on the most accurate models for assessing the effectiveness of training (each expert had to choose the most accurate models, in his opinion, from the proposed list). The survey data is presented below (see Figure 5):
Figure 5. Survey results on the accuracy of training models
This survey showed that the most accurate learning models are the Kirkpatrick model and the Phillips model, and the least accurate are the Stufflebeam model and the Berne model.
Conclusion
In conclusion, we can say that it is necessary to take into account the differences between Western learning culture and ours. In the West, corporate training has long become an indispensable condition and part of business life; there is no need to explain many things to employees. In Russia, the attitude towards business training has now begun to change towards interest, and the technologies themselves are just beginning to be introduced. Thus, the interviewed respondents showed that mainly Western companies most often use effective training models. The most accurate models are those of Kirkpatrick and Phillips. The least accurate models include the Stufflebeam and Berne models; according to respondents, these models are least often used both in the West and in Russia.
Kirkpatrick's model shows how to turn the training process into an effective business tool and integrate individual training into an organization. Phillips' model helps an organization evaluate the financial results of training and development. The Stufflebeam model allows you to evaluate both the results and the learning process itself. Berne's model helps us determine the goals, opportunities, and opinions of training participants. Thus, we can say that each of the presented models has its own advantages and disadvantages. Currently, there is no ideal model that would include all the parameters for assessing effective training at once.
Reviewers:
Putilov A.V., Doctor of Technical Sciences, Professor, Dean of the Faculty of Management and Economics of High Technologies, Ministry of Education and Science of the Russian Federation, Federal State Autonomous Educational Institution of Higher Professional Education "National Research Nuclear University "MEPhI", Moscow ;
Tupchienko V.A., Doctor of Economics, Professor, Professor of the Department of “Business Project Management”, Ministry of Education and Science of the Russian Federation, Federal State Autonomous Educational Institution of Higher Professional Education “National Research Nuclear University “MEPhI”, Moscow.
Bibliographic link
Udovidchenko R.S., Kireev V.S. COMPARATIVE ANALYSIS OF MODELS FOR ASSESSING THE EFFECTIVENESS OF STAFF TRAINING // Modern problems of science and education. – 2014. – No. 6.;URL: http://science-education.ru/ru/article/view?id=16909 (date of access: November 24, 2019). We bring to your attention magazines published by the publishing house "Academy of Natural Sciences"