Evaluation ... The Never-Ending Story

One of the most important, yet in my opinion most overlooked aspects of instructional design is evaluations, also known as assessments. One of the first topics we are taught within instructional design is about formative and summative evaluation. Formative evaluation is the assessment process while a course or curriculum is being developed. With each chapter, module, or exercise written, an assessment needs to occur to ensure that the information is accurate, it meets the objectives of the course, and that it is presented in the best possible way to ensure human performance improvement (meaning the learner can apply the new information). Then, after the course is published, a summative evaluation occurs with a knowledge check, quiz, or skills exam to see if they can perform what was taught. Typically, it ends there. There is no follow up to see if 6 -12 months later, the information learned was helpful for job performance, or if it is having an impact on the ROI for the organization. There are two assessment models I use with each project I develop: Stufflebeam’s CIPP and Kirkpatrick’s Training Evaluation Model. Kirkpatrick’s is a very popular model, so I will focus this blog on Stufflebeam’s.

In my occupation, as an instructional designer for a major tech organization, I find Stufflebeam’s CIPP Evaluation Model clear and helpful and easy to follow as I collaborate with subject matter experts, technical experts and even marketing. There are four simple steps:

· Context evaluation

· Input evaluation

· Process evaluation and

· Product evaluation.

Context evaluation

Context evaluation is an assessment of the setting where the program will be utilized. By evaluating the setting, whether it be the location, the audience, the system, we can determine what is needed for an effective course, as well as set the objectives. This is also called a needs assessment. Why is this step so important? Imagine filling the billet to create a course, because someone asked for it, only to learn that knowledge, skills or ability was not the problem to begin with.

When you conduct a context evaluation, you bring all the players together and determine what is the problem? As an example, let’s say in an organization, the problem is the call center representatives are not able to hit their target answer and close out times. Management requests training for the staff. Training is developed and provided, and yet the numbers never change. When you

conduct a needs assessment or context evaluation, you interview or follow the customer service staff and learn they know all the procedures and requirements very well. As you sit and watch them work, you notice they are switching between many screens, the system is slow, and they must wait for information to load. They are not able to enter information quickly and efficiently because of the software system being used. Rather than spend the time creating a course, you reveal the root cause of the problem is an old software system that desperately needs to be updated. Management can save money by not having the training department create another training course, which pulls customer service representatives off the phones, further harming the data metrics. Instead, they can show data to justify the need for a new program.

For some, this context evaluation is quick and easy. A new product is launched, and a course must be written on how to use it. But this step should never be overlooked. It sets the vision for the entire group collaborating on the course, because you can always come back to requests to add to the course and say, “is this needed to support our objectives?” If the answer is no, then let the team know it can be added in future release, or that this is scope creep that is not useful. This becomes the footing for the entire project.

Input evaluation

Input evaluation is where questions are asked as to what resources will be required to design, develop and implement the training. Who are the people involved? What funds are needed? What location, space and equipment is required? How will learning be tracked to make sure it is effective? If it is an eLearning, what program will be utilized to create it, and how will it tracked by the learners (LMS)? Will these resources be enough to meet the objectives, and can the results be tracked? Here is a really important question to be asked at this time: Will the benefits of this course outweigh the cost of development? When, as the project lead, you are able to lay out these questions, and document the answers, you can create a list of resources you already have, whether it be trainers or equipment, and what you need. If there are any gaps, put in the request of what is needed and do not proceed until you get approval, and those items are in the process of being procured. There is nothing worse than moving forward and fully developing a course, only to not be able to release it on time because the resources are not available or were unknown. Once you have the approval, and items are on order, then you can begin to put the project in a Gannt Chart, JIRA or whatever project management system is used for the next step, process evaluation.

Process evaluation

Process evaluation is very much like formative assessment. As the course is being built, you bring in your SME’s and technical experts to make sure it is accurate. You have marketing look at it to make sure it meets branding guidelines, and you do this in an iterative process. Have the editing department review it for syntax, and making sure, if you are creating a course to be used globally, that it translates well to all cultures. My position requires the end product contains a PowerPoint with speaker notes for the instructor, a student guide for the learners to follow along, a lab guide for the step-by-step exercises to follow, and an eLearning for learners who opt not to attend the in-person or webinar. There are a lot of products to review, and I have each module/chapter reviewed by all the various areas upon completion of the module. As I move on to the next module, they are reviewing the previous one and when I get all the feedback, I can make the corrections, and give them another module. This is very helpful, especially with a twenty-module course. Imagine getting all 20 modules at once, being a SME, and having to review it while balancing your existing workload. This iterative process evaluation helps respect everyone’s time, and leads to less rework at the end of the final product, which leads us to product evaluation.

Product evaluation

Product evaluation is what occurs when the course is delivered, and results start coming in. Do not stop at collecting quiz score results, this is when you dive into deeper data and answer the questions: Do the results show the objectives were met? Are there any unintended outcomes or consequences that were not accounted for? Is the cost analysis proving the course to impact ROI positively? What is the effectiveness (merit) of the course? The product evaluation is not a summative evaluation. However, summative evaluation can help answer questions for the product evaluation. Also, if the course is ongoing the product evaluation is not a one-time event. After each cycle of the course, you continue to gather metrics and determine what changes need to be made in order to improve the course.

Stufflebeam’s CIPP Evaluation Model shows just how important assessments are from the start, until the lifetime of a course. It provides four clear evaluation steps that are relatively easy to document and follow. As a final thought, unless you work for a non-profit, an organization’s goal is to make a profit. When the profit is not what is expected, there is typically a reduction in force, and in my experience, training has been one of the areas historically that take a hard hit. When you are able to regularly show in your evaluation process the positive financial impact your work has on the organization, it is no longer a guessing game as to the impact you have on profit, you have data and numbers to back it up. If you are a trainer, instructional designer, or project manager that involves education, I urge you not to overlook the important of a great evaluation model.