A Pilot Study Regarding Educational Effectiveness
Table 1: Analysis of Web-based e-Learning Modules Using Bloom’s Taxonomy
Bloom’s Taxonomy* Within the e-Learning Modules
The web-based e-Learning modules provide knowledge about defensive driving and office ergonomics. Pre-assessments, post-assessments, and review questions throughout the training will allow the learner to demonstrate prior and newly gained knowledge.
The information presented is factual, and graphics, tables, clip art, and video support the content that is presented.
The web-based e-Learning modules will engage the learner’s prior knowledge through the use of introductory questions about the learner’s current practices or behaviors. The main section of each module will then focus on the content and how the learner can change his/her practices or behaviors to apply the content.
Each web-based e-Learning module challenges the learner to reflect upon his/her practices or behaviors as well as analyze various examples and respond to questions about these examples.
The “How” and “Why” questions within the modules allow the learner to reconstruct the information in order to develop new knowledge.
The pre-assessment enables the learner to reflect on prior knowledge of the subject. There are questions throughout the training to allow the learner to reflect on what s/he has learned. The post-assessment allows the learner to demonstrate mastery of the content. The post-assessment asks the participant to reflect upon the new information they have just learned. A notable change in behavior and attitude is encouraged in order to promote long-lasting improvements.
*Adapted from: Bloom, B.S. (Ed.) (1956) Taxonomy of educational objectives: The classification of educational goals: Handbook I, cognitive domain. New York ; Toronto: Longmans, Green.
The modules also incorporate the descriptions regarding best practice principles. These principles are applicable to the e-learning process, as adapted from the Zemelman, Daniels, and Hyde text book (1998, p. 8).
The training is student-centered. The information in the modules is of interest to the participants from a professional as well as personal viewpoint. Throughout the course, questions will be asked to determine prior knowledge about the subject as well as self-reflection exercises.
The training is interactive. Audio, visual, and tactile learning styles will be incorporated.
Although the modules are divided into various sections, these sections are tied to the general theme of each module. The information promotes whole ideas, events, and materials in a purposeful context.
Throughout the training, the learner will be guided to reflect upon his/her experiences so that s/he may evaluate past behaviors. The learner is also encouraged to continually reflect, debrief, and abstract from their experiences concerning any new knowledge of the subject matter as they progress through the content of the module.
The training is completed individually; however, the company encourages employees to share information learned with fellow co-workers.
The e-Learning module contains a pre-assessment, which is a series of approximately five questions regarding any prior knowledge the student has regarding the content. As the student progresses through the module, new information is presented which builds upon previously learned information. The student retains this information for subsequent use and retrieval throughout the module. The student needs to assimilate the newly learned knowledge and accommodate their activities in their work environment.
The training material is defined but not rigidly presented. The activities within the training allow the learner to review the content and reflect upon his/her practices or behaviors.
Throughout the modules, the learner will have the opportunity to take the new learning and incorporate it into his/her base of prior knowledge. The learner can re-create and build upon the new knowledge they encounter.
The learner will be challenged as s/he continuously assesses his/her present knowledge about either defensive driving or office ergonomics and his/her need to alter current practices and behaviors in order to maintain a safe and healthy lifestyle.
*Zemelman, S., Daniels, H., Hyde, A. (1998). Best practice: New standards for teaching and learning in America’s schools. . . Portsmouth, NH: Heinemann.
Upon completion of either the Defensive Driving or Office Ergonomics web-based e-Learning modules, each participant received and completed a Post-module Survey. The Post-module Survey provided participants an opportunity to share what they had learned about either Defensive Driving or Office Ergonomics as well as an opportunity to inform the researchers as to the amount of time it took to complete the web-based e-Learning and how that amount of time compared to previous experiences with training. Once the Pre-module and Post-module surveys were collected, the researchers compiled and analyzed the information. This analysis began by examining basic facts regarding the completion of the modules, and it concluded by examining the learner’s prior knowledge, expectations, and actual learning.
The data regarding the results of the project were compiled from the information obtained from the Post-module Surveys. As indicated in Figure 1a, it took most participants less than a half hour to complete the training. Figure 1b shows that each learner found the training to be “somewhat” or “very” easy.
Within the Post-module Survey, each participant explained why s/he considered the web-based e-Learning module either “somewhat” or “very” easy. Seventy-eight (78) percent of the participants stated that because they were familiar with the content, the training went quickly and smoothly. Also, the length of the actual training module was a factor, as well as the use of descriptive graphics and clearly written text. The majority of the participants stated that the modules were well organized and easy to navigate. In addition, one participant commented that the interactivity of the web-based e-Learning modules was conducive to holding his/her interest. Eighty-three (83) percent of the participants responded that previous experience with computers made the completion of the web-based e-Learning modules easier. Eighty-three (83) percent of the participants responded that completing previous training through the use of computer-based e-Learning or other web-based e-Learning made the completion of the modules easier.
Simplicity and the length of time it took to complete the training were two factors that were mentioned as positive aspects of the web-based e-Learning. Also, the participants noted that the information contained in the module was pertinent to the subject and effectively increased awareness of the importance of ergonomics in the workplace and safe driving practices.
Sixty-one (61) percent of the participants provided suggestions and comments pertaining to the aesthetics of the training. In one instance, a participant mentioned that the text and graphics made the modules more interesting. However, two of the participants stated the consistency of the text, graphics, and the navigation between the pages of each module should be reconsidered. Several of the participants thought that the assessment questions needed to be more challenging. In addition, it was suggested that more interactivity could possibly better hold the learner’s interest and increase the retention of new learning.
The surveys gathered information about each learner’s prior knowledge, expectations, and actual learning. The following questions provided a way to examine these three items:
What do you already know about this training topic?
What do want to learn about this training topic?
What did you learn by completing the training on this topic?
By compiling the Pre-module and Post-module Survey results, a “Know, Want to Know, and Learned” (KWL) (Ogle, 1986) Chart was created.
Table 2a: KWL Chart for Defensive Driving web-based e-Learning Module
Defensive Driving Module
Want to Know
I have taken driver safety classes before.
I don't believe that I am going to learn anything new
To keep our wheels straight when stopped and before making a turn. Also, to look left first when at a stop sign
Somewhat knowledgeable about being aware of road hazards.
How to react to driving situations.
That paying attention is a critical behavior necessary for safe driving.
[I] Have some knowledge of road hazard.
Some of the common tasks such as checking the vehicle before driving.
Just whatever I have gained from years of driving experience.
Most current and comprehensive information on the subject.
Several critical factors to be prepared for driving.
I know to walk around your car to make sure that it doesn't have any flat tires.
I want to learn the "newest" ideas about being ready to drive.
I learned that I need to be more aware of my vehicle through regular maintenance, and that I also need to do a much better job using my mirrors.
I'm familiar with most traffic rules and regulations governing the highway.
To prepare for bad weather driving conditions.
When waiting to turn left across traffic to keep front tires straight so to keep your turning options open in case of an accident.
I know that driving readiness is vital in order to drive safely.
I want to learn how driving readiness will help me to become a better driver.
I learned that there are seven critical behaviors that will help reduce or prevent driving accidents.
·Basically what I learned in Drivers' Ed in high school -nothing supplemental.
·How to better anticipate and handle sticky driving situations.
·The 360 Degree Scanning Techniques.
KWL Chart for Office Ergonomics Web-based e-Learning Module
Office Ergonomics Module
Want to Know
· I have taken Office Ergonomic classes before.
· Somewhat knowledgeable about being aware of office hazards.
· A considerable amount.
· Expert – I have taught the subject in the traditional classroom.
· I don't believe that I am going to learn anything new.
· How to sit properly at the computer.
· Most current and comprehensive information on the subject.
· How to properly lift objects.
· Review of proper sitting posture at workstation.
· For me, nothing as I have taught this course in detail. I would expect that others would learn quite a bit, however.
· Repetitive motion injury is interesting. I have learned to be more aware of how I sit and position myself in front of the computer.
The above KWL charts show that the participants had prior knowledge of the training, as well as expectations of what new learning the training would present. The chart also shows that learning did occur.
Although the participants expected the basics of both Defensive Driving and Office Ergonomics to be covered in the web-based e-Learning modules, they were curious about each topic and wanted to engage in new learning. To gain a better understanding of how learner expectations were and were not met, the Post-module Survey asked the following questions:
What things that you expected to learn were covered by this training?
What things that you expected to learn were not covered by this training?
These questions were compiled to create Table 3a and 3b.
Wanted to Learn and was within
Wanted to Learn but was not within
· Cell phone policy, checking out car, etc.
· Normal common sense reactions to driving situations.
· Individual awareness and behavior when driving.
· Checking car ahead of time. Actual driving.
· Things to do before you drive.
· Being alert, buckling seatbelts, checking mirrors, checking tire inflation and fluids.
· How to reduce and prevent accidents.
· No cell phone use.
· Checking out road and weather conditions prior to travel.
· Defensive driving practices.
· More accident avoidance.
· I was surprised that the module did not mention anything about the 2 second following rule.
· Night driving and bad weather driving.
· I cannot think of anything that was not covered during the module.
Office Ergonomics Module
Wanted to Learn and was within
Wanted to Learn but was not within
· Head position; arm position; leg position; back position.
· Recognition of stressors.
· Proper posture while sitting and computer set-up.
· Seat adjustability; seat position relative to the height of the monitor.
· Keyboard position; document position; angle position (relative to the seat).
· Proper lifting techniques, (office supplies, luggage, etc.).
· More on solutions and controls, specific equipment maybe a link to ergo web page.
· Common hazard in office areas (slips, trips and falls.
· Types of stretches that would be beneficial to relieve stressors.
· Suggestions for eliminating the potential for strain.
The left sides of the above charts show that the participants had some of their expectations met through the training. In contrast, the right sides of the above charts show that the company has an opportunity to improve the training so that it meets additional learner needs.
The classic evaluation model developed by Donald Kirkpatrick looks at four levels: student reaction, knowledge transfer, behavioral change, and business results (Kirkpatrick, 1975). Critics of the Kirkpatrick model say that it does not take the business impact far enough and that the final step in any training program should be a “fifth level” of evaluation – financial return. This ultimate evaluation determines the financial return on investment (RIO) of the training program. (Kruse, 2002, Art. 5.1, ¶ 1)
The first step in cost-benefit analysis is to measure all direct and indirect costs involved in the design, development, delivery, and maintenance of the program. Because different industries have different ways of doing business, this process called for some careful examination of how the company goes about its daily work activities. If the company more efficiently trains its employees, the time saved can be used for productive work. More work time is then translated into a financial benefit. (Kruse, 2002, Art. 5.2, ¶ 1)
In the article “Is E-Learning Right for Your Organization (2002),” Terri Anderson asks, “Can the company afford an e-Learning initiative?” (Cost, ¶. 1) Demonstration of the potential cost savings, combined with the educational effectiveness of internally created e-Learning, presents a strong business case for utilizing e-Learning. The following is an example of the company’s projected cost savings gained by launching one web-based e-Learning module via the intranet. These figures are based on the elimination of redundant tracking systems and the reduction of the use of external content developers and instructors. These numbers represent approximate figures rounded to the nearest thousand dollars, and the figures depict the costs of solely using Instructor-led Training, using a blend of Instructor-led Training and Web-based e-Learning, and solely using e-Learning. Since the company currently utilizes and expects to continue utilizing a blend of Instructor-led Training and Web-based e-Learning, the company expects to achieve the lower end of the potential cost savings. At the time these figures were gathered, savings due to a reduction of travel expenses had not yet been calculated.
Capital Expenditure of 100% Instructor-led Training . . . . . . . . . . . . . . . . . . . . . .
Capital Expenditure of Training Comprised of
Capital Expenditure of 100% Web-based e-Learning . . . . . . . . . . . . . . . . . . . . . .
Potential Cost Savings Through Utilizing
The Fortune 500 Company saw the utilization of e-Learning as an educationally effective and economical way to deploy training to its locations. Upon developing and deploying two internally developed web-based e-Learning modules, the company wanted to ensure that the training was educationally effective. The company enlisted the assistance of two graduate students from Duquesne University who developed and implemented pilot research projects that would enable the company to better determine the educational effectiveness of its e-Learning.
The pilot research projects each utilized a different web-based e-Learning module, but the Pre-module and Post-module Surveys contained the same questions. Before taking the training, each of the 18 participants completed a Pre-module Survey that provided a baseline of each participant’s prior experience with training and training technology as well as prior knowledge of the training content and expected learning gained by completing the training. Upon completion of the training, each participant completed a Post-module Survey. This survey gathered information about the amount of time it took to complete the training and the level of difficulty of completing the training. More important, the Post-module Survey gathered information about what each participant learned, and what expected information was contained and not contained within the training.
Overall, participants found the training to take less time than other forms of training in which they participated. Participants also found the training to be easier than other training they completed, for these reasons: the training content was familiar and was well-organized, or the participants had previous experience with computers and/or e-Learning. According to survey responses, the participants did gain new knowledge from the training, and gave the company suggestions for improving the training so that it better meets the needs of the employees.
E-Learning is an effective, economical method for providing training to employees. For the training to be successful, the information contained with a web-based e-Learning module needs to be pertinent to the topic, interesting to read, and easy to navigate. In addition, multi-media and interactivity should be included in the e-Learning module, but these items should only enhance the content, not overshadow it. Web-based e-Learning may also reduce resource expenditures by streamlining administrative overhead while simultaneously promoting interactivity between co-workers and providing motivation and teamwork within its workforce. E‑learning is a method of training that can enhance the learning process and produce positive results within a company and its employees.
Anderson, Terri. (January 2002). “Is e-learning right for your organization?” Learning Circuits. http://www.learningcircuits.org/2002/jan2002/anderson.html.
Bloom, B.S. (Ed.) (1956) “Taxonomy of educational objectives: The classification of educational goals”: Handbook I, cognitive domain. New York ; Toronto: Longmans, Green. http://www.coun.uvic.ca/learn/program/hndouts/bloom.html.
Douglas, Merrill. (January 2003). “E-volution at corporate u.” e-learning. http://www.elearningmag.com.
Francis, Laura and Emelo, Randy. (January 2002). “Buy versus build: a battle of needs.” Learning Circuits. http://www.learningcircuits.org/2002/jan2002/elearn.html.
Kruse, Kevin. (July 2002). “Using the web for learning: advantages and disadvantages.” e-Learning Guru. http://www.e-learningguru.com/articles/art1_9.htm.
Kruse, Kevin. (September 2002). “Beyond Kirkpatrick: measuring the financial returns of e-learning.” e-Learning Guru. http://www.e-learningguru.com/articles/art5_1.htm.
Kruse, Kevin. (December 2002). “Measuring the total cost of e-learning.” e-Learning Guru. http://www.e-learningguru.com/articles/art5_2.htm
Ogle, D. S. (1986). K-W-L group instructional strategy. In A. S. Palincsar, D. S. Ogle, B. F. Jones, & E. G. Carr (Eds.), Teaching reading as thinking (Teleconference Resource Guide, pp. 11-17). Alexandria, VA: Association for Supervision and Curriculum Development. http://www.ncrel.org/sdrs/areas/issues/students/learning/lr2refer.htm.
Zemelman, S., Daniels, H., Hyde, A. (1998). Best practice: New standards for teaching and learning in America’s schools. Portsmouth, NH: Heinemann. http://www.talpiot.macam.ac.il/hishtalmut/principles.htm.
Vince Kwisnek is a recent graduate of Duquesne University in Pittsburgh, Pennsylvania, where he earned a Master’s Degree in Instructional Technology. He is currently an e-learning developer and co-administrator of a Learning Management System for a Fortune 500 Company. Vince is dedicated to utilizing technology to enable learners to more easily access necessary training and helping others better understand the field of e-learning and how to better assess the effectiveness of e-learning.
Sharon Miller is a recent graduate of Duquesne University in Pittsburgh, Pennsylvania, where she earned a Master’s Degree in Instructional Technology. She is currently a distance education instructor at a community college located in Pittsburgh, Pennsylvania. Sharon’s focus is enabling educators to gain a better understanding of distance learning and empowering them to build successful distance education lessons and courses.