Editor’s Note:  From a practical perspective, companies need data to support effectiveness and cost-benefits of e-learning as compared to traditional training programs. One advantage of e-learning is that recorded data can be used for continuous quality improvement.

A Pilot Study Regarding Educational Effectiveness
of Web-Based E-Learning

Vincent Kwisnek and Sharon Miller

Key Words: Business, Company, Computer-based e-Learning, Corporation, Distance Education, Educational Effectiveness, Educational Technology, Electronic Mail (Email), Evaluation, Instructor-led Training, Internet, Project, Research, Survey, Web-based e-Learning

Introduction: The Company’s E-Learning Initiative

The Fortune 500 Company for which the pilot study was completed is headquartered in Pennsylvania, and has facilities located around the world. With a workforce of 30,000 employees and an emphasis on manufacturing, an important goal of the company is to operate the safest possible workplace environment. To best meet this goal, the company has utilized face-to-face instructor-led training to convey safety and health practices and requirements. According to Merrill Douglas, author of the article “E-volution at Corporate U (2003),” many businesses are seeking training methods that will, “…save money; reduce travel, educate large, dispersed work forces; and deliver training exactly when employees need it (¶. 3).” This company is no exception, as it sees effective, economical safety and health training as a way to remain competitive as a manufacturer and reduce employee turnover. Safety and health training needs to provide flexibility so that employees may complete the training at their convenience.

Computer-based e-Learning and web-based e-Learning were seen as solutions, so the company purchased and developed safety and health training modules. When compared to traditional instructor-led training, web-based training can be self-paced, highly interactive, result in increased retention rates, and reduce travel costs for either trainer or employees. The company defined computer-based e-Learning as computer-based training accessible independent of the internet or intranet, and defined web-based e-Learning as any computer-based training dependent upon access to the internet or intranet. When compared to computer-based training, web-based training provides easy access to the content and requires no distribution of physical materials. This means that web-based training yields additional benefits over computer-based training, among them:

• Access is available anytime, anywhere, around the globe.

• Per-employee equipment costs are lower.

• Employee participation and completion can be more easily tracked.

• Possible “learning object” architecture supports on-demand, personalized learning.

• Content is easier to update. (Kruse, 2002, Art. 1.9, ¶ 1)

Though the company has no conclusive evidence that the seven items listed below are benefits of using web-based e-Learning, it is utilizing web-based e-Learning as a way to meet the following objectives:

• Minimize potential litigation liability through meeting governmental training requirements.

• Rapidly deploy knowledge about the company’s products and services across the business to support strategic cross-selling.

• Share support services so that the company can avoid redundant technology investments and streamline administrative processes.

• Encourage a cultural shift from location-based collaboration to cooperative teamwork between locations.

• Enhance morale and motivation through providing accessible, timely, and individualized learning for employees.

• Attract and retain top talent by providing effective training that meets the needs of each employee’s demanding schedule.

• Accelerate the ability to implement change through maintaining a focus on results.

After years of unchecked computer-based e-Learning purchasing and web-based e-Learning development, the company informally examined employee perceptions of such training methods. Most employees saw e-Learning as ineffective, unavailable, unsupported, cost-prohibitive, and too high-tech for the hardware and software that was available at the company’s locations. In reaction to these negative perceptions, the company formed an action team, which was comprised of employees who supported the utilization of computer-based training methods. The action team spent three months examining the possibilities of improving the purchase of computer-based e-Learning and the development of web-based e-Learning. By March 2001, the team concluded that the best solution would be to halt the development of web-based e-Learning and only purchase and deploy commercially produced training provided by outside vendors.

Because of this conclusion, the company spent the spring of 2001 evaluating various types of web-based e-Learning and computer-based e-Learning products. After two months of evaluation, the following discoveries and conclusions were reached. First, content providers typically lease or license e-Learning through a per-user or yearly fee, which proved cost prohibitive for the company. Second, content providers tend to distribute basic, generic content so they can in order to maximize their profits offer the content to a variety of businesses, with little or no specific connection to the company’s employees. Third, each vendor that was evaluated utilized proprietary technology, which the company saw as limiting in regard to the ways that training could be deployed.

Once again, the company questioned which approach would be best: purchasing or developing e-Learning. In their article “Buy Versus Build: A Battle of Needs (2002),” Laura Francis and Randy Emelo state that the decision to purchase or develop e-Learning “…can be boiled down into three factors to consider: needs, resources, and uniqueness (¶. 1).” To assist in determining the best course of action, the company contacted various distance education professionals. Since much of the company’s safety and health training needs were unique and the company had the personnel and the time required to develop e-Learning, the distance education professionals advised that over time, the internal development of web-based e-Learning would be the most effective and economical method of deploying safety and health training within the company. Internally developed content could be modified to meet the needs of the learners at a lower cost than purchasing and modifying computer-based e-Learning or web-based e-Learning through a vendor. Based on this recommendation, the company spent the summer of 2001 developing and demonstrating internally created web-based e-Learning modules designed through the use of various e-Learning development software.

These demonstrations validated the need for further internal development of e-Learning and also raised the following question: how could the company be sure that internally developed web-based e-Learning was educationally effective? To begin answering this question, a pilot study was developed that would provide the company with preliminary results and a method to further investigate the educational effectiveness of internally developed web-based e-Learning.

Initiating the Pilot Study

A total of 18 employees participated in the pilot research projects, which were approved by the Institutional Review Board. These projects were developed to evaluate the educational effectiveness of internally created web-based e-Learning. Each participant received an invitation letter, a consent form, and a welcome letter that were sent via either email or inter-office mail. Throughout these communications, the researchers informed each participant that there were no known risks in participating in the research study and that participation would enable the company to better understand the effectiveness of web-based e-Learning. They were informed that participation in this project would neither require nor provide monetary compensation, and that only data that could not identify any individual or group of individuals would appear in any summary of the data. Each participant was also informed that s/he was under no obligation to participate in the study and that s/he was free to withdraw consent to participate at any time. The researchers concluded these communications by stating that a summary of the results of this project would be supplied to each participant, at no cost, upon request.

Employees confirmed their willingness to participate by signing and returning the consent form. Once the completed consent form was received, the participant was sent a Pre-module Survey. The data gathered via the Pre-module Survey enabled the researchers to gather a baseline of each participant’s experience with web-based e-Learning and computer-based e-Learning, prior knowledge of Defensive Driving and Office Ergonomics, and expected learning gained from completing either of the modules. Upon completing and returning the Pre-module Survey, each participant was provided with either the Defensive Driving or Office Ergonomics module.

Web-Based e-Learning Modules

The Defensive Driving and Office Ergonomics e-Learning modules are a combination of behavioral, cognitive, and humanistic learning styles. As shown in Table 1, the Defensive Driving and Office Ergonomics web-based e-Learning modules were developed in accordance to the cognitive levels of Bloom's Taxonomy (Bloom, 1956).

The modules also incorporate the descriptions regarding best practice principles. These principles are applicable to the e-learning process, as adapted from the Zemelman, Daniels, and Hyde text book (1998, p. 8).

• The training is student-centered. The information in the modules is of interest to the participants from a professional as well as personal viewpoint. Throughout the course, questions will be asked to determine prior knowledge about the subject as well as self-reflection exercises.

• The training is interactive. Audio, visual, and tactile learning styles will be incorporated.

• Although the modules are divided into various sections, these sections are tied to the general theme of each module. The information promotes whole ideas, events, and materials in a purposeful context.

• Throughout the training, the learner will be guided to reflect upon his/her experiences so that s/he may evaluate past behaviors. The learner is also encouraged to continually reflect, debrief, and abstract from their experiences concerning any new knowledge of the subject matter as they progress through the content of the module.

• The training is completed individually; however, the company encourages employees to share information learned with fellow co-workers.

• The e-Learning module contains a pre-assessment, which is a series of approximately five questions regarding any prior knowledge the student has regarding the content. As the student progresses through the module, new information is presented which builds upon previously learned information. The student retains this information for subsequent use and retrieval throughout the module. The student needs to assimilate the newly learned knowledge and accommodate their activities in their work environment.

• The training material is defined but not rigidly presented. The activities within the training allow the learner to review the content and reflect upon his/her practices or behaviors.

• Throughout the modules, the learner will have the opportunity to take the new learning and incorporate it into his/her base of prior knowledge. The learner can re-create and build upon the new knowledge they encounter.

• The learner will be challenged as s/he continuously assesses his/her present knowledge about either defensive driving or office ergonomics and his/her need to alter current practices and behaviors in order to maintain a safe and healthy lifestyle.

*Zemelman, S., Daniels, H., Hyde, A. (1998). Best practice: New standards for teaching and learning in America’s schools. . . Portsmouth, NH: Heinemann.

Upon completion of either the Defensive Driving or Office Ergonomics web-based e-Learning modules, each participant received and completed a Post-module Survey. The Post-module Survey provided participants an opportunity to share what they had learned about either Defensive Driving or Office Ergonomics as well as an opportunity to inform the researchers as to the amount of time it took to complete the web-based e-Learning and how that amount of time compared to previous experiences with training. Once the Pre-module and Post-module surveys were collected, the researchers compiled and analyzed the information. This analysis began by examining basic facts regarding the completion of the modules, and it concluded by examining the learner’s prior knowledge, expectations, and actual learning.

Results of the Projects

The data regarding the results of the project were compiled from the information obtained from the Post-module Surveys. As indicated in Figure 1a, it took most participants less than a half hour to complete the training. Figure 1b shows that each learner found the training to be “somewhat” or “very” easy.

Within the Post-module Survey, each participant explained why s/he considered the web-based e-Learning module either “somewhat” or “very” easy. Seventy-eight (78) percent of the participants stated that because they were familiar with the content, the training went quickly and smoothly. Also, the length of the actual training module was a factor, as well as the use of descriptive graphics and clearly written text. The majority of the participants stated that the modules were well organized and easy to navigate. In addition, one participant commented that the interactivity of the web-based e-Learning modules was conducive to holding his/her interest. Eighty-three (83) percent of the participants responded that previous experience with computers made the completion of the web-based e-Learning modules easier. Eighty-three (83) percent of the participants responded that completing previous training through the use of computer-based e-Learning or other web-based e-Learning made the completion of the modules easier.

Simplicity and the length of time it took to complete the training were two factors that were mentioned as positive aspects of the web-based e-Learning. Also, the participants noted that the information contained in the module was pertinent to the subject and effectively increased awareness of the importance of ergonomics in the workplace and safe driving practices.

Sixty-one (61) percent of the participants provided suggestions and comments pertaining to the aesthetics of the training. In one instance, a participant mentioned that the text and graphics made the modules more interesting. However, two of the participants stated the consistency of the text, graphics, and the navigation between the pages of each module should be reconsidered. Several of the participants thought that the assessment questions needed to be more challenging. In addition, it was suggested that more interactivity could possibly better hold the learner’s interest and increase the retention of new learning.

The surveys gathered information about each learner’s prior knowledge, expectations, and actual learning. The following questions provided a way to examine these three items:

• What do you already know about this training topic?

• What do want to learn about this training topic?

• What did you learn by completing the training on this topic?

By compiling the Pre-module and Post-module Survey results, a “Know, Want to Know, and Learned” (KWL) (Ogle, 1986) Chart was created.

The above KWL charts show that the participants had prior knowledge of the training, as well as expectations of what new learning the training would present. The chart also shows that learning did occur.

Although the participants expected the basics of both Defensive Driving and Office Ergonomics to be covered in the web-based e-Learning modules, they were curious about each topic and wanted to engage in new learning. To gain a better understanding of how learner expectations were and were not met, the Post-module Survey asked the following questions:

• What things that you expected to learn were covered by this training?

• What things that you expected to learn were not covered by this training?

These questions were compiled to create Table 3a and 3b.

The left sides of the above charts show that the participants had some of their expectations met through the training. In contrast, the right sides of the above charts show that the company has an opportunity to improve the training so that it meets additional learner needs.

Cost Savings

The classic evaluation model developed by Donald Kirkpatrick looks at four levels: student reaction, knowledge transfer, behavioral change, and business results (Kirkpatrick, 1975). Critics of the Kirkpatrick model say that it does not take the business impact far enough and that the final step in any training program should be a “fifth level” of evaluation – financial return. This ultimate evaluation determines the financial return on investment (RIO) of the training program. (Kruse, 2002, Art. 5.1, ¶ 1)

The first step in cost-benefit analysis is to measure all direct and indirect costs involved in the design, development, delivery, and maintenance of the program. Because different industries have different ways of doing business, this process called for some careful examination of how the company goes about its daily work activities. If the company more efficiently trains its employees, the time saved can be used for productive work. More work time is then translated into a financial benefit. (Kruse, 2002, Art. 5.2, ¶ 1)

In the article “Is E-Learning Right for Your Organization (2002),” Terri Anderson asks, “Can the company afford an e-Learning initiative?” (Cost, ¶. 1) Demonstration of the potential cost savings, combined with the educational effectiveness of internally created e-Learning, presents a strong business case for utilizing e-Learning. The following is an example of the company’s projected cost savings gained by launching one web-based e-Learning module via the intranet. These figures are based on the elimination of redundant tracking systems and the reduction of the use of external content developers and instructors. These numbers represent approximate figures rounded to the nearest thousand dollars, and the figures depict the costs of solely using Instructor-led Training, using a blend of Instructor-led Training and Web-based e-Learning, and solely using e-Learning. Since the company currently utilizes and expects to continue utilizing a blend of Instructor-led Training and Web-based e-Learning, the company expects to achieve the lower end of the potential cost savings. At the time these figures were gathered, savings due to a reduction of travel expenses had not yet been calculated.

Conclusion

The Fortune 500 Company saw the utilization of e-Learning as an educationally effective and economical way to deploy training to its locations. Upon developing and deploying two internally developed web-based e-Learning modules, the company wanted to ensure that the training was educationally effective. The company enlisted the assistance of two graduate students from Duquesne University who developed and implemented pilot research projects that would enable the company to better determine the educational effectiveness of its e-Learning.

The pilot research projects each utilized a different web-based e-Learning module, but the Pre-module and Post-module Surveys contained the same questions. Before taking the training, each of the 18 participants completed a Pre-module Survey that provided a baseline of each participant’s prior experience with training and training technology as well as prior knowledge of the training content and expected learning gained by completing the training. Upon completion of the training, each participant completed a Post-module Survey. This survey gathered information about the amount of time it took to complete the training and the level of difficulty of completing the training. More important, the Post-module Survey gathered information about what each participant learned, and what expected information was contained and not contained within the training.

Overall, participants found the training to take less time than other forms of training in which they participated. Participants also found the training to be easier than other training they completed, for these reasons: the training content was familiar and was well-organized, or the participants had previous experience with computers and/or e-Learning. According to survey responses, the participants did gain new knowledge from the training, and gave the company suggestions for improving the training so that it better meets the needs of the employees.

E-Learning is an effective, economical method for providing training to employees. For the training to be successful, the information contained with a web-based e-Learning module needs to be pertinent to the topic, interesting to read, and easy to navigate. In addition, multi-media and interactivity should be included in the e-Learning module, but these items should only enhance the content, not overshadow it. Web-based e-Learning may also reduce resource expenditures by streamlining administrative overhead while simultaneously promoting interactivity between co-workers and providing motivation and teamwork within its workforce. E‑learning is a method of training that can enhance the learning process and produce positive results within a company and its employees.

References

Anderson, Terri. (January 2002). “Is e-learning right for your organization?” Learning Circuits. http://www.learningcircuits.org/2002/jan2002/anderson.html.

Bloom, B.S. (Ed.) (1956) “Taxonomy of educational objectives: The classification of educational goals”: Handbook I, cognitive domain. New York ; Toronto: Longmans, Green. http://www.coun.uvic.ca/learn/program/hndouts/bloom.html.

Douglas, Merrill. (January 2003). “E-volution at corporate u.” e-learning. http://www.elearningmag.com.

Francis, Laura and Emelo, Randy. (January 2002). “Buy versus build: a battle of needs.” Learning Circuits. http://www.learningcircuits.org/2002/jan2002/elearn.html.

Kruse, Kevin. (July 2002). “Using the web for learning: advantages and disadvantages.” e-Learning Guru. http://www.e-learningguru.com/articles/art1_9.htm.

Kruse, Kevin. (September 2002). “Beyond Kirkpatrick: measuring the financial returns of e-learning.” e-Learning Guru. http://www.e-learningguru.com/articles/art5_1.htm.

Kruse, Kevin. (December 2002). “Measuring the total cost of e-learning.” e-Learning Guru. http://www.e-learningguru.com/articles/art5_2.htm

Ogle, D. S. (1986). K-W-L group instructional strategy. In A. S. Palincsar, D. S. Ogle, B. F. Jones, & E. G. Carr (Eds.), Teaching reading as thinking (Teleconference Resource Guide, pp. 11-17). Alexandria, VA: Association for Supervision and Curriculum Development. http://www.ncrel.org/sdrs/areas/issues/students/learning/lr2refer.htm.

Zemelman, S., Daniels, H., Hyde, A. (1998). Best practice: New standards for teaching and learning in America’s schools. Portsmouth, NH: Heinemann. http://www.talpiot.macam.ac.il/hishtalmut/principles.htm.

About the Authors

Vince Kwisnek is a recent graduate of Duquesne University in Pittsburgh, Pennsylvania, where he earned a Master’s Degree in Instructional Technology. He is currently an e-learning developer and co-administrator of a Learning Management System for a Fortune 500 Company. Vince is dedicated to utilizing technology to enable learners to more easily access necessary training and helping others better understand the field of e-learning and how to better assess the effectiveness of e-learning.

Sharon Miller is a recent graduate of Duquesne University in Pittsburgh, Pennsylvania, where she earned a Master’s Degree in Instructional Technology. She is currently a distance education instructor at a community college located in Pittsburgh, Pennsylvania. Sharon’s focus is enabling educators to gain a better understanding of distance learning and empowering them to build successful distance education lessons and courses.
 

go top

Aug 2004 Index

Home Page