Editor’s Note: Up till now, much of the literature on Reusable Learning Objects has focused on research, definition and instructional design. This paper takes a practical application to determine the effectiveness of objects used in different course contexts.

Reusing Learning Objects in Three Settings:
Implications for Online Instruction

Simone Conceição, Christine Olgren, Patricia Ploetz

Abstract

This paper describes the implementation and evaluation of sharable content objects (SCO) used as prototypes to test overall design considerations and learning effectiveness. The prototypes were evaluated in three settings: (a) a blended online and classroom model, (b) an online collaborative model, and (c) an online self-paced model. The results of the evaluation demonstrated that SCOs can be effectively reused in multiple environments and that their pedagogical value reflects the context of use. Implications for online instruction related to reusability, design, and learner outcomes are presented.

Keywords: sharable content objects, prototype evaluation, learning objects

Introduction

To meet the growing demand for educational resources, colleges and universities around the world are turning to online technologies to replace and/or enhance the traditional classroom experience. An important component of this new e-learning environment is the use of learning objects (Hamel & Ryan-Jones, 2002).

A great deal has been written about the concept of learning objects, their definition, and their potential application. However, much less is known about their actual use and re-use in educational contexts. Few studies have been conducted to date to research the application of learning objects in actual instructional settings and their value in practice (Mason, Pegler, & Weller, 2005; Nurmi & Jaakkola, 2005).

As Mason, Pegler, and Weller (2005) point out, one benefit of learning objects is to create short learning events that can be adapted to different learning needs and contexts. This flexibility of learning objects contributes to their potential use and re-use in different instructional contexts. The re-use of course material should improve the cost-efficiency of course development because the content chunks or learning events can be used across several courses to reduce development time and expenses.

According to the Department of Defense (DOD), learning objects have the following characteristics or “ilities”: reusability, accessibility, interoperability, and durability. These “ilities,” explains Kaiser (2000), result in activity-sized learning objects, just large enough to be a lesson, that retain their utility over time, are easy to locate and use, can be used on a variety of platforms or course management systems, and are able to be reused in different learning contexts. These characteristics are in line with the literature on learning objects (Downes, 2003; Hamel & Ryan-Jones, 2002; Ploetz, 2003; South & Monson, 2000).

Of the four “ilities,” the most misunderstood is interoperability. In order to be fully interoperable, a learning object (the content) and its associated metadata file must be packaged according to the Sharable Content Object Reference Model (SCORM). Packaging is the process that brings together the learning object and metadata files, then it creates a manifest file that tells the learning management system how the learning content is organized and how the content is presented to the user. When a learning object and its metadata have been packaged according to the SCORM, the result is a SCORM conformant sharable content object, or SCO. Learning objects are packaged so that they can be easily uploaded and used in SCORM conformant Learning Management System (LMS).

The ability to easily upload and use learning objects developed by different proprietary systems into a SCORM conformant LMS promotes reusability across programs and systems (Murphy, 2004; Ploetz, 2003). Therefore, in order to be truly reusable a learning object must be interoperable. The learning objects for this project conform to the SCORM standards for interoperability and, therefore, they will be identified as sharable content objects (SCOs) throughout the remainder of this paper.

This paper describes the implementation and evaluation of the SCOs created for a project entitled, An Investigation of the Pedagogical and Economic Effectiveness of Sharable Content Objects Using Standards in Online Instruction (Meachen, Olgren, & Ploetz, 2004). The project activities included developing and evaluating a library of SCOs for faculty training in 14 instructional areas (or modules) that are central to effective practices in online course conversion, design, teaching, learner support, and assessment. The module Online Facilitation and Communication Techniques was used as a prototype to test overall design considerations in the use of SCOs and was piloted under several conditions to assess reusability, faculty satisfaction, and effectiveness. The conditions involved three settings, or instructional models, in which SCOs are likely to be used: (a) a blended online and classroom model, (b) an online collaborative model, and (c) an online self-paced model.

The Three Settings

The Online Facilitation and Communication Techniques module comprised 11 SCOs. Each SCO was designed to be a self-contained lesson, or instructional event, that included an overview, objectives, content, practice activities, and/or self-check quiz, and a summary taking an estimated average of 20-30 minutes to complete (see Table 1 for a listing of the SCOs). The 11 SCOs were packaged to be SCORM conformant in meeting interoperability standards. The SCOs were made available to three instructional settings: a blended online and classroom course, an online collaborative course, and an online self-paced course at three higher education institutions in the United States to evaluate reusability and effectiveness in different instructional contexts.

Blended Online and Classroom Course

The blended course was part of a curricular redesign program and its goal was to provide higher education faculty new to the online environment the rationale and methods for redesigning one of their face-to-face courses for the online environment. Faculty met one day a week for four weeks for presentations and instruction; the rest of their work week was spent in the LMS Desire2Learn (D2L) reading articles, viewing SCOs, participating in discussion forums, and working on redesigning their face-to-face course.

Online Collaborative Course

The online collaborative course was part of a master’s degree program in adult and continuing education. The purpose of the online collaborative course was to allow students to analyze concepts, theories, and research on distance education; and develop and assess distance education programs. The course met face-to-face for an orientation in the beginning of the semester and on the last day of the class. The remainder of the course was totally online using D2L. The course was divided into five modules for which readings and team tasks were assigned. Students participated in online group discussions, creation of concept maps, and a team project. SCOs were used as instructional aids in the online collaborative course. Students were encouraged to view SCOs as part of the online course orientation and throughout the semester at their own pace.

Table 1

Online Facilitation and Communication Techniques Module

Online Self-Paced Course

The online self-paced course was part of a Distance Education Certificate Program that provides a curriculum of online collaborative and self-paced courses for professional development in online teaching, technology, instructional design, evaluation, and management. The prototype was set up as a self-paced course for online learning. The course consisted of 11 lessons, where each lesson was a SCO. The 11 SCOs were the sole source of content and activities. The only other instructional resource was an overview of the course procedures and content. Participants could complete the course at their own pace, at any time during a one-month period. They used D2L to access the lesson content, submit lesson evaluations, and post optional messages to a discussion board.

Methodology

Online survey research was used to collect data from participants. In order to capture participants’ perceptions of the SCOs, the online survey was administered after interacting with individual SCOs. The survey questionnaire included questions related to the design layout, content presentation and relevancy, and features that were relevant to respondents’ learning and future application (see Table 2 for Online Survey Questionnaire). The same online questionnaire was used for all three settings. The questionnaire was designed to be short in asking nine questions, but it provided responses to key elements related to the design and content of the SCOs. Survey respondents were all non-traditional adult learners.

Table 2

Online Survey Questionnaire

Prototype Participants

Fourteen faculty participated in a month-long blended course: four were tenured professors, seven were in tenure track positions, and the remaining three were classified as teaching academic staff. There were six female (43%) and eight male (57%) participants. Five of the participants had previously used the Internet and public folders (early discussion forums) as instructional aids to existing face-to-face courses or had previously used an LMS such as WebCT or Blackboard. During the first week of the program, faculty were instructed to view three of the eleven SCOs of their choice and complete the online questionnaire for each SCO viewed. There were a total of 47 evaluations.

For the online collaborative course, participants evaluated the 11 prototype SCOs immediately after viewing them. A reminder about viewing and evaluating the SCOs was emailed to students at the beginning of each course module. The online collaborative course included 13 graduate students (15% male; 85% female) pursuing a certificate program in health professional education and non profit management (15%), a master’s degree program in adult education (62%), and a PhD program in education (23%). A total of 120 SCO evaluations accounted for the graduate students who participated in the online collaborative course.

The online self-paced course involved 21 participants who volunteered to participate in the course to increase their knowledge of facilitation strategies. Each participant had to complete and evaluate all 11 lessons. All of the participants had prior experience in using an LMS for online learning. Of the 21 participants, 17 completed the course. The completers comprised 14 females and 3 males representing the following types of organizations and job functions: higher education (53%), business/industry (18%), government/military (18%), and other non-profit (12%). Participants of the self-paced online course were involved in the following types of job functions: design/development (35%), management (24%), faculty training (18%), and teaching (24%). The self-paced participants submitted 205 SCO evaluations.

Prototype Evaluation Results

Quantitative and qualitative records were used to analyze data from participants’ responses to the survey items. The survey items provided data about the following elements: overall rating of each SCO, the SCO design and content, and learning outcomes in relation to knowledge gains and applications.

Overall Rating

Participants in all three settings rated the SCOs favorably (see Table 3 for ANOVA Comparison of Mean Scores for the Three Settings). For the overall rating in Question 9, participants were asked to rate each SCO on a scale of 1 to 4. All three formats were rated at least “good,” with average scores of 3.00 to 3.46. The Self-Paced group average rating of 3.46 was significantly higher than either the Blended or the Online (Blended and Online did not differ significantly). Interestingly, the Self-Paced mode was perceived as being of higher quality overall, where the SCOs stood alone as the sole source of content and activities.

Table 3

ANOVA Comparison of Mean Scores for the Three Settings

 Notes:

Questions 1-6 used a five-point scale of
strongly disagree=1, disagree=2, neutral=3, agree=4, and strongly agree=5.

The anchors used for questions 7-9 are shown in the table.

The < or > symbols denote significant differences between means in the indicated direction
with significance level of p<.01

SCO Design and Content

Five questions on the survey asked participants to rate elements of the SCO design and content related to: (1) learning objectives, (2) layout and navigation, (3) motivational engagement of the content and activities, (4) amount of content, and (5) depth of content. Each element was rated on a five-point Likert scale ranging from 1 = strongly disagree to 5 = strongly agree.

For the SCO design, the participants in all three settings agreed (minimum rating of 4.0) the objectives were clearly stated, the SCOs were easy to navigate, and the content and activities were engaging. The ratings for the three settings were quite similar. However, when asked to rate whether they agreed or disagreed with the statement, "The content presentation and activities are engaging," those in the Self-Paced group rated this statement significantly higher than those in the Blended or in the Online groups. Differences in ratings between the Blended and Online groups were not significant.

Questions 7 and 8 asked about perceptions of the content. All three groups responded, on average, that the amount of content and depth of content were “about right” for the prototype SCOs.

Knowledge Gains and Applications

Participants found the SCOs to be effective in learning outcomes. They agreed (minimum rating of 4.0) the content was relevant to their needs, added to their knowledge or skills, and provided practical information applicable to their own work. There was no significant difference among the three groups in their ratings of effectiveness (see Table 3).

The quantitative data about SCO effectiveness was supported by the qualitative comments made by participants. The qualitative comments also provided insights into how the SCOs were used. In the online collaborative course, respondents evaluated the content in the SCOs in relationship to their learning in the course and application of the material during online discussions. For example, two of the students stated how the SCOs were used as a learning tool: “I knew most of this material already, but was able to pick up a few tips and it was a nice refresher of things I already knew” and “This learning objective was applicable to what has been discussed in this [course] module and helped me apply what I have learned in a clearer manner.” Another student revealed how the content in the SCO could be applied during the online course discussions: “I was surprised by a few of the 'correct' answers, and hope to facilitate Module 2 discussion on those points.”

During an online program session on multimedia in the online environment, faculty were asked to compare the effectiveness of the SCOs to other media. Faculty overwhelmingly agreed that the SCOs were more effective in providing instruction than other types of media including online video/audio lectures and PowerPoint presentations. Comparing SCOs to other forms of multimedia allowed faculty to become aware of the potential use of SCOs for online instruction.

It was evident that students in the online course were looking at the SCO content as a tool that contained information they could apply in the future. One student said that the information in the SCO will assist in teaching online courses. Another student stated, “I thought that the case studies were very helpful and provided great insight as to what I could do when I become an instructor.”

Consistency in Layout and Variety in Design

The last item on the survey asked respondents to provide suggestions for improvement of the SCOs. Most of the qualitative responses involved participants’ perceptions of the SCO layout, content, and design features that affected their learning. A theme emerging from the qualitative responses brought up issues related to the consistency in layout and variety in design.

The SCOs were designed using the same instructional template to enhance consistency in content layout, but they varied in terms of instructional activities. Consistency in content layout is defined in terms of font color and page layout. Design variety means using a variety of instructional activities in each SCO. When different instructional activities were used in the design of the SCOs users could interact with the content in a variety of ways. These activities included case studies, quizzes, matching activities, puzzles, game-like strategies, and analogies. It was evident that depending on the learning preference of the user, certain methods were favored. Responses that indicated learning preference for certain methods compared strategies used in one SCO. For example, a student in the online collaborative course said, “I thought that quizzes were sufficient in testing if information was learned; I didn't think that we needed to go through the scenarios.” Another student stated, “I thought the cake analogy was a bit overdone, but enjoyed constructing paragraph responses for the offensive scenarios.”

Depending on the type of analogy and topic introduced in the SCO, course participants had different reactions. One of the SCOs compared team building to a basketball team, which made the information transferable to people who were familiar with team sports. For some people the crossword puzzles were enjoyable; for others they were annoying. One respondent said that “asking to 'fill in the blanks' would eliminate the hassle of scrolling, clicking tabs, and tinkering with arrows to try to fill in letter boxes.” Other respondents stated that action examples were insightful. For some, the use of scenarios was real and thought provoking.

Several participants in the online self-paced course made comments about consistency and variety from an instructional design perspective. Because the self-paced participants were involved in a certificate program to gain knowledge and skills in online instruction, their evaluation of the SCOs included comments about design considerations.

One self-paced participant liked the variety of instructional activities, saying: “By nature I would prefer consistency, but I think variety adds a lot. It makes the learning process interesting, prevents monotony.” Another person saw benefits to both, where consistency made it easier to navigate the SCOs and to know what to expect, while variety added interest. As she said, “I'm not sure which side of this fence I want to fall on and am trying to think of some ideas to promote consistency while keeping the variety…At the very least, I think the colors and navigation where it is similar need to stay consistent.”

A third person wondered how a designer would approach the question of balance between the two: “I know that the idea with SCOs is that they be granular, standalone pieces that can be used on their own or reassembled into a course, but I'm not sure how that approach played into the creation of this course. For example, I really liked the graphics of the people—because they were done in a consistent style, they lent coherence to the modules. On the other hand, one module had a basketball theme, one module had a cake baking theme, etc. That made the modules feel somewhat disjointed. How did the project team approach this balance between coherence and independence among the SCOs?”   

More research is needed on the instructional design of learning objects, including the balance of consistency and variety. Participants in the self-paced course indicated that an appropriate balance affected their level of satisfaction with the learning experience, ease of navigating the content, motivational interest, and level of understanding both within and between the SCOs.

Implications for Online Instruction

This study examined the implementation of prototype SCOs in three instructional settings to test overall design considerations and evaluate their learning effectiveness. The results of the evaluation demonstrated that SCOs can be effectively reused in multiple environments and that their pedagogical value reflects the context of use.

Reusability

A key benefit of learning objects, in theory, is their reusability in different instructional contexts. As Nurmi and Jaakkola (2005) state, a major reason for creating learning objects is to achieve a “learning object economy” that maximizes the reuse of learning resources to reduce the costs and time of course development. However, in practice, questions exist about the feasibility and effectiveness of reusing learning objects in different contexts (Christiansen & Anderson, 2004; Mason, Pegler, & Weller, 2005).

This study found that SCOs can be effectively reused in different learning environments that involved a blended online and face-to-face mode, a collaborative online mode, and a self-paced online mode. The learning environments had different audiences (faculty, graduate students, and trainers) and were offered by three different institutions. Although the settings were quite different, the SCOs were evaluated as being effective in achieving learning objectives. The study found that reusability was aided by packaging the SCOs to meet SCORM standards, which made them easily uploaded and unpacked in an LMS. Because all three settings used an LMS from the same vendor, the study did not examine interoperability across platforms.

The study also found that the SCOs could be reused in different ways to aggregate course content. In the blended and online collaborative modes, lesson-sized SCOs supported instructor flexibility in combining the SCOs with other resources. The instructors remained in control of the course design and used the SCOs as a supplement or a complement to other aspects of instruction (Collis & Strijker, 2003). The online self-paced mode capitalized on the value of the SCOs for just-in-time learning and as a sole source of content. The SCOs were organized into a logical sequence of lessons, but no other instructional element was used except for a narrative course overview. In all three contexts, the SCOs served as a source of content that met course goals and achieved learning objectives. In addition, because the SCOs were designed to be self-contained units of instruction, they demonstrated the value of having a higher level of granularity where each SCO contained all the necessary elements of a holistic learning experience (Mason, Pegler, & Weller, 2005). While South and Monson (2000) argue that a smaller or more granular object is more reusable, this study found that a higher level of granularity was effective.

Design

Consistency of presentation and variety of instructional activities proved to be two important elements in the design of the SCOs. In this study, a template approach to content presentation provided consistency in layout and look, which learners said clarified navigation and helped them to know what to expect. On the other hand, the SCOs incorporated a variety of instructional strategies to add variety and learner-to-content interactions, such as drag and drop activities, crossword puzzles, quizzes, animations, scenarios, analogies, and case studies. Learners had mixed feelings about the instructional variety. Some found it enhanced motivation and understanding, some expressed preferences for certain types of strategies but not others, and some students wondered if the variety interfered with coherence. The need to balance variety with consistency is identified as a major design challenge by Weller, Pegler, and Mason (2003). As they said, variety is important to student motivation, while consistency is important to a cohesive design approach. In this study, the balance between variety and consistency emerged as a major consideration that requires further research.

Learner Outcomes

Critics of learning objects claim that the learning object approach may break down the narrative flow that holds individual elements together in a course, leaving a series of disconnected pieces that affect the learning process (Mason, Pegler, & Weller, 2005; Nurmi & Jaakola, 2005). Our experience reveals that the SCO content, as experienced by our learners, can achieve a specific learning outcome and provide a higher level of granularity where each object contains the necessary elements of a holistic learning experience that can be meaningful and relevant to the learner. The respondents in all three contexts evaluated the SCOs positively in terms of learning for understanding and application. The self-paced mode was particularly interesting because the SCOs were sequenced as a series of lessons with no other source of content or narrative transitions (except for a course overview). The self-paced mode found the SCOs worked together to provide instruction in a format that did not seem fragmented to the learners, but was experienced as integrated and effective in learning about the content.

Conclusions

Much has been written about the potential of learning objects to increase the efficiency of instruction through the reuse of learning resources for course development and delivery. However, research on the actual use of learning objects in instructional contexts is just beginning to address questions related to their pedagogical value and learning effectiveness.

This study found that SCOs can be reused effectively in multiple environments, and they can serve as flexible sources of content in achieving learning objectives. Within the context of use, the pedagogical value of SCOs is related to how they are employed to help accomplish particular learning goals. In addition, as this study found, SCOs can be used in at least two different ways for content presentation: (1) as a series of lessons arranged in a logical sequence in serving as a sole source of content; and (2) as aggregated with other learning elements in serving as part of the content. In either case, a SCO can be experienced as a holistic learning experience if the level of granularity contains all the elements needed to support the objectives.

However, issues of learner preferences need to be taken into consideration. Designers may want to consider the balance between consistency in SCO structure and variety in instructional methods. The prototype evaluation presented here describes the use of SCOs in an LMS; however, a knowledge repository, where SCOs can be stored and searched, would give more options for the type of methods that appeal to specific instructors or learners.

References

Christiansen, J., & Anderson, T. (2004). Feasibility of course development based on learning objects. International Journal of Instructional Technology and Distance Learning 1(3). Retrieved on February 25, 2006 from
http://www.itdl.org/Journal/Mar_04/article02.htm

Collis, B., & Strijker, A. (2003). Re-usable learning objects in context. International Journal on E-Learning, 2(4), 5-16. Retrieved on February 11, 2006 from http://dl.aace.org/14190

Hamel, C. J., & Ryan-Jones, D. (2002) Designing instruction with learning objects. International Journal of Educational Technology, 3(1). Retrieved on February 9, 2006 from
http://www.ao.uiuc.edu/ijet/v3n1/hamel/index.html

Kaiser, G. E. (2002). Constructing learning objects. Retrieved February 11, 2006 from http://student.ccbcmd.edu/~gkaiser/LO_05.html

Longmire, W. (2000). A primer on learning objects. ASTD Learning Circuits. Retrieved September 1, 2002 from www.learningcircuits.org/mar2000/primer.html

Mason, R., Pegler, C., & Weller, M. (2005), A learning object success story. JALN, 9(1), 97-105.

Meachen, E., Olgren, C., & Ploetz, P. (2004). An investigation of the pedagogical and economic effectiveness of sharable content objects, using standards, in online instruction. Fund for the Improvement of Postsecondary Education (FIPSE). U.S. Department of Education. Grant No. P116B020126. Available at
http://www.fipse.aed.org/grantshow.cfm?grantNumber=P116B020126

Metros, S. E. (2005, July-August). Learning objects: A rose by any other name. Educause Review, 40(4), 12-13.

Murphy, E.(2004). Moving from theory to practice in the design of web-based learning using a learning object approach. e-Journal of Instructional Science and Technology, 7(1). Retrieved on February 10, 2006 from
http://www.usq.edu.au/electpub/e-jist/docs/Vol7_No1/FullPapers/Theory_to_practice.htm

Nurmi, S., & Jaakola, T. (2005). Problems underlying the learning object approach. International Journal of Instructional Technology and Distance Learning, 2(11). Retrieved February 11, 2006 from http://www.itdl.org/Journal/Nov%5F05/article07.htm

Oakes, K. (2002). An objective view of learning objects. American Society for Training and Development, 56(5), 103-105.

Ploetz, P. (2004) Faculty development and learning object technology: Bridging the gap. Teaching with Technology Today 10(4).

South, J. B., & Monson, D. W. (2000). A university-wide system for creating, capturing and delivering learning objects. Retrieved February 11, 2006 from http://72.14.203.104/search?q=cache:P34hz0u6QSEJ:
www.reusability.org/read/chapters/south.doc+reusability,+accessibility,
+interoperability+and+durability+&hl=en&gl=us&ct=clnk&cd=12&client=firefox-a

Weller, M., Pegler, C., & Mason, R. (2003). Working with learning objects: Some pedagogical suggestions. Retrieved February 25, 2006, from the World Wide Web: http://iet.open.ac.uk/pp/c.a.pegler/ukeu/ALTC_2003.doc

 About the Authors

Simone Conceição, Ph.D. is an assistant professor of Adult and Continuing Education Leadership at the University of Wisconsin-Milwaukee. She teaches courses in the areas of distance education, use of technology with adult learners, instructional design, and adult learning. Her research interests include adult learning, distance education, impact of technology on teaching and learning, instructional design, and learning objects. Dr. Conceição has researched and identified many aspects of good practice in online environments, and she is an expert in helping instructors and trainers understand web-based technology tools, software, and design processes.

e-mail: simonec@uwm.edu, website: http://www.uwm.edu/~simonec Telephone: 414-229-4615

Location: Enderis Hall 639, 2400 E. Hartford Ave.
Mailing Address: UWM School of Education, P.O. Box 413, Milwaukee, WI 53201-0413

Christine Olgren, Ph.D., is Director of the Distance Education Certificate Program, School of Education, University of Wisconsin-Madison. She is responsible for program management, instructional development, teaching, evaluation, and marketing. She has taught online since 1995 and has developed courses for both collaborative and self-paced learning. Her research interests include the cognitive, metacognitive, and motivational aspects of learning with technology. She recently completed a three-year FIPSE grant as co-project manager to develop and evaluate learning objects for online instruction.

e-mail: cholgren@wisc.edu Website: http://www.wisc.edu/depd/ Telephone: 608-262-8530

Address: UW-Madison, B136 Lathrop Hall, 1050 University Avenue, Madison, WI 53706

Patricia Ploetz, ABD, is the Interim Director of the Center for Academic Excellence and the Coordinator of the Teaching & Learning Resource Network at the University of Wisconsin-Stevens Point (UWSP). Patricia works with UWSP’s academic and information technology departments to: 1) create a faculty driven Center for Academic Excellence that advances the learning environment at UWSP by supporting the campus’s strong culture of teaching and 2) provides guidance and direction in the growth and development of faculty educational technology initiatives. Patricia’s research interests include changing faculty roles in higher education, faculty development, and the use of technology to enhance instruction.

email: pploetz@uwsp.edu, website: http://www.uwsp.edu/it/tlrn Telephone: 715-346-4930

Location: Learning Resource Center Room 500

Mailing Address: University of Wisconsin Stevens Point, 900 Reserve Street, LRC 500, Stevens Point, WI 54481