Editor’s Note: Digital technologies have revolutionized distance learning and impacted traditional classrooms. Interactive multimedia, learning management systems, and computer-managed diagnostic-prescriptive learning with learning-objects have provided alternative solutions for quality education. Government, corporate, and academic administrators continue to question the cost-benefits. Are these the tools that will revolutionize teaching and learning in the twenty-first century? Where is the break-even point for current investments, and what profits can we expect in terms of quality, accelerated learning, and cost. Here is a view from Finland too stimulate the dialog.

Do Investments in Digital Learning Resources Pay Back?
Comparing Learning Objects and Traditional Classroom Teaching

Sami Nurmi, Tomi Jaakkola

Finland

Abstract

Nowadays resources are allocated for developing, distributing, standardizing and implementing learning objects (LOs)for the needs of schools. These investments can only be deemed worthwhile if LOs can elaborate teaching practices or have some positive impacts on students’ academic performance. Hitherto there is rather limited understanding and a lack of empirical studies on the instructional value of LOs. In this paper the effectiveness of LOs on students’ learning performance in mathematics and language learning was studied and compared to the traditional classroom teaching. According to the results the students using paper-and-pencil tasks in traditional classroom environment outperformed the students using drill-and-practice LOs. Implication of the results and the crucial roles of learning context and available instructional support are discussed.

Keywords: Learning objects, Educational Technology, Experimental study, Mathematics, Language, Traditional classroom teaching, Learning Outcomes, Educational Effectiveness

Introduction

Ever since Thomas Edison declared in 1922 that motion picture will revolutionize educational systems and substitute for textbooks, there has been a search for advanced technologies to improve teaching and learning (Bernard et al., 2007). Although the role of content is not stressed in contemporary learning theories, content and the format in which it is presented is important whether learning is technology-enhanced or not (Kalz et al., 2008). During the last twenty years considerable investments have been made to introduce new learning technologies and Information and Communication Technologies (ICT) into schools through acquiring computer hardware and software, developing digital learning materials, setting up network connections and providing staff training. More recently, financial resources have been allocated for developing, distributing, standardizing and implementing learning objects for the needs of schools especially in Europe, North America and Australia (see e.g. McCormick & Li, 2006; Rehak, 2006). These investments and spent resources can only be deemed worthwhile if there is evidence that ICT infrastructure and learning objects have made some positive impact on the academic performance of students, teaching and learning practices, and work load of teachers.

The impact and effectiveness of ICT in education has been studied intensively ever since early 1980s and the findings vary from highly positive and optimistic to negative and skeptical (Sclater et al., 2006). However, the general trend is that technology-supported learning environments are more effective than or at least as effective as traditional instruction in terms of student outcomes (e.g. Waxman et al., 2003). Recent reviews of the impact of ICT on education provide some summary findings that are interesting in the context of this paper. The EU review published by European Schoolnet (Balanskat et al., 2006) provided statistical evidence that, generally, ICT can enhance students’ attainment in the primary school level. The evidence suggested that the impact is most positive particularly in science and in English as a home language, but not as compelling in all other subjects, e.g. in mathematics. In addition to the benefits on students’ attainment, the overwhelming majority of the studies reviewed in the report confirmed wider positive advantages of using ICT on students’ motivation, skills, concentration, cognitive processing, independent and student-centred learning, critical thinking and teamwork. It was also stated that ICT can benefit both academically strong and weak students as well as students with special needs. The Becta commissioned study of ICT impact in the United Kingdom (Condie & Munro, 2007) reported that the evidence on students’ attainment is somewhat inconsistent. In mathematics, foreign languages, science, history, geography, physical education and arts ICT had positive impact. However, the evidence of ICT impacts on intermediate outcomes, such as students’ motivation, engagement with and independence in learning is reported to be greater and more persuasive. Based on the ambiguous results of these reviews and other previous ICT impact studies, it is still clear that simply installing hardware and software and having access to computers does not produce desired outcomes and improved students’ academic achievement (e.g. Clark, 1983), but successful and effective learning with ICT must rely on sound instructional strategies and implementation in environments that are based on contemporary learning theories.

Promises of learning objects

Although there is no consensus regarding the exact definition, learning objects (LOs) are generally understood as digital learning resources that can be shared and accessed through the Internet and reused in multiple learning contexts. Therefore the learning object concept is more about use, not the objects themselves (Parrish, 2008). The core idea of LOs is to make educational materials broadly accessible, searchable, durable, and reusable beyond their origins for different people, for different purposes, and in different contexts (Bennett & McGee, 2005; Nurmi & Jaakkola, 2006; Rehak & Mason, 2003). Littleton (2003) has summarized this vision of reusability arguing that ‘LOs, produced by publishers, teachers, support staff and students themselves, would be stored in digital repositories, where they could be easily accessed, recombined and reused within online courses’, and they ‘would be designed so that they could be adapted to fit different educational models, subject disciplines and levels of study’ (p. 2).  Furthermore, an LO does not have any particular format, but a LO can, for example, be granular digital resources aimed at one specific learning objective and designed to been able to integrate, aggregate and sequenced together to form various customized ‘units of learning’ according to the given learning needs (Margaryan & Littlejohn, 2007).

These promises of unlimited universal access to online instructional materials, increased productivity among educators and simplified solutions for individualized learning have raised worldwide enthusiasm. It is said that LOs can fulfil the long-promised rewards of eLearning by offering means to make instruction adaptive to individual learners, generated even on the fly, and scalable for mass education without proportional increase in cost, in addition to increased collaboration between educators in terms of the sharing and modifying of available content (Parrish, 2004).

While LOs hold tremendous promise, they have also raised criticism among academics (e.g. Butson, 2003; Collis & Strijker, 2004; Lambe, 2002; Nurmi & Jaakkola, 2006; Parrish, 2004; 2008). In contrast to the above mentioned general advantages of LOs, there are also claims that eLearning can be detrimental not only to the achievement of specific instructional objectives, but also to broader educational goals ranging from brain development to social development. In these anti-computer views the claim is that ICT instruction is likely to be ineffective at best and harmful at worst (Triona & Klahr, 2003). The biggest drawbacks of a majority of LOs (due to their emphasis on knowledge transmission and drilling of learners’ level of knowledge mastery) are their underlying views of knowledge, learning and teaching which are argued to be flawed and outdated (Nurmi & Jaakkola, 2006). In the worst case, LOs can reduce teaching to content delivery and transmission, and learning to simple information acquisition and memorization neglecting learner’s active role in personal knowledge construction and meaning making.

Despite the zest for LOs among different focus groups ranging from academics and educators to corporate leaders, there is still rather limited understanding of the instructional value and effectiveness of LOs (Kay & Knaack, 2007; Nurmi & Jaakkola, 2005; 2006), and questions about whether and how the LOs influence students’ learning have only begun to be addressed. The majority of the LO debate is focused on theoretical ideas and technical aspects behind LOs, and empirical evidence is very sparse (Butson, 2003; Collis & Strijker, 2004). For example, in his recent review of LO literature, Kay (2007) found only two articles out of 58 that examined the impact of LOs on learning.

In the evaluation studies where users evaluate the effectiveness of LOs the findings tend to be very positive. For example, in one such study (Kay, 2007) two-thirds of the high school students answered that they benefited from using LOs, and valued most the motivational, interactive and visual qualities of LOs. Unfortunately the majority of the evaluation studies rely only on descriptive data and anecdotal reports, and are therefore lacking reliability and validity, as well as statistical analysis (Kay & Knaack, 2007). Apart from these evaluation studies, there practically are no empirical studies where LOs’ impact on learning outcomes is studied rigorously. As a consequence, there exists a clear lack of strict empirical studies, making it difficult to have confidence in findings of the positive educational value of LOs. Also, without empirical evidence on the impact that LOs have on learning and analysis of the instructional aspects of LO implementation, we are at the risk of having our digital repositories filled with easy-to-find LOs that we do not know how to use meaningfully in the classroom (Agostinho et al., 2004; Richards, 2002).

Therefore in this research the effectiveness of LOs on students’ learning performance in two subjects was studied and compared to more traditional classroom teaching. The selected subjects were mathematics and language, because according to the previous studies those were found to be subjects where ICT can have positive impact.

Overview of the studies

This paper reports and discusses the results of two experimental studies[1] that investigated the effectiveness of drill-and-practice LOs in comparison to more traditional classroom teaching in two subjects. Drill-and-practice LOs were selected because they represent the most common LO type available at the moment (McCormick, 2008). Drill-and-practice LOs are typically rather simple game-like programs whose main aim is to transmit the content efficiently from the LO to the receiving learner. One important reason for their possible efficiency is that students, especially younger ones, are likely to be motivated and engaged by such tasks that incorporate game-like features (see e.g. Stipek, 1993), and learning motivation has found to have a direct impact on learners’ learning results in online learning environments (e.g. Wang et al., 2008). Due to their design, drill-and-practice LOs are mainly suitable for mechanistic skill rehearsal and fact-oriented knowledge acquisition in narrowly defined topic areas. Although these kind of LOs are quite contrary to the ideas of contemporary learning theories and have only a limited potential to encourage deeper-level understanding and knowledge construction in complex content areas, they can be very effective in some learning contexts with certain audience and content areas when used appropriately. The topics in both the studies required rather mechanistic learning skills including learning of facts and rules and ways to apply them in straightforward way, and therefore drill-and-practice LO was the suitable LO type to be selected for these studies.

The first study was conducted in mathematics and the second in the Finnish language. In study I the specific topic was fractions, and students needed first to understand the concept of fraction, then to convert fractions to mixed numbers, and finally conduct simple calculations with fractions. In study II the topic was noun cases of the Finnish language, and students were required to learn specific grammatical rules and be able to use cases in a correct context. In both studies, half of the students worked individually with drill-and-practice LOs and half worked under teacher-led classroom instruction where typical paper-and-pencil exercises were completed. In order to obtain more robust evidence with a larger sample size on the impact of learning environments, the results of the individual studies were combined.

All of the used LOs were originally designed for different learning contexts, and they were reused and recontextualized to meet the needs of the specific learning environments of the studies. First the LOs were searched and selected by researchers from one learning portal, and then integrated and aggregated into the school’s learning management system (LMS) to form new LO entities or ‘units of learning’. Finally students individually used LOs online within LMS with a web browser.

The procedure in both studies followed the same pattern. In the first session students were given a pre-test. In order to ensure that different learning conditions within each study had the same spread of achievement – that is, students in all learning conditions were equal at the baseline – students were first classified according to their pre-test scores and were then placed evenly into learning conditions. The actual intervention phase, in which students worked in two different learning conditions, took place one week after the pre-test and lasted two hours. A post-test was administered to students one day after the intervention. The pre-test–post-test design with control and experimental conditions allowed us to evaluate the effectiveness of LOs on students’ learning outcomes and their possible differences in comparison to traditional classroom teaching environments. In addition to overall effectiveness, the studies were aimed at revealing whether the environments have different effects on the learning outcomes of students with different levels of prior subject knowledge. Therefore, the level of the prior knowledge parameter was included in the analyses. The division of students into low and high prior knowledge group was based on the median split of students’ pre-test scores. Also, general observations based on researchers’ notes were recorded during the intervention.

In this context, the research questions investigated in this chapter were:

1.      Are there differences in students’ learning outcomes between LO and traditional classroom environments?

2.      Are there differences between LO and traditional environment in learning outcomes of students with different level of prior knowledge?

The results are expected to provide valuable information for teachers, instructional designers and content producers as to what aspects they should consider when designing and implementing LOs in education. The details of both studies are provided in the next sections.

Study I. Mathematics: Fractions and mixed numbers

The first study was conducted with 35 10-year-old (fourth grade) Finnish elementary school students. A subject knowledge test that measured students’ understanding of fractions and mixed numbers was administered before and after the intervention phase. The pre-test consisted of six different sections and each of them included several questions. The pre-test focused on identification and marking of fractions and comparing different fractions and sorting them in size order. The maximum score for the test was six (one point for each section). The post-test included ten sections with various numbers of questions. The maximum score of the post-test was ten where each section corresponded to one point. In addition to the basics of fractions, the post-test dealt with identification of mixed numbers being more challenging than the pre-tests. The intervention phase consisted of two one-hour sessions. The teacher started both sessions in both conditions with an introductory instruction in which he presented the content to the students. After the introduction, students solved content assignments individually. Students were taught by the same teacher for both studies, in order to control the possible effect that differing teaching styles might have had. To ensure that the conditions were comparable, the assignments used in the separate classes were carefully chosen to cover the same topics.

1)      In the Learning Object condition (n = 19), students worked in the computer laboratory with LOs (three LOs per session). The LOs covered fractions and mixed numbers and were principally quite simple ‘game like’ drill-and-practice programs that provided instant feedback for students’ input/answers. Even though the students could proceed at their own pace, the order of the LOs was pre-determined. There was no direct teaching and no teacher-controlled tasks during and after the working phase. The LOs provided very simple feedback, indicating only whether an answer had been correct or incorrect.

2)      In the Traditional Classroom condition (n = 16), students worked in a normal classroom. Here students individually completed different paper-and-pencil tasks concerning fractions and mixed numbers. Students were allowed to seek help from the teacher during the working phase (although they were not encouraged to do so), and at the end of the lessons, the students marked their work using answers provided to the class by the teacher.

Study II. Finnish language: Cases/grammar

The content of study II was Finnish grammar, more specifically noun cases. Cases are a vital part of the Finnish language and are considered to be very difficult to learn. The participants in the study were 37 11-year-old (fifth grade) students from average Finnish elementary schools. A subject knowledge test that measured students’ understanding of cases was administered before and after the intervention phase. The pre-test consisted of three sections with multiple questions. The first section measured students’ prior knowledge of cases and the second their understanding of parts of speech and the third their reading comprehension skills. The maximum score was six, and each section was weighted for two points. The post-test focused only on cases and students were required to identify the cases and inflect different words according to different cases. The post-test involved five different sections with numerous questions and the maximum score was 32. The intervention phase itself consisted of two one-hour sessions. The lessons in both conditions started with a teacher-led introduction that included collective sentence completion during which the teacher asked students to propose which cases fit in particular contexts. After the introduction, students solved content assignments individually. Again, in this study, students were taught by one teacher and the assignments that were used in both conditions were carefully chosen to cover the same topics.

1)      In the Learning Object condition (n = 19), students worked individually in the computer laboratory with LOs containing case identification tasks. There were five LOs for both sessions. Even though the students could proceed at their own pace, the order of the LOs was pre-determined. The LOs were again simple drill-and-practice games or drag-and-drop applications that gave instant feedback or scores for each student action. As in study I, there was no direct teaching and no teacher-controlled tasks during and after the working phase, and the only feedback came from LOs.

2)      In the Traditional Classroom condition (n = 18), students worked in a normal classroom. Here students were individually assigned to solve case identification tasks, which were to be completed in paper-and-pencil format. As in Study I students were allowed to seek help from the teacher during the working phase (although they were not encouraged to do so), and at the end of the lessons, the students marked their work using answers provided to the class by the teacher.

Results

In order to examine the effect of different learning conditions on students’ learning outcomes, the students' subject knowledge post-test scores in both studies are compared. Analysis of covariance (ANCOVA), with subject knowledge pre-test as a covariate, was used to investigate post-test differences. First the results of the individual studies are introduced separately, and then the individual results are combined to get more reliable conclusion on the impact of the compared learning environments. The results of the individual studies are presented in Table 1.

Although students working in the traditional classroom environment slightly outperformed students in the learning object environment in both studies, the differences were not statistically significant (p > .05). In the study I (mathematics) students in the traditional classroom environment scored better than students in the learning object environment both within low and high prior knowledge groups. However, in the study II (language) the learning object environment was more beneficial than the traditional classroom environment for the students with low level of prior knowledge. Among the high prior knowledge level students, the traditional classroom environment was more effective. Nevertheless, the differences between the compared learning environments within the prior knowledge groups were not significant.

As the studies were similar in their designs and required same kind of learning skills, it is possible to combine the results of the individual studies. Instead of focusing only on the results of individual studies, it is more beneficial to investigate the impact of identical parameters across the studies simultaneously. By combining the results from individual studies we increase the sample size, which enables us to make firmer conclusions on the effectiveness of the compared learning environments and detect more easily statistical differences. The Stouffer method allows combination of p-values from multiple studies and computation of an average p-value[2] for these studies (p-value is a direct function of sample size). Combined results are presented in Table 2.

As can be seen that overall, there is significant difference in learning outcomes between the LO and the traditional classroom condition. The investigation of combined average results reveals that the students using paper-and-pencil tasks in traditional classroom environment outperformed the students using drill-and-practice LOs (p < .05). But how much more effective is the classroom environment? The mean difference - expressed in standard deviation units - is called standardized mean difference effect size which is reported in the table. As a general rule of thumb, a standardized mean difference effect size (ES) of .20 should be interpreted as small, .50 as medium, and .80 as large (Cohen, 1988). If we interpret the magnitude of the effect in that way, the average difference between the means of the traditional classroom condition and the LO condition is of medium size (ES = .47) in favour of the traditional classroom group. Another useful, and perhaps more concrete, way to interpret the effect magnitude is to consider the percentage of overlap between the scores (or distributions) of two conditions. Using this logic, an ES of .47 means that 68% of the students in the traditional classroom environment did better than the average student in the LO environment.

The more detailed investigation of impact on students’ learning outcomes within prior knowledge level groups shows that overall traditional classroom teaching was more effective than LOs within both low and high prior knowledge groups. However, the differences between the conditions are not significant, partly due to the fact that the sample sizes of the level groups remained small.

Discussion

Recently, considerable investments have been placed on building up ICT infrastructure and developing sharable digital learning resources for the needs of education all around the world. Although these new instructional technologies raise huge optimism and dazzle us with their promises (e.g. Parrish, 2008), these eLearning investments can only be considered justifiable if they succeed in introducing improvements on teaching practices and enhancements on students’ learning outcomes in comparison to normal classroom teaching activities. However, to date there has only been sparse empirical evidence on the effectiveness of learning objects on learning performance. Therefore the main aim of this article was to investigate the effectiveness of LOs in comparison to traditional classroom teaching.

Although the individual studies did not highlight significant differences, the pooled results from both studies showed that students using traditional paper-and-pencil tasks outperformed the students working with drill-and-practice LOs. The results demonstrated that traditional classroom teaching is at least as effective as LOs in implementing expository teaching activities and fact-oriented learning behaviour. Therefore using LOs to replicate traditional teaching activities which rely on presentation, transmission, exercising, rehearsal and reproduction of knowledge does not seem appropriate.

Why were traditional classroom activities more effective than using drill-and-practice LOs in these two studies? Firstly, based on researchers’ general observation during the interventions there seemed to be differences between the studied environments in the level of students’ engagement. It can be concluded that students working with LOs had difficulties in concentrating on the content to be learned and the atmosphere was somewhat restless. Students seemed to be hurrying through the LOs and they were even competing who was the fastest in completing all the LO exercises. Students were also interested in solving how the LOs works, i.e. what was the logic behind given LOs, more than the learned content itself. Instead in traditional classroom environments there were not such difficulties with the students’ concentration.

Secondly the available instructional support and control may have affected on the students’ learning behaviour. In the classroom conditions the teacher led the class and therefore the teaching-learning activities were rather strictly controlled.  The less-controlled LO environments placed more requirements on students’ self-regulation and self-discipline, whereas in the classroom contexts, the teacher controlled activities. In this way, the instructional support was slightly different in the LO condition as compared to the traditional condition. In the classroom contexts students were allowed to seek help from the teacher during the working phase, and at the end of the lessons, the tasks were collectively checked, whereas in the LO contexts students received no support from the teacher and only elementary feedback from the LO itself.  These differences between the conditions in their level of control and instructional support can be a critical factor in explaining the differences in the successfulness of the learning environments.

A third possible reason may be associated with students’ learning habits. It is clear that students were more accustomed to typical classroom activities with paper-and-pencil assignments. It is likely that taking advantage of technology completely requires some time and very short-term eLearning interventions are found to be predominantly ineffective, as shown in a classic review study by Khaili and Shashaani (1994). According to their findings the ICT impact increased decidedly when the intervention duration expanded from couple of days to four to seven weeks. It may be that the duration of the LO interventions in our studies was too short to reveal the real effectiveness of such environments. Furthermore, there may be inherent problems in the mechanistic learning behaviour that both studies required. Fact-oriented learning and rehearsal activities do not always motivate students enough and in addition, they cannot understand the purpose or objective of their learning, for example, learning of grammatical rules. As a result of low motivation and meaningfulness learning behaviour students’ focus may have drifted away from the actual content to be learned.

The fourth explanations can relate to the game-like features of the LOs used in these studies. Although LOs’ game-like features are designed to raise learners’ motivation, they could also bring their own challenges and limitations. Students are accustomed to play computer games in their free time where gaming means relaxation and is entertaining. Consequently, when computer games are used in education, there easily exist discrepancies between the expectations of educators and students. Instead of using games for learning purposes, students often seek entertainment as they would in their free time gaming, and then, as a consequence, they do not regard the use of educational games as important learning situations. Students’ and educators’ aims can also conflict. Sometimes students may not try to achieve the actual objectives of the educational games, but are aiming for loss or negative feedback if they feel it is somehow more rewarding. For example, in our studies it was observed that students were making mistakes because they were willing to see the negative feedback within an LO as they regarded it as funny or entertaining. Based on our results on the effectiveness of game-like drill-and-practice Los, it can be argued that learning resources aiming at ‘edutainment’ are not effective in terms of content learning when compared to the academic performance achieved in normal classroom contexts.

However, the whole question about the effectiveness of eLearning is problematic, because research has shown that technology as such does not have any particular impact on learning, but the impact is always related to the ways of using ICT as a part of certain, emerged learning environments. Therefore, the focus of the research should be placed on the effectiveness of whole learning environments, not just on the type of eLearning technology used. LOs are just a new chapter in the story of educational technology innovations that do not necessarily lead to students to improved academic achievement (c.f. Clark, 1983). It remains evident that, in order to have effective LOs as well as all educational technology, applications require sound instructional design strategies founded on contemporary learning theories and research-based evidence. These findings highlight again the crucial significance of context. As we have found out the available instructional support is a critical factor in explaining the successfulness of learning environments. Our students in LO conditions were required to work in self-directed ways, however with more structured instructional guidance in using Los, students’ learning performance could be better. As Wang et al. (2008) argued it is important to help and support learners to adapt to and cope with the open self-directed learning environments.

No technology is inherently good or bad, but its applications can be judged good or bad. LOs hold many promises and possibilities in various learning contexts when used according to appropriate instructional strategies, but they should not be seen as the primary or only solution for the challenges of learning (Parrish, 2008). This point is related to any eLearning innovation, since taken to its extremes any technology ends up reversing its original benefits (McLuhan and McLuhan, 1988).

Although these results did not support promises of LOs to enhance students’ learning outcomes when compared to traditional instruction, there are other important elements that can be accomplished by using LOs. For example, using LOs can provide ways to enrich and diversify daily instruction practices; can develop students’ technical skills and more generally, can improve their attitudes towards technology; may increase the interaction among students and/or between students and teacher; and offers possibilities to create positive learning atmosphere where students are motivated to work towards attaining desired learning objectives. However, more research is needed on the interaction between various LO types, ways to implement LOs and learning outcomes.

In addition to these learning perspectives, LOs also provide means to reuse once produced learning materials as mentioned in myriad of LO literature. However, it should be borne in mind that beside promised benefits (at least on a rhetorical level) of cost savings and quickness in lesson and material preparation through content reusability and easiness of updating (e.g. Weller, 2004), developing and implementing new learning contexts with existing LOs will always be difficult, costly, time-consuming and technically demanding (Tompsett, 2005; Wilhelm & Wilde, 2005).

References

Agostinho, S., Bennett, S., Lockyer, L. & Harper, B. (2004). Developing a learning object metadata application profile based on LOM suitable for the Australian higher education market. Australasian Journal of Educational Technology, 20(2), 191-208.

Balanskat, A., Blamire, R. & Kefala, S. (2006) The ICT impact report. A review of studies of ICT impact on schools in Europe. Available: http://insight.eun.org/shared/data/pdf/impact_study.pdf

Bennett, K. & McGee, P. (2005) Transformative power of the learning object debate. Open Learning, 20(1), 15-30.

Bernard, R. M., Abrami, P. C. & Wade, C. A. (2007) A summary of review of e-learning in Canada. A rough sketch of the evidence, gaps, and promising directions. Horizons, 9 (3), 32-43. Available: http://search.ccl-cca.ca/NR/rdonlyres/845D31C6-67D5-47E8-B37F-0A5FA061F06C/0/ELearningSummaryFromHorizons_V9N3_e.pdf

Butson, R. (2003) Colloquium. Learning objects: weapons of mass instruction. British Journal of Educational Technology, 34(5), 667–669.

Clark, R. E. (1983) Reconsidering research on learning from media. Review of Educational Research, 53(4), 445-459.

Cohen, J. (1988) Statistical power analysis for the behavioural sciences (2^nd edition). (Hillsdale, NJ, Erlbaum).

Collis, B., & Strijker, A. (2004) Technology and human issues in reusing learning objects. Journal of Interactive Media in Education, 4. Available:
http://www-jime.open.ac.uk/4

Condie, R. & Munro, B. (2007) The impact of ICT in schools – A landscape review. Available: http://publications.becta.org.uk/display.cfm?resID=28221&page=1835

Kalz, M., Drachsler, J., van Bruggen, J., Hummel, H. & Koper, R. (2008) Wayfinding services for open educational practices. International Journal of Emerging Technologies in Learning, 3(2). Available:
http://online-journals.org/i-jet/article/view/220

Kay, R. (2007) A systematic evaluation of learning objects for secondary school students. Journal of Educational Technology Systems, 35(4), 411-448.

Kay, R. H. & Knaack, L. (2007) Evaluating the learning in learning objects. Open Learning, 22(1), 5–28.

Khaili, A. & Shashaani, L. (1994) The Effectiveness of Computer Applications: A Meta-Analysis. Journal of Research on Computing in Education, 27(1), 48-61.

Lambe, P. (2002) The autism of knowledge management. Available: http://www.straitsknowledge.com

Littleton, A. (2003) Issues in reusing online resources. In A. Littleton (Ed.) Reusing online resources (London, Kogan Page), 1-6.

McCormick, R. (2008) Evaluation of a Large-scale European Learning Object production, distribution and use. In L. Lockyer, S. Bennett, S. Agostinho & B. Harper (Eds.) Handbook of Research on Learning Design and Learning Objects: Issues, Applications and Technologies (Information Science Reference).

McCormick, R. & Li, N. (2006) An evaluation of European learning objects in use. Learning, Media and Technology, 31(3), 213–231.

McLuhan, M. & McLuhan, E. (1988) Laws of media: The new science (Toronto, University of Toronto Press).

Nurmi, S. & Jaakkola, T. (2005) Problems underlying the learning object approach. International Journal of Instructional Technology & Distance Learning, Nov. 2005, 2(11), 61-66. Available: http://www.itdl.org/Journal/Nov_05/

Nurmi, S. & Jaakkola, T. (2006) Promises and pitfalls of LOs. Learning, Media and Technology, 31(3), 269–285.

Parrish, P. E. (2004) The trouble with learning objects. Educational Technology Research & Development, 52(1), 49-67.

Parrish, P. E. (2008) Learning with objects. In S. Carliner & P. Shank (Eds.) Handbook of E-Learning. Past Promises, Present Challenges (San Francisco, CA, Pfeiffer), 215-240.

Rehak, D. (2006) Challenges for ubiquitous learning and learning technology. Educational Technology, January-February, 43–49.

Rehak, D. & Mason, R. (2003) Keeping the learning in learning objects, in: A. Littlejohn (Ed) Reusing online resources: A sustainable approach to e-learning (London, Routledge), 20-34.

Richards, G. (2002). Editorial: The challenges of the learning object paradigm. Canadian Journal of Learning and Technology, 28(3). Available: http://www.cjlt.ca/content/vol28.3/editorial.html

Rosenthal, R. (1984) Meta-analytic procedures for social research. Applied Social Research Methods Series. Volume 6 (London, Sage).

Sclater, J., Sicoly, F., Abrami, P. C. & Wade, C. A. (2006) Ubiquitous technology integration in Canadian public schools: Year one study. Canadian Journal of Learning and Technology, 32(1) Winter. Available: http://www.cjlt.ca/content/vol32.1/sclater.html

Stipek, D. J. (1993) Motivation to learn. From theory to practice. Second edition (Needham Heights, MA, Allyn and Bacon).

Tompsett, C. (2005) Reconfigurability: Creating new courses from existing learning objects will always be difficult! Journal of computer assisted learning, 21(6), 440-448.

Triona, L. M. & Klahr, D. (2003) Point and click or grab and heft: Comparing the influence of physical and virtual instructional materials on elementary school students’ ability to design experiments. Cognition and Instruction, 21(2), 149-173.

Wang, Y., Peng, H., Huang, R., Hou, Y. & Wang, J. (2008) Characteristics of distance learners: Research on relationship of learning motivation, learning strategy, self-efficacy, attribution and learning results. Open Learning, 23(1), 17-28.

Waxman, H. C., Lin, M. & Michko, G. M. (2003) A meta-analysis of the effects of teaching and learning with technology on students outcomes. Available: http://www.ncrel.org/tech/effects2/waxman.pdf

Weller, M. (2004) Learning objects and the e-learning cost dilemma. Open Learning, 19(3), 293-302.

Wilhelm, P. & Wilde, R. (2005) Developing a university course for online delivery based on learning objects: From ideals to compromises. Open Learning, 20(1), 65-81. 

About the Authors

Sami Nurmi, M.Ed. was formerly a researcher at the Centre for Learning Research of the University of Turku, Finland. Currently Nurmi is studying to be an airline pilot and at the same time preparing his educational sciences PhD dissertation on learning with simulation learning objects. Correspondence should be sent via email tosami.nurmi@utu.fi

Tomi Jaakkola, M.Ed. is a researcher at the Centre for Learning Research of the University of Turku, Finland. Jaakkola is currently preparing his educational sciences PhD dissertation on the use of learning objects (LOs) and computer simulations to promote conceptual change and students’ understanding of complex scientific concepts.
Email: tomi.jaakkola@utu.fi