Editor’s Note: This paper is a reminder that the adoption of new technologies requires planning to ensure that the infrastructure will support the technology, and that administrators, faculty, students and support staff are properly trained. Course management systems make learning accessible and efficient for anywhere-anytime learning. With the proper technology and support, online learning and course management systems allow universities to grow when classrooms and faculty office space is impacted.

Evaluating the Impact on Users from Implementing a Course Management System

Richard E. West, Greg Waddoups, Meghan M. Kennedy, Charles R. Graham

USA

Abstract

Nearly all colleges and universities are using some form of e-learning system, usually an expensive course management system (CMS), to create online course offerings or to enhance regular, classroom-oriented courses. Our university has invested a large amount of resources into purchasing and supporting one of the two most popular CMS vendors, and it has become imperative to understand what the effects from using this technology have been, as well as how we can improve the integration of this and other educational technologies into different instructional contexts. This project, through a combination of surveys, call-log analysis, and interviews, was used to investigate the following questions: How has the implementation of a CMS (Blackboard) impacted students and faculty at the university? What are the benefits and challenges from supporting this tool on an institution-wide basis? Our findings conclude that instructors and students are moderately satisfied with the course management tool, but only if it is stable. However, for all of the features available through the CMS, there are only four features regularly used, primarily to increase the efficient transfer of information from teacher to student. We discuss the benefits and challenges reported by students and faculty, along with the acknowledgment that the tool has become critical to many participants’ practices.

Keywords: higher education, evaluation, CMS, course management system, e-learning, distance learning, online learning, instructional technology, educational technology, blackboard, blended learning.

Introduction

In the last decade, there has been a rush by higher education institutions to adopt Course Management System (CMS) technologies in an effort to more easily transition courses into a blended, or part online, experience. Now, with over 95% of colleges and universities in the United States using some form of e-learning system (Pollack, 2003), usually an expensive course management system, it is imperative to look closely at the CMS technologies available and evaluate whether these CMS tools are creating effective e-learning environments.

E-learning can take many different forms, and scholars vary in how they define a course management system. For example, one author wrote that a CMS is a “comprehensive software package that supports some or all aspects of course preparation, delivery, communication, participation and interaction and allows these aspects to be accessible via a network” (Collis & de Boer, 2004). For these authors, a CMS is an efficiency tool, designed to help a professor prepare and deliver instruction more efficiently. John Meerts (2003) described a CMS as simply a way to enable teachers who do not have web design skills to be able to easily create a web accompaniment to their courses. Other writers describe CMS tools as having the ability to actually improve the learning of students. For example, Pollack (2003) defined a CMS as “a technology tool that supports and enhances the learning process” (p. 5). Many other instructors, administrators, and educational researchers define CMS tools as something in between—a tool that can support some kinds of learning and provide some efficiency benefits.

Recently, two companies have grown to dominate the young CMS market: Blackboard and WebCT, each being used now at over 2,000 different academic institutions (Arnone, 2002; Pollack, 2003). More recently, these two companies have announced their merger, creating one tool that dominates the market. Besides Blackboard and WebCT, there are several other CMS companies with much smaller market shares, including Desire2Learn, eCollege, and Jenzabar. There are also many open-source or freely distributed CMS products, such as the Manhattan Virtual Classroom Project, Sakai, OpenCourseWare, and Moodle. Finally, there are several universities that develop their own CMS systems. Examples are Penn State (Angel). Websites such as Edutools (http://www.edutools.info/course/productinfo/) provide lists of many available CMS products.

When institutions rushed to adopt these, or other, CMS tools, it was not clearly understood how much of a commitment was required. At one time, commercial CMS products were relatively affordable—much more affordable to purchase than it would have been for most institutions to develop their own CMS-type of system. However, in recent years Blackboard and WebCT have both raised their prices dramatically. Colleges and universities are now finding themselves in a difficult situation as the prices of CMS systems are increasing at the same time that instructors are becoming more dependent on these tools for their teaching. Young (2002) commented that this escalating cost structure would have a significant impact on institutions that have already integrated a CMS into their institution. “CMSs have become mission critical systems for many institutions,” Young said, meaning that the need for course management systems has become a critical one for colleges and universities, and going back to instruction without an adequate CMS is no longer seen as acceptable by most students and faculty members.

While the open-source and free alternatives to Blackboard and WebCT might appear to be a much cheaper option for institutions, some, including Meerts, 2003, have predicted that the benefit is small, if there is one at all, because of the cost needed for a university to install, train, and support a free CMS software program, and that home-grown systems often come with their own set of problems (Olsen, 2001). The rising cost of purchasing or supporting a CMS has become a common problem for many institutions. “Course management systems are likely to become as commonplace as email and the web. No institution of higher education will be able to do without either an open source or commercial version of the software,” Meerts said. Similarly, Young (2002) reported that the CMS is now a “fixture” on many campuses. This leaves many institutions scrambling to find the most effective, and affordable, tool for their e-learning support needs.

When considering the cost required to purchase or upkeep a CMS on campus, institutions need to know what the benefit might be to having a CMS, in order to evaluate whether the money spent on such a system is justifiable. Several small studies have found positive impacts from using CMSs in specific types of courses. These studies reported that using a CMS can be helpful for improving communication and collaboration in a course (Hutchins, 2001; Anderson, 2003; Pollack, 2003); increasing student preparation for class and improving the quality of in-class time (Massimo, 2003); enhancing class lectures and feedback to students about grades (Morgan, 2003); giving students greater access to materials (Yip, 2004); and improving learning in other ways (Klecker, 2002). However, other studies have found no significant difference between the grades of students using a CMS and students who did not (Vessell, 2001), and that the benefits of using a CMS can be counter-balanced by many flaws in the software, causing slowness or instability (Dutton, Cheong, & Park, 2004).

Despite these studies, it is surprising how little research and evaluation has been done about the implications of using these tools. A search of several of the major databases in the fall of 2005 (ERIC, Education Full Text, WebSPIRS, PsychInfo and Ingenta) found 164 published articles that mentioned course management systems, Blackboard, WebCT, Moodle, CMS, or other similar terms in the abstracts. But of these, only 74 appeared to be data-driven articles, and most of these were quick evaluations of how a CMS impacted a particular class or context. Less than 10 studies, that we could find, seemed to attempt a more general evaluation of the impact from using a CMS over multiple contexts, such as multiple university departments. Two of the more extensive evaluations of CMS technologies have been completed by the Educause Center for Applied Research, but not, as far as we know, yet published through peer review (Morgan, 2003; ECAR, 2005). In these reports, the authors have used survey research to conclude there are many positive effects from using course management systems, and that the majority of instructors and students are satisfied with these technologies. More studies of this scope, completed by a variety of researchers and published in accessible sources, would improve our understanding of how the use of these technologies is impacting higher education.

Considering the rapid adoption by almost all institutions of higher education of CMS tools, there needs to be more thought, reflection, and research about what the impact might be. How well do these colleges and universities really understand their implementation of their chosen CMS? After spending the money to purchase and support a CMS, are instructors and students using it? Which features and in what ways? Is it changing how people teach or learn? Are there costs or benefits that we were not prepared for? These are the kinds of questions that need to be asked about a collection of technologies that are becoming as ubiquitous in higher education as the Internet or office software.

Research Questions

Like many others, our university has invested a large amount of resources into purchasing and supporting Blackboard, one of the two most popular CMS vendors. This study was designed to help us better understand the effects on our institution from using this technology, as well as provide ideas for how we can improve the integration of this and other educational technologies into different instructional contexts. Specifically, we investigated the following questions: How has the implementation of a CMS (Blackboard) impacted students and faculty at the university? What are the benefits and challenges from supporting this tool on an institution-wide basis?

Methods

Data Collection

Our methods for collecting data included using open-ended/closed-ended survey questions (n=124 instructors; 163 students) administered to students and instructors over two semesters; an analysis of the calls reported to the Blackboard support center on campus (n=1,341) and semi-structured interviews with instructors (n=48) as well as several brief intercept interviews with students (n=17). We adopted a mixed methods approach to evaluation using both descriptive analysis of survey responses and qualitative analysis of interviews to triangulate our analysis and strengthen our findings (Greene, Benjamin & Goodyear, 2001).

Semester Surveys

We administered the surveys to random samples of instructors and students near the end of the semester in fall, 2004 and winter, 2005. These samples were drawn from the population of Blackboard users on campus (see Figures 1 and 2 for descriptions of the demographics of faculty survey participants).

The selection process included identifying all faculty and students who had at least one Blackboard course and then randomly selecting 200 faculty and 600 students as participants. From this sample, 124 instructors and 163 students responded to surveys in both semesters. The survey questions considered for this article focused on the following categories that were identified as important by institutional stakeholders:

• Satisfaction: How students and teachers feel about Blackboard and whether they like and prefer to use it

• Knowledge: Whether students and teachers know how to use and operate Blackboard and if they understand its features

• Usage: What features in Blackboard students and teachers are using most or least

• Efficiency: Whether Blackboard is saving students and teachers time or is increasing their workload

• Stability: How well Blackboard is operating and what technical problems students and teachers are experiencing

Survey questions were developed and checked for face and content validity by members of the evaluation team and selected university stakeholders.

Figure 1. Description of the colleges represented by faculty who participated in the surveys. These are colleges represented by the abbreviations: Biology and Agriculture; Business; Education; Engineering and Technology; Family, Home, and Social Sciences; Fine Arts and Communications, Health and Human Performance, Humanities, International, Law, Nursing, Physical and Mathematical Sciences, and Religious Education.

Figure 2. Description of the positions held by the participants in the surveys.
 

Review of Calls to the Support Center

The Instructional Media Center (IMC) at our university has been responsible for helping instructors and students with Blackboard support issues. The IMC employees note each call that is received, and categorize these calls as belonging to different categories such as course setup, uploading and downloading materials, and use of specific features. We included these calls as data by noting the number of calls received each semester, and the number received for each category. This data provides an easy way to regularly monitor which Blackboard problems are most prevalent on campus.

Faculty and Student Interviews

The interviews with faculty and students were conducted in 28 different departments on campus. Participants were sampled from every college on campus. Initially the sampling was random, but we also used theoretical sampling to find representatives of demographics not represented very well in the random sample and to develop our emerging theories (Glazer & Strauss, 1967). This sampling yielded 36 participants, and our interview protocol is represented in Table 1.

We also utilized some interviews (n=12) from a previous study by two of the authors (see Table 2 for a description of participants’ demographics). This previous study investigated patterns of use from instructors who integrated various kinds of technology on campus, and we selected those interviews that focused specifically on the use of a CMS to be included in the data pool of the present study. All of the interviews were semi-structured, one-on-one interviews, and the interviewers used a series of prompts about each category of interest to guide the discussion, while at the same time keeping the interview casual enough for the participants and interviewers to explore the topics. The student interviews (n=17) were shorter, intercept interviews where students were intercepted in popular places on campus and asked two or three questions about their experiences with Blackboard.

Table 2

Description of the demographics of faculty interview participants
by college and gender.

Together the student and faculty surveys, support center call record, and faculty interviews provide a rich set of data to understand the implementation of Blackboard at our institution.

Data Analysis

Quantitative Data Analysis

Quantitative data from the surveys and from the calls to the campus Blackboard support center were analyzed using descriptive data analysis techniques, an important method often used in educational research (Knupfer & McLellan, 1996). In this project, we reported the quantitative data from students and faculty as means and percentages, and then compared the data to identify patterns.

Qualitative Data Analysis

After transcribing the interviews, we began coding the data using a constant-comparison approach (Glaser & Strauss, 1967) to determine the sub-categories, properties, and relationships existing in the main categories, which had been determined by our research questions. Once all of the transcripts were coded, each domain or main category was re-analyzed to synthesize and determine the relationships between the codes and to identify the key findings relevant to each particular domain.

Establishing Trustworthiness of Qualitative Data

For the qualitative data in this project, we used methods suggested by Lincoln and Guba (1985) for establishing the trustworthiness of the research. These authors propose four standards for establishing trustworthiness: credibility, transferability, dependability, and confirmability. To meet these standards, we have emphasized using triangulation, member checking, and thick description. We triangulated the findings by using multiple sources of qualitative and quantitative data (interviews and open-ended survey questions) from multiple sites and contexts (28 university departments). To validate the main case studies, we checked the final descriptions of the cases with the participants to ensure they were accurate portrayals of their experiences. Finally, we have attempted to provide as much thick description as we could while still addressing all of the diverse aspects of our findings by utilizing quotes from the interviews and case studies. A research journal was also kept to describe the steps we took to complete this project.

Findings and Discussion

Our research purpose was to investigate the impacts from implementing a course management system on an institution-wide level. It might first be helpful, however, to review the context of our university’s association with Blackboard. In general, the use of Blackboard is fairly ubiquitous on campus. Our university has an average enrollment each semester of 30,000 students, who participated in just over 8,000 academic courses for both fall and winter semesters in 2004. There were also slightly more than 4,000 Blackboard courses created during each semester, meaning about half of the courses on campus were using the tool. Through the 2004-2005 school year, over 300,000 students and faculty are estimated to have used Blackboard at least once. In surveys given during fall and winter semester of the 2004-05 school year to faculty and student Blackboard users, instructors reported averaging nearly two Blackboard courses each while students reported being enrolled in an average of five courses each.

To foster more effective uses of Blackboard at our institution, we provided a wide range of Blackboard training opportunities for faculty. These training offerings included training for new users to familiarize them with how to set up a course and how to use the functionality within the Blackboard system. Additionally, there were training for more advanced users of the system that focused on using communication, and assessment tools as well as posting pedagogical content. Training and consultation was also offered that focused on technology integration and instructional design-related issues to help faculty better utilize the course management system.

Instructor and student usage of the CMS

We have learned that the truth about “whether Blackboard is used by instructors” depends on how usage is defined. For example, most instructors at our university only used four features: email, announcements, posting of course documents, and the online gradebook (see Figure 3), but, according to our interviews, they often only used the gradebook for communicating scores to students and not for actually calculating grades.

Another tricky aspect of understanding how much instructors actually used Blackboard was the impact of the teaching assistants (TAs). It is difficult to completely define how many instructors “used” Blackboard, because while they may have had Blackboard courses listed under their names, some did not actually create or maintain the courses. Rather, they let their TAs run the online portion of class. We believe this was fairly common because instructors were more likely to use Blackboard in large courses, large courses often have TAs, and TAs usually are involved in grading, distributing documents, and other tasks that instructors usually associate with Blackboard. Because we found TAs to sometimes be the person coordinating instruction through the CMS, we feel that future training and CMS support will be more effective if an emphasis is also made to train the teaching assistants on campus, who could then help train the professors.

Satisfaction with the Tool

To answer our question about what the impact has been from using a CMS, we first will consider the satisfaction of the faculty and students. When Blackboard worked without technical difficulties—and this is an important “when” clause that we discuss later in this paper—most students and instructors at our university reported being satisfied with the tool because it was convenient for them to use, easy to learn, and helpful in their studies. Sixty-six percent of student survey respondents in the winter semester (67% in fall semester) preferred that their instructors use the tool (see Table 3), and 73% (winter) and 75% (fall) of students said it was easy to use Blackboard. Most instructors also indicated that they were comfortable and confident using Blackboard in their teaching, and that it was not difficult for them to use the CMS (see Table 4).

Table 3.

How satisfied students are with Blackboard
and how comfortable they feel using the tool.

Note. SD=Strongly Disagree, D=Disagree, N=Neither, A=Agree, SA=Strongly Agree.

Table 4.

How satisfied faculty are with Blackboard
and how comfortable they feel using the tool.

Note. SD=Strongly Disagree, D=Disagree, N=Neither, A=Agree, SA=Strongly Agree.

The features of Blackboard with which instructors and students were most satisfied were the announcements, course documents, gradebook, and sometimes email features (see Figure 3). The features with which they were most dissatisfied were the ones commonly referred to on campus as the pedagogical features: synchronous chatting tools, discussion boards, assessments, and the digital dropbox (see Figure 4).

Figure 3. Instructors and students most favor using the announcements,
course documents, gradebook, and email features of Blackboard.

Figure 4. The features of Blackboard with which
instructors and students are most dissatisfied.

Stability of the Tool

Closely aligned with faculty and student satisfaction of the CMS was its stability. At our university, the implementation of a CMS has been a very large project. Many other services on campus, such as the library, registration, etc., have been integrated into this CMS. With a system this large, there are bound to be some stability problems. However, it seems that the stability problems with Blackboard were excessive. In Fall 2005, 76% of instructors said they had experienced technical problems with Blackboard, but this jumped to a full 100% of winter respondents (see Figures 5 and 6). In the winter semester, over half of the instructors indicated they had encountered technical problems in just the previous week (see Figure 6). Students seemed slightly less likely to have difficulties with Blackboard, but this number also jumped in winter semester to 90% of students having encountered problems (see Figures 5 and 6).

Figure 5. Percentage of students and faculty who experienced
technical problems with Blackboard through Fall 2004.

Figure 6. Frequency of technical problems experienced
by  survey respondents from the Winter 2005 semester.

One of the biggest frustrations during this school year was that Blackboard was unstable in some of its most critical areas, and there were many sad cases of students’ test scores being lost after a long exam, of critical emails with important information being lost without any warning given to the instructor, and of students being locked out of courses or exams (see Figures 7 and 8 for data on the most problem-plagued features). These stability problems negated, for many, the benefits derived from using a CMS, and most students and instructors, when asked if they like the tool, respond, “When it’s stable.”

Figure 7. Number of problems reported, according to category, to the
university’s support center for Blackboard on campus from 2003 to 2005.

Figure 8. Types of technical problems reported to the Blackboard support center
on campus in the winter of 2005.

We have learned that stability has much more at stake than simple convenience. One instructor shared his story of how when Blackboard lost 25% of his students’ test scores, and several critical emails were not delivered, his relationship with his students was permanently damaged. "Whatever teacher ratings I would get this semester, Blackboard will take 1/2 a point off. It looks like it works great, but there are glitches," he said. Consequently, he was much less motivated to use Blackboard, or any computer-based educational technology in the future. “It's made me think I need to go back to the Stone Age and use hard copies. Then if I lose something, I'VE lost it," he said.

Blackboard can be Used to Improve Efficiency

While stability was a challenge, there were still many overall benefits to purchasing and supporting Blackboard at our university. The biggest benefits we found were gains in efficiency, convenience, and class organization and management. Many instructors felt that Blackboard was a tool that saved them in-class and out-of-class time and made teaching more convenient. Exactly half of the instructors responding to the fall survey and 41% of the winter respondents felt this way, compared with about 26% of instructors who felt Blackboard cost them time overall (see Figure 9). The majority of students also felt that using Blackboard helped them be more efficient (65% fall semester and 57% winter semester). However, for both instructors and students, this was not as strong of a majority as might be expected, probably because of stability issues, which increased in winter semester.

Figure 9. Most students and instructors feel Blackboard
helps them be more efficient.

There were several methods for using Blackboard that seemed to increase efficiency benefits, and here we will explain four that seemed to be most prevalent. First, it was very common for instructors and students to feel that Blackboard saved time in the distribution of class materials. As one instructor said, “I don't have to worry about creating a packet or publishing a textbook or creating a lengthy syllabus and pass[ing] out paperwork all during the semester. It's a major convenience for me.” A second benefit was that course materials could be updated electronically, and then students could be directed to the newest versions without the instructor needing to print copies of every document every time there is an update. Third, using Blackboard was helpful in pacing the instructor and the students and in keeping the course organized.

For example, in courses that had multiple labs connected to a single course, Blackboard was a useful tool for organizing the different lab sections, as well as for monitoring the TA activity in the different sections. However, even in one-section courses, using Blackboard helped the instructor pace the teaching because everybody was aware of the schedule for the class. One student commented that “It allows (and makes) professors be more organized. For the class that doesn't use Blackboard, I am often confused at what material I should know, assignments, etc.”

Finally, Chickering and Gamson (1987) wrote that feedback was a critical element to effective undergraduate instruction. Many instructors and students felt that Blackboard made the giving of feedback more efficient, thus allowing instructors to give feedback more quickly and more often. Instructors found that when they used Blackboard to communicate scores to students, there were less disagreements about mistakes, because the students were more aware of their standing in the course at all times. One instructor said, “It's a good way to keep track of grades, and make the students responsible for their own grades so that they can look, and if I haven't recorded something, it's their job to call me. . . . It saves time at the end of the semester so you don't get a whole bunch of students saying, ‘C+, I thought I had a B.’” Also the email feature and discussion board features allowed instructors and students to more easily communicate outside of class time, increasing the interaction and opportunity for feedback on performance.

Conclusions

In this study we investigated the ways that a CMS has been used on campus and their knowledge of how to use the tool. We also studied the impacts from using a CMS on faculty and student efficiency and satisfaction. Faculty and students overall were moderately satisfied with Blackboard as a course management tool, but only if it is stable. We learned that stability was a very major concern for instructors and students, and must be more adequately addressed. At our university, instructors and students were using the tool in many classes, but usually only for the posting of course documents, communicating of student grades, announcements and email. These features were primarily used to increase the efficient transfer of information from teacher to student.

Even though many instructors responded that they liked Blackboard overall when it was stable, most found some aspects of the tool that they did not like. For example, one instructor said, “I've used Blackboard for three years now. Some aspects I really appreciate, and other aspects I have grown to dislike intensely." However, even though Blackboard was not a perfect tool for most people, many instructors in interviews expressed concerns about the possibility that our university might leave Blackboard for another CMS option after they have spent a lot of time learning the tool, preparing materials for online distribution, and uploading many years’ worth of these materials onto their Blackboard courses. For example, one instructor said, “If we get rid of Blackboard, it would be discouraging. . . . You don't want to do this but once in a lifetime. I've put hours and hours of busy work into it. I'd like to be able to use it next year." Another instructor remarked, “It dramatically reduces the effectiveness of the tool [to change to a new tool]. The purpose of the tool, as I understand it, is to save time and effort; but every time you update it, you redo everybody's effort. Please don't do that.”

In this way, because of the large-scale adoption of a single CMS at our university, the tool became somewhat “mission critical,” and many instructors were dependant upon the tool. In fact, it was because so many felt that their instruction was now tied to the tool that stability was such a major issue, for losing Blackboard at critical moments for many instructors was like having the lights turned off in the classroom or the textbook not available in the bookstore. To some degree, these instructors felt married to Blackboard, for better and for worse.

A main motivation for this research study was to provide data for university stakeholders to decide whether or not they should continue paying the price tag every year for a large, commercial CMS, and whether the costs for purchasing and supporting a CMS are outweighed by the benefits. These are difficult questions to answer. If the goal of university administrators and teacher development services is to improve the efficient transfer of documents, grades, and communication between students and instructors, then using a CMS was an improvement and enhanced these types of educational activities. However, if the goal is to help educators to move towards more student-centered, constructivist, or exploratory instruction, then using a CMS did not appear to be as helpful, at least not the way it was implemented by most instructors in this study.

An alternative possible lens for interpreting these results may be provided by the Concerns-based Adoption Model (CBAM). In this model, Gene Hall and Shirley Hord (1987) describe a series of concerns that instructors have as they adopt an innovation. In order, these concerns are awareness, informational, personal, management, consequences, collaboration, and refocusing. The earlier concerns focus mostly on becoming aware of an innovation, and simply learning to survive successfully with that innovation. Later, instructors acquire concerns about consequences, leading them to eventually refocus their use of the innovation to improve consequences, such as learning outcomes. It could be that because the adoption of CMS technologies was still a relatively new occurrence for many instructors on our campus, instructors and students still had basic concerns of simply managing the technology, and were consequentially using the tool for those tasks that were easiest, such as to communicate with students about grades or announcements or to “hand-out” articles to read. This could explain why the tool was mostly used for teacher-centered instructional activities. This hypothesis leads us to hope that more creative exploration of the technologies may still be in the future as instructors’ concerns shift towards refocusing their practice so that other, more student-centered, learning outcomes can be accomplished by using the tool.

No matter which hypothesis is accepted, it is crucial for universities that purchase and support CMS technologies to continue conducting studies to assess the costs and benefits of these administrative decisions. We believe that the dimensions outlined in this paper including  student and faculty satisfaction, knowledge, usage, efficiency, and stability are important vital signs of CMS implementation that could guide similar large-scale evaluations. Like medical vital signs, these are attributes indicating a healthy implementation of an educational technology. Continual assessment in each of these vital areas can allow an institution to quickly notice problems and spend its efforts on collecting additional data about the areas of greatest concern so that prescriptive strategies can be created. These evaluation methods are scalable and could facilitate decisions, made by administrators, that will have significant impacts on improving how we integrate CMS tools on campuses.

References

Arnone, M. (2002). Course-management outfits still seek elusive profits. The Chronicle of Higher Education. July 12, 2002.

Brown, D. (2002). File cabinets in the sky. Syllabus. June 2002 (p. 22).

Carmean, C. & Haefner, J. (2002). Mind over matter: Transforming course management systems into effective learning environments. Educause Review. 37(6). 26-34.

Chickering, A. W. & Gamson, Z. F. (1987). Seven principles for good practice. AAHE Bulletin, 39, 3-7.

Collis, B. & Boer, W. D. (2004). Teachers as learners: Embedded tools for implementing a CMS.  TechTrends, 48(6), 7-12.

Dutton, W. H., Cheong, P. H., & Park, N. (2004). The social shaping of a virtual learning  environment: The case of a university-wide course management system. Electronic Journal of e-Learning, 2(1), 69-80.

Educause Center for Applied Research (ECAR). (2005). ECAR Study of Students and  Information Technology, 2005: Convenience, Connection, Control, and Learning. Accessed February 24, 2006, from http://www.educause.edu/ers0506.

Glaser. B. G. & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for  qualitative research. Mill Valley, CA: Sociology Press.

Greeene, J.C., Benjamin, L. & Goodyear, L. (2001). The merits of mixed methods in evaluation. Evaluation, 7 (11), 25-44.

Hall, G. & Hord, S. (1987). Change in schools: Facilitating the process. Albany, NY: State University of New York Press.

Hutchins, H. M. (2001). Enhancing the business communication course through WebCT.  Business Communications Quarterly, 64(3), 87-95.

Klecker, B. M. (2002). Evaluation of electronic Blackboard enhancement of a graduate course in school counseling. Paper presented at the conference for the Mid-South Educational Research Association, held at Chattanooga, TN.

Knupfer, N. N. & McLellan, H. (1996). Descriptive research methodologies. In Handbook of Research for Educational Communications and Technology, edited by David H. Jonassen, 1196-1212. New York: Macmillan.

Massimo, V. S. (2003). Integrating the WebCT discussion feature into social work courses: An assessment focused on pedagogy and practicality. Journal of Technology in Human Services. 22(1), 49-64.

Meerts, J. (2003). Course management systems (CMS). An Evolving Technologies White Paper for Educause. October 2003.

Merriam, S. B. (1998). Qualitative research and case study applications in education. San Francisco: Jossey-Bass.

Morgan, G. (2003). Faculty use of course management systems. Denver: Educause Center for Applied Research.

Olsen, F. (2001). Getting ready for a new generation of course-management systems. The Chronicle of Higher Education, 48(17).

Pollack, T. A. (2003). Using a course management system to improve instruction. Paper presented at the annual conference of the Association of Small Computer Users in Education, held at Myrtle Beach, South Carolina.

Vessell, D. C. (2001). Comparing the WWW and WebCT to traditional methods of supporting an  undergraduate psychology course: Is it effective? Unpublished dissertation.

Yip, M. C. W. (2004). Using WebCT to teach courses online. British Journal of Educational  Technology, 35(4), 497-501.

Young, J. (2002). Pricing shifts by Blackboard and WebCT cost some colleges much more. The Chronicle of Higher Education. April 19, 2002.
 

About the Authors