“No Significant Difference” Only on the Surface
Thomas K. Ross and Paul D. Bell
The purpose of this study was to compare the effectiveness of an asynchronous online platform and its traditional F2F classroom based counterpart in promoting deep learning or higher levels of learning according to Bloom’s taxonomy. Equivalency between the two modes of instruction was assessed using course grades as the outcome measure. The ANOVA results suggested that on-line education is comparable to classroom education at lower levels of abstraction (recollection) but as work progresses to higher orders of abstraction (application) significant performance differences arise that place on-line learners at a disadvantage. Regression analysis was then employed to understand why the performance difference between the two groups existed or if it existed when other explanatory variables were introduced. Finally, regression results were analyzed and reasons offered for why this difference in learning outcomes occurred for the two instructional delivery modes.
Keywords: surface learning, deep learning, no significant difference, asynchronous web based learning, face to face learning, Bloom’s taxonomy, learning outcomes, ANOVA and regression.
The growth in asynchronous web based undergraduate courses has been dramatic. It has, in large part, been spurred by the motivation to hold down costs while simultaneously expanding access to higher education (Jones, 2003). However, the legitimacy of this expansion in Asynchronous Web-Based Learning (AWBL) has been based on a genre of academic research known as “media comparison studies” (MCS). This genre of research is based on the finding of no significant difference between average achievement scores for learners in online versus classroom-based versions of the same course. The conclusion reached based on this lack of difference in student achievement is that online learning is as effective a modality for learning as is conventional face to face learning (Bell, 2001; Hadidi & Sung, 2000).
Some have argued that MCS (Diaz, 2002; Bendixen and Hartley, 2001) are inappropriate for establishing the effectiveness of asynchronous online courses. This is because such studies fail to assess the learning environment according to some specific blueprint or definition of effectiveness. Biggs, 1999; Ramsden, 1992; and Weigel, 2002 have further argued that the true measure of course effectiveness is related more to whether it promotes deep learning over surface learning than to whether a particular medium serves as the learning platform. They have defined deep learning as the critical analysis of new ideas and linking them to already known concepts and principles. It leads to understanding and long-term retention of concepts that can be used for problem solving in unfamiliar contexts. Deep learning promotes understanding and application for life. In contrast, surface learning is the tacit acceptance of information and memorization as isolated and unlinked facts. It leads to superficial retention of material for examinations and does not promote understanding or long-term retention of knowledge and information. Thus, a better gauge of course effectiveness would appear to be how well it promotes deep learning over surface learning. Accordingly a better method of comparing different course learning environments should include a comparison of how successful each is in promoting deep rather than surface learning.
Bloom’s taxonomy provides a practical rubric for measuring deep learning. It specifies six types of learning (or levels of abstraction) that can be used to describe one’s depth of learning. These include knowledge, comprehension, application, analysis, synthesis, and evaluation (Huitt, 2004) In the current study learners were compared on five of the six types of learning. They included knowledge, application and synthesis.
As different levels of abstraction require different study skills, the depth of student learning can be assessed according to the study skill employed. For example, Bloom’s taxonomy is used to categorize test questions that are reflective of a particular taxonomy or level of learning. Furthermore, if a student successfully answers a question at a certain taxonomy level then it can be inferred that specific study skills have been mastered. The mastery of certain study skills can, in turn, give information about the particular level of learning achieved according to Blooms taxonomy. Thus, an analysis of the number and type of questions and problems correctly answered by both face to face (F2F) learners and distance education (AW) learners can then be used to compare the depth of learning in each group.
The purpose of this study was to assess the ability of an asynchronous online platform and its traditional classroom counterpart to promote deep learning or higher levels of learning according to Bloom’s taxonomy. Equivalency between the two modes of instruction will be assessed using course grades as the outcome measure.
A Quality Management course delivered in Spring 2007 provides an appropriate case for determining if asynchronous web based (AW) online courses are equivalent to face to face classroom based (F2F) education. In this course the only difference between the AW and F2F students was the latter group came to class and the AW students viewed the in-class lectures via the internet. The class was built on the Blackboard course management system and all lectures were recorded using Mediasite. PowerPoint slides to accompany the lectures were provided in the Course Docs section of Blackboard and available to all students prior to the lectures. The Mediasite lectures were streamed almost simultaneously with the class lectures and remained on-line for the duration of the course. F2F students had the option to: 1) attend class, 2) view the lectures on Mediasite or 3) attend class and review lectures via Mediasite. Class attendance was not mandatory and was not considered in calculating student grades.
Class lectures began with a review of the homework assigned in the prior class (if applicable) and proceeded to new material. The new material drew heavily on the text (theory) and was documented in the PowerPoint slides. Class lectures provided the opportunity to flesh out ideas, draw additional examples of applications, and demonstrate cross-industry uses of quality techniques. Some of the lecture material may not have been covered or only introduced briefly in the PowerPoint slides.
The study group/descriptive statistics
There were 79 students enrolled in the course, 44 F2F and 35 AW. All but two of the students were junior or senior majors in the department offering the course and the two non-majors were enrolled in a five year accelerated Masters program within the same college.
* Final sample reduced by four due to withdraws and non-majors.
Student performance in the course was based on ten homework assignments (10% of total course grade), a midterm exam (40%), and a final exam (50%). Assignments and due dates were identical for both groups. The same exams were given to both groups and both groups completed the tests on-line. Grades on exams were determined largely on mastery of the subject rather than effort while timeliness and effort were major components of homework grades.
Two AW students withdrew after the midterm exam due to low scores. The average grade for this course was 76.11% and the standard deviation was 8.31%. On the standard four point GPA scale the grade was 2.16. Exhibit 2 displays the distribution of scores. The grades show wide variation in performance providing a good sample to study differences in performance between groups and identify the potential explanatory variables that could impact grades.
Grade Distribution by Setting
AW (n = 32)
F2F (n = 43)
Lectures viewed online
As expected course grade was positively correlated with GPA, and overall course score increased by 5.66 points for each one point change in GPA (p = 0.012). Unexpectedly overall course grade decreased by 0.37 points for every one year increase in age (p = 0.007). This is unexpected as the descriptive statistics, exhibit 1, demonstrate that the older group of students, the AW students, had the higher GPA, 3.23 versus 2.78 for the younger F2F students. The intercept, the expected grade when the explanatory variables are zero, was 26.25 and significant (p = 0.001).
For the F2F students, the four independent variables explained 36.3% of the difference in course grades. All variables were significant. Unlike the AW results course grade decreased by 0.54 points for every lecture viewed on-line (p = 0.040) by F2F students. The relationship between homework performance and overall course grade was positive and significant (p = 0.015) but the coefficient was much lower for F2F students. F2F student grades increase by 0.18 points for every point earned on homework assignments versus 0.36 for AW students.
The coefficients on GPA and age were in the same direction and had a similar magnitude to those observed in the AW class. The coefficient for GPA and probability was 6.58, p = 0.002 for F2F students versus 5.66, p = 0.005 for AW students. The coefficient for age and probability was -0.37, p = 0.020 for F2F students versus -0.36, p = 0.007 for AW students. The intercept was 55.71, considerably higher than the AW class, and significant (p = 0.000).
The major differences between the two groups were the impact of on-line lectures and homework assignments and other unmeasured variables. In both regressions the intercept is positive and significant. The intercept for the F2F class is more than twice as large as the AW coefficient (55.72 versus 26.25). Obviously excluded variables have a major impact as the intercept predicts much higher performance in the F2F class.
The ANOVA results suggest that on-line education is comparable to classroom education at lower levels of abstraction (recollection) but as work progresses to higher orders of abstraction (application) significant performance differences arise that place on-line learners at a disadvantage.
This finding may be a result of the differences between how the two groups of learners chose to engage in their learning. That is, whether or not learners attended lectures, accessed them via Mediasite electronic recordings, and/or whether they completed homework assignments.
Course content was presented in face-to-face class lectures, video recordings of class lectures, and summarized in PowerPoint slides. Moreover students had the opportunity to practice and learn course material via homework assignments. These different modes of learning content can be conceptualized in terms of how well they support active learning. If these modes of learning are viewed on a continuum of active to passive learning the most active form of learning would be doing homework assignments followed by attending the in-class lectures, viewing the recorded lectures, and lastly, reading the PowerPoint slides for course information content. This is because each type of learning listed above requires less purposeful educational activity on the part of students. Prior research on college student development shows that the time and energy that students devote to educationally purposeful activities is the single best predictor of their learning success (Kuh, Kinzie, Schuh,Whitt et al, 2005) In keeping with this prior research, the current results suggest that those students who took advantage of opportunities to engage in active learning were the most successful learners regardless of whether they were AW or F2F students.
Differences between F2F and AW students
Despite this global finding, a key difference between the two classes is the adjusted R2 (coefficient of determination) statistic. The adjusted R2 for AW students is nearly twice as large as that for the F2F students. However, it should be recognized that the decrease in the adjusted R2 in the F2F group is probably related to the fact that classroom attendance was not required or recorded. Had classroom attendance been recorded we would expect class performance would vary positively with attendance and a larger percentage of the variance in course grade would be explained. Furthermore, while the larger adjusted R2 for the AW group might have been impacted by viewing on-line lectures, it should be noted that other non-quantified factors such as reading the textbook and amount of time spent viewing lectures, could affect student performance.
While viewing on-line lectures was a positive and important factor in predicting an AW student’s grade, the negative coefficient for F2F students suggests that viewing on-line lectures is not a good substitute for class attendance. For example, a AW student who viewed all the lectures and received the maximum available points from homework assignments could expect to score 13.3 points lower than a F2F student who viewed no on-line lectures holding other variables constant. The difference in course grades would narrow as F2F students view the on-line lectures that is, substitute on-line lectures for class attendance. If a F2F student elected to view all the lectures on-line rather than attend class the difference in course score would shrink to 6.3 points.
As stated earlier AW students only accessed 68.3% of the recorded lectures indicating that the on-line lectures were frequently not used to obtain course material. Many students preferred to access course material via the PowerPoint slides. The low use of on-line lectures by the AW students may be attributed to a difference in opinion between the students and professor concerning the most effective and efficient means for learning course material. The professor viewed on-line lectures as the primary information delivery vehicle for AW learning; the on-line lecture usage statistics indicate that AW students often felt the PowerPoint slides were the primary vehicle for learning course material. The problem and risk of relying on this passive method of obtaining information is that students may believe that a cursory reading of slides is sufficient for attaining mastery of the subject. Yet, if they do not attend lectures or at least view the video recordings, they will miss extemporaneous remarks made during the lectures that do not appear in the prepared PowerPoint slides. Relying solely on PowerPoint slides, failing to attend or view entire lectures, and inability to recognize the importance of in-class remarks presented a major obstacle to successful test performance.
On the other hand, in-class students who were physically present for the lectures not only were exposed to in-class remarks but could also pick up the verbal and non-verbal clues from the instructor and/or their classmates as to the usefulness of these remarks. Obviously the availability of the professor’s notes via PowerPoint and the internet have increased the ease and risk of passive acquisition of information. Class notes, whether those of their fellow students or the professor, are inferior substitutes for actively attending class or viewing a lecture.
In addition to how course material was accessed (in person or via video recording), course scores were also correlated with homework assignments. As stated earlier both groups had similar performance on homework assignments but similar to the utilization of on-line lectures, regression shows these assignments had a larger impact on the overall course score for AW students. An AW student earning 90 points on homework could expect this performance to translate into a 32.4 point increase in their overall course grade, a F2F student earning the same number of homework points could only expect a 16.2 increase in their course grade.
The larger impact for AW students may emanate from the reduced opportunity for exposure to course material in this group. F2F students were observed working together in the classroom, consequently, it was probably easier for students to build study and support groups in the classroom setting. Expanded opportunity for extra-classroom cooperation may improve outcomes given the greater ability to learn from one’s peers or the opportunities for guiding and coaching one’s fellow students (and thus reinforce material in the student’s mind) in a face to face setting. The reduced opportunities for exposure and interaction (especially those of a spontaneous nature) in on-line environments may place additional importance on those instructional devices that are readily available to AW students.
Another prime difference between F2F and AW students involves the student’s ability to selectively view course materials. Students who attend class have little choice but to review prior assignments and be exposed to the elaborations on the new material (obviously daydreaming would be one way of maintaining physical presence without paying attention). In the asynchronous on-line environment a student who assumes he or she knows the prior material could skip that part of the lecture or put it on fast play in addition to not attending (viewing) or paying attention. Similarly AW students could decide after the basic material was introduced that further elaborations were not needed and terminate the session. On the other hand, premature termination in the classroom, i.e. walking out, is constrained by social convention.
Reason for selecting one medium vs. another
Previous research concerning the reasons for taking an on-line course has demonstrated an association between learning achievement and student reason for selecting a particular medium for his/her learning. Studies involving on-line learners [Roblyer 1999 and Collins & Pascarella 2003] showed that those learners who earned the highest learning outcome scores in on-line learning environments belonged to those who actively selected on-line learning because they wanted to regulate their own learning. In the group of AW learners some actively chose to learn on-line instead of taking a face-to-face class. On the other hand, other students had little or no choice of their learning medium because they lived at a great geographic distance from campus or were full time working adults with little opportunity to come on campus for their learning. Regardless of the reason, self selection by learners into one medium of learning versus another may have played a role in affecting the outcomes achieved by both groups of learners.
Asynchronous web-based learning is different than learning in a classroom!
Another factor that may have affected the study results is on-line learning is different than learning in a traditional classroom setting. That is, processing information from on-line lectures is more challenging than from classroom lectures. For example, in the classroom the number of distracters is deliberately kept to a minimum ( no TV, computer, phone, refrigerator although we are aware that students surf the internet and use cell phones in class). Each lecture was approximately three hours and students in the classroom setting were committed to spending this time in their learning with classroom breaks allotted in a controlled fashion. Moreover, the presence of other students paying attention in class provided an example and support for maintaining attention in class. To the contrary, AW students have to rely on their own capacity to self regulate and control their learning. Not all students are prepared to exercise such control (Turnbull, 2003). Therefore, some AW students may not have been prepared to control their learning and needed external control imposed by the instructor in the classroom to focus their attention. Furthermore, these AW students may not have anticipated putting in the same number of hours as “traditional” classroom students. Trouble in self regulating their learning is evidenced by their belief that reviewing the PowerPoint slides was sufficient to master the course material.
On the other hand, many AW students may have approached their learning in the same manner as they would traditional classroom courses. They recognized that PowerPoint slides were supplements to lectures and invested time and effort in viewing the on-line lectures. Differences in the willingness to learn from on-line lectures demonstrated that some students were adequately prepared for the independent environment of on-line learning while others were less capable of handling the demands of self-regulation required by on-line classes. This could partially explain why variance in learning outcomes was greater for AW students than F2F students.
Many educators assume that older students are more diligent in their study habits than younger students because they are more autonomous, self directed, and goal oriented in their learning compared to younger students (Knowles, 1984). As a result they are expected to succeed at a higher rate in their learning than their younger cohorts. However, previous research in on-line learning environments has yielded mixed results in this area. Some studies indicate that older learners are more successful learners (Alstete & Beutell, 2004),while other studies find no difference in rates of learning achievement among age groups (Tucker, 2001). In the current study older students had lower course scores than their younger colleagues. The age coefficients in the regression analyses were negative, similar in magnitude, and significant for both groups of students. This finding may reflect the higher burdens that older students face when they go back to school. As opposed to traditional college students, these students may have full-time jobs and families that require attention.
Previous academic achievement
Prior academic achievement was associated with higher course scores in the present study. Prior course grades, as measured by academic GPA, were a strong predictor of grades in the course: the difference between a B student (3.0) and a C student (2.0) translated into an additional 5.66 points on course grade for AW students and 6.58 for F2F students. This supports earlier research findings that demonstrate importance of previous academic achievement for predicting subsequent academic learning achievement.(Bell, 2006; Ishitani, 2003; Garavila & Gredler, 2002)
The present study demonstrates a large and statistically significant performance difference between AW and F2F students where the only difference between the two groups was the means by which lectures were delivered. F2F students performed better than their AW counterparts which is attributed to the more active learning environment provided by the traditional classroom setting versus computer based, on-line learning. Additionally it was shown besides performing at a higher level, F2F students were more consistent performers than AW students.
Similar to previous studies that found “no significant difference” between traditional and on-line class environments, this study found comparable performance for certain levels of learning. Specifically at lower levels of learning, knowledge and comprehension, performance was comparable across learning environments. However outcome differences arose when students were asked to apply, analyze, and synthesize information. While we show that higher level performance is positively correlated to use of course materials, completion of homework assignments, and prior academic performance, the learning environment continues to play a large and significant role in student performance.
We cannot conclude that the learning environments are comparable at anything other than the lower levels of learning. If we wish to achieve “no significant difference” between environments this study suggests that we will need to implement additional activities into the on-line environment to more actively engage AW students or become more selective in accepting students into on-line degree programs. This study shows identical activities (except for in person versus recorded lectures) do not result in similar student outcomes so more activity will be required in the less active, on-line learning environment. The study also demonstrates that certain members of the on-line community are well prepared for the demands of on-line learning so another possible step toward “no significant difference” may be to identify and accept only those students who have a high probability of succeeding in this environment. This study demonstrates that using the same learning and student selection techniques in traditional classroom and on-line learning environments produces dramatically different student outcomes when higher order learning is required.
Alstete, J.W., Beutell, N. (2004), "Performance indicators in online distance learning courses: a study of management education", Quality Assurance in Education, Vol. 12 No. 1, pp. 6-14.
Bell, P.D. (2001). Learning medical terminology in a traditional classroom versus the World Wide Web: Is there a difference in PA student performance? Perspectives on Physician Assistant Education, 12(4), 232-235.
Bell, P. D., (2006). Can factors related to self-regulated learning and epistemological beliefs predict learning achievement in undergraduate asynchronous Web-based courses? Proquest Digital Dissertations, retrieved on July 13, 2007 from
Bendixen, L., & Hartley, K. (2003). Successful learning with hypermedia: The role of epistemological beliefs and metacognitive awareness. Journal of Educational Computing Research, 28(1), 15-30.
Biggs, J. (1999). Teaching for Quality Learning at University, SHRE and Open University Press
Collins, J., & Pascarella, E. (2003). Learning on campus and learning at a distance: A randomized instructional experiment. Research in Higher Education, 44(3), 315-325.
Diaz, D. (2002, May/June). On-line drop rates revisited. Technology Source. Retrieved November 14, 2004, from http://ts.mivu.org/default.asp?show=article&id=981.
Garavilia, L. S., & Gredler, M. E. (2002). Prior achievement, aptitude and use of learning strategies as predictors of college student achievement. College Student Journal, 36(4), 616-625.
Hadidi, R., & Sung, C. (2000, August). Pedagogy of on-line instruction: Can it be as good as face to face? Paper presented at the Americas Conference on Information Systems, Long Beach, CA.
Huitt, W. (2004). Bloom et al.'s taxonomy of the cognitive domain. Educational Psychology Interactive. Valdosta, GA: Valdosta State University. Retrieved [date], from
Ishitani, T. (2003). A longitudinal approach to assessing attrition behavior among first-generation students: Time varying effects of pre-college characteristics. Research in Higher Education, 44(4), 433-449.
Jones, R. (2003). A recommendation for managing the predicted growth in college enrollment at a time of adverse economic condition. On-line Journal of Distance Learning Administration, 6(1), 1-12.
Knowles, M. (1984). The Adult Learner: A Neglected Species (3rd Ed.). Houston, TX: Gulf Publishing.
Kuh, G. D., Kinzie, J., Schuh, J. H., Whitt, E. J. & Associates (2005). Student success in college: Creating conditions that matter. San Francisco: Jossey-Bass.
Ramsden, P. (1992). Learning to Teach in Higher Education, Routledge .
Roblyer, M. D. (1999). Is choice important in distance learning? A study of student motives for taking Internet-based courses at the high school and community college levels. Journal of Research on Computing in Education, 32(1), 157-171.
Tucker, S. (2001) Distance Education: Better, Worse, Or As Good As Traditional Education? Online Journal of Distance Learning Administration, Volume IV, Number IV, Winter 2001 Retrieved 7/12/07.
Turnbull, E. D. (2003). Investigating the potential for improving experiential undergraduate curriculum through the concept of personality. Unpublished doctoral dissertation. University of Western Sydney, Sydney, Australia.
Wang, A. Y., & Newlin, M. H. (2000). Characteristics of students who enroll and succeed in psychology Web-based courses. Journal of Educational Psychology 92(1), 137-143.
Wang, A. Y., Newlin, M. H. (2002). Predictors of performance in the virtual classroom [Electronic version]. T.H.E. Journal, Retrieved July 13, 2007, from http://www.thejournal.com/magazine/vault/M2643.cfm.
Weigel, V. (2002). Deep learning for a digital age: Technology's untapped potential to enrich higher education. San Francisco: Jossey-Bass.
Thomas K. Ross, PhD, is Assistant Professor of Health Services and Information Management in the School of Allied Health Sciences at East Carolina University in Greenville, NC. Email: email@example.com
Paul D. Bell, PhD, RHIA, CTR, is Associate Professor of Health Services and Information Management in the School of Allied Health Sciences at East Carolina University in Greenville, NC. Email: firstname.lastname@example.org
Paul David Bell, PhD, RHIA, CTR
East Carolina University
Department of Health Services & Information Management
4340E Health Sciences Building
Greenville, North Carolina 27858