Editor’s Note: Learning architects continue to ask the question; ”What is significant?” In research we also ask the question; What is NOT significant?” From the results we can design more effective lessons for traditional and distance learning. This comparative study provides useful data that, when compared and combined with findings of previous studies, advances our knowledge of design and learning.

A Comparative Study Between Traditional and Distance Education Instructional Environments Involving Two Graduate Level Learning Disabilities Classes

Larry S. Tinnerman

Abstract

As education enters the 21^st Century, teachers are challenged daily to keep up with the rapid changes that are evolving at an even increasing rate. Learning theory and age old principles of education are being made to accommodate to new and faster means of information delivery, often utilizing technology. This, unfortunately, comes sometimes without full regard to the learner of the resulting qualitative impacts on teaching. This study focused on comparing various dynamics between two graduate-level courses in Learning Disabilities, taught by the same professor to both a distance education and a traditional class. The total class sizes were 25 traditional students and 22 distance education students. While the researcher attempted to have all class members participate in the study, seven distance students and three traditional students declined to do so. There were twenty-three traditional students involved in the pre and post testing sessions. The additional participant failed to respond to the other instruments used in this study.

Some of the findings have their support in the literature, including “no significant difference” (Shearer, 2000) (Phipps & Merisotis 1999). This was true when comparing both pre and post test outcomes using t-test analysis at the p<.05. Other items of significance using chi square at p<.05 centered on the following:

§         Student-to-student communication

§         Student-to-teacher communication

§         Student-to-content understanding of communicational issues

§         Impromptu and exploratory communication

§         Time spent in defining and understanding course content and expectations

§         Continuity of curriculum for instructor to instructor

§         Technology issues, including experience and expertise of both the students and the instructor

§         Perceptions of distance educational effectiveness

During the course of the research, there were other factors that were observed for distance education students including:

§         The awkwardness of the communication effectively using a nonverbal typed method such as chat or discussion forum

§         The compatibility of various software packages

§         The difficulty of teacher modeling for the students

§         The demand for the instructor to be both a content expert and a technical expert
 

KeyWords: distance learning, distance education, no significant difference, DL, DI, online learning, virtual classroom, asynchronous learning, synchronous learning, graduate study, computer learning, education in the 21^st century.
 

The Problem

The problem under investigation in this study was a comparison of a Masters level class being offered in both traditional and online delivery modalities. Factors being examined included student composition, methods of content delivery, instructional effectiveness, student achievement on both a pre and post test, student attitudes and perceptions in regards to the opposite educational model (traditional attitudes on distance and distance attitudes on traditional) and comparative advantages and drawbacks to each instructional model (distance education and traditional education). Please note that distance education and online are terms that are used interchangeably.

Importance and Relevance of this Study

For the past thirty years, there has been a growing interest in distance education. At first the idea was embraced in the business world as a way to train employees with a minimum disruption to employment productivity. It was considered cost prohibitive and ineffective to send individual employees across the country to training seminars and conferences. With the advent of the World Wide Web (WWW), it became more productive and effective to have workers receive training as part of a distance learning approach.

In recent years, academic institutions have begun to explore the possibility of not only offering training on line, but entire degree programs as well. This has spurred a grad deal of discussion and controversy among educators, administrators, and potential employers as to the relative effectiveness of distance education programs when applied to the academic arena. (Twigg, 2003) There have been many studies and articles published in the past few years which debate the issue of “no significant difference: in education (Brown, 2000). However, the debate over effectiveness is particularly critical when considering the preparation of future educators, particularly for educators dealing with students with special needs in the field of Special Education. It is important to know the limitations and strengths of such an endeavor, since the consequences of inadequately trained individuals in special education could have a profound impact on many children in our society.

The General Problem under Investigation

Is there a significant difference between students choosing to receive their degrees using a distance educational model as compared to a more traditional classroom approach? Further, are there differences in content delivery leading to variations in the quality and effectiveness of learning for students choosing to receive their degrees via distance education as compared to the traditional classroom approach? Particular attention will be paid to obstacles encountered by faculty, students or both.

Research Questions

1. What educational similarities and differences exist for individuals enrolled in distance education classes versus the traditional classroom setting, in particular, educational background, professional status in the field of education and professional status in the area of special education in particular?

2. How does the educational background in regards to parents and spouses compare between each of the two groups under study?

3. What demographic similarities and differences exist for the individuals enrolled in each educational setting? In particular: gender, age, marital status.

4. What are the perceived and specific time management differences that exist for the individuals enrolled in each educational setting? Particularity in regards to academic tasks performed and time spent in communication with instructors and classmates.

5. How do students compare on standard evaluations of content material when looking at students enrolled in each educational setting?

6. What are the perceptions and opinions of students in each educational setting of those students enrolled in the opposite educational setting (online, traditional course format) of the same course? Include in this section, perceptions of advantages in terms of quality and depth of instruction, relevance and educational obstacles.

Assumptions

1. Students in the on-line program were adhering to the specified conditions of both the pre and post tests, i.e. that both tests were to be taken without books or notes.

2. Students in both classroom answered questions honestly and openly under the condition of anonymity.

3. The on-line students’ activity logs reflected the actual time spent in each section of the course.

4. On-line students were actually who they say they were and did not have outside individuals complete their assignments for them.

5. The pre and post test instruments validly and reliably measure similar content.

6. The instructor delivered course content in a consistent manner despite the knowledge that this study was taking place.

7. The student participants in this study had no vested interest in the outcome of the study.

Limitations

1. The inability to observe distance learners as they complete their tasks.

2. Due to the anonymous nature of the sample, the inability to reconnect a student with their participant number if that student ‘forgets’ their participant number.

3. The inability to mandate that a student completes the project.

4. The inability of the researcher to observe all classroom interactions in both settings.

5. The duration of time available to complete the study due to the length of the semester in which the course was offered.

6. Consistent contact with distance learners is dependent on electronic communications such as on-line surveys or emails which had the potential to constituting significant delays.

§         Students in the population received instruction either by the traditional classroom approach offered in a campus classroom or a distance educational model offered via computer.

§         Students in the population ranged between 18 and 55 years of age, of mixed gender, marital status, socio-economic backgrounds, professional educational experience and employment status.

Participant Selection Process

Students were selected for this study using a census approach as constituted by their enrollment in the course by either traditional or distance instruction. It was this researcher’s intent to get as close to total participation from all the students in both settings as possible.

§         Participant Target Number

§         Traditional face-to-face class – 25 students

§         Distance Education class – 21 students

§         Total anticipated population – 46 students

Research Methodology

The study incorporated two integrated approaches

§         Experimental – The independent variable was the method of instructional delivery, i.e. the traditional classroom instructional approach or the distance method of instruction. Enrollment in the class, the instructor and course content were the dependant variables. Since the class was offered by the same instructor to both groups at the same time, this provided an excellent opportunity to examine the similarities and differences between the groups in the areas of content delivery, student performance and student satisfaction.

§         Descriptive – Attitudes and opinions of students in both groups were investigated through both open ended and Likert-like scale questions.

Procedure

1.       Students in both groups were randomly assigned participant ID numbers to assure their anonymity during the study.

2.       Week One – A Pretest of content materials projected to be covered during the first 9 week of class. This instrument was use to measure prior knowledge and to establish a baseline for both groups. The pretest was comprised of 20 multiple choice questions covering content materials taken from the course test bank for Chapters one thru six. The identical instrument was used by both groups of students.

3.       Week Three – A Demographic Survey was distributed to each group. Students again only used their participant ID numbers to assure anonymity.

4.       Week Seven – A Student Perception Survey was distributed to each group. Participants in both groups were provided the entire spring break period to complete their surveys (March 5^th – March 16^th, 2003)

5.       Week Nine – A Post test of content materials covered in the first nine weeks of the semester was given to both groups. The post test was comprised of 20 multiple choice questions covering content materials taken from the course test bank for Chapters one thru six. The identical instrument was used by both groups of students.

6.       Post week nine – It was planned to distribute a Faculty Feedback Questionnaire to all university professors in the Department of Special Education and School Psychology who were currently teaching distance education courses for the department to collect perceptions of the experience from the instructors point of view, but time limitations made this exercise impossible.

Data Instruments

1.       Pretest - comprised of 20 multiple choice questions covering content materials taken from the course test bank for Chapters one thru six. The identical instrument was used by both groups of students.

2.       Demographic Questionnaire – This tool was constructed by the researcher based on the questions on which this study was based. There was an attempt at developing construct validity based on the issues being investigated. This instrument has no formal measure for validity or reliability.

3.       Student Perception Survey – This tool was constructed by the researcher based on the questions on which this study was based. There was an attempt at developing construct validity based on the issues being investigated. This instrument has no formal measure for validity or reliability.

4.       Post test – comprised of 20 multiple choice questions covering content materials taken from the course test bank for Chapters one thru six. This instrument has no formal measure for validity or reliability.

Potential Threats to Internal Validity

1. Instrumentation – There were no formal measures of validity or reliability, although there has been an attempt at developing construct validity based on the fact the questions were designed to directly study the major questions being investigated.

2. Attrition - As all the students were assigned anonymously, the researcher had no method to guarantee that all participants would finish the entire process.

3. Testing – The instruments used have no formal measure for validity or reliability; however, they did possess content validity base on the fact that they were derived for the course content in both the pre and post tests and were identical for both groups.

4. Experimenter Effect –

   1. Observer bias – as the researcher also works closely with the department offering this course, there was the potential threat of bias in interpreting the data. For this reason, data collection was largely objective using Likert scales. The subjective sections of the study only included opinions of students and faculty.

   2. Hawthorne Effect – It was anticipated that the researcher was to attempt to observe the traditional classroom directly to document instructional procedures, however, it was noted that the presence of the observer in the classroom was potentially having an effect on the instructional methodologies utilized by the faculty member. Therefore, this entire planned component of the study was eliminated.

Generalizability of This Study to Other Sites/Subjects

While this study is particularly restricted to one course by one professor teaching in the two investigated instructional methodologies, it is believed that certain aspects of this study will generalize to other sites and similar subjects. To be sure, individual instructor approaches to the tools and instructional methods available will have a wide impact on the effectiveness of instruction, regardless of the mode of delivery. This can be said of even comparisons of traditional classroom environments across varied teaching methodologies. However, certain pervasive points should emerge, especially in the areas of perceived effectiveness.

Basic advantages, obstacles and/or barriers to the distance educational modality of instruction can be kept in mind by instructors when designing future criteria to either maximize or minimize their impact through creative use of instructional methodologies. This study can be used as a tool in conjunction with other like investigations through other instructional venues in order to provide a cross comparison of similarities and differences. In particular, this research should help in the planning and construction of future courses offered by universities at the graduate level to make the course offering the most effective experience for the student that is possible in both the traditional and distance educational environment. This comparative study can supply hints as to what is instructionally effective regardless of the method of delivery.
 

Findings

Overviews of Procedures

The methods of analysis included chi-square analysis of the questionnaire data and t-test analysis of the pre and post test data. Both analysis methods were performed at the p<.05 level of significance.

Sample size: N=37 students (15 from the distance education class and 22 from the traditional class settings). The total class sizes were 25 traditional students and 22 distance education students. While the researcher attempted to have all class members participate in the study, 7 distance students and 3 traditional students declined to do so.

Departure for the original planned procedure included the following decisions:

1. Not to pursue the time study option due to an inability to accurately assess the time spent by on-line participants. In addition, the tools necessary to monitor the time spent in the traditional classroom were not accessible.

2. Not to administer the faculty questionnaire due to both faculty and personal time constraints and faculty availability due to the ‘crush’ at the end of the term.
    

Presentation of Data

The Demographic Questionnaire

A base demographic analysis of both groups was conducted. In regard to gender, females constituted the majority for both groups. However, the genders were somewhat more balanced in the traditional classroom setting. The differences between the groups were, however, not significant. The racial composition for both groups was 100% Caucasian. The marital status of participants between groups varied a bit, but again, was not significant. In the traditional class, the majority of the students were single, while in the on-line class, the majority of the students were married. There was no significant difference between the two classes in regards to the number of children in the household, with the majority of both groups reporting “no children.”

The educational background of parents and spouses of both groups was examined. When looking at the highest level of education completed the participant’s mothers, the majority of the respondents in the traditional class indicated high school while college was the highest reported statistic for mother’s education of the on-line learners. However, the differences were not statistically different between the two groups. While the similarities between the two groups in regards to their fathers’ educational background was more similar, the on-line group seemed to indicate a higher percentage of fathers completing advanced degrees at both the Masters and Doctorate levels when compared to the traditional group. As far as the educational levels of spouses, both groups were similar without any significant differences or trends. One observed factor, however, was that in the traditional classroom, the majority of students were single.

Looking at the years since undergraduate school offered no significant difference between the two groups with the majority of the traditional participants indicating a rage of 1-3 years and a majority of on-line participants divided between 4-5 years and 10 or more years. Examining further educational background of students for both traditional and on-line classes, several factors offered significant differences. Credits taken towards the masters degrees between the two croups varied significantly. The data indicates that traditional students have taken significantly more graduate course work in preparation of their graduate degree with 11 members (50%) of the participants having taken more than 28 credit hours. This contrasts with the 14 members (93%) of the on-line students taking between 10 and 27 credit hours.

Years teaching also varied significantly between the two groups in this study. In the traditional group, only 4 (18%) of the participants had ever taught, while 14 (93%) of the on-line participants had formal teaching experience.

When considering certification in the field of Special Education, only 1 of the traditional students had special education certification while 4 students in the online class had special Education Certification. However, this difference did not rise to the level of significance.

When certification is generalized to fields other than Special Education, the differences between the two groups was found to be significant with 5 (23%) of the traditional group and 12 (80%) of the on-line group claiming such certification. When asked if the participant functioned in the role as a special education teacher, there were no traditional students functioning in this role as compared the distance education group in which 9 (60%) of the 15 participants functioned in some role as a special educator. This variance was determined to be significant.

When asked if there is or has been a special needs child in the family, both groups appeared to have between 36% and 40% representation. While the differences between the groups were insignificant, it is interesting to note that the presence of such a trend existed in both groups.

Opinion Survey

The opinion survey was divided into the following subdivisions:

The students’ perception of time management (8 questions)

The following items indicated no significant difference between the two groups:

§         Perceived time spent reading class text

§         Perceived time spent reading supplemental assigned materials

§         Perceived time spent writing assignments for class

§         Perceived time spent in communication with instructor

§         Perceived time spent in class research activities

The following items indicated a significant difference between the two groups:

§         When looking at participants’ perceived time spent in dealing with technology issues, 100% of the traditional participants indicated time spent to less than 1 hour per week. However, 60% of the on-line participants reported to spend from 1 to 2 hours per week dealing with technology concerns.

§         When looking at participants’ perceived time spent in communication with other class members, 100% of traditional participants indicated less than 2 hours per week. However, 40% of the on-line participants reported spending time in excess of 2 hours a week. This variance between the two groups was determined to be significant.

§         When looking at participants’ perceived time spent each week in total class activities (Including time spent in class in chats, discussions, and other time related to class activities) 59% of the traditional participants’ reported less than 2 hours which when contrasted to the on-line participants responses where 40% reported 2-5 hours, 20% reported 6-10 hours and another 40% reported more than 10 hours.

The students’ perception of student to student communication (5 questions)

The following items indicated no significant difference between the two groups:

§         Accessibility of other students in your class

§         Effectiveness of student to student discussions to progress or interest in class

§         Frequency of contacting other classmates in completion of assignments

The following items indicated a significant difference between the two groups:

§         When participants’ perceptions of other classmates having a broad scope of knowledge that helps make the course topics relevant, only 27% of the traditional participants strongly agreed while 60% of the on-line participants strongly agreed.

§         Further inquiry was made as to participants’ contacting other classmates during the week in a social or causal context, 67% of the on-line participants claimed that they somewhat agree or strongly agree as compared to 32% of the traditional participants.

The students’ perception of student to instructor communication (5 questions)

The following items indicated no significant difference between the two groups:

§         Instructor accessibility

§         Prompt and helpful feedback from the instructor in a timely manner

§         Instructional methods learn and understandable

§         Instructor sympathetic to needs and situations of a graduate student

§         Instructor provides ample opportunity for questions and the voicing of concerns and opinions.

There were no items indicating a significant variance in this category of questions.

The students’ perception of course effectiveness and relevance (7 questions)

The following items indicated no significant difference between the two groups:

§         Factors beyond the control of the student, instructor or university that contribute to frustration and confusion

§         Relevance of material presented in regards to personal career goals

§         Generalizability of material to other professional and or personal life endeavors

§         Material found to be thought provoking and intellectually stimulating

§         Satisfaction as to how class is progressing

§         Perception of the difficulty of participation in the on-line version of this class

The following items indicated a significant difference between the two groups:

§         Ninety-five percent of the traditional participants found that they strongly agreed that they found the course content to be clear and understandable. This strongly contrasted with only 53% of the on-line participants. While the majority of the respondents in both groups strongly agreed, the variance between the two groups was found to be significant.

The students’ perception of the varied delivery method and overall satisfaction (2 questions)

The following items indicated no significant difference between the two groups:

§         Overall satisfaction level with the class (100% of both groups indicated that they were either Somewhat Satisfied or Very Satisfied.

The following items indicated a significant difference between the two groups:

§         When asked for their perception as to which model would offer the greatest depth of information for the student, 100% of the traditional participants indicated that the traditional classroom would be best as compared to 73% of the on-line participants. It should be noted that 27% of the on-line participants’ believed that the distance learning approach offered greater depth.

In a descriptive section of the questionnaire, students were asked to comment on the perceived advantages and disadvantages found between the two different modalities. The following items help summarize comments made by both groups.

Traditional –

§         More face to face interaction in traditional setting

§         No need to travel to class

§         Ability to work at times convenient to student for on-line setting

§         Ability to ask questions and seek clarification in traditional setting

On-line –

§         Time, flexibility, lessened inhibitions online

§         No child care expenses for on-line format

§         More face to face interaction in traditional setting

§         Working at one pace for LD students online

§         On-line class more time consuming

§         Stronger foundation in learning having to find information in varied locations

§         Note taking difficult in traditional class, can’t keep up

§         Clarity of instructors directions more difficult on-line sometimes

§         Immediate feedback available in traditional setting

§         No classes to attend, therefore no penalty of missing class online

§         Immediate dialog in traditional setting

When asked to make suggestions as to how to make this class more effective, the following summarizes comments from both groups.

Traditional –

§         Instructor needs to be more specific when giving research instructions

§         More discussion of interest to each field

§         I am happy with the class

§         Less time reviewing previous material

§         Clearer syllabus (research assignments required)

On-line –

§         Less work for on-line class

§         Better explanation on research project expectations

§         Open office hours on-line / instructor availability

§         No mandatory chats for online

§         Get graduation requirements out of the way sooner

§         Greater coordination between different professors

§         Other on-line classes need to be as structured as and as well organized as this one

§         More chat and discussion sessions to share concerns and advice among students

§         The course if fine as it is.

Pre and Post test analysis

Table 1

Descriptive Statistics of Pre and Post-test Performance
(N=37, Online Class = 15, Traditional Class = 22)

Table 2

t-test Analysis of Pre and Post-test Performance
(N=37, Online Class = 15, Traditional Class = 22, P<.05)

Two Sample t-test

Distance Education Pretest to Traditional Education Pretest: t=.80, p=0.431. df=32

There was no significant difference between the performance on the pretest for the distance education group and the pretest for the traditional education group at least p=.05

Two Sample t-test

Distance Education Post-test to Traditional Education Post-test: t=1.19, p=0.242, df=35

There was no significant difference between the performance on the post-test for the distance education group and the post test for the traditional education group at least p=.05

Two Sample t-test

Distance Education Pretest to Distance Education Post-test: t=4.01, p=.001

There was a significant increase in performance from the pretest to the post-test for the distance education group at least p=.05

Two Sample t-test

Traditional Education Pretest to Traditional Education Post-test: t=4.90, p=.000

There was a significant increase in performance from the pretest to the post-test for the traditional education group at least p=.05

Implications and Concluding Comments

Suggestions for future study include:

1. The study of the relationship between instruction velocity and rigor in the traditional classroom and the impact of the ability to regulate that in the distance education environment. Educational research has shown that the truly effective teacher establishes a pace of instruction with their students that “ebbs and flows” as needed. This concept is difficult to monitor in the on-line environment.

2. The study of the impact of learning styles with distance education students. Currently, the technology is a bit limited in addressing or considering this issue. What happens to the student who happens to be an auditor learner? What is the impact of distance learning that is largely tactile and visual?

3. The continued study of the on-line student group from this study as it comes to the end of their educational pursuit. A follow-up investigation would determine on how students in the online program performed on their competency exam as compared to their traditional counterparts?

4. A study to examine the effectiveness of new technologies and ways to deliver these technologies despite the limitations that the internet now imposes on the methods of delivery.

5. A study to investigate the optimal situation of using distance learning as merely a supplemental tool for the traditional classroom or hybrid.
    

“The marketing strategy in the on-line community must become… “Focus attention on what kinds of education people need, want, and for which they are willing to pay.” The pitfall is the notion of technology for technologies sake and forgetting the learners”. (Wilson, 2003, p.3 )

While students were able to perform equitably on pre and post test analysis, it is important to point out that the reproduction of fact on a test do not necessarily represent the fullness of the learning experience. This is a particular concern when examining the learning experience for the on-line students as compared to that of the face-to-face traditional class. As technology continues to surge forward, there are broad implications to today’s education administrator, educator and student (Twigg, 2003). The driving force behind education must continue to be research supported educational methods and strategies. Technology must never become the controlling force alone. It must be remembered that there is a difference between the teacher and the tools that the teacher uses.

While many implications can be derived from this study, the key points must focus on quality instruction, regardless of modality. In the past, education has made assumptions based on the convenience. While the endless practice and drill sheets were convenient, today we know they were not a best practice in education. Just because distance education is convenient, it must be effective as well to survive. Issues surrounding communication have a centerpiece in the discussion. A technological limitation such as uniform bandwidth which hinders video transfer tends to challenge today’s distance educator. Distance educators must continue to tackle these impacting issues with as much fervor as their mainstream educational colleagues to assure that both the student and the instructor are relating in as optimal am manner as can be designed.
 

References

Brown, B.L. (2000) Web-based training. ERIC Digest No. 218, ERIC Clearinghouse on Adult Career and Vocational Education, Columbus, OH.

Phipps, R. & Merisotis, J. (1999). What's the difference? A review of contemporary research on the effectiveness of distance learning in higher education . The Institute for Higher Education Policy. Retrieved February 2004 from: http://www.ihep.com/Pubs/PDF/Difference.pdf

Shearer, R. (2002, September 15). No significant difference and distance education. Retrieved February 9, 2003 from
http://www.distance-educator.com/dnews/

Twigg, Carol A. (2003) Improving learning and reducing costs: New models for online learning. Educause Review , 38:5. Retrieved February 2004 from: http://www.educause.edu/ir/library/pdf/erm0352.pdf

Wilson, J.M. (2003). Is there a future for on-line ed? University Business. 6(3), 7

About the Author

Larry S. Tinnerman is a Doctoral Candidate at the University of Indiana.
Mr. Tinnerman graduated from Edinboro State College in 1976 with a BA in Psychology and a teaching certification in Secondary Social Studies. In 1998, Mr. Tinnerman taught computer science at a Junior College in Erie, PA. In 2003 Mr. Tinnerman earned his M.Ed. in Special Education and helped develop the departments distance learning program. Mr. Tinnerman has also taught 6^th and 7^th grade learning support classes in the public schools. Mr. Tinnerman is currently a doctoral student at Indiana University of Pennsylvania in Curriculum and Instruction where he currently teaches courses in elementary education as a teaching associate.