Editor’s Note: Peer assessment has been used to increase student autonomy and promote active learning. It has positive results when used as a collaborative process, especially for team assignments. This study looks at pros and cons of various approaches to make it’s research findings.

The Implementation and Evaluation of
a Peer Assessment Based Online Course

Harun YILMAZ

Turkey

Abstract

This paper focuses on the role of the peer assessment in an asynchronous course in higher education. The course was designed for adult learners who have a job which is mostly teaching-related in the K-12 setting and pursue their master’s degree in education. In this course, the students were exposed to the instructional planning skills and knowledge to develop educational courses and materials. The paper addresses the different aspects of peer assessment in this particular setting where the author, as a facilitator, had to create and maintain an environment in which students were expected to work effectively to accomplish tasks, interact with each other, share their knowledge so that promote peer learning, provide reflections related to their learning and their team’s work, and also they were required to meet the attendance and participation requirement. The study indicates that use of collaborative assignments and peer assessment not only facilitates student learning of course content and group work dynamics including communication, interaction, timing, diversity, leadership, conflict management, and a common goal, but also helps students develop lifelong learning skills to assess others and improve their own work based on peers’ feedback.

Keywords: peer assessment, e-learning, adult learning, online course, asynchronous learning environment, self assessment, collaborative learning, team work, higher education, online student reflections, instructional design.

Introduction

Peer assessment in an online environment has become one of the most popular methods to make students active in their learning process, as well as playing a role in their work assessment (Boud & Falchikov, 2007; Davies, 2006). In peer assessment, students are required to assess the rest of group members (McLuckie & Topping, 2004). While students are encouraged in giving feedback on peers’ work, they are realized the strengths and weaknesses of their work during the peer assessment process. Thus, they have chance to make their work better and facilitate their learning (Van Den Berg, 2006). In addition, by peer assessment, students are encouraged to engage in learning activities in class rather than assignment products or grades (Chen, 2010).

Peer assessment is considered as an innovative method to enhance learning by giving students some autonomy to provide feedback on each other’s work, as well as their work in the assessment process and to promote learning by involving students more in their learning process (McDowell & Mowl, 1996). According to Topping (1996) there are several advantages of peer assessment including that it helps students to develop evaluating and justifying skills, and to utilize content knowledge in order to conduct peer assessment. Also, in such an environment in which peer assessment being conducted, students are expected to share and discuss what they know with each other, help peers regarding assignments, and students gain insight.

Although peer assessment has many benefits, there are some drawbacks that need to be considered to enhance learning. Especially the reliability and validity issues need to be taken into account to make students feel that they have been assessed by peers accurately and assessment is valuable for them (Grieves, McMillan, & Wilding, 2006). Elliot and Higgins (2005) argue that accuracy of peer assessment is problematic, while Topping (1998) and Sahin (2008) found that there is high correlation between peer assessment scores and lecturer scores. In order to minimize the reliability and validity problems in peer assessment, Sluijsmans (2006) suggested providing students criteria explicitly defined for their use during peer assessment. Since criteria are used to correspond with students to create a common understanding, the success of peer assessment mostly depends on how much students use criteria in the peer assessment process.

In addition to peer assessment, using collaboratively designed assessments increases students’ motivation and satisfaction level; therefore, learning from collaborative group projects or assignments is facilitated more in an online environment (Fisher, Phelps, & Ellis, 2000). Since students interact with each other to complete their own tasks and accomplish a common goal which is submission of a group project to instructor on time in such an asynchronous environment, students become more active in their learning process. In this process, students not only learn to work with peers to, but also they overcome some group related challenges including being on time, giving feedback, and achieving tasks. Thus, students have some group experiences which can be used in their work and learn the course content as well in a collaborative learning environment (Murphy, Mahoney, & Harvell, 2000).

The Case

The course was aimed to provide students with the instructional planning skills which they are required to use when developing educational courses and related materials. The 14 participant students were distance students who were mostly teachers in the K-12 settings and pursue a master’s degree in education. They were grouped into three teams by the author and for the team formation, geographical regions and time zones were considered. Each team was specifically assigned to the Team Room in which they were required to interact online on a regular basis to complete their team assignments and evaluate their individual assignments and provide feedback to each other. Also, students were encouraged to coordinate their work without the use of teleconferencing or instant messaging, since this was an asynchronous learning environment. In addition, it was announced that amongst the team members, tasks should be allocated and every student should be able to do their tasks on their own time, and post questions and discussions on this coordination in their team room. In order for the team members to post their discussions in the team room was important because the participation of students was accounted for grade based on these posts.

The students were required to post their thoughts and reflections related to the facilitator’s questions and students’ responses to the Main Room individually. In this room, students were expected to participate in the discussions with minimal submissions and all communications are in the form of text-based.

Several assignments in the class were completed by teams of four to five students. After each team assignment, students were asked to complete a peer evaluation form to assess the contributions of each member of team. These forms were taken into account when assessing individual contributions to the team assignments. Each student was required to complete a project individually and this project was broken into four parts. In relation to each part, students were required to submit their partial assignment to the Team Room where team members were expected to assess the assignment and provide feedback. Based on the feedback, each team member was expected to revise his/her assignment.

Also, students were required to submit a paper summarizing what they learned by evaluating other students’ assignments. In addition to performance of the assignments, students were evaluated for the quality of their participation in the Main Room in which most interactions and knowledge transmissions took place, as well as in the Team Room in which team related activities and interactions took place.

The third room was a Chat Room in which students interact with each other to meet and share their biography with others. This room is used for social activities such as personal introduction and sharing course related web links or resources.

Methodological Approach

In this paper, both a qualitative and quantitative case study approach (Erickson, 1998) was presented. In order to understand what the peer assessment role is for student learning in the online environment, students’ contributions in the course rooms are presented and analyzed.

Berge’s four dimensions of online instructor roles (1995) were used to analyze the data. In the same way, Ashton and her friends (1999) suggested the ‘four hats’ of the instructor model consisting of four dimensions consisting of pedagogical, social, managerial, and technical. In order to understand the role of instructor in the online environment, these dimensions are helpful. The first dimension, pedagogical, gives instructors a facilitator role requiring asking questions to students for critical thinking and making students focus on course content. The second dimension is a social role that is required to create a warm and social environment in which students interact with each other to promote learning via collaboration. The third is a managerial role, involving developing assignments and setting deadline for each assignment and online activities. The last role of instructor is a technical role which helps students use online environment without any technical problem, lets technology be transparent in students’ online learning process.

Implementation

In this case, the instructor is responsible for developing a course syllabus that includes a course description, objectives, assignments and their points, course policies, information about attendance and participation, instructor’s contact information including name, title, email, phone, and available times. In addition, course syllabus provides some information about team assignments and contact information for technical problems. The instructor is also responsible for creating a learning environment in which students feel comfortable to interact with peers and instructor. When a course is electronically open to students, the instructor submits a welcome message to everybody and his biography to the Chat Room. He also asks students to introduce themselves to the class. When each student submits his or her bio, he submits a follow up message and asks some questions so that students quickly become accustomed to the class environment.

After the class meets each other, the instructor assigns students to teams including four or five students considering their geographical regions and time zones then he gives some directions about how they work together in their group room. Once teams are formed, he submits some questions to the Main Room. All students are required to answer these questions and these answers should be supported by course readings. The instructor gives some feedback on students’ answers and asks some triggering questions to enable students to understand course content well and have students use critical thinking skills. If needed, the instructor refers the course readings for question answers.

Being on time in assignment submissions is important and if students submit their individual and group assignments late, their points are cut based on how many days they are late. This penalty policy is written on the course syllabus and also it is submitted to the Main Room as a message so that all students have more chance to read.

Quantitative Analysis

The participation level of 14 students was different in the asynchronous classroom. Although the teams were formed based on geographical regions and time zones, some teams seemed to be more active than others. In Table 1 the number of students’ message submissions to the groups is given. According to the table, the instructor participation level is higher than % 65 of the students. Also, his participation is higher than the average of the student participation in the Chat Room.  Since the instructor only observes the Team Room and participation of students in this group, there is no message submitted to the Team Rooms. Instead, he sent general warning messages to the Main Room to draw attention.

Table 1

The number of students’ message submissions

In addition to the Table 1, to demonstrate the level of team participation in class groups, Table 2 is given. It seems that Team A produced the highest rank in the Main Room even though they had four members. On the other hand, Team C had the lowest rank in the Team Room and the Chat Room. It is interesting to see that while Team B had the lowest rank in the Main Room, they generated the highest rank in the Team Room.

Table 2

The team ranks

Differences between team ranks in the three different rooms are tested by using Kruskal-Wallis H test (Siegel & Castellan, 1988). According to results represented in Table 3, there is a significant difference in the Main Room submissions (Chi² _{2, 14}; p<0.05). Also, the Team A members submitted more messages than other team members. While in the Team Room, there is a significant difference (Chi² _{2, 14}; p<0.05), there is no significant difference in the Chat Room (Chi² _{2, 14}; p=0.25).

Table 3

The comparisons of team ranks

After weekly team assignments, students were asked to submit their evaluations on team members’ performance. When students were evaluating the peers’ performance, they considered six items including preparation, presence, contribution, timeliness, interpersonal relations, and feedback. For each item, a 4-point Likert type scale (1: Strongly disagree, 2: Disagree, 3: Agree, and 4: Strongly agree) was used to evaluate a student performance. The average of Team A Peer Evaluation is 3.958333, the average of Team B Peer Evaluation is 3.8, and the average of Team C Peer Evaluation is 3.986667. It is interesting to have these results. Although Team C had the lowest mean rank which is 3.00, they had the highest average peer evaluation result which is 3.958333. On the other hand, while Team B produced the highest mean rank, 10.80, the same team had the lowest average peer evaluation result which is 3.8.

Qualitative Analysis

Students were required to submit their individual assignments to their team for review. Based on peer reviews, they were supposed to make some improvements in their assignments before submitting them to the course instructor for grade. 14 students were asked to submit their peer evaluations regarding the most important concepts that they learned from the peer review process. Time was revealed as one of the most important concepts from these evaluations. This concept becomes very important when students work for their team assignment from different regions and time zones. One student summarized this view:

I have really learned that time is of the essence. It is better to complete your work early allowing time for changes and or peer review. Also, it is amazing how we are all working in different parts of the country, yet we are completing thorough, cohesive assignments through collaboration.

Another student confirmed this opinion: “Everyone finishing their parts on time so the whole assignment can be reviewed before the assignment is due”.

The second most important concept that emerged in the peer evaluations was importance of feedback. A majority of students stated giving and receiving feedback was crucial for the success of collaborative team work, even though some students may not be willing to share what they know with others. As one student explained:

We are all willing to see others view point and be willing to put ourselves into their classroom situations. I sometimes think we all take pride in our work and it is very hard to put a lot of effort into something and then have it changed.

The same student also added: “I learned that feedback is so key into really understanding items that can confusing”. On the other hand, one student put a negative side of giving feedback into consideration: “It is uncomfortable to point out mistakes in other people’s work but my group members seemed to take it in stride”. Benefits of feedback were expressed by students as “it is helpful to assist the writer in fixing his or her paper”, “reviewing others work helps you to review your own work with a more critical eye”, and “it assists everyone in writing a clear, precise paper”.

During the peer evaluation process, students stated that they learned why a team requires understanding and commitment, the power of leadership, organization, and communication. One student explained how communication plays an important role in a team: “Sharing information about your personal life helps to establish the lines of communication and I believe that my team members feel confident in each other and his/her abilities.  Everyone has really “stepped up” to the plate”. Since team work requires different people in terms of gender, age, and culture to work together, a team work becomes a melting pot for these people. It was interesting to have one student stated: “Patience is a very important concept to keep in mind when working with a group. As we are all busy individuals, coming together as a group can be difficult due to individual time constraints”.

In addition to concepts that students learned from the peer review process, students were also asked to reflect how they will use this learning in their future team work. A majority of students stated that time management at the online environment and for the peer review process was important. One student reflected this issue: “I must be apt to getting my work done in a more timely fashion rather than waiting a day or two before its due. I plan to really set a schedule and space out my work so it does not affect the team’s peer review of my work”. Another student confirmed this student: “I plan to work on my time management so I can put my input earlier for a better discussion”.

Another learning experience that arose in the peer evaluations was being flexible while working at the asynchronous environment. One student asserted that “Hope to give my students a better understanding of why they should be flexible when working with new people”. One female student considered the peer review process as a good learning opportunity for herself and she reflected: “Working with people who have different priorities in life and different styles is helpful in any educational or work setting”.

In addition to the peer evaluation reflections, students were asked to write a paper to summarize what they learned by evaluating and reviewing other students` instructional plans. A majority of the students declared that reviewing and evaluating peers’ work enabled them to learn creating a well-documented instructional plan. In addition, they liked the process of reviewing and evaluation other’s work. One student stated: “Reviewing and evaluation other students’ instructional plans is an important tool in reflecting and evaluating my own work. By reflecting and evaluation my own instructional plan, I am able to determine the strength and weaknesses within my own instructional designs”. Another student confirmed this view: “Evaluating and reviewing other student’s instructional plans is a very valuable and enriching experience. Since each student’s plan applies the principles of instructional design to a different content area, peer review offers the opportunity to develop a deeper understanding and insight into the design process. By discovering strengths and weaknesses in each plan, reviewers are able to reevaluate their own understanding of the process and improve their own instructional plan. Although the strengths and weaknesses of the instructional plans varied based on the individual student, some commonalities became apparent”.

One student pointed a negative side of the reviewing and evaluating others’ instructional plans: “Reviewing and commenting on a peer’s work product can be difficult. Finding a balance between making suggestions for improvement and being supportive is a skill teachers employ daily with their learners in the classroom but seldom practice with their peers. Some reviewers skirted this difficult task by concentrating on mechanics rather than content”.

Discussion and Conclusion

In this paper, peer assessment and collaborative learning have been argued to implement successful online courses in which the instructor is concerned with how well students are learning new concepts and skills. Students were required to assess both their own performance and peers’ performance to learn concepts and skills and suggest necessary corrections to peers. In addition, peer assessment is the way to evaluate the collaborative team assignments and of how much and each student has contributed to the final of the team assignments. Overall, students participated more in the Main Room to give responses to the questions that were submitted by the instructor were accounted to the final class grade. However, team members who participated in less than other teams tend to give more points, when they evaluate peer performance in the team. A reason for this tendency may be that students achieved the team objective with less but constructive participation or they may have more motivation to complete team assignments (Fisher, Phelps, & Ellis, 2000) because the same team participated more in the Main Room.

It seems that a leadership role plays an important role in the online courses. In each team one or two students got involved more either in the Main Room or the Team Room and acted a leadership role in completing assignments, giving feedback, and submitting the weekly team assignments to the instructor. Therefore, the online course structure needs to encourage students to take steps in the online teams. Also, class policies, information about participation, attendance, team construction and dynamics are important to provide students beforehand that the class starts. During the peer assessment process, to prevent reliability and validity problems, it is crucial to provide students with rubrics that communicate with students and explain what students are expected when using rubrics to evaluate team members (Grieves, McMillan, & Wilding, 2006).

The concepts of timeliness, constructive feedback, being flexible and patient, and communication have been raised in this study. A majority of the students stated that being on time is the essence of online courses so the online class implementation should consider this issue and encourages students extrinsically such as giving extra points or sending warning messages to all students. Another issue is giving feedback and interacting with class and team members via constructive messages. Students need to differentiate between class related submissions and personal submissions. In order for students to consider these differences, the online environment needs to be divided into several parts including the Main Room, the Team Room, and the Chat Room. In addition, there should be another room for the assignments that is restricted to students to see and open others’ documents. Before the class starts, it is important to provide students with information about how team works and how communication occurs and what they need to do in case of team conflicts. This support can be done by providing some tutorials and readings, as well as proactively sending messages or giving some directions regarding these issues to the class by the instructor.

References

Ashton, S., T. Roberts, & L. Teles. (1999). Investigating the role of the instructor in collaborative online environments. Poster session presented at Computer Support For Collaborative Learning (CSCL) 1999 Conference, Stanford University, CA.

Berge, Z. (1995). Facilitating computer conferencing: Recommendations from the field. Educational Technology, 35, 22-30.

Boud, D., & Falchikov, N. (Eds.). (2007). Rethinking assessment for higher education: Learning for the longer term. London: Routledge

Chen, C.-h. (2010). The implementation and evaluation of a mobile self- and peer-assessment system. Computers & Education, 55, 229–236.

Davies, P.  (2006). Peer assessment: Judging the quality of students' work by comments rather than marks. Innovations in Education and Teaching International, 43 (1), 69-82.

Elliott, N. and Higgins, A. (2005) Self and peer assessment: Does it make a difference to student group work?. Nurse Education in Practice, 5(1), 40-48.

Erickson, F. (1998). Qualitative research methods for science education. In B. J. Fraser & K. G. Tobin (Eds.), The international handbook of science education (1155-1173). Dordrecht, The Netherlands: Kluwer.

Fisher, K., Phelps, R., & Ellis, A. (2000). Group processes online: Teaching collaboration through collaborative processes. Educational Technology & Society, 3(3), 1-16.

Grieves, J., McMillan, J. & Wilding, P. (2006). Barriers to learning: Conflicts that occur between and within organisational systems. International Journal of Learning and Intellectual Capital, 3(1),
86-103.

McDowell, L., & Mowl, G. (1996).  Innovative assessment: Its impact on students.  In   G. Gibbs (Ed.), Improving student learning through assessment and evaluation (131-147). Oxford: The Oxford Centre for Staff Development.

McLuckie, J. & Topping, K. J. (2004). Transferable skills for online peer learning. Assessment & Evaluation in Higher Education, 29(5), 563-584.

Murphy, K. L., S. E. Mahoney, & T. J. Harvell. (2000). Role of contracts in enhancing community building in web courses. Educational Technology and Society, 3(3), 409-421.

Sahin, S. (2008). An application of peer assessment in higher education. The Turkish Online Journal of Educational Technology, 7(2), 5-10.

Siegel, S. & Castellan, N. J. (1988). Nonparametric statistics for the behavioral sciences (second edition). New York: McGraw-Hill.

Sluijsmans, D. (2006). A conceptual framework for integrating peer assessment in teacher education. Studies in Educational Evaluation, 32(1), 6-22.

Topping, K. (1998). Peer assessment between students in colleges and universities. Review of Educational Research, 68, 249-276.

Van den Berg, I. (2006). Design principles and outcomes of peer assessment in Higher Education. Studies in Higher Education, 31(3), 341-356.

About the Author