Editor’s Note: At the beginning of my teaching career I had the good fortune to watch a tutor teach lip-reading to 3 year old deaf child at the John Tracy Clinic for the Deaf at University of Southern California.
At the end of the session, the teacher came into our viewing room behind a one-way glass mirror. Her colleague immediately went through a list of suggestions to make the lesson more effective. I was shocked.
I thought the teacher had done a brilliant job. When the teacher left I questioned the critical comments.
“We do this for each other at every session. It is the only way we can learn; the only way we can improve.” At that moment I realized that mutual trust and a common goal opened the way for continuous improvement of a life-changing process for these children. The process brought out the best thinking of teachers-helping-teachers, resulting in outstanding performance. That is what this article is about.

Don Perrin, Executive Editor

 

Flying or Falling:
Benefits and Pitfalls of Online Peer Review Programs
in Distance Education

Kim Blum and Brent Muirhead

Introduction

One author of this article had a rare opportunity to go river rafting in Australia. On the second trip, the river guiding company placed the author in a raft with the author’s grown son and six members of an All Aussie Rules over-30 football team. An Aussie All Rules Football game is played with few fixed rules, no padding, and men over seven feet tall that weight at least 250 pounds of muscle. They have massive scars and scrapes from playing the game.

The author felt, as a result of a previous successful rafting trip where rapids were very difficult, this lower-classed river grade should be quite easy. Rivers for rafting are graded by difficulty, one being the easiest and six almost impossible to navigate by a paddle and raft. The author felt that previous experience on a class six river ensured that, because of this river’s easier classification, the raft would fly down the river in an enjoyable manner. The author was wrong.

After boarding the raft, the river guide instructed his team on how to use the paddle, lean right, lean left (to avoid rocks and tilt the boat), forward paddle, and back paddle. The worst-case instruction was given and all rafting team members practiced this successfully in the entrance’s calm pool waters – get down and hold on (to the rope), lift the paddle up out of the water. When the guide issues the instruction to hold on, rafters are typically afraid because hold on means the upcoming rapids are very difficult and wild. After the initial training, the team felt ready to face the turbulent waters of the first rapids. The author’s previous experience added confidence. The raft was flying along the river headed towards the rapids and confidence levels were high.

As the raft approached the rapids, excitement and cheering erupted from paddlers of the All Aussie Rules Football team. All rafting paddlers closely followed the guide’s instructions except the author, who promptly fell out of the raft and into the swirling rapids, failing to succeed in spite of previous training, and in error about level of ability.

Similar to the author’s rafting experience, a successful online peer review is received by online faculty as a wonderful flying feeling of success comparable to the feeling of teamwork and exhilaration of a successful manned raft. An online peer review program can successfully further develop faculty after initial training (Carr, 2005). Unfortunately, an online peer review can also resemble falling rafters as resentful faculty members receive an evaluation instead of coaching on online best practices.

This article discusses the pitfalls and benefits of online peer review programs, sharing experiences from administration in higher education, and comparing administration experiences with an online faculty member who has been through the online peer review process. Implications of Peer Review Programs for higher education online faculty and administration are included.

Flying –Purpose of a Successful Online Peer Review Program

According to the U.S. Department of Education (1999), one in three higher education institutions in the United States offered some type of distance education during the 1997-98 academic years. Nearly 80% of all 4-year and almost two thirds of 2-year public institutions made distance education available to students during this period. Of the institutions that did not offer distance education, 20% planned to offer some type of remote delivery service by 2002 (U.S. Department of Education, 1999). Ross and Klug (1999) found that one of the most effective things an institution can do to promote greater receptivity and support for distance education is to enhance faculty knowledge of distance education best practices. Peer review programs are one of the tools designed with the goal of helping and coaching faculty with best practices for success after initial online faculty training (Peer Monitoring of TRIO Programs, 2005). This is typically implemented when the faculty member has taught at least one online course.

A successful online peer review program rests on the concept of equal-in-status faculty members coaching other faculty members on best practices, giving tips on how to handle online discussions, assignments, feedback and materials, and communicating to the peer reviewee with a positive tone in a collaborative manner. Adams (2002) suggested that the best online faculty support systems are those where online faculty participate in a collaborative team, Faculty in the study expressed a strong degree of satisfaction with support provided by the collaborative model.

At University of Phoenix, Peer Reviewers in the School of Advanced Studies (SAS) observe the online class and provide tips to the faculty member written in a positive tone. Praises are included with all tips, based on a guideline of best practices proven effective online. A successful online peer review program should result in faculty incorporating suggestions and feeling that the results are not punitive in nature. Proponents maintain that peer assistance and reviews will help floundering teachers and possibly save their jobs (Pushing for Peer Reviews, 2005). “Opponents argue that it will pit one teacher against another and threaten the unity of local union associations”
(para 10). The faculty member should feel that despite the rocks and bumps in the river, the river guide – the reviewer – gave good instructions to keep from falling. In the case of the author, the guide hauled the rafter back in by the life jacket, and proceeded to re-coach the rafter on how to stay in the raft so that the raft could continue to fly with success down the river rapids. A successful online peer review should help faculty fly instead of fall.

Man the Paddles – The Processes of Online Peer Reviews

All faculty in SAS have yearly Online Peer Reviews. Newer faculty are reviewed after three months in order to provide more tips on best practices at an earlier stage. The faculty member receives notification of the review in the last week of an 8-week course, student feedback for one successful student and one struggling student is requested, and the reviewer observes the online class newsgroups and feedback. The trained peer reviewer observes the newsgroups in an unbiased manner, using a checklist of best practices as a guide for key areas including materials, discussions, classroom management, tone, and feedback. For example, SAS has found that online students achieve maximum student learning following the adult as an active leaner model (Knowles, 1984), when the professor facilitates discussion by relating work experiences, theories, and asks questions to stimulate higher levels of Bloom, Bertram and Krathwohl (1964) levels of critical thinking from application to synthesis and evaluation.

The online peer reviewer at SAS reports on the findings and incorporates praise and suggestions. The review is sent anonymously by a processor (the reviewer’s name is not released) to the faculty member for review and signature. Follow-up procedures include coaching, addressing questions, and explaining processes and purposes if the faculty member is unclear or has some trepidation about the review.

Fly Instead of Fall: Pitfalls to Avoid in Online Peer Review Programs

At the School of Advanced Studies (SAS), the underling mission of the online doctoral peer review program ensures that subsequent evaluative assessments by administration, triggered by results of a peer review, do not occur as an evaluation. It is critical that administration does not use peer review for any purpose other than faculty development. Online faculty fear that evaluations can result in punitive scheduling or pay reductions. The word evaluation is not consistent with goals of the peer review program. Its core mission is faculty-helping-faculty.

SAS determined that administration must avoid acting on information presented in peer reviews. The goal is to help faculty succeed and findings should be highly confidential. A distinct and separate faculty evaluation process must be clearly differentiated from the Online Peer Review program. The Faculty Evaluation process is based on findings of the 1981 Teacher Peer Review program that successfully developed teachers (Pushing for Peer Reviews, 2005) positing that peer reviews can be positive developmental experiences if used in this manner.

For example, one author of this article participated in an additional online peer review program. One hundred and sixty-five faculty complaints resulted from peer reviews because of the evaluative nature of the comments recorded by the peer reviewers. In spite of reassurance from administration, follow-up administrative actions stemmed from information noted in the reviews. The cycle of mistrust escalated until peer review processes and communications were changed.

The opposite occurred in SAS. After two years of online peer reviews, SAS has not received any faculty complaints. Comparing patterns in responses and plans, the author concluded that there were several reasons why one online peer review program succeeded where another failed. The successful peer review program had key differences not found in the unsuccessful program:

• Invitations to become SAS Online Peer Reviewers were the result of months of research on all SAS faculty. Only the best of the best faculty were invited based on SEOCS, faculty evaluations and observing many of their online classes.

• Extensive training of Online Peer Reviewers in a formal training workshop presented the purpose and guidelines as well as additional time to practice reviews. Follow-up trainee responses received individual coaching on tone, how to avoid certain words that could be perceived as negative, and tips on how to formulate coaching to ensure a positive review. As part of the training, instruction and practice focused on the sandwich method of praising, tips, and ending in praise as an effective online coaching method. This follows finding by Wolf (2003) that the choice of words in a peer review is critical.

• Peer Reviewers conducted one real review, the trainer edited the review, and peer reviewers received additional coaching. A decision to continue with the peer reviewer or decline to award additional reviews depended on the outcome of the analysis.

• Establishment and maintenance of a Peer Review lounge and a questions contact person.

• SAS faculty were continually reminded that the purpose of the Peer Review Program is faculty helping faculty. This is not an evaluation. Peer reviews are never punitive in nature, and scheduling or pay is never affected. Reminders came to all faculty from top leadership in SAS on a frequent basis.

• Faculty could see, after an extended period, that administration truly modeled the purpose of the peer review with faculty coaching faculty with no punitive outcomes., faculty were never contacted after any review and coached as an evaluation and scheduling or pay was not affected. Reductions in the levels of faculty fear resulted, and the grapevine did the rest as faculty relaxed and accepted coaching in reviews.

Faculty Perceptive of Online Peer Reviews:
Benefits (Flying instead of Falling)

Reflective online faculty have a positive and visionary perspective on professional development. Educators realize that teaching and learning is an evolving process which requires constant attention, experimenting with various instructional strategies and investigation to acquire more effective methods (Brookfield, 1995). Today’s online instructors should strive to be life-long learners who realize that it takes time and diligent study and practice to become an expert. Fear of falling must be ignored as the online educator receives tips from the reviewer – the guide.

Cognitive psychologists stress that it often takes ten years for a person to become an expert (Anderson, 2005; Schacter, 1996). Experts have “ …a highly refined and powerful form of elaborate encoding that enables experts to pick out key information efficiently and to imbue it with meaning by integrating it with preexisting knowledge” (Schacter 1996, p. 49). Experts possess two kinds of expertise: routine and adaptive. Routine expertise enables the individual to do problem solving in an effective and timely manner. Adaptive expertise skills are those which help people to develop strategies that fit the particular situation (Eysenck, 2001).

Online distance faculty should consider peer reviews as an intentional way to cultivate their expertise. Teaching should be considered a craft and the word craft highlights that teaching requires the acquisition and refinement of unique skills and knowledge. It brings a sense of dignity to teaching as people focus upon producing quality instructional materials, intellectually stimulating online discussions and relevant feedback on student assignments. Online faculty who embrace teaching as a craft will be more likely to operate by an internal standard of excellence that helps them to cultivate a work ethic and be a colleague who willingly shares best practices with others. Additionally, a growing sense of confidence is characteristic of reflective teachers because it enables instructors to avoid the paralyzing effect of always having to prove or compare themselves to others (Sennett, 2003).

Research studies on experts in distance education have found that skill development and developing expertise are tied closely to the timing, quality and quantity of deliberate practice. The use of mentors plays a vital role by providing guidance, monitoring progress and establishing appropriate goals that promote optimal growth. Bruning, et al (2004) noted that research indicates deliberate practice can help less talented people surpass the achievements of those who are more talented. Skill acquisition among young athletes, mathematicians and musicians indicates that individuals follow a similar learning process. The key is having the appropriate guidance and intentional practices that cultivate superior performance. "The best practice occurs under the watchful guidance of a skilled mentor who helps the developing expert set goals and monitor improvement" (Bruning et al, 2004, p. 177).

Professional staff development programs can utilize some of the principles found in effective mentoring through the peer review process. Online distance education faculty can benefit from the insights of a peer reviewer who can provide clarity to their work and being receptive to insights from the review on what the faculty member is doing well and what areas that for improvement is critical. Online teachers can be proactive by participating in conferences, reading literature, sharing with their colleague’s best practices and instructional resources and sharing tips learned in peer reviews with others.

The University of Phoenix encourages its SAS online distance education faculty to refine facilitation skills and instructional practices using online peer reviews, and by encouraging faculty to participate in a variety of free online workshops that are held both online. The following list of workshops is a partial list of the professional activities that are available free of charge to online faculty members (Faculty Development Workshops, 2005):

New Student Facilitation: Helping Learners Succeed - focuses on resources and strategies facilitators can use with new students. Addresses common characteristics of new SAS students and present methods for helping entry-level students develop core competencies.

Online Tone - emphasizes importance of proper tone in an online classroom, focusing on the application of appropriate tone when responding to and offering feedback to students. It will enable faculty to identify and develop communication skills that are necessary for teaching in the online classroom.

Critical Thinking - introduces the components of the critical thinking process and identifies various methods for teaching critical thinking skills. Topical areas include taxonomies and frameworks for understanding critical thinking, and cognitive abilities and affective dispositions in critical thinking.

Learning Teams - facilitation techniques and evaluation methods. Participants will explore Learning Team processes, including conflict resolution, behavioral guidelines, and factors affecting team interaction.

Difficult Student - various approaches for resolving several types of conflicts. Participants will examine team dynamics and strategies for helping students with team approaches.

Evaluating student writing - develop clear writing assignments, assess student papers effectively, and help students improve their writing skills. The workshop provides a review of writing principles and includes materials to assist participants to establish clear grading criteria for written work and provide effective feedback to students.

Plagiarism - provides the knowledge and tools necessary to detect plagiarism.

Student Evaluation - addresses grading plans, criteria for grade changes and grade grievances, and qualities of effective feedback

Some online educators have legitimate concerns about peer review evaluations being accurate and objective toward the teaching and learning process. Educators fear having subjective reviews that are politically motivated or have excessive focus on minor administrative details (i.e. alternative email address in a syllabus). It is important that the review process be based on objective and measurable evaluation standards and that administered by trained personnel who understand the dynamics of the teaching. Teachers can profit from constructive insights into their online work and sharing with colleagues who relate to the challenges of distance education. Brookfield (1995) eloquently describes how reflective teachers maintain a sense of high academic expectations and a positive mindset toward the educational process of online successful acceptance and embracement of the results of an online peer review:

Critically reflective teachers learn from the past but live in the present with an eye to the future. Because they know that every class has its own dynamic, they cease to rely only on methods and activities that have worked in the past. Their practice is infused with a sense of excitement and purpose. There is a continual checking of assumptions, a continual viewing of practice through different lenses, and a continual rethinking of what works, and why. Knowing that each new group of students brings its own challenges, they see their life as lived in forward motion. Because tomorrow is unpredictable, there is always the chance for new learning from practice. (p. 265)

Implications and Conclusions

Online Peer Reviews can be a win-win situation for faculty and administration if the program is established in a similar manner to the rafting guide coaching the rafters. Fear of falling must be overcome by supportive administration, careful peer review selection and training, and communication to online faculty about the non-evaluative nature of the peer review. Online educators should be educated about coaching and helpful objectives of the online peer review. They should be encouraged to share tips and best practices with colleagues and be receptive to suggestions and changes. Administration should carefully plan and monitor any online peer review program to ensure that it is effective in meeting its goals. The successful online peer review is to help faculty fly instead of fall, and encourage online faculty to participate in faculty workshops for further development.

References

Adams, J.R. (2002). Implementing a collaborative support model for online faculty: a case study. Dissertation. University of Virginia. Available at Proquest.

Anderson, J. R. (2005). Cognitive psychology and its implications (6th ed.). New York, NY: Worth Publishers.

Bloom, B.S., Bertram B. Mesia, & D. R. Krathwohl (1964). Taxonomy of Educational Objectives (two vols: The Affective Domain & The Cognitive Domain). New York. David McKay.

Brookfield, S. (1995). Becoming a critically reflective teacher. San Francisco, CA: Jossey-Bass.

Bruning, R. H., Schraw, G. J., Norby, M. N., & Ronning, R. R. (2004). Cognitive psychology and instruction (4th ed.). Upper Saddle River, NJ: Pearson.

Carr, J. F. (2005). Standards to practice. Journal of Staff Development, 26 (1), pg 48.

Eysenck, M. W. (2001). Principles of cognitive psychology (2^nd ed.). New York, NY: Psychology Press.

Knowles, M. (1984). The Adult Learner: A Neglected Species (3rd Ed.). Houston, TX: Gulf Publishing. Retrieved March 3, 2005, from: http://tip.psychology.org/knowles.html

Peer Monitoring of TRIO Programs (2005). Peer Monitoring of TRIO Programs Helps

Find Ways to Save Money; Ensures Compliance with Federal Regulations.

Northwest Association of Special Programs. Retrieved, March 3, 2005, from: http://users.moscow.com/mareese/bestpractices/peer_review.html

Pushing for Peer-Review (2005). School Administrators Article. Education World, Retrieved March 3, 2005, from http://www.education world.com/a_admin/admin047.shtml

Russ, G. J. & Klug, M.G. (1999). Attitudes of business college faculty and administrators toward distance education: A national survey, Distance Education, 20 (1) 109-129.

Schacter, D. L. (1996). Searching for memory: The brain, the mind, and the past. New York, NY: Basic Books.

Sennett, R. (2003). Respect in a world of inequality. New York, NY: W. W. Norton.

University of Phoenix Online (2005). Faculty Development Workshops.

U.S. Department of Education, National Center for Education Statistics. (1999). Distance education at postsecondary education institutions: 1997-98. NCES 2000-13. Washington, DC: Author.

Wolf, W. J. (2003). Student Peer Reviews in an Upper-Division Mathematics Class, Exchanges, The Online Journal of Teaching and Learning in the CSU, September. Retrieved March 3, 2005, from http://www.exchangesjournal.org/classroom/1156_Wolfe.html
 

About the Authors