Editor’s Note: This is an interesting, well-researched study. Hopefully, well-designed commercial games will be more widely accepted and used in academia. Formal collaboration between video game designers and academia should be encouraged. Educators should research, validate, and actively implement video games for learning. There have been many attempts to do this, including the Ngee Ann Polytechnic in Singapore in the 1990s and the University of Southern California in this millennium.

Survey of Commercial off-the-shelf Video Games:
 Benefits and Barriers in Formal Educational Settings

Albert D. Ritzhaupt, Erin Gunter, Greg Jones

USA

Abstract

This paper examines the use of commercial-off-the-shelf (COTS) games in formal educational settings. A survey method was employed to determine which COTS video games are appropriated for formal educational settings, and the benefits and barriers to integration in formal educational settings. Ninety-nine members of the Special Interest Group of Games and Simulations (SIGGS) in the International Society for Technology in Education (ISTE) responded to the survey. The results suggest that greatest benefits to integrating COTS videos games include developing cognitive skills, teaching complex problem-solving, accepting and learning from mistakes, and learning by doing. The greatest barriers include teachers’ background in games and technology, perception of games, lack of financial resources, alignment with the curriculum, lack of evidence to support their use, and lack of time to integrate into the classroom. Conclusions are provided.

Keywords: Commercial-off-the-shelf, formal educational settings, benefits, barriers.

Introduction

In recent years, attention has been placed on the utility of commercial-off-the-shelf (COTS) video games in formal and informal educational settings (Becker, & Jocaobsen, 2005; Becker, 2006; Sanford, Ulicsak, Facer, & Rudd, 2006; Ertzberger, 2008; Charsky & Mims, 2008), particularly for military training applications (Barlow, Morrison, & Easton, 2002; Morrison & Barlow, 2004; Fong, 2006). COTS video games can be purchased from brick-and-mortar businesses, online, and via catalogs. COTS video games, unlike serious games (Michael & Chen, 2005), are not designed for the sole purpose of training, educating or informing, but rather for the primary goal of entertainment. Some COTS video games are deemed appropriate for formal educational settings (Sanford, Ulicsak, Facer, & Rudd, 2006). Charsky and Mims (2008) note “creating game-based learning environments or experiences using COTS games is becoming an increasingly tenable, valuable, and popular instructional strategy” (p. 38). COTS video games for formal educational settings should exhibit broad appropriateness for school settings, engage players with authentic content and challenges, and show the presence of clear causal relationships between game variables (Sanford, Ulicsak, Facer, & Rudd, 2006).

However, as discussed by Becker (2006), there remains resistance to the use of COTS games in formal educational settings. Squire (2003) suggests that integrating games into formal educational settings potentially raises as many problems as it solves. COTS video games may not appeal to every student equally (Ke, 2008) and students may be distracted by game-playing, and consequently, not achieve the educational goals (Miller, Lehman, & Koedinger, 1999). Egenfeldt-Nielsen (2004) states there are numerous barriers to successfully integrating games; including scheduling, setting, class expectations, teacher background, genre knowledge, technical problems, teacher preparation, perception of games, class size, and priority issues – all of which place a tremendous burden on teachers. Thus, the use of COTS video games in formal educational setting remains a topic of debate among researchers and educators alike.

Commercial-Off-The-Shelf Games in Education

Becker and Jacobsen (2005) surveyed 109 educators about the utility of computer and video games in a classroom setting. Fifty-seven percent of their respondents had used computer or video games in their classroom practice. More than half of the educators indicated that the teachers themselves, either through their own efforts, through professional development opportunities, or through help from their students or nearby colleagues provided the greatest amount of help in integrating computer or video games into their classroom practice. When asked about barriers, more than 70% of the respondents indicated that there was not enough time to integrate games, not enough classroom computers, not enough games within the schools, or that integrating games was not a school priority.

Ertzberger (2008) surveyed 390 educators from 110 different schools about the utility of video games as instructional tools. His results suggest the most important factors in promoting the use of video games are the games alignment to curriculum, reliability of the technology, and hands on training available. Results indicate the biggest deterrents to the use of video games were the expense of video games, lack of time to create video games, lack of relevance to curriculum, and lack of the needed technology. Participants boasted the ability to individualize instruction, and increase motivation as benefits to video game use.

Future Lab, a non-profit organization for innovation in teaching, conducted a survey of primary and secondary teachers’ (N = 1,000) attitudes towards games in education (Future Lab, 2005). Their survey was not specifically about COTS video games. Their findings suggest that 72% of the teachers do not play video games in their leisure time and 69% have not used games in their classrooms for educational purposes. Though 59% of the teachers would consider using games in their classroom, 49% of same population see the largest integration barrier as access to equipment (Future Lab, 2005).

Kerbitchi (2009) examined factors affecting teachers’ adoption of computer games using a case study method. The results were compared with the existing literature on the adoption of educational software, and the barriers in the use of educational computer games in formal K-12 settings. The findings showed that adoption attributes for the games and other educational software had a similar pattern from high to low significance on relative advantage, compatibility, complexity, trialability and observability (Rogers, 2003) and that the game adoption factors were more inclusive than the barriers of using the computer games.

While these studies have certainly informed educational research, they leave much to be desired. Not one of the aforementioned studies specifically examined the role of COTS video games in formal educational settings. Further, in another study, the results confirmed that the majority of the educators were not gamers (72%) and had not integrated games into their classrooms (69%). These findings are alarming and confirm there is a dearth of existing research examining role of COTS video games in formal educational settings.

Research Questions

Therefore, this research sought to address the following research questions:

1.  What commercial-off-the-shelf video games do professionals deem appropriate for formal educational settings?

2.  What are the benefits of integrating commercial-off-the-shelf games in formal educational settings?

3.  What are the barriers to integrating commercial-off-the-shelf games in formal educational settings?

Method

Participants

Participants included members of the Special Interest Group of Games and Simulations (SIGGS) in the International Society for Technology in Education (ISTE). One-hundred fifty-four participants opened the survey, and of those participants, N=137 provided a response to at least one item. During this period, the SIGGS had approximately 1,400 members, which is a response rate of approximately 10%. Those participants that did not complete at least 75% of the survey items were removed from the analysis resulting in a final set of n=99 participants.

As can be gleaned in Table 1, approximately 68% of the participants were female. The majority of the participants are in the age range of 36 to 65 years old. More than 80% of the participants reported salary ranges from $30,000 to $100,000 per year. More than 80% of the participants reported a White/Caucasian ethnicity. One hundred percent of the sample held at least a bachelors degree with approximately 49% holding a masters degree.

Table1

Demographic characteristics of sample.

Ninety-six percent of the participants were from the United States with the remaining from Canada. Approximately 68% of the participants reported playing computer games and 51% reported playing video games. The job titles of the participants represented a wide range of careers from teachers to college professors to educational technologists. Participants reported an average of 15.36 years (SD = 9.35) in educational technology-related professions.

Instrument

The instrument was divided into four sections: background information, COTS, COTS barriers, and COTS benefits. The background information section included items pertaining to age, gender, ethnicity, salary range, educational attainment, game play information, and job title information. The COTS section included 19 popular COTS games for multiple platforms derived from multiple sources (Kirriemuir, 2005; Prensky, 2010) and with a response scale of Very Appropriate, Appropriate, Neutral, Inappropriate, and Very Inappropriate.

The COTS games barriers section included 23 potential barriers to the integration of COTS games in formal educational settings with a response scale of Very Significant Barrier, Significant Barrier, Neutral, Insignificant Barrier, and Very Insignificant Barrier. Finally, the COTS games benefits section included 18 potential benefits of integrating COTS games into a formal educational setting with a response scale of Very Significant Benefit, Significant Benefit, Neutral, Insignificant Benefit, and Very Insignificant Benefit. Both the COTS games benefits and barriers scales were derived from previous research (Becker, & Jocaobsen, 2005; Ertzberger, 2008). All scales also included a Do not Know option.

Procedures

The survey was made accessible in a web-based format using SelectSurvey. The survey was available during a three-week period, and during this time, two reminder emails were sent to members of the SIGGS. Participants were informed the purpose of this project was to examine characteristics related to the use of COTS games in formal educational settings.

Data Analysis

This study employed a mixed-method approach, (Tashakkori & Teddlie, 1998), involving both quantitative and qualitative procedures used concurrently and independently. Quantitative analyses of the data included descriptive analysis of response frequencies and measures of variation and central tendency, (Knupfer & McLellan, 1996), and internal consistency reliability analysis, (Cronbach’s alpha). Three open-ended survey items were included in the instrument for the purpose of gaining further insight into COTS games, benefits and barriers. The data were analyzed using a constant comparative method (Glaser, 1965; Glaser, 1967).

Results

The results of this research must be viewed in light of its limitations. The sample had a low response rate. Prior research suggests that average online response rates fall somewhere within the range of 24% – 39% (Cook et al., 2000, Sheehan, 2001). Our research achieved 10% response rate. This low response may not accurately represent the perceptions of the population. Additionally, the survey itself is a limitation. The questions were few and the content of a limited nature. Further research is necessary to validate an instrument for wide scale use. In light of these limitations, this research has resulted in several interesting findings.

Commercial-off-the-shelf (COTS) Video Games

The internal consistency reliability for the scale was high at α = .89. As can be seen in Table 2, many of the participants were not familiar with a handful of the COTS games listed in the survey. Big Brain Academy, Brain Age, and Oregon Trail had item averages at or above 4.0, indicating that those that were aware of the games, deemed them appropriate for a formal educational setting. Other games had high averages (>> 3.5), including Civilization III, Dimension M, Railroad Tycoon, Rise of Nations, The Sims, Wii Sports, and Zoo Tycoon. The only game identified as not appropriate was the Grand Theft Auto series with an item average of 1.83 (SD = 1.22).

Table 2

COTS games scale with descriptive statistics.

n=Number of respondents, DK = Do not Know, VI = Very Inappropriate, I = Inappropriate, N = Neutral, A = Appropriate, VA = Very Appropriate, M = Item average, SD = Item Standard deviation, *Note the average and standard deviations do not include the Do not Know response.

Participants also had the option of listing COTS games that were not listed in the scale. These responses are summarized in Table 3. Only those with two or more responses are shown. In total, 22 unique games were provided. Of the 99 participants, only 23 provided a response with an average of 5.13 words (SD = 4.74). Of those listed, both Spore and Risk occurred most.

Table

Benefits to COTS Video Games

The internal consistency reliability for the scale was also high at α = .93. Table 4 shows the results. The highest rated item averages (>> 4.0) for the support of COTS in formal educational settings are to develop cognitive skills (M = 4.31, SD = 0.64), develop spatial relation skills (M = 4.14, SD = 0.72), develop motor skills (M = 4.12, SD = 0.73), knowledge acquisition (M = 4.18, SD = 0.76), increase memory capacity (M = 4.02, SD = 0.85), teaching complex problem-solving (M = 4.45, SD = 0.66), increase creativity (M = 4.15, SD = 0.91), transfer knowledge to real-world situations (M = 4.17, SD = 0.77), accept and learn from mistakes (M = 4.28, SD = 0.75), learning by doing (M = 4.26, SD = 0.70), increase self-esteem and self-confidence (M = 4.00, SD = 0.88), incorporate technology student’s use every day (M = 4.18, SD = 0.83), and to promote differentiated instruction (M = 4.17, SD = 0.81). Notably, none of the potential benefits listed are below the central point (3.0).

Table 4

Benefits to COTS video games.

n=Number of respondents, DK = Do not Know, VIB = Very Insignificant Benefit, IB = Insignificant Benefit, N = Neutral, SB = Significant Benefit, VIB = Very Significant Benefit, M = Item average, SD = Item Standard deviation, *Note the average and standard deviations do not include the Do not Know response.

Participants also had the option of listing potential benefits not listed in the scale. Eighteen participants provided free-form responses averaging 11.05 (SD=9.94) words in length. Results suggest, in addition to those listed, increased communication among students and teachers, increased teamwork and collaboration among students, stealth assessment, and students learning the design of games themselves are additional potential benefits.

Barriers to COTS Video Games

The Cronbach’s alpha was high at α = .92. Table 5 shows the results. The greatest barriers (4 >>) to the integration of COTS games include teachers’ background in games and technology (M = 4.33; SD = 0.86), perception of games (M = 4.23; SD = 0.85), lack of financial resources (M = 4.13; SD = 1.01), alignment with the curriculum (M = 4.09; SD = 1.00), lack of evidence to support their use (M = 4.04; SD = 0.89), and lack of time to integrate into the classroom (M = 4.23; SD = 0.85).

Participants also had the option of listing potential barriers not listed in the scale. Twelve participants provided responses with an average length of 16.08 (SD=19.28) words. Additional barriers to integrating COTS video games include gender issues, equity issues for students with special needs (e.g., visually impaired), lack of games aligned with standardized tests, unhealthy competition, and Internet safety.

Discussion

One respondent captured the essence of perhaps the most significant barrier to using COTS video games in formal settings by stating “I must teach to these academic standards; my principal will give me a bad evaluation if I am not teaching the standards and we are playing games instead” (Respondent, 2009). This statement reiterates the ongoing challenge educators face in attempting to pioneer innovative instructional approaches employing technology. The perception of COTS video games is itself a major barrier as confirmed by our findings. Educators must make the case to leadership of the value of integrating COTS video games in the classroom. This inevitably requires more evidence to substantiate the use of COTS video games. This lack of documentation has been identified as another significant barrier. We believe these findings, at minimum, make a call to educators and researchers alike to spearhead this complicated domain.

Table 5

Barriers to COTS video games.

n=Number of respondents, DK = Do not Know, VIB = Very Insignificant Barrier, IB = Insignificant Barrier, N = Neutral, SB = Significant Barrier, VIB = Very Significant Barrier, M = Item average, SD = Item Standard deviation, *Note the average and standard deviations do not include the Do not Know response.

The results of our research corroborate the findings of prior research. For instance, lack of time to integrate into the curriculum was perceived as a major barrier in Becker and Jocaobsen findings and in our own results. Additionally, Ertzberger (2008) found that lack of alignment to the curriculum and lack of financial resources were also significant barriers. Our findings provide further evidence of these barriers in integrating video games in formal educational settings. In addition, our research has identified a teachers’ background in games and technology, perception of games in general, and lack of evidence to support their use as significant barriers.

In exploring potential benefits, the greatest benefits identified include developing cognitive skills, teaching complex problem-solving, accepting and learning from mistakes, and learning by doing. Additionally, qualitative findings suggested increased communication among students and teachers, increased teamwork and collaboration among students, stealth assessment, and students learning the design of games themselves are additional potential benefits. These findings are analogous to prior research suggesting video games are a tool to “use action instead of explanation, create personal motivation and satisfaction, provide interactive decision-making context” (Kebritchi, 2008, p. 15), and can promote collaboration among learners (Kaptelin & Cole, 2002). While this research has explored the many perceived benefits of COTS video games, further research is necessary to examine the efficacy of these benefits in experimental and quasi-experimental settings. Put simply, more research is necessary.

Our research has also identified COTS video games that are deemed appropriate for formal educational settings. These games include Big Brain Academy, Brain Age, Oregon Trail Civilization III, DimensionM, Railroad Tycoon, Rise of Nations, The Sims, Wii Sports, and Zoo Tycoon. These games are a list of tools that educators can choose to employ in their classrooms. Further, the list also provides a list of appropriate games to investigate from a research perspective as more evidence to substantiate their use is indispensable.

We believe that our findings should be used to create a larger validated instrument that can be used on a wider audience of educational professionals on the benefits and barriers to integrating COTS video games. This paper has documented a dearth of literature on the use of COTS video games in educational settings. Our instrumentation is a starting place for researchers to document and validate a tool to collect this pertinent information from education professionals.

As noted by Charsky and Mims (2008), “COTS game are not a panacea, just a strategy with which we, and others, have had success; a strategy that may be successful for some educators, especially those educators who are already playing video games outside of work.” COTS video games should be perceived as one tool among many different instructional strategies - a tool that has the potential to truly engage our students.

References

Barlow, M., Morrison, P. & Easton, A. (2002). 1st-Person tactical shooters: COTS games with military training potential? In Proceedings of SimTecT.

Becker, K. (2006). Pedagogy in commercial video games. In Gibson, D., Aldrich, C., & Prensky, M. (Eds.). Games and simulations in online learning: Research and development frameworks (). Hershey, PA: Information Science Publishing.

Becker, K. & Jocaobsen, D. M. (2005). Games for learning: Are schools ready for what’s to come? Proceedings of DiGRA 2005 Conference: Changing Views – Worlds in Play.

Charsky, D. & Mims, C. (2008). Integrating commercial-off-the-shelf video games into school curriculums. TechTrends, 52(5), 38 – 44.

Cook, C., Heath, F. & Thompson, R. L. (2000). A meta-analysis of response rates in web- or Internet-based surveys, Educational and Psychological Measurement, 60, 6, 821-836.

Ertzberger, J. (2008). An investigation into factors affecting teachers’ use of video games as instructional tools, (Doctoral dissertation, Pepperdine University).

Fong, G. (2006). Adapting COTS games for military experimentation. Simulation & Gaming, 37(4), 452-465.

Future Lab (2005). Teaching with games: Survey on teachers' attitudes to games and learning. Retrieved on July 11, 2009 from: http://www.futurelab.org.uk/resources/documents/project_reports/teaching_with_games/
TWG_teachers_survey.pdf

Glaser, B. G. (1965). The constant comparative method of qualitative analysis. Social Problems, 12(4), 436-445.

Glaser, B. G. (1967). The constant comparative method of qualitative analysis. In B. Glaser & A. Strauss (Eds.), The discovery of grounded theory: Strategies for qualitative research
(pp. 101- 116). Chicago: Aldine.

Kaptelin, V., & Cole, M. (2002). Individual and collective activities in educational computer game playing. In T. Kosmann, R. Hall, & N. Miyake (Eds.), g2057CSCL 2: Carrying forward the conversation (pp. 303–316). Mahwah, NJ: Lawrence Erlbaum.

Ke, F. (2008). A case study of computer gaming for math: Engaged learning from gameplay? Computers & Education, 51(4), Pages 1609-1620.

Kebritchi, M. (2009). Factors affecting teachers’ adoption of educational computer games: A case study. British Journal of Educational Technology.

Kebritchi, M. (2008). Effects of a computer game on mathematics achievement and class motivation: An experimental study. Unpublished doctoral dissertation, University of Central Florida.

Kirriemuir, J. (2005). Commercial games in the classroom. Interact, 31, 20-21.

Knupfer, N. N., & McLellan, H. (1996). Descriptive research methodologies. In D. H. Jonassen (Ed.). Handbook of research for educational communications and technology (pp. 693-719). New York: Macmillan. Michael, D. & Chen, S. (2005). Serious Games: Games that educate, train, and inform. Thomson: Course Technology.

Miller, C. S., Lehman, J. F., & Koedinger, K. R. (1999). Goals and learning in microworlds. Cognitive Science, 23(3), 305–336.

Morrison, P. & Barlow, M. (2004). Child's Play? Coercing a COTS Game into a Military Experimentation Tool. In SimTecT 2004 Conference Proceedings, 72-79.

Niederhauser , D. S., & Stoddart, T. (2001). Teachers' instructional perspectives and use of educational software. Teaching and Teacher Education, 17(1), 15 – 31.

Prensky, M. (2010). Video games. Retrieved on April 23, 2010 from http://www.marcprensky.com/.

Sanford, R., Ulicsak, M., Facer, K., & Rudd, T. (2006). Teaching with Games: Using commercial off-the-shelf computer games in formal education. Entertainment Arts: FutureLab.

Sheehan, K. (2001). E-mail survey response rates: A review. Journal of Computer-Mediated Communication, 6(2), Available at: http://jcmc.indiana.edu/vol6/issue2/sheehan.html.

Squire, K. D. (2003). Gameplay in context: Learning through participation in communities of civilization III players. Unpublished PhD dissertation. Instructional Systems Technology Department, Indiana University.

Tashakkori, A. & Teddlie, C. (1998). Mixed Methodology: Combining Qualitative and Quantitative Approaches. Thousand Oaks, California: Sage.

About the Authors

Albert D. Ritzhaupt, School of Teaching and Learning, University of Florida.

aritzhaupt@gmail.com

Erin Gunter, Watson School of Education, University of North Carolina, Wilmington.

gunterin@gmail.com

Greg Jones, College of Information, University of North Texas.

gjones@unt.edu