Mar05_02

Editor’s Note: This research combines two instruments to detect aspects of critical thinking in online interactions. It explores theoretical and practical constructs for mentoring discussions, analysis of critical thinking processes, and interpreting the findings. By codifying each sentence in an online discussion, it is possible to generate statistics and descriptors for the critical thinking process – trigger, exploration, integration, and resolution. These findings have value for design and implementation of online learning
and mentoring.

Two Methods for Assessing Critical Thinking in Computer-Mediated Communications (CMC) Transcripts

Patrick J. Fahy

Abstract

Critical thinking, though critical in education, is especially difficult to detect in online learning and teaching based on computer-mediated communication (CMC). As a latent construct, critical thinking must be inferred by analysis of the “traces” of higher-level cognitive activity found in transcripts. Two models are presented for describing and analyzing critical thinking, the practical inquiry (PI) model (Garrison, Anderson, & Archer, 2001), and the Transcript Analysis Tool (TAT) (Fahy, Crawford, & Ally, 2001). The models reveal different aspects of online interaction: the PI model determines the proportions of four phases found in transcripts of the critical thinking process, while the TAT adds detail, from the sentence level, about communication strategies and patterns within postings. Principal findings and suggestions for further research focus on triggers and postings classified as other in the PI model.

Keywords: CMC; online interaction analysis; transcript analysis; community of inquiry; cognitive presence; practical inquiry model; Transcript Analysis Tool (TAT); network interaction; social presence

Background

Critical or higher-order thinking has consistently been cited as a prime objective of all types of education, including education at a distance (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956; Gibson, 1996; Bostock, 1997; Romiszowski, 1997; Haughey & Anderson, 1998; Marttunen, 1998; Collison, Elbaum, Haavind, & Tinker, 2000; Strother, 2002; Roblyer & Schwier, 2003). In pursuit of a better understanding of this critical construct, the Canadian research group of Garrison, Anderson, Rourke, and Archer have articulated, in three important papers, a conceptual framework for the context within which they argue critical thinking is likely to be found, a community of inquiry (Rourke, Anderson, Garrison, & Archer, 1999; Anderson, Rourke, Garrison, & Archer, 2001; Garrison, Anderson, & Archer, 2001). In their paper of interest here (Garrison, et al., 2001), they stated that such a community, engaging in critical thinking, is an “extremely valuable, if not essential” element of higher education. Their work focuses on the importance, to communities of inquiry, of computer-mediated communications (CMC) as a means for creating and sustaining cognitive presence, and as a vehicle for engaging in critical thinking (Gunawardena, Lowe, & Anderson, 1997; Fahy, 2001).

The efforts of Garrison et al. (2001) are important, as they directly address a fundamental problem encountered by all attempts to detect or assess latent constructs such as cognitive presence and critical thinking in online contexts (Rourke & Anderson, 2004). Because they are only indirectly knowable, latent constructs must be known from their “traces,” “symptoms,” or “indicators” (2001, p. 12, 13). This process, as the authors admit, is inherently subjective, inductive, and prone to error (p. 12) – as is, one might add, the associated research.

This paper was prompted by two objectives. The first was a wish to apply further, and perhaps refine, Garrison et al.’s (2001) practical inquiry (PI) model (the term for the operationalized procedure for using this construct, grounded in the critical thinking literature; p. 8). Garrison et al. conducted an admittedly weak initial pilot application of their model, commenting that it should “not be seen as immutable” (p. 9), and concluding their paper with the comment, “this tool is worth further investigation” (p. 22). A review of the literature since its appearance reveals that to date the model has not received the further testing the authors hoped and expected it would, a situation, it is hoped, this paper will in part redress.

A second reason for this paper was to present a comparison of Garrison et al.’s method for the detection and assessment of latent projective variables (including, but not limited to, cognitive presence; Fahy, 2003) with that of another model, the Transcript Analysis Tool (TAT) (Fahy, Crawford, & Ally, 2001). The comparison is guided by Thorngate’s (1976) postulate of commensurate complexity (cited in Weick, 1979, p. 35ff.): “It is impossible for a theory of social behavior to be simultaneously general, accurate, and simple” (p. 35). The TAT model requires each sentence be considered (coded). In comparison to the PI model, the TAT strives for accuracy and generalizability, at the expense of the reliability that greater simplicity would confer. In presenting the PI model, Garrison et al. wrote of their intention to avoid some of the complexities of other approaches, for example by restricting the number of phases (and therefore the number of coding distinctions) to four, a relatively low number in transcript analysis studies (Fahy, 2001; Rourke, Anderson, Garrison, & Archer, 2001); by coding whole postings rather than component parts; and by developing a “heuristic” for dealing with “contradictory categorization cues or evidence of multiple phases” (p. 17), that required “coding up” or “coding down” (to resolve situations where coding was ambiguous). While their approach produced good reliability figures, the authors recognized the implications of their approach in relation to Thorngate’s third criterion, accuracy: “Submessage level units [i.e., sentences] may be introduced in future confirmatory studies if increased precision is warranted” (p. 17).

The importance of critical thinking as a component and outcome of online interaction has piqued previous research interest. Various attempts have been made to operationalize critical thinking in order to see it more clearly in online group behaviour, occasionally (as will be seen in the following list) simply by attributing it to certain activities or to specific strategies or technologies (Simon & Berstein, 1985). Examples of de facto perception of critical thinking to activities and tools include: questioning and challenging (Blanchette, 2001); constructivist dialogue in case-based learning contexts (Commonwealth of Learning, 1993; Jonassen, Davidson, Collins, Campbell, & Bannan Haag, 1995; Jonassen, 1998); collaborations of various forms and under different conditions (Bullen, 1998; Curtis & Lawson, 2001; Rose, 2004); group focus and reflection on transcript contents (Davie & Wells, 1992); uses of various media (Dede, 1996; Mayer, 2001); and approaches to group-mediated strategic thinking (Gunawardena, et al., 1997).

The above shows how widely researchers have ranged to find evidence of cognition in CMC. In this paper, the two methods used to detect critical thinking in an online community focus on “the nature and quality of critical discourse” found in the transcript itself. While their methods differ, the purpose of both methods is to identify elements of postings that create identifiable and (more or less) predictable responses from others in the online community. Both models address the organization of online interaction, “a systematic account of some rules and conventions by which sets of interlocked behaviours are assembled to form social processes” (Weick, 1979, p. 3). They focus on different elements of the transcript (postings in the Garrison et al. model, sentences in the TAT), but the intention in both cases is a “systematic account” of some aspect of communication in the online community, derived directly from transcript data. Their use together here is exploratory; consequently, parts of the analysis assume that exploratory studies may claim some latitude in interpretation in the interests of fairly testing their potential value (Rourke, et al., 1999).

Two models for detecting critical thinking in online interaction

The Practical Inquiry model. Garrison et al. (2001) operationalized critical thinking through a model of practical inquiry, recognizing such thinking to be both a process and an outcome of online communities engaged in reflective critical discourse (p. 7, 8). In critical communities of inquiry, they reasoned, participants apply reflection and action to facts and ideas, often (especially in educational environments) under the direction of a moderator or instructor. (This concept is similar to that of McKlin, Harmon, Evans, & Jones [2002, p. 2], who linked “sustained reflection and discourse” to cognitive activity.)

The phases of this model of critical thinking are as follows (Garrison et al., 2001, pp. 10 – 11):

The PI model was initially tested by its developers on a corpus of 95 postings, small enough to be called by the authors a “methodological weakness” (Garrison, et al., 2001, p. 18). The researchers coded messages from the transcript using the four original categories (soon adding a fifth category, other, to accommodate messages not fitting elsewhere). When (perhaps not unexpectedly; see Fahy, 2001; Rourke, et al., 2001; Puustjärvi, 2004) difficulties were encountered with classification of whole postings into single categories, further clarification in the form of “descriptors” and the “perspective” of the participants was added to assist coding. The phases (codes), with the adjunctive “descriptors” and “perspectives,” are shown below.

Table 1

Phases, descriptors, and perspectives of the
Practical Inquiry model

Phase	Descriptor	Perspective
Triggers	Evocative	Shared world
Exploration	Inquisitive	Private world
Integration	Tentative	Reflection
Resolution	Committed	Discourse
Other	(Postings not fitting another category)

The validity of the process appeared promising in the initial application: codings in the pilot test of the model yielded coefficients of reliability (CR, a ratio of agreement to total number of judgments made by raters) ranging from 0.45 to 0.84, and kappa values from 0.35 to 0.74 (Garrison, et al., 2001, p. 18). (Cohen’s kappa is a chance corrected measure of agreement [University of Colorado, 1999; Agreement observer, 2000], especially useful where the number of coding decisions is limited, thus making chance a potentially important factor in the classification process).

The results of the initial pilot analysis (see Table 2, in “Findings,” below) showed that most of the postings (42%) were exploration, and that the next most common category (consisting of eight postings, or one-third of the total) was other, postings that could not be classified in any of the other four phases (p. 19). In the pilot test of the model, the authors wrote that their intention in offering the PI model was to suggest an approach that might be useful in facilitating the process of higher-order online learning (Garrison et al., 2001, p. 8), and that the model’s phases reflected an “idealized” critical inquiry process which “must not be seen as immutable” (p. 9), words encouraging to the present study.

The TAT model. Another approach to understanding the content and social processes in online interaction, including thinking processes, is the TAT (Transcript Analysis Tool). The TAT, based on a concept originated by Zhu (1996), has been applied during its development to a variety of CMC-based interaction analysis problems (Fahy, et al., 2001; Fahy, 2002a; Fahy, 2002b; Fahy, 2003; Fahy, 2004; Fahy & Ally, in press). Application of the TAT involves coding each sentence of a transcript into one of 8 categories (five major): 1) questions (horizontal or vertical), 2) statements (referential or non-referential), 3) reflections, 4) scaffolding comments, or
5) paraphrases and citations.

Type 1 - Questions:

1A includes vertical questions, which assume a “correct” answer exists, and the question can be answered if the right (knowledgeable) individual is asked, or the right source contacted.

1B are horizontal questions: recognizes there may not be one right answer; others are invited to help provide a plausible or alternate “answer” or explanation, or to help shed light on the question.

Type 2 - Statements:

2A (non-referential statements) contain little self-revelation and usually do not invite response or dialogue; the main intent is to impart facts or information. The speaker may take a matter-of-fact, didactic, or pedantic stance, providing information or correction to an audience assumed to be uninformed or in error, but curious, interested, and otherwise open to correction. Statements may contain implicit values or beliefs, but usually these must be inferred, and are not as explicit as they are in reflections (TAT type 3).

2B (referential statements) comprise direct answers to questions, or comments making reference to specific preceding statements.

Type 3 - Reflections (significant personal revelations)

Type 3 sentences show the speaker expressing thoughts, judgments, opinions, or information which are personal and are usually guarded or private. The speaker may also reveal personal values, beliefs, doubts, convictions, or ideas acknowledged as personal. The listener/reader receives both information about some aspect of the world (in the form of opinions), and insights into the speaker. Listeners are assumed to be interested in and empathetic toward these personal revelations, and are expected to respond with understanding and acceptance. The speaker implicitly welcomes questions (even personal ones), as well as self-revelations in turn, and other supportive responses.

Type 4 - Scaffolding/engaging

Scaffolding/engaging sentences are intended to initiate, continue, or acknowledge interpersonal interaction, to “warm” and personalize the discussion by greeting or welcoming, and to support and maintain the online network by enhancing inclusiveness. Scaffolding/engaging comments connect or agree with, thank, or recognize someone else, and encourage or acknowledge the helpfulness, ideas and comments, capabilities, and experience of others. Also included are comments without real substantive meaning (“phatic communion,” “elevator/weather talk,” salutations/greetings, and closings/signatures), and devices such as obvious rhetorical questions and emoticons, whose main purpose is maintenance of the interpersonal health of the online community.

Type 5 - Quotations/citations:

5A: quotations or paraphrases of others’ words or ideas, including print and non-print sources.

5B: citations or attributions of quotations or paraphrases, in a formal or reasonable complete informal manner.

The TAT uses sentences; each sentence in the transcript is assigned to one (or more) TAT categories (about 6% of sentences in this transcript received more than one TAT code, a typical proportion). Unitizing, the process of selecting elements of the transcript to code, has sometimes proven problematic (Rourke, et al., 2001; Fahy, 2001). While the debate has not been resolved, problems have been identified with units greater than the sentence, such as “units of meaning” (Henri, 1992), “segments” (Borg & Gall, 1989, cited in Garrison et al., 2001), “thematic units” (Rourke, et al., 1999), or “phases” (Gunawardena, et al., 1997). Although Garrison et al. coded their transcript at the level of the posting, for reasons of consistency and due to concern for validity, they acknowledged (2001, p. 17), as noted earlier, the advantages of sentence-level analysis for revealing more accurately subtle nuances in the transcript (Fahy, 2001, 2002a, 2002b).

Theoretical context for an analysis of critical thinking in CMC

Garrison and his colleagues posited in the PI model that critical thinking would involve a progression through four phases, beginning with a trigger, moving through exploration, to integration, and achieving final resolution. They reasoned that higher-order learning required questioning and challenging of assumptions, through the dual processes of engagement in internal reflection and community-based discourse (via CMC), resulting in further (re)constructing of experience and knowledge. Critical thinking, in this view, requires interaction with a community, drawing upon the resources of the community to test the content of individual contributions (the quality of ideas, the soundness of reasoning, the universality of experience, cogency of argument, eloquence, etc.).

In proposing four main phases for this process, the PI model presents a cyclical concept of thinking (resolution, the final phase, may reveal new dichotomies or discontinuities, producing a new triggering event); in general in this model, groups are assumed to be seeking resolution. While each phase of the model is accompanied by concurrent cognitive and social outcomes, the implication is clear that the overall process is incomplete if it stalls prior to completion of a full cycle ending with resolution (p. 9).

The initial pilot application of the PI model revealed little integration (Table 2), and even less resolution (Garrison, et al., 2001, p. 18). This finding may not be surprising, for theoretical reasons which others (including one of Garrison’s co-authors) have identified. Kanuka and Anderson (1998) examined a transcript generated in a moderated online forum (CMC conference), whose purpose was to support professional development among distance education professions. The researchers sought evidence of a five-phase knowledge-construction process, based on constructivist theory:

In fact, about 93% of the transcript postings (191 of 216; an “overwhelming number,” according to the authors [p. 65]) fell into the first category. This single phase, sharing/comparing of information, as defined in the study by the researchers, consisted of various preparatory activities, including several reminiscent of those found in the triggering phase of the PI model: stating observations or opinions, expressing agreement or support, identifying problems, defining, describing, corroborating, and clarifying questions. The other four phases, including especially those equivalent to what Garrison et al. termed integration and resolution, comprised as little as 3% of the transcript, depending upon the proportion deemed exploration (Kanuka & Anderson, 1998, p. 66). These results suggested that the analytic approach used in the study may not have discriminated adequately to permit real insights into the quality of the online interaction, a previously described problem in transcript analysis studies (Fahy, 2001; Rourke, et al., 2001).

Another example was reported by Gunawardena, et al. (1997). Using a similar analytic approach, they attempted to use the structure of a stringently moderated online debate to examine the social construction of knowledge in an international group of experienced distance education professionals. The authors held that knowledge results from interaction, stating emphatically: “Interaction is the process through which negotiation of meaning and co-creation of knowledge occurs" (p. 405). They assumed knowledge construction would occur in this group despite the debate structure, since the interaction was collaborative as opposed to one-way (p. 400 - 401).

Of particular interest in this study was the finding that participants obviously resisted the debate format, attempting to reach compromise and consensus despite the persistent efforts of the debate leaders “to keep the two sides apart” (p. 417). In effect, the researchers reported, the moderators’ attempts to base the discussion on discord ran counter to the group’s preference for synthesis. Even in a formal debate, these findings showed, the group’s propensity may be to avoid dwelling on differences, and to seek commonalities.

The work of Fulford and Zhang (1993) may partially explain these findings. Fulford and Zhang studied perceptions of interaction among teachers involved in professional development, by examining the interaction of the variables personal interaction, overall interaction, and satisfaction. The findings of interest were, first, that perceptions of personal and overall interaction were positively correlated (“people who see themselves as active participants tend to have a more positive perception of overall interaction” [p. 14]); second, that satisfaction was more attributable to perceived overall interactivity than to individual participation, leading to the conclusion that “learners who perceive interaction to be high will have more satisfaction with the instruction than will learners who perceive interaction to be low” (p. 18). An encouraging and intriguing finding for instructor/moderators was the observation that involving all students in direct instructor-student interaction might not be necessary to produce positive perceptions of overall group interactivity: “Vicarious interaction may result in greater learner satisfaction than would the divided attention necessary to ensure the overt engagement of each participant [by the instructor]” (p. 19).

The above suggests that in their cognitive behaviours online groups may have a disposition (a tropism, in biological terms) toward consensus, agreement, synthesis, and accord, and an aversion to discord, conflict, and argument. Rather than seeking a clash of viewpoints in CMC, participants apparently prefer to attempt to build solidarity. As Gunawardena, et al. noted, in group interactions “the situation itself exerts a strong mediation effect upon individual cognitive and conceptual processes” (p. 407), favouring sharing and concord. The relative lack of conflict in instructor-moderated academic interactions, especially in comparison with the Mardi Gras-like atmosphere often seen in unmoderated list-based discussions (Walther, 1996; Yates, 1997; Schrage, 2003), may be seen as further evidence of this preference (Garton, Haythornthwaite, & Wellman, 1997).

The finding of Garrison et al., (2001), Kanuka and Anderson (1998), and Gunawardena, et al. (1999), that online groups appear “comfortable remaining in a continuous exploration mode” (Garrison, et al., 2001, p. 10), requiring moderator intervention (or “teaching presence”; Anderson, et al., 2001) to move to more advanced stages of critical thinking, is one of several generalities following from these studies. Others include:

For individuals, the process of critical thinking involves both private reflection and public interaction, the latter within a community;

Efforts to observe interaction associated with critical thinking often produce results which do not discriminate well (a few interaction categories [codes] account for a large proportion of the observations), or expose weak or faulty instruments, or poor observational procedures;

CMC participants engaged in a process of critical thinking seem to prefer to share and compare, and to avoid conflicts, differences of opinion, or disagreements of interpretation;

The tendency to avoid overt disagreement and discord may be based on a group preference for a climate where the quality of general social interaction is more important to satisfaction than opportunities for personal interaction (a climate that is more epistolary than expository) (Fahy, 2002a).

This present study was designed to explore the behaviour of an online community engaged in critical thinking, as reflected in the transcript of its online CMC interactions, by the application of two different but similarly purposed analytic models. The portion of the total intra-group interaction that occurred is not known, as students had the option of communicating by other means not assessed in the study (e-mail, telephone, even face-to-face meetings). The assumption here, as in similar studies, was that the transcript would contain evidence – “traces” (Garrison et al., 2001, p. 12) – showing how the community of inquiry was functioning as a unit in relation to its sociocognitive purposes, and that these two tools would reveal important, but different, elements of that functioning.

Method

The study corpus used was a transcript of 462 postings, comprising 3,126 sentences containing approximately 54,000 words, generated by a group of thirteen students and an instructor/moderator, engaged in a 13-week distance education graduate credit course delivered totally at a distance. All of the students were experienced CMC users, and the instructor was an experienced distance educator who had used CMC to instruct graduate courses at a distance for over five years.

Each posting of the study transcript was coded into one of the PI model’s categories (trigger, exploration, integration, resolution); each sentence was also coded with the TAT (5.3% of the sentences received more than one TAT code). A code-recode method was used: the author did the initial coding of the transcript using both models, then recoded it again more than two months later. For the TAT, coefficient of reliability (CR) values ranging from of .70 to .94 have been reported (Keller, 1999; Fahy, Crawford, Ally, Cookson, Keller, & Prosser, 2000; Fahy, Crawford, & Ally, 2001; Poscente, 2003). In this case, the agreement level (CR) was 81% with the TAT (Fahy, et al., 2001).

For the PI model, the whole posting was coded into one of the model’s five categories. As noted above, the process of fitting whole postings into one code can be problematic: postings often contain multiple elements, and forcing a whole post into one category may ignore nuances or shadings of meaning. The PI model’s authors recognized this problem, recommending “coding down” to an earlier phase when it is not clear which phase is reflected, and “coding up” to a later phase when evidence of multiple phases was detected (Garrison et al., 2001, p. 17). (The frequency with which coding up or down was applied was not reported in the original paper.) In this study, coding up and down was applied as described when required, and an overall code-recode reliability of 86% was achieved with the PI model.

Coding for both models was accomplished with ATLAS.ti, and quantitative analyses were conducted with SPSS-PC and Excel.

Findings

Table 2 shows the results obtained from the application of the PI model to the study transcript, compared to the findings reported from the initial small pilot implementation of the model at the time of its initial appearance (Garrison, et al., 2001).

As shown in Table 2, while the proportions of postings in the categories of trigger, integration, and integration/resolution are remarkably similar in both studies, exploration was clearly affected by the large difference in the postings coded as other. In the original study, the process of coding three transcripts to refine the process produced interrater reliabilities from .45 to .84 (Garrison, et al., 2001, p. 18); the most frequent interrater disagreement during the refinement process reportedly occurred between the phases exploration and integration (p. 19). As well, during development and refinement of the model the category of other was added to the initial four phases; by the third transcript coding there was no reported disagreement among the coders in identifying postings placed in this category (p. 19).

Table 2

Practical Inquiry (PI) model results

Table 3 shows the occurrence of TAT categories, at the level of the sentence, within each of the five phases of the PI model.

Table 3

TAT Results

A comparison of Tables 2 and 3 shows some small discrepancies in the proportion of sentences (Table 3), compared with the frequency of the phases (Table 2): while triggers constituted over 9% of the phases, they comprised only 6.3% of the sentences; exploration tended to contain more sentences than its proportion of the phases (75.3% vs. 71.6%, respectively); integration was almost identically in proportion (14.1 of phases and 14.5% of sentences); resolution contained a higher proportion of sentences than its share of phases (2.4% vs. 1.7%); and other postings, while comprising 3.5% of the phases, constituted only 1.5% of the sentences. The pattern suggests that triggers, resolution, and other postings tended to be shorter (in numbers of sentences), while exploration and resolution postings tended to be lengthier. This finding is not surprising: one would expect that the processes of exploring and achieving resolution of issues would require more interaction (as seen in the number of sentences), while initiating the process, or comments orthogonal to the topic, would require less.

In order to provide a standardized method of assessing the proportions observed in Table 3, and to identify potentially salient findings for further investigation in this exploratory study, z (standard) scores were calculated. The z statistic shows the distance of the figure of interest (in this case, the percentages shown in Table 3, reflecting the proportion of TAT sentences within each phase) from the mean, in standard deviation units (Best, 1970). Table 4 shows the z scores for these percentages. (Cells of interest in relation to the following discussion are shown left-aligned and in bold in the following Table.)

Table 4

TAT results converted to Z scores

As can be seen, the phase with the greatest TAT variations was trigger postings, while the least variation was found in exploration postings. As described below, the phase other also contains some intriguing findings. The following summarizes the differences noted in the Table. (For this exploratory study, a z score of ±1.5 standard deviations is termed salient, while a difference of ±2.0 S.D. is considered significant).

Table5 summarizes the findings in relation to the TAT analysis, for significant and salient results.

Table 5

Summary of differences in TAT sentence types within PI phases (z ≥ 1.50)

PI Phase	Valence	TAT Category	Effect Size (z score)
Triggers	More:	Horizontal questions (1B)	2.06
		Citations (5B)	1.92
		Quotations and paraphrases (5A)	1.59
	Fewer:	Non-referential statements (2A)	-2.26
		Vertical questions (1A)	-1.71
Other	More:	Scaffolding/engaging (4)	2.26
	Fewer:	Reflections (3)	-1.89

Most triggers originated with the instructor/moderator, in accord with the predictions of Garrison et al. (2001): in the study transcript, 74% of the trigger postings were made by the instructor/moderator, 26% by students. This was the only phase where such a marked difference was noted, and conforms to the description of triggers in the PI model as a primary pedagogical responsibility of the instructor/moderator.

Four other findings in Table 3 are discussed here briefly, as suggestive in relation to the significant and salient findings reported earlier (the z scores associated with these differences were less than 1.5, but were in the same direction as the other findings, perhaps warranting further investigation (Riffe, Lacy, & Fico, 1998, in Rourke, et al., 1999, p. 66). In relation to triggers, two other TAT categories were also less common: referential statements (z = -1.47) and reflections (z = -1.19). Added to the previous significant and salient findings, these suggest triggers may also comprise more horizontal questions, quotations/paraphrases, and citations, and less of the other TAT categories, a finding similar to Poscente’s (2003).

Integration was also found to contain a somewhat lower proportion of non-referential statements (2A; z = -1.20), with a slightly elevated level of referential statements (2B; z = 0.82). These differences support a view of integration as a phase of interactive construction of meaning, involving assessing, connecting, and describing emerging understandings (Garrison, et al., 2001, p. 10), through both referential and non-referential statements.

Finally, resolution contained fewer vertical questions (1A; z = -1.10). As the phase in which consensus is built by vicarious or actual application of the knowledge developed in the other phases, this fact, and the presence of somewhat more referential statements and reflections (Table 4), are together not unexpected.

The above analysis permits the following summary of the nature of the online interaction observed here:

The frequencies of the PI model’s phases were similar to those noted in the original report, with the bulk of all postings constituting exploration, and triggers and integration/resolution comprising much smaller proportions of the interaction.

The contents of the category other in the PI model warrants further investigation, especially in regard to the apparently greater social and network orientation of this phase (revealed by the slightly higher proportion of scaffolding/engaging sentences).

The TAT analysis showed a tendency in exploration and resolution postings for more sentences, and in triggers for fewer. (The relation of posting length to type or contents remains unresolved, and in need of further study.)

On the basis of relative differences among TAT categories, revealed by z scores, triggers differed most from the other phases in terms of the TAT constituents, containing significantly more horizontal questions, quotations and paraphrases, and citations, and significantly fewer vertical questions and non-referential statements.

Discussion

The two different approaches to the analysis of the same study transcript revealed different aspects of the kind and quality of the online interaction that generated it. The PI model showed similar relative proportions of most of the phases as were found in an initial application, but the reduced occurrence of the phase other raises questions about the nature of this category, and about activities within the online community itself. The task of analysis was made more difficult by the fact that little information was provided regarding the type of postings which were classified other in the original work; the discrepancy found here could therefore be due to a lack of agreement about what other comprises (resulting in this study in the coding into one of the four principal phases material that was not coded that way by the authors of the original study), or it may reflect a genuine difference between this transcript and the one used by Garrison et al. (2001) in their initial paper.

Other comments are inherently difficult to classify, being defined by what they are not (one of the other four phases). A clue to the nature of these postings, and to a fundamental difference in the two analytic approaches, was the significantly higher occurrence in other postings of TAT scaffolding/engaging sentences, the type which addresses network maintenance and inclusiveness in the online community. These may indicate that the PI model does not provide for such factors within its four main phases. The fact that the TAT was able to identify the greater presence of the scaffolding/engaging sentence type suggests a difference, and perhaps an advantage, in relation to detection of specific kinds of interpersonal content in transcripts. These results are preliminary; further studies are clearly needed, carefully examining coding decisions relating to the other category. (Garrison et al. commented, “Content analysis is a difficult process under the best of circumstances” [2001, p. 18]; one suspects that grappling with complexities such as other content might have prompted that observation.)

Other findings at the level of the sentence seemed to confirm that the TAT and the PI model were both sensitive to similar processes within postings, and that these processes were consistent with their notional designations. This was especially evident in regard to triggers. In the PI model, triggers are sui generis, initiated by the instructor/moderator to focus group attention on a problem or phenomenon. In this study, the task of triggering the group was clearly one predominantly – though not exclusively – exercised by the teacher/moderator, and this pattern was detected equally well, although in different ways, by both tools.

Characteristics of integration and resolution postings were also revealed by the dual analysis. First, there was some evidence of reliability: the proportions of these two phases were found to be similar in both studies. Second, somewhat lower levels of non-referential statements and vertical questions were found in these phases, accompanied by more referential statements. These interactive processes, made apparent by the TAT analysis, may be the actual communicative strategies, or linguistic “moves” (Herring, 1996), by which critical thinking is conducted in communities of inquiry. If confirmed in future studies, this finding would constitute another insight gained through sentence-level analysis by the TAT.

Conclusion

In developing the practical inquiry model, Garrison et al. (2001) wrote that the fundamental problem was to see and assess thought processes “through the traces of the process that are made visible and public in the transcript” (p. 12). They went on to note that this process was “inevitably inductive and prone to error,” due to the subjective judgments necessarily involved. They also acknowledged that the transcript was itself an incomplete and imperfect record of the group’s interactions, and consequently of its learnings, since it lacked a record of all the other interactions engaged in by the participants. Perhaps in response to these perceptions, their analytic model appeared to prize simplicity and generalizability, at the expense of accuracy (by Thorngate’s principle of compensatory complexity; Thorngate, 1976, cited in Weick, 1979).

Despite problems with interaction analysis as a means of judging the qualities of online learning experiences, use of transcripts in this way remains one of the few methods available to study important social and cognitive aspects in online learning situations. Problems are greater when the focus is on latent projective variables like critical thinking, whose presence must be inferred from other indicators (Rourke, et al., 2001). In such studies, the more indicators incorporated in the analysis the more likely that accurate analytic judgments will be made, as more potentially causal factors are considered in the research process. (This process is termed overdetermination by Weick, 1979, p. 37). In this study, the use of the two models, with their different foci and processes, provided a high level of overdetermination, as shown both by the areas of consensus and by the unique contributions made by each.

This paper offers evidence that aspects of the PI model’s phases may be usefully elaborated at the level of the sentence by the TAT. In some cases, the greater detail provided by the TAT showed some of the concrete communications and interpersonal strategies (Witte, 1983) on which the phases of the PI model were based (especially in relation to the nature of triggers, and the interpersonal and network focus of postings coded other). It also appeared that the iterative nature of the PI model, and the conceptual interconnectedness of the model’s phases, provide a promising conceptual guide for researchers studying the “sociocognitive process” (Garrison et al., 2001, p. 13) of interaction through CMC. While questions and even equivocalities remain (Garrison, et al., 2001, p. 11), these are not signs of failure, but of the “dilemmas that face those who choose as their topic of interest phenomena that are complex, fluid, collective" (Weick, 1979, pp. 11 –12).

References

Anderson, T., Rourke, L., Garrison, D.R. & Archer, W. (2001). Assessing teaching presence in a computer conferencing context. Journal of Asynchronous Learning Networks, 5(2), pp. 1- 17.

Best, J.W. (1970). Research in education (2^nd edition). Englewood Cliffs, N.J.: Prentice-Hall, Inc.

Bloom, B.S., Engelhart, M.D., Furst, E.J., Hill, W.H. & Krathwohl, D.R. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook 1: Cognitive domain. New York: David McKay Co., Inc.

Borg, W. & Gall, M. (1989). The methods and tools of observational research. In W. Borg & M. Gall (Eds.), Educational Research: An introduction (5^th ed., pp. 473 – 530). London: Longman. (Cited in Garrison, D. R., Anderson, T., & Archer, W., 2001).

Bostock, S. (1997). Designing web-based instruction for active learning. In Badrul Khan (Ed.), Web-based instruction (pp. 225-230). Englewood Cliffs, NJ: Educational Technology Publications.

Bullen, M. (1998). Participation and critical thinking in on-line university distance education. Journal of Distance Education, 13(2), pp. 1-32.

Collison, G., Elbaum, B., Haavind, S. & Tinker, R. (2000). Facilitating online learning: Effective strategies for moderators. Madison, Wisc.: Atwood Publishing.

Commonwealth of Learning. (1993). Student support services: Towards more responsive systems. Vancouver: Author.

Davie, L. & Wells, R. (1992). Empowering the learner through computer-mediated communication. In M.G. Moore (Ed.), Distance education for corporate and military training (Readings in Distance Education, No. 3, pp. 104 - 112). The Pennsylvania State University, American Centre for the Study of Distance Education.

Dede, C. (1996). The evolution of distance education: Emerging technologies and distributed learning. The American Journal of Distance Education, 10(2), pp. 4 - 36.

Fahy, P. J. (2002a). Epistolary and expository interaction patterns in a computer conference transcript. Journal of Distance Education, 17(1), pp. 20 – 35.

Fahy, P. J. (2002b). Use of linguistic qualifiers and intensifiers in a computer conference. The American Journal of Distance Education, 16(1), pp. 5 – 22.

Fahy, P. J. (2003). Indicators of support in online interaction. International Review of Research in Open and Distance Learning, 4(1). Available: http://www.irrodl.org/content/v4.1/fahy.html.

Fahy, P. J. (2004). Use of the Bales model for analysis of small group communications in the analysis of interaction in computer-based asynchronous conferences. Paper presented to the IASTED International Conference on Knowledge Sharing And Collaborative Engineering (KSCE 2004), November 22-24, St. Thomas, US Virgin Islands. In Boumedine, M and Ranka, S. (Eds.), Proceedings of the IASTED International Conference on Knowledge Sharing And Collaborative Engineering (pp. 44 – 49). Calgary: ACTA Press.

Fahy, P. J., Crawford, G., Ally, M., Cookson, P., Keller, V. and Prosser, F. (2000). "The Development and Testing of a Tool for Analysis of Computer Mediated Conferencing Transcripts." Alberta Journal of Educational Research, 46(1), pp. 85 - 88.

Fahy, P. J., Crawford, G. & Ally, M. (2001). Patterns of interaction in a computer conference transcript. International Review of Research in Open and Distance Learning, 2(1). Available: http://www.irrodl.org/content/v2.1/fahy.html.

Fahy, P. J., & Ally, M. Student learning style and asynchronous CMC in distance education. American Journal of Distance Education (in press).

Fulford, C. P. & Zhang, S. (1993). Perception of interaction: The critical predictor in distance education. The American Journal of Distance Education, 7(3), pp. 8 – 21.

Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of distance education, 15(1), pp. 7 – 23.

Garton, L., Haythornthwaite, C., & Wellman, B. (1997). Studying online social networks. Journal of Computer-Mediated Communications, 3(1). Retrieved December 17, 1999, from: http://www.ascusc.org/jcmc/vol3/issue1/garton.html

Gibson, C. C. (1996). Collaborative learning in action via computer mediated conferencing. In Proceedings of the 12^th Annual Conference on Distance Teaching and Learning: Designing for Active Learning (pp. 121 – 125). Madison: University of Wisconsin.

Gunawardena, C., Lowe, C. & Anderson, T. (1997). Analysis of a global on-line debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing. Journal of Educational Computing Research, 17(4), pp. 395-429.

Haughey, M. & Anderson, T. (1998). Networked Learning: The pedagogy of the internet. Montreal: Chenelière/McGraw-Hill.

Henri, F. (1992). Computer conferencing and content analysis. In A. Kaye (Ed.), Collaborative learning through computer conferencing: The Najaden papers (pp. 117-136). Berlin: Springer-Verlag.

Herring, S. C. (1996). Two variants of an electronic message schema. In Susan C. Herring (Ed.), Computer-mediated communication (pp. 81-106). Philadelphia: John Benjamins Publishing Co..

Jonassen, D., Davidson, M., Collins, M., Campbell, J. & Bannan Haag, B. (1995). Constructivism and computer-mediated communication in distance education. American Journal of Distance Education,9(2), pp. 7 – 26.

Jonassen, D. (1998). Designing constructivist learning environments. In C.M. Reigeluth (Ed.), Instructional Theories and Models, 2^nd ed (pp. 1-21). Mahwah, NJ: Lawrence Erlbaum. Available: http://www.ed.psu.edu/insys/who/jonassen/cle/cle.html.

Kanuka, H. & Anderson, T. (1998). Online social interchange, discord, and knowledge construction. Journal of Distance Education, 13(1), pp. 57 – 74.

Keller, V. (1999). Adaptation and application of a transcript analysis tool to analyze a computer-mediated communication (CMC) distance education course transcript. Unpublished master's thesis, Athabasca University, Athabasca, Alberta.

Marttunen, M. (1998). Learning of argumentation in face-to-face and e-mail environments. (ERIC Documents: ED 422791.)

Mayer, R. E. (2001). Multimedia learning. New York: Cambridge University Press.

McKlin, T., Harmon, S. W., Evans, W. & Jones, M. G. (2002). Cognitive presence in Web-based learning: A content analysis of students’ online discussions. ITFORUM Paper #60. Downloaded March 22, 2002, from: http://it.coe.uga.edu/itforum/paper60/paper60.htm.

Poscente, K. (2003). Trigger analysis in computer-mediated conferencing. Unpublished master’s thesis, Athabasca University, Athabasca, Alberta, Canada.

Puustjärvi, J. (2004). Dynamic ontologies in information retrieval. Proceedings of the IASTED International Conference on Knowledge Sharing and Collaborative Engineering, St. Thomas, USVI, November 22 – 24, pp. 115 – 120.

Riffe, D., Lacy, S., & Fico, F. (1998). Analyzing media messages: Using quantitative content analysis in research. Mawah, N.J.: Erlbaum. (Cited in Rourke, et al., 2001.)

Roblyer, M. D. & Schwier, R. A. (2003). Integrating educational technology into teaching, Integrating educational technology into teaching, Canadian edition. Toronto: Pearson Education Canada Inc.

Romiszowski, A. (1997). Web-based distance learning and teaching: Revolutionary invention or reaction to necessity? In Badrul Khan (Ed.), Web-based instruction (pp. 25 - 37). Englewood Cliffs, NJ: Educational Technology Publications.

Rose, M. A. (2004). Comparing productive online dialog in two groups styles: Cooperative and collaborative. The American Journal of Distance Education, 18(2), pp. 73 -- 88.

Rourke, L., Anderson, T., Garrison, D. R. & Archer, W. (1999). Assessing social presence in asynchronous text-based computer conferencing. Journal of Distance Education, 14(2), pp. 50 - 71.

Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (2001). Methodological issues in the content analysis of computer conference transcripts. International Journal Of Artificial Intelligence and Education, 12. Available from: http://cbl.leeds.ac.uk/ijaied/.

Rourke, L., & Anderson, T. (2004). Validity in quantitative content analysis. Educational Technology Research and Development, 52(1), pp. 5 - 18.

Simon, J. & Burstein, P. (1985). Basic research methods in social science (3rd Edition). Toronto: McGraw-Hill, Inc.

Strother, J. (2002). An assessment of the effectiveness of e-learning in corporate training programs. International Review of Research in Open and Distance Learning, 3(1). Downloaded March 11, 2003, from http://www.irrodl.org/content/v3.1/strother.html.

Thorngate, W. 1976. “In general” vs. “it depends”: Some comments on the Gergen-Schlenker debate. Personality and Social Psychology Bulletin 2, pp. 404 – 410. (Cited in Weick, 1979).

Walther, J. B. (1996). Computer- mediated communication: Impersonal, interpersonal and hyperpersonal interaction. Communication Research, 20(1), pp. 3 – 43.

Weick, K. E. (1979). The social psychology of organizing, 2^nd edition. Reading, Mass.: Addison-Wesley Pub. Co.

Witte, S. (1983). Topic structure and revision: an exploratory study. College Composition and Communications (34), 3, pp. 313 - 341.

Yates, S. J. (1997). Gender, identity and CMC. Journal of Computer-Assisted Learning, 13, pp. 281 – 290.

Zhu, E. (1996). Meaning negotiation, knowledge construction, and mentoring in a distance learning course. In Proceedings of Selected Research and Development Presentations at the 1996 National Convention of the Association for Educational Communications and Technology (18th, Indianapolis, IN). (ERIC Documents: ED397849.)

About the Author

Patrick J. Fahy is an Associate Professor in the Centre for Distance Education (CDE), Athabasca University. His career has included teaching and research at all levels of adult education, private sector management and training experience, and consulting with government and private sector clients. Presently, in addition to developing and teaching educational technology courses in the CDE, Pat coordinates the Advanced Graduate Diploma in Distance Education (Technology) program, and the CDE‘s annual Athabasca University Distance Education Technology Symposium (DETS). His research interests include online interaction media, and CMC transcript analysis. He is a former president of the Alberta Distance Education and Training Association (ADETA). He can be contacted as follows:

Patrick J. Fahy, Ph.D., Associate Professor
Centre for Distance Education
Athabasca University
1 University Drive
Athabasca, Alberta, Canada T9S 3A3