Editor’s Note: The computer offers logical options for testing that increase reliability and scalability for  large numbers of learners in training and education environments.  Immediate feedback stimulates motivation and supports learning as Sydney Pressey discovered in 1925.  This article goes in depth into different kinds of computer assisted testing and feedback to support learning.

Adaptive Feedback Characteristics in CAT

Anastasios A. Economides

Greece

Abstract

Feedback is an essential element in learning. The educator (real or artificial) responds to the learner’s actions, emotions, intentions, etc. in order to help her to become self-aware and improve. Recently, Computer Adaptive Testing (CAT) is gaining high popularity due to its efficiency in testing large number of examinees. This paper examines the attributes of Adaptive Feedback in CAT. Designers and developers of CAT systems may rely on these attributes to produce effective feedback adapted to the learner or the educational context.

Keywords: adaptive feedback, CAT, adaptive testing, cognition, affection, emotion, conation.

Introduction

Many schools, organizations, licensing boards and states are currently using CAT (Computer Adaptive Testing) (Schaeffer et al., 1995, 1998; Dunkel, 1997; Wainer at al., 2000; Sri Krishna, 2001; Rudner, 2002; Di Challis, 2003). This growth of CAT will continue propelled by the huge number of applicants. The popularity of CAT is due to a number of reasons. It provides highly reliable and valid results. The learner can take the test at her pace, and receive her score immediately. She can be challenged, but not discouraged or get bored by the presentation of questions that are far above or below her ability level. The cheating is restricted because the learner sees different questions each time she takes the test, and different questions from the other learners. Also, the questions can include multimedia and interactivity.

Assessment is not only aiming at measuring the performance of the learner, but also assisting her to learn, create knowledge and abilities. Feedback is an important tool to support these aims. This paper analyzes the characteristics of feedback in CAT. Designers and developers of CAT systems would be inspired and guided by these feedback characteristics in order to incorporate effective feedback into the CAT systems. This feedback would be based on the individual learner characteristics and/or on the specific educational context.

The usual definition for the feedback in education relates the teacher’s response to the learner’s actions. Here, we introduce a more general definition. Feedback is the educator’s (real or artificial) response to the learner’s actions, thoughts, emotions, needs, attitudes, wills, intentions etc. It may aim to control, guide and regulate the learner, or instruct and teach her, or help and support her. It may inform the learner about her progress, her strengths and weaknesses. It may also try to develop, enhance and improve the learner’s strengths as well as reduce and correct her weaknesses.

Previous research (Kulhavy and Stock, 1989; Mason and Bruning, 2001) classifies feedback into the following categories: No feedback, Knowledge of response, Answer until correct, Knowledge of correct response, Topic contingent, Response contingent, Bug related and Attribute isolation.

Carter (1984) discusses four characteristics of feedback: function, timing, schedule, and type. Schimmel (1988) recommends allowing high ability learners to select desired feedback types because they have extensive prior knowledge and enhanced metacognitive skills. In contrast, since low ability learners tend to be less confident in their own academic skills and less aware of their metacognitive processes, they may be inclined to select feedback that provides them with the correct answer as opposed to the type of feedback that promotes the greatest learning. Freedman (1989, 1991) found that girls tend to focus on the static images in computer graphics that use lighter colors, while boys tend to like computer graphics that use bold colors and smaller, animated objects. Bationo (1992) found that the combination of written and spoken feedback was more effective than only written or only spoken feedback for the immediate recall of the learning material. However, there was no significant difference for the retention of the learning material. Sales (1993) suggests considering individual differences such as ability, learning style, and motivation in designing adaptive feedback. Azevedo and Bernard (1995) suggest that immediate feedback provides the best instructional advantage to the student. Morrison et al. (1995) found that knowledge-of-correct-response and delayed (providing feedback at the end of the testing session) feedback produced greater learning than answer-until-correct or no-feedback for lower level questions (verbatim post test questions). For higher-level questions (paraphrased or transformed posttest questions), however, there were no learning differences with respect to the various feedback types. Butler and Winne (1995) assign five functions to the feedback: Confirming conditions, Adding information, Replacing or overwriting prior knowledge, Tuning understandings, and Restructuring schemata. Narciss and Huth (2002) suggest design principles for effective feedback. They also consider the nature and quality of the feedback, the characteristics of the instructional context, and the individual learner characteristics. Herschell, et al. 2002 suggest that specific feedback is much more likely to influence student performance than haphazard, general feedback.

In the next Section 2, we introduce the feedback characteristics. Designers and developers of CAT systems would consider these characteristics to design and develop effective CAT systems. The CAT systems would adapt these feedback characteristics to the learner and/or the educational context.

Adaptive Feedback Characteristics

In this section, we define and analyze characteristics (attributes) for adaptive feedback in CAT (Computer Adaptive Testing). Designers and developers of CAT systems may rely on these attributes to provide adaptive feedback based on the learner and/or the educational context. For example, if the learner is answering correctly all answers then minimum feedback would be presented to her. Otherwise, if she is not doing well, then elaborative feedback would be presented to help her. The following table presents the proposed feedback characteristics:

Table 1

Proposed Feedback Characteristics

1. Reason & Cause

The first question to ask is what may cause the activation of feedback. It may be a “good” or a “bad” action, behavior or emotion of the learner. For example, the learner may answer correctly or wrongly a question. Or, the learner may cross a threshold. Or, the learner may accidentally push a button on the interface. Or, the learner may become frustrated.

2. Result, Effect & Outcome

The next question to ask is what should be the result of the feedback. We consider the following feedback types with respect to the result:

Confirmation and Verification feedback: it confirms and verifies a correct action, or assures the successful route of the learner.

Maintaining feedback: it supports the learner to maintain her successful progress.

Enhancing feedback: it helps the learner to enhance (strengthen, improve, amplify, augment, increase, magnify) her expertise, abilities, skills, strengths, competencies, and masteries, and gives her a leg up.

Prevention and Avoidance feedback: it helps the learner to prevent and avoid an error, failure or inefficiency.

Error recognition feedback: it helps the learner to recognize an error, failure or inefficiency.

Weakening feedback: it helps the learner to weaken (reduce, lower, lessen, decrease, diminish) her weaknesses, disabilities and inefficiencies. 

Alleviation feedback: it helps the learner to alleviate and relief an error, failure or inefficiency.

Correction feedback: it helps the learner to solve and correct an error, failure or inefficiency.

For example, the feedback may notify the learner of a misconception she may has. Or, it may try to relief the learner’s anxiety.

3. Reinforcement Method

Depending on the adopted didactic method, the feedback may be: 1) Reward (positive), 2) Neutral, or 3) Punishment (negative). For example, if the learner behaves well, then the feedback may praise her or even may challenge her. If the learner behaves improperly, then the feedback may excuse her or criticize her. If the learner makes a mistake, then the feedback may take it up or blame her. Then it may help her to correct this mistake. If the learner performs a successful action, then the feedback may congratulate her or push her further. If the learner fails, then the feedback may ignore her failure or scold her.

4. Mind Dimension 

The feedback may be of cognitive, affective and emotional, or conational dimension.

4.1. Cognitive Feedback

Cognitive feedback is related to the mental activities of the learner. It may inform the learner about the CAT system, the test, the content, the question sequencing, etc. It may help her to answer the questions. It may explain the answers. It may inform the learner on her progress, her achievements, her status in comparison to other learners, etc.  Extending Bloom’s taxonomy, we consider the following cognitive feedback:

Informative feedback: it provides new and unknown information and knowledge, e.g. on how to use the CAT system.

Remembering feedback: it helps the learner to remember, recall and retrieve her acquired knowledge, e.g. it provides definitions, keywords, names, etc.

Comprehension feedback: it helps the learner to comprehend and understand her knowledge, e.g. it provides a similar question with the correct answer, interpretation of the question, etc.

Application feedback: it helps the learner to apply her knowledge, e.g. it applies data on a formula, graph, or program, it simplifies or solves a mathematical formula, etc.

Analysis feedback: it helps the learner to analyze the problem, e.g. it analyzes the assumptions and hypotheses of a question, it compares the possible answers, it provides special cases of the question, etc.

Synthesis feedback: it helps the learner to synthesize the problem solution, e.g. it combines concepts and ideas, it integrates mathematical formulas, it provides generalizations of the question, etc.

Evaluation feedback: it advises the learner on evaluating and selecting the correct answer, e.g. it critiques an answer, it provides criteria for judging, etc.

Creativity feedback: it guides the learner to explore and innovate, e.g. it pushes the learner to take an initiative, it asks the learner to imagine and create a virtual situation, etc.

Collaboration feedback: it supports the learner to collaborate with her classmates to solve a problem, e.g. it asks the learner to share her work, it pushes the learner to express her opinion, it gives credit to every member of a group, etc.

Management feedback: it supports the learner to develop her managerial abilities to organize projects and people, e.g. it supports the learner to organize the tasks of a group, it nominates the learner as the leader of a group, it helps the learner to guide other learners, etc.

4.2. Affective & Emotional Feedback.

Affect refers to the emotional interpretation of perceptions, information, or knowledge. Recently, there is a lot of research interest on affective computing. Researchers are trying to design and develop instruments that will monitor, record, analyze and respond to the user emotions. For example, measuring the timing of the changes in a voice’s pitch may detect and recognize if a person is angry. If the person is angry a more careful treatment of her should be made. We consider the following affective feedback types:

Positive emotions feedback: it acts and expresses positive emotions to the learner trying to develop, maintain and increase the learner’s positive emotions, such as the following: 1) Enthusiasm and Fascination, 2) Happiness and Joy, 3) Satisfaction and Fulfillment, 4) Calmness, 5) Hope and Optimism, 6) Sympathy and Love. Types of positive emotions feedback are the following: 1) Humour and Jokes, 2) Entertainment and Games, 3) Reward, 4) Sympathy and Goodwill, 5) Positive Surprise, 6) Encouragement, 7) Acceptance, 8) Praise and Congratulations.

Negative emotions feedback: it expresses negative emotions to the learner trying to increase the learner’s effort and commitment. Types of negative emotions feedback are the following: 1) Criticism, 2) Blame, 3) Anger, and 4) Punishment.  

Negative-to-Positive emotions feedback: it tries to decrease the learner’s negative emotions, such as the following: 1) Anxiety and Stress, 2) Sadness, Sorrow, Melancholy and Depression, 3) Anger and Madness, 4) Boredom, 5) Fear, Worry, and Doubt, 6) Confusion, 7) Pessimism,  Defeatism and Self-pity, 8) Frustration, Despair, Hopelessness, and Panic, 9) Astonishment, 10) Shame, Guilt and Embarrassment, 11) Disgust and Aversion, 12) Hate. We classify the negative-to-positive emotions feedback into 4 stages. The feedback in the first stage should try to avoid and prevent the development of negative emotions. The feedback in the second stage should try to control and manage the learner’s negative emotions not allowing them to grow. The feedback in the third stage should try to relief and alleviate the learner’s negative emotions lowering them. The feedback in the fourth stage should try to transform the learner’s negative emotions to positive ones.

4.3. Conational Feedback

Conational feedback would try to support and enhance the learners’ will to learn and succeed in the CAT. We consider the following conational feedback types:

Positive conational feedback: it tries to enhance the learner’s self, such as the following: 1) Self-awareness and Self-consciousness, 2) Interest, Will and Volition, 3) Self-efficacy, Self-esteem and Confidence, 4) Motivation, 5) Self-direction and Goal-orientation, 6) Commitment, Dedication, Determination and Persistence, and 7) Self-regulation.

For example, the feedback may inform the learner about the meaningfulness, usefulness, importance and significance of the learning (in general as well as for the specific subject) and of succeeding in the CAT. It may increase the learner’s trust on the CAT validity and reliability. It may spur the curiosity of the learner. It may challenge her. It may increase her belief on herself and on her specific abilities. It may increase her belief that her effort leads to success and on her expectancy for success. It may motivate the learner showing to her the rewards, gains, profits, earnings and benefits from learning (in general as well as the specific subject) and succeeding in the CAT. It may support her orientation towards learning and succeeding in the CAT. It may help her to plan and implement her learning and test taking strategies. It may support and reward her efforts, courage and patience. It may help her to manage and control her time and actions. It may gain the learner’s attention, focus and concentration. It may reward her discipline. It may offer alternatives (with or without) arguments to the learner to choose from.

Negative-to-positive conational feedback: it tries to decrease the learner’s negative attitudes such as the following: 1) Self-ignorance, 2) Disinterest, 3) Self-doubt and Insecurity, 4) Discouragement, 5) Disorientation and Distraction, 6) Reluctance and Hesitance, and 7) Disorganization.

For example, the feedback may decrease the doubt of the learner on her abilities and on her success. It may decrease her insecurity. It may try to keep her focused on the learning and on succeeding in the CAT. It may try to overcome her reservations and hesitation to move forward to the CAT. It may try to trig up her organizational chaos.

5. Quantity and Amount

The quantity and amount of the feedback would vary based on the learner’s state. It would vary from short (brief) to extensive (detailed). For an expert, the quantity may be minimal. On the contrary, for a novice the quantity may include extensive explanations and examples. For example, the feedback to the learner after her answer to a question may be: i) null, ii) score, iii) score and the correct answer, iv) score and explanation about the correct and wrong responses, v) the same as in iv plus references, additional material, examples, cases, etc.

6. Simplicity

The feedback would be simple or complex. For example, it would be simple with respect to the syntax, grammar, concepts and ideas. The same content may be given either in simple “words” or in complicated, elaborate and rare “words”.

7. Presentation, Media, Layout and Format

Various format and layout types may be used for the feedback. Also, various multimedia (text, pictures, figures, animation, audio, video, etc.) may be incorporated in the feedback depending on the learner and the situation. For example, if the learner is a verbal person, then the question and possible responses may also be spoken.

8. Proximity and Relevance

The feedback would be exactly to the point or roundabout. For example, the hint to answer a question would be exact and precise or fuzzy and approximate. Depending on the level of help, the system would provide feedback that is more or less close to the answer. Another example is that the feedback to a question would be a mathematical formula, a graph, or a number. The feedback proximity (relevance) has the following dimensions:

i) specific and concrete -versus- general and abstract,

ii) clear -versus- ambiguous and fuzzy, 

iii) exact and precise -versus- approximate,

iv) direct, straightforward and explicit -versus- indirect and implicit.

9. Area

The feedback would be triggered by the learner’s actions on one of the CAT areas, e.g. on the user interface, on her answers, on her communication and collaborative activities. The feedback may inform, advise and help the learner in any of the following CAT areas: 1) User Interface, 2) System (hardware and software), 3) Content, 4) Presentation, 5) Navigation, 6) Orientation, 7) Sequencing, 8) Communication and Collaboration, 9) Test Duration, 10) Test Control, and 11) Scoring. For example, an agent may help the learner on formatting or spell checking her answer. A guide may tour the learner on the CAT system. A search engine may help the learner on finding useful reference material to accomplish her task. An avatar may help the learner to answer a question.

10. Frequency

The feedback may be triggered and presented to the learner rarely or frequently. The frequency of its appearance depends on the learner state. For example, an independent learner may prefer low system intervention. On the other hand, a scared learner may need continuous guidance. So, the system should provide her feedback very often.

11. Scheduling and Triggering instances

The feedback may appear at scheduled or dynamic instances. Its appearance may also be deterministic (fixed) or probabilistic.

In scheduled feedback, the system may help the learner at pre-specified instances. For example, the system may inform or advise the learner at the beginning of the test, or after every answer, or after every N questions, or after every subtopic, or at the end of the test, etc.

In dynamic feedback, the system or the learner may initiate the feedback depending on the learner state. So, the system may continually evaluate the learner’s state and help her whenever she passes some threshold (e.g. time threshold, stress threshold, knowledge threshold). Also, the system may intervene in case of learner’s error or right action. On the other hand, the learner may ask for help whenever she decides (e.g. needs additional information, feels desperate or disoriented).

In deterministic feedback, the feedback appears definitively if it was so decided by the system or the learner.

In probabilistic feedback, the feedback does not always appear. When the system or the learner decides to invoke the feedback, a coin is tossed. Depending on the result, the feedback may or may not appear. Introducing randomness in the CAT, the learner is kept alert and challenged. For example, if the learner invokes for help, then the system flips a coin and either decides to help her or propose her to try more. Similarly, in scheduled feedback, the system may flip a coin after every answer and decides to inform or not the learner about the correct answer.

12. Timing, Synchronism and Currency

There are three feedback types with respect to their timing and currency: i) in advance (preview) that appears beforehand of an action, ii) immediate (instantaneous) that appears immediately after of an action, and iii) delayed that appears some time after the action has taken place. The feedback may inform, notify, advise, recommend, motivate, alert or warn the learner about several issues. So, we may have the following feedback types:

In advance informative feedback: it informs and notifies the learner beforehand of an action. For example, it may inform the learner about the whole test before the test initiates or about a question before it appears. It may inform her about the educational goals and objectives of the test, what subjects and abilities are to be measured, how they will be measured, how they will be interpreted. It may also inform her about the required subject prerequisites, the required computer expertise to use the CAT system, about its access, features, tools, resources, help and hints available, about how to use it. Furthermore, it may inform her about the question types, the media and format, the test type, the test duration and deadlines, FAQ, etc.

In advance advisory, suggesting and recommending feedback: it advises, suggests and recommends alternatives to the learner beforehand of an action. For example, it may advise her about effective test strategies, time management, etc.

In advance alerting, alarming, and warning feedback: it alerts and warns the learner about common errors and inappropriate actions beforehand of an action.

In advance emotional touching and sensitizing feedback: it supports emotionally the learner beforehand of an action. For example, it enhances her enthusiasm, hope and optimism. It may reduce her stress, relax her anxiety and fear.

In advance motivating feedback: it motivates the learner beforehand of an action. For example, it explains to her the test’s usefulness, meaningfulness, appropriateness, reliability, validity, accuracy, fairness, security and confidentiality. It may also inspire her curiosity and gain her attention.

Immediate informative feedback: it informs the learner immediately after of an action. For example, it informs the learner after her response to every question about her current state, her score, the correct answer, explanation and elaboration on the answer, clarification about misconceptions, her progress, comparison to other learners, the remaining time, subjects and questions, the resources availability and restrictions, etc.

Immediate advisory, suggesting and recommending feedback: it advises, suggests and recommends alternatives to the learner immediately after of an action. For example, it may advise the learner after her answer to a question about references on bibliography, additional activities, etc.

Immediate alerting, alarming, and warning feedback: it alerts, alarms and warns the learner immediately after of an action. For example, it may warn the learner if she is trying to cheat, or to access prohibited resources. It may also alarm her about trivial errors (e.g. spelling).

Immediate emotional touching and sensitizing feedback: it supports emotionally the learner immediately after of an action. For example, it may comfort the learner. It may reduce her panic. It may encourage, praise and congratulate her on her effort, on her results, etc. However, it may also criticize and blame her.

Immediate motivating feedback: it motivates the learner immediately after of an action. For example, it may assure her that she is doing well, that she is on the proper route. It may also, stimulate and challenge her.

Delayed informative feedback: it informs the learner after some time of an action. For example, after the test, it informs her about her score, strengths and weaknesses, test duration, for the total test, for each ability and subject tested, for each subtopic, for each question, in comparison to her past performance as well as to other learners.

Delayed advisory, suggesting and recommending feedback: it advises, suggests and recommends alternatives to the learner after some time of an action. For example, after answering questions related to a subject, it advises her about additional studies, work to do, activities, etc.

Delayed alerting, alarming and warning feedback: it alerts, alarms and warns the learner after some time of an action. For example, it alerts the learner about misconceptions and false ideas she may have. It may also warn her after the test about failing the class if she is not studying harder.

Delayed emotional touching and sensitizing feedback: it supports emotionally the learner after some time of an action. For example, it may try to cool down her after the test. Or, it may try to tranquilize an agitated learner. It may congratulate or criticize her about her effort or results. 

Delayed motivating feedback: it motivates the learner after some time of an action. For example, it may enhance the learner’s self-direction and confidence. It may reduce the learner’s shame, guilt and embarrassment.  

13. Duration

The feedback may be provided for the whole duration of the test or during some period of the test (e.g. during the first 10 questions). This feedback duration may be short or long. It may be fixed or dynamic depending on the learner state. It may also be deterministic or probabilistic. For example, a learner who answers correctly all 10 first questions may no need feedback. A learner who answers correctly the 9 first questions and wrongly the 10^th question may need (or not) feedback. This decision would be made probabilistically.

14. Lifetime and Phase

Each time the feedback appears it lasts for a period of time. This feedback lifetime may be short or long. It may be fixed (e.g. 1 minute) or dynamic (e.g. the learner decides). Also, it may vary deterministically or probabilistically.

The feedback lifetime follows the following phases: 1) Cause Recognition, 2) Feedback Activation and Initiation, 3) Feedback Application, 4) Management, 5) Feedback Termination, and 6) Result Evaluation.

15. Memory and its Depth

The feedback may be based only on the current state of the learner or also consider the learner’s past. In the memory-less feedback, the system helps the learner ignoring her past. The decisions are based only on the current learner state. In the memory-based feedback, the system helps the learner taking into account her past and her progress up to the current moment. The depth of the memory is also another attribute. It may be one-step or many-steps back.

16. Comparativeness

The feedback may track only the learner in isolation from other learners or keep comparing her to other learners. In the isolated feedback, it does not take into account the other learners. In the comparative feedback, it informs the learner about her current state and her progress in comparison to other learners.

This comparison may be simple or comprehensive. For example, it may compare only the learner’s score to the average score among all learners. Or, it may compare the learner’s full state to those of other learners.

Furthermore, it may be based on the current state or the progress. For example, it may compare the learner’s current score to those of other learners. Or, it may compare the learner’s route of the score, time per question, correct-wrong answers, etc. to those of other learners.

Regarding the other learners, the comparison may be with respect to: 1) the average learner, 2) the mean learner, 3) the low 20% learners, 4) the top 20% learners, or 5) an expert.

17. Initiator and Controller

The feedback can be initiated and/or controlled: 1) by the system, 2) by a person, or 3) cooperatively by both. This person may be the learner, a classmate, the tutor, the teacher, the examiner, the parents, the employer, etc. In a cooperative mode, the system may propose a set of help alternatives, and the learner selects one of these. In a different cooperative mode, sometimes the learner may call for help, and other times the system may take the initiative to guide the learner.

18. Interactivity

The interactivity between the learner and the system may be at a single or at multiple levels. In a single level interactivity, when the learner performs an action, behavior or emotion, then the system presents to her a single feedback instance (snapshot).

In a multiple level interactivity, the system and the learner have a dialogue. So, depending on the responses of the learner the system presents to her a sequence of feedback instances. For example, the CAT system may present a difficult question to the learner at once or in a sequence of increasing difficulty sub-questions. If the learner has difficulty to answer the question at once, the system may help her at an increasing dose. 

19. Connectivity

The feedback would be connected to other resources or people. For example, the action of the learner may trigger the intervention of a real or artificial tutor (avatar) who will help the learner to accomplish her task. Also, the feedback would initiate a program to run with input variables given by the learner. Finally, the feedback would be connected to additional reference material on the Web, on a database, on a library, etc.

20. Personalization

The feedback would be personalized to the learner and/or the teacher/examiner. So, the various attributes of the feedback are adapted to the learner’s state. For example, the quantity or the proximity would be adapted to the learner’s previous knowledge, abilities, emotions, and values. Also, the presentation would be adapted to the learner’s intelligence (visual, verbal, mathematical, kinesthetic, etc.). Or, if the learner is frustrated, then emotional touching and sensitizing feedback would alleviate her frustration. Furthermore, the feedback attributes would be also adapted to the teacher’s or examiner’ state. Instead of providing only the standard and “objective” point-of-view, it would provide the subjective teacher’s or examiner’s point of view. In this way, pluralism and freedom of expression would be sustained.

21. Educational Context

The feedback would be adapted to the educational context. Obviously, the feedback should correspond to the content of the test questions. It should take into account the prerequisites, the purpose and the expected outcomes of the test. It should be in agreement with the pedagogical and didactic theories of the test. For example, the reinforcement method should agree with the didactic method.

Conclusions

Feedback is a powerful tool to improve learning. This paper analyzes the characteristics of the feedback in CAT. Twenty-one characteristics (attributes) have been analyzed. Designers and developers of CAT systems may rely on this analysis to produce adaptive feedback. The CAT systems would adapt the feedback characteristics to the individual learner and/or the specific educational context. Of course, these feedback characteristics may also be used in other educational (or not) situations.

References

Azevedo, R. and Bernard, R.M.: “A meta-analysis of the effects of feedback in computer-based instruction”, Journal of Educational Computing Research, Vol. 13, No 2, pp. 111 - 127 (1995).

Bationo, B.: “The effects of three feedback forms on learning through a computer-based tutorial”, Calico Journal, Vol. 10, No 1, pp. 45-52 (1992). 

Butler, D. and Winne, P.: “Feedback and Self-Regulated Learning: A Theoretical Synthesis”, Review of Educational Research, Vol. 65, No 3, pp. 245-281 (1995).

Carter, J.: “Instructional learner feedback: A literature review with implications for software development”, The Computing Teacher, Vol. 12, No 2, pp. 53-55 (1984).

Di Challis: “Committing to quality learning through adaptive online assessment”, in Proceedings Second ATN Evaluations and Assessment Conference, (2003).

Dunkel, P.: “Computer-Adaptive Testing of listening comprehension: A Blueprint for CAT development”, The Language Teacher Online, (1997).

Freedman, K.: “Microcomputers and the dynamics of image making and social life in three art classrooms”, Journal of Research on Computing in Education, Vol. 21, No 3, pp. 290-298 (1989).

Freedman, K.: “Possibilities of interactive computer graphics for art instruction:  A summary of research”, Art Education, Vol. 44, No 3, pp. 41-47 (1991).

Kulhavy, R. W. and Stock, W. A.: “Feedback in written instruction: The place of response certitude”, Educational Psychology Review, Vol. 1, No 4, pp. 279 - 308 (1989).

Mason, B.J. and Bruning, R.: “Providing feedback in computer-based instruction: What the research tells us”, (2001), http://dwb.unl.edu/Edit/MB/MasonBruning.html

Morrison, G. R., Ross, S. M., Gopalakrishnan, M. and Casey, J.: “The effects of feedback and incentives on achievement in computer-based instruction”, Contemporary Educational Psychology, Vol. 20, pp. 32 - 50 (1995).

Narciss, S. and Huth, K.: “How to design informative tutoring feedback for multi-media learning”, in Instructional design for multimedia learning, eds. Niegemann,H., Brunken, R. and Leutner, D., Munster: Waxmann, (2002).

Rudner, L.M. An On-line Interactive Computer Adaptive Testing Tutorial. http://edres.org/scripts/cat/catdemo.htm

Rudner, L.M. (2002). An Examination of Decision-Theory Adaptive Testing Procedures, Conference of American Educational Research Association, New Orleans, LA April 1-5.

Sales, G.C.: “Adapted and adaptive feedback in technology-based instruction”, in Interactive instruction and feedback, eds. Dempsey, J.V. and Sales, G.C., pp. 159-175. Englewood Cliffs, NJ: Educational Technology Publications, (1993).

Schaeffer, G.A. et al.: “The introduction and comparability of the computer adaptive GRE general test”, GRE Board Professional Report No.88-08aP, Educational Testing Service, Princeton, New Jersey, (1995).

Schaeffer, G.A. et al.: “Comparability of paper-and-pencil and computer adaptive test scores on the GRE general test”, GRE Board Professional Report No. 95-08P, Educational Testing Service, ETS Research Report 98-38, Princeton: ETS., (1998).

Schimmel, B.J.: “Providing meaningful feedback in courseware”, in Instructional designs for microcomputer courseware, ed. Jonassen, D.H., pp. 183-196. Hillsdale, NJ: Lawrence Erlbaum Associates, (1988).

Sri Krishna, T.M.: “Web-based adaptive testing”, Learning Technology newsletter, Vol. 3, No 3, pp. 44-47 (2001).

Wainer, H. et al.: “Computer Adaptive Testing”, Second Edition. London: Lawrence Erlbaum Associates, Inc. (2000).
 

About the Author

Prof. Anastasios A. Economides received the Diploma degree in Electrical Engineering from Aristotle University of Thessaloniki, in 1984. Holding a Fulbright and a Greek State Fellowship, he received a M.Sc. and a Ph.D. degree in Computer Engineering from the University of Southern California, Los Angeles, in 1987 and 1990, respectively. Currently, he is Vice-Chairman of the Postgraduate Program in Information Systems at the University of Macedonia, Thessaloniki, Greece. He has published over 90 peer-reviewed papers.

Prof. Anastasios A. Economices
Information Systems Department
University of Macedonia, Thessaloniki 54006, Greece

economid@uom.gr