The effect of competition and context of advisement in a simulation/simulation game to promote transfer of mathematics skills in middle school students.
 

Richard Van Eck
(Meet the Author)

Assistant Professor

University of Memphis


Abstract

This study looked at the effect of contextual advisement and competition on transfer of mathematics skills in a computer-based instructional simulation game and simulation in which game participants helped their “aunt and uncle” fix up a house. Competition referred to whether or not the participant was playing against a computer character, and context of advisement referred to whether the participant had access to a reference book and video clips, or just the reference book. The video consisted of advice on how to solve the problem and was delivered by the “aunt and uncle.” One hundred and twenty-three seventh- and eighth-grade students were randomly assigned to one of five conditions formed by crossing the two independent variables and adding a control group. Results indicated that non-competitive conditions may be best for transfer learning and that high-contextual advisement (video) may promote transfer.

Purpose

The primary purpose of this study was to determine if a computer-based instructional mathematics simulation game or simulation (delineated by the presence or absence of competitive elements) could promote transfer by including built-in advisement and by situating the transfer opportunities and advisement in a meaningful, authentic context. 

Transfer

Despite the importance of transfer of learning in education, learners in general do not transfer learning (Asch, 1969, Gick & Holyoak 1980; Perfetto, Bransford, and Franks, 1983; Reed, Ernst, & Banerji, 1974; Simon & Hayes, 1976; Thurman, 1993; Weisberg, DiCamillo, & Phillips, 1978) including, according to the Cognition and Technology Group at Vanderbilt (CTGV) mathematics (CTGV, 1992a, 1992b; Van Haneghan, 1990).

Evidence suggests that problem solving and transfer are largely domain specific, so transfer is not likely to occur as the result of general problem-solving instruction (Larkin, 1989), but instead requires multiple practice opportunities in a variety of contexts (Gagné, Briggs, & Wager, 1992).

Formal learning frequently assumes that abstract principles and skills are applicable across multiple domains and that transfer will emerge automatically from the acquisition of these general skills. Although some cognitive psychologists disagree that knowledge is wholly tied to the context in which it is learned (e.g., Anderson et al., 1996), many researchers have found that knowledge and transfer are strongly tied to context and domain (e.g., Bransford, Franks, Vye, & Sherwood, 1989; Bransford, Sherwood, Vye, & Rieser, 1986; Brown, Collins, & Duguid, 1989; Perkins & Salomon, 1989).

Royer (1979) defines transfer in general as “the extent to which the learning of an instructional event contributes to or detracts from subsequent problem solving or the learning of subsequent instructional events” and says that “transfer of learning is evidenced by the ability to apply a particular skill, or bit of knowledge, to situations differing from those encountered during original learning” (p. 53).

Transfer in this study is categorized as positive, horizontal transfer. Horizontal transfer (Gagné, 1965) refers to “the sort of transfer that occurs when a child recognizes that the fractions he is learning about in school are relevant to the problem of deciding how to divide up a prized, but jointly owned, marble collection” (Royer, 1979, p. 54). Positive transfer refers to the successful (i.e., correct and appropriate) application of prior knowledge to novel contexts and problems.

In this study, the participants applied previously learned mathematics skills in a new context beyond what may be found in most traditional mathematics instruction. Students who had studied area, volume, perimeter, addition, subtraction, multiplication, division, and calculation of equivalent measurements were required to apply this prior learning to determine the amount of paint and wallpaper border needed to remodel a room in a house.

Operational Definition: Transfer of Mathematics Skills. The activation, retrieval, and application of previously acquired mathematics skills to the successful solution of a problem set in a novel context. Novel refers to a performance context in which the stimulus elements differ from those in the original learning context. In this study, transfer as defined above was measured by transfer scores, which were measured by the ability to select and apply relevant formulas to two problems in a simulation or simulation game.

Anchored Instruction

One way to address failure to transfer is through the use of authentic learning paradigms such as anchored instruction, which is related to situated cognition, a theory proposed by Lave and Wenger (1991). The emphasis in anchored instruction is to design learning and teaching activities around an authentic situation. The learning events, or “anchors,” are embedded in problem-solving environments that the learner is free to explore.

Anchored instruction has been experimentally shown to promote performance and transfer (Sherwood & the Cognition and Technology Group at Vanderbilt [CTGV], 1991; CTGV, 1993; Van Haneghan et al., 1992) and to be more effective in teaching mathematical problem-solving skills than traditional instruction. 

Instructional simulations and games present an excellent means for promoting problem-solving skills and transfer of prior learning by accommodating anchored instruction principles. Anchored instruction requires that the learning take place in a realistic problem-solving situation and that the learner be able to explore the environment. Computer-based games allow for the former through the use of graphics, sound, text, and video, and for the latter through navigational options (e.g., clicking on different parts of the screen to navigate to different places in the environment). While, in theory, well-designed games should function similarly to anchored instruction, no research that had examined this could be found.

Advisement
One problem with using computer-based instruction, however, is that the learner must function more autonomously since, in many cases, access to an instructor is limited. While programs can be written to provide adaptive kinds of intervention (i.e., to provide remediation and/or challenge based on student performance in the instruction to that point), such programs are complex to program and can be prohibitively costly (e.g., Fingar, 1999; Tennyson, 1984) and, as Tennyson (1981) points out, ignore “the important educational goal of student responsibility in learning” (p. 426). 
The problem with learner control is that research has shown that novice learners are ill-equipped to manage their own instruction, at least without feedback (Carrier, 1984; Fisher et al., 1975; Park & Tennyson, 1980; Reinking, 1983; Tennyson, 1980a, 1980b; Tennyson and Buttrey, 1980). According to Peters (1988), the problem with learner control may lie in the “learners’ failure to make effective use of the control given them” (p. 3). Researchers have looked to advisement, or “coaching,” to provide enough information to learners for them to effectively manage their own learning in computer-based instruction. Advisement can help learners process content or clear up misconceptions—roles usually addressed by a teacher in traditional instruction. 

There is some support for integrating video advisors of this nature into computer-based training (CBT). A model for human-computer interaction proposed by Streitz (1988) posits that interaction problems require the learner to build a representation of the tutoring system. In addition to the "learner" and the "system," the model proposes a human tutor who functions as a problem mediator, making suggestions or asking questions about specific content domains. It is this type of advisement that this study and others explore (Bennett, 1992; Boulet, 1993, 1994; Clariana, 1989; Fingar, 1999).

Such forms of advisement may have special relevance for promoting transfer. One means of promoting transfer of learning involves making the connection between the learning context and performance context explicit (Adams et al., 1988; Brown, 1989; Gick & Holyoak, 1980; Hayes & Simon, 1977; Lockhart et al., 1987; Perfetto et al., 1983; Reed et al., 1974; Simon & Hayes, 1976; Weisberg et al., 1978). Likewise, for insight problems (those that require reconceptualizing the problem), helping learners to think about the problem in a new way has been shown to increase transfer of learning (Lockhart et al.). 

If, as some researchers suggest (e.g., Black & Schell, 1995; Perkins & Salomon 1989), transfer is highly context dependent and specific, and requires guidance and cueing, then it seems reasonable to assume that a computer-based simulation game with some kind of simulated teacher, or advisor could promote transfer. No research has examined this to date, however. But while research would seem to suggest that the learning and performance contexts should remain as functionally identical as is feasible to promote transfer, it is unclear whether this should be extended to the context of the advisement itself. For instance, advisement could have little to do with the context of the game (e.g., be delivered in the form of text-based prompts and resources) or be intrinsically embedded in the game (e.g., be delivered by a character who is part of the game context). This study examined both forms of advisement.

Operational Definition: Advisement. Advisement is solicited help provided to the learner regarding how to go about solving a problem. As such, it may consist of prompts or cues to reformulate the problem, modeling of problem-solving behavior, and identification of tools and knowledge needed to solve the problem.

Operational Definition: Context of Advisement. Context of advisement refers to the internal and external events associated with the delivery of solicited advice. In this research, advisement has either a high or low level of congruency between the learning context (a game that relies on a storyline about painting and redecorating a room in a house) and the type of advisement (either low: text-based listing of relevant and irrelevant formulas, or high: text-based listing of relevant and irrelevant formulas plus a video of two carpenter/decorators discussing the process).

Competition & Games
What role does competition play in all this? A great deal of research supports the positive effects of individual competition on performance (e.g., Fisher, 1976; Hurlock, 1927; Julian & Perry, 1967; Kraft Miller 1981; Spalt, 1987/1988; Wilkes, 1965), while others show no difference (Craig, 1967) or even negative effects (e.g., Cartmill, 1994; Keefer & Karabenick, 1998; Thompson, 1972).

For competition to promote motivation, performance, and learning, students must perform at less than their maximum level of performance in noncompetitive conditions, which may not always be the case. Competition alone cannot make learners function beyond their maximum ability unless they have help, such as a coach, mentor, or advisor. It may be that competition can improve performance, but that the means and extent to which it does so are at least partially determined by the content, the complexity of the learning, familiarity with the content, the nature of who is competing against whom, and other situational characteristics. Likewise, it seems logical to conclude that there may be some conditions (e.g., learner characteristics, domain) under which competition can be detrimental.

The research studies that show benefits of competition appear to focus on knowledge measures and content in non-problem solving contexts (i.e., at the rule and verbal information levels) and in nonauthentic contexts (i.e., school-based contexts rather than “real world” contexts). It might be argued that such learning requires less cognitive processing than higher-order learning such as problem solving (the most common venue for transfer learning studies).

Operational Definition: Competition. Any condition in which learners are able to compare their performance to some internal or external standard, or to others in their social environment, in such a way that they can tell if they are below, at, or above a reference performance mark.

Operational Definition: Simulation. An interactive experience that contains some representation of a world, real or imagined, that behaves according to a coherent (although not necessarily realistic) set of rules, in which the participant(s) have a clear goal, the pursuit and attainment of which results in an entertaining, rewarding experience.

Operational Definition: Simulation Game. An interactive experience that contains some representation of a world, real or imagined, that behaves according to a coherent (although not necessarily realistic) set of rules, in which the participant(s) have a clear goal, the pursuit and attainment of which results in an entertaining, rewarding experience, and that includes some form of competition.

Alternate Hypotheses

1.Participants who use advisement more often than others will have higher transfer of mathematics scores.

2.Participants in the high-contextual advisement conditions will have higher transfer of mathematics scores than those in the low-contextual advisement conditions.

3.Participants in the non-competitive simulation game conditions will have higher transfer of mathematics scores than participants in the competitive simulation game conditions.

4.Participants in the competitive and non-competitive simulation game conditions will have higher transfer of mathematics scores than participants in the control conditions.

Method

Population

The target population for this study is middle-school-aged children in grades 7 through 8, with a range in age from 11 years to 14 years old. This population was available at several middle schools in a Gulf Coast city, of which four were selected: School A (n = 50), School B (n = 75), School C (n = 123), and School D (n = 80). Schools A and B were used for pilot testing and field trials (respectively) of the game, and School D was unable to participate. Accordingly, the sample for this study included students at School C only. Participants had regular access to the computer lab and access to an edutainment game on math as well as other knowledge and entertainment games during free lab time as part of their normal studies. Demographic data were collected via self-reported instruments developed for this study.

Lesson Content

The content of the lesson was delivered via a computer-based instructional simulation and simulation game and was developed using the National Council of Teachers of Mathematics (NCTM) 2000 mathematics curriculum standards. In particular, the content covered portions of NCTM 2000 content strands 1, 2, and 3.[1]
Problems based on these goals and standards were developed and integrated into an instructional simulation game in which participants played a peer-aged character working for their aunt and uncle’s home remodeling business. Participants were given a room in a house for which they must calculate how much paint was needed to paint the room and how much wallpaper border was needed to put a border around the room at ceiling height.

A computer-based instructional simulation game was developed using Macromedia Authorware 5.1 for Windows 95/98. This simulation game made extensive use of graphics, sound, video, and interactivity. Participants entered a computer-generated room in a “house” and navigated around in it by clicking in the direction they wanted to go. They were able to use a variety of “tools” in the simulation game, including a tape measure to measure walls, doors, and windows, a workbook in which to record information used to solve the problem, a reference book to look up facts and formulas, a calculator, and, in some conditions, a walkie-talkie to call the video advisors for advice. Participants used these tools to learn about the environment (how long/high a wall is, for instance) and they recorded their observations in the workbook built into the simulation game. Participants in the control group were given word problems identical to those in the computer simulation game in the form of a computer tutorial to minimize any differences or resentment due to medium.

Context of Advisement

Advisement was available to all participants in the simulation game. A reference book containing a variety of facts and formulas was provided to all participants. The goal was to measure the ability to apply prior knowledge, not to see if participants had memorized the relevant formulae. Such support devices are common in many computer games. Half the participants were limited to this form of advisement. The rest of the participants received video-based advisement in which their Aunt Ann and Uncle Bob appear to walk into the room and discuss the problem and the solution process. This type of advisement has a high contextual relevance to the storyline of the game itself. These two conditions of advisement are referred to in this study as high- and low-contextual advisement (HCA and LCA, respectively).

Competition

Competition was determined by the presence or absence of a computer-generated competitor. In the competitive environment, participants were told to work quickly because they were competing against a computer character (whose ability level they chose). They were asked to indicate the level of competitor they wanted: below average, average, or above average. In the noncompetitive environment, participants had no opponent to compete against for time or accuracy, but they were encouraged to work quickly and accurately. The two competition conditions are referred to as with competition (WC) and no competition (NC). The four cells formed by crossing the two advisement conditions and two competition conditions are referred to as high-contextual advisement with competition (HCAWC), high-contextual advisement with no competition (HCANC), low-contextual advisement with competition (LCAWC), and low-contextual advisement with no competition (LCANC).

Controls

The control group was given computer-based word problems that were identical in content to those in the simulation game and simulation. They had no access to advisement, nor was any element of competitioninvolved. 
Instruments

In order to collect data for possible use as covariates and for post hoc examinations, a demographic survey was developed to collect data on age, sex, ethnic background, computer experience, mathematics experience, game playing behavior, hours spent on schoolwork and other activities. This scale had a Flesch-Kincaid Grade Level reading score of 3.1.

A pretest was developed to assess incoming mathematics skills and to verify that students were capable performing the mathematical computations required in the simulation game and simulation. This instrument was content validated by the teachers at the participants’ schools and by a professor who teaches mathematics instruction to K-12 teachers at a Gulf Coast university. This instrument had a Flesch-Kincaid Grade Level reading score of 5.2.

Students completed the simulation (NC) or simulation game (WC). Transfer of mathematics skills was then assessed via a second computer-based instructional simulation identical in structure and general content but differing in the setting. Whereas the simulation game and simulation context in the intervention consisted of a room in a house, the transfer posttest was assessed by a simulation set in a movie theater, where participants calculated the amount of material to buy to replace the movie curtain and the number of aisle carpet rolls needed to replace the carpet running around the outside of the theater seating area. No advisement was available, nor was there any element of competition present in this simulation. 

Transfer was measured both by the ability to select the correct formula and to solve the problem correctly (i.e., either was counted as correct). While transfer might theoretically be measured by the selection of the formula alone, some participants are more sophisticated problem-solvers and may be able to solve the problem intuitively (i.e., without selecting the formula from the reference book). Because no formulas beyond the correct one for a given problem would produce the same answer, and because the likelihood of guessing the right answer without using the correct formula was small, a correct answer indicated having used the correct formula.

Research Design

The experimental design was a randomized pretest-posttest design with two independent variables and one dependent variable, resulting in a non-crossed 2 X 2 with control group design (see Table 1). Participants were randomly assigned to conditions beforehand, but participated as a class during their normal class time. Independent variables include context of advisement (advisement as either a text-based formula or text-based formula plus video-based discussion of problem, process, and formula) and mode of instruction (with or without competition). The dependent variable was transfer scores. 
Table 1

Research Design of Study
 

Competition

Competition

Context of Advisement

With Competition (WC)

No Competition (NC)

Control

Low-Contextual Advisement (LCA):
(Reference Book Only)

26

24

24

High-Contextual Advisement (HCA):
(Reference Book & Video Discussion)

25

24

The simulation game was piloted on twenty members of the target population and revised accordingly. The simulation game was then formatively evaluated on 10 members of the target population and modified further. The simulation game was then field tested on 75 members of the target population, and minor changes were made based on observations. The simulation game and the simulation were identical except for the presence or absence of a competitor.

Experiment
Participants were selected from seventh- and eighth-grade classes at a Gulf Coast middle school and assigned in a stratified random manner to one of five conditions: low-contextual advisement with competition, low-contextual advisement without competition, high-contextual advisement with competition, high-contextual advisement without competition, or control. After random assignment to conditions, participants were randomly assigned and run as an intact group for the duration of the study.

Pretest

During the first session (day one), participants received orienting instructions explaining the purpose and process of the study, were given the opportunity to ask questions, and were then given the demographics survey and the pretest, all in computer form. 

Simulation/Simulation Game

Participants returned two days later for the second session (day two) and completed a five-minute computer-based simulation game tutorial, which oriented them to the game interface, including all tools within the game and navigation. They were unable to proceed to the simulation or simulation game until they had demonstrated the use of each tool and element of the interface one time. Participants then began playing the simulation (NC) or simulation game (WC) or worked the identical on-line word problems (controls). Data were collected during the game via the computer and stored as text files for later retrieval. A debriefing form was given to the participants to be filled out and returned later. The teachers were instructed not to discuss or teach the content of the game (i.e., area and perimeter) between sessions.

Posttest

The third session (day three) occurred one week after the second session, when the posttest (the transfer simulation) was administered. Participants were then debriefed about the actual nature of the study. 
Data from the instruments and the game and computer-based word problems (controls) were input directly from the computer-generated files into SPSS. After data screening for outliers and normality, and after checking for appropriate statistical assumptions, ANOVA, bi-variate correlation, and chi-square analyses were performed to test the null hypotheses.

Results

Outliers were removed on a case-by-case basis. Assumptions for the statistical measures used were checked. All fell within acceptable parameters for the inferential statistics used. Tables 2 and 3 present demographic data. Tables 4 and 5 present means and standard deviations for transfer scores.

Table 2


Age, Gender, Grade, and Pretest Math Scores



Table 3

Ethnicity of Participants

African American
Hispanic
Asian
Caucasian
Other
5
2
1
99
4

Table 4

Advisor Use and Transfer of Mathematics Scores by Condition

Condition

(n = 20)

(n = 17)

(n = 17)

(n = 12)

(n = 18)

Total 

(n = 84)

Dependent Variable

M

SD

M

SD

M

SD

M

SD

M

SD

M

SD

Transfer Score
.88
.68
1.23
.85
.76
.83
.71
.69
.92
.84
.88
.79

Table 5

Number of Successful Transfer Participants by Condition 
 
Condition 

Successful Transfer (n / %)

Control 

10 (50%)
High-Contextual Advisement, No Competition (HCANC)
9 (53%)
High-Contextual Advisement, With Competition (HCAWC)
13 (77%)
Low-Contextual Advisement, No Competition (LCANC)
9 (75%)
Low-Contextual Advisement, With Competition (LCAWC)
11 (61%)

To control for differences in treatment time, only those participants who had completed the game or simulation (i.e., had not been forced to quit the game because of a computer problem or who had not accidentally exited the game prior to completing the problems) were included. This resulted in 16 participants not being included for analyses involving transfer. Table 6 presents the number excluded by condition.

Of those that were excluded, none had answered the first or second problem. Only one had used advisement (once) prior to exiting the game, and participants excluded were evenly split between males and females. No other discernable characteristics appeared to differentiate these participants from those included in the analysis.

Table 6

Participants Excluded from Analyses Involving Transfer by Condition

Condition
n
1 (High-contextual advisement without competition) 

1

2 (High-contextual advisement with competition)
3
3 (Low-contextual advisement without competition)
7
4 (Low-contextual advisement with competition)
5

Statistical analysis indicated no significant correlation for hypothesis one and no significant differences between groups for null hypotheses one through four. Null hypothesis five, that there would be no interaction of context of advisement and competition on transfer of mathematics scores, was examined using a 2 x 2 ANOVA. The analysis indicated no significant interaction of competition and context of advisement. A similar 2 X 2 ANOVA post-hoc analysis was run using a transfer score based solely on the ability to complete the problems in the game correctly. Because participants were not required to select formulae, it was felt that those who chose correct formulae may have done so by chance or some other unforeseen reason. Likewise, those who selected incorrect formulae may have realized their error but not bothered to then select the correct formula, choosing instead to work the calculations on their scratch paper. Levene’s test of equality of error variances was significant, indicating the error variance of the dependent variable was not equal across groups. The cell numbers were large and equal. This analysis yielded a significant interaction of competition and context of advisement, F (3, 60) = 4.528, MSE = 3.024, p = .037 (see Table 7 and Figure 1). This null hypothesis was not supported. There was no alternate hypothesis proposed.

Table 7

ANOVA Table of Competition and Context of Advisement on Transfer Score
 

df

F

Significance

Competition 

1

.178

.674

Context
1
.414
.522
Interaction
1
4.528
.037


Figure 1. Interaction of competition and context of advisement on transfer of mathematics score.

Participants in the HCANC condition had higher transfer of mathematics scores than participants in the LCAWC condition. Participants in the LCAWC condition had higher transfer of mathematics scores than those in the both the HCANC and LCANC conditions. No other differences were detected between or among the other conditions. 

To test whether pretest performance was responsible for any transfer effects, a chi-square of the two problems relating to area and perimeter in the posttest and in the game was conducted. There were no significant relationships between pretest and posttest scores on area and perimeter. No significant correlations were found between overall transfer scores and overall pretest scores, either. Finally, a regression of pretest scores on posttest transfer scores also failed to yield any significant predictive relationship.

Discussion

Alternate hypothesis 1, that participants who use advisement more often than others would have higher transfer of mathematics scores, was not supported. Participants who selected advisement more often than others were no more likely to have higher transfer of mathematics scores than were any other participants.

Advisement in this question was measured by the number of times the participants selected either the high-contextual advisement (video-based) or the reference book. Participants in the high-contextual advisement conditions had access to both the video-based advisement and the reference book of facts and formulae, while those in the low-contextual advisement conditions only had access to the reference book. Transfer was measured in a different context, at a different time, using different problems. The transfer variable ranged from 0 to 2, which may not have allowed for enough variability to detect differences, at least with this number of participants. Additional research is needed over a longer period of time in order to allow for more transfer items and more practice opportunities. Also, the advisement itself was not piloted and evaluated using participants to determine if the advisement is effective in reformulating the problem space.

Alternate hypothesis 2, that participants in the high-contextual advisement conditions would have higher transfer of mathematics scores than those in the low-contextual advisement conditions, was partially supported. When transfer was measured solely by the participants’ ability to solve the problem correctly, statistical analysis indicated that those in the high-contextual advisement conditions had higher transfer of mathematics scores than those in the low-contextual advisement conditions. This only occurred in the simulation condition (i.e., without competition). There was no significant difference in transfer of mathematics scores between high-contextual advisement and low-contextual advisement conditions under the competitive condition.

It may be that the presence of competition creates an affective environment in which high-contextual advisement cannot be fully attended to or processed because learners are concerned about the time they have taken (which is displayed on screen) and with beating the competitor. The competitor character in the simulation game in this study was always visible at the bottom right of the screen and randomly commented on how he or she (the competitor) was doing on the problem. Because participants were so conscious of the time factor, waiting for the advisement video to finish playing may have caused stress that interfered with accurate processing of the information. This may account for why the low-contextualized advisement in the competitive simulation game condition resulted in higher transfer of mathematics scores than it did in the non-competitive simulation condition, since learners did not have video advisement and were in control of how much time they spent in the reference book.

Alternate hypothesis 3, that participants in the non-competitive simulation game conditions would have higher transfer of mathematics scores than participants in the competitive simulation game conditions, was not supported. Participants in the non-competitive simulation game condition did best when they had access to high-contextual advisement. Participants in the competitive simulation game condition did best on transfer tasks when they had access to low-contextual advisement.

It appears that advisement should be modified according to whether competition is present or not. Games that make use of a time element may be incompatible with high-contextual advisement, which by its nature takes longer and may be perceived as less relevant. Alternatively, it may be that time constraints and competition may be better suited for building fluency and automaticity than for learning relatively new material and processes, as the transfer problems in this study might well be considered, given the application of learned material in a new setting or context.

Further research examining competition and cooperative learning might also help to explain these results, as some researchers maintain that cooperative learning is best for promoting problem-based learning and transfer (Bransford & Stein, 1993; Dalton, Hannafin, & Hooper, 1989; Reid, 1992; Young, 1993).

Alternate hypothesis 4, that participants in the competitive and non-competitive simulation game conditions would have higher transfer of mathematics scores than participants in the control conditions, was not supported. No significant differences in transfer were found between the control conditions and the combined competitive and non-competitive conditions.

Given that there were no differences in transfer of mathematics scores solely as a result of either competition or context of advisement (main effects), it is perhaps not so surprising that controls did not differ significantly from the other conditions, although controls did have lower transfer scores than any other conditions, with a mean transfer score of .1335, while the transfer of mathematics scores for the other conditions ranged from .25 to .82. It may be that the measure of transfer in this study does not vary enough to detect differences because of a restriction of range. Transfer of mathematics scores ranged from 0 to 2, as they were based on the ability to select and apply the correct formulas for two problems. This was necessary because the intervention was limited by the schools to one 50-minute session, and situated learning is complex and requires elaborate processin,g. Given this and the fact that the problems themselves were complex (e.g., the area problem involved calculating area for unpainted surfaces (windows, etc.) on all walls and ceilings and subtracting that from overall area, which then had to be divided by the square feet per gallon of paint) more than two problems could not have been finished by the learners in the allotted time. 

Limitations

The advisement itself was not validated for effectiveness with problem solving, although most (17 of 29) of those asked indicated that the advisement was good and was helpful or somewhathelpful (18 of 27). A pilot study to evaluate the effectiveness of the advisement would have made the study stronger. This study also did not examine qualitative measures regarding advisement. Debriefing forms asking about the qualitative aspects and affective responses to the instruction were distributed and collected later because there was not enough time in class, but the return rate was low.

The mathematics content of the simulation and game focused on solving two problems: one requiring area and the other requiring perimeter. While all participants were in seventh- and eighth-grade, and thus should have been familiar with these concepts, some were in semi-remedial classes and were still working with these problems, while some others were in advanced mathematics classes. Because participants were randomly assigned, ability was controlled for throughout the conditions, but this did introduce some potential error into the statistical analyses. It would have been better to train the learners to mastery in the content, and then run the intervention weeks or months later.

There was also not enough time available for the learners to work at their own pace. The school required that all sessions take place with intact classes, during the regular 50-minute class periods. As a result, some participants were unable to complete the game, and most had little time for reflection and processing, focusing instead on getting the work done in the allotted time. Those who did not finish the game were excluded from the analyses to minimize error. This may have resulted in an overly conservative test for differences among groups. The fidelity of the treatment condition would have been higher had students been able to work at their own pace over a longer period of time. Because there was not enough time to do more than two problems, transfer of mathematics scores had a restricted range, potentially leading to low variance and validity for this variable. Participants may also have used advisement less because they wee concerned about running out of time.

Good interface design dictates that items and tools should be a logical extension of the metaphor being used. Accordingly, advisement was selected by clicking on a reference book (low-contextual advisement) or clicking on either the reference book and/or a walkie-talkie (high contextual advisement). In order to allow the learners to move about the room to measure and collect information, it was necessary to give the tools as small a “footprint” as possible. While this did not prevent users from finding or using advisement, it may not have been as obvious as prior research has suggested it should be (Dempsey & Van Eck, 1998). Consequently, advisement may not have been selected as often as it might otherwise have been.

Although participants had all received at least one year of training in the content, no external criteria of mastery was available. The study would have been stronger if it had been possible to provide training to mastery prior to the intervention. Finally, transfer may require longer periods of time and multiple practice oportunities and interventions (Gagné et al., 1992; Larkin, 1989). The intervention was limited in this study because the schools could only provide three class days out of their normal curriculum. The pretest instruments alone required one class period, leaving one class period for the game and one class period for the posttest. More interventions over a longer period of time for longer periods of time and the inclusion of qualitative measures may have produced larger changes in transfer.

Because participants were able to type in their answer to either of the two transfer problems in the game and in the posttest without doing any calculations on screen and without selecting any formulae or facts, it is possible some participants entered answers that amounted to guesses. It was not possible to determine with any accuracy whether participants were guessing at the answers because some may have used their scratch paper to do the calculations. While this scratch paper was retained by the researcher, it is problematic to evaluate these sheets for this purpose.

Conclusions

It appears that transfer can be promoted through computer-mediated intervention. One of the factors associated with increased transfer of mathematics scores seems to be whether and to what extent the learners avail themselves of advisement. Instruction that attempts to build in advisement should also explore ways to promote its use; the mere presence of advisement is not enough.

It also appears that contextual advisement can promote transfer under non-competitive conditions. High-contextual advisement in non-competitive conditions produced the highest transfer of mathematics scores. This is probably a function both of the newness of the instruction and of the complexity of the instruction as much as it is a function of the competition. Transfer is a form of problem solving, which is in this case a higher-order intellectual skillinvolving accurate problem space representation, recall of prior knowledge, and the formulation of rules about when and where to apply that knowledge. Accordingly, the cognitive load involved may be higher than for lower-level intellectual skills. Competition may create an affective state of anxiety and pressure that is detrimental to the processing necessary for transfer learning to occur. There were no detectable differences between high-contextual and low-contextual advisement conditions in the competitive simulation game condition.

In summary, for transfer training of this nature, non-competitive simulation games might be a better choice than simulation games that include a time-pressure factor. Advisement seems to be a good way to promote transfer. High-contextual advisement, that is, advisement that is metaphorically tied to the context in which it is found and is interesting, may be the best form of advisement. This is true regardless of the presence or absence of competition but perhaps particularly so for non-competitive simulation games. It tends to promote advisement use, which in turn is associated with transfer. Finally, simulation games seem to be capable of representing authentic contexts, with and without competition, and may be useful in promoting transfer in a variety of subject areas.

Future Research

Taken in conjunction with previous studies on advisement (e.g., Boulet, 1993; Boulet et al., 1990; Dempsey & Van Eck, 1998; Tennyson, 1980a, 1980b, 1981) it would seem that advisement can help learners manage their own instruction, increase performance, and promote transfer. The issue may no longer be whether or not advisement is necessary, but why it is, and how its use can be promoted. Future studies should examine other ways to promote advisement in simulations and games. An earlier study showed that making the advisement option obvious on the screen can increase advisement use (Dempsey & Van Eck, 1998), but this may be contraindicated in simulations and games, where a premium is placed on the immersive quality of the experience. It may be possible to build a kind of adaptive advisement system similar to that developed by Tennyson (1980a, 1980b), but which sends contextual prompts to the learner (e.g., after three errors and/or long periods of inactivity, voices come over the walkie-talkie asking if they need any help). A similar form of advisement has been utilized in a game called Hangtown (Doolittle, 1995).

Further research is needed to determine which factors of the high-contextual advisement used in this study are responsible for the effects observed. High-contextual advisement could be delivered by sound only with no loss in contextual relevance. This would help to determine what kinds of novelty or modality effects may be at work. Similar video clips of people who are generic advisors unconnected to the context of the simulation or game might also be useful.

Competition may inhibit elaboration. Future research might examine the role competition plays in elaborative processing. This should be done taking into account both time stress and competition as separate variables. While this study looked at competition as a factor, it might also be beneficial to examine cooperative learning in similar contexts. Research has shown that cooperative learning may be best for promoting transfer (Bransford & Stein, 1993; CTGV, 1992b; Keller, 1990; Young, 1993).

Further research should consider a mixed methods approach, using think-aloud protocols, observational measures, and oral debriefing to examine the why and how of the trends discussed in this study. Future research might also consider tracking errors and looking for patterns which might then be used to develop adaptive advisors. Future studies might also examine transfer issues in a more longitudinal fashion, perhaps over the course of one or more years.

Further research should be done to examine what kinds of gender differences there are in advisor preference and preference for competition. There was a participant/competitor gender effect; it may be reasonable to expect the same kind of relation between the gender of the participant and the gender of the advisor. It would be useful to examine whether this had any effect on advisor use, which was one of the more robust variables in this study. Such an effect might also impact affect as well. Future studies might provide different gendered advisors and run conditions where gender of advisor and participant were crossed. 

The population for this study are private Catholic school students. There may be a variety of cultural beliefs and attitudes in this population which might be expected to impact the variables in this study. Catholic school students may be less likely to be questioning of teachers, thereby leading to differences in advisement use. Private school students may be more advanced and have better problem-solving skills than public school students. Private school students may also have higher computing skills and abilities because computing technology is more prevalent in private than public schools. Public school populations should be studied in similar fashion to strengthen generalizability of results.

The population under study was aged 12 to 14. The effectiveness of training and instruction using simulations and games should be studied at different age groups. Younger students exposed to this kind of training during instruction on the topic of interest, in this case area and perimeter, might be more successful transferring knowledge than those in this study, who were exposed after having studied the content exclusively in the abstract.

References

Adams, L. T., Kasserman, J. E., Yearwood, A. A., Perfetto, G. A., Bransford, J. D., & Franks, J. J. (1988). Memory access: The effects of fact-oriented versus problem-oriented acquisition. Memory and Cognition, 16(2), 167-175.

Anderson, J. R., Reder, L. M., & Simon, H. A. (1996). Applications and misapplications of cognitive psychology to mathematics education [1]. Unpublished paper, retrieved May 7, 1996, from the World Wide Web: http://act.psy.cmu.edu/ACT/papers/misapplied-abs-ja.html

Asch, S. E. (1969). A reformulation of the problem of associations. American Psychologist, 24, 92-102.

Bennettt, K. (1992). The use of on-line guidance, representation, aiding, and discovery learning to improve the effectiveness of simulation training. In J. W. Regian and V. J. Shute (Eds.), Cognitive approaches to automated instruction (217-244). Hillsdale, NJ: Erlbaum.

Black, R. S., & Schell, J. W. (1995, December). Learning within a situated cognition framework: Implications for adult learning. Paper presented at the annual meeting of the American Vocational Association, Denver, CO. (ERIC Document Reproduction Service No. ED 389 939)

Boulet, M. M. (1993). Intelligent advisor systems and transfer of knowledge. In Verbal-visual literacy: Understanding and applying new educational communication media technologies. Selected readings from the Symposium of the International Visual Literacy Association, Delphi, Greece, June 25-29, 1993. (ERIC Document Reproduction Services No. ED 393 425)

Boulet, M. M. (1994). Providing just-in time help to case tool users. Journal of Computing in Higher Education, 6(1), 44-62.

Boulet, M. M., Barbeau, L., & Slobodrian, S. (1990). Advisor system: Conception of an intervention module. Computers and Education, 14(1), 17-29.

Bransford, J. D., Franks, J. J., Vye, N. J., & Sherwood, R. D. (1989). New approaches to instruction: Because wisdom can’t be told. In S. Vosniadou & A. Ortany (Eds.), Similarity and analogical reasoning (pp. 470-497). New York: Cambridge University Press.

Bransford, J., Sherwood, R., Vye, N., & Rieser, J. (1986). Teaching thinking and problem solving. American Psychologist, 41(10), 1078-1089.

Bransford, J. D., & Stein, B. S. (1993). The ideal problem solver: A guide for improving thinking, learning, and creativity, (2nd ed.). New York: W. H. Freeman.

Brown, A. L. (1989). Analogical learning and transfer: What develops? In S. Vosniadou & A. Ortony (Eds.), Similarity and analogical reasoning (pp. 369-413). Cambridge, MA: Cambridge University Press.

Brown, J. S., Collins, A., & Duguid, P. (1989).Situated cognition and the culture of learning. Educational Researcher, 18, 32-42.

Carrier, C. (1984). Do learners make good choices? Instructional Innovator, 29(2), 15-17, 48.

Cartmill, C. (1994). Cognitive performance in women as influenced by gender-role identity, competition conditions, and task difficulty (Doctoral dissertation, California School of Professional Psychology, 1994). Dissertation Abstracts International, 55(02), 627B.

Clariana, R. B. (1989). Computer simulations of laboratory experiences. Journal of Computers in Mathematics and Science Teaching, 8(2), 14-19.

Cognition and Technology Group at Vanderbilt. (1992a). An anchored instruction approach to cognitive skill acquisition and intelligent tutoring. In J. W. Regian and V. J. Shute (Eds.), Cognitive approaches to automated instruction (pp. 135-170). Hillsdale, NJ: Erlbaum.

Cognition and Technology Group at Vanderbilt. (1992b). The Jasper experiment: An exploration of issues in learning and instructional design. Educational Technology Research and Development, 40(1), 65-80.

Cognition and Technology Group at Vanderbilt. (1993). Anchored instruction and situated cognition revisited. Educational Technology, 33(3), 52-70.

Craig, G. J. (1967). The effect of distribution of rewards, grade level, and achievement level on small group applicational problem solving (Doctoral dissertation, University of Massachusetts, 1967). Dissertation Abstracts International, 28(09), 3894B. (University Microfilms, No. 68-2755)

Dalton, D. W., Hannafin, M. J., & Hooper, S. (1989). Effects of individual and cooperative computer-assisted instruction on student performance and attitudes. Educational Technology Research and Development, 37(2), 15-24.

Dempsey, J. V., & Van Eck, R. N. (1998, April). Advisor use in CBT: Modality and placement. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.

Doolittle, J. H. (1995). Using riddles and interactive computer games to teach problem-solving skills. Teaching of Psychology, 22(1), 33-36.

Fingar, P. (1999, April). Intelligent agents: The key to open ecommerce. Component Strategies, 29-35.

Fisher, M. D., Blackwell, L. R., Garcia, A. B., & Greene, J. C. (1975). Effects of student control and choice on engagement in a CAI arithmetic task in a low-income school. Journal of Educational Psychology, 67(6), 776-783.

Gagné, R. M. (1965). The conditions of learning. New York: Holt, Rinehart, and Winston.

Gagné, R. M., Briggs, L. J., & Wager, W. W. (1992). Principles of instructional design (4th ed.). Fort Worth, TX: Harcourt Brace Jovanovich.

Gick, M. L., & Holyoak, K. J. (1980). Analogical problem solving. Cognitive Psychology, 12, 306-355.

Hayes, J. R., & Simon, H. A. (1977). Psychological differences among problem isomorphs. In N. J. Castellan, Jr., D. B. Pisoni, & G. R. Potts (Eds.), Cognitive theory (Vol. 2, pp. 21-41). Hillsdale, NJ: Erlbaum.

Hurlock, E. B. (1927). The use of group rivalry as an incentive. Journal of Abnormal Social Psychology, 22, 278-90.

Johansen, K. J., & Tennyson, R. D. (1983). Effect of adaptive advisement on perception in learner-controlled, computer-based instruction using a rule-learning task. Educational Communications and Technology Journal, 31(4), 226-236.

Julian, J. W., & Perry, F. A. (1967). Cooperation contrasted with intra-group and inter-group competition. Sociometry, 30, 79-90.

Keefer, J. A., & Karabenick, S. A. (1998). Help seeking in the information age. In Strategic help seeking: Implications for learning and teaching (pp. 219-250). Mahwah, NJ: Erlbaum.

Keller, J. K. (1990). Characteristics of Logo instruction promoting transfer of learning: A research review. Journal of Research on Computing in Education, 23(1), 55-71.

Kraft Miller, J. L. (1981). Competition, maturation rate, and motive to avoid success in mathematics as predictors of performance on a spatial task (Doctoral dissertation, University of Washington, 1981). Dissertation Abstracts International, 42(06), 2602B.

Larkin, J. H. (1989). What kind of knowledge transfers? In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser (pp. 283-305). Hillsdale, NJ: Erlbaum.

Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge, MA: Cambridge University Press.

Lockhart, R. S., Lamon, M., & Gick, M. L. (1987). Contextual transfer in simple insight problems. Memory and Cognition, 16(1), 36-44.

Park, O., & Tennyson, R. D. (1980). Adaptive design strategies for selecting number and presentation order of examples in coordinate concept acquisition. Journal of Educational Psychology, 73, 745-753.

Perfetto, G. A., Bransford, J. D., & Franks, J. J. (1983). Constraints on access in a problem solving context. Memory and Cognition, 11(1), 24-31.

Perkins, D. N., & Salomon, G. (1989). Are cognitive skills context-bound? Educational Researcher, 18(1), 16-25.

Peters, C. L. (1988). The effects of advisement, content mapping, and interactive video on learner control and achievement in computer-based instruction (Doctoral dissertation, University of Georgia, 1988). Dissertation Abstracts International, 50(2), 348A.

Reed, S. K., Ernst, G. W., & Banerji, R. (1974). The role of analogy in transfer between similar problem states. Cognitive Psychology, 6, 436-450.

Reid, J. (1992). The effects of cooperative learning with intergroup competition on the math achievement of seventh grade students. (Research Report). (ERIC Document Reproduction Service No. ED 355 106).

Reinking, D. P. (1983). The effects of computer mediated text on measures of reading comprehension and reading behavior (Doctoral dissertation, University of Minnesota, 1983). Dissertation Abstracts International, 44(11), 3339A.

Royer, J. M. (1979). Theories of the transfer of learning. Educational Psychologist, 14, 53-69.

Sherwood, R. D., & Cognition and Technology Group at Vanderbilt. (1991, April). The development and preliminary evaluation of anchored instruction environments for developing mathematical and scientific thinking. Paper presented at the annual meeting of the National Association for Research in Science Teaching, Lake Geneva, WI.

Simon, H. A., & Hayes, J. R. (1976). The understanding process: Problem isomorphs. Cognitive Psychology, 8(2), 165-190.

Spalt, S. K. (1988). The effects of academic competition upon school system support, teacher-sponsor operational behavior, secondary school curriculum and student performance on the American College Test (Doctoral Dissertation, Southern Illinois University at Carbondale, 1987. Dissertation Abstracts International, 48(11), 2801A.

Streitz, N. A. (1988). Mental models and metaphors: Implications for the design of adaptive user-system interfaces. In H. Mandl & A. Lesgold (Eds.), Learning issues for intelligent tutoring systems (pp. 164-186). New York: Springer-Verlag.

Tennyson, R .D. (1980a, April). Advisement and management strategies as design variables in computer-assisted instruction. A paper presented at the annual meeting of the American Educational Research Association, Boston, MA.

Tennyson, R.D. (1980b). Instructional control strategies and content structure as design variables in concept acquisition using computer-based instruction. Journal of Educational Psychology, 72(4), 525-532.

Tennyson, R. D. (1981). Use of adaptive information for advisement in learning concepts and rules using computer-assisted instruction. American Educational Research Journal, 18(4), 425-438.

Tennyson, R. D. (1984). Artificial intelligence methods in computer-based instructional design: The Minnesota Adaptive Instructional System. Journal of Instructional Development, 7(3), 17-22.

Tennyson, R. D., & Buttrey, T. (1980). Advisement and management strategies as design variables in computer-assisted instruction. Educational Communications and Technology, 28, 169-176.

Thompson, G. B. (1972). Effects of co-operation and competition on pupil learning. Educational Research, 15(1), 28-36.

Thurman, R. A. (1993).  Instructional simulation from a cognitive psychology viewpoint. Educational Technology Research and Development, 41(4), 75-89.

Van Haneghan, J. P. (1990). Third and fifth graders’ use of multiple standards of evaluation to detect errors in word problems. Journal of Educational Psychology, 82(2), 352-358.

Van Haneghan, J. P., Barron, L., Young, M. F., Williams, S. M., Vye, N. J., & Bransford, J. D. (1992). The Jasper series: An experiment with new ways to enhance mathematical thinking. In D. F. Halpern (Ed.), Enhancing thinking skills in the sciences and mathematics (pp. 15-38). Hillsdale, NJ: Erlbaum.

Weisberg, R., DiCamillo, M., & Phillips, D. (1978). Transferring old associations to new situations: A nonautomatic process. Journal of Verbal Learning and Verbal Behavior, 17, 219-228.

Wilkes, G. N. (1965). An experimental study of the effect of competition on the learning of a selected physical education activity skill (Doctoral dissertation, Peabody College for Teachers of Vanderbilt University, 1965). Dissertation Abstracts International, 27(04), 1198B.

Young, M. F. (1993). Instructional design for situated learning. Educational Technology Research and Development, 41(1), 43-58.



[1] Numbers and letters represent the numbering scheme developed for the NCTM standards. Not every component of every strand was represented in this study, which is why some letters do not appear in sequence (e.g., a, d, e instead of a, b, c, d, e).



ITFORUM PAPER #49 - The effect of competition and context of advisement in a simulation/simulation game to promote transfer of mathematics skills in middle school students by Richard Van Eck. Posted on ITFORUM January 13, 2001. The author retains all copyrights of this work. Used on ITFORUM by permission of the author.Visit the ITFORUM WWW Home Page at http://itforum.coe.uga.edu/home.html