Research Statement

My research interests focus on how to improve educational technology by combining intelligent cognitive tutors with educational games. Cognitive tutors use computational models of expertise to provide explicit feedback to learners when solving problems, while educational games provide imaginary worlds and minimal feedback about progress toward some goal (but not on how to achieve it). The challenge in combining a cognitive tutor with a game is to find the proper balance between the intrusive didactic feedback of the tutor, which we know is effective for learning, with the more open-ended game environment, which we know to be engaging.

To investigate these questions, I have constructed Policy World, a platform within which I can combine several variants of a cognitive tutor embedded in several variants of an educational game. In Policy World, students search for evidence in the form of newspaper reports that they then use to debate policy recommendations against a computer opponent. Policy World teaches students to make evidence-based recommendations by constructing causal diagrams, which helps them overcome a demonstrable bias to ignore evidence that contradicts their initial beliefs. A pure cognitive tutor version of Policy World corrects the student immediately after making a mistake, never allowing the student to fail the game, making sure the student collects all the credible evidence, ensuring that the student creates an accurate diagram, and so on. A more game-like version of Policy World allows the student to make many mistakes and even fail the game before offering explicit tutoring, but provides multiple subgoals and goals, all of which come with a "score" reward. Contrasting more tutor-like and more game-like versions of Policy World allows us to determine how different tutoring components can be used to maximize both learning and engagement.

Policy World provides a platform within which I can investigate several broad research questions including: (1) what principles optimize the benefits of tutors and games with respect to learning, (2) what pedagogical techniques help students become proficient at converting informally presented information (e.g., an algebra story problem or a policy brief) into a powerful formal representation (an algebra equation, or a causal diagram), and (3) what is the appropriate model of cognitive expertise in policy deliberation?

Cognitive games for teaching policy deliberation

My existing work has made several contributions toward scientifically-based instruction for policy deliberation by: (a) defining a cognitive framework for policy deliberation, (b) showing that although confirmation bias often prevents students from recognizing when evidence contradicts their policy beliefs, causal diagrams can help students to make evidence-based arguments, (c) developing intelligent cognitive tutoring software that teaches policy deliberation, and (d) showing how embedding this tutor into a game affects motivation while maintaining efficiency in learning.

A cognitive framework for deliberation

Engaged citizenship requires a host of skills including critical thinking, speaking and debating, and organizing. With respect to policy deliberation problems, critical thinking can be thought of as a process of posing questions, searching for information, comprehending information, synthesizing information into causal models of the policy domain, and deciding upon recommendations. This process can be augmented with diagrams to improve reasoning in the same way that equations improve reasoning about algebra problems. Much of my work focuses on identifying and developing educational technology to teach the scores of cognitive sub skills required for to perform each of the steps in this process.

Bias in search and analysis of evidence

Where does bias occur during the policy deliberation process? In Easterday, Aleven, Scheines & Carver (2009), undergraduates played a computer game in which they assumed the role of policy analysts who had to determine whether four different policies: reducing class size, increasing teacher qualifications, increased funding or providing vouchers, would increase school performance. Unbeknownst to the students, half the evidence mostly supported one of their policy beliefs, e.g., about class size, and half the evidence mostly undermined one of their policy beliefs, e.g. about teacher qualifications.

I found, unsurprisingly, that students' recommendations are biased by their prior beliefs, but not necessarily in the way predicted by the literature. Measurements of students' beliefs showed that students change their beliefs in response to particular reports in a roughly rational manner, increasing their confidence after reading a confirming report and decreasing their confidence after reading a disconfirming report. The problem is not so much that students discount disconfirming evidence as that they start with extremely high confidence in their initial beliefs. Furthermore, students do not maintain an accurate picture of the evidence read, so when asked to recall the evidence read, students answers merely rationalize their final position.

With respect to the theoretical framework, the study suggests that students do not have as much difficulty with search, comprehension and evaluation as they do with synthesis. More precisely, students' synthesized causal models do not include an accurate mental representation of the evidence making them susceptible to bias. This result suggests a possible need for external representations of the evidence.

Diagrams improve reasoning

Can diagrams improve reasoning? Easterday, Aleven, Scheines & Carver (in press) examined whether causal diagrams help students coordinate a more complex set of causal claims.

Students were given a set of causal claims from multiple sources about a policy problem and asked them to determine which policy should be pursued, assuming that a given set of sources are credible. Some students received the claims as text, while other students received the claims as text plus a causal diagram, and other students received the claims as text with a diagram tool with which to construct their own diagrams.

This study found that causal diagrams do indeed improve student's ability to make evidence-based policy recommendations and also that practice constructing diagrams improves students' evidence comprehension skills. In other words, causal diagrams can be used as an equations language for policy reasoning. This study also revealed that students had great difficulty constructing accurate diagrams themselves.

A follow-up protocol study uncovered ways in which confirmation bias affects students' and experts' use of diagrams by comparing how several experts and novices analyze causal evidence. I found that reasoners are heavily influenced by background knowledge in both normative ways, such as when experts overrule information based on superior prior knowledge, and non-normative ways, such as when novices reinterpret the diagrams in a biased way to support their prior beliefs.

Policy World: A Cognitive game for teaching deliberation

Taken together, the framework and experimental results show that confirmation bias does indeed hinder students' synthesis of evidence and that while diagrams can help students coordinate multiple policy claims, diagram construction proves difficult. These findings suggest that a computer tutor for policy reasoning should focus on the construction of external representations to synthesize evidence.

I'm currently developing and testing a cognitive game (intelligent tutor + video game) to teach students how to deliberate, that is, use evidence from multiple conflicting sources to make policy decisions, primarily by constructing causal diagrams.

Future research

Much of my graduate work focused on building the Policy World platform, which integrates a diagramming tool, cognitive tutor and an educational game for studying policy deliberation. With this platform in hand, I am now in a position to carry out a number of studies over the next 5 years. For example, to examine just one step of the cognitive model of deliberation, say comprehension, one could substitute the newspaper like-articles about empirical evidence that are currently used with different types of raw information, i.e., one could insert more realistic "talking-head" debates that occur on the news that include more complex rhetoric and moral arguments; one could also insert teachable-agents that the student must interview and persuade in order to teach students how to work with community organizations; one could insert historical narratives to explore causal reasoning in history; one could allow the student to collect "data" to teach how science affects policy, and so on. To investigate my second broad research question on constructing external representations, the same technologies used in Policy World to teach causal diagramming skills can be applied to other domains such as argument mapping (which is currently taught in CMU philosophy classes using my iLogos software, but without tutoring or game elements). Finally, to address the primary research question, Policy World allows me to develop design principles for combining tutors and games by testing how different sets of tutoring and game features affect learning and motivation. My current work only scratches the surface of the questions that Policy World allows us to explore.